JP2023012795A - Training device, abnormal behavior assessment device, method, and program - Google Patents

Abstract

To assess an abnormal behavior with good accuracy that does not conform to procedures.SOLUTION: A procedural sentence database 24 stores a plurality of procedural sentences for each of a series of procedure that represent at least one action included in the procedure, or a vector that represents each of the plurality of procedural sentences. A label detection unit 28 detects an action label related to a human action from video data or audio data for training that represents the actions included in the procedure. A model learning unit 30 learns a language model that generates a sentence or a sentence vector from the action label, so as to generate the procedural sentences preliminarily determined for the video data or audio data for learning, or the sentence vectors that represent the procedural sentences.SELECTED DRAWING: Figure 2

Description

The disclosed technology relates to a learning device, an abnormal behavior determination device, a method, and a program.

In recent years, with the spread of high-definition cameras, there is a growing need for technology that analyzes human behavior using video and audio captured by cameras. For example, detection of criminal behavior with a surveillance camera, detection of dangerous behavior at a construction site, and the like. Discovering these behaviors requires observing large amounts of video and audio. Furthermore, it is necessary for a person who understands the definition of abnormal behavior to observe the behavior in the video and detect the abnormal behavior. However, since manual detection is time-consuming and labor-intensive, a method of detecting abnormal behavior by constructing an algorithm that automatically detects it is conceivable.

In recent years, an abnormal behavior detection technique using a neural network has been proposed (Non-Patent Document 1). In the method of Non-Patent Document 1, abnormal behavior is detected with high accuracy by clustering videos.

Zaheer M.Z., Mahmood A., Astrid M., Lee SI. CLAWS: Clustering Assisted Weakly Supervised Learning with Normalcy Suppression for Anomalous Event Detection. ECCV 2020.

In the conventional method for detecting abnormal behavior in images, which is shown in Non-Patent Document 1, procedures and actions are not clearly distinguished. Therefore, for example, if there is a series of steps such as (step 1) standing a stepladder on the floor, (step 2) tightening the safety belt, and (step 3) climbing the stepladder, many actions are required in each step. Therefore, it is difficult to judge whether the order of actions included in the procedure is correct. Specifically, (procedure 1) includes a number of actions in which a person bends his knees, grabs a stepladder, lifts the stepladder, and fixes it. Similarly, the procedure (Procedure 2) of tightening the safety belt and climbing the stepladder also includes a series of actions of holding the safety belt and fixing it to the human body. (Procedure 3) includes a number of actions of walking to a stepladder, placing the foot on the step, and climbing up while holding the stepladder in the hand. In this way, it is necessary to group behaviors to some extent as a procedure and check whether the order of the behaviors included in the procedure is correct. It is the center, and there is no examination of abnormality determination as to whether or not a plurality of actions are different from the integrated procedure. Therefore, if the safety belt is tightened after climbing the stepladder, it is difficult to determine from the video an abnormal behavior such as dangerous behavior, unlike the procedure.

The disclosed technology has been made in view of the above points, and aims to provide a learning device, an abnormal behavior determination device, a method, and a program that can accurately determine abnormal behavior that differs from a procedure. do.

A first aspect of the present disclosure is a learning device, for each of a series of procedures, a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences a label detection unit that detects action labels related to human actions from a procedure sentence database that stores actions included in the procedure, video data or audio data for learning representing actions included in the procedure; and a model learning unit that learns a language model for generating a sentence or a sentence vector from the action label so as to generate the predetermined procedural sentence or a sentence vector representing the procedural sentence.

A second aspect of the present disclosure is an abnormal behavior determination device, for each of a series of procedures, representing a plurality of procedural sentences representing at least one action included in the procedure, or each of the plurality of procedural sentences Based on a procedural sentence database storing sentence vectors and an action label related to human action detected from video data or audio data representing human action, a sentence or sentence vector is generated from the action label. a generation unit that generates the sentence or sentence vector using a language model; the generated sentence or sentence vector; the procedural sentence stored in the procedural sentence database; or a sentence vector representing the procedural sentence; and an abnormality determination unit that determines whether or not the behavior of the person is abnormal based on the similarity calculated by the similarity calculation unit.

A third aspect of the present disclosure is a learning method, which includes, for each of a series of procedures, a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences A learning device that includes a procedural statement database that stores the action label for human action from learning video data or audio data representing actions included in the procedure, and stores the action label in the learning video data or audio data A language model for generating a sentence or a sentence vector from the action label is learned so as to generate the predetermined procedural sentence or a sentence vector representing the procedural sentence.

A fourth aspect of the present disclosure is a method for determining abnormal behavior, wherein for each of a series of procedures, a plurality of procedural sentences representing at least one behavior included in the procedure, or each of the plurality of procedural sentences An abnormal behavior determination device including a procedural sentence database that stores sentence vectors, based on an action label related to human behavior detected from video data or audio data representing human behavior, extracts a sentence or a sentence vector from the action label. Using the generated pre-trained language model to generate the sentence or sentence vector, representing the generated sentence or sentence vector and the procedural sentence stored in the procedural sentence database or the procedural sentence A degree of similarity with the sentence vector is calculated, and whether or not the person's behavior is abnormal is determined based on the calculated degree of similarity.

A fifth aspect of the present disclosure is a program for causing a computer to function as the learning device of the first aspect or the abnormal behavior determination device of the second aspect.

According to the disclosed technique, it is possible to accurately determine abnormal behavior that differs from the procedure.

1 is a schematic block diagram of an example of a computer functioning as a learning device and an abnormal behavior determination device according to the first embodiment and the second embodiment; FIG. 1 is a block diagram showing the configuration of a learning device according to a first embodiment; FIG. It is a block diagram showing the configuration of the abnormal behavior determination device of the first embodiment and the second embodiment. 6 is a flow chart showing a sentence vector generation processing routine of the learning device of the first embodiment; 4 is a flow chart showing a learning processing routine of the learning device of the first embodiment; 4 is a flow chart showing the flow of processing for updating the language model of the learning device of the first embodiment; 4 is a flowchart showing an abnormal behavior determination processing routine of the abnormal behavior determination device of the first embodiment; 4 is a flow chart showing the flow of processing for generating a sentence vector of the abnormal behavior determination device of the first embodiment; 4 is a flowchart showing the flow of processing for determining abnormal behavior by the abnormal behavior determination device of the first embodiment; 9 is a flow chart showing a learning processing routine of the learning device of the second embodiment; 10 is a flow chart showing the flow of processing for updating the language model of the learning device of the second embodiment; 9 is a flowchart showing an abnormal behavior determination processing routine of the abnormal behavior determination device of the second embodiment; It is a flowchart which shows the flow of the process which produces|generates a sentence of the abnormal behavior determination apparatus of 2nd Embodiment.

An example of embodiments of the technology disclosed herein will be described below with reference to the drawings. In each drawing, the same or equivalent components and portions are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may differ from the actual ratios.

[First embodiment]
<Overview of this embodiment>
In this embodiment, a group of action labels extracted from video or audio is input, and a sentence vector of the action label group is generated. For example, a group of human-object-interaction labels (HOI labels), which are combinations of human labels, object labels, and human action labels for objects extracted from video, is input to generate a sentence vector of action labels. do. Then, the sentence vector and the sentence vector of the procedural sentence to be compared are used as input to calculate the similarity between the sentence vectors. .

Here, a sentence vector of a procedure sentence is a sentence vector of a procedure sentence representing each of a series of procedures, and one procedure includes at least one action.

<Configuration of learning device according to the present embodiment>
FIG. 1 is a block diagram showing the hardware configuration of the learning device 10 of this embodiment.

As shown in FIG. 1, the learning device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface ( I/F) 17. Each component is communicatively connected to each other via a bus 19 .

The CPU 11 is a central processing unit that executes various programs and controls each section. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 performs control of each configuration and various arithmetic processing according to programs stored in the ROM 12 or the storage 14 . In this embodiment, the ROM 12 or storage 14 stores a learning program for learning a language model, which is a neural network. The learning program may be one program, or may be a program group composed of a plurality of programs or modules.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a work area. The storage 14 is configured by a HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various inputs.

The input unit 15 receives video data for learning as an input. Specifically, the input unit 15 receives video data for learning representing actions included in the procedure represented by the procedure sentence of the work manual. Information (for example, ID) indicating which procedural sentence in the work manual corresponds to the video data for learning is given.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may employ a touch panel system and function as the input unit 15 .

The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark), for example.

Next, the functional configuration of the learning device 10 will be described. FIG. 2 is a block diagram showing an example of the functional configuration of the learning device 10. As shown in FIG.

Functionally, as shown in FIG. 2, the learning device 10 includes a work manual database (DB) 20, a sentence vector generator 22, a procedural sentence database (DB) 24, a learning database (DB) 26, and a label detector. 28 and a model learning unit 30 .

The work manual database 20 stores a plurality of procedural sentences included in a work manual describing a series of procedures. Here, the procedure sentences are stored in the work manual database 20 as text data.

The sentence vector generation unit 22 extracts the procedural sentences one by one from the work manual database 20, and uses the pre-trained model to generate a sentence vector representing the procedural sentence.

Specifically, the sentence vector generation unit 22 prepares in advance a language model such as BERT (Bidirectional Encoder Representations from Transformers) trained with a large amount of text as a trained model, and generates a procedural sentence using the language model. Generate a sentence vector to represent This language model is used only for generating sentence vectors representing procedural sentences without performing any learning beyond pre-learning. The sentence vector generation unit 22 stores the generated sentence vectors in the procedural sentence database 24 .

The procedural sentence database 24 stores sentence vectors generated by the sentence vector generation unit 22 for each of a plurality of procedural sentences included in the work manual.

Here, it is assumed that sentence vectors are stored in the procedural sentence database 24 in a form that matches the granularity of sentence vectors generated by a language model, which will be described later.

The learning database 26 stores a plurality of input video data for learning. The video data for learning may be input for each video, may be input for each divided video segment, or may be input for each video frame. Here, the video segment is a unit obtained by dividing a video into a plurality of frames. For example, 32 frames are defined as one segment.

The label detection unit 28 detects action labels related to human actions from each of the plurality of video data for learning stored in the database 26 for learning. Specifically, the label detection unit 28 extracts video data for learning from the database 26 for learning, and outputs a group of action labels, which is a group of action labels detected within a certain period of time, which is the length of the video data. . An example of a group of activity labels is a group of HOI labels. A HOI label group is a set of HOI labels detected within a certain period of time.

For example, if person is person, ladder is ladder, and climbing is climb, the HOI label of <person, ladder, climb> is detected as the activity label. Depending on whether the verb of the action label included in the HOI label is intransitive or transitive, the presence or absence of the object label changes. In the example above, the action label is a transitive verb, but it can also be an intransitive verb.

The model learning unit 30 learns a language model for generating sentence vectors from action labels so as to generate sentence vectors representing procedural sentences predetermined for video data for learning.

Specifically, the model learning unit 30 receives action labels detected from the video data for learning from the label detection unit 28 . The model learning unit 30 also retrieves from the procedure sentence database 24 the procedure sentence vector of the correct procedure sentence associated with the video data for learning. The model learning unit 30 generates a sentence vector from the action label group detected from the video data for learning, using the language model to be learned. The model learning unit 30 compares and evaluates the procedural sentence vector of the correct procedural sentence and the generated sentence vector so that the procedural sentence vector of the correct procedural sentence and the generated sentence vector match each other. , update the language model and output the language model. Here, the language model is a neural network model or the like.

For example, as a base model for the language model to be learned, a language model that has undergone self-supervised learning with a large amount of text is used. A typical model is BERT in Non-Patent Document 2. For BERT, action labels are rearranged in the order <person label action label object label> and input. In other words, it is an arrangement considering the subject-predicate-object relationship, and in the above example, it is <person climb ladder>. Also, if the procedure "fasten the safety belt" precedes "the person climbs the ladder", the action label <person tight safety belt> can be similarly detected. safety belt person climb ladder>, one sentence vector can be obtained. Also, as an input method, <person tight safety belt> and <person climb ladder> may be divided and input as separate sentences. Specifically, a separator that can be recognized by the BERT model is inserted between the HOI labels and input. By doing so, it is possible to generate a sentence vector that considers a plurality of procedures.

[Non-Patent Document 2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL2019.

<Configuration of Abnormal Behavior Determining Device According to First Embodiment>
FIG. 1 above is a block diagram showing the hardware configuration of the abnormal behavior determination device 50 of the first embodiment.

As shown in FIG. 1, the abnormal behavior determination device 50 has the same configuration as the learning device 10, and the ROM 12 or the storage 14 stores an abnormal behavior determination program for determining abnormal behavior.

The input unit 15 receives as an input a behavior label group detected from video data representing human behavior. Specifically, the input unit 15 receives as an input an action label group, which is a group of HOI labels detected within a certain period of time, which is the length of video data.

Next, the functional configuration of the abnormal behavior determination device 50 will be described. FIG. 3 is a block diagram showing an example of the functional configuration of the abnormal behavior determination device 50. As shown in FIG.

The abnormal behavior determination device 50 functionally includes a procedural statement database (DB) 60, a generation unit 62, a similarity calculation unit 64, and an abnormality determination unit 66, as shown in FIG.

Similar to the procedure sentence database 24, the procedure sentence database 60 stores sentence vectors generated by the sentence vector generation unit 22 for each of a plurality of procedure sentences included in the work manual.

The generation unit 62 generates a sentence vector based on the input action label group and using a language model pre-learned by the learning device 10 .

The similarity calculator 64 calculates the similarity between the sentence vector generated by the generator 62 and the sentence vector representing the procedural sentence stored in the procedural sentence database 60 .

Based on the degree of similarity calculated by the degree-of-similarity calculation unit 64, the abnormality determination unit 66 determines whether or not the human behavior is abnormal unlike the procedure. For example, the abnormality determination unit 66 compares the calculated degree of similarity with a predetermined threshold to determine whether or not the behavior of a person is abnormal, and assigns a label with a value of 1 or 0. Output. Here, the label is 1 when abnormal, and the label is 0 when normal.

<Action of the learning device according to the first embodiment>
Next, operation of the learning device 10 according to the first embodiment will be described.

FIG. 4 is a flow chart showing the flow of sentence vector generation processing by the learning device 10 . The CPU 11 reads out the learning program from the ROM 12 or the storage 14, develops it in the RAM 13, and executes it, thereby performing sentence vector generation processing.

In step S100, the CPU 11, as the sentence vector generation unit 22, extracts the procedure sentences one by one from the work manual database 20 and inputs them.

In step S102, the CPU 11, as the sentence vector generation unit 22, inputs procedural sentences to the pre-trained model one by one to generate a sentence vector representing the procedural sentences. Here, a pre-trained model is a language model such as BERT that has been pre-trained only with a large amount of common texts.

In step S104, the CPU 1 stores the sentence vector generated for each procedural sentence in the procedural sentence database 24. FIG.

FIG. 5 is a flowchart showing the flow of learning processing by the learning device 10. As shown in FIG. The learning process is performed by the CPU 11 reading the learning program from the ROM 12 or the storage 14, developing it in the RAM 13, and executing it. Also, a plurality of video data for learning are input to the learning device 10 and stored in the learning database 26 .

In step S110, the CPU 11, as the label detection unit 28, extracts a plurality of video data for learning from the database 26 for learning and uses each of them as an input.

In step S112, the CPU 11, as the label detection unit 28, detects an action label group from each of the plurality of video data for learning.

In step S114, the CPU 11, as the model learning unit 30, acquires a group of action labels detected from each of the plurality of video data for learning, and inputs them.

In step S116, the CPU 11, as the model learning unit 30, retrieves from the procedure sentence database 24 the procedure sentence vector of the correct procedure sentence associated with each of the plurality of video data for learning. The CPU 11 generates a sentence vector from action labels detected from each of a plurality of video data for learning, using a language model to be learned. The CPU 11 compares and evaluates the procedural sentence vector of the correct procedural sentence and the generated sentence vector, and sets the language model so that the procedural sentence vector of the correct procedural sentence and the generated sentence vector match. to update.

In step S118, the CPU 11, as the model learning unit 30, outputs the updated language model. Here, the label detection unit 28 may detect the action label using a method such as Non-Patent Document 3 that can detect the HOI label from the video data. Any output unit such as video frame or video segment may be used.

[Non-Patent Document 3] Georgia Gkioxari, Ross Girshick, Piotr Dollar, Kaiming He. Detecting and Recognizing Human-Object Interactions. CVPR2018.

FIG. 6 shows detailed operations in the processing of steps S114 and S116.

In step S120, the CPU 11, as the model learning unit 30, acquires a group of action labels detected from each of the plurality of video data for learning, and inputs them.

In step S122, the CPU 11, as the model learning unit 30, rearranges each action label in the action label group in the order <person label action label object label>.

In step S124, the CPU 11, as the model learning unit 30, determines whether the action label group is longer than the upper limit of input length of the language model to be learned. If the action label group is longer than the upper limit of the input length of the language model to be learned, the process proceeds to step S126. On the other hand, when the action label group is equal to or less than the upper limit of the input length of the language model to be learned, the process proceeds to step S128.

In step S126, the CPU 11, as the model learning unit 30, aggregates the appearance frequency of each action label in the action label group, arranges each action label in order of appearance frequency, and performs The top N action labels are extracted, action labels other than the extracted action labels are deleted, and a new action label group is created.

In step S128, the CPU 11, as the model learning unit 30, inputs the rearranged action label group to the learning target language model to generate a sentence vector.

In step S130, the CPU 11, as the model learning unit 30, performs the following calculation between the generated sentence vector and the sentence vector of the correct procedural sentence linked to the learning video data acquired from the procedural sentence database 24: , to calculate the loss by the evaluation function.

Here, the evaluation function is represented by a distance function, and the square of the Euclidean distance, Contrastive Loss of Non-Patent Document 4, or the like can be used. It is not limited to these distance functions, and other distance functions may be used as long as they can express distance and are differentiable, in other words, a function that can calculate the gradient required for backpropagation in deep learning. .

[Non-Patent Document 4] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. CVPR2020.

In step S132, the CPU 11, as the model learning unit 30, calculates a gradient from the obtained loss and updates the parameters of the language model by back propagation.

At step S134, the CPU 11, as the model learning unit 30, outputs the updated language model.

<Action of Abnormal Behavior Determining Device According to First Embodiment>
Next, the operation of the abnormal behavior determination device 50 according to the first embodiment will be described.

FIG. 7 is a flowchart showing the flow of abnormal behavior determination processing by the abnormal behavior determination device 50. As shown in FIG. The CPU 11 reads out an abnormal behavior determination program from the ROM 12 or the storage 14, develops it in the RAM 13, and executes the abnormal behavior determination process. Also, a group of behavior labels detected from video data representing human behavior is input to the abnormal behavior determination device 50 . Here, it is assumed that the action label group has already been detected by the method described in Non-Patent Document 3.

In step S<b>140 , the CPU 11 inputs the input action label group to the generation unit 62 .

In step S142, the CPU 11, as the generation unit 62, generates a sentence vector of the action label group.

In step S<b>144 , the CPU 11 inputs the sentence vector of the action label group to the similarity calculation unit 64 .

In step S<b>146 , the CPU 11 extracts sentence vectors representing procedural sentences from the procedural sentence database 60 and inputs them to the similarity calculation unit 64 .

Here, the order of procedures may be taken into consideration when retrieving sentence vectors of procedure sentences from the procedure sentence database 60 . For example, the ID of the sentence vector of the procedural sentence judged to have the highest similarity with respect to the action label group of the previous video segment is stored, and T procedural sentences before and after the procedure of the relevant procedural sentence are represented. A sentence vector may be retrieved. This reduces the amount of calculation of the similarity between the procedural sentence and the sentence vector.

In step S148, the CPU 11, as the similarity calculation unit 64, calculates the degree of similarity between the sentence vector representing the procedural sentence and the sentence vector of the action label group for each sentence vector representing the extracted procedural sentence. For example, cosine similarity is calculated as an index of similarity. It should be noted that the index of similarity may not be cosine similarity as long as similarity between vectors can be compared.

In step S<b>150 , the CPU 11 , acting as the similarity calculation unit 64 , outputs the calculated similarity to the abnormality determination unit 66 .

In step S152, the CPU 11, as the abnormality determination unit 66, compares the similarity with a threshold value, determines whether or not the human behavior is abnormal unlike the procedure, outputs a label indicating the determination result, Terminate the action determination process.

FIG. 8 shows detailed operations in the processing of steps S140 and S142. The operation shown in FIG. 8 is repeatedly performed for the action label group for each video segment. Note that the segment length of the video segment depends on the application.

In step S<b>160 , the CPU 11 inputs the action label group to the generation unit 62 .

In step S162, the CPU 11, as the generation unit 62, rearranges each action label in the action label group in the order <person label action label object label>. For example, when there are two person labels in the action label group of one video segment, they are arranged as <person label 1, action label 1, object label 1, person label 2, action label 2, object label 2>. With regard to combinations of actions and objects, possible combinations of action labels and object labels are stored in advance as a table, and action labels are determined by referring to the table. In addition, image features and the like may be taken into consideration.

A sentence vector is obtained by inputting the rearranged behavior label group to a trained language model such as BERT. In this embodiment, the action label group is input by linking it in the form of <action label 1><action label 2> . Therefore, in step S164, the CPU 11, as the generation unit 62, determines whether or not the action label group is longer than the upper limit of the input length of the learned language model. If the action label group is longer than the upper limit of the input length of the learned language model, the process proceeds to step S166. On the other hand, when the action label group is equal to or less than the upper limit of the input length of the learned language model, the process proceeds to step S168.

In step S166, the CPU 11, as the generation unit 62, aggregates the appearance frequency of each action label in the action label group, arranges each action label in order of appearance frequency, and arranges each action label so as to fit within the input length upper limit of the learned language model. N action labels are extracted, action labels other than the extracted action labels are deleted, and a new action label group is created.

In step S168, the CPU 11, as the generation unit 62, inputs the rearranged action label group to the learned language model to generate a sentence vector.

At step S170, the CPU 11, as the generator 62, outputs the obtained sentence vector.

There are two methods for determining abnormal behavior in the processes of steps S150 and S152. In one method, when the degree of similarity between the procedural sentence and the sentence vector is low with respect to all the extracted procedural sentences, the action that does not correspond to any procedural sentence is determined to be an abnormal action.

The other method considers the order of steps. FIG. 9 shows detailed operations in the processing of steps S150 and S152 when the order of procedures is considered.

In step S<b>180 , the CPU 11 inputs the top M calculated similarities in descending order to the abnormality determination unit 66 .

In step S182, the CPU 11, as the abnormality determination unit 66, determines whether or not the input action label group belongs to the video segment extracted from the beginning of the video (hereinafter referred to as the leading segment). If the input action label group is for the leading segment of the video, the process proceeds to step S188. On the other hand, if the input action label group is not the leading segment of the video, the process proceeds to step S184.

In step S184, the CPU 11, as the abnormality determination unit 66, compares the sentence vector of the procedural sentence with the highest similarity with the previous video segment among the M input similarities with the same sentence vector. Determine if there is a degree If there is a similarity compared with the same sentence vector as the sentence vector of the procedural sentence with the highest similarity for the previous video segment among the M similarities that have been input, the process proceeds to step S186. On the other hand, if there is no similarity compared with the same sentence vector as the sentence vector of the procedural sentence with the highest similarity for the previous video segment among the M similarities that have been input, the process proceeds to step S188. do.

In step S186, the CPU 11, as the abnormality determination unit 66, compares the sentence vector of the procedural sentence with the highest similarity with respect to the previous video segment among the M input similarities with the same sentence vector. is higher than the same similarity for the previous video segment. Among the M input similarities, the sentence vector of the procedural sentence with the highest similarity for the previous video segment and the same sentence vector have the same similarity for the previous video segment. If it is higher than the degree, the process proceeds to step S192. On the other hand, among the M input similarities, the similarity when compared with the same sentence vector as the sentence vector of the procedural sentence with the highest similarity for the previous video segment is If the similarities are less than or equal to the same degree, the process proceeds to step S188.

In step S188, it is determined whether or not the highest similarity among the input M similarities is lower than a threshold. If the highest similarity among the M similarities that are input is lower than the threshold, the process proceeds to step S190. On the other hand, if the highest similarity among the M similarities that are input is greater than or equal to the threshold, the process proceeds to step S192.

In step S190, the CPU 11, as the abnormality determination unit 66, outputs a label indicating that the human behavior is abnormal. In step S192, the CPU 11, as the abnormality determination unit 66, outputs a label indicating that the behavior of the person is normal.

As described above, the learning apparatus according to the first embodiment detects action labels related to human actions from learning video data representing actions included in a procedure, and predetermines action labels for the learning video data. We learn a language model that generates sentence vectors from action labels so as to generate sentence vectors that represent procedural sentences. As a result, it is possible to learn a language model for accurately determining abnormal behavior that differs from procedures.

Further, the abnormal behavior determination device according to the first embodiment includes a pre-learned language model that generates a sentence vector from the behavior label based on the behavior label related to the behavior of the person detected from the video data representing the behavior of the person. is used to generate a sentence vector, the similarity between the generated sentence vector and the sentence vector representing the procedural sentence stored in the procedural sentence database is calculated, and based on the calculated similarity, human behavior is abnormal. As a result, abnormal behavior different from the procedure can be determined with high accuracy in the work with the procedure photographed in the video.

[Second embodiment]
Next, a second embodiment will be described. Parts having the same configuration as in the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted.

In the second embodiment, instead of using the similarity of sentence vectors, a procedural sentence is generated from an action label group, the similarity of the procedural sentence is calculated, and whether or not a person's action is different from the procedure is abnormal. It is different from the first embodiment in that whether or not is determined.

<Overview of Second Embodiment>
In the second embodiment, a sentence is generated with an action label group as an input, a similarity is calculated with the generated sentence and a procedure sentence as an input, and whether or not a person's behavior is abnormal is determined with the calculated similarity as an input. judge.

<Structure of Learning Apparatus According to Second Embodiment>
The learning device of the second embodiment is the same as the learning device 10 of the first embodiment except that the sentence vector generation unit 22 is not required.

The procedure sentence database 24 of the learning device 10 of the second embodiment stores each of a plurality of procedure sentences included in the work manual.

The model learning unit 30 learns a language model that generates sentences from action labels so as to generate predetermined procedural sentences for video data for learning.

Specifically, the model learning unit 30 receives action labels detected from the video data for learning from the label detection unit 28 . In addition, the model learning unit 30 extracts a correct procedural sentence associated with the video data for learning from the procedural sentence database 24 . The model learning unit 30 generates sentences from the action label group detected from the video data for learning, using the language model to be learned. For example, the generated sentence is represented by the occurrence probability of each word. The model learning unit 30 compares and evaluates the procedural sentence that is the correct answer and the generated sentence, and updates the language model so that the procedural sentence that is the correct answer matches the generated sentence. Output. Here, the language model is, for example, an Encoder-Decoder type neural network model used for translation, as in Non-Patent Document 5.

[Non-Patent Document 5] Jinhua Zhu and Yingce Xia and Lijun Wu and Di He and Tao Qin and Wengang Zhou and Houqiang Li and Tieyan Liu. Incorporating BERT into Neural Machine Translation. ICLR2020.

<Configuration of Abnormal Behavior Determining Device According to Second Embodiment>
Since the abnormal behavior determination device of the second embodiment is the same as the abnormal behavior determination device 50 of the first embodiment, the same reference numerals are assigned and the description thereof is omitted.

Like the procedure sentence database 24, the procedure sentence database 60 stores each of a plurality of procedure sentences included in the work manual.

The generation unit 62 generates a sentence based on the input action label group using a language model pre-learned by the learning device 10 .

The similarity calculator 64 calculates the similarity between the sentence generated by the generator 62 and the procedural sentences stored in the procedural sentence database 60 .

Based on the degree of similarity calculated by the degree-of-similarity calculation unit 64, the abnormality determination unit 66 determines whether or not the human behavior is abnormal unlike the procedure.

<Action of the learning device according to the second embodiment>
Next, the operation of the learning device 10 according to the second embodiment will be described. The same reference numerals are given to the same processing as in the first embodiment, and detailed description thereof will be omitted.

FIG. 10 is a flowchart showing the flow of learning processing by the learning device 10. As shown in FIG. The learning process is performed by the CPU 11 reading the learning program from the ROM 12 or the storage 14, developing it in the RAM 13, and executing it. Also, a plurality of video data for learning are input to the learning device 10 and stored in the learning database 26 .

In step S110, the CPU 11, as the label detection unit 28, extracts a plurality of video data for learning from the database 26 for learning and uses each of them as an input.

In step S112, the CPU 11, as the label detection unit 28, detects an action label group from each of the plurality of video data for learning.

In step S114, the CPU 11, as the model learning unit 30, acquires a group of action labels detected from each of the plurality of video data for learning, and inputs them.

In step S200, the CPU 11, as the model learning unit 30, retrieves from the procedure sentence database 24 the correct procedure sentence associated with each of the plurality of video data for learning. The CPU 11 generates sentences from action labels detected from each of a plurality of video data for learning, using a language model to be learned. The CPU 11 compares and evaluates the correct procedure sentence and the generated sentence, and updates the language model so that the correct procedure sentence and the generated sentence match.

In step S118, the CPU 11, as the model learning unit 30, outputs the updated language model.

FIG. 11 shows detailed operations in the processing of steps S114 and S200.

In step S120, the CPU 11, as the model learning unit 30, acquires a group of action labels detected from each of the plurality of video data for learning, and inputs them.

In step S122, the CPU 11, as the model learning unit 30, rearranges each action label in the action label group in the order of <person label, action label, object label>.

In step S124, the CPU 11, as the model learning unit 30, determines whether the action label group is longer than the upper limit of input length of the language model to be learned. If the action label group is longer than the upper limit of the input length of the language model to be learned, the process proceeds to step S126. On the other hand, when the action label group is equal to or less than the upper limit of the input length of the language model to be learned, the process proceeds to step S210.

In step S126, the CPU 11, as the model learning unit 30, aggregates the appearance frequency of each action label in the action label group, arranges each action label in order of appearance frequency, and performs The top N action labels are extracted, action labels other than the extracted action labels are deleted, and a new action label group is created.

In step S210, the CPU 11, as the model learning unit 30, inputs the rearranged action label group to the learning target language model to generate a sentence.

In step S212, the CPU 11, as the model learning unit 30, compares the generated sentence with the correct procedural sentence linked to the learning video data acquired from the procedural sentence database 24 using an evaluation function. Calculate loss. Specifically, a loss is calculated by an evaluation function that evaluates the degree of difference between the appearance probability of each word represented by the generated sentence and each word of the correct procedural sentence. For example, with the evaluation function, the closer the appearance probability of each word in the correct procedural sentence is to 1, the lower the loss, and the closer to 0 the appearance probability of each word that is not included in the correct procedural sentence, the lower the loss. to calculate

In step S132, the CPU 11, as the model learning unit 30, calculates a gradient from the obtained loss and updates the parameters of the language model by back propagation.

At step S134, the CPU 11, as the model learning unit 30, outputs the updated language model.

<Action of Abnormal Behavior Determining Device According to Second Embodiment>
Next, the operation of the abnormal behavior determination device 50 according to the second embodiment will be described.

FIG. 12 is a flow chart showing the flow of abnormal behavior determination processing by the abnormal behavior determination device 50 . The CPU 11 reads out an abnormal behavior determination program from the ROM 12 or the storage 14, develops it in the RAM 13, and executes the abnormal behavior determination process. Also, a group of behavior labels detected from video data representing human behavior is input to the abnormal behavior determination device 50 . Here, it is assumed that the action label group has already been detected by the method described in Non-Patent Document 3.

In step S<b>220 , the CPU 11 inputs the input action label group to the generation unit 62 . Then, the CPU 11, as the generation unit 62, generates a sentence from the action label group.

In step S<b>222 , the CPU 11 inputs the generated sentence to the similarity calculation unit 64 .

In step S<b>224 , the CPU 11 extracts each procedural sentence from the procedural sentence database 60 and inputs it to the similarity calculation unit 64 . Here, the order of procedures may be taken into account when retrieving procedure statements from the procedure statement database 60 . For example, the ID of the procedural sentence judged to have the highest degree of similarity with respect to the action label group of the previous video segment is held, and T procedural sentences before and after the procedure of the relevant procedural sentence are taken out. good too. This can reduce the amount of calculation of the degree of similarity with the procedural sentence.

In step S226, the CPU 11, as the similarity calculation unit 64, calculates the similarity between each extracted procedural statement and the generated procedural statement. As a similarity index, BLEU in Non-Patent Document 6, ROUGE in Non-Patent Document 7, etc. used in machine translation may be used, and other indices that can measure similarity between sentences may be used. .

[Non-Patent Document 6] Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. BLEU: a method for automatic evaluation of machine translation. ACL2002.

[Non-Patent Document 7] Lin, Chin-Yew. ROUGE: A Package for Automatic Evaluation of summaries. ACL Workshop: Text Summarization Braches Out 2004.

In step S<b>150 , the CPU 11 , acting as the similarity calculation unit 64 , outputs the calculated similarity to the abnormality determination unit 66 .

In step S152, the CPU 11, as the abnormality determination unit 66, compares the similarity with a threshold value, determines whether or not the human behavior is abnormal unlike the procedure, outputs a label indicating the determination result, Terminate the action determination process.

FIG. 13 shows detailed operations in the process of step S200. The operation shown in FIG. 13 is repeatedly performed for the action label group for each video segment. Note that the segment length of the video segment depends on the application.

In step S<b>160 , the CPU 11 inputs the action label group to the generation unit 62 .

In step S162, the CPU 11, as the generation unit 62, rearranges each action label in the action label group in the order <person label action label object label>.

In step S164, the CPU 11, as the generation unit 62, determines whether or not the action label group is longer than the input length upper limit of the learned language model. If the action label group is longer than the upper limit of the input length of the learned language model, the process proceeds to step S166. On the other hand, when the action label group is equal to or less than the upper limit of the input length of the learned language model, the process proceeds to step S230.

In step S166, the CPU 11, as the generation unit 62, aggregates the appearance frequency of each action label in the action label group, arranges each action label in order of appearance frequency, and arranges each action label so as to fit within the input length upper limit of the learned language model. N action labels are extracted, action labels other than the extracted action labels are deleted, and a new action label group is formed.

In step S230, the CPU 11, as the generation unit 62, inputs the rearranged action label group into the learned language model to generate a sentence.

In step S232, the CPU 11, as the generator 62, outputs the obtained sentence.

As described above, the learning apparatus according to the second embodiment detects action labels related to human actions from learning video data representing actions included in a procedure, and predetermines action labels for learning video data. We learn a language model that generates sentences from action labels so as to generate ordered procedural sentences. As a result, it is possible to learn a language model for accurately determining abnormal behavior that differs from procedures.

Further, the abnormal behavior determination device according to the second embodiment uses a pre-learned language model for generating sentences from action labels based on action labels related to human actions detected from video data representing human actions. to generate a sentence, calculate the degree of similarity between the generated sentence and the procedural sentence stored in the procedural sentence database, and determine whether or not the human behavior is abnormal based on the calculated similarity judge. As a result, abnormal behavior different from the procedure can be determined with high accuracy in the work with the procedure photographed in the video.

<Modification>
The present invention is not limited to the above-described embodiments, and various modifications and applications are possible without departing from the gist of the present invention.

For example, action labels may be detected from audio data instead of video data. In this case, a combination of the action label detected from the voice data and the person label may be used as the action label. Also, an action label may be detected from a combination of video data and audio data. In this case, the action label may be a combination of the action label and person label detected from the audio data and the object label detected from the video data.

In addition, although the case where the learning device and the abnormal behavior determination device are configured as separate devices has been described as an example, the present invention is not limited to this, and the learning device and the abnormal behavior determination device are configured as one device. good too.

Further, the various processes executed by the CPU by reading the software (program) in each of the above-described embodiments may be executed by various processors other than the CPU. Processors in this case include GPUs (Graphics Processing Units), FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacturing, and specific circuits such as ASICs (Application Specific Integrated Circuits). A dedicated electric circuit or the like, which is a processor having a circuit configuration exclusively designed for executing the processing of , is exemplified. In addition, the learning process and the abnormal behavior determination process may be executed by one of these various processors, or a combination of two or more processors of the same or different types (for example, multiple FPGAs, CPUs and combination with FPGA, etc.). More specifically, the hardware structure of these various processors is an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in each of the above-described embodiments, a mode in which the learning program and the abnormal behavior determination program are stored (installed) in advance in the storage 14 has been described, but the present invention is not limited to this. The program is stored in non-transitory storage media such as CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), and USB (Universal Serial Bus) memory. may be provided in the form Also, the program may be downloaded from an external device via a network.

The following additional remarks are disclosed regarding the above embodiments.

(Appendix 1)
A learning device including a procedural sentence database storing a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences, for each of a series of procedures,
memory;
at least one processor connected to the memory;
including
The processor
Detecting action labels related to human actions from learning video data or audio data representing actions included in the procedure,
learning a language model for generating a sentence or a sentence vector from the action label so as to generate the procedural sentence predetermined for the video data or audio data for learning, or a sentence vector representing the procedural sentence; A learning device configured to:

(Appendix 2)
A procedural sentence database for storing a plurality of procedural sentences representing at least one action included in each of a series of procedures, or a sentence vector representing each of the plurality of procedural sentences, for each of a series of procedures so as to execute a learning process. A non-transitory storage medium storing a computer-executable program containing
The learning process includes
Detecting action labels related to human actions from learning video data or audio data representing actions included in the procedure,
learning a language model for generating a sentence or a sentence vector from the action label so as to generate the procedural sentence predetermined for the video data or audio data for learning, or a sentence vector representing the procedural sentence; Non-transitory storage media.

(Appendix 3)
An abnormal behavior determination device including a procedural sentence database that stores a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences, for each of a series of procedures. hand,
memory;
at least one processor connected to the memory;
including
The processor
Based on action labels related to human actions detected from video data or audio data representing human actions, using a pre-learned language model that generates sentences or sentence vectors from the action labels, the sentences or sentences generate a vector,
calculating a degree of similarity between the generated sentence or sentence vector and the procedural sentence stored in the procedural sentence database or a sentence vector representing the procedural sentence;
An abnormal behavior determination device configured to determine whether the person's behavior is abnormal based on the calculated similarity.

(Appendix 4)
A step of storing a plurality of procedural sentences representing at least one action included in said procedure, or a sentence vector representing each of said plurality of procedural sentences, for each of a series of procedures, so as to execute abnormal behavior determination processing. A non-transitory storage medium storing a computer-executable program containing a sentence database,
The abnormal behavior determination process includes:
Based on action labels related to human actions detected from video data or audio data representing human actions, using a pre-learned language model that generates sentences or sentence vectors from the action labels, the sentences or sentences generate a vector,
calculating a degree of similarity between the generated sentence or sentence vector and the procedural sentence stored in the procedural sentence database or a sentence vector representing the procedural sentence;
A non-temporary storage medium for determining whether or not the person's behavior is abnormal based on the calculated similarity.

10 learning device 11 CPU
14 Storage 20 Work manual database 22 Sentence vector generation unit 24 Procedure sentence database 26 Learning database 28 Label detection unit 30 Model learning unit 50 Abnormal behavior determination device 60 Procedure sentence database 62 Generation unit 64 Similarity calculation unit 66 Abnormality determination unit

Claims (8)

a procedural sentence database that stores a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences, for each of a series of procedures;
a label detection unit that detects an action label related to a person's action from video data or audio data for learning representing actions included in the procedure;
learning a language model for generating a sentence or a sentence vector from the action label so as to generate the procedural sentence predetermined for the video data or audio data for learning, or a sentence vector representing the procedural sentence; a model learning unit;
Learning device including. 2. The learning device according to claim 1, wherein the sentence vector representing the procedural sentence predetermined for the video data or audio data for learning is generated using a pre-trained model for generating sentence vectors. . When the label detection unit detects an action label group composed of a plurality of action labels, the model learning unit converts the sentence or the sentence vector generated by the language model from the concatenation of the action label group to: 3. The learning device according to claim 1, wherein the language model is learned so as to match the procedural sentence predetermined for the video data or audio data for learning, or a sentence vector representing the procedural sentence. . The model learning unit, when the action label group is detected by the label detection unit and when the concatenated action label group exceeds the input length upper limit of the language model, The sentence or sentence vector generated by the language model from the concatenation of the action labels extracted so that the input length is equal to or less than the upper limit is determined in advance for the video data or audio data for learning. 4. The learning device according to claim 3, wherein the language model is trained so as to match procedural sentences or sentence vectors representing the procedural sentences. a procedural sentence database that stores a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences, for each of a series of procedures;
Based on the behavior label related to human behavior detected from video data or audio data representing human behavior,
a generation unit that generates the sentence or sentence vector using a pre-learned language model that generates a sentence or sentence vector from the action label;
a similarity calculation unit that calculates a similarity between the generated sentence or sentence vector and the procedural sentence stored in the procedural sentence database or a sentence vector representing the procedural sentence;
an abnormality determination unit that determines whether the behavior of the person is abnormal based on the similarity calculated by the similarity calculation unit;
Abnormal behavior determination device including. a learning device including a procedural sentence database storing a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences, for each of a series of procedures;
Detecting action labels related to human actions from learning video data or audio data representing actions included in the procedure,
learning a language model for generating a sentence or a sentence vector from the action label so as to generate the procedural sentence predetermined for the video data or audio data for learning, or a sentence vector representing the procedural sentence; learning method. an abnormal behavior determination device including a procedural sentence database that stores a plurality of procedural sentences representing at least one action included in the procedure, or a sentence vector representing each of the plurality of procedural sentences, for each of a series of procedures;
Based on action labels related to human actions detected from video data or audio data representing human actions, using a pre-learned language model that generates sentences or sentence vectors from the action labels, the sentences or sentences generate a vector,
calculating a degree of similarity between the generated sentence or sentence vector and the procedural sentence stored in the procedural sentence database or a sentence vector representing the procedural sentence;
An abnormal behavior determination method, comprising: determining whether or not the person's behavior is abnormal based on the calculated degree of similarity. A program for causing a computer to function as the learning device according to any one of claims 1 to 4 or the abnormal behavior determination device according to claim 5.

Images