JP2020135747A - Action analysis device and action analysis method - Google Patents

Abstract

To provide an action analysis device that can more accurately recognize an action of a person included in an image than in a conventional art.SOLUTION: An action analysis device 1 that is the action analysis device to analyze an action of a person included in an image comprises: an original image acquisition section 11 that acquires a plurality of pieces of original image data 131 different in imaging time; a skeleton image generation section 12 that generates skeleton image data 132 on the person within each original image data; a behavior image generation section 13 that generates behavior image data 133 indicating time change of a skeleton of the person on the basis of each skeleton image data; and a model generation section 15 that generates a predetermined model enabling learning and inference of an action pattern of the person on the basis of each original image data, each skeleton image data and each behavior image data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a behavior analyzer and a behavior analysis method.

A technique for recognizing what kind of behavior a person in an image has performed is known (Patent Document 1). Patent Document 1 proposes a technique for recognizing a person's behavior by extracting and learning an image in which a change in behavior is detected from a time-series image in a moving image. There is also known a technique of determining the possibility of a driver taking a specific action such as using a mobile phone based on an image of the driver and outputting an alarm (Patent Document 2).

A technique for detecting a posture in a two-dimensional image by estimating the skeleton of a person in an image is also known (Non-Patent Document 1).

International Publication No. 2017/150211 Japanese Unexamined Patent Publication No. 2009-37534

Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields, CVPR20

In Patent Document 1, since the behavior of the target person is recognized based on the captured image (original image), if an object other than the target person (for example, the person's belongings, background, etc.) exists in the image, the object other than the target person is present. The object becomes noise. For example, when a person riding on a moving object such as a train or a vehicle is photographed, the scenery outside the vehicle changes, so that it becomes difficult to accurately recognize the behavior of the target person. Further, for example, even when the brightness around the target person changes drastically, the background of the target person fluctuates greatly, so that it is difficult to accurately recognize the behavior of the target person.

Further, in Patent Document 1 using only the original image, it is difficult to accurately recognize the characteristics of the target person and the movement of the target person. Further, in Patent Document 1 using only the original image, it is difficult to accurately capture the time change of the behavior of the target person. As described above, it is difficult to accurately recognize the behavior of the target person with the technique of Patent Document 1.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a behavior analysis device and a behavior analysis method capable of recognizing the behavior of a person included in an image with higher accuracy than before. There is.

In order to solve the above problem, the behavior analysis device according to one viewpoint of the present invention is a behavior analysis device that analyzes the behavior of a person included in an image, and is an original image that acquires a plurality of original image data having different shooting times. An acquisition unit, a skeletal image generation unit that generates skeletal image data of a person in each original image data, and a behavior image generation unit that generates behavior image data indicating a time change of a person's skeleton based on each skeletal image data. It has a model generation unit that generates a predetermined model capable of learning and inferring a person's behavior pattern based on each original image data, each skeleton image data, and each behavior image data.

According to the present invention, not only the original image data but also the skeleton image data and the behavior image data generated from the original image data are used to generate a predetermined model capable of learning and inferring the behavior pattern. It is possible to analyze the behavior of the person included in the above more accurately.

It is explanatory drawing which shows the whole structure of the action recognition device. It is a block diagram of the hardware and software of an action recognition device. It is explanatory drawing of the original image data. It is explanatory drawing of the skeleton image data. It is explanatory drawing of the behavior image data. This is a configuration example of correct answer data. This is a configuration example of sequence data for learning. This is a configuration example of model data. This is an example of sequence data for inference. It is a flowchart which shows the process of generating the skeleton image data. It is a flowchart which shows the process of generating the behavior image data. It is a flowchart which shows the process of generating sequence data. It is a flowchart which shows the process of generating model data. An example of configuring a neural network is shown. It is a flowchart which shows the inference process. FIG. 5 is an overall configuration diagram of the behavior recognition device according to the second embodiment. It is a flowchart which shows the behavior monitoring process. FIG. 3 is an overall configuration diagram of the behavior recognition device according to the third embodiment. It is a flowchart which shows the behavior monitoring process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The behavior analyzer according to the present embodiment uses not only the original image data including the person but also other data (skeleton image data, behavior image data) derived from the behavior of the person in the original image data to obtain the original image. Analyze the behavior of the person inside.

In the present embodiment, since other data derived from the behavior of the person is also used, the behavior of the person can be accurately analyzed even when the image (background, landscape) other than the person changes. Then, the behavior analysis device according to the present embodiment can be applied to a system for monitoring the behavior of various persons such as a driver, a passenger, a pedestrian, and a shopper.

That is, the present embodiment provides a device that recognizes human behavior in a time-series image group. In this embodiment, the behavior of a person shown in a moving image is recognized by using machine learning such as a neural network.

The behavioral analyzer according to the present embodiment reads an image group divided into frames from a moving image, applies a skeleton estimation technique, extracts a main human skeleton from the image, and images the image as a skeleton. Further, in the present embodiment, changes in movement between images such as optical flow are extracted from continuous skeleton images on the time axis and imaged as a behavior image.

In the present embodiment, the image groups (original image group, skeleton image group, behavior image group) are summarized as time-series sequence data, and the relationship between the input data and the behavior is achieved by a machine learning technique such as a neural network. To learn.

By including the skeleton image data and the behavior image data as the data to be input to machine learning, it is possible to suppress the influence of noise caused by objects other than humans and noise caused by changes in the background. Further, by using the posture information based on the human skeleton and the posture change information for machine learning, it is possible to recognize the behavior with higher accuracy than the conventional technique using only the original image data.

The behavior analysis device according to the present embodiment may be realized by using a computer including a processor and a storage device. The processor performs, for example, processing of the original image data and learning and inference by machine learning. The storage device stores, for example, each image data, each intermediate data, a machine learning model, and an inference result.

The processor calculates skeleton image data and behavior image data for each original image data extracted from the moving image data, for example. Next, the processor collects each original image data, each skeleton image data, and each behavior image data as time series data (sequence data). The processor learns the relationship between the time series data and the human behavior shown in the original image data by using machine learning such as a neural network, and calculates the model data. At the time of inference, the processor inputs the time series data into the model data to calculate the recognition result of the human behavior reflected in the original image data.

The first embodiment will be described with reference to FIGS. 1 to 15. It should be noted that the present embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention.

In the following description, "computer program" may be used as the subject. A computer program is executed by a processor to perform a defined process while using a memory and a communication port (communication control device). Therefore, a processor can be described as a subject instead of a computer program, or a computer having a processor can be described as a subject.

At least a part or all of the computer program may be realized by dedicated hardware. The computer program may be modular. The computer program may be fixedly distributed on a recording medium, or may be distributed from a program distribution server via a communication network. When the processor reads and executes the computer program, the functions 11 to 16 described later are realized.

FIG. 1 shows the overall configuration of the behavior recognition device 1 as a “behavior analysis device”. The action recognition device 1 recognizes what kind of action the person's action included in the original image data is.

The action recognition device 1 may include, for example, an original image acquisition unit 11, a skeleton image generation unit 12, a behavior image generation unit 13, a sequence data generation unit 14, a model generation unit 15, and an inference unit 16. it can.

The original image acquisition unit 11 acquires a plurality of original image data 131 having different shooting times. The original image acquisition unit 11 acquires image data 131 of the same subject having different shooting times from, for example, a moving image file, a still image file continuously shot, and the like. Since the skeleton image data 132 is created based on the image data 131 having different shooting times, it is called the original image data 131. The original image data may be stored in the action recognition device 1, or may be stored in an external storage device accessible to the action recognition device 1.

The skeleton image generation unit 12 estimates the skeleton of the person 1311 shown in the original image data 131 based on the original image data 131 acquired by the original image acquisition unit 11, and generates the estimated skeleton image data 132. To do.

The behavior image generation unit 13 generates behavior image data 133 showing the time change (movement direction, behavior) of the skeleton based on the skeleton image 132 generated by the skeleton image generation unit 12.

The sequence data generation unit 14 generates sequence data including each original image data 131, each skeleton image data 132, and each behavior image data 133.

The model generation unit 15 generates data of a predetermined model capable of learning and inferring a person's behavior pattern based on the learning sequence data (described later in FIG. 7) generated by the sequence data generation unit 14.

The inference unit 16 is based on the inference sequence data (described later in FIG. 9) created by the sequence data generation unit 14 and the model data generated by the model generation unit 15, and the behavior of the person included in the inference sequence data. Is recognized and the recognition result is output.

<Configuration of behavior recognition device>

FIG. 2 is a configuration example (functional block diagram) of the hardware and software of the action recognition device 1.

The action recognition device 1 includes, for example, a central processing unit 110, an input / output device 120, and a storage device 130.

The central processing unit 110 has a microprocessor and a program memory (both not shown), and performs necessary arithmetic processing and control processing for functioning as the action recognition device 1. The central processing unit 110 executes predetermined computer programs 111 to 116. Each computer program 111-116 corresponds to each function 11-16 described in FIG.

The original image acquisition program 111 is a computer program that acquires the original image data. The original image acquisition program 111, for example, reads the original image data 131 stored in the storage device 130 as data to be recognized (analyzed). The original image acquisition program 111 may read the original image data captured by the camera 140, or may read the original image data stored in the storage device 130. The original image acquisition program 111 may be realized as a function of the operating system or a function of the device driver or the like. Alternatively, the original image acquisition program 111 may be provided as a part of the skeleton image generation program 112.

The skeleton image generation program 112 is a computer program that recognizes parts of a human main part (for example, face, limbs, etc.) shown in the original image data 131, extracts them as a skeleton, and generates skeleton image data 132.

The behavior image generation program 113 is a computer program that extracts the behavior representing the movement of the skeleton based on the temporally continuous skeleton image data 132 and generates the behavior image data 133.

The sequence data generation program 114 is a computer program that aggregates time-series image data 131 to 133 representing a series of movements to generate sequence data 135 and 137.

The model generation program 115 is a computer program (learning program) that machine-learns the movement of a human being as a subject from sequence data and generates model data 136 as a "model".

The inference program 116 is a computer program that recognizes human movements in each sequence by inputting inference sequence data into the model.

The input / output device 120 is a device that inputs / outputs information to / from the user. The input / output device 120 includes an information output device 121 and an information input device 122. Examples of the information output device 121 include a display and a printer (both not shown). Examples of the information input device 122 include a keyboard, a mouse, a touch panel, a camera, and a scanner (all not shown). A device that also serves as both an information output device and an information input device may be used. It is also possible to have the moving image file captured by the external camera 140 input to either or both of the storage device 130 and the central processing unit 110.

The storage device 130 is, for example, a device that stores data to be processed by the central processing unit 110 and data after processing.

In the storage device 130, for example, the original image data 131, the skeleton image data 132, the behavior image data 133, the correct answer data 134, the learning sequence data 135, the model data 136, and the inference sequence data 137 are stored. It is stored.

As described above, the original image data 131 is an original image group obtained by dividing a moving image into, for example, an image unit for each frame. The skeleton image data 132 is image data obtained by extracting the main skeleton of a person based on the original image data 131. The behavior image data 133 is image data obtained by extracting information on what kind of change has occurred in the movement of a human being in a temporally continuous image based on the skeleton image data 132.

The correct answer data 134 is data indicating the correct answer of the human behavior pattern included in the original image data 131. The learning sequence data 135 is data processed as a set of time-series data based on the original image data 131, the skeleton image data 132, the behavior image data 133, and the correct answer data 134. The model data 136 is the data of the trained model obtained by machine learning the training sequence data 135. The inference sequence data 137 is data processed as a set of time series data based on the original image data 131, the skeleton image data 132, and the behavior image data 133.

At least a part or all of the above-mentioned computer program and data can be stored and distributed in a recording medium MM such as a flash memory device, a hard disk, a magnetic tape, or an optical disk. At least a portion of computer programs and data can also be distributed over communication networks.

<Original image data>

FIG. 3 shows an example of the original image data 131. The original image data 131 is, for example, data obtained by dividing moving image data (moving image file) into frames and storing it as a file.

3 (1) and 3 (2) show the original image data 131 (1) and 131 (2) having different shooting times. The original image data 131 (1) and 131 (2) include the target person 1311 (1) and 1311 (2) and the background 1312. FIG. 3 cites, for example, an image of a human being 1311 running across the front of a streetlight 1312. In this case, since the original image data 131 (1) and 131 (2) capture a moving person, the images of the human 1311 are different from each other. On the other hand, the background 1312 of the fixed street light or the like does not change between the original image data 131 (1) and 131 (2).

As shown in FIG. 3 (1), the file name F131 is automatically assigned to each original image data 131. In the example shown in FIG. 3, the file name of the original image data is generated by adding a numerical value indicating the temporal order to "raw" indicating that the image is the original image.

<Skeletal image data>

FIG. 4 shows an example of skeleton image data 132. The skeleton image data 132 (1) shown in FIG. 4 (1) is data generated from the skeleton information obtained from the original image data 131 (1) shown in FIG. 3 (1). The skeleton image data 132 (2) shown in FIG. 4 (2) is data generated from the skeleton information obtained from the original image data 131 (2) shown in FIG. 3 (2).

A person 1322 (1) having only a skeleton is generated from a person 1311 (1) included in the original image data 131 (1) of FIG. 3 (1). Similarly, a person 1322 (2) having only a skeleton is generated from a person 1311 (2) included in the original image data 131 (2) of FIG. 3 (2).

The skeletal image is a plot of major joints in major parts such as the human head and limbs with dots, and some dots connected by lines. Since the skeleton image data 132 does not include the background in the original image data 131, information that causes noise in recognizing human behavior is excluded. Furthermore, by extracting the skeletal information, the human posture can be recognized more clearly.

As shown in FIG. 4 (3), the file name F132 is automatically assigned to the skeleton image data 132 as well. In the example shown in FIG. 4, the file name of the skeleton image data is generated by adding a numerical value indicating the temporal order to "pose" indicating that the image is a skeleton image.

<Behavior image data>

FIG. 5 shows an example of the behavior image data 133. The behavior image data 133 is data obtained by extracting changes in the movement of the human skeleton as behavior information from images that are adjacent in time from each image of the skeleton image data 132, and storing the extracted behavior information as an image. ..

The behavior image data 133 (1) shown in FIG. 5 (1) has an optical flow from the skeleton image data 132 (1) shown in FIG. 4 (1) and the skeleton image data 132 (2) shown in FIG. 4 (2). It is the data extracted and the optical flow for each extracted pixel is represented by an arrow. Similarly, the behavior image data 133 (2) shown in FIG. 5 (2) is an optical flow from the skeleton image data 132 (2) shown in FIG. 4 (2) and the next skeleton image data (not shown) in terms of time. It is the data generated by extracting.

As shown in FIG. 5 (3), the file name F133 is automatically assigned to the behavior image data 133 as well. In the example shown in FIG. 5, the file name of the behavior image data is generated by adding a numerical value indicating the temporal order to "flow" indicating that the behavior image is used.

<Correct answer data>

FIG. 6 shows an example of correct answer data 134. The correct answer data 134 is composed of the file name 1341 of the original image data 131 and the correct answer 1342.

The correct answer 1342 is an identifier (ID) that classifies human behavior in the image. In the correct answer 1342, any classification can be defined, for example, "0" for a walking person, "1" for a running person, and "2" for a sitting person. In the example of FIG. 6, each file 1341 is given an ID "1" indicating a running person.

Other than these, define behavioral patterns such as jumping, crouching, standing up, sitting, lifting something, trying to put something down, An identifier may be assigned to the behavior pattern.

<Sequence data for learning>

FIG. 7 shows an example of the learning sequence data 135. The learning sequence data 135 is generated based on a series of time series data (original image data 131, skeleton image data 132, behavior image data 133). Model data that recognizes human behavior is generated by performing machine learning using the learning sequence data 135.

The training sequence data 135 includes, for example, a sequence identifier 1351 (sid in the figure), a temporal sequence identifier 1352 (tid in the figure), an original image data file name 1353, and a skeleton image data file name 1354. , The behavior image data file name 1355 and the classification class (classification result) 1356 are provided. The identifiers 1351 and 1352 may be unique within the action recognition device 1.

In the example of FIG. 7, the number of the temporal sequence identifier 1352 is "3". The number of the identifier 1352 may be a number other than "3". In the example of FIG. 7, the skeleton image data and the behavior image data are image data generated from the original image data at the same time, respectively.

<Model data>

FIG. 8 shows an example of model data 136. The model data 136 includes, for example, a data type 1361, a data item 1362, and a value 1363.

The data type 1361 includes setting data 13611 of the model obtained by machine learning and the trained model 13612. The configuration data 13611 and the trained model 13612 each include a data item 1362 and its value 1363.

The data item 1362 of the setting data 13611 includes, for example, the original image shape 1362A, the skeleton image shape 1362B, the behavior image shape 1362C, the output shape 1362D, and the processing contents 1362E and 1362F of each layer.

The original image shape 1362A represents the structure of the original image data 131. For example, (256,256,3) is set in the value 1363 of the original image shape 1362A. This means that it has 256 pixels in height, 256 pixels in width, and 3 channels (usually RGB).

The same applies to the skeleton image shape 1362B and the behavior image shape 1362C. That is, the skeleton image shape 1362B shows the structure of the skeleton image data 132. The behavior image shape 1362C shows the structure of the behavior image data 133. Since the behavior image data is a gray image, it has one channel.

(10) is set in the value 1363 of the output shop 1362D. This means that there are 10 types of behavior patterns.

When the machine learning algorithm is a neural network, the processing content of each layer is represented as the first layer processing 1362E and the second layer processing 1362F. In addition, various settings related to the processing content are described in the setting data 13611.

The trained model 13612 stores model parameters for obtaining a human behavior recognition result (behavior identifier) from sequence data. The value 1363 of the trained model 13612 is not stored until the learning process by machine learning is performed. After the learning process is performed, the automatically calculated value is stored in the value 1363. The model parameters do not need to be known by the user and are automatically used in the computer program when the machine learning model is called.

<Sequence data for inference>

FIG. 9 shows an example of inference sequence data 137. The inference sequence data 137 is data used for discriminating human behavior included in the image, and is input to the model described in FIG.

The inference sequence data 137 is similar to the learning sequence data 135 described with reference to FIG. 7, for example, a sequence identifier 1371, a temporal sequence identifier 1372, an original image data file name 1373, and a skeleton image data file name. It includes 1374, a behavior image data file name 1375, and a classification class 1376. In the classification class 1376, an identifier that is a determination result of the behavior pattern is stored after the inference processing.

<Outline of processing in the behavior recognition device>

The processing outline of the action recognition device 1 will be described. The central processing unit 110 reads the original image data 131 from the storage device 130 by using the original image acquisition program 111 called by the skeleton image generation program 112. Subsequently, the central processing unit 110 generates the skeleton image data 132 from the original image data 131 by using the skeleton image generation program 112, and stores the generated skeleton image data 132 in the storage device 130. Next, the central processing unit 110 executes the behavior image generation program 113, reads the skeleton image data 132 from the storage device 130, and generates the behavior image data 133 from the skeleton image data 132. The central processing unit 110 stores the generated behavior image data 133 in the storage device 130.

The central processing unit 110 executes the sequence data generation program 114. The central processing unit 110 reads the original image data 131, the skeleton image data 132, the behavior image data 133, and the correct answer data 134 from the storage device 130, and generates the learning sequence data 135. The central processing unit 110 stores the generated learning sequence data 135 in the storage device 130.

The central processing unit 110 executes the model generation program 115. The central processing unit 110 reads the learning sequence data 135 and the model data 136 from the storage device 130, performs machine learning, and obtains the model data 136. The central processing unit 110 overwrites and stores the newly generated model data 136 in the storage device 130.

The central processing unit 110 executes the inference program 116. The central processing unit 110 reads the model data 136 and the inference sequence data 137 from the storage device 130, and obtains the recognition result (behavior classification class) in each sequence. The central processing unit 110 overwrites and saves the generated inference sequence data 137 in the storage device 130. Each process will be described in detail below.

<Skeletal estimation processing>

FIG. 10 is a flowchart showing a skeleton image data generation process executed by the skeleton image generation program 112. The operating subject here is the skeleton image generation program 112 executed by the central processing unit 110. In the skeleton image king data generation process, the coordinates of the human skeleton reflected in each original image data are estimated from the original image data group as shown in FIG. 3 and drawn as a skeleton image.

The skeleton image generation program 112 reads the original image data 131 from the storage device 130 (S21). Hereinafter, it is assumed that the original image data 131 described in FIG. 3 is read into the central processing unit 110.

The skeleton image generation program 112 calculates the skeleton coordinates of the human subject from each original image data 131, and generates the skeleton image data 132 (S22).

There are various methods for obtaining human skeleton coordinates, and for example, the method described in Non-Patent Document 1 may be used. In this method, the position of each part of the human body and the characteristics of the relationship between each part of the human body shown in the image are extracted, and the skeletal coordinates of each part of the human body are obtained.

The part of the skeleton to be calculated can be changed according to the application. In this embodiment, for example, when acquiring a total of 14 points of nose, neck, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right waist, left waist, right knee, left knee, right foot, left foot. Will be explained. After extracting the coordinates of each part to be calculated, the extracted skeleton coordinate group is plotted, and the skeleton image data 132 is generated by connecting some of the plotted skeleton coordinate groups with a straight line. ..

For example, when the original image data 131 (1) and (2) shown in FIGS. 3 (1) and 3 (2) are calculated based on the skeleton coordinates after extraction, the results are shown in FIGS. 4 (1) and (2). The skeleton image data 132 (1) and 132 (2) shown are generated. The background of the skeleton image data 132 may be, for example, white or black.

Finally, the skeleton image generation program 112 stores the skeleton image data 132 generated in step S22 in the storage device 130 (S23).

<Behavior extraction process>

FIG. 11 is a flowchart showing a behavior image data generation process executed by the behavior image generation program 113. The operating subject here is the behavior image generation program 113 executed by the central processing unit 110. In the behavior image data generation processing, the temporal change of each part is extracted from the skeleton image group shown in FIG. 4 and drawn as the behavior image data 133.

The behavior image generation program 113 reads the skeleton image data 132 from the storage device 130 (S31). In the following, for example, data such as the skeleton image data 132 (1) and (2) shown in FIG. 4 will be described as being read.

The behavior image generation program 113 extracts the optical flow as the movement of the skeleton from each skeleton image data 132, and generates the behavior image data 133 (S32). Here, the optical flow is a vector representation of the movement of an object in a temporally continuous image. There are various methods for calculating the optical flow, and for example, the Lucas-Kanade method can be used. When the optical flow is obtained based on the skeleton image data 132 (1) and (2) shown in FIGS. 4 (1) and 4 (2), the behavior image data 133 (1) shown in FIG. 5 (1) is generated. The arrow in the behavior image data indicates that the pixels in the image have moved from the start point to the end point of the arrow.

The behavior image generation program 113 stores the behavior image data 133 generated in step S32 in the storage device 130 (S33).

<Sequence generation process>

FIG. 12 is a flowchart showing a sequence data generation process executed by the sequence data generation program 114. The operating subject here is the sequence data generation program 114 executed by the central processing unit 110. In the sequence data generation process, the learning sequence data 135 described in FIG. 7 or the inference sequence data 137 described in FIG. 9 is generated.

First, the sequence data generation program 114 reads the original image data 131, the skeleton image data 132, the behavior image data 133, and the correct answer data 134 from the storage device 130 (S41). For example, it will be described below assuming that the data described in FIGS. 3, 4, 5, and 6 has been read.

The sequence data generation program 114 substitutes “1” for the variable t uniquely assigned to each image data 131, 132, 133 to represent the time series (S42). The sequence data generation program 114 acquires data within a predetermined time configured as one sequence (S43). Here, the data from "t" to "t + n-1" are treated as one sequence. The variable n represents the length of one sequence. For example, in the case of "t = 1, n = 3", one sequence is data of time "1", "2", "3".

For example, in FIG. 7, among the original image data file name 1353, the skeleton image data file name 1354, and the behavior image data file name 1355, the values indicating the temporal order in each file name are "0001" and "0002". , "0003" are used as one sequence. That is, in this case, it becomes "1", "2", and "3".

The sequence data generation program 114 sets a sequence identifier (sid) and a temporal sequence identifier (tid) of the sequence data (S44). A value that uniquely identifies each sequence data is stored in the sequence identifier. As shown in the sequence identifier 1351 of FIG. 7, the storage is performed in order from "1". The temporal order identifier is a value representing the order in each sequence data. As shown in the temporal order identifier 1352 of FIG. 7, the time is set as "1", "2", "3" in order from the oldest one.

The sequence data generation program 114 sets a classification class for each sequence data (S45). The classification class is an identifier representing a human behavior pattern in each sequence. For example, the classification class is set such that "walking person" is "0" and "running person" is "1".

The value to be stored in the classification class differs between the learning process and the inference process. At the time of inference processing, the value of the classification class is not set and is left blank. At the time of learning process, the value of the classification class is set based on the correct answer data. There are various methods for determining the value of the classification class, but for example, a method of determining the classification class by majority voting within the same sequence can be considered.

The case of “sid = 1” in FIG. 7 will be described. There are three original image data, "raw_0001.jpg", "raw_0002.jpg", and "raw_0003.jpg". When these three original image data are collated with the correct answer data, the correct answer is "1". Therefore, the classification class of the sequence of "sid = 1" becomes "1" by the majority vote of the three files. If the classification class cannot be determined by majority vote, for example, it may be randomly determined from the candidate classification classes, or the classification class having the maximum "tid" in the sequence may be used.

Subsequently, the sequence data generation program 114 substitutes “t + n” for the variable t (S46). That is, the start time of the next sequence is set.

The sequence data generation program 114 determines whether "t + n-1" is larger than the maximum time tmax (S47). When this condition is satisfied (S47: YES), it means that the sequence cannot be set any more. If the condition is not satisfied (S47: NO), the sequence data generation program 114 returns to step S43 and generates the next sequence. If the condition is satisfied (S47: YES), the process proceeds to step S48.

The sequence data generation program 114 stores the generated sequence data in the storage device 130 (S48). When the sequence data generation process is called for the learning process, it is stored in the storage device 130 as the learning sequence data 135. When the sequence data generation process is called for the inference process, it is stored in the storage device 130 as the inference sequence data 137.

<Learning process>

FIG. 13 is a flowchart showing a model generation process (learning process) executed by the model generation program 115. The operating subject here is the model generation program 115 executed by the central processing unit 110. In the model generation process, after the learning sequence data 135 described in FIG. 7 is generated, the model data 136 described in FIG. 8 is generated from the learning sequence data 135.

The model generation program 115 causes the sequence data generation program 114 to generate the learning sequence data 135 (S51). In this step S51, the sequence data generation process described in FIG. 12 is called, and the above-described process is executed.

The model generation program 115 reads the generated learning sequence data 135 from the storage device 130 (S52), and generates model data 136 from the learning sequence data 135 by machine learning (S53).

There are various machine learning methods, but for example, deep learning can be used. When using deep learning, various models can be defined. FIG. 14 shows a configuration example of the neural network.

In FIG. 14, CNN (Convolutional Neural Network) 21 is used as multi-channel image data in which the original image data 131, the skeleton image data 132, and the behavior image data 133 are combined into one for each temporal order identifier (tide) as input. The features are extracted by inputting to, and further, the features are extracted as a time series by LSTM (Long Short-Term Memory) 22 and the final classification class is output (23, 24). At the time of processing with CNN and LSTM, processing such as activation function or pooling or dropout may be added.

Set the classification class of each sequence data as the output of deep learning. The configuration of FIG. 14 is an example, and various changes can be made. By inputting the training sequence data 135 into the neural network described in FIG. 14 and training it by an error backpropagation method or the like, model data 136 capable of estimating the human behavior pattern in the image is generated.

Return to FIG. The model generation program 115 stores the model data 136 generated in step S53 in the storage device 130 (S54).

<Inference processing>

FIG. 15 is a flowchart showing an inference process executed by the inference program 116. The operating subject is the inference program 116 executed by the central processing unit 110. In the inference process, after the inference sequence data 137 described in FIG. 9 is generated, the classification class in each sequence is estimated based on the inference sequence data 137 and the model data 136 described in FIG.

The inference program 116 causes the sequence data generation program 114 to generate inference sequence data 137 (S61). In step S61, the sequence data generation process is called and the above-mentioned process is executed.

The inference program 116 reads the inference sequence data 137 and the model data 136 from the storage device 130 (S62), inputs the inference sequence data 137 to the model data 136 by inference processing, and obtains the classification class of each sequence. (S63).

When the original image data, the skeleton image data, and the behavior image data are input to the neural network described in FIG. 14 as a set of multi-channel image data for each temporal order identifier (ted), softmax The process 23 outputs the probability of the classification class of each sequence. For example, if the probability of the classification class "1" representing "running" is "0.9", it can be determined that the human behavior included in the input data group is "running". The identifier indicating the recognition result of the action is stored in the classification class.

The inference program 116 overwrites and saves the inference sequence data 137 generated in step S63 in the storage device 130 (S64).

As described above, according to the present embodiment, the skeleton image data 132 and the behavior image 133 data are generated from the original image data 131, and the model data 136 by machine learning is generated, so that the image data is reflected in the time series image data. It is possible to judge the behavior of a human being.

In this embodiment, since the skeleton image data 132 is used for recognizing human behavior, background information unrelated to humans can be excluded. Furthermore, in this embodiment, since the characteristics of the human skeleton are extracted, human behavior can be determined with high accuracy. Further, in this embodiment, since the behavior image data 133 is also used, the characteristics of human movement in time series can be extracted, and the determination accuracy can be further improved.

The second embodiment will be described with reference to FIGS. 16 and 17. In each of the following examples including this embodiment, the differences from the first embodiment will be mainly described. In this embodiment, when the time change of the background in each original image data 131 is equal to or greater than a predetermined value, the action recognition device 1A is operated to recognize the human action captured in the image. The behavior recognition device 1A according to the present embodiment may be used as a monitoring system for a driver who drives various moving objects such as a passenger car, a commercial vehicle, and a construction machine.

The sensor data from the sensor 31 is input to the action recognition device 1A. The sensor 31 is provided on the moving body 30 driven by a human being, which is the monitoring target of the action recognition device 1A, and corresponds to, for example, a speed sensor, an acceleration sensor, a position sensor, and the like. The action recognition device 1A can be provided on the moving body 30 or can be provided outside the moving body 30.

When the action recognition device 1A is provided on a server outside the moving body 30, for example, the sensor data from the sensor 31 of the moving body 30 and the original image data taken by the camera 140 or the like are subjected to action recognition in the server via a communication network. It is transmitted to the device 1A. The action recognition device 1A recognizes the driver's action based on the sensor data and the original image data received from the moving body 30, and transmits the recognition result (classification class) to the moving body 30 via the communication network. The information output device in the moving body 30 outputs an alarm according to the recognition result of the action.

FIG. 17 is a flowchart showing the behavior monitoring process S70 when the behavior recognition device 1A is used as the monitoring system. Here, a case where the action recognition device 1A is provided on the moving body 30 will be described as an example.

When the action recognition device 1A acquires the sensor data from the sensor 31 (S71), the action recognition device 1A determines whether the moving body 30 is moving based on the sensor data (S72). Here, as an example, when the moving body 30 is not stopped, that is, when the speed of the moving body 30 exceeds "0", it is determined that the moving body 30 is "moving". Instead of this, it can be determined that the moving body 30 is moving when the speed is equal to or higher than a predetermined speed set to an arbitrary natural number.

When the moving body 30 is not moving (S72: NO), the behavior monitoring process ends. On the other hand, when the moving body 30 is moving (S72: YES), the behavior recognition device 1A is divided into the original image data 131, the skeleton image data 132, and the behavior image data 133 as described in the first embodiment. By applying the based inference sequence data 137 to the model data 136, the behavior of the driver of the moving body 30 which is the target person is recognized (S73). Here, as the classification of behavior, "looking forward", "looking away", "operating a smartphone", "eating and drinking", "looking down", etc. are listed. ..

The action recognition device 1A determines whether the action recognized in step S73 is a normal action (S74). Here, it is assumed that "looking forward" is preset as a normal behavior, and other behaviors are set as not normal behaviors.

When the action recognition device 1A determines that the driver's action is a normal action (S74: YES), the action recognition device 1A ends this process. On the other hand, when the behavior recognition device 1A determines that the driver's behavior is not a normal behavior (S74: NO), the behavior recognition device 1A outputs an alarm (S75). The alarm can be output, for example, through an information output device mounted on the moving body 30 such as a car navigation system.

According to the present embodiment configured as described above, the moving body 30 can recognize the driver's behavior while moving and output information according to the recognition result. As described in the first embodiment, since the behavior recognition device 1A recognizes the driver's behavior by using not only the original image data but also the skeleton image data and the behavior image data, the moving body 30 moves and operates. Even when the background of the hand changes, the driver's behavior can be appropriately recognized.

A third embodiment will be described with reference to FIGS. 18 and 19. The behavior recognition device 1B according to the present embodiment detects the line of sight of the target person (here, the driver) from the original image data 131, and uses the detected line of sight to determine the behavior of the target person.

The behavior recognition device 1B shown in FIG. 18 is also used as a monitoring system for monitoring the behavior of the driver of the moving body 30 as in the second embodiment. The sensor data from the sensor 31 is input to the action recognition device 1B. Further, the behavior recognition device 1B is provided with a line-of-sight analysis unit 17 that detects the line of sight by analyzing the original image data 131. The line-of-sight analysis unit 17 is a function realized by the central processing unit 110 executing a predetermined computer program (line-of-sight analysis program (not shown)).

FIG. 19 is a flowchart of the behavior monitoring process S70B according to this embodiment. This process includes all steps S71 to S74 described in FIG. In this embodiment, a plurality of types of alarms are output instead of step S75 described in FIG. 17 (S78, S79). In addition, this example has novel steps S76 and S77.

The action recognition device 1B determines whether the driver's behavior obtained by applying the inference sequence data 137 to the model data 136 is normal (S74), and if it is determined that the behavior is not normal (S74). : NO), the analysis result of the driver's line of sight is read from the line of sight analysis unit 17 (S76).

The behavior recognition device 1B determines whether the analysis result of the line of sight is normal (S77). Here, the normal line of sight is a state in which the moving body 30 is facing the moving direction. Even if the driver's behavior is not normal (S74: NO), the behavior recognition device 1B maintains a certain level of safety as long as the line of sight is in the direction of movement (S77: YES). It is determined that the vehicle is present, and an alarm calling attention is output (S78).

On the other hand, when the driver's behavior is not normal (S74: NO) and the line of sight is also not normal (S77: NO), an alarm indicating a warning is output to the driver in order to improve safety. (S79). The warning indicating caution or warning may be output only to the driver in the moving body 30, or may be output to the management system that manages the moving body 30 or the information processing device owned by the administrator. ..

According to this embodiment configured in this way, when it is determined that the driver's behavior is not normal, an alarm is output in consideration of the driver's line of sight, so that safety is actually ensured. It is possible to suppress the situation where a warning is issued under the circumstances, reduce the discomfort or discomfort given to the driver, and improve the usability.

The present invention is not limited to the embodiment as it is, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. In addition, various inventions can be formed by appropriately combining the plurality of components disclosed in the embodiments. For example, some components may be removed from all the components shown in the embodiments. In addition, components across different embodiments may be combined as appropriate.

Each configuration, function, processing unit, processing means, etc. shown in the embodiment may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function is stored in a memory, hard disk, recording or storage device such as SSD (Solid State Drive), or recording or storage medium such as IC card, SD card, DVD, etc. be able to.

Further, in the above-described embodiment, the control lines and information lines are shown as necessary for explanation, and not all the control lines and information lines are necessarily shown in the product. All configurations may be interconnected.

Each component of the present invention can be arbitrarily selected, and an invention having the selected configuration is also included in the present invention. Further, the configurations described in the claims can be combined in addition to the combinations specified in the claims.

1,1A, 1B: Behavior recognition device, 11: Original image acquisition unit, 12: Skeletal image generation unit, 13: Behavior image generation unit, 14: Sequence data generation unit, 15: Model generation unit, 16: Inference unit, 30 : Moving body, 31: Sensor, 110: Central arithmetic processing device, 120: Input / output device, 130: Storage device, 131: Original image data, 132: Skeletal image data, 133: Behavior image data, 134: Correct answer data, 135 : Training sequence data, 136: Model data, 137: Inference sequence data

Claims (7)

A behavior analyzer that analyzes the behavior of a person included in an image.
An original image acquisition unit that acquires multiple original image data with different shooting times,
A skeleton image generation unit that generates skeleton image data of a person in each of the original image data,
A behavior image generation unit that generates behavior image data indicating a time change of the skeleton of the person based on each skeleton image data.
A model generation unit that generates a predetermined model capable of learning and inferring a person's behavior pattern based on the original image data, the skeleton image data, and the behavior image data.
Behavior analyzer with. It further includes an inference unit that estimates the behavior of a person in each of the original image data to be analyzed based on the plurality of original image data to be analyzed and the predetermined model, and outputs the estimation result.
The behavior analyzer according to claim 1. When the time change of the background in each of the original image data is equal to or greater than a predetermined value, the inference unit is generated by the skeleton image generation unit from each of the original image data of the analysis target and each of the original image data of the analysis target. It is included in each image data to be analyzed based on the plurality of skeleton image data to be generated, the plurality of behavior image data generated by the behavior image generation unit from each skeleton image data, and the predetermined model. Estimate the behavior of a person and output the estimation result,
The behavior analyzer according to claim 2. A line-of-sight analysis unit that analyzes the line of sight of the person included in each original image data to be analyzed is further provided.
The inference unit estimates the behavior of the person based on the analyzed line of sight, each original image data of the analysis target, and the predetermined model, and outputs the estimation result.
The behavior analyzer according to claim 2. The inference unit
When the time change of the background in each of the original image data is equal to or greater than a predetermined value, each of the original image data to be analyzed and a plurality of skeletons generated by the skeleton image generation unit from each of the original image data to be analyzed Based on the image data, a plurality of behavior image data generated by the behavior image generation unit from each of the skeleton image data, and the predetermined model, the behavior of the person is estimated and the estimation result is output.
When the estimation result does not indicate a normal state, the first alarm is output when the analyzed line of sight is within a preset predetermined range, and when the analyzed line of sight is out of the predetermined range. Outputs a second alarm,
The behavior analyzer according to claim 4. Further equipped with an arithmetic unit and a storage device,
When the arithmetic unit executes a predetermined computer program stored in the storage device, the original image acquisition unit, the skeleton image generation unit, the behavior image generation unit, and the model generation unit are realized.
The original image data, the skeleton image data, the behavior image data, and the predetermined model are stored in the storage device.
The behavior analyzer according to claim 1. It is a behavior analysis method that analyzes the behavior of a person included in an image using a computer.
The calculator
Acquire multiple original image data with different shooting times,
The skeleton image data of the person in each of the original image data is generated,
Based on each of the skeleton image data, behavior image data showing the time change of the skeleton of the person is generated.
Based on the original image data, the skeleton image data, and the behavior image data, a predetermined model capable of learning and inferring the behavior pattern of a person is generated.
Acquire a plurality of original image data to be analyzed and the predetermined model,
The behavior of the person in each original image data to be analyzed is estimated and the estimation result is output.
Behavior analysis method.

Images