JP2023182085A - Behavior recognition device, learning device, and behavior recognition method - Google Patents

Abstract

To reduce storage capacity and to accelerate behavior recognition in the case of accurately recognizing a plurality of kinds of behaviors of a recognition object in which a portion of its shape is missing.SOLUTION: A behavior recognition device can access a behavior classification model which is trained by using a component group about a learning object obtained from a shape of the learning object by component analysis for generating a statistical component by multivariate analysis, and a behavior of the learning object, detects the shape of the learning object from analysis object data, generates missing position information showing the position of a missing place in the shape of the recognition object, interpolates the missing place from non-missing information, updates the interpolated non-missing information as the shape of the recognition object, generates a component group about the same number of recognition objects as component groups about the learning object on the basis of the shape of an interpolated recognition object by component analysis, and outputs a recognition result showing a behavior of the recognition object by inputting the component group of the generated recognition object, and the missing position information to the behavior classification model.SELECTED DRAWING: Figure 1

Description

The present invention relates to a behavior recognition device, a learning device, and a behavior recognition method.

As background art in this technical field, Patent Document 1 discloses an action recognition device that highly accurately recognizes multiple types of actions to be recognized. This behavior recognition device uses a component group obtained from the shape of the learning target through component analysis that generates statistical components through multivariate analysis, and the behavior of the learning target, and learns behavior classification for each component group. has access to a model group, detects the shape of the recognition target from the data to be analyzed, and generates one or more components and the contribution rate of each component based on the shape of the recognition target through component analysis; Based on the cumulative contribution rate obtained from each contribution rate, an ordinal number indicating each dimension of one or more components is determined, and a component group that is the same as a specific component group that includes one or more components with an ordinal number indicating the determined dimension is determined. The specific behavior classification model learned in is selected from the behavior classification model group, and a specific component group is input to the specific behavior classification model, thereby outputting a recognition result indicating the behavior to be recognized.

Japanese Patent Application Publication No. 2022-43974

However, the above-mentioned Patent Document 1 is a technology for recognizing behavior by selecting a specific behavior classification model from a group of behavior classification models, and does not take into consideration the point of recognizing behavior by making full use of one behavior classification model. .

An object of the present invention is to reduce memory capacity and speed up behavior recognition when recognizing multiple types of behaviors of recognition targets whose shapes are partially missing with high accuracy.

A behavior recognition device that is an aspect of the invention disclosed in this application is a behavior recognition device that includes a processor that executes a program, and a storage device that stores the program, and that calculates statistical components by multivariate analysis. It is possible to access a behavior classification model learned using a component group related to the learning target obtained from the shape of the learning target by the generated component analysis and the behavior of the learning target, and the processor a detection process for detecting the shape of the recognition target; a missing position information generation process for generating missing position information indicating a position of a missing part in the shape of the recognition target detected by the detection process; By interpolating the missing part from non-missing information that is a part other than the missing part in the shape of the recognition target, and updating the non-missing information after interpolation as the shape of the recognition target, and the component analysis, a component analysis process that generates the same number of component groups related to the recognition target as component groups related to the learning target based on the shape of the recognition target interpolated by the interpolation process; The present invention is characterized in that by inputting the generated component group related to the recognition target and the missing position information, a behavior recognition process is executed to output a recognition result indicating the behavior of the recognition target.

A learning device that is one aspect of the invention disclosed in this application is a learning device that has a processor that executes a program, and a storage device that stores the program, and the processor is configured to learn the shape and behavior of a learning target. an acquisition process for acquiring teacher data including; a deletion process for deleting the shape of the learning target acquired by the acquisition process; and a deletion position indicating the position of the missing part deleted from the shape of the learning target by the deletion process. a missing position information generation process that generates information, and interpolation from non-missing information that is a location other than the missing location that was lost by the missing process in the shape of the learning target, and the non-missing information after interpolation is used for the learning. A component group related to the learning target is generated based on the shape of the learning target interpolated by the interpolation process, using interpolation processing to update the shape of the target and component analysis to generate statistical components by multivariate analysis. Learning the behavior of the learning target based on a component analysis process, a component group related to the learning target generated by the component analysis process, the behavior of the learning target, and the missing position information, and performing the learning. The present invention is characterized by executing a behavior learning process that generates a behavior classification model for classifying target behavior.

According to a typical embodiment of the present invention, when recognizing multiple types of actions of a recognition target whose shape is partially missing with high accuracy, it is possible to reduce memory capacity and speed up action recognition. can. Problems, configurations, and effects other than those described above will become clear from the description of the following examples.

FIG. 1 is an explanatory diagram showing an example of a system configuration of an action recognition system according to a first embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. FIG. 3 is an explanatory diagram showing an example of learning data. FIG. 4 is a block diagram showing an example of the functional configuration of the action recognition system according to the first embodiment. FIG. 5 is a block diagram showing a detailed functional configuration example of the skeleton information processing section. FIG. 6 is an explanatory diagram showing a detailed method for calculating joint angles executed by the joint angle calculating section. FIG. 7 is an explanatory diagram illustrating a detailed example of a method for calculating the amount of movement between frames, which is executed by the movement amount calculating section. FIG. 8 is an explanatory diagram showing a detailed method of normalizing skeleton information executed by the normalization unit. FIG. 9 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB. FIG. 10 is an explanatory diagram showing an example in which principal components generated by a principal component analysis section using a teacher signal as input data are plotted on a principal component space. FIG. 11 is an explanatory diagram showing data outputted by the principal component analysis section and the missing position information generation section to the behavior learning section. FIG. 12 is an explanatory diagram showing a detailed method for the behavior learning unit to learn behavior and for the behavior recognition unit to classify the behavior. FIG. 13 is a graph showing changes in the cumulative contribution rate used by the principal component analysis unit when determining the number of dimensions. FIG. 14 is a flowchart illustrating a detailed processing procedure example of learning processing by the server (learning device) according to the first embodiment. FIG. 15 is a flowchart illustrating a detailed processing procedure example of skeleton information processing according to the first embodiment. FIG. 16 is a flowchart illustrating an example of a behavior recognition processing procedure by a client (behavior recognition device) according to the first embodiment. FIG. 17 is a block diagram showing an example of the functional configuration of the action recognition system according to the second embodiment. FIG. 18 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB according to the third embodiment. FIG. 19 is an explanatory diagram showing data that the principal component analysis section and the missing position information generation section according to the third embodiment output to the behavior learning section. FIG. 20 is a block diagram showing an example of the functional configuration of the action recognition system according to the third embodiment. FIG. 21 is a flowchart illustrating an example of a behavior recognition processing procedure by a client (behavior recognition device) according to the third embodiment. FIG. 22 is a block diagram showing an example of the functional configuration of the skeleton information processing section according to the fourth embodiment. FIG. 23 is a flowchart illustrating a detailed processing procedure example of the skeleton information processing unit according to the fourth embodiment. FIG. 24 is a block diagram showing an example of the functional configuration of the action recognition system according to the fifth embodiment. FIG. 25 is a block diagram showing an example of the functional configuration of the action recognition system according to the sixth embodiment. FIG. 26 is an explanatory diagram showing a decision tree, which is a basic method for classifying behaviors by the behavior learning unit and the behavior recognition unit. FIG. 27 is an explanatory diagram showing a detailed method for developing classification using a decision tree. FIG. 28 is an explanatory diagram showing ensemble learning and a method used by the behavior learning section and the behavior recognition section to classify behaviors. FIG. 29 is a block diagram showing an example of the functional configuration of the action recognition system according to the seventh embodiment. FIG. 30 is a flowchart illustrating a detailed processing procedure example of learning processing by the server (learning device) according to the seventh embodiment.

Embodiments according to the present invention will be described below based on the drawings. In addition, in all the figures for explaining the embodiment, the same members are given the same reference numerals in principle, and repeated explanations thereof will be omitted. In addition, in the following embodiments, the constituent elements (including elemental steps, etc.) are not necessarily essential, except when explicitly stated or when it is clearly considered essential in principle. Needless to say. In addition, when we say "consists of A," "consists of A," "has A," or "contains A," other elements are excluded, unless it is specifically stated that only that element is included. Needless to say, this is not something you should do. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., the shape, positional relationship, etc. of components, etc. are referred to, unless specifically stated or when it is considered that it is clearly not possible in principle. This shall include things that approximate or are similar to, etc.

In this specification, etc., expressions such as "first," "second," and "third" are used to identify constituent elements, and do not necessarily limit the number, order, or content thereof. isn't it. Further, numbers for identifying components are used for each context, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, this does not preclude a component identified by a certain number from serving the function of a component identified by another number.

The position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings or the like.

<Action recognition system>
FIG. 1 is an explanatory diagram showing an example of a system configuration of an action recognition system according to a first embodiment. The behavior recognition system 100 includes a server 101 and one or more clients 102. The server and the client are communicably connected via a network 105 such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network). The server 101 is a computer that manages the clients 102. The client 102 is a computer that is connected to the sensor 103 and acquires data from the sensor 103.

The sensor 103 detects data to be analyzed from the analysis environment. The sensor 103 is, for example, a camera that captures still images or moving images. Further, the sensor 103 may detect sound or smell. The teacher signal DB 104 is a database that holds a combination of learning data (person's skeletal information) and behavior information (for example, a person's posture or motion such as "standing" or "falling down") as a teacher signal. The teacher signal DB 104 may be stored in the server 101 or may be connected to a computer that can communicate with the server 101 or the client 102 via the network 105.

The behavior recognition system 100 has a learning function using the teacher signal DB 104 and an action recognition function using a behavior classification model obtained by the learning function. A behavior classification model is a learning model for classifying the behavior of recognition targets such as people and animals. The learning function and the behavior recognition function may be implemented in either the server 101 or the client 102 as long as they are implemented in the behavior recognition system 100. For example, the server 101 may implement a learning function, and the client 102 may implement an action recognition function. Further, the server 101 may implement a learning function and a behavior recognition function, and the client 102 may transmit data from the sensor 103 to the server 101 or receive behavior recognition results from the server 101 using the behavior recognition function. .

Further, the client 102 may implement a learning function and a behavior recognition function, and the server 101 may manage behavior classification models and behavior recognition results from the client 102. Note that a computer implementing a learning function is referred to as a learning device, and a computer implementing at least an action recognition function of the learning function and the action recognition function is referred to as an action recognition device. Further, in FIG. 1, a client-server type behavior recognition system 100 is taken as an example, but a stand-alone type behavior recognition device may be used. In the first embodiment, for convenience of explanation, a behavior recognition system 100 will be described as an example in which the server 101 implements a learning function (learning device) and the client 102 implements a behavior recognition function (behavior recognition device).

<Example of computer hardware configuration>
FIG. 2 is a block diagram showing an example of the hardware configuration of a computer (server 101, client 102). The computer 200 includes a processor 201, a storage device 202, an input device 203, an output device 204, and a communication interface (communication IF) 205. The processor 201, storage device 202, input device 203, output device 204, and communication IF 205 are connected by a bus 206. Processor 201 controls computer 200 . Storage device 202 serves as a work area for processor 201 . Furthermore, the storage device 202 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the storage device 202 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. Input device 203 inputs data. Examples of the input device 203 include a keyboard, mouse, touch panel, numeric keypad, and scanner. Output device 204 outputs data. Examples of the output device 204 include a display, a printer, and a speaker. Communication IF 205 connects to network 105 and transmits and receives data.

<Learning data>
FIG. 3 is an explanatory diagram showing an example of learning data. The learning data 380 includes skeletal information 320 and joint angles 370 for each subject. Skeletal information 320 is detected based on analysis target data acquired from sensor 103. Joint angle 370 is calculated based on skeletal information 320. The learning data 380 for one subject is configured, for example, by a combination of skeletal information 320 and joint angles 370 obtained from each of a plurality of time-series frames in which the subject is the subject.

The skeleton information 320 includes, for each of a plurality of (18 in this example) skeleton points 300 to 317, a name 321, an x-coordinate value 322 on the x-axis, a y-coordinate value 323 on the y-axis perpendicular to the x-axis, has. The joint angle 370 also has a name 371 for each of the plurality of (18 in this example) skeletal points 300-317. In addition, in the name 371, ∠a-b-c (a, b, c are the names 321 of skeleton points) is the joint angle 370 of the skeleton point b formed by the line segment ab and the line segment bc. Note that the skeletal information 320 may include, for example, finger joints. Further, the joint angle 370 may also include joint angles 370 other than these.

Note that in FIG. 3, the coordinate values of the skeleton points 300 to 317 are two-dimensional positional information (a combination of x-coordinate values and y-coordinate values), but they may be three-dimensional positional information. Specifically, for example, a z-coordinate value in the z-axis (eg, depth direction) perpendicular to the x-axis and the y-axis may be added.

<Functional configuration example of behavior recognition system 100>
FIG. 4 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the first embodiment. The server 101 includes a teacher signal acquisition unit 401, a loss generation unit 402, a loss control unit 421, a loss position information generation unit 422, a loss information interpolation unit 423, a skeleton information processing unit 403, and a principal component analysis unit 404. and a behavior learning unit 406. The client 102 includes a skeleton detection section 451, a missing position information generation section 462, a missing information interpolation section 461, a skeleton information processing section 453, a principal component analysis section 454, and an action recognition section 457.

Specifically, these are realized, for example, by causing the processor 201 to execute a program stored in the storage device 202 shown in FIG. 2. First, an example of the functional configuration of the server 101 will be described.

The teacher signal acquisition unit 401 acquires one or more teacher signals used for learning from the teacher signal DB 104 and outputs the selected teacher signal to the loss generation unit 402 . The skeleton information 320 in the teacher signal is non-defective information without any missing skeleton points.

The loss control unit 421 sets skeleton points to be lost according to random numbers or predetermined numbers, and outputs them as loss occurrence information to the loss generation unit 402 and the loss position information generation unit 422. The number of skeleton points to be deleted may be single, multiple, or zero indicating that no skeleton points are to be deleted. The predetermined number here indicates, for example, a skeletal position where there is a high possibility that detection will be missed when the skeletal detection unit 451 (described later) detects the skeletal information 320 of a person appearing in the data to be analyzed acquired from the sensor 103. Let it be a number.

The loss generation unit 402 causes skeleton points to be lost in the skeleton information 320 of the teacher signal acquired from the teacher signal acquisition unit 401 according to the loss occurrence information. The loss generating unit 402 updates the skeleton information 320 after the loss (including the case where the number of skeleton points to be deleted is zero) as the skeleton information 320 in the teacher signal. Furthermore, in order to strengthen noise resistance, when a skeleton point is deleted, the skeleton information 320 may be updated by adding noise that shifts the position of the skeleton point that is not to be deleted. The missing skeleton information 320 or pieces of skeleton information 320 will be referred to as "missing information."

Then, the defect generation unit 402 outputs a teacher signal containing the missing information, which is the name 321 and position information (x coordinate value 322, y coordinate value 323) of the missing skeleton point, to the missing information interpolation unit 423.

The missing information interpolation unit 423 interpolates missing information from the non-missing skeleton information 320. One or more pieces of skeleton information 320 that are not missing are referred to as non-missing information.

Specifically, for example, the missing information interpolation unit 423 extracts missing information from a skeleton point that was connected to a skeleton point of the missing information or a skeleton point located near the skeleton point of the missing information, out of the non-missing information. May be interpolated.

Furthermore, the missing information interpolation unit 423 may substitute predetermined position information for the missing information. Furthermore, the missing information interpolation unit 423 may interpolate the previously acquired skeletal information 320 of another frame using the missing information of the skeletal information 320 that has been determined to include missing information. In this way, the interpolation method for missing information is not limited. The missing information interpolation unit 423 outputs the interpolated skeleton information 320 to the skeleton information processing unit 403.

Based on the defect occurrence information from the defect control section 421, the defect position information generation section 422 outputs the position information of the missing skeleton point to the behavior learning section 406 as defect position information.

The skeleton information processing unit 403 processes the skeleton information 320 after interpolation by the missing information interpolation unit 423. Specifically, for example, the skeletal information processing unit 403 calculates the joint angle 370 and the amount of movement between frames from the skeletal information 320 of the acquired interpolated teacher signal. Furthermore, the skeleton information processing unit 403 performs normalization on the skeleton information 320, excluding absolute position information so that the size of the skeleton information 320 is constant. Then, the skeletal information processing unit 403 outputs the joint angle 370, the amount of movement between frames, and the normalized skeletal information 320 to the principal component analysis unit 404.

FIG. 5 is a block diagram showing a detailed functional configuration example of the skeleton information processing units 403 and 453. The skeletal information processing units 403 and 453 include a joint angle calculation unit 501, a movement amount calculation unit 502, and a normalization unit 503.

The joint angle calculation unit 501 calculates a joint angle 370 from the interpolated skeletal information 320 of the acquired teacher signal, and outputs it to the principal component analysis unit 404 via the movement amount calculation unit 502 and the normalization unit 503.

The movement amount calculation unit 502 calculates the interframe movement amount from the interpolated skeleton information 320 of the acquired teacher signal, and outputs it to the principal component analysis unit 404 via the normalization unit 503.

The normalization unit 503 excludes absolute position information from the interpolated skeletal information 320 from the acquired teacher signal, and performs normalization such that the size of the interpolated skeletal information 320 is constant. It is output to the principal component analysis section 404.

Returning to FIG. 4, the principal component analysis unit 404 uses as input data the normalized skeletal information 320, the joint angles 370, and the amount of movement between frames among the teacher signals acquired from the skeletal information processing unit 403. A principal component analysis is performed to generate one or more principal components, and the generated principal components are output to the behavior learning unit 406. Note that among the skeletal information 320, the joint angles 370, and the amount of movement between frames, at least the normalized skeletal information 320 may be input data.

In principal component analysis, as shown in equation (1) below, input data x _i is multiplied by coefficient w _ij and added to generate principal component y _i . The general formula for principal component analysis is shown in formula (2) below. As shown in equation (3) below, the coefficient w _ij is determined so that when the variance of y _i is defined as V(y _i ), the variance V(y _i ) is maximized.

However, if there is no constraint on the coefficient wij, the absolute amount of the variance V(y _i ) can be infinitely large, and the coefficient _wij cannot be uniquely determined, so the constraint in equation (4) below It is desirable to add . Furthermore, in order to eliminate duplication of information, it is desirable to impose the following formula (5) constraint such that the covariance between the newly generated principal component y _k and the previously generated principal component y _k is 0.

However, the above equations (4) and (5) that are added as constraints are not limited to these, and the coefficient w _ij may be calculated by adding another constraint condition or removing the constraints. When the variance V(yj) of the new principal component y _j generated in this way is separately defined as λ _j as shown in equation ₍ 6) below, the variance V(x _j ) and the sum of λ _j are equal.

Here, p is the number of input data _xj . The higher the variance V (y _j ) of the newly generated principal component y _j , the more the original information is reflected, and the principal component with the highest variance value is the first, second, ..., m-th principal component. That's what it means. The ratio of the variance of the newly generated variable y _j to the variance of the original data is called the contribution rate, and is expressed by the following equation (8). Further, the result of adding the contribution rate from the contribution rate of the first principal component in descending order of the variance value (in ascending order of the ordinal number m of the principal component) is called the cumulative contribution rate, and is expressed by the following formula (9).

The contribution rate and the cumulative contribution rate are measures of how much the newly generated principal component y _j or the generated plurality of principal components represent the information amount of the original data, and are generated together with the principal component. Although principal component analysis was applied as an example of component analysis that generates statistical components in multivariate analysis, independent component analysis, which is also an example of component analysis, may be performed instead of principal component analysis. .

In the case of independent component analysis, the principal components are independent components. A contribution rate may be used as an index indicating how much influence this independent component has on the input data xi. In the independent component analysis, the sum of squares of the mixing coefficient matrix in the independent component analysis for each independent component becomes the strength of each independent component.

The strength of the independent component indicates the variance of the independent component in the input data x _i . That is, since all the independent components obtained by the independent component analysis have a unified variance of 1, the sum of the squares of the mixing coefficients becomes the variance of the input data x _i . Then, the contribution ratio of the independent variable may be determined by dividing the strength of the independent component by the sum of the strengths of all independent components.

The principal component analysis unit 404 controls an ordinal number k indicating the dimension of each of one or more components. Specifically, for example, the principal component analysis unit 404 determines how many dimensions of the principal components to be used for learning in the behavioral learning unit 406 among the generated principal components, in descending order of variance value, and determines the first principal component. The principal components from the component to the k-th principal component whose ordinal number is the determined dimension k (k is an integer greater than or equal to 1) are output to the behavior learning unit 406 in descending order of variance value. Specifically, for example, the principal component analysis unit 404 employs the dimension k of the principal component that is equal to or greater than the cumulative contribution rate threshold and is closest to the threshold. Alternatively, the principal component analysis unit 404 adopts the dimension k of the principal component that is less than or equal to the cumulative contribution rate threshold and closest to the threshold.

The behavior learning unit 406 performs learning by associating the principal component acquired from the principal component analysis unit 404, the missing position information acquired from the missing position information generation unit 422, and the behavior information in the teacher signal acquired from the teacher signal DB 104. do. Specifically, for example, the behavior learning unit 406 uses data obtained by connecting the principal component group from the first principal component to the k-th principal component obtained from the principal component analysis unit 404 and the missing position information as an explanatory variable, and A behavior classification model is generated by machine learning using the behavior information in the teacher signal acquired from the signal DB 104 as a target variable. The behavior learning unit 406 outputs the behavior classification model generated as a result of learning to the behavior recognition unit 457.

Next, an example of the functional configuration of the client 102 will be described. The skeleton detection unit 451 detects the skeleton information 320 of a person appearing in the data to be analyzed acquired from the sensor 103 and outputs it to the missing information interpolation unit 461 and the missing position information generating unit 462. To detect the skeletal information 320, a neural network (NN) that can estimate the human skeletal information 320 generated by machine learning may be used, or markers may be added to the skeletal points of the person to be detected, and markers that appear in the image may be used. The skeletal information 320 may be detected from the position, and the method of detecting the skeletal information 320 is not limited. Skeletal points in the skeleton information 320 whose x-coordinate values 322 and y-coordinate values 323 are not detected and become undefined values are missing information, and skeleton points in the skeleton information 320 excluding the missing information are non-defective information. For example, if only the x-coordinate value 322 and y-coordinate value 323 of the skeleton point 310 among the skeleton points 300-317 are indefinite values, the skeleton point 310 is missing information, and the skeleton points 300-309, 311-317 are non-constant. This is missing information.

The missing information interpolation unit 461 has the same function as the missing information interpolation unit 423. The missing information interpolation unit 461 performs the same processing as the missing information interpolation unit 423 on the skeleton information 320 from the skeleton detection unit 451, and converts missing information (skeletal point 310 in the above example) into non-missing information. Interpolation is performed from the skeleton points 300 to 309 and 311 to 317 in the above example, and the interpolated skeleton information 320 is output to the skeleton information processing section 453.

The missing position information generating section 462 has the same function as the missing position information generating section 422. The missing position information generating unit 462 performs the same processing as the missing position information generating unit 422 on the skeleton information 320 from the skeleton detecting unit 451, and detects occlusions in the skeleton information 320 detected by the skeleton detecting unit 451. If there is a skeleton point that could not be acquired, the system generates missing position information indicating the position of the skeleton point as a missing point, and outputs it to the behavior recognition unit 457. .

The skeletal information processing unit 453 has the same functions as the skeletal information processing unit 403. The skeleton information processing unit 453 executes the same processing as the skeleton information processing unit 403 on the skeleton information 320 detected by the skeleton detection unit 451, and calculates the joint angle 370, the amount of movement between frames, and the normalized skeleton. The information 320 is output to the principal component analysis section 454.

Principal component analysis section 454 has the same functions as principal component analysis section 404. The principal component analysis unit 454 performs the same processing as the principal component analysis unit 404 on the output data from the skeletal information processing unit 453 to generate one or more principal components, and outputs the generated principal components to the behavior recognition unit 457. .

The principal component analysis unit 454 determines an ordinal number k indicating each dimension of one or more components based on the cumulative contribution rate obtained from each contribution rate. Specifically, for example, the principal component analysis unit 454 determines, based on the acquired contribution rate and cumulative contribution rate, up to which dimension of the acquired principal components should be output to the behavior recognition unit 457 in descending order of variance? Determine the number of dimensions k that represents . The number of dimensions k is an ordinal number k indicating the dimension of the principal component. For example, for the first principal component, the number of dimensions (ordinal number) k=1, and for the second principal component, the number of dimensions (ordinal number) k=2. The principal component analysis unit 454 outputs a group of principal components from the first principal component to the k-th principal component to the behavior recognition unit 457 in order of descending variance.

The behavior recognition unit 457 selects the analysis target obtained from the sensor 103 based on the behavior classification model generated by the behavior learning unit 406, the principal component group from the first principal component to the k-th principal component, and the missing position information. Recognize human behavior reflected in data. Specifically, for example, the behavior recognition unit 457 inputs the principal component group (first principal component to k-th principal component) and missing position information obtained from the data to be analyzed into the selected behavior classification model. As a result, a predicted value indicating the behavior of the person reflected in the data to be analyzed is output as a recognition result.

<Example of joint angle calculation>
FIG. 6 is an explanatory diagram showing a detailed calculation method of the joint angle 370 executed by the joint angle calculation unit 501. The joint angle calculation unit 501 calculates the joint angle θ at three connected skeleton points 600 to 602. The skeleton information 620 of the skeleton points 600 to 602 is defined as position vectors O, A, and B with the origin 630 as a reference. The joint angle calculation unit 501 calculates a relative vector with the skeleton point 600 as the origin as shown in the following equations (10) and (11), and from the calculated vector, the following equation (12) is established, and the following equation (13) is established. The joint angle θ is calculated by calculating the arc cosine as shown in .

<Example of calculating the amount of movement between frames>
FIG. 7 is an explanatory diagram illustrating a detailed example of a method for calculating the amount of movement between frames, which is executed by the movement amount calculation unit 502. The movement amount calculation unit 502 uses skeleton information 701 of the Nth frame and skeleton information 702 of the NMth frame regarding the same subject in calculating the movement amount between frames. N and M are integers of 1 or more, and N>M. The value of M can be set arbitrarily. As shown in equations (14) to (16) below, the movement amount calculation unit 502 calculates the distances between the same skeletal points 300 to 317 of the same person shown between each frame. The amount of movement of the 18 skeleton points 300 to 317 between frames is the amount of movement between frames for the person.

However, the amount of movement between frames executed by the movement amount calculation unit 502 is not limited to this, and as shown in equation (17) below, the movement amount calculation unit 502 can calculate the amount of movement of the same person shown between each frame. The distances between the same skeletal points 300 to 317 may be calculated, and the sum of the inter-frame movement amounts of all 18 skeletal points 300 to 317 may be used as the inter-frame movement amount for the person.

Further, the movement amount calculation unit 502 may use center-of-gravity skeleton information 711 and center-of-gravity skeleton information 712, which are the center of gravity, out of the skeleton information 701 of the n-th frame and the skeleton information 702 of the nm-th frame. Specifically, for example, the movement amount calculation unit 502 calculates the center of gravity for each person as shown in equations (18) to (19) below, and calculates the center of gravity for each person as shown in equation (20) below. , the amount of movement between frames for the person may be calculated.

<Example of normalization>
FIG. 8 is an explanatory diagram showing a detailed method of normalizing the skeleton information 320 performed by the normalization unit 503. First, the normalization unit 503 (a) calculates the center of gravity from all or part of the skeleton information 320, and (b) converts the center of gravity into relative coordinates with the origin as the origin. After that, the normalization unit 503 divides (d) the position information of each skeleton point in the skeleton information 320 by (c) the length L of the minimum rectangular diagonal that surrounds the 18 skeleton points 300 to 317. If the skeleton information 320 obtained in (d) is used as a teacher signal, the position information of the skeleton points 300 to 317 after division will also be incorporated.

For example, if the normalization unit 503 is not executed and learning for skeleton detection and behavior classification is executed for an action such as "a 180 cm person sits at point A", "a person who is 180 cm tall sits at point A" and "does not sit at any place other than point A" There is a possibility that a judgment will be made that ``people who do not sit down will not sit down''. In order to eliminate these limitations and provide versatility in behavior classification, the normalization unit 503 normalizes the skeleton information 320 in order to remove absolute position information in the image and information regarding the size of the skeleton. do.

<Teacher signal held by the teacher signal DB 104>
FIG. 9 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB 104. For the person appearing in (a) image 900, which is data to be analyzed, (b) skeletal information 320A, joint angles 370 (not shown), and (c) behavior information 901 (“standing”) associated with skeletal information 320A. , becomes the teacher signal. Similarly, for the person appearing in (a) image 910, which is the data to be analyzed, (b) skeletal information 320B, joint angles 370 (not shown), and (c) behavioral information 911 (“fall down”) associated with skeletal information 320B. ”) becomes the teacher signal.

<Dimension reduction by principal component analysis unit 404>
FIG. 10 is an explanatory diagram showing an example in which the principal components generated by the principal component analysis unit 404 using the teacher signal as input data are plotted on the principal component space. The legend indicates behavior information 1000 to 1004 included in the teacher signal.

In FIG. 10, (a) shows an example in which the first principal component is plotted on the X axis and the second principal component is plotted on the Y axis, and information up to the second principal component is plotted on a two-dimensional plane. (b) is an example in which the first principal component is plotted on the X-axis, the second principal component is plotted on the Y-axis, and the third principal component is plotted on the Z-axis, and the information up to the third principal component is plotted on a three-dimensional space. show.

In (a), it can be seen that standing 1000, sitting 1001, and falling 1004 can be separated even on a two-dimensional plane up to the second principal component, but walking 1002 and squatting 1003 are two-dimensional up to the second principal component. It can be seen that separation is difficult on a dimensional plane. Here, in (b), if walking 1002 and crouching 1003 are plotted on a three-dimensional space including up to the third principal component, the possibility of separation may increase.

Therefore, if a large number of principal components generated by the principal component analysis unit 404 are used, highly accurate behavior classification is possible. However, as the ordinal number k, which indicates the dimension of the principal component, increases, the amount of calculation increases, so it is necessary to consider the extent of the principal component based on the accuracy and amount of calculation, and determine in what dimensional space the behavior should be represented.

Therefore, the principal component analysis unit 404 changes the maximum ordinal number of the principal components used for learning by the behavior learning unit 406, and outputs a principal component group from the first principal component to the principal component with the maximum ordinal number to the behavior learning unit 406. Specifically, for example, the required accuracy of the behavior classification described above (for example, an ordinal number indicating the minimum required principal component dimension) and/or the allowable amount of calculation are set in advance, and the principal component analysis unit 404 The learning unit 406 changes the maximum ordinal number of the principal component used for learning, and determines the ordinal number that satisfies the required accuracy and/or allowable amount of calculation to the maximum extent.

For example, if the required accuracy is the ordinal number "3" indicating the dimension (third principal component), the principal component analysis unit 404 determines the maximum ordinal number to be "3" and The principal component group of is output to the behavior learning unit 406.

In addition, when the allowable amount of calculation is set as a condition, the principal component analysis unit 404 sequentially acquires the amount of calculation in ascending order starting from the first principal component, and determines the maximum ordinal number to be the ordinal number ( For example, the ordinal number (for example, "4") is determined to be one less than "5" (for example, "4"), and the principal component group from the first principal component to the fourth principal component with the maximum ordinal number k=4 is output to the behavior learning unit 406. .

In addition, if the required accuracy is the ordinal number "3" indicating the dimension (third principal component) or more, and the allowable calculation amount is set as a condition, the cumulative calculation amount up to the third principal component is the allowable calculation amount. If it is below, the principal component analysis unit 404 changes the maximum ordinal number from "3" to "4". Then, if the cumulative amount of calculation up to the fourth principal component exceeds the allowable amount of calculation, the principal component analysis unit 404 determines the maximum ordinal number k to be "3" and The group is output to the behavior learning unit 406.

On the other hand, if the cumulative amount of calculation up to the third principal component exceeds the allowable amount of calculation, the principal component analysis unit 404 changes the maximum ordinal number from "3" to "2". Then, if the cumulative amount of calculation up to the second principal component is less than the allowable amount of calculation, the principal component analysis unit 404 determines the maximum ordinal number k to be "2" and The component group is output to the behavior learning unit 406.

Note that the principal component group output to the behavior learning unit 406 does not need to be limited to ascending order starting from the first principal component. For example, the principal component analysis unit 404 may extract a specific number of predetermined principal component groups. Further, the principal component analysis unit 404 may decide the principal component group to be output to the behavior learning unit 406 after excluding a specific principal component group. In this way, the principal component groups output to the behavior learning unit 406 are not limited to principal component groups in ascending order from the first principal component.

Also in this case, if the allowable amount of calculation is set as a condition, the principal component analysis unit 404 performs calculations in ascending ordinal order for principal component groups that are not limited to ascending order from the first principal component described above. The amount is sequentially acquired, and the principal component group up to the ordinal number immediately before the ordinal number when the allowable calculation amount is exceeded for the first time is output to the behavior learning unit 406. For example, if the principal component group consists of the second principal component, third principal component, and fifth principal component, the second principal component does not exceed the allowable amount of calculation, and the second and third principal components do not exceed the allowable amount of calculation. If the allowable calculation amount is exceeded for the first time in the second principal component, third principal component, and fifth principal component without exceeding the allowable amount of calculation, the principal component analysis unit 404 Up to three principal components may be determined as the principal component group to be output to the behavior learning unit 406.

<Data output to behavior learning unit 406>
FIG. 11 is an explanatory diagram showing data that the principal component analysis section 404 and the missing position information generation section 422 output to the behavior learning section 406. The principal component analysis unit 404 executes a principal component analysis using the interpolated skeleton information 320, the joint angles 370, and the amount of movement between frames as input data to generate one or more principal components 1101.

The missing position information generation unit 422 generates missing position information 1100. The missing position information 1100 is indicated by, for example, variables prepared as many times as the number of skeletons as flag information indicating whether or not each skeleton in the skeleton information 320 is missing. For example, if the skeleton information 320 is composed of 18 skeleton points, the flag information is prepared in 18 variables, and for each skeleton point, if there is coordinate information of the skeleton point without missing, "1" is set, If it is missing, the missing position information 1100 is generated as flag information of "0".

Furthermore, the missing position information 1100 may be generated as a multivalued variable, with each value indicating which skeleton point is missing, and the data format of the missing position information 1100 is not limited. In this way, the behavior learning unit 406 is outputted with data 1102 that is a compilation of the principal component 1101 generated by the principal component analysis unit 404 and the missing position information 1100 generated by the missing position information generating unit 422. Note that the data that the principal component analysis unit 454 and the missing position information generation unit 462 output to the behavior recognition unit 457 is also data similar to the data 1102.

The behavior learning unit 406 performs learning using the data 1102, generates a behavior classification model, and outputs it to the behavior recognition unit 457.

FIG. 12 is an explanatory diagram showing a detailed method for the behavior learning unit 406 to learn behaviors and for the behavior recognition unit 457 to classify the behaviors. In FIG. 12, (a) shows the distribution of plot points of the behavior up to the second variable after dimension reduction, and (b) shows the behavior classification result taking into account the missing position information 1100. For each action on the principal component space, the action learning unit 406 generates a boundary line 1210 and a boundary plane 1220 in order to classify each action into regions. The method for learning and classifying behaviors may be any of the k-means method, support vector machine, decision tree, random forest, etc., and the behavior learning method is not limited.

The missing position information 1100 during behavior learning by the behavior learning unit 406 is plotted on the principal component space and given to each piece of behavior information. Points plotted for each action information (standing, sitting, walking, falling, raising hand, crouching) are plotted points 1200 to 1205. For the sake of simplicity, plot points 1200 to 1203 of behavior when there is no skeleton point loss are shown as white points, and plot points 1204 and 1205 of behavior when skeleton point loss occurs are filled out. express.

The action plot point 1204 for which the skeleton point is missing is assigned the action "raise hand", and the plot point 1205 for the action for which the skeleton point is missing is assigned the action information "squat". has been done. Note that plot points (1200, 1204) and (1201, 1205) having the same shape indicate similar numerical information when the skeleton information 320 after interpolation is subjected to principal component analysis.

Skeletal points of the missing skeleton information 320 are interpolated by a missing information interpolation unit 423. However, since interpolated skeletal points may contain a larger amount of error than detected skeletal points, behavior classification accuracy may deteriorate when classifying behaviors in the same dimension as behaviors that do not include defects. There is sex. Therefore, by classifying the skeletal information 320 into interpolated information or detected information, it is possible to improve the accuracy of behavior classification. In other words, by adding the missing position information 1100 to the principal component containing the interpolated error, information of another dimension is added to the principal component, and a boundary plane 1220 for classification is provided on the new dimension, which facilitates behavior classification. Accuracy improves.

Note that the missing position information 1100 in FIG. 12 is expressed by filling in the plot points 1204 and 1205, so it is treated as 1-bit information. However, for example, the missing position information 1100 is a variable prepared for the number of skeleton points. In this case, it is treated as multivalued information. In this case, each action is plotted on a multidimensional space, and the missing position information 1100 enables more detailed action classification.

Note that the behavior is held in the teacher signal DB 104 as a combination of learning data (human skeletal information 320 and joint angles 370) and behavior, and is changed by the processing of the defect control unit 421 and the defect position information generation unit 422. It's not a thing. The defect control unit 421 and the defect generation unit 402 intentionally generate a defect, and the defect position information generation unit 422 generates the defect position information 1100 of the generated defect in the principal component containing the error resulting from the defect and interpolation. This is also to correctly perform behavior classification using a new axis that takes into account the missing position information 1100.

The behavior recognition unit 457 recognizes behavior using the behavior classification model learned and generated by the behavior learning unit 406. Specifically, for example, the client 102 interpolates missing information if the missing information interpolation unit 461 has missing information regarding the newly input skeleton information 320. The behavior recognition unit 457 applies principal component analysis to the interpolated skeletal information 320 and inputs the newly generated principal component and missing position information 1100 to the behavior classification model. Thereby, the behavior recognition unit 457 determines to which region the newly input skeleton information 320 belongs according to the boundary line 1210 and boundary plane 1220 set by the behavior classification model, and recognizes the behavior according to the determined region. do.

FIG. 13 is a graph showing changes in the cumulative contribution rate used by the principal component analysis unit 404 when determining the number of dimensions. The cumulative contribution rate is a measure of how much the newly generated principal components represent the amount of information in the original data. Therefore, even if the number of principal components is increased to increase the number of dimensions for behavior classification, if there is no significant change in the cumulative contribution rate, no significant improvement in accuracy can be expected.

Therefore, the principal component analysis unit 404 determines the number of dimensions by using as many principal components as are necessary to exceed a predetermined cumulative contribution rate threshold. For example, if the predetermined cumulative contribution rate threshold is "0.8", the condition is satisfied as long as the second principal component is reached, so the number of dimensions k here is assumed to be "2", and the first principal component and 2 principal components to the behavior learning unit 406.

Note that the principal component group output to the behavior learning unit 406 does not need to be limited to ascending order starting from the first principal component. For example, the principal component analysis unit 404 may determine a combination of ordinal numbers k of principal components that does not exceed a predetermined cumulative contribution rate threshold and has a maximum cumulative contribution rate. Further, the principal component analysis unit 404 may select such a combination of ordinal numbers k of principal components from a group of principal components applied to the behavior classification model. In this way, the principal component groups output to the behavior learning unit 406 are not limited to principal component groups in ascending order from the first principal component.

Note that the method for selecting one or more principal component groups that the principal component analysis section 454 outputs to the behavior recognition section 457 is the same as the method used by the principal component analysis section 404.

<Learning process>
FIG. 14 is a flowchart illustrating a detailed processing procedure example of learning processing by the server 101 (learning device) according to the first embodiment. The server 101 uses the teacher signal acquisition unit 401 to acquire one or more teacher signals used for learning from the teacher signals acquired from the teacher signal DB 104 (step S1400).

In the server 101, the loss control unit 421 sets skeleton points to be lost according to a random number or a predetermined number, and the loss generation unit 402 adds the skeleton information 320 in the teacher signal acquired in step S1400 according to the setting result. In contrast, the skeleton points are deleted, and the deleted skeleton information 320 is updated as the skeleton information 320 in the teacher signal (step S1401).

The server 101 uses the defect position information generation unit 422 to generate the position information of the skeleton point to be deleted, which is set by the defect control unit 421, as the defect position information (step S1410).

The server 101 uses the missing information interpolation unit 423 to interpolate missing information (the missing skeleton information or pieces of skeleton information 320) from the non-missing information (step S1411). The teacher signal processed by the missing information interpolation unit 423 is referred to as an updated teacher signal.

The server 101 uses the skeleton information processing unit 403 to execute skeleton information processing for each updated teacher signal (step S1402). Specifically, for example, the server 101 executes processing by a joint angle calculation unit 501, a movement amount calculation unit 502, and a normalization unit 503.

FIG. 15 is a flowchart illustrating a detailed processing procedure example of skeleton information processing according to the first embodiment. The server 101 uses the joint angle calculating unit 501 to calculate the joint angle 370 from the skeleton information 320 in the updated teaching signal for each updated teaching signal (step S1501). Next, in the server 101, the movement amount calculation unit 502 calculates the movement amount between frames for each updated teacher signal from the skeleton information 320 in the updated teacher signal (step S1501).

Then, the server 101 uses the normalization unit to exclude absolute position information from the skeletal information 320 for each updated teacher signal, and performs normalization such that the size of the skeletal information 320 is constant (step S1403 ). As a result, the joint angle 370, the amount of movement between frames, and the normalized skeletal information 320 are obtained for the updated teacher signal. Then, the process moves to step S1403 in FIG.

Returning to FIG. 14, the server 101 uses the normalized skeletal information 320, the joint angles 370, and the amount of movement between frames as input data, and executes a principal component analysis using the principal component analysis unit 404. or generate multiple principal components. Among the generated principal components, decide how many dimensions of the principal components to be used for learning in order of the highest variance value, and select the principal components up to the determined k dimension (1st principal component to k-th principal component) with the highest variance value. They are selected in order (step S1403).

In step S1407, if there is skeletal information 320 that has not been deleted yet for the skeletal information 320 that has been deleted in step S1401 (step S1407: No), the process returns to step S1401, and the server 101 determines whether or not the skeletal information 320 has been deleted so far. Skeletal points that do not exist are deleted (step S1401).

On the other hand, if all of the skeleton information 320 is deleted (step S1407: Yes), the process advances to step S1408. However, the determination of the process in step S1407 is not limited to this, and the server 101 may determine whether to return to step S1401 or proceed to step S1408 according to a predetermined number of repetitions. Alternatively, skeletons to be deleted may be determined in advance, and the server 101 may determine whether to return to step S1401 or proceed to step S1408 depending on whether all the predetermined skeletons have been deleted.

In step S1408, if there is a teacher signal that has not been selected yet among the teacher signals selected in step S1400 (step S1408: No), the server 101 selects a teacher signal that has not been selected so far (step S1400). . On the other hand, if all teacher signals have been selected (step S1408: Yes), the process advances to step S1405. However, the determination of the process in step S1408 is not limited to this, and the server 101 may determine whether to return to step S1400 or proceed to the process in step S1405 according to a predetermined number of repetitions.

Note that the processing in steps S1407 and S1408 causes repeated processing, and the principal component analysis processing in step S1403 occurs as many times as the number of repetitions, but the results of the principal component analysis processed each time are additionally held, All values are held in principal component analysis section 404. When all the repetitive processing is completed, all values held in the principal component analysis section 404 are output to the behavior learning section 406.

The server 101 uses the behavior learning unit 406 to use the principal component input from the principal component analysis unit 404 and the missing position information input from the missing position information generation unit 422 as explanatory variables, and uses the behavioral information in the updated teacher signal as an objective variable. , and as a result of the learning, a behavior classification model is generated (step S1405).

<Action recognition processing>
FIG. 16 is a flowchart illustrating an example of a behavior recognition processing procedure by the client 102 (behavior recognition device) according to the first embodiment. The client 102 uses the skeleton detection unit 451 to detect the skeleton information 320 of the person appearing in the analysis target data acquired from the sensor 103 (step S1600). Next, the client 102 causes the missing position information generation unit 462 to generate position information of the skeleton points that could not be detected due to occlusion or the like out of the detected skeleton information 320 as missing position information (step S1601).

The client 102 uses the missing information interpolation unit 423 to interpolate missing information from non-defective information (step S1610).

Next, the client 102 causes the skeleton information processing unit 453 to perform skeleton information processing on the skeleton information 320 with the missing information interpolated in step S1610, similar to the process in step S1402 (step S1602). Specifically, for example, the client 102 executes processing (steps S1501 to S1503) by the joint angle calculation unit 501, movement amount calculation unit 502, and normalization unit 503, as shown in FIG.

Next, the client 102 uses the skeletal information 320 normalized in step S1602, the joint angles 370, and the amount of movement between frames as input data to cause the principal component analysis unit 454 to perform principal component analysis, and then generate the principal components of Regarding the generated principal components, the principal component analysis unit 454 determines the first to kth principal components as principal components to be used in action recognition (step S1606), based on the ordinal number k determined in step S1403. (Step S1603).

Next, the client 102 uses the behavior recognition unit 457 to obtain information from the sensor 103 based on the behavior classification model learned in step S1305, the principal component generated in step S1603, and the missing position information generated in step S1601. The behavior of the person reflected in the data to be analyzed is recognized (step S1606). The client 102 may transmit the recognition result in step S1606 to the server 101, and may also use the recognition result to control devices connected to the client 102.

For example, if the analysis environment in which the sensor 103 is installed is a factory, the behavior recognition system 100 can be applied to monitoring the work of workers in the factory, inspecting products for defects, etc. using the recognition results. . When the analysis environment is a train, the behavior recognition system 100 can be applied to monitoring passengers on the train, monitoring in-car equipment, detecting disasters such as fire, etc. using the recognition results.

In this manner, according to the first embodiment, multiple types of behaviors to be recognized can be recognized with high accuracy. Furthermore, it is not necessary to generate multiple behavior classification models in order to perform accurate behavior recognition even in a partially missing shape. Thereby, it is possible to shorten the learning period and reduce the storage area of the behavior classification model. In particular, even if some of the skeleton points 300 to 317 are missing due to occlusion or the like, multiple types of behaviors can be recognized with high accuracy according to the missing skeleton points without increasing the number of behavior classification models.

The second embodiment will be described focusing on the differences from the first embodiment. Note that the same points as in the first embodiment are given the same reference numerals, and the explanation thereof will be omitted.

FIG. 17 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the second embodiment. In the second embodiment, the teacher signal DB 104 holds position information of skeleton points to be deleted by the loss control unit 421, and the position information is output to the loss control unit 421.

The possibility that the skeleton detection unit 451 misses detection when detecting the human skeleton information 320 appearing in the analysis target data acquired from the sensor 103 depends on the characteristics of the skeleton and the analysis target data. For example, there is a higher possibility of detection failure at skeletal points at the extremities of the human body, such as the wrist or ankle. Also, for example, if the data to be analyzed is from a high angle, there is a high probability that the skeleton points of the lower body will be hidden by the upper body and will not be detected. Missing detection of skeletal points on the half body or left half of the body is likely to occur.

If these characteristics are known in advance, it is possible to improve the accuracy of action recognition by generating such deficits and learning intensively. For this reason, the teacher signal DB 104 stores the position information of the skeleton points to be deleted by the deletion control unit 421. The positional information of the skeleton points to be deleted by the deletion control unit 421 is referred to as deletion suggestion information. The loss control unit 421 acquires loss suggestion information from the teacher signal DB 104 and outputs it to the loss generation unit 402. The defect generation unit 402 causes a skeleton point to be deleted based on the position information of the skeleton point to be deleted. Thereby, it is possible to accurately recognize behavior even in a situation where a specific defect is likely to occur in advance.

As described above, according to the second embodiment, in a situation where a specific defect is likely to occur, the information is stored in the teacher signal DB 104 in advance, and the stored information is used to intentionally generate a defect to perform learning. This allows for highly accurate behavior recognition.

Embodiment 3 will be described focusing on the differences between Embodiment 1 and Embodiment 2. Note that the same points as in Examples 1 to 3 are given the same reference numerals, and the explanation thereof will be omitted.

FIG. 18 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB 104 according to the third embodiment. In the third embodiment, the teacher signal DB 104 holds environmental information 1801 and 1811 to be used as explanatory variables in learning by the behavior learning unit 406. The environment information 1801 and 1811 is information about the environment from which the data to be analyzed is acquired, such as information about the angle at which the sensor 103, which is the source to acquire the data to be analyzed, is installed, and information about objects that exist around the sensor 103. There is shape information, etc. In addition, information on the acquisition time period of the data to be analyzed and variables generated independently by the developer may be added, and the environment information 1801 and 1811 is not limited to these.

FIG. 19 is an explanatory diagram showing data outputted to the behavior learning unit 406 by the principal component analysis unit 404 and the missing position information generation unit 422 according to the third embodiment. The environment information 1900 (1801, 1811) held by the teacher signal DB 104 is output to the principal component analysis unit 404 along with the skeleton information 320. The environmental information 1900 is not subjected to principal component analysis processing, but is output to the behavior learning unit 406 as an explanatory variable together with the principal component 1101 and missing position information 1100 generated by the principal component analysis unit 404.

FIG. 20 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the third embodiment. In the third embodiment, the teacher signal acquisition section 401 is changed to a teacher signal acquisition section 2000, and an environmental information detection section 2001 is added.

The teacher signal acquisition unit 2000 acquires one or more teacher signals to be used for learning regarding the teacher signal including environmental information from the teacher signal DB 104, and outputs the selected teacher signal to the loss generating unit 402. Note that the environmental information 1900 is input to the behavior learning unit 406 and used to generate a behavior classification model.

The environmental information detection unit 2001 detects environmental information 1900 defined in the teacher signal DB 104 from the analysis target data acquired from the sensor 103, and outputs it to the skeleton detection unit 451. Note that the environmental information 1900 may depend on the installation position of the sensor 103, etc. In this case, the environmental information detection unit 2001 stores in advance the environmental information 1900 when the sensor 103 is installed, not the data to be analyzed. The environmental information 1900 may be output to the skeleton detection unit 451, and the method of detecting the environmental information 1900 is not limited.

<Action recognition processing>
FIG. 21 is a flowchart illustrating an example of a behavior recognition processing procedure by the client 102 (behavior recognition device) according to the third embodiment. In the third embodiment, in FIG. 16, environmental information detection processing (step 3300) is executed prior to skeleton detection (step S1600). The client 102 uses the environmental information detection unit 2001 to detect the environmental information 1900 defined in the teacher signal DB 104 from the analysis target data acquired from the sensor 103 (step S2100). After this, steps S1600 to S1606 are executed. The environmental information 1900 is used as an explanatory variable in behavior recognition by the behavior learning unit 406 (step S1606).

In this way, according to the third embodiment, the teacher signal DB 104 holds the environmental information 1900, and by adding it as an explanatory variable during behavior learning, the environmental information 1900 is added to the principal component as information of another dimension in addition to the missing position information. , boundaries for classification on a new dimension are set. Therefore, it is possible to improve the accuracy of behavior classification.

Example 4 will be explained focusing on the differences from Examples 1 to 3. Note that the same points as in Examples 1 to 3 are given the same reference numerals, and the explanation thereof will be omitted.

FIG. 22 is a block diagram showing an example of the functional configuration of the skeleton information processing section according to the fourth embodiment. In the fourth embodiment, the skeleton information processing units 403 and 453 include a mutual information normalization unit 2204. The mutual information normalization unit 2204 normalizes the range of the skeletal information 320, joint angles 370, and inter-frame movement amounts to be within a certain range, which are output to the principal component analysis unit 404.

The range of the skeleton information 320 and the amount of movement between frames depends on the resolution of the data to be analyzed. On the other hand, the range of the joint angle 370 is from 0 to 2π, or from 0 degrees to 360 degrees. If there is a large difference in the range of the data to be subjected to principal component analysis, the influence on the principal components of the original data may be biased depending on the data type.

In order to eliminate this bias, the mutual information normalization unit 2204 performs normalization to bring the value range of the data to be applied to the principal component within a certain range. For example, the mutual information normalization unit 2204 unifies the range of the original data from 0 to 2π using the skeleton information 320 according to the following formulas (21) to (22) and the inter-frame movement amount according to the following formula (23).

However, the normalization method executed by the mutual information normalization unit 2204 is not limited to this; for example, the mutual information normalization unit 2204 may adjust the joint angle 370 The value range of may be normalized to a constant value.

FIG. 23 is a flowchart illustrating a detailed processing procedure example of the skeleton information processing unit according to the fourth embodiment. In the fourth embodiment, in the skeleton information processing (steps S1402, S1602), the client 102 performs mutual information normalization (step S2304) after normalization (step S1503). In mutual information normalization (step S2304), the possible value ranges of the skeletal information 320, joint angles 370, and inter-frame movement amounts normalized by the normalization unit are normalized to a constant value.

In this way, according to the fourth embodiment, by unifying the range of possible values of the original data (skeletal information 320, joint angles 370, amount of movement between frames) on which principal component analysis is performed, a wide range of values can be achieved. It is possible to eliminate bias in the influence of specific data on the principal components and to discriminate between multiple types of behavior with high accuracy.

Example 5 will be explained focusing on the differences from Examples 1 to 4. Note that the same reference numerals are given to the same points as in Examples 1 to 4, and the explanation thereof will be omitted.

FIG. 24 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the fifth embodiment. In the fifth embodiment, the principal component analysis units 404 and 454 are changed to dimension reduction units 2400 and 2401. Dimensionality reduction is a process of reducing the number of original variables or the number of original dimensions while maintaining the original amount of information as much as possible. This is a concept that includes component analysis.

The dimension reduction unit 2400 executes dimension reduction using the normalized skeleton information 320, joint angles 370, and inter-frame movement amounts as input data among the teacher signals acquired from the skeleton information processing unit 403. One or more variables are generated and output to the behavior learning unit 406.

The dimension reduction methods performed by the dimension reduction unit 2400 include SNE (Stochastic Neighbor Embedding), t-SNE (t-Distributed Stochastic Neighbor Embedding), and UMAP (Uniform Manifold). Approximation and Projection), Isomap, LLE (Locally Linear Embedding), Techniques include Laplacian Eignmap, LargeVis, and Diffusion Map. The dimension reduction unit 2400 may perform dimension reduction by combining t-SNE or UMAP with principal component analysis or independent component analysis. Below, each dimension reduction method and a dimension reduction method performed by combining each method will be explained.

The SNE processing will be explained using the following equations (24) to (28).

Let the similarity between two x coordinate values 322 (input data) x _i and x _j be the conditional probability p _{j |i} of selecting x _j as a neighbor when x _i is given. The conditional probability p _{j |i} is shown in the above equation (24). At this time, it is assumed that x _j is selected based on a normal distribution centered on x _i . Next, the similarity between the two y-coordinate values 323 (principal components) of y _i and y _j after dimension reduction is also calculated by the above equation (25), as well as the similarity between x _i and x _j before dimension reduction. Let the conditional probability q _j|i be However, the variance of the coordinate values after dimension reduction is fixed at 1/√2 to simplify the equation.

If y for dimension reduction is generated so as to maintain the distance relationship before and after dimension reduction, it is possible to reduce dimension while maintaining the amount of information as much as possible. In order to perform dimension reduction while suppressing a reduction in the amount of information, the dimension reduction unit 2400 performs processing so that p _{j |i} = q _{j |i} . KL divergence, which is a measure of how similar two probability distributions are, is used for dimension reduction.

The above equation (26) shows an equation in which the probability distribution before and after dimension reduction is adapted using the KL divergence as a loss function. The dimensionality reduction unit 2400 minimizes the loss function expressed in equation (26) using stochastic gradient descent. This gradient varies y _i using the above equation (27) in which the loss function is differentiated by y _i . The updating formula for this variation is shown by the above equation (28).

As described above, the above formula (28) is updated while varying y _i , and dimension reduction is performed by obtaining y _i that minimizes the above formula (27), and a new variable is obtained. However, in the case of SNE, unlike principal component analysis, the number of dimensions (variables) after reduction is two or three types due to processing characteristics. Therefore, when performing dimension reduction by SNE, a predetermined number of dimensions (variables) is output to the behavior learning unit 406.

However, in SNE, it is difficult to minimize the loss function, and there is a problem that the skeleton points specified by the x-coordinate value 322 and the y-coordinate value 323 become dense when trying to maintain equidistantness during dimension reduction. t-SNE is a method for solving this problem.

The t-SNE processing will be explained using the following equations (29) to (33).

To simplify loss function minimization, we make the loss function symmetric. In the loss function symmetrization process, the distance between x _i and x _j is represented by a joint probability distribution p _ij , as shown in equation (29) above. p _j|i is similar to the above equation (24) and can be expressed by the above equation (30). Further, the distance between y _i and y _j after dimension reduction is expressed by the joint probability distribution q _ij shown in the above equation (31).

The distance between points after dimension reduction assumes Student's t distribution. The Student's t distribution is characterized by a higher probability of the existence of values that deviate from the mean value than the normal distribution, and this feature allows for a long distance distribution of the distance between data after dimension reduction. becomes possible.

In t-SNE, the dimensionality reduction unit 2400 performs dimensionality reduction by minimizing the loss function shown in the above equation (32) using pij and q _ij calculated using the above equations (29) to (31). . The dimension reduction unit 2400 uses the stochastic gradient descent method shown in the above equation (33) similarly to SNE to minimize the loss function.

As described above, by obtaining y _i that minimizes the above equation (33), the dimension reduction unit 2400 performs dimension reduction and obtains a new variable. Like SNE, t-SNE also has two or three types of dimensions (variables) after reduction due to processing characteristics. Therefore, when performing dimension reduction by t-SNE, a predetermined number of dimensions (variables) is output to the behavior learning unit 406.

t-SNE maintains the high-dimensional local structure before dimension reduction and captures the global structure as much as possible, so it is possible to reduce the dimension accurately, but depending on the number of dimensions before dimension reduction. There is a problem that calculation time increases. UMAP is a method for solving the calculation time problem of dimension reduction. The UMAP processing will be explained using the following equations (34) to (36).

Among all possible values A, there is a high-dimensional set X (formula (34) above). Let μ be a membership function that, when arbitrary data is extracted from A, outputs the degree to which it is included in the set X in a range of 0 to 1. For input X shown in equation (1) above, Y shown in equation (2) above is prepared. Y is a set of m (<p) points existing in a space with a lower dimension than X, and is a set of data after dimension reduction. Then, by setting the membership function of Y to ν, the dimension reduction unit 2400 performs dimension reduction by determining Y such that the above equation (36) is minimized, and obtains a new variable.

When performing dimension reduction using UMAP, the dimension reduction unit 2400 may output a predetermined number of dimensions (variables) to the behavior learning unit 406, similar to SNE and t-SNE, or The number of dimensions (variables) such that the membership function ν of is greater than or equal to a predetermined range may be output to the behavior learning unit 406 as the required number of dimensions.

Isomap processing will be explained. The dimensionality reduction unit 2400 calculates the shortest distance between nearby data for arbitrary data, and performs dimensionality reduction by expressing the calculated distance in a geodesic distance matrix using multidimensional scaling (MDS). Get variable. When performing dimension reduction using Isomap, the dimension reduction unit 2400 outputs a predetermined number of dimensions (variables) to the behavior learning unit 406.

LLE will be explained using the following formulas (37) to (43).

Points in the vicinity of x _i are approximately represented by the above equation (37) by linear combination. Here, by minimizing the above equation (39) under the constraint of the above equation (38), the approximate value of x _i before dimension reduction is determined. Next, regarding y _i after dimension reduction, the dimension reduction unit 2400 minimizes the above equation (40) in order to maintain the linear adjacency relationship of x _i as much as possible even after dimension reduction. This solution is obtained as in the above equation (42) by extracting the eigenvectors of the above equation (41) from the second smallest eigenvalue v _i to the (d+1)th v _d , and the dimension reduction unit 2400 As shown in equation (43), y _i after dimension reduction is obtained.

When implementing dimension reduction by LLE, the dimension reduction unit 2400 outputs a predetermined number of dimensions (variables) to the behavior learning unit 406, and the behavior learning unit 406 selects the variables to be used according to the predetermined number of dimensions. All you have to do is decide on the number of .

The processing of the Laplacian eigenmap will be explained using the following equations (44) to (49).

Each edge x _i x _j of the neighborhood graph generated by the data before dimension reduction is assigned to the above equation (44) or the above equation (45). Introducing the graph Laplacian of the above equation (46) for the assigned weights, and extracting the eigenvectors of the graph Laplacian (the above equation (47)) from the second smallest eigenvalue v _i to the (d+1)th v _d is obtained according to the above equation (48), and the dimension reduction unit 2400 obtains the value y _i after dimension reduction according to the above equation (49).

When performing dimension reduction using the Laplacian eigenmap, the dimension reduction unit 2400 outputs a predetermined number of dimensions (variables) to the behavior learning unit 406.

The processing of LargeVis will be explained. LargeVis is a method that improves the calculation time of t-SNE. In t-SNE, the distance between data points is determined, so the calculation time increases depending on the number of data. In LargeVis, the dimension reduction unit 2400 divides data into regions using a K-NN graph from neighboring data, and performs dimension reduction for each data model divided into regions using a method similar to t-SNE.

When implementing dimension reduction using LargeVis, the dimension reduction unit 2400 outputs a predetermined number of dimensions (variables) to the behavior learning unit 406.

The diffusion map will be explained using equations (50) to (55) below.

A weight W _ij is assigned to each edge x _i x j of the neighborhood graph composed of x _i before dimension reduction and x _j in the neighborhood, and this is normalized to create an N×N transition probability matrix shown in equation (50) above. Make P. Let p _t (x _i x _j ) represent the probability of starting from x _i and arriving at x _j after t steps by a random walk on the graph represented by P. From the properties of the transition matrix, p _t (x _i x _j ) converges to a stationary distribution φ ₀ (x _j ) at t→∞. At this time, the diffusion distance of the point x _i x _j is defined by the above equation (51). Let the eigenvalue of the transition probability matrix P be the above equation (52), and the eigenvector be the above equation (53). At this time, the above formula (54) holds true. Since the absolute value of λ _i is less than or equal to 1, the dimension reduction unit 2400 takes the eigenvectors up to an appropriate dimension d(t) smaller than N, performs dimension reduction according to the above equation (55), and creates a new variable. get.

When performing dimension reduction using a diffusion map, a predetermined number of dimensions (variables) is output to the behavior learning unit 406.

The dimension reduction unit 2400 may perform a combination of the principal component analysis, independent component analysis, t-SNE, UMAP, Isomap, LLE, Laplacian eigenmap, LargeVis, diffusion map, etc. described above. For example, the dimension reduction unit 2400 uses principal component analysis to reduce dimensions to 10 dimensions for high-dimensional data with 36 dimensions or 36 variables, and then uses UMAP to reduce dimensions to 2 dimensions. , the combination of methods used for dimensionality reduction is not limited. In this way, by combining various methods during dimensionality reduction, we can expect a composite effect on performance and calculation time.

Furthermore, these dimension reduction methods are not limited to the scope described in Example 5, and may include, for example, simply counting, subtracting, multiplying, or dividing high-dimensional information, or convolving it according to predetermined coefficients. However, the dimension reduction method is not limited as long as it is a method that generates high-dimensional data or multiple variables, lower-dimensional data, or a small number of variables, such as the method described in Example 5.

The dimension reduction unit 2401 has the same function as the dimension reduction unit 2400. The dimensionality reduction unit 2401 executes the same processing as the dimensionality reduction unit 2400 on the output data from the skeleton information processing unit 453, and generates one or more new variables that are smaller than before the dimensionality reduction. . Further, the dimension reduction unit 2401 outputs a new variable determined from the contribution rate and cumulative contribution rate generated together with the principal component to the behavior recognition unit 457.

As described above, according to the fifth embodiment, by changing the dimension reduction method, it is possible to reduce the dimension effectively or reduce the calculation time according to the data acquired from the skeleton information processing unit 403, and to reduce the complexity. behavior can be determined with high accuracy.

Example 6 will be explained focusing on the differences from Examples 1 to 5. Note that the same reference numerals are given to the points common to Examples 1 to 5, and the explanation thereof will be omitted.

FIG. 25 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the sixth embodiment. In the sixth embodiment, the behavior learning section 406 and the behavior recognition section 457 are changed to the behavior learning section 2500 and the behavior recognition section 2501. A detailed method by which the behavior learning unit 2500 and the behavior recognition unit 2501 classify behaviors will be explained using FIGS. 26 to 28.

FIG. 26 is an explanatory diagram showing a decision tree, which is a basic method by which the behavior learning unit 2500 and the behavior recognition unit 2501 classify behaviors. We will explain the behavior classification method using decision trees. In the decision tree, for each behavior in the variable space newly generated after dimension reduction, (a) the boundary line 2610 is generated.

(a) A method of generating the boundary line 2610 will be explained. The decision tree classifies behaviors step by step so that the impurity of a variable group 2621 input from a population 2620 including behaviors (plot points 1200 to 1203) is minimized. In the first stage, variable group 2621 is classified into variable group 2622 and variable group 2623 on the second variable axis, and in the second stage, variable group 2622 and variable group 2623 are classified into variable groups 2624 to 2623 on the first variable axis. Classified into 2627. (a) A boundary line 2610 is generated using the discriminant obtained in the process of classifying so that the impurity is minimized. Note that there is no limitation on which axis the behavior is classified at each stage, and there is no limitation on the behavior classification on each axis by a prescribed number of times such as once.

FIG. 27 is an explanatory diagram showing a detailed method for developing classification using a decision tree. Decision trees include level-wise 2700, which grows a decision tree for each level (depth), and leaf-wise 2701, which grows a decision tree for each leaf (data group after branching). Learning by stacking classifiers such as decision trees is called ensemble learning.

FIG. 28 is an explanatory diagram showing ensemble learning and a method used by the behavior learning unit 2500 and the behavior recognition unit 2501 to classify behaviors. Ensemble learning includes bagging 2801 that uses classification trees such as decision trees in parallel, and boosting 2802 that updates learning results by inheriting previous results. The random forest of the first embodiment is a method that uses bagging 2801 for decision trees, and the behavior learning unit 2500 and the behavior recognition unit 2501 of the sixth embodiment are a classification method that uses boosting 2802.

When the behavior learning unit 2500 learns the behavior and the behavior recognition unit 2501 classifies the behavior, it is possible to grow each decision tree levelwise and classify the input variables by boosting multiple decision trees. Alternatively, each decision tree may be grown leafwise, and input variables may be classified by boosting, which overlaps a plurality of decision trees.

Note that when employing boosting, in which each decision tree is grown levelwise and multiple decision trees are stacked, as a behavior classification method, it may be implemented using the software library xgboost. On the other hand, when employing boosting, in which each decision tree is grown leafwise and multiple decision trees are stacked, as a behavior classification method, it may be implemented using the software library LightGBM. However, implementation methods are not limited to these.

In this way, according to the sixth embodiment, by using boosting as the behavior classification method and overlapping a plurality of decision trees, complex behaviors can be determined with high accuracy.

Example 7 will be explained focusing on the differences from Examples 1 to 6. Note that the same reference numerals are given to the points common to Examples 1 to 6, and the explanation thereof will be omitted.

FIG. 29 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the seventh embodiment. In the seventh embodiment, the dimensionality reduction unit 2400, behavior learning unit 406, dimensionality reduction unit 2401, and behavior recognition unit 457 are changed to dimensionality reduction unit 2900, behavior learning unit 2901, dimensionality reduction unit 2903, and behavior recognition unit 2904. Ru.

The dimensionality reduction unit 2900 performs dimensionality reduction using any of the methods of Examples 1 to 6 according to a predetermined number of dimensions, and outputs the variable after dimensionality reduction to the behavior learning unit 2901.

The behavior learning unit 2901 generates a boundary line for behavior classification by machine learning from the given behavior type together with the obtained variable after dimension reduction, and generates a behavior classification model. At this time, the behavior classification accuracy, which indicates how accurately the behavior can be predicted, is calculated for the generated behavior classification model.

The behavior learning unit 2901 may calculate the behavior classification accuracy using the variables used to generate the behavior classification model. The behavior learning unit 2901 may not use some of the variables acquired from the dimension reduction unit 2900 for behavior classification model generation, and may calculate behavior classification accuracy using variables that are not used for behavior classification generation. However, the method of calculating behavior classification accuracy is not limited to these. If the calculated behavior classification accuracy is higher than the predetermined accuracy, the behavior learning unit 2901 outputs the generated behavior classification model to the behavior recognition unit 2904. In addition, at this time, the behavior learning unit 2901 outputs to the dimension reduction unit 2900 that the obtained number of dimensions and behavior classification accuracy are passed.

On the other hand, if the calculated behavior classification accuracy is lower than the predetermined accuracy, the behavior learning unit 2901 outputs to the dimension reduction unit 2900 that the behavior classification accuracy has failed. However, if behavior classification models are generated using all settable dimensions (variables) and the behavior classification accuracy fails in all of them, the behavior learning unit 2901 The behavior classification model with the highest behavior classification accuracy among the models is output to the behavior recognition unit 2904, and the number of dimensions (variables) used when outputting is output to the behavior learning unit 2901 together with all learning completion information. Note that without determining the behavior classification accuracy for determining pass/fail, the behavior learning unit 2901 performs learning with all settable dimensions, calculates the behavior classification accuracy, and then creates a behavior classification model according to the calculated behavior classification accuracy. The determined behavior classification model may be determined to be acceptable.

If the dimension reduction unit 2900 acquires pass or all learning completion information according to the pass/fail information and all learning completion information acquired from the behavior learning unit 2901, the dimension reduction unit 2900 outputs the acquired dimension number information to the dimensionality reduction unit 2903, and If it passes, the number of dimensions used for dimension reduction is changed, dimension reduction is performed again, and the generated variables are output to the behavior learning unit 2901.

The dimension reduction unit 2903 performs dimension reduction on the data obtained from the skeleton information processing unit 453 according to the number of dimensions k (variable) obtained from the dimension reduction unit 2900 using the dimension reduction methods of Examples 1 to 6. , and outputs the generated variables to the behavior recognition unit 2904.

The behavior recognition unit 2904 executes behavior recognition using the behavior classification model that the behavior learning unit 2901 determines to be acceptable and the variables input from the dimension reduction unit 2903.

Note that the behavior classification accuracy calculated by the behavior learning unit 2901 may be regarded as the contribution rate described in the first embodiment. For example, by associating the obtained variable after dimension reduction with the behavior classification accuracy calculated using it, the calculated behavior classification accuracy is the contribution rate to the original information of the variable after dimension reduction used for calculation. do. The dimensionality reduction unit 2900 determines which of the variables after dimensionality reduction will be used for control according to the contribution rate estimated in this way.

<Learning process>
FIG. 30 is a flowchart illustrating a detailed processing procedure example of learning processing by the server 101 (learning device) according to the seventh embodiment. The server 101 uses the dimension reduction unit 2900 to determine the number of dimensions k. At this time, when performing dimension reduction for the first time, a predetermined number of dimensions k is determined, and when dimension reduction is performed for the second time or later, a previously undetermined number of dimensions k is determined. The dimension reduction unit 2900 performs dimension reduction according to the determined number of dimensions k, and generates a new variable (number of dimensions k after dimension reduction) (S3001).

In step S3002, the server 101 makes a pass/fail judgment on the behavior classification accuracy acquired from the behavior learning unit 2901. If it passes, the learning process ends, and if it fails, the process returns to step S3001.

In this manner, according to the seventh embodiment, by changing the number of dimensions in accordance with the target behavior classification accuracy and repeating dimension reduction, complex behaviors can be determined with high accuracy.

Furthermore, the behavior recognition device and learning device of Examples 1 to 7 described above can also be configured as shown in [1] to [14] below.

[1] The behavior recognition device (client 102) having a processor 201 that executes a program and a storage device 202 that stores the program performs component analysis (principal component analysis or The processor 201 can access a behavior classification model learned using the behavior of the learning target and a component group related to the learning target obtained from the shape of the learning target (skeletal information 320) by independent component analysis). includes a detection process that detects the shape of the recognition target (skeletal information 320) from the analysis target data obtained from the sensor 103, and missing position information indicating the position of the missing part of the skeleton information 320 detected by the detection process. a process for generating missing position information; and an interpolation process for interpolating the missing location, interpolating the missing location from the non-missing information after interpolation, and updating the non-missing information after interpolation as the shape of the recognition target. , a component analysis process that generates the same number of component groups related to the recognition target as component groups related to the learning target based on the shape of the recognition target interpolated by the interpolation process, and the behavior classification model; A behavior recognition process is executed to output a recognition result indicating the behavior of the recognition target by inputting the component group related to the recognition target generated by the component analysis process and the missing position information.

As a result, compared to the case where a specific behavior classification model is selected from a plurality of behavior classification models by component analysis and the behavior recognition process is executed, the usage amount of the storage device 202 is reduced and the amount of time required to output the recognition result is reduced. Processing speed can be increased. Furthermore, it is possible to highly accurately recognize multiple types of behaviors of recognition targets whose shapes are partially missing.

[2] In the behavior recognition device according to [1] above, in the behavior recognition process, the processor includes environmental information indicating the environment from which the analysis target data is obtained, a component group related to the recognition target, and the learning target. By inputting the behavior and the missing position information into the behavior classification model, a recognition result indicating the behavior of the recognition target is output.

As a result, a behavior classification model that takes environmental information into account is prepared, so that the behavior of the recognition target according to the environment of the recognition target can be recognized with high accuracy.

[3] In the behavior recognition device of [1] above, the behavior classification model uses dimension reduction (principal component analysis, independent component analysis, SNE (Stochastic Neighbor Embedding) or t- SNE (t-Distributed Stochastic Neighbor Embedding) or UMAP (Uniform Manifold Approximation and Projection) or Isomap or LLE (Locally L obtained from the shape of the learning object by inear Embedding or Laplacian Eignmap or LargeVis or Diffusion map). Learning is performed using a component group in ascending order from a first variable related to the learning target and the behavior of the learning target, and in the generation process, the processor performs interpolation by the interpolation process by the dimension reduction. Based on the shape of the recognition target, the same number of component groups in ascending order from the first variable regarding the recognition target as the component groups in ascending order from the first variable regarding the learning target are generated.

As a result, compared to the case where a specific behavior classification model is selected from a plurality of behavior classification models by dimension reduction and the behavior recognition process is executed, the usage amount of the storage device 202 is reduced and the amount of time required to output the recognition result is reduced. Processing speed can be increased.

[4] In the behavior recognition device of [1] above, each behavior classification model of the behavior classification model group is obtained from the shape of the learning target and the angles (joint angles 370) of a plurality of vertices constituting the shape. Learning is performed using a component group related to the learning target and the behavior of the learning target, and the processor 201 determines the shape of a plurality of vertices constituting the shape of the recognition target based on the shape of the recognition target. A calculation process for calculating an angle (joint angle 370) is executed, and in the component analysis process, the processor 201 calculates the shape of the recognition target and the angle of the vertex of the recognition target calculated by the calculation process. Based on the recognition target, a component group related to the recognition target is generated.

Thereby, multiple types of actions to be recognized can be recognized with high accuracy according to changes in shape caused by the angle of the vertices.

[5] In the behavior recognition device of [1] above, each behavior classification model of the behavior classification model group includes a component group related to the learning object obtained from the shape of the learning object and the amount of movement of the learning object; The processor 201 executes a calculation process of calculating the movement amount of the recognition target based on a plurality of shapes of the recognition target at different times, and In the component analysis process, the processor 201 generates a component group regarding the recognition target based on the shape of the recognition target and the amount of movement of the recognition target calculated by the calculation process.

Thereby, multiple types of actions of the recognition target can be recognized with high accuracy according to changes in shape over time due to movement.

[6] In the behavior recognition device of [1] above, the processor 201 executes a first normalization process to normalize the size of the shape of the recognition target, and in the component analysis process, the processor 201 A component group related to the recognition target is generated based on the shape of the recognition target after first normalization by the first normalization process.

Thereby, by improving the versatility of behavior classification, it is possible to suppress misrecognition.

[7] In the behavior recognition device of [1] above, the processor 201 executes a second normalization process to normalize the range of possible values of the shape of the recognition target and the angle of the vertex, and in the component analysis process, The processor 201 generates a component group regarding the recognition target based on the shape of the recognition target and the vertex angle (joint angle 370) after second normalization by the second normalization process.

As a result, it is possible to suppress bias in the range of different data types such as shape and angle, and it is possible to improve the accuracy of behavior recognition.

[8] In a learning device that includes a processor 201 that executes a program and a storage device 202 that stores the program, the processor 201 performs an acquisition process that acquires teacher data including the shape and behavior of a learning target; a deletion process that deletes the shape of the learning target acquired by the acquisition process; a deletion position information generation process that generates deletion position information indicating a position of a defective part deleted from the shape of the learning target by the deletion process; interpolation processing that interpolates from non-missing information that is a part of the shape of the learning target other than the missing part that has been deleted by the missing processing, and updating the non-missing information after interpolation as the shape of the learning target; A component analysis process that generates a component group related to the learning target based on the shape of the learning target interpolated by the interpolation process by component analysis (principal component analysis or independent component analysis) that generates statistical components by variable analysis. The behavior of the learning target is learned based on the component group related to the learning target generated by the component analysis process, the behavior of the learning target, and the missing position information, and the behavior of the learning target is learned. and a behavior learning process that generates a behavior classification model for classifying.

This eliminates the need for the behavior recognition device to select a specific behavior classification model from a plurality of behavior classification models.

[9] In the learning device of [8] above, the processor 201 performs a calculation process of calculating angles (joint angles 370) of a plurality of vertices constituting the shape of the learning target based on the shape of the learning target. In the component analysis process, the processor 201 generates a component group regarding the learning target based on the shape of the learning target and the angle of the vertex of the learning target calculated by the calculation process. .

This makes it possible to prepare a behavior classification model that responds to changes in shape caused by the angle of the vertices, so it is possible to accurately classify multiple types of behaviors in response to changes in shape caused by the angle of the vertices to be recognized. can be recognized.

[10] In the learning device of [8] above, the processor 201 executes a calculation process to calculate the movement amount of the learning target based on a plurality of shapes of the learning target at different times, and performs the component analysis process. Then, the processor 201 generates a component group regarding the learning object based on the shape of the learning object and the movement amount of the learning object calculated by the calculation process.

As a result, it is possible to prepare a behavior classification model that responds to changes in shape over time due to movement, so multiple types of behaviors can be recognized with high accuracy according to changes in shape over time due to movement. can do.

[11] In the learning device according to [8] above, the processor 201 executes a first normalization process to normalize the size of the shape of the learning target, and in the component analysis process, the processor 201 A component group related to the learning target is generated based on the shape of the learning target after first normalization by a first normalization process.

Thereby, by improving the versatility of behavior classification learning, it is possible to suppress erroneous learning.

[12] In the learning device according to [9] above, the processor 201 executes a second normalization process to normalize the range of possible values of the shape of the learning target and the angle of the vertex, and in the component analysis process, the The processor 201 generates a component group regarding the learning target based on the shape of the learning target and the angle of the vertex after second normalization by the second normalization process.

Thereby, it is possible to suppress bias in the range of different data types such as shape and angle, and it is possible to improve the accuracy of behavior classification learning.

[13] In the learning device according to [8] above, in the deletion process, the processor deletes a specific part of the shape of the learning target.

Thereby, it is possible to learn the shape of the learning target that is likely to be lost, and it is possible to improve the accuracy of behavior classification learning.

[14] In the learning device according to [8] above, when acquiring the shape of the learning target in the acquisition process, the processor 201 acquires environmental information at the time when the shape of the learning target was acquired, and A behavior classification model is generated based on the group, the behavior of the learning target, the missing position information, and the environment information.

Thereby, it is possible to generate a behavior classification model according to the shape of the learning target and environmental information, and it is possible to improve the accuracy of behavior classification learning that is tailored to various environments.

Note that the present invention is not limited to the embodiments described above, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the embodiments described above have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Further, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Further, the configuration of one embodiment may be added to the configuration of another embodiment. Furthermore, other configurations may be added to, deleted from, or replaced with some of the configurations of each embodiment.

Further, each of the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing an integrated circuit, for example, and the processor 201 may implement each function. It may be realized by software by interpreting and executing a program to be realized.

Information such as programs, tables, and files that realize each function is stored in storage devices such as memory, hard disks, and SSDs (Solid State Drives), or on IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs). It can be stored on a medium.

Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, almost all configurations can be considered interconnected.

100 Behavior recognition system 101 Server 102 Client 103 Sensor 104 Teacher signal DB
201 Processor 202 Storage device 320 Skeletal information 401, 2000 Teacher signal acquisition unit 402 Deficit generation unit 421 Deficit control unit 403, 453 Skeleton information processing unit 404, 454 Principal component analysis unit 406, 2500, 2901 Behavior learning unit 422, 462 Defective position Information generation section 423, 461 Missing information interpolation section 451 Skeleton detection section 455 Dimension number determination section 457, 2501, 2904 Action recognition section 501 Joint angle calculation section 502 Movement amount calculation section 503 Normalization section 2001 Environment information detection section 2204 Mutual information normalization conversion section 2400, 2401, 2900, 2903 dimension reduction section

Claims (15)

An action recognition device comprising a processor that executes a program, and a storage device that stores the program,
It is possible to access a behavior classification model learned using a component group related to the learning target obtained from the shape of the learning target by component analysis that generates statistical components by multivariate analysis, and the behavior of the learning target. ,
The processor includes:
Detection processing that detects the shape of the recognition target from the analysis target data,
a defective position information generation process that generates defective position information indicating a position of a defective part in the shape of the recognition target detected by the detection process;
An interpolation process of interpolating the missing part from non-defective information that is a part other than the missing part in the shape of the recognition target including the missing part, and updating the non-defective information after interpolation as the shape of the recognition target. ,
a component analysis process that generates the same number of component groups related to the recognition target as component groups related to the learning target, based on the shape of the recognition target interpolated by the interpolation process, through the component analysis;
a behavior recognition process that outputs a recognition result indicating the behavior of the recognition target by inputting a component group related to the recognition target generated by the component analysis process and the missing position information to the behavior classification model; ,
An action recognition device characterized by performing. The behavior recognition device according to claim 1,
In the behavior recognition process, the processor classifies environmental information indicating the environment from which the analysis target data is obtained, a component group related to the recognition target, the behavior of the learning target, and the missing position information into the behavior classification. outputting a recognition result indicating the behavior of the recognition target by inputting it to a model;
An action recognition device characterized by: The behavior recognition device according to claim 1,
The behavior classification model includes a group of components in ascending order from a first variable related to the learning target obtained from the shape of the learning target by dimension reduction that generates statistical components by multivariate analysis, and a behavior of the learning target. It is learned using
In the component analysis process, the processor calculates the same number of recognition targets as the number of component groups in ascending order from the first variable related to the learning target based on the shape of the recognition target interpolated by the interpolation process through the dimension reduction. generate a group of components in ascending order from the first variable with respect to
An action recognition device characterized by: The behavior recognition device according to claim 1,
The behavior classification model is trained using a component group related to the learning target obtained from the shape of the learning target and the angles of a plurality of vertices constituting the shape, and the behavior of the learning target,
The processor includes:
Based on the shape of the recognition target, performing a calculation process of calculating angles of a plurality of vertices forming the shape of the recognition target;
In the component analysis process, the processor generates a component group related to the recognition target based on the shape of the recognition target and the angle of the vertex of the recognition target calculated by the calculation process.
An action recognition device characterized by: The behavior recognition device according to claim 1,
The behavior classification model is trained using a component group related to the learning target obtained from the shape of the learning target and the amount of movement of the learning target, and the behavior of the learning target,
The processor includes:
executing a calculation process for calculating a movement amount of the recognition target based on a plurality of shapes of the recognition target at different times;
In the component analysis process, the processor generates a component group related to the recognition target based on the shape of the recognition target and the movement amount of the recognition target calculated by the calculation process.
An action recognition device characterized by: The behavior recognition device according to claim 1,
The processor includes:
performing a first normalization process to normalize the size of the shape of the recognition target;
In the component analysis process, the processor generates a component group regarding the recognition target based on the shape of the recognition target after normalization by the first normalization process.
An action recognition device characterized by: The behavior recognition device according to claim 1,
The processor executes a second normalization process to normalize the range of possible values of the shape of the recognition target and the angle of the vertex,
In the component analysis process, the processor generates a component group regarding the recognition target based on the shape and angle of the vertex of the recognition target after second normalization by the second normalization process.
An action recognition device characterized by: A learning device comprising a processor that executes a program and a storage device that stores the program,
The processor includes:
an acquisition process that acquires training data including the shape and behavior of the learning target;
a deletion process that deletes the shape of the learning target acquired by the acquisition process;
a defective position information generation process that generates defective position information indicating a position of a defective part deleted from the shape of the learning target by the defective process;
Interpolation processing that performs interpolation from non-missing information that is a location other than the missing location that has been deleted by the loss processing in the shape of the learning target, and updates the non-missing information after interpolation as the shape of the learning target;
Component analysis processing that generates a component group related to the learning object based on the shape of the learning object interpolated by the interpolation processing by component analysis that generates statistical components by multivariate analysis;
Learning the behavior of the learning target based on the component group related to the learning target generated by the component analysis process, the behavior of the learning target, and the missing position information, and classifying the behavior of the learning target. a behavioral learning process that generates a behavioral classification model;
A learning device characterized by performing the following. The learning device according to claim 8,
The processor executes a calculation process to calculate angles of a plurality of vertices forming the shape of the learning target based on the shape of the learning target,
In the component analysis process, the processor generates a component group regarding the learning target based on the shape of the learning target and the angle of the vertex of the learning target calculated by the calculation process.
A learning device characterized by: The learning device according to claim 8,
The processor executes a calculation process that calculates a movement amount of the learning target based on a plurality of shapes of the learning target at different times, and in the component analysis process, the processor calculates the shape of the learning target, generating a component group related to the learning target based on the movement amount of the learning target calculated by the calculation process;
A learning device characterized by: The learning device according to claim 8,
The processor executes a first normalization process to normalize the size of the shape of the learning target, and in the component analysis process, the processor executes the learning after the first normalization by the first normalization process. generating a component group related to the learning target based on the shape of the target;
A learning device characterized by: The learning device according to claim 9,
The processor executes a second normalization process that normalizes a range of possible values of the shape of the learning target and the angle of the vertex, and in the component analysis process, the processor generating a component group related to the learning target based on the shape of the learning target after transformation and the angle of the vertex;
A learning device characterized by: The learning device according to claim 8,
In the deletion process, the processor deletes a specific part of the learning target shape;
A learning device characterized by: The learning device according to claim 8,
In the acquisition process, the processor acquires environmental information when the shape of the learning target is acquired;
In the behavior learning process, the processor learns the behavior of the learning target based on the component group related to the learning target, the behavior of the learning target, the missing position information, and the environment information, generating a behavior classification model for classifying the behavior of the learning target;
A learning device characterized by: An action recognition method executed by an action recognition device having a processor that executes a program, and a storage device that stores the program, the method comprising:
It is possible to access a behavior classification model learned using a component group related to the learning target obtained from the shape of the learning target by component analysis that generates statistical components by multivariate analysis, and the behavior of the learning target. ,
The processor includes:
Detection processing that detects the shape of the recognition target from the analysis target data,
a defective position information generation process that generates defective position information indicating a position of a defective part in the shape of the recognition target detected by the detection process;
An interpolation process of interpolating the missing part from non-defective information that is a part other than the missing part in the shape of the recognition target including the missing part, and updating the non-defective information after interpolation as the shape of the recognition target. ,
a component analysis process that generates the same number of component groups related to the recognition target as component groups related to the learning target, based on the shape of the recognition target interpolated by the interpolation process, through the component analysis;
a behavior recognition process that outputs a recognition result indicating the behavior of the recognition target by inputting a component group related to the recognition target generated by the component analysis process and the missing position information to the behavior classification model; ,
An action recognition method characterized by performing the following.

Images