JP7439004B2 - Behavior recognition device, learning device, and behavior recognition method - Google Patents

Description

The present invention relates to a behavior recognition device, a learning device, and a behavior recognition method.

As background art in this technical field, Patent Document 1 discloses an intention estimation device that identifies whether a human movement is intended without relying on biological signals such as surface myoelectric potential. This intention estimation device acquires movement information using a method of measuring the position and angle of a person's movement, limits the person's movement to a range that is achievable by the person, and calculates the joint angles of the person during the movement. We extract the positional information of the tip position of the moving part and use a multivariate analysis method, and then use a threshold to identify whether the person's movement is intentional or not. By identifying whether the movement is intended or not, it is possible to identify whether the movement is intended or not, without relying on biological signals such as surface myoelectric potential.

Japanese Patent Application Publication No. 2012-101284

In the technology described in Patent Document 1 for estimating the intention of a person's movements, it is difficult to classify the intentions of multiple types of complex movements in order to make a binary judgment as to whether the action is intended or not. This may result in a significant decrease in the accuracy of motion intention estimation.

An object of the present invention is to highly accurately recognize multiple types of behaviors to be recognized.

A behavior recognition device that is an aspect of the invention disclosed in this application is a behavior recognition device that includes a processor that executes a program, and a storage device that stores the program, and that calculates statistical components by multivariate analysis. A behavior classification model group learned for each component group can be accessed using a component group obtained from the shape of the learning target by the generated component analysis and the behavior of the learning target, and the processor can access the behavior classification model group learned for each component group. One or more components and a contribution rate of each of the components are determined by a detection process of detecting the shape of the recognition target from data and the component analysis based on the shape of the recognition target detected by the detection process. a component analysis process to generate; a determination process that determines an ordinal number indicating each of the one or more dimensions based on a cumulative contribution rate obtained from each of the contribution rates; and an ordinal number indicating the dimension determined by the determination process. A selection process of selecting a specific behavior classification model learned with the same component group as a specific component group containing one or more components from the behavior classification model group, and a specific behavior classification model selected by the selection process. The present invention is characterized in that, by inputting the specific component group, a behavior recognition process is executed that outputs a recognition result indicating the behavior of the recognition target.

Another aspect of the invention disclosed in this application is an action recognition device that includes a processor that executes a program, and a storage device that stores the program, and that performs multivariate analysis to calculate statistical components. The behavior classification model group learned for each component group can be accessed by using the component group in ascending order from the first variable obtained from the shape of the learning target by dimension reduction to generate , and the behavior of the learning target. The processor performs a detection process of detecting the shape of the recognition target from the analysis target data, and performs the dimension reduction to determine one or more components and the component based on the shape of the recognition target detected by the detection process. and a determination process that determines an ordinal number indicating the dimension of the component in ascending order from the first variable among the one or more components based on the contribution rate of each of the components. , a selection process of selecting a specific behavior classification model trained with the same component group as a specific component group from the first variable to an ordinal component indicating the dimension determined by the determination process from the behavior classification model group; and a behavior recognition process that outputs a recognition result indicating the behavior to be recognized by inputting the specific component group to a specific behavior classification model selected by the selection process. do.

A learning device that is one aspect of the invention disclosed in this application is a learning device that has a processor that executes a program, and a storage device that stores the program, and the processor is configured to learn the shape and behavior of a learning target. Component analysis that generates one or more components based on the shape of the learning target acquired by the acquisition process, by an acquisition process that acquires training data containing the data, and a component analysis that generates statistical components by multivariate analysis. a control process for controlling ordinal numbers indicating each of the one or more dimensions based on an allowable amount of calculation; a component group including one or more ordinal components indicating the dimensions controlled by the control process; and the learning. The present invention is characterized in that a behavior learning process is executed for learning the behavior of the learning target based on the behavior of the target and generating a behavior classification model for classifying the behavior of the learning target.

A learning device according to another aspect of the invention disclosed in this application is a learning device including a processor that executes a program, and a storage device that stores the program, and the processor is configured to control the shape and behavior of a learning target. A dimension that generates one or more components based on the shape of the learning target acquired by the acquisition process, by an acquisition process that acquires training data including , and a dimension reduction that generates statistical components by multivariate analysis. a reduction process; a control process that controls an ordinal number indicating the dimension of an ascending component from a first variable among the one or more components based on an allowable amount of calculation; Behavior learning processing that generates a behavior classification model that classifies the behavior of the learning target by learning the behavior of the learning target based on a component group up to an ordinal component indicating a dimension and the behavior of the learning target. It is characterized by executing the following.

According to a typical embodiment of the present invention, multiple types of behaviors to be recognized can be recognized with high accuracy. Problems, configurations, and effects other than those described above will become clear from the description of the following examples.

FIG. 1 is an explanatory diagram showing an example of a system configuration of an action recognition system according to a first embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. FIG. 3 is an explanatory diagram showing an example of learning data. FIG. 4 is a block diagram showing an example of the functional configuration of the action recognition system according to the first embodiment. FIG. 5 is a block diagram showing a detailed functional configuration example of the skeleton information processing section. FIG. 6 is an explanatory diagram showing a detailed method for calculating joint angles executed by the joint angle calculating section. FIG. 7 is an explanatory diagram illustrating a detailed example of a method for calculating the amount of movement between frames, which is executed by the movement amount calculating section. FIG. 8 is an explanatory diagram showing a detailed method of normalizing skeleton information executed by the normalization unit. FIG. 9 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB. FIG. 10 is an explanatory diagram showing an example in which principal components generated by a principal component analysis section using a teacher signal as input data are plotted on a principal component space. FIG. 11 is an explanatory diagram showing a detailed method for the behavior learning unit to learn behaviors and for the behavior recognition unit to classify the behaviors. FIG. 12 is a graph showing changes in the cumulative contribution rate used by the dimension number determining unit when determining the number of dimensions. FIG. 13 is a flowchart illustrating a detailed processing procedure example of learning processing by the server (learning device) according to the first embodiment. FIG. 14 is a flowchart illustrating a detailed processing procedure example of skeleton information processing according to the first embodiment. FIG. 15 is a flowchart illustrating an example of a behavior recognition processing procedure by a client (behavior recognition device) according to the first embodiment. FIG. 16 is a block diagram showing an example of the functional configuration of the action recognition system according to the second embodiment. FIG. 17 is a flowchart illustrating a detailed processing procedure example of learning processing by the server (learning device) according to the second embodiment. FIG. 18 is a flowchart illustrating an example of a behavior recognition processing procedure by a client (behavior recognition device) according to the second embodiment. FIG. 19 is a block diagram showing an example of the functional configuration of the skeleton information processing section according to the fourth embodiment. FIG. 20 is a flowchart illustrating a detailed processing procedure example of the skeleton information processing unit according to the fourth embodiment. FIG. 21 is a block diagram showing an example of the functional configuration of the action recognition system according to the fifth embodiment. FIG. 22 is a block diagram showing an example of the functional configuration of the action recognition system according to the sixth embodiment. FIG. 23 is an explanatory diagram showing a decision tree, which is a basic method by which the behavior learning unit and the behavior recognition unit classify behaviors. FIG. 24 is an explanatory diagram showing a detailed method for developing classification using a decision tree. FIG. 25 is an explanatory diagram showing ensemble learning and a method used by the behavior learning section and the behavior recognition section to classify behaviors. FIG. 26 is a block diagram showing an example of the functional configuration of the action recognition system according to the seventh embodiment. FIG. 27 is a flowchart illustrating a detailed processing procedure example of learning processing by the server (learning device) according to the seventh embodiment.

Embodiments according to the present invention will be described below based on the drawings. In addition, in all the figures for explaining the embodiment, the same members are given the same reference numerals in principle, and repeated explanations thereof will be omitted. In addition, in the following embodiments, the constituent elements (including elemental steps, etc.) are not necessarily essential, except when explicitly stated or when it is clearly considered essential in principle. Needless to say. In addition, when we say "consists of A," "consists of A," "has A," or "contains A," other elements are excluded, unless it is specifically stated that only that element is included. Needless to say, this is not something you should do. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., the shape, positional relationship, etc. of components, etc. are referred to, unless specifically stated or when it is considered that it is clearly not possible in principle. This shall include things that approximate or are similar to, etc.

In this specification, etc., expressions such as "first," "second," and "third" are used to identify constituent elements, and do not necessarily limit the number, order, or content thereof. isn't it. Further, numbers for identifying components are used for each context, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, this does not preclude a component identified by a certain number from serving the function of a component identified by another number.

The position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings or the like.

<Action recognition system>
FIG. 1 is an explanatory diagram showing an example of a system configuration of an action recognition system according to a first embodiment. The behavior recognition system 100 includes a server 101 and one or more clients 102. The server and the client are communicably connected via a network 105 such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network). The server 101 is a computer that manages the clients 102. The client 102 is a computer that is connected to the sensor 103 and acquires data from the sensor 103.

The sensor 103 detects data to be analyzed from the analysis environment. The sensor 103 is, for example, a camera that captures still images or moving images. Further, the sensor 103 may detect sound or smell. The teacher signal DB 104 is a database that holds a combination of learning data (person's skeletal information and joint angles) and behavior information (for example, person's postures and movements such as "standing" and "falling down") as a teacher signal. The teacher signal DB 104 may be stored in the server 101 or may be connected to a computer that can communicate with the server 101 or the client 102 via the network 105.

The behavior recognition system 100 has a learning function using the teacher signal DB 104 and an action recognition function using a behavior classification model obtained by the learning function. A behavior classification model is a learning model for classifying the behavior of recognition targets such as people and animals. The learning function and the behavior recognition function may be implemented in either the server 101 or the client 102 as long as they are implemented in the behavior recognition system 100. For example, the server 101 may implement a learning function, and the client 102 may implement an action recognition function. Further, the server 101 may implement a learning function and a behavior recognition function, and the client 102 may transmit data from the sensor 103 to the server 101 or receive behavior recognition results from the server 101 using the behavior recognition function. .

Further, the client 102 may implement a learning function and a behavior recognition function, and the server 101 may manage behavior classification models and behavior recognition results from the client 102. Note that a computer implementing a learning function is referred to as a learning device, and a computer implementing at least an action recognition function of the learning function and the action recognition function is referred to as an action recognition device. Further, in FIG. 1, a client-server type behavior recognition system 100 is taken as an example, but a stand-alone type behavior recognition device may be used. In the first embodiment, for convenience of explanation, a behavior recognition system 100 will be described as an example in which the server 101 implements a learning function (learning device) and the client 102 implements a behavior recognition function (behavior recognition device).

<Example of computer hardware configuration>
FIG. 2 is a block diagram showing an example of the hardware configuration of a computer (server 101, client 102). The computer 200 includes a processor 201, a storage device 202, an input device 203, an output device 204, and a communication interface (communication IF) 205. The processor 201, storage device 202, input device 203, output device 204, and communication IF 205 are connected by a bus 206. Processor 201 controls computer 200 . Storage device 202 serves as a work area for processor 201 . Furthermore, the storage device 202 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the storage device 202 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. Input device 203 inputs data. Examples of the input device 203 include a keyboard, mouse, touch panel, numeric keypad, and scanner. Output device 204 outputs data. Examples of the output device 204 include a display, a printer, and a speaker. Communication IF 205 connects to network 105 and transmits and receives data.

<Learning data>
FIG. 3 is an explanatory diagram showing an example of learning data. The learning data 380 includes skeletal information 320 and joint angles 370 for each subject. Skeletal information 320 is detected based on analysis target data acquired from sensor 103. Joint angle 370 is calculated based on skeletal information 320. The learning data 380 for one subject is configured, for example, by a combination of skeletal information 320 and joint angles 370 obtained from each of a plurality of time-series frames in which the subject is the subject.

The skeleton information 320 includes, for each of a plurality of (18 in this example) skeleton points 300 to 317, a name 321, an x-coordinate value 322 on the X-axis, a y-coordinate value 323 on the y-axis perpendicular to the X-axis, has. The joint angle 370 also has a name 371 for each of the plurality of (18 in this example) skeletal points 300-317. In addition, in the name 371, ∠a-b-c (a, b, c are the names 321 of skeleton points) is the joint angle 370 of the skeleton point b formed by the line segment ab and the line segment bc. Note that the skeletal information 320 may include, for example, finger joints. Further, the joint angle 370 may also include joint angles 370 other than these.

Note that in FIG. 3, the coordinate values of the skeleton points 300 to 317 are two-dimensional positional information (a combination of x-coordinate values and y-coordinate values), but they may be three-dimensional positional information. Specifically, for example, a z-coordinate value in the z-axis (eg, depth direction) orthogonal to the X-axis and the y-axis may be added.

<Functional configuration example of behavior recognition system 100>
FIG. 4 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the first embodiment. The server 101 includes a teacher signal acquisition section 401, a missing information control section 402, a skeleton information processing section 403, a principal component analysis section 404, a dimension number control section 405, and a behavior learning section 406. The client 102 includes a skeleton detection unit 451, a missing information determination unit 452, a skeleton information processing unit 453, a principal component analysis unit 454, a dimensionality determination unit 455, a behavior classification model selection unit 456, and a behavior recognition unit 457. and has.

Specifically, these are realized, for example, by causing the processor 201 to execute a program stored in the storage device 202 shown in FIG. 2. First, an example of the functional configuration of the server 101 will be described.

The teacher signal acquisition unit 401 acquires one or more teacher signals used for learning from the teacher signal DB 104 and outputs the selected teacher signal to the missing information control unit 402 .

The missing information control unit 402 causes arbitrary skeleton points to be missing from the skeleton information 320 in the teacher signal acquired from the teacher signal acquisition unit 401. The number of skeleton points to be deleted may be singular, plural, or zero. The missing information control unit 402 updates the skeleton information 320 after the loss (including the case where there are no skeleton points to be deleted) as the skeleton information 320 in the teacher signal. Further, in order to strengthen noise resistance, the skeleton information 320 may be updated by adding noise that shifts the position of the skeleton points to the skeleton information 320 when deleting information.

Then, the missing information control unit 402 outputs a teacher signal including missing information, which is the name 321 and position information (x coordinate value 322, y coordinate value 323) of the missing skeleton point, to the skeleton information processing unit 120. Furthermore, the missing information control unit 402 outputs the missing information to the behavior learning unit 406 via the skeleton information processing unit 403, principal component analysis unit 404, and dimension number control unit 405.

The skeleton information processing unit 403 processes the updated skeleton information 320. Specifically, for example, the skeleton information processing unit 403 calculates the joint angle 370 and the amount of movement between frames from the skeleton information 320 of the acquired updated teacher signal. Furthermore, the skeleton information processing unit 403 performs normalization on the skeleton information 320, excluding absolute position information so that the size of the skeleton information 320 is constant. Then, the skeletal information processing unit 403 outputs the joint angle 370, the amount of movement between frames, and the normalized skeletal information 320 to the principal component analysis unit 404.

FIG. 5 is a block diagram showing a detailed functional configuration example of the skeleton information processing units 403 and 453. The skeletal information processing units 403 and 453 include a joint angle calculation unit 501, a movement amount calculation unit 502, and a normalization unit 503.

The joint angle calculation unit 501 calculates a joint angle 370 from the skeleton information 320 of the acquired teacher signal, and outputs it to the principal component analysis unit 404 via the movement amount calculation unit 502 and the normalization unit 503.

The movement amount calculation unit 502 calculates the inter-frame movement amount from the skeleton information 320 of the acquired teacher signal, and outputs it to the principal component analysis unit 404 via the normalization unit 503.

The normalization unit 503 excludes absolute position information from the acquired teacher signal with respect to the skeleton information 320, performs normalization so that the size of the skeleton information 320 is constant, and sends it to the principal component analysis unit 404. Output.

Returning to FIG. 4, the principal component analysis unit 404 uses as input data the normalized skeletal information 320, the joint angles 370, and the amount of movement between frames among the teacher signals acquired from the skeletal information processing unit 403. A principal component analysis is performed to generate one or more principal components, and the generated principal components are output to the number of dimensions control unit 405 . Note that among the skeletal information 320, the joint angles 370, and the amount of movement between frames, at least the skeletal information 320 may be input data.

In principal component analysis, as shown in equation (1) below, input data x _i is multiplied by coefficient w _ij and added to generate principal component y _i . The general formula for principal component analysis is shown in formula (2) below. As shown in equation (3) below, the coefficient w _ij is determined so that when the variance of y _i is defined as V(y _i ), the variance V(y _i ) is maximized.

However, if there is no constraint on the coefficient wij, the absolute amount of the variance V(y _i ) can be infinitely large, and the coefficient _wij cannot be uniquely determined, so the constraint in equation (4) below It is desirable to add . Furthermore, in order to eliminate duplication of information, it is desirable to impose the following formula (5) constraint such that the covariance between the newly generated principal component y _k and the previously generated principal component y _k is 0.

However, the above equations (4) and (5) that are added as constraints are not limited to these, and the coefficient w _ij may be calculated by adding another constraint condition or removing the constraints. When the variance V(yj) of the new principal component y _j generated in this way is separately defined as λ _j as shown in equation ₍ 6) below, the variance V(x _j ) and the sum of λ _j are equal.

Here, p is the number of input data _xj . The higher the variance V (y _j ) of the newly generated principal component y _j , the more the original information is reflected, and the principal component with the highest variance value is the first, second, ..., m-th principal component. That's what it means. The ratio of the variance of the newly generated variable y _j to the variance of the original data is called the contribution rate, and is expressed by the following equation (8). Further, the result of adding the contribution rate from the contribution rate of the first principal component in descending order of the variance value (in ascending order of the ordinal number m of the principal component) is called the cumulative contribution rate, and is expressed by the following equation (9).

The contribution rate and the cumulative contribution rate are measures of how much the newly generated principal component y _j or the generated plurality of principal components represent the information amount of the original data, and are generated together with the principal component. Although principal component analysis was applied as an example of component analysis that generates statistical components in multivariate analysis, independent component analysis, which is also an example of component analysis, may be performed instead of principal component analysis. .

In the case of independent component analysis, the principal components are independent components. A contribution rate may be used as an index indicating how much influence this independent component has on the input data xi. In the independent component analysis, the sum of squares of the mixing coefficient matrix in the independent component analysis for each independent component becomes the strength of each independent component.

The strength of the independent component indicates the variance of the independent component in the input data x _i . That is, since all the independent components obtained by the independent component analysis have a unified variance of 1, the sum of the squares of the mixing coefficients becomes the variance of the input data x _i . Then, the contribution ratio of the independent variable may be determined by dividing the strength of the independent component by the sum of the strengths of all independent components.

The dimension number control unit 405 controls an ordinal number k indicating the dimension of each of one or more components. Specifically, for example, the dimension number control unit 405 determines how many dimensions of the principal components to be used for learning in the behavioral learning unit 406 among the acquired principal components, in descending order of variance value, and selects the first principal component. The principal components from the component to the k-th principal component whose ordinal number is the determined dimension k (k is an integer greater than or equal to 1) are output to the behavior learning unit 406 in descending order of variance value.

The behavior learning unit 406 learns the principal components acquired from the number of dimensions control unit 405 and the behavior information in the teacher signal acquired from the teacher signal DB 104 in association with each other. Specifically, for example, the behavior learning unit 406 uses the principal component group from the first principal component to the k-th principal component obtained from the number of dimensions control unit 405 as input data, and uses the information in the teacher signal obtained from the teacher signal DB 104 as input data. A behavior classification model is generated by machine learning using behavior information as output data. The behavior learning unit 406 associates the behavior classification model generated as a result of learning with the missing information acquired from the missing information control unit 402 and outputs it to the behavior classification model selection unit 456.

Next, an example of the functional configuration of the client 102 will be described. The skeleton detection unit 451 detects the skeleton information 320 of a person appearing in the analysis target data acquired from the sensor 103 and outputs it to the missing information determination unit 452. To detect the skeletal information 320, a neural network (NN) that can estimate the human skeletal information 320 generated by machine learning may be used, or markers may be added to the skeletal points of the person to be detected, and markers that appear in the image may be used. The skeletal information 320 may be detected from the position, and the method of detecting the skeletal information 320 is not limited.

The missing information determining unit 452 determines whether there is a skeleton point that cannot be acquired due to occlusion or the like in the skeleton information 320 detected by the skeleton detecting unit 451, and if there is a skeleton point that could not be acquired, the position information of the skeleton point is determined. The skeleton information 320 detected by the skeleton detection section 451 is output to the skeleton information processing section 453 as missing information. Furthermore, the missing information determining unit 452 outputs the missing information to the behavior classification model selecting unit 456 via the skeletal information processing unit 453, the principal component analyzing unit 454, and the number of dimensions determining unit 455.

The skeletal information processing unit 453 has the same functions as the skeletal information processing unit 403. The skeleton information processing unit 453 executes the same processing as the skeleton information processing unit 403 on the skeleton information 320 detected by the skeleton detection unit 451, and calculates the joint angle 370, the amount of movement between frames, and the normalized skeleton. The information 320 is output to the principal component analysis section 454.

Principal component analysis section 454 has the same functions as principal component analysis section 404. The principal component analysis section 454 performs the same processing as the principal component analysis section 404 on the output data from the skeleton information processing section 453 to generate one or more principal components. Further, the principal component analysis unit 454 outputs the contribution rate and cumulative contribution rate generated together with the principal component to the number of dimensions determination unit 455.

The dimension number determination unit 455 determines an ordinal number k indicating each dimension of one or more components based on the cumulative contribution rate obtained from each contribution rate. Specifically, for example, the dimension number determining unit 455 outputs the principal components of the acquired principal components in descending order of variance from the acquired contribution rate and cumulative contribution rate to the behavior classification model selection unit 456. Determine the number of dimensions k that indicates whether The number of dimensions k is an ordinal number k indicating the dimension of the principal component. For example, for the first principal component, the number of dimensions (ordinal number) k=1, and for the second principal component, the number of dimensions (ordinal number) k=2. The number of dimensions determining unit 455 outputs a group of principal components from the first principal component to the k-th principal component to the behavior classification model selecting unit 456 in descending order of variance.

The behavior classification model selection unit 456 selects a behavior classification model associated with the missing information generated by the missing information control unit 402, which is associated with the same missing information as the missing information obtained from the missing information determination unit 452, and whose dimension number is The determining unit 455 selects a behavior classification model that has undergone behavioral learning using the group of principal components up to the kth dimension (first principal component to kth principal component) determined by the determining unit 455. The behavior classification model selection unit 456 outputs the selected behavior classification model along with the principal component group from the first principal component to the k-th principal component to the behavior recognition unit 457.

In particular, in a two-dimensional image, there is a possibility that all defined skeleton points cannot be obtained due to occlusion, etc., and the skeleton detection unit 451 may generate skeleton information 320 in which some skeleton points that could not be obtained are missing. There is. When performing behavior recognition on the skeleton information 320 in which some skeleton points are missing, the client 102 uses the behavior learning model associated with the missing information in the missing skeleton information 320 detected by the skeleton detection unit 451 to perform behavior recognition. Recognize. As a result, highly accurate behavior recognition is achieved even for the skeleton information 320 in which some skeleton points are missing.

Note that among the behavior classification models associated with the missing information generated by the missing information control unit 402 obtained from the behavior learning unit 406, the same missing information as the missing information obtained from the missing information determining unit 452 is associated, and the number of dimensions is It is also assumed that a behavior classification model in which behavior learning is performed using the same principal components (first principal component to k-th principal component) as the ordinal number k indicating the dimension of the principal component determined by the determining unit 455 is not generated.

In this case, the behavior classification model selection unit 456 selects a behavior classification model that is closest to this condition (for example, a behavior classification model in which the position information of the missing skeletal point and the missing information within a predetermined distance are associated, the first principal component to the (k-1) A behavior classification model in which behavior learning is performed using principal components, etc.) may be selected.

The behavior recognition unit 457 recognizes the behavior of the person reflected in the analysis target data acquired from the sensor 103 based on the selected behavior classification model and the principal component group from the first principal component to the k-th principal component. Specifically, for example, the behavior recognition unit 457 inputs the principal component group (first principal component to k-th principal component) obtained from the data to be analyzed into the selected behavior classification model. A predicted value indicating the behavior of the person shown in the image is output as a recognition result.

<Example of joint angle calculation>
FIG. 6 is an explanatory diagram showing a detailed calculation method of the joint angle 370 executed by the joint angle calculation unit 501. The joint angle calculation unit 501 calculates the joint angle θ at three connected skeleton points 600 to 602. The skeleton information 620 of the skeleton points 600 to 602 is defined as position vectors O, A, and B with the origin 630 as a reference. The joint angle calculation unit 501 calculates a relative vector with the skeleton point 600 as the origin as shown in the following equations (10) and (11), and from the calculated vector, the following equation (12) is established, and the following equation (13) is established. The joint angle θ is calculated by calculating the arc cosine as shown in .

<Example of calculating the amount of movement between frames>
FIG. 7 is an explanatory diagram illustrating a detailed example of a method for calculating the amount of movement between frames, which is executed by the movement amount calculation unit 502. The movement amount calculation unit 502 uses skeleton information 701 of the Nth frame and skeleton information 702 of the NMth frame regarding the same subject in calculating the movement amount between frames. N and M are integers of 1 or more, and N>M. The value of M can be set arbitrarily. As shown in equations (14) to (16) below, the movement amount calculation unit 502 calculates the distances between the same skeletal points 300 to 317 of the same person shown between each frame. The amount of movement of the 18 skeleton points 300 to 317 between frames is the amount of movement between frames for the person.

However, the amount of movement between frames executed by the movement amount calculation unit 502 is not limited to this, and as shown in equation (17) below, the movement amount calculation unit 502 can calculate the amount of movement of the same person shown between each frame. The distances between the same skeletal points 300 to 317 may be calculated, and the sum of the inter-frame movement amounts of all 18 skeletal points 300 to 317 may be used as the inter-frame movement amount for the person.

Further, the movement amount calculation unit 502 may use center-of-gravity skeleton information 711 and center-of-gravity skeleton information 712, which are the center of gravity, out of the skeleton information 701 of the n-th frame and the skeleton information 702 of the nm-th frame. Specifically, for example, the movement amount calculation unit 502 calculates the center of gravity for each person as shown in equations (18) to (19) below, and calculates the center of gravity for each person as shown in equation (20) below. , the amount of movement between frames for the person may be calculated.

<Example of normalization>
FIG. 8 is an explanatory diagram showing a detailed method of normalizing the skeleton information 320 performed by the normalization unit 503. First, the normalization unit 503 (a) calculates the center of gravity from all or part of the skeleton information 320, and (b) converts the center of gravity into relative coordinates with the origin as the origin. After that, the normalization unit 503 divides (d) the position information of each skeleton point in the skeleton information 320 by (c) the length L of the minimum rectangular diagonal that surrounds the 18 skeleton points 300 to 317. If the skeleton information 320 obtained in (d) is used as a teacher signal, the position information of the skeleton points 300 to 317 after division will also be incorporated.

For example, if the normalization unit 503 is not executed and learning for skeleton detection and behavior classification is executed for an action such as "a 180 cm person sits at point A", "a person who is 180 cm tall sits at point A" and "does not sit at any place other than point A" There is a possibility that a judgment will be made that ``people who do not sit down will not sit down''. In order to eliminate these limitations and provide versatility in behavior classification, the normalization unit 503 normalizes the skeleton information 320 in order to remove absolute position information in the image and information regarding the size of the skeleton. do.

<Teacher signal held by the teacher signal DB 104>
FIG. 9 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB 104. For the person appearing in (a) image 900, which is data to be analyzed, (b) skeletal information 320A, joint angles 370 (not shown), and (c) behavior information 901 (“standing”) associated with skeletal information 320A. , becomes the teacher signal. Similarly, for the person appearing in (a) image 910, which is the data to be analyzed, (b) skeletal information 320B, joint angles 370 (not shown), and (c) behavioral information 911 (“fall down”) associated with skeletal information 320B. ”) becomes the teacher signal.

<Number of dimensions control by the number of dimensions control unit 405 and behavior learning by the behavior learning unit 406>
FIG. 10 is an explanatory diagram showing an example in which the principal components generated by the principal component analysis unit 404 using the teacher signal as input data are plotted on the principal component space. The legend indicates behavior information 1000 to 1004 included in the teacher signal.

In FIG. 10, (a) shows an example in which the first principal component is plotted on the X axis and the second principal component is plotted on the Y axis, and information up to the second principal component is plotted on a two-dimensional plane. (b) is an example in which the first principal component is plotted on the X-axis, the second principal component is plotted on the Y-axis, and the third principal component is plotted on the Z-axis, and the information up to the third principal component is plotted on a three-dimensional space. show.

In (a), it can be seen that standing 1000, sitting 1001, and falling 1004 can be separated even on a two-dimensional plane up to the second principal component, but walking 1002 and squatting 1003 are two-dimensional up to the second principal component. It can be seen that separation is difficult on a dimensional plane. Here, in (b), if walking 1002 and crouching 1003 are plotted on a three-dimensional space including up to the third principal component, the possibility of separation may increase.

Therefore, if a large number of principal components generated by the principal component analysis unit 404 are used, highly accurate behavior classification is possible. However, as the ordinal number k, which indicates the dimension of the principal component, increases, the amount of calculation increases, so it is necessary to consider the extent of the principal component based on the accuracy and amount of calculation, and determine in what dimensional space the behavior should be represented.

Therefore, the dimension number control unit 405 changes the maximum ordinal number of the principal components used for learning by the behavior learning unit 406, and outputs the principal component group from the first principal component to the principal component with the maximum ordinal number to the behavior learning unit 406. Specifically, for example, the required accuracy of the behavior classification described above (for example, an ordinal number indicating the minimum required dimension of the principal component) and/or the allowable amount of calculation are set in advance, and the number of dimensions control unit 405 controls the behavior classification. The learning unit 406 changes the maximum ordinal number of the principal component used for learning, and determines the ordinal number that satisfies the required accuracy and/or allowable amount of calculation to the maximum extent.

For example, if the required accuracy is an ordinal number "3" indicating a dimension (third principal component), the dimension number control unit 405 determines the maximum ordinal number to be "3" and The principal component group of is output to the behavior learning unit 406.

In addition, when the allowable amount of calculation is set as a condition, the number of dimensions control unit 405 sequentially acquires the amount of calculation in ascending order from the first principal component, and sets the maximum ordinal number to the ordinal number ( For example, the ordinal number (for example, "4") is determined to be one less than "5" (for example, "4"), and the principal component group from the first principal component to the fourth principal component with the maximum ordinal number k=4 is output to the behavior learning unit 406. .

In addition, if the required accuracy is the ordinal number "3" indicating the dimension (third principal component) or more, and the allowable calculation amount is set as a condition, the cumulative calculation amount up to the third principal component is the allowable calculation amount. If it is below, the dimension number control unit 405 changes the maximum ordinal number from "3" to "4". Then, if the cumulative amount of calculation up to the fourth principal component exceeds the allowable amount of calculation, the dimension number control unit 405 determines the maximum ordinal number k to be "3", and The group is output to the behavior learning unit 406.

On the other hand, if the cumulative amount of calculation up to the third principal component exceeds the allowable amount of calculation, the number of dimensions control unit 405 changes the maximum ordinal number from "3" to "2". Then, if the cumulative amount of calculation up to the second principal component is less than the allowable amount of calculation, the number of dimensions control unit 405 determines the maximum ordinal number k to be "2", and The component group is output to the behavior learning unit 406.

Note that the principal component group output to the behavior learning unit 406 does not need to be limited to ascending order starting from the first principal component. For example, the number of dimensions control unit 405 may extract a specific number of predetermined principal component groups. Further, the number of dimensions control unit 405 may decide the principal component group to be output to the behavior learning unit 406 after excluding a specific principal component group. In this way, the principal component groups output to the behavior learning unit 406 are not limited to principal component groups in ascending order from the first principal component.

Also, in this case, if the allowable amount of calculation is set as a condition, the number of dimensions control unit 405 calculates the principal component groups in ascending order of ordinal numbers, which is not limited to the ascending order from the first principal component described above. The amount is sequentially acquired, and the principal component group up to the ordinal number immediately before the ordinal number when the allowable calculation amount is exceeded for the first time is output to the behavior learning unit 406. For example, if the principal component group consists of the second principal component, third principal component, and fifth principal component, the second principal component does not exceed the allowable amount of calculation, and the second and third principal components do not exceed the allowable amount of calculation. If the allowable calculation amount is exceeded for the first time in the second principal component, third principal component, and fifth principal component without exceeding the allowable amount of calculation, the dimension number control unit 405 Up to three principal components may be determined as the principal component group to be output to the behavior learning unit 406.

The behavior learning unit 406 performs behavior learning under a plurality of conditions in advance, generates a behavior classification model, and outputs it to the behavior classification model selection unit 456. By selecting a behavior classification model according to the situation from the multiple behavior classification models generated in this way, versatile and highly accurate behavior recognition is achieved.

FIG. 11 is an explanatory diagram showing a detailed method for the behavior learning unit 406 to learn behaviors and for the behavior recognition unit 457 to classify the behaviors. Regarding each action on the principal component space, the action learning unit 406 classifies each action into regions using (a) a boundary line 1101 and (b) a boundary plane 1102. The method for learning and classifying behaviors may be any of the k-means method, support vector machine, decision tree, random forest, etc., and the behavior learning method is not limited.

Using the behavior classification model learned and generated by the behavior learning unit 406, the behavior recognition unit 457 recognizes the behavior. Specifically, for example, the client 102 applies principal component analysis to the newly input skeletal information 320, and uses the newly generated principal components according to the boundary line 1101 and boundary plane 1102 set by the behavior classification model. Determine which region it belongs to and recognize the behavior according to the determined region.

FIG. 12 is a graph showing changes in the cumulative contribution rate used by the dimension number determining unit 455 when determining the number of dimensions. The cumulative contribution rate is a measure of how much the newly generated principal components represent the amount of information in the original data. Therefore, even if the number of principal components is increased to increase the number of dimensions for behavior classification, if there is no significant change in the cumulative contribution rate, no significant improvement in accuracy can be expected.

Therefore, the number of dimensions determining unit 455 determines the number of dimensions by using only the number of principal components necessary to exceed a predetermined cumulative contribution rate threshold. For example, if the predetermined cumulative contribution rate threshold is "0.8", the condition is satisfied as long as the second principal component is reached, so the number of dimensions k here is set to "2", and the first principal component and 2 principal components to the behavior classification model selection unit 456.

Note that the principal component groups output to the behavior classification model selection unit 456 do not need to be limited to ascending order starting from the first principal component. For example, the number of dimensions determination unit 455 may determine a combination of ordinal numbers k of principal components that does not exceed a predetermined cumulative contribution rate threshold and has a maximum cumulative contribution rate. Further, the dimension number determination unit 455 may select such a combination of ordinal numbers k of principal components from a group of principal components applied to the behavior classification model. In this way, the principal component groups output to the behavior classification model selection unit 456 are not limited to principal component groups in ascending order from the first principal component.

<Learning process>
FIG. 13 is a flowchart illustrating a detailed processing procedure example of learning processing by the server 101 (learning device) according to the first embodiment. The server 101 uses the teacher signal acquisition unit 401 to acquire one or more teacher signals used for learning from the teacher signals acquired from the teacher signal DB 104 (step S1300).

The server 101 uses the missing information control unit 402 to cause information to be missing from the skeleton information 320 in the acquired teacher signal, updates the missing skeleton information 320 as the skeleton information 320 in the teacher signal, and updates the missing skeleton as the skeleton information 320 in the teacher signal. The point name 321 and position information (x coordinate value 322, y coordinate value 323) are set as missing information (step S1301). The teacher signal on which the missing information control unit is executed is referred to as an updated teacher signal.

The server 101 uses the skeleton information processing unit 403 to execute skeleton information processing for each updated teacher signal (step S1302). Specifically, for example, the server 101 executes processing by a joint angle calculation unit 501, a movement amount calculation unit 502, and a normalization unit 503.

FIG. 14 is a flowchart illustrating a detailed processing procedure example of skeleton information processing according to the first embodiment. The server 101 uses the joint angle calculating unit 501 to calculate the joint angle 370 from the skeleton information 320 in the updated teaching signal for each updated teaching signal (step S1401). Next, in the server 101, the movement amount calculation unit 502 calculates the movement amount between frames for each updated teacher signal from the skeleton information 320 in the updated teacher signal (step S1401).

Then, the server 101 uses the normalization unit to exclude absolute position information from the skeletal information 320 for each updated teacher signal, and performs normalization such that the size of the skeletal information 320 is constant (step S1303 ). As a result, the joint angle 370, the amount of movement between frames, and the normalized skeletal information 320 are obtained for the updated teacher signal. Then, the process moves to step S1303 in FIG. 13.

Returning to FIG. 13, the server 101 uses the normalized skeletal information 320, the joint angles 370, and the amount of movement between frames as input data, and executes a principal component analysis using the principal component analysis unit 404. Alternatively, multiple principal components are generated (step S1303).

Next, the server 101 uses the dimension number control unit 405 to determine how many principal components to use for learning among the generated principal components in descending order of variance value, and determines the principal components up to the determined k dimension (the 1st principal component to kth principal component) are selected in descending order of variance value (step S1304).

Then, the server 101 uses the behavior learning unit to perform learning based on the selected principal component and the behavior information in the updated teacher signal, and as a result of the learning, generates a behavior classification model and associates it with the missing information (step S1305).

In the principal component analysis (step S1303), it is possible to generate principal components with the same number of dimensions k as the information before performing the principal component analysis. Therefore, in step S1306, the server 101 uses the dimension number control unit 405 to control the dimension number k of the principal components used for learning determined in step S1304, if there is a dimension of the principal components that has not been determined so far, ( Step S1306: No), the process returns to step S1304, and the dimensions of the principal components that have not been determined so far are determined (step S1304).

On the other hand, if the dimensions of all principal components used for determinable learning have been determined (step S1306: Yes), the process advances to step S1307. However, the determination of the process in step S1306 is not limited to determining the next process based on whether or not to determine the dimensions of all principal components used for decidable learning. For example, if the number of repetitions is predetermined and step S1304 is repeated the predetermined number of times, the process may proceed to step S1307.

In step S1307, if there is skeletal information 320 that has not been deleted yet for the skeletal information 320 that has been deleted in step S1301 (step S1307: No), the process returns to step S1301, and the server 101 determines whether or not the skeletal information 320 has been deleted so far. Skeletons that do not exist are deleted (step S1301).

On the other hand, if all of the skeleton information 320 is deleted (step S1307: Yes), the process advances to step S1308. However, the determination of the process in step S1307 is not limited to this, and the server 101 may determine whether to return to step S1301 or proceed to step S1308 according to a predetermined number of repetitions. Alternatively, skeletons to be deleted may be determined in advance, and the server 101 may determine whether to return to step S1301 or proceed to step S1308 depending on whether all the predetermined skeletons have been deleted.

In step S1308, if there is a teacher signal that has not been selected yet among the teacher signals selected in step S1300 (step S1308: No), the server 101 selects a teacher signal that has not been selected so far (step S1300). . On the other hand, if all teacher signals have been selected (step S1308: Yes), the server 101 ends the behavioral learning process. However, the determination of the process in step S1308 is not limited to this, and the server 101 may determine whether to return to step S1300 or terminate the behavior learning process according to a predetermined number of repetitions.

<Action recognition processing>
FIG. 15 is a flowchart illustrating an example of a behavior recognition processing procedure by the client 102 (behavior recognition device) according to the first embodiment. The client 102 uses the skeleton detection unit 451 to detect the skeleton information 320 of the person appearing in the analysis target data acquired from the sensor 103 (step S1500). Next, in the client 102, the missing information determining unit 452 determines that the position information of the skeletal points that could not be detected due to occlusion or the like in the detected skeletal information 320 is missing information (step S1501).

Next, the client 102 uses the skeleton information processing unit 453 to perform skeleton information processing on the skeleton information 320 detected in step S1500, similar to the process in step S1302 (step S1502). Specifically, for example, the server 101 executes processing by a joint angle calculation unit 501, a movement amount calculation unit 502, and a normalization unit 503, as shown in FIG.

Next, the client 102 uses the skeletal information 320 normalized in step S1502, the joint angles 370, and the amount of movement between frames as input data to cause the principal component analysis unit 454 to perform principal component analysis, and then A principal component is generated, and a contribution rate and a cumulative contribution rate are also calculated along with the principal component (step S1503).

Next, the client 102 uses the dimension number determination unit 455 to determine how many principal components to use from among the generated principal components in descending order of variance from the calculated contribution rate and cumulative contribution rate (step S1504). .

Next, the client 102 uses the behavior classification model selection unit 456 to determine whether the behavior classification model generated by behavior learning is associated with the same missing information as the missing information detected in step S1501, and which is associated with the main component determined in step S1504. A behavior classification model that has undergone behavior learning using principal components having the same number of dimensions as the number of dimensions is selected (step S1505).

Next, the client 102 uses the behavior recognition unit 457 to recognize the behavior of the person reflected in the analysis target data acquired from the sensor 103 based on the behavior classification model and principal components selected in step S1505 (step S1506). The client 102 may send the recognition result to the server 101, and may also use the recognition result to control devices connected to the client 102.

For example, if the analysis environment in which the sensor 103 is installed is a factory, the behavior recognition system 100 can be applied to monitoring the work of workers in the factory, inspecting products for defects, etc. using the recognition results. . When the analysis environment is a train, the behavior recognition system 100 can be applied to monitoring passengers on the train, monitoring in-car equipment, detecting disasters such as fire, etc. using the recognition results.

In this manner, according to the first embodiment, multiple types of behaviors to be recognized can be recognized with high accuracy. In particular, even if some of the skeleton points 300 to 317 are missing due to occlusion or the like, it is possible to recognize a plurality of types of actions with high accuracy depending on the missing skeleton points.

The second embodiment will be described focusing on the differences from the first embodiment. Note that the same points as in the first embodiment are given the same reference numerals, and the explanation thereof will be omitted.

FIG. 16 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the second embodiment. In the second embodiment, the missing information control section 402 is deleted, and the missing information determining section 452 is changed to the missing information interpolating section 1652. As a result, when measuring the position of a person's movement, if part of the skeleton cannot be measured due to occlusion or the like and contains missing information, the missing information interpolation unit 1652 can remove the missing information from the measurable skeletal information 320. Interpolate information.

Specifically, for example, the missing information interpolation unit 1652 uses the position information of a skeleton point that could not be acquired due to occlusion or the like in the skeleton information 320 obtained from the skeleton detection unit 451 as missing information, and interpolates the missing information. It is output to the skeleton information processing section 453. The missing information interpolation unit 1652 may, for example, interpolate the missing information from skeleton points to be connected or skeleton points located near the missing information in the acquired skeleton information 320.

Furthermore, the missing information interpolation unit 1652 may substitute predetermined position information for the missing information. Furthermore, the missing information interpolation unit 1652 may interpolate the previously acquired skeletal information 320 of another frame using the missing information of the skeletal information 320 that has been determined to include missing information. In this way, the interpolation method for missing information is not limited.

<Learning process>
FIG. 17 is a flowchart illustrating a detailed processing procedure example of learning processing by the server 101 (learning device) according to the second embodiment. In the second embodiment, missing information control (step S1301) is not performed, and skeleton information processing (step S1302) is performed for the teacher signal selected in step S1300. That is, in the second embodiment, the behavior learning unit 406 generates one behavior classification model without distinguishing the skeletal information 320 regardless of the presence or absence of missing skeletal points.

<Action recognition processing>
FIG. 18 is a flowchart illustrating an example of a behavior recognition processing procedure by the client 102 (behavior recognition device) according to the second embodiment. In the second embodiment, missing information determination (step S1501) is changed to missing information interpolation (step S1801). The client 102 interpolates the position information of the skeleton points that could not be acquired due to occlusion or the like out of the skeleton information 320 detected in the skeleton detection (step S1500) with missing information, and updates it to the interpolated skeleton information 320 (step S1801). ). In the skeleton information processing (step S1502), a teacher signal including the interpolated skeleton information 320 is used.

In this manner, according to the second embodiment, by interpolating the skeleton information 320 that is missing due to occlusion or the like, it is not necessary to generate a behavior classification model for each piece of missing information. Thereby, it is possible to reduce the processing load of the learning function and speed up the action recognition function.

Example 3 is an example in which Example 1 and Example 2 are combined. Specifically, for example, in the behavior recognition system 100 according to the third embodiment, a first mode in which the learning process and behavior recognition process according to the second embodiment are executed, and a first mode in which the learning process and the behavior recognition process according to the second embodiment are executed by user operation. It is possible to switch to a second mode in which processing is executed.

In this way, according to the third embodiment, if you want to consider missing information, you can obtain highly accurate action recognition results by selecting the first mode, and if you want to interpolate the missing information, you can select the second mode. By doing so, action recognition results can be obtained efficiently.

Example 4 will be explained focusing on the differences from Examples 1 to 3. Note that the same points as in Examples 1 to 3 are given the same reference numerals, and the explanation thereof will be omitted.

FIG. 19 is a block diagram showing an example of the functional configuration of the skeleton information processing section according to the fourth embodiment. In the fourth embodiment, the skeleton information processing units 403 and 453 include a mutual information normalization unit 1904. The mutual information normalization unit 1904 normalizes the range of the skeletal information 320, joint angles 370, and inter-frame movement amounts to be within a certain range, which are output to the principal component analysis unit 404.

The range of the skeleton information 320 and the amount of movement between frames depends on the resolution of the data to be analyzed. On the other hand, the range of the joint angle 370 is from 0 to 2π, or from 0 degrees to 360 degrees. If there is a large difference in the range of the data to be subjected to principal component analysis, the influence on the principal components of the original data may be biased depending on the data type.

In order to eliminate this bias, the mutual information normalization unit 1904 performs normalization to bring the value range of the data to be applied to the principal component within a certain range. For example, the mutual information normalization unit 1904 unifies the range of the original data from 0 to 2π using the skeleton information 320 according to the following formulas (21) to (22) and the inter-frame movement amount according to the following formula (23).

However, the normalization method executed by the mutual information normalization unit 1904 is not limited to this; for example, the mutual information normalization unit 1904 may adjust the joint angle 370 The value range of may be normalized to a constant value.

FIG. 20 is a flowchart illustrating a detailed processing procedure example of the skeleton information processing unit according to the fourth embodiment. In the fourth embodiment, in the skeleton information processing (steps S1302, S1502), the client 102 performs mutual information normalization (step S2004) after normalization (step S1403). In mutual information normalization (step S2004), the possible value ranges of the skeletal information 320, joint angles 370, and inter-frame movement amounts normalized by the normalization unit are normalized to a constant value.

In this way, according to the fourth embodiment, by unifying the range of possible values of the original data (skeletal information 320, joint angles 370, amount of movement between frames) on which principal component analysis is performed, a wide range of values can be achieved. It is possible to eliminate bias in the influence of specific data on the principal components and to discriminate between multiple types of behavior with high accuracy.

Example 5 will be explained focusing on the differences from Examples 1 to 4. Note that the same reference numerals are given to the same points as in Examples 1 to 4, and the explanation thereof will be omitted.

FIG. 21 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the fifth embodiment. In the fifth embodiment, the principal component analysis section 404 and the principal component analysis section 445 are changed to a dimension reduction section 2100 and a dimension reduction section 2101. Dimensionality reduction is a process of reducing the number of original variables or the number of original dimensions while maintaining the original amount of information as much as possible. This is a concept that includes component analysis.

The dimension reduction unit 2100 executes dimension reduction using the normalized skeleton information 320, joint angles 370, and inter-frame movement amounts as input data among the teacher signals acquired from the skeleton information processing unit 403. One or more variables are generated and output to the number of dimensions control unit 405.

The dimension reduction methods performed by the dimension reduction unit 2100 include SNE (Stochastic Neighbor Embedding), t-SNE (t-Distributed Stochastic Neighbor Embedding), and UMAP (Uniform Manifold). Approximation and Projection), Isomap, LLE (Locally Linear Embedding), Techniques include Laplacian Eignmap, LargeVis, and Diffusion Map. The dimension reduction unit 2100 may reduce the dimension by combining t-SNE or UMAP with principal component analysis or independent component analysis. Below, each dimension reduction method and a dimension reduction method performed by combining each method will be explained.

The SNE processing will be explained using the following equations (24) to (28).

Let the similarity between two x coordinate values 322 (input data) x _i and x _j be the conditional probability p _{j |i} of selecting x _j as a neighbor when x _i is given. The conditional probability p _{j |i} is shown in the above equation (24). At this time, it is assumed that x _j is selected based on a normal distribution centered on x _i . Next, the similarity between the two y-coordinate values 323 (principal components) of y _i and y _j after dimension reduction is also calculated by the above equation (25), as well as the similarity between x _i and x _j before dimension reduction. Let the conditional probability q _j|i be However, the variance of the coordinate values after dimension reduction is fixed at 1/√2 to simplify the equation.

If y for dimension reduction is generated so as to maintain the distance relationship before and after dimension reduction, it is possible to reduce dimension while maintaining the amount of information as much as possible. In order to perform dimension reduction while suppressing a reduction in the amount of information, the dimension reduction unit 2100 performs processing so that p _{j |i} = q _{j |i} . KL divergence, which is a measure of how similar two probability distributions are, is used for dimension reduction.

The above equation (26) shows an equation in which the probability distribution before and after dimension reduction is adapted using the KL divergence as a loss function. The dimensionality reduction unit 2100 minimizes the loss function expressed in equation (26) using stochastic gradient descent. This gradient varies y _i using the above equation (27) in which the loss function is differentiated by y _i . The updating formula for this variation is shown by the above equation (28).

As described above, the above formula (28) is updated while varying y _i , and dimension reduction is performed by obtaining y _i that minimizes the above formula (27), and a new variable is obtained. However, in the case of SNE, unlike principal component analysis, the number of dimensions (variables) after reduction is two or three types due to processing characteristics. Therefore, when performing dimension reduction by SNE, a predetermined number of dimensions (variables) is output to the dimension number control unit 405, and the dimension number control unit 405 uses the number of variables to be used according to the predetermined number of dimensions. All you have to do is decide.

However, in SNE, it is difficult to minimize the loss function, and there is a problem that the skeleton points specified by the x-coordinate value 322 and the y-coordinate value 323 become dense when trying to maintain equidistantness during dimension reduction. t-SNE is a method for solving this problem.

The t-SNE processing will be explained using the following equations (29) to (33).

To simplify loss function minimization, we make the loss function symmetric. In the loss function symmetrization process, the distance between x _i and x _j is represented by a joint probability distribution p _ij , as shown in equation (29) above. p _j|i is similar to the above equation (24) and can be expressed by the above equation (30). Further, the distance between y _i and y _j after dimension reduction is expressed by the joint probability distribution q _ij shown in the above equation (31).

The distance between points after dimension reduction assumes Student's t distribution. The Student's t distribution is characterized by a higher probability of the existence of values that deviate from the mean value than the normal distribution, and this feature allows for a long distance distribution of the distance between data after dimension reduction. becomes possible.

In t-SNE, the dimensionality reduction unit 2100 performs dimensionality reduction by minimizing the loss function shown in the above equation (32) using pij and q _ij obtained from the above equations (29) to (31). . The dimension reduction unit 2100 uses the stochastic gradient descent method shown in the above equation (33) similarly to SNE to minimize the loss function.

As described above, by obtaining y _i that minimizes the above equation (33), the dimension reduction unit 2100 performs dimension reduction and obtains a new variable. Like SNE, t-SNE also has two or three types of dimensions (variables) after reduction due to processing characteristics. Therefore, when performing dimension reduction by t-SNE, a predetermined number of dimensions (variables) is output to the dimension number control unit 405, and the dimension number control unit 405 selects the variables to be used according to the predetermined number of dimensions. All you have to do is decide on the number of .

t-SNE maintains the high-dimensional local structure before dimension reduction and captures the global structure as much as possible, so it is possible to reduce the dimension accurately, but depending on the number of dimensions before dimension reduction. There is a problem that calculation time increases. UMAP is a method for solving the calculation time problem of dimension reduction. The UMAP processing will be explained using the following equations (34) to (36).

Among all possible values A, there is a high-dimensional set X (formula (34) above). Let μ be a membership function that, when arbitrary data is extracted from A, outputs the degree to which it is included in the set X in a range of 0 to 1. For input X shown in equation (1) above, Y shown in equation (2) above is prepared. Y is a set of m (<p) points existing in a space with a lower dimension than X, and is a set of data after dimension reduction. Then, by setting the membership function of Y to ν, the dimension reduction unit 2100 performs dimension reduction by determining Y such that the above equation (36) is minimized, and obtains a new variable.

When performing dimension reduction using UMAP, the dimension reduction unit 2100 may output a predetermined number of dimensions (variables) to the dimension number control unit 405 as in SNE and t-SNE, or may perform dimension reduction. The number of dimensions (variables) such that the subsequent membership function ν is greater than or equal to a predetermined range may be output to the dimension control unit 405 as the required number of dimensions. At this time, the dimension number control unit 405 may determine the number of dimensions (number of variables) to be used according to the number of dimensions (variables) output by the dimension reduction unit 2100.

Isomap processing will be explained. The dimensionality reduction unit 2100 calculates the shortest distance between neighboring data in arbitrary data, and performs dimensionality reduction by expressing the calculated distance in a geodesic distance matrix using multidimensional scaling (MDS). Get variable. When performing dimension reduction using Isomap, the dimension reduction unit 2100 outputs a predetermined number of dimensions (variables) to the dimension number control unit 405, and determines the number of variables to be used according to the predetermined number of dimensions. do it.

LLE will be explained using the following formulas (35) to (41).

Points in the vicinity of x _i are approximately represented by the above equation (35) by linear combination. Here, by minimizing the above equation (37) under the constraint of the above equation (36), the approximate value of x _i before dimension reduction is determined. Next, regarding y _i after dimension reduction, the dimension reduction unit 2100 minimizes the above equation (38) in order to maintain the linear adjacency relationship of x _i as much as possible even after dimension reduction. This solution is obtained as in the above equation (40) by extracting the eigenvectors of the above equation (39) from the second smallest eigenvalue v _i to the (d+1)th v _d . As shown in equation (41), y _i after dimension reduction is obtained.

When performing dimension reduction by LLE, the dimension reduction unit 2100 outputs a predetermined number of dimensions (variables) to the dimension number control unit 405, and the dimension number control unit 405 uses the dimension number according to the predetermined number of dimensions. All you have to do is decide the number of variables to use.

The processing of the Laplacian eigenmap will be explained using the following equations (42) to (47).

Each edge x _i x _j of the neighborhood graph generated by the data before dimension reduction is assigned to the above equation (42) or the above equation (43). Introducing the graph Laplacian of the above equation (44) for the assigned weights, and extracting the eigenvectors of the graph Laplacian (the above equation (45)) from the second smallest eigenvalue v _i to the (d+1)th v _d is obtained according to the above equation (46), and the dimension reduction unit 2100 obtains the value y _i after dimension reduction according to the above equation (47).

When implementing dimension reduction using the Laplacian eigenmap, the dimension reduction section 2100 outputs a predetermined number of dimensions (variables) to the dimension number control section 405, and the dimension number control section 405 follows the predetermined number of dimensions. , just decide the number of variables to use.

The processing of LargeVis will be explained. LargeVis is a method that improves the calculation time of t-SNE. In t-SNE, the distance between data points is determined, so the calculation time increases depending on the number of data. In LargeVis, the dimension reduction unit 2100 divides data into regions from neighboring data using a K-NN graph, and performs dimension reduction for each data model divided into regions using a method similar to t-SNE.

When implementing dimension reduction using LargeVis, the dimension number control unit 405 outputs a predetermined number of dimensions (variables) to the dimension number control unit 405, and determines the number of variables to be used according to the predetermined number of dimensions. All you have to do is decide.

The diffusion map will be explained using the following equations (48) to (53).

A weight W _ij is assigned to each edge x _i x _j of the neighborhood graph consisting of x _i before dimension reduction and the neighboring x _j , and this is normalized to obtain the N×N transition probability shown in equation (48) above. Create matrix P. Let p _t (x _i x _j ) represent the probability of starting from x _i and arriving at x _j after t steps by a random walk on the graph represented by P. From the properties of the transition matrix, p _t (x _i x _j ) converges to a stationary distribution φ ₀ (x _j ) at t→∞. At this time, the diffusion distance of the point x _i x _j is defined by the above equation (49). Let the eigenvalue of the transition probability matrix P be the above equation (50), and the eigenvector be the above equation (51). At this time, the above formula (52) holds true. Since the absolute value of λ _i is less than or equal to 1, the dimension reduction unit 2100 takes the eigenvectors up to an appropriate dimension d(t) smaller than N, performs dimension reduction according to the above equation (53), and creates a new variable. get.

When performing dimension reduction using a diffusion map, a predetermined number of dimensions (variables) is output to the dimension number control unit 405, and the dimension number control unit 405 controls the number of variables to be used according to the predetermined number of dimensions. All you have to do is decide.

The dimension reduction unit 2100 may perform a combination of the principal component analysis, independent component analysis, t-SNE, UMAP, Isomap, LLE, Laplacian eigenmap, LargeVis, diffusion map, etc. described above. For example, the dimension reduction unit 2100 uses principal component analysis to reduce the dimension to 10 dimensions for high-dimensional data with 36 dimensions or 36 variables, and then reduces the dimension to 2 dimensions by using UMAP. , the combination of methods used for dimensionality reduction is not limited. In this way, by combining various methods during dimensionality reduction, we can expect a composite effect on performance and calculation time.

Furthermore, these dimension reduction methods are not limited to the scope described in Example 5, and may include, for example, simply counting, subtracting, multiplying, or dividing high-dimensional information, or convolving it according to predetermined coefficients. However, the dimension reduction method is not limited as long as it is a method that generates high-dimensional data or multiple variables, lower-dimensional data, or a small number of variables, such as the method described in Example 5.

The dimension reduction unit 2101 has the same function as the dimension reduction unit 2100. The dimensionality reduction unit 2101 executes the same processing as the dimensionality reduction unit 2100 on the output data from the skeleton information processing unit 453, and generates one or more new variables that are smaller than before the dimensionality reduction. . Further, the principal component analysis unit 454 outputs the contribution rate and cumulative contribution rate generated together with the principal component to the number of dimensions determination unit 455.

The dimension reduction unit 2101 has the same function as the dimension reduction unit 2100. The dimension reduction unit 2101 performs the same processing as the dimension reduction unit 2100 on the output data from the skeleton information processing unit 453 to generate one or more new variables. In addition, the dimension reduction unit 2101 outputs information on the number of dimensions (variables) required by the dimension number determination unit 455 to the dimension number determination unit 455 along with new variables using a method similar to that of the dimension reduction unit 2100.

Based on the obtained number of dimensions (variables), the number of dimensions determination unit 455 determines the number of dimensions k indicating how many of the obtained variables are to be output to the behavior classification model selection unit 456, and newly outputs the determined number of variables. The generated variables are output to the behavior classification model selection unit 456.

As described above, according to the fifth embodiment, by changing the dimension reduction method, it is possible to reduce the dimension effectively or reduce the calculation time according to the data acquired from the skeleton information processing unit 403, and to reduce the complexity. behavior can be determined with high accuracy.

Example 6 will be explained focusing on the differences from Examples 1 to 5. Note that the same reference numerals are given to the points common to Examples 1 to 5, and the explanation thereof will be omitted.

FIG. 22 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the sixth embodiment. In the sixth embodiment, the behavior learning section 406 and the behavior recognition section 457 are changed to the behavior learning section 2200 and the behavior recognition section 2201. A detailed method by which the behavior learning unit 2200 and the behavior recognition unit 2201 classify behaviors will be explained using FIGS. 23 to 25.

FIG. 23 is an explanatory diagram showing a decision tree, which is a basic method by which the behavior learning unit 2200 and the behavior recognition unit 2201 classify behaviors. We will explain the behavior classification method using decision trees. In the decision tree, for each action in the variable space newly generated after dimension reduction, a boundary line 2310 (a) is generated using variables 2300 to 2303 given the type of action in advance.

(a) A method of generating the boundary line 2310 will be explained. The decision tree classifies actions step by step so that the impurity of the input variable group 2321 is minimized. In the first stage, the behavior is classified into variable group 2322 and variable group 2323 on the second variable axis, and in the second stage, the variable group 2322 and variable group 2323 are classified into variable groups 2324 to 2327 on the first variable axis. Classify. (a) A boundary line 2310 is generated using the discriminant obtained in the process of classifying so that the impurity is minimized. Note that there is no limitation on which axis the behavior is classified at each stage, and there is no limitation on the behavior classification on each axis by a prescribed number of times such as once.

FIG. 24 is an explanatory diagram showing a detailed method for developing classification using a decision tree. Decision trees include level-wise 2400, which grows a decision tree for each level (depth), and leaf-wise 2401, which grows a decision tree for each leaf (data group after branching). Learning by stacking classifiers such as decision trees is called ensemble learning.

FIG. 25 is an explanatory diagram showing ensemble learning and a method used by the behavior learning unit 2200 and the behavior recognition unit 2201 to classify behaviors. Ensemble learning includes bagging 2401 that uses classification trees such as decision trees in parallel, and boosting 2402 that updates learning results by inheriting previous results. The random forest of the first embodiment is a method that uses bagging 2401 for decision trees, and the behavior learning section 2200 and the behavior recognition section 2201 of the sixth embodiment are a classification method that uses boosting 2402.

When the behavior learning unit 2200 learns the behavior and the behavior recognition unit 2201 classifies the behavior, it is possible to grow each decision tree levelwise and classify the input variables by boosting multiple decision trees. Alternatively, each decision tree may be grown leafwise, and input variables may be classified by boosting, which overlaps a plurality of decision trees.

Note that when employing boosting, in which each decision tree is grown levelwise and multiple decision trees are stacked, as a behavior classification method, it may be implemented using the software library xgboost. On the other hand, when employing boosting, in which each decision tree is grown leafwise and multiple decision trees are stacked, as a behavior classification method, it may be implemented using the software library LightGBM. However, implementation methods are not limited to these.

In this way, according to the sixth embodiment, by using boosting as the behavior classification method and overlapping a plurality of decision trees, complex behaviors can be determined with high accuracy.

Example 7 will be explained focusing on the differences from Examples 1 to 6. Note that the same reference numerals are given to the same points as in Examples 1 to 6, and the explanation thereof will be omitted.

FIG. 26 is a block diagram showing an example of the functional configuration of the behavior recognition system 100 according to the seventh embodiment. In the seventh embodiment, the dimension reduction unit 2100, the dimension number control unit 405, the behavior learning unit 406, and the dimension number determination unit 455 are changed to the dimension reduction unit 2600, the dimension number control unit 2601, the behavior learning unit 2602, and the dimension reduction unit 2603. be done.

The dimensionality reduction unit 2600 performs dimensionality reduction using any of the methods of Examples 1 to 6 according to a predetermined number of dimensions, and outputs new variables generated after dimensionality reduction to the dimensionality number control unit 2601. The number of dimensions control unit 2601 outputs the variable after dimension reduction to the behavior learning unit 2602 according to the obtained number of dimensions.

The behavior learning unit 2602 generates a boundary line for behavior classification by machine learning from the given behavior type together with the obtained variable after dimension reduction, and generates a behavior classification model. At this time, the behavior classification accuracy, which indicates how accurately the behavior can be predicted, is calculated for the generated behavior classification model.

The behavior learning unit 2602 may calculate the behavior classification accuracy using the variables used to generate the behavior classification model. The behavior learning unit 2602 may not use some of the variables acquired from the dimension control unit 2600 for behavior classification model generation, and may calculate behavior classification accuracy using variables that are not used for behavior classification generation. However, the method of calculating behavior classification accuracy is not limited to these. If the calculated behavior classification accuracy is higher than the predetermined accuracy, the behavior learning unit 2602 outputs the generated behavior classification model to the behavior classification model selection unit 456. At this time, the behavior learning unit 2602 outputs to the dimension control unit 2601 that the acquired number of dimensions and behavior classification accuracy are acceptable.

On the other hand, if the calculated behavior classification accuracy is lower than the predetermined accuracy, the behavior learning unit 2602 outputs to the dimension control unit 2601 that the behavior classification accuracy has failed. However, if behavior classification models are generated using all settable dimensions (variables) and the behavior classification accuracy fails in all of them, the behavior learning unit 2602 Outputs the behavior classification model with the highest behavior classification accuracy among the models to the behavior classification model selection unit 456, and outputs the number of dimensions (variables) used at the time of output to the number of dimensions control unit 2601 together with all learning completion information. .

When the dimension control unit 2601 acquires pass or complete learning information according to the pass/fail information and all learning completion information acquired from the behavior learning unit 2602, it outputs the acquired dimension number information to the dimension reduction unit 2603 and If it passes, a dimension reduction command is output to the dimension reduction unit 2600 to change the number of dimensions used for dimension reduction and perform dimension reduction again.

In accordance with the obtained dimension reduction command, the dimension reduction unit 2600 sets the number of dimensions that have not been set so far, performs dimension reduction again, and outputs the generated variables to the dimension number control unit 2601.

The dimension reduction unit 2603 performs dimension reduction on the data obtained from the skeleton information processing unit 452 according to the number of dimensions (variables) obtained from the number of dimensions control unit 2601 using the dimension reduction methods of Examples 1 to 6. , outputs the generated variables to the behavior classification model selection unit 456. Note that without determining the behavior classification accuracy for determining pass/fail, the dimension reduction unit 2603 performs learning with all settable dimensions, calculates the behavior classification accuracy, and then creates a behavior classification model according to the calculated behavior classification accuracy. The number of dimensions may also be determined.

The behavior classification accuracy calculated by the behavior learning unit 2602 may be regarded as the contribution rate described in the first embodiment. For example, by associating the obtained variable after dimension reduction with the behavior classification accuracy calculated using it, the calculated behavior classification accuracy is the contribution rate to the original information of the variable after dimension reduction used for calculation. do. The dimension control unit 2601 determines which of the variables after dimension reduction is used for control according to the contribution rate estimated in this way.

<Learning process>
FIG. 27 is a flowchart illustrating a detailed processing procedure example of learning processing by the server 101 (learning device) according to the seventh embodiment. The server 101 determines the number of dimensions using the number of dimensions control unit 2601. At this time, when performing dimension reduction for the first time, a predetermined number of dimensions is determined, and when dimension reduction is performed for the second time or later, a previously undetermined number of dimensions is determined (step S2700).

Next, the server 101 performs dimension reduction in the dimension reduction unit 2601 according to the determined number of dimensions, and generates a new variable (S2701). In step S2702, the server 101 makes a pass/fail judgment on the behavior classification accuracy acquired from the behavior learning unit 2602. If it passes, the process advances to step S1307, and if it fails, the process returns to step S2700.

In this manner, according to the seventh embodiment, by changing the number of dimensions in accordance with the target behavior classification accuracy and repeating dimension reduction, complex behaviors can be determined with high accuracy.

Furthermore, the behavior recognition devices and learning devices of Examples 1 to 7 described above can also be configured as shown in (1) to (14) below.

(1) A behavior recognition device (client 102) having a processor 201 that executes a program and a storage device 202 that stores the program performs component analysis (principal component analysis or The behavior classification model group learned for each component group can be accessed using the component group obtained from the shape of the learning target (skeletal information 320) by independent component analysis) and the behavior of the learning target, and the behavior classification model group learned for each component group can be accessed. The processor 201 performs a detection process to detect the shape of the recognition target (skeletal information 320) from the analysis target data obtained from the sensor 103, and performs the component analysis based on the shape of the recognition target detected by the detection process. , an ordinal number k indicating each of the one or more dimensions based on a component analysis process that generates one or more components and a contribution rate of each of the components, and a cumulative contribution rate obtained from each of the contribution rates. and a specific behavior classification model learned with the same component group as a specific component group containing one or more components with an ordinal number k indicating the dimension determined by the determination process, from the behavior classification model group. A selection process for selecting, and a behavior recognition process for outputting a recognition result indicating the behavior to be recognized by inputting the specific component group to a specific behavior classification model selected by the selection process. .

Thereby, since a behavior classification model is prepared according to the shape of the learning target, multiple types of behaviors to be recognized can be recognized with high accuracy.

(2) In the behavior recognition device of (1) above, each behavior classification model of the behavior classification model group is obtained from the shape of the learning target and the angles (joint angles 370) of a plurality of vertices constituting the shape. Learning is performed for each component group using a component group and the behavior of the learning target, and the processor 201 determines the shape of a plurality of vertices constituting the shape of the recognition target based on the shape of the recognition target. A calculation process for calculating an angle (joint angle 370) is executed, and in the component analysis process, the processor 201 calculates the shape of the recognition target and the angle of the vertex of the recognition target calculated by the calculation process. Based on the above, the one or more components and the contribution rate are generated.

Thereby, multiple types of actions to be recognized can be recognized with high accuracy according to changes in shape caused by the angle of the vertices.

(3) In the behavior recognition device of (1) above, each behavior classification model of the behavior classification model group includes a component group obtained from the shape of the learning target and the amount of movement of the learning target, and the behavior of the learning target. are learned for each component group using In the component analysis process, the processor 201 generates the one or more components and the contribution rate based on the shape of the recognition target and the movement amount of the recognition target calculated by the calculation process. do.

Thereby, multiple types of actions of the recognition target can be recognized with high accuracy according to changes in shape over time due to movement.

(4) In the behavior recognition device of (1) above, the processor 201 executes a first normalization process to normalize the size of the shape of the recognition target, and in the component analysis process, the processor 201 The one or more components and the contribution rate are generated based on the shape of the recognition target after first normalization by the first normalization process.

Thereby, the versatility of behavior classification is improved, and erroneous recognition can be suppressed.

(5) In the behavior recognition device of (2) above, the processor 201 executes a second normalization process to normalize the range of possible values of the shape of the recognition target and the angle of the vertex, and in the component analysis process, The processor 201 generates the one or more components and the contribution rate based on the shape of the recognition target and the angle of the vertex (joint angle 370) after second normalization by the second normalization process. do.

As a result, it is possible to suppress bias in the range of different data types such as shape and angle, and it is possible to improve the accuracy of behavior recognition.

(6) In the behavior recognition device of (1) above, in the determination process, the processor 201 determines an ordinal number k indicating the dimension of the component necessary for the cumulative contribution rate to exceed a threshold value.

The cumulative contribution rate is a measure of how much the newly generated components represent the amount of information in the original data, so by referring to the cumulative contribution rate, it is possible to suppress the increase in the number of dimensions. be able to.

(7) In the behavior recognition device of (1) above, each behavior classification model of the behavior classification model group uses a component group obtained from a partially missing shape of the learning target and the behavior of the learning target. The processor 201 performs learning for each combination of the partially missing shape and the component group, and the processor 201 performs the judgment process of determining the partially missing shape of the recognition target and the component analysis process. generates the one or more components and the contribution rate of each of the one or more components based on the partially missing shape of the recognition target determined by the determination process, and in the selection process, The processor 201 selects a specific behavior classification model from the behavior classification model group that has been trained using the same missing shape as the partially missing shape of the recognition target and the same component group as the specific component group.

Even if the shape of the recognition target is partially missing, highly accurate behavior recognition can be performed using a behavior classification model that corresponds to the partially missing shape.

(8) In the behavior recognition device of (1) above, the processor 201 executes an interpolation process to interpolate if there is a partial defect in the shape of the recognition target, and in the component analysis process, the processor 201 The one or more components and the contribution rate are generated based on the shape of the recognition target after interpolation by interpolation processing.

Thereby, appropriate input can be given to the behavior classification model generated by the learning target with no defects in shape, and a decrease in behavior recognition accuracy can be suppressed.

(9) The behavior recognition device (client 102), which includes a processor 201 that executes a program and a storage device 202 that stores the program, performs dimension reduction (principal component analysis or Independent component analysis or SNE (Stochastic Neighbor Embedding) or t-SNE (t-Distributed Stochastic Neighbor Embedding) or UMAP (Uniform Manifold Approximation) on and Projection) or Isomap or LLE (Locally Linear Embedding) or Laplacian Eignmap or LargeVis A behavior classification model group learned for each component group using the component group in ascending order from the first variable obtained from the shape of the learning target (skeletal information 320) using a diffusion map) and the behavior of the learning target. The processor 201 performs a detection process of detecting the shape of the recognition target (skeletal information 320) from the analysis target data obtained from the sensor 103, and performs the dimension reduction to detect the shape of the recognition target from the analysis target data obtained from the sensor 103, and the Dimension reduction processing that generates one or more components and a contribution rate of each of the components based on the shape of the recognition target, and a dimension reduction process that generates one or more components and a contribution rate of each of the components, and a a determination process for determining an ordinal number k indicating the dimension of the component in ascending order from the first variable among the components; and a specific component group from the first variable to the component with the ordinal number k indicating the dimension determined by the determination process. A selection process of selecting a specific behavior classification model learned with the same component group from the behavior classification model group, and inputting the specific component group to the specific behavior classification model selected by the selection process, and a behavior recognition process of outputting a recognition result indicating the behavior of the recognition target.

Thereby, since a behavior classification model is prepared according to the shape of the learning target, multiple types of behaviors to be recognized can be recognized with high accuracy.

(10) In the behavior recognition device of (9) above, each behavior classification model of the behavior classification model group is obtained from the shape of the learning target and the angles (joint angles 370) of a plurality of vertices constituting the shape. Learning is performed for each component group using the component group in ascending order from the first variable and the behavior of the learning target, and the processor 201 determines the shape of the recognition target based on the shape of the recognition target. In the dimension reduction process, the processor 201 calculates the shape of the recognition target and the recognition target calculated by the calculation process. The one or more components and the contribution rate are generated based on the angle of the vertex.

Thereby, multiple types of actions to be recognized can be recognized with high accuracy according to changes in shape caused by the angle of the vertices.

(11) In the behavior recognition device of (9) above, each behavior classification model of the behavior classification model group is a component group in ascending order from a first variable obtained from the shape of the learning target and the amount of movement of the learning target. and the behavior of the learning target are learned for each component group, and the processor 201 calculates the amount of movement of the recognition target based on a plurality of shapes of the recognition target at different times. In the dimension reduction process, the processor 201 calculates the one or more components based on the shape of the recognition target and the movement amount of the recognition target calculated by the calculation process. The contribution rate is generated.

Thereby, multiple types of actions of the recognition target can be recognized with high accuracy according to changes in shape over time due to movement.

(12) In the behavior recognition device of (9) above, the processor 201 executes a first normalization process to normalize the size of the shape of the recognition target, and in the dimension reduction process, the processor 201 The one or more components and the contribution rate are generated based on the shape of the recognition target after first normalization by the first normalization process.

Thereby, by improving the versatility of behavior classification, it is possible to suppress misrecognition.

(13) In the behavior recognition device of (10) above, the processor 201 executes a second normalization process to normalize the range of possible values of the shape of the recognition target and the angle of the vertex, and in the dimension reduction process, The processor 201 generates the one or more components and the contribution rate based on the shape of the recognition target and the angle of the vertex (joint angle 370) after second normalization by the second normalization process. do.

As a result, it is possible to suppress bias in the range of different data types such as shape and angle, and it is possible to improve the accuracy of behavior recognition.

(14) In the behavior recognition device of (9) above, in the determination process, the processor 201 selects the values in ascending order from the first variable that are necessary for the cumulative contribution rate from the first variable to exceed a threshold. Determine the ordinal number k indicating the dimension of the component.

The cumulative contribution rate is a measure of how much the newly generated components represent the amount of information in the original data, so by referring to the cumulative contribution rate, it is possible to suppress the increase in the number of dimensions. be able to.

(15) In the behavior recognition device according to (9) above, each behavior classification model of the behavior classification model group includes a component group in ascending order from a first variable obtained from a partially missing shape of the learning target, and the learning object. The behavior of the target is learned for each combination of the partially missing shape and the component group, and the processor 201 performs a judgment process for determining the partially missing shape of the recognition target, and the dimension In the reduction process, the processor 201 generates the one or more components and the contribution rate of each of the one or more components based on the partially missing shape of the recognition target determined by the determination process. In the selection process, the processor 201 selects a specific behavior classification model learned using the same missing shape as the partially missing shape of the recognition target and a combination of the same component group as the specific component group. Select from a group of classification models.

Even if the shape of the recognition target is partially missing, highly accurate behavior recognition can be performed using a behavior classification model that corresponds to the partially missing shape.

(16) In the behavior recognition device of (9) above, the processor 201 executes an interpolation process to interpolate if there is a partial defect in the shape of the recognition target, and in the dimension reduction process, the processor 201 The one or more components and the contribution rate are generated based on the shape of the recognition target after interpolation by interpolation processing.

Thereby, appropriate input can be given to the behavior classification model generated by the learning target with no defects in shape, and a decrease in behavior recognition accuracy can be suppressed.

(17) In a learning device that includes a processor 201 that executes a program and a storage device 202 that stores the program, the processor 201 performs an acquisition process that acquires teacher data including the shape and behavior of a learning target, and A component analysis process that generates one or more components based on the shape of the learning target acquired by the acquisition process by component analysis (principal component analysis or independent component analysis) that generates statistical components by variable analysis; , a control process for controlling ordinal numbers indicating dimensions of each of the one or more components based on an allowable amount of calculation; a component group including one or more ordinal components indicating the dimensions controlled by the control process; and the learning. and a behavior learning process of learning the behavior of the learning target based on the behavior of the target and generating a behavior classification model for classifying the behavior of the learning target.

As a result, it is possible to prepare a plurality of types of behavior classification models according to the shape of the learning target, and therefore, it is possible to recognize multiple types of recognition target behaviors with high accuracy.

(18) In the learning device of (17) above, the processor 201 performs a calculation process of calculating angles (joint angles 370) of a plurality of vertices constituting the shape of the learning target based on the shape of the learning target. In the component analysis process, the processor 201 generates the one or more components based on the shape of the learning target and the angle of the vertex of the learning target calculated by the calculation process.

As a result, multiple types of behavior classification models can be prepared according to changes in shape caused by the angle of the vertex to be recognized. It can be recognized with high precision.

(19) In the learning device of (17) above, the processor 201 executes a calculation process for calculating the movement amount of the learning target based on a plurality of shapes of the learning target at different times, and performs the component analysis process. Then, the processor 201 generates the one or more components based on the shape of the learning target and the amount of movement of the learning target calculated by the calculation process.

As a result, it is possible to prepare multiple types of behavior classification models according to changes in shape over time due to movement. Can be recognized with precision.

(20) In the learning device of (17) above, the processor 201 executes a first normalization process to normalize the size of the shape of the learning target, and in the component analysis process, the processor 201 performs the first normalization process to normalize the size of the shape of the learning target. The one or more components are generated based on the shape of the learning target after first normalization by a first normalization process.

Thereby, by improving the versatility of behavior classification learning, it is possible to suppress erroneous learning.

(21) In the learning device of (18) above, the processor 201 executes a second normalization process to normalize the range of possible values of the shape of the learning target and the angle of the vertex, and in the component analysis process, the The processor 201 generates the one or more components based on the shape of the learning target and the angle of the vertex after second normalization by the second normalization process.

Thereby, it is possible to suppress bias in the range of different data types such as shape and angle, and it is possible to improve the accuracy of behavior classification learning.

(22) In the learning device of (17) above, the processor 201 executes a loss control process for partially missing the shape of the learning target, and in the component analysis process, the processor 201 performs a loss control process by the loss control process. The one or more components are generated based on the obtained partially missing shape of the learning target, and in the behavior learning process, the processor 201 generates the one or more components based on the component group and the behavior of the learning target. Then, the behavior of the learning target is learned, the behavior classification model is generated, and the behavior classification model is associated with the missing information regarding the partially missing shape.

By generating a shape with some parts intentionally missing, it is possible to increase the number of types of behavior classification models. This enables highly accurate behavior recognition that corresponds to various shapes of recognition targets.

(23) In a learning device that includes a processor 201 that executes a program and a storage device 202 that stores the program, the processor 201 performs an acquisition process that acquires teacher data including the shape and behavior of a learning target, and Dimensionality reduction (Principal Component Analysis or Independent Component Analysis or SNE (Stochastic Neighbor Embedding) or t-SNE (t-Distributed Stochastic Neighbor Embedding) or UMAP (Uniform Mani fold Approximation and Projection) or Isomap or a dimension reduction process that generates one or more components based on the shape of the learning target acquired by the acquisition process using LLE (Locally Linear Embedding), Laplacian Eignmap, LargeVis, or Diffusion Map); , a control process that controls an ordinal number indicating the dimension of the component in ascending order from the first variable among the one or more components based on an allowable amount of calculation; and a control process that indicates the dimension controlled by the control process from the first variable. Behavior learning processing that generates a behavior classification model that classifies the behavior of the learning target by learning the behavior of the learning target based on a component group up to an ordinal component and the behavior of the learning target. Execute.

As a result, it is possible to prepare a plurality of types of behavior classification models according to the shape of the learning target, and therefore, it is possible to recognize multiple types of recognition target behaviors with high accuracy.

(24) In the learning device of (23) above, the processor 201 performs a calculation process of calculating the angles (joint angles 370) of a plurality of vertices constituting the shape of the learning target based on the shape of the learning target. In the dimension reduction process, the processor 201 generates the one or more components based on the shape of the learning target and the angle of the vertex of the learning target calculated by the calculation process.

As a result, multiple types of behavior classification models can be prepared according to changes in shape caused by the angle of the vertex to be recognized. It can be recognized with high precision.

(25) In the learning device of (23) above, the processor 201 executes a calculation process for calculating the movement amount of the learning target based on a plurality of shapes of the learning target at different times, and performs the dimension reduction process. Then, the processor 201 generates the one or more components based on the shape of the learning target and the amount of movement of the learning target calculated by the calculation process.

As a result, it is possible to prepare multiple types of behavior classification models according to changes in shape over time due to movement. Can be recognized with precision.

(26) In the learning device of (23) above, the processor 201 executes a first normalization process to normalize the size of the shape of the learning target, and in the dimension reduction process, the processor 201 The one or more components are generated based on the shape of the learning target after first normalization by a first normalization process.

Thereby, by improving the versatility of behavior classification learning, it is possible to suppress erroneous learning.

(27) In the learning device of (24) above, the processor 201 executes a second normalization process to normalize the range of possible values of the shape of the learning target and the angle of the vertex, and in the dimension reduction process, the The processor 201 generates the one or more components based on the shape of the learning target and the angle of the vertex after second normalization by the second normalization process.

Thereby, it is possible to suppress bias in the range of different data types such as shape and angle, and it is possible to improve the accuracy of behavior classification learning.

(28) In the learning device of (23) above, the processor 201 executes a loss control process for partially missing the shape of the learning target, and in the dimension reduction process, the processor 201 performs the loss control process by the loss control process. The one or more components are generated based on the obtained partially missing shape of the learning target, and in the behavioral learning process, the processor 201 generates the components from the first variable to the ordinal component indicating the dimension. The behavior of the learning target is learned based on the component group and the behavior of the learning target, the behavior classification model is generated, and the behavior is associated with the missing information regarding the partially missing shape.

By generating a shape with some parts intentionally missing, it is possible to increase the number of types of behavior classification models. This enables highly accurate behavior recognition that corresponds to various shapes of recognition targets.

Note that the present invention is not limited to the embodiments described above, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the embodiments described above have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Further, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Further, the configuration of one embodiment may be added to the configuration of another embodiment. Furthermore, other configurations may be added to, deleted from, or replaced with some of the configurations of each embodiment.

Further, each of the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing an integrated circuit, for example, and the processor 201 may implement each function. It may be realized by software by interpreting and executing a program to be realized.

Information such as programs, tables, and files that realize each function is stored in storage devices such as memory, hard disks, and SSDs (Solid State Drives), or on IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs). It can be stored on a medium.

Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, almost all configurations can be considered interconnected.

100 Behavior recognition system 101 Server 102 Client 103 Sensor 104 Teacher signal DB
201 Processor 202 Storage device 320 Skeleton information 401 Teacher signal acquisition section 402 Missing information control section 403, 453 Skeletal information processing section 404, 454 Principal component analysis section 405, 2601 Number of dimensions control section 406, 2200, 2602 Behavior learning section 451 Skeleton detection Unit 452 Missing information judgment unit 455 Dimension number deciding unit 456 Behavior classification model selection unit 457, 2201 Behavior recognition unit 501 Joint angle calculation unit 502 Movement amount calculation unit 503 Normalization unit 1652 Missing information interpolation unit 1904 Mutual information normalization unit 2100, 2101, 2600, 2603 Dimension reduction part

Claims (20)

An action recognition device comprising a processor that executes a program, and a storage device that stores the program,
Using the component group obtained from the shape of the learning target through component analysis that generates statistical components through multivariate analysis and the behavior of the learning target, it is possible to access the behavior classification model group learned for each component group. and
The processor includes:
Detection processing that detects the shape of the recognition target from the analysis target data,
A component analysis process that generates one or more components and a contribution rate of each of the components based on the shape of the recognition target detected by the detection process by the component analysis;
a determination process of determining an ordinal number indicating the dimension of each of the one or more components based on the cumulative contribution rate obtained from each of the contribution rates;
a selection process of selecting a specific behavior classification model trained with the same component group as a specific component group that includes one or more ordinal components indicating the dimension determined by the determination process from the behavior classification model group;
a behavior recognition process that outputs a recognition result indicating the behavior to be recognized by inputting the specific component group to a specific behavior classification model selected by the selection process;
An action recognition device characterized by performing. The behavior recognition device according to claim 1,
Each of the behavior classification models in the behavior classification model group uses a component group obtained from the partially missing shape of the learning target and the behavior of the learning target to combine the partially missing shape and the component group. It is learned every
The processor includes:
a determination process for determining a partially missing shape of the recognition target;
In the component analysis process, the processor calculates the one or more components and the contribution rate of each of the one or more components based on the partially missing shape of the recognition target determined by the determination process. generate,
In the selection process, the processor selects a specific behavior classification model learned using the same missing shape as the partially missing shape of the recognition target and the same component group as the specific component group, into the behavior classification model group. choose from;
An action recognition device characterized by: The behavior recognition device according to claim 1,
The processor includes:
Performing interpolation processing to interpolate if there is a partial defect in the shape of the recognition target,
In the component analysis process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target after interpolation by the interpolation process.
An action recognition device characterized by: An action recognition device comprising a processor that executes a program, and a storage device that stores the program,
Each component group is learned using the component group in ascending order from the first variable obtained from the shape of the learning target by dimension reduction that generates statistical components through multivariate analysis, and the behavior of the learning target. Has access to a group of behavioral classification models,
The processor includes:
Detection processing that detects the shape of the recognition target from the analysis target data,
Dimension reduction processing that generates one or more components and a contribution rate of each of the components based on the shape of the recognition target detected by the detection processing by the dimension reduction;
a determination process of determining an ordinal number indicating the dimension of the component in ascending order from the first variable among the one or more components, based on each of the contribution rates;
a selection process of selecting a specific behavior classification model learned with the same component group as a specific component group from the first variable to an ordinal component indicating the dimension determined by the determination process from the behavior classification model group; ,
a behavior recognition process that outputs a recognition result indicating the behavior to be recognized by inputting the specific component group to a specific behavior classification model selected by the selection process;
An action recognition device characterized by performing. The behavior recognition device according to claim 4,
Each behavior classification model of the behavior classification model group includes a component group in ascending order from a first variable obtained from the shape of the learning target and the angles of a plurality of vertices constituting the shape, the behavior of the learning target, is learned for each component group using
The processor includes:
Based on the shape of the recognition target, performing a calculation process of calculating angles of a plurality of vertices forming the shape of the recognition target;
In the dimension reduction process, the processor calculates the one or more components and the contribution rate based on the shape of the recognition target and the angle of the vertex of the recognition target calculated by the calculation process. generate,
An action recognition device characterized by: The behavior recognition device according to claim 4,
Each behavior classification model of the behavior classification model group uses a component group in ascending order from a first variable obtained from the shape of the learning target and the amount of movement of the learning target, and the behavior of the learning target, It is learned for each component group,
The processor includes:
executing a calculation process for calculating a movement amount of the recognition target based on a plurality of shapes of the recognition target at different times;
In the dimension reduction process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target and the movement amount of the recognition target calculated by the calculation process. do,
An action recognition device characterized by: The behavior recognition device according to claim 4,
The processor includes:
performing a first normalization process to normalize the size of the shape of the recognition target;
In the dimension reduction process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target after first normalization by the first normalization process.
An action recognition device characterized by: The behavior recognition device according to claim 5,
The processor includes:
performing a second normalization process to normalize the range of possible values of the shape of the recognition target and the angle of the vertex;
In the dimension reduction process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target and the angle of the vertex after second normalization by the second normalization process. do,
An action recognition device characterized by: The behavior recognition device according to claim 4,
In the determination process, the processor determines an ordinal number indicating the dimension of an ascending component from the first variable necessary for the contribution rate to exceed a threshold;
An action recognition device characterized by: The behavior recognition device according to claim 4,
Each of the behavior classification models of the behavior classification model group uses a component group in ascending order from the first variable obtained from the partially missing shape of the learning target and the behavior of the learning target. It is learned for each combination of shape and component group,
The processor includes:
a determination process for determining a partially missing shape of the recognition target;
In the dimension reduction process, the processor calculates the one or more components and the contribution rate of each of the one or more components based on the partially missing shape of the recognition target determined by the determination process. generate,
In the selection process, the processor selects a specific behavior classification model learned using the same missing shape as the partially missing shape of the recognition target and the same component group as the specific component group, into the behavior classification model group. choose from,
An action recognition device characterized by: The behavior recognition device according to claim 4,
The processor includes:
Performing interpolation processing to interpolate if there is a partial defect in the shape of the recognition target,
In the dimension reduction process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target after interpolation by the interpolation process.
An action recognition device characterized by: A learning device comprising a processor that executes a program and a storage device that stores the program,
The processor includes:
an acquisition process that acquires training data including the shape and behavior of the learning target;
a component analysis process that generates one or more components based on the shape of the learning target acquired by the acquisition process by a component analysis that generates statistical components by multivariate analysis;
A control process that controls ordinal numbers indicating dimensions of each of the one or more components based on an allowable amount of calculation;
The behavior of the learning target is learned and the behavior of the learning target is classified based on the behavior of the learning target and a component group including one or more ordinal components indicating the dimension controlled by the control process. Behavior learning processing that generates a behavior classification model;
A learning device characterized by performing the following. The learning device according to claim 12,
The processor includes:
Execute a loss control process to partially lose the shape of the learning target,
In the component analysis process, the processor generates the one or more components based on the partially missing shape of the learning target obtained by the loss control process,
In the behavior learning process, the processor:
learning the behavior of the learning target based on the component group and the behavior of the learning target, generating the behavior classification model, and associating it with missing information regarding the partially missing shape;
A learning device characterized by: A learning device comprising a processor that executes a program and a storage device that stores the program,
The processor includes:
an acquisition process that acquires training data including the shape and behavior of the learning target;
Dimension reduction processing that generates one or more components based on the shape of the learning target acquired by the acquisition process by dimension reduction that generates statistical components by multivariate analysis;
A control process that controls ordinal numbers indicating dimensions of components in ascending order from a first variable among the one or more components based on an allowable amount of calculation;
Learning the behavior of the learning target based on a component group from the first variable to an ordinal component indicating a dimension controlled by the control process and the behavior of the learning target, and learning the behavior of the learning target. a behavior learning process that generates a behavior classification model that classifies the
A learning device characterized by performing the following. The learning device according to claim 14,
The processor includes:
Based on the shape of the learning target, performing a calculation process of calculating angles of a plurality of vertices forming the shape of the learning target,
In the dimension reduction process, the processor generates the one or more components based on the shape of the learning target and the angle of the vertex of the learning target calculated by the calculation process.
A learning device characterized by: The learning device according to claim 14,
The processor includes:
Executing a calculation process to calculate a movement amount of the learning target based on a plurality of shapes of the learning target at different times,
In the dimension reduction process, the processor generates the one or more components based on the shape of the learning target and the amount of movement of the learning target calculated by the calculation process.
A learning device characterized by: The learning device according to claim 14,
The processor includes:
performing a first normalization process to normalize the size of the shape of the learning target;
In the dimension reduction process, the processor generates the one or more components based on the shape of the learning target after first normalization by the first normalization process.
A learning device characterized by: The learning device according to claim 15,
The processor includes:
performing a second normalization process to normalize the range of possible values of the shape of the learning target and the angle of the vertex;
In the dimension reduction process, the processor generates the one or more components based on the shape of the learning target and the angle of the vertex after second normalization by the second normalization process.
A learning device characterized by: The learning device according to claim 14,
The processor includes:
Execute a loss control process to partially lose the shape of the learning target,
In the dimension reduction process, the processor generates the one or more components based on the partially missing shape of the learning target obtained by the loss control process,
In the behavior learning process, the processor:
Learning the behavior of the learning target based on a component group from the first variable to an ordinal component indicating the dimension and the behavior of the learning target to generate the behavior classification model; Correlate with missing information regarding the missing shape,
A learning device characterized by: An action recognition method executed by an action recognition device having a processor that executes a program, and a storage device that stores the program, the method comprising:
Using the component group obtained from the shape of the learning target through component analysis that generates statistical components through multivariate analysis and the behavior of the learning target, it is possible to access the behavior classification model group learned for each component group. and
The behavior recognition method includes:
The processor,
Detection processing that detects the shape of the recognition target from the analysis target data,
A component analysis process that generates one or more components and a contribution rate of each of the components based on the shape of the recognition target detected by the detection process by the component analysis;
a determination process of determining an ordinal number indicating the dimension of each of the one or more components based on the cumulative contribution rate obtained from each of the contribution rates;
a selection process of selecting a specific behavior classification model trained with the same component group as a specific component group that includes one or more ordinal components indicating the dimension determined by the determination process from the behavior classification model group;
a behavior recognition process that outputs a recognition result indicating the behavior to be recognized by inputting the specific component group to a specific behavior classification model selected by the selection process;
An action recognition method characterized by performing the following.

Images