JP2022043974A - Behavior recognition apparatus, learning apparatus, and behavior recognition method - Google Patents

Abstract

To recognize behavior of multiple kinds of recognition objects highly accurately.SOLUTION: A behavior recognition apparatus can access a behavior classification model group learned for each component group by using a behavior for learning and a component group obtained from a shape for learning by component analysis which generates statistical components by multivariable analysis, detects a shape of a recognition object from data for analysis, generates one or more components and contribution rates of each component on the basis of the shape of the recognition object by component analysis, determines an ordinal number indicating dimensions of one or more components on the basis of a cumulative contribution rate obtained from the contribution rates, selects a specific behavior classification model leaned with the same component group as the specific component group including one or more components of the determined ordinal number indicating the dimensions, from the behavior classification model group, and inputs the specific component group to the specific behavior classification model to output a recognition result indicating behavior of the recognition object.SELECTED DRAWING: Figure 4

Description

The present invention relates to a behavior recognition device, a learning device, and a behavior recognition method.

As a background technique in the present technical field, Patent Document 1 discloses an intention estimation device that discriminates whether or not a person's movement is intended without relying on a biological signal such as a surface myoelectric potential. This intention estimation device acquires motion information using a method of measuring the position and angle at which a person is moving, limits the motion of the person to a range that can be realized by the person, and determines the joint angle of the person during the motion. A person's movement is intended by extracting the position information of the tip position of the moving part, using a multivariate analysis method, and using a threshold to identify whether or not the person's movement is intended. By identifying whether or not the movement is intended, it is possible to identify whether or not the operation is intended without relying on biological signals such as surface myoelectric potential.

Japanese Unexamined Patent Publication No. 2012-101284

In the technique described in Patent Document 1 for estimating an intention in a person's movement, it is possible to classify the intentions of a plurality of types of complicated movements in order to determine binarization as to whether or not the intention is an action. This is not possible, and there is a possibility that the accuracy of motion intention estimation will be significantly reduced.

An object of the present invention is to recognize a plurality of types of actions to be recognized with high accuracy.

The behavior recognition device according to one aspect of the invention disclosed in the present application is a behavior recognition device having a processor for executing a program and a storage device for storing the program, and a statistical component is obtained by multivariate analysis. The behavior classification model group learned for each component group can be accessed by using the component group obtained from the shape of the learning target by the generated component analysis and the behavior of the learning target, and the processor is the analysis target. A detection process for detecting the shape of the recognition target from the data, and one or more components and the contribution ratio of each of the components based on the shape of the recognition target detected by the detection process by the component analysis. A determination process for determining a sequence indicating each of the above 1 or more dimensions based on a component analysis process to be generated and a cumulative contribution rate obtained from each of the contribution rates, and a decision process indicating a dimension determined by the determination process. A specific behavior classification model learned in the same component group as a specific component group containing one or more components of is selected from the behavior classification model group and a specific behavior classification model selected by the selection process. By inputting the specific component group, an action recognition process for outputting a recognition result indicating the action of the recognition target is executed.

The behavior recognition device according to another aspect of the invention disclosed in the present application is a behavior recognition device having a processor for executing a program and a storage device for storing the program, and is a statistical component in multivariate analysis. It is possible to access the behavior classification model group learned for each component group by using the component group in ascending order from the first variable obtained from the shape of the learning target by the dimension reduction to generate and the behavior of the learning target. The processor has one or more components and the component based on the shape of the recognition target detected by the detection process by the detection process of detecting the shape of the recognition target from the analysis target data and the dimension reduction. And the dimension reduction process to generate each contribution rate, and the decision process to determine the order indicating the dimension of the ascending component from the first variable among the one or more components based on each contribution rate. , A selection process for selecting from the behavior classification model group a specific behavior classification model learned in the same component group as the specific component group from the first variable to the component of the order indicating the dimension determined by the determination process. And, by inputting the specific component group into the specific action classification model selected by the selection process, the action recognition process that outputs the recognition result indicating the action of the recognition target is executed. do.

The learning device according to one aspect of the invention disclosed in the present application is a learning device having a processor for executing a program and a storage device for storing the program, and the processor determines the shape and behavior of a learning target. Component analysis that generates one or more components based on the shape of the learning target acquired by the acquisition process by the acquisition process that acquires the teacher data including and the component analysis that generates statistical components by multivariate analysis. The learning, a control process for controlling the order indicating each dimension of 1 or more based on the processing, an allowable calculation amount, a component group containing 1 or more components of the order indicating the dimension controlled by the control process, and the learning. It is characterized by executing a behavior learning process of learning the behavior of the learning target and generating a behavior classification model for classifying the behavior of the learning target based on the behavior of the target.

A learning device according to another aspect of the invention disclosed in the present application is a learning device having a processor for executing a program and a storage device for storing the program, wherein the processor has a shape and behavior of a learning object. A dimension that generates one or more components based on the shape of the learning target acquired by the acquisition process by the acquisition process that acquires teacher data including the above and the dimension reduction that generates statistical components by multivariate analysis. It was controlled by the reduction process, the control process that controls the order indicating the dimension of the ascending component from the first variable among the one or more components based on the allowable calculation amount, and the control process from the first variable. A behavior learning process that learns the behavior of the learning target based on the component group up to the component of the order indicating the dimension and the behavior of the learning target, and generates a behavior classification model that classifies the behavior of the learning target. And, it is characterized by executing.

According to a typical embodiment of the present invention, it is possible to recognize a plurality of types of actions to be recognized with high accuracy. Issues, configurations and effects other than those described above will be clarified by the description of the following examples.

FIG. 1 is an explanatory diagram showing a system configuration example of the behavior recognition system according to the first embodiment. FIG. 2 is a block diagram showing an example of a computer hardware configuration. FIG. 3 is an explanatory diagram showing an example of learning data. FIG. 4 is a block diagram showing a functional configuration example of the behavior recognition system according to the first embodiment. FIG. 5 is a block diagram showing a detailed functional configuration example of the skeleton information processing unit. FIG. 6 is an explanatory diagram showing a detailed calculation method of the joint angle executed by the joint angle calculation unit. FIG. 7 is an explanatory diagram showing an example of a detailed calculation method of the movement amount between frames executed by the movement amount calculation unit. FIG. 8 is an explanatory diagram showing a detailed method of normalizing the skeleton information executed by the normalization unit. FIG. 9 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB. FIG. 10 is an explanatory diagram showing an example in which the principal component generated by the principal component analysis unit using the teacher signal as input data is plotted on the principal component space. FIG. 11 is an explanatory diagram showing a detailed method for the behavior learning unit to learn the behavior and the behavior recognition unit to classify the behavior. FIG. 12 is a graph showing the transition of the cumulative contribution rate used by the dimension number determination unit when determining the dimension number. FIG. 13 is a flowchart showing a detailed processing procedure example of the learning process by the server (learning device) according to the first embodiment. FIG. 14 is a flowchart showing a detailed processing procedure example of the skeleton information processing according to the first embodiment. FIG. 15 is a flowchart showing an example of an action recognition processing procedure by a client (behavior recognition device) according to the first embodiment. FIG. 16 is a block diagram showing a functional configuration example of the behavior recognition system according to the second embodiment. FIG. 17 is a flowchart showing a detailed processing procedure example of the learning process by the server (learning device) according to the second embodiment. FIG. 18 is a flowchart showing an example of an action recognition processing procedure by a client (behavior recognition device) according to the second embodiment. FIG. 19 is a block diagram showing a functional configuration example of the skeleton information processing unit according to the fourth embodiment. FIG. 20 is a flowchart showing a detailed processing procedure example of the skeleton information processing unit according to the fourth embodiment. FIG. 21 is a block diagram showing a functional configuration example of the behavior recognition system according to the fifth embodiment. FIG. 22 is a block diagram showing a functional configuration example of the behavior recognition system according to the sixth embodiment. FIG. 23 is an explanatory diagram showing a decision tree, which is a basic method for the behavior learning unit and the behavior recognition unit to classify behaviors. FIG. 24 is an explanatory diagram showing a detailed development method of classification by a decision tree. FIG. 25 is an explanatory diagram showing ensemble learning and a method used by the behavior learning unit and the behavior recognition unit to classify behaviors. FIG. 26 is a block diagram showing a functional configuration example of the behavior recognition system according to the seventh embodiment. FIG. 27 is a flowchart showing a detailed processing procedure example of the learning process by the server (learning device) according to the seventh embodiment.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In addition, in all the drawings for explaining the embodiment, the same members are designated by the same reference numerals in principle, and the repeated description thereof will be omitted. Further, in the following embodiments, the constituent elements (including element steps and the like) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. Needless to say. In addition, when saying "consisting of A", "consisting of A", "having A", and "including A", other elements are excluded unless it is clearly stated that it is only that element. It goes without saying that it is not something to do. Similarly, in the following embodiments, when the shape, positional relationship, etc. of the components or the like are referred to, the shape is substantially the same, except when it is clearly stated or when it is considered that it is not clearly the case in principle. Etc., etc. shall be included.

Notations such as "first", "second", and "third" in the present specification and the like are attached to identify components, and do not necessarily limit the number, order, or contents thereof. is not it. Further, the numbers for identifying the components are used for each context, and the numbers used in one context do not always indicate the same composition in the other contexts. Further, it does not prevent the component identified by a certain number from functioning as the component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range and the like disclosed in the drawings and the like.

<Behavior recognition system>
FIG. 1 is an explanatory diagram showing a system configuration example of the behavior recognition system according to the first embodiment. The action recognition system 100 includes a server 101 and one or more clients 102. The server and the client are communicably connected via a network 105 such as the Internet, a LAN (Local Area Network), and a WAN (Wide Area Network). The server 101 is a computer that manages the client 102. The client 102 is a computer that is connected to the sensor 103 and acquires data from the sensor 103.

The sensor 103 detects the data to be analyzed from the analysis environment. The sensor 103 is, for example, a camera that captures a still image or a moving image. Further, the sensor 103 may detect voice or odor. The teacher signal DB 104 is a database that holds a combination of learning data (human skeleton information and joint angle) and behavior information (for example, a person's posture or movement such as "standing" or "falling") as a teacher signal. The teacher signal DB 104 may be stored in the server 101, or may be connected to a computer capable of communicating with the server 101 or the client 102 via the network 105.

The behavior recognition system 100 has a learning function using the teacher signal DB 104 and a behavior recognition function using the behavior classification model obtained by the learning function. The behavior classification model is a learning model for classifying the behavior of a recognition target such as a person or an animal. The learning function and the behavior recognition function may be implemented in either the server 101 or the client 102 as long as they are implemented in the behavior recognition system 100. For example, the server 101 may implement the learning function, and the client 102 may implement the action recognition function. Further, the server 101 may implement the learning function and the action recognition function, and the client 102 may send the data from the sensor 103 to the server 101 or receive the action recognition result by the action recognition function from the server 101. ..

Further, the client 102 may implement the learning function and the behavior recognition function, and the server 101 may manage the behavior classification model and the behavior recognition result from the client 102. A computer that implements the learning function is referred to as a learning device, and a computer that implements at least the behavior recognition function among the learning function and the behavior recognition function is referred to as a behavior recognition device. Further, in FIG. 1, the client-server type action recognition system 100 is taken as an example, but a stand-alone type action recognition device may also be used. In the first embodiment, for convenience of explanation, the behavior recognition system 100 in which the server 101 implements the learning function (learning device) and the client 102 implements the behavior recognition function (behavior recognition device) will be described as an example.

<Computer hardware configuration example>
FIG. 2 is a block diagram showing a hardware configuration example of a computer (server 101, client 102). The computer 200 has a processor 201, a storage device 202, an input device 203, an output device 204, and a communication interface (communication IF) 205. The processor 201, the storage device 202, the input device 203, the output device 204, and the communication IF 205 are connected by the bus 206. The processor 201 controls the computer 200. The storage device 202 serves as a work area for the processor 201. Further, the storage device 202 is a non-temporary or temporary recording medium for storing various programs and data. Examples of the storage device 202 include a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a flash memory. The input device 203 inputs data. The input device 203 includes, for example, a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device 204 outputs data. The output device 204 includes, for example, a display, a printer, and a speaker. The communication IF 205 connects to the network 105 and transmits / receives data.

<Learning data>
FIG. 3 is an explanatory diagram showing an example of learning data. The learning data 380 is composed of skeletal information 320 and joint angle 370 for each subject. The skeleton information 320 is detected based on the analysis target data acquired from the sensor 103. The joint angle 370 is calculated based on the skeletal information 320. The learning data 380 for one subject is composed of, for example, a combination of skeletal information 320 and joint angle 370 obtained from each of a plurality of time-series frames in which the subject is a subject.

The skeleton information 320 includes the name 321, the x-coordinate value 322 on the X-axis, and the y-coordinate value 323 on the y-axis orthogonal to the X-axis for each of the plurality of (18 in this example) skeleton points 300 to 317. Has. The joint angle 370 also has the name 371 for each of the plurality (18 in this example) skeletal points 300-317. In the name 371, ∠a-bc (a, b, c are the names of the skeleton points 321) is the joint angle 370 of the skeleton point b formed by the line segment ab and the line segment bc. The skeleton information 320 may include, for example, a knuckle. Further, the joint angle 370 may also include a joint angle 370 other than these.

In FIG. 3, the coordinate values of the skeleton points 300 to 317 are used as two-dimensional position information (combination of x-coordinate value and y-coordinate value), but may be used as three-dimensional position information. Specifically, for example, a z-coordinate value on the z-axis (for example, in the depth direction) orthogonal to the X-axis and the y-axis may be added.

<Example of functional configuration of behavior recognition system 100>
FIG. 4 is a block diagram showing a functional configuration example of the behavior recognition system 100 according to the first embodiment. The server 101 includes a teacher signal acquisition unit 401, a missing information control unit 402, a skeletal information processing unit 403, a principal component analysis unit 404, a dimension number control unit 405, and a behavior learning unit 406. The client 102 includes a skeleton detection unit 451, a defect information determination unit 452, a skeleton information processing unit 453, a principal component analysis unit 454, a dimension number determination unit 455, an action classification model selection unit 456, and an action recognition unit 457. And have.

Specifically, these are realized, for example, by causing the processor 201 to execute a program stored in the storage device 202 shown in FIG. First, a functional configuration example on the server 101 side will be described.

The teacher signal acquisition unit 401 acquires a single teacher signal or a plurality of teacher signals used for learning about the teacher signal acquired from the teacher signal DB 104, and outputs the selected teacher signal to the missing information control unit 402.

The missing information control unit 402 deletes an arbitrary skeleton point with respect to the skeleton information 320 among the teacher signals acquired from the teacher signal acquisition unit 401. The number of skeletal points to be deleted may be singular, plural, or zero. The defect information control unit 402 updates the skeleton information 320 after the defect (including the case where the number of skeleton points to be deleted is 0) as the skeleton information 320 in the teacher signal. Further, in order to enhance the noise immunity, the skeleton information 320 may be updated by adding noise that shifts the skeleton point position with respect to the skeleton information 320 when the information is lost.

Then, the defect information control unit 402 outputs a teacher signal including the name 321 of the deleted skeleton point and the defect information which is the position information (x coordinate value 322, y coordinate value 323) to the skeleton information processing unit 120. Further, the defect information control unit 402 outputs the defect information to the behavior learning unit 406 via the skeleton information processing unit 403, the principal component analysis unit 404, and the dimension number control unit 405.

The skeleton information processing unit 403 processes the updated skeleton information 320. Specifically, for example, the skeleton information processing unit 403 calculates the joint angle 370 and the amount of movement between frames from the skeleton information 320 in the acquired updated teacher signal. Further, the skeleton information processing unit 403 excludes the absolute position information from the skeleton information 320, and executes normalization in which the size of the skeleton information 320 is constant. Then, the skeleton information processing unit 403 outputs the joint angle 370, the amount of movement between frames, and the normalized skeleton information 320 to the principal component analysis unit 404.

FIG. 5 is a block diagram showing a detailed functional configuration example of the skeleton information processing units 403 and 453. The skeleton information processing units 403 and 453 have a joint angle calculation unit 501, a movement amount calculation unit 502, and a normalization unit 503.

The joint angle calculation unit 501 calculates the joint angle 370 from the skeleton information 320 among the acquired teacher signals, and outputs the joint angle 370 to the principal component analysis unit 404 via the movement amount calculation unit 502 and the normalization unit 503.

The movement amount calculation unit 502 calculates the movement amount between frames from the skeleton information 320 in the acquired teacher signal, and outputs the movement amount to the principal component analysis unit 404 via the normalization unit 503.

The normalization unit 503 excludes the absolute position information from the acquired teacher signal with respect to the skeleton information 320, performs normalization to make the size of the skeleton information 320 constant, and causes the principal component analysis unit 404 to perform normalization. Output.

Returning to FIG. 4, the principal component analysis unit 404 uses the normalized skeleton information 320, the joint angle 370, and the movement amount between frames among the teacher signals acquired from the skeleton information processing unit 403 as input data. Principal component analysis is executed to generate one or more principal components and output to the dimension number control unit 405. Of the skeleton information 320, the joint angle 370, and the amount of movement between frames, at least the skeleton information 320 may be input data.

In the principal component analysis, as shown in the following equation (1), the principal component y _i is generated by multiplying the input data x _i by the coefficient _wij and adding them. The general formula for principal component analysis is shown in the following formula (2). As shown in the following equation (3), the coefficient _wij is determined so that the variance V (y _i ) becomes the maximum when the variance of y _i is defined as V (y _i ).

However, if the coefficient wij is not constrained, the absolute amount of the variance V (y _i ) can be infinitely large, and the coefficient _wij cannot be uniquely determined. Therefore, the constraint of the following equation (4) It is desirable to add. Further, in order to eliminate duplication of information, it is desirable to impose the constraint of the following equation (5) that the covariance between the newly generated principal component y _k and the previously generated principal component y _k is 0.

However, the above equation (4) and the above equation (5) attached as constraints are not limited to this, and there is no problem in calculating the coefficient _wij by adding another constraint condition or removing the constraint. When the variance V (yj) of the new principal component y _j thus generated is separately defined as λ _j as shown in the following equation (6), the variance V (x _j ) of the input data x _j is shown in the following equation (7). ) And the sum of λ _j are equal.

Here, p is the number of input data x _j . The higher the variance V (y _j ) of the newly generated principal component y _j , the more the original information is reflected. That is. The ratio of the variance of the newly generated variable y _j to the variance of the original data is called the contribution ratio and is expressed by the following equation (8). Further, the result of adding the contribution rate from the contribution rate of the first principal component to the descending order of the dispersion values (the ascending order of the ordinal number m of the principal component) is called the cumulative contribution rate and is represented by the following equation (9).

The contribution rate and the cumulative contribution rate are measures such as how much the newly generated main component y _j and the generated plurality of main components represent the amount of information in the original data, and are generated together with the main components. Although principal component analysis was applied as an example of component analysis that produces statistical components in multivariate analysis, independent component analysis, which is also an example of component analysis, may be performed instead of principal component analysis. ..

In the case of independent component analysis, the principal component is an independent component. The contribution rate may be used as an index showing how much this independent component affects the input data xi. In the independent component analysis, the sum of squares of the mixing coefficient matrix in the independent component analysis for each independent component is the intensity of each independent component.

The intensity of the independent component indicates the variance of the independent component in the input data x _i . That is, since the variances of all the independent components obtained by the independent component analysis are unified to 1, the variance of the input data x _i can be obtained by taking the sum of squares of the mixing coefficients. Then, the value obtained by dividing the intensity of the independent component by the sum of the intensities of all the independent components may be used as the contribution rate of the independent variable.

The dimension number control unit 405 controls an ordinal number k indicating each dimension of one or more components. Specifically, for example, the dimension number control unit 405 determines, among the acquired principal components, the number of dimensions of the principal components used for learning by the behavior learning unit 406 in descending order of the variance value, and the first principal component is used. The principal components from the components to the k-th principal component having the determined dimension k (k is an integer of 1 or more) as an order are output to the behavior learning unit 406 in descending order of the variance value.

The behavior learning unit 406 learns by associating the main component acquired from the dimension number control unit 405 with the behavior information in the teacher signal acquired from the teacher signal DB 104. Specifically, for example, the behavior learning unit 406 uses the principal component group from the first principal component to the kth principal component acquired from the dimension number control unit 405 as input data, and is in the teacher signal acquired from the teacher signal DB 104. Using behavior information as output data, a behavior classification model is generated by machine learning. The behavior learning unit 406 associates the behavior classification model generated as a result of learning with the defect information acquired from the defect information control unit 402, and outputs it to the behavior classification model selection unit 456.

Next, an example of a functional configuration on the client 102 side will be described. The skeleton detection unit 451 detects the human skeleton information 320 reflected in the analysis target data acquired from the sensor 103, and outputs the information to the defect information determination unit 452. For the detection of the skeleton information 320, an NN (neural network) capable of estimating the human skeleton information 320 generated by machine learning may be used, or a marker is added to the skeleton point of the person to be detected and the marker appears in the image. The skeleton information 320 may be detected from the position, and the method for detecting the skeleton information 320 is not limited.

The defect information determination unit 452 determines whether or not there is a skeleton point that cannot be acquired due to occlusion or the like among the skeleton information 320 detected by the skeleton detection unit 451. If there is a skeleton point that could not be acquired, the position information is obtained. The skeleton information 320 detected by the skeleton detection unit 451 is output to the skeleton information processing unit 453 as missing information. Further, the defect information determination unit 452 outputs the defect information to the behavior classification model selection unit 456 via the skeleton information processing unit 453, the principal component analysis unit 454, and the dimension number determination unit 455.

The skeletal information processing unit 453 has the same function as the skeletal information processing unit 403. The skeleton information processing unit 453 performs the same processing as the skeleton information processing unit 403 on the skeleton information 320 detected by the skeleton detection unit 451 to obtain the joint angle 370, the amount of movement between frames, and the normalized skeleton. Information 320 and are output to the principal component analysis unit 454.

The principal component analysis unit 454 has the same function as the principal component analysis unit 404. The principal component analysis unit 454 executes the same processing as the principal component analysis unit 404 on the output data from the skeleton information processing unit 453 to generate a single or a plurality of principal components. Further, the principal component analysis unit 454 outputs the contribution rate and the cumulative contribution rate generated together with the principal component to the dimension number determination unit 455.

The dimension number determination unit 455 determines the ordinal number k indicating each dimension of one or more components based on the cumulative contribution rate obtained from each contribution rate. Specifically, for example, the dimension number determination unit 455 outputs the principal components of the acquired principal components from the acquired contribution rate and the cumulative contribution rate to the action classification model selection unit 456 in descending order of variance. The number of dimensions k indicating whether or not to do is determined. The dimension number k is an ordinal number k indicating the dimension of the principal component. For example, in the case of the first principal component, the number of dimensions (ordinal number) k = 1, and in the case of the second principal component, the number of dimensions (ordinal number) k = 2. The dimension number determination unit 455 outputs the principal component groups from the first principal component to the kth principal component to the behavior classification model selection unit 456 in descending order of variance.

The behavior classification model selection unit 456 is associated with the same defect information as the defect information acquired from the defect information determination unit 452 among the behavior classification models associated with the defect information generated by the defect information control unit 402, and has the same number of dimensions. A behavior classification model in which behavior learning is performed in the principal component group (first principal component to kth principal component) up to the kth dimension determined by the determination unit 455 is selected. The behavior classification model selection unit 456 outputs the behavior classification model selected together with the principal component group from the first principal component to the kth principal component to the behavior recognition unit 457.

Especially in a two-dimensional image, there is a possibility that all the defined skeleton points cannot be acquired due to occlusion or the like, and the skeleton information 320 in which some of the skeleton points that could not be acquired is missing may be generated by the skeleton detection unit 451. There is. When performing behavior recognition for the skeleton information 320 lacking some of the skeleton points, the client 102 behaves using the behavior learning model associated with the missing information of the missing skeleton information 320 detected by the skeleton detection unit 451. Do recognition. As a result, highly accurate action recognition is realized even for the skeleton information 320 in which some skeleton points are missing.

Of the behavior classification models associated with the defect information generated by the defect information control unit 402 acquired from the behavior learning unit 406, the same defect information as the defect information acquired from the defect information determination unit 452 is associated with the number of dimensions. It is also assumed that a behavior classification model in which behavior learning is performed with the same principal components (first principal component to k-th principal component) as the order number k indicating the dimension of the principal component determined by the determination unit 455 has not been generated.

In this case, the behavior classification model selection unit 456 is a behavior classification model closest to this condition (for example, a behavior classification model in which the position information of the missing skeleton point and the missing information within a predetermined distance are associated, the first principal component to the first. (K-1) A behavior classification model in which behavior learning is performed with the main component) may be selected.

The behavior recognition unit 457 recognizes the behavior of a person reflected in the analysis target data acquired from the sensor 103 based on the selected behavior classification model and the principal component group from the first principal component to the kth principal component. Specifically, for example, the behavior recognition unit 457 inputs the principal component group (first principal component to kth principal component) obtained from the analysis target data into the selected behavior classification model, thereby analyzing the analysis target data. The predicted value indicating the behavior of the person reflected in is output as the recognition result.

<Example of joint angle calculation>
FIG. 6 is an explanatory diagram showing a detailed calculation method of the joint angle 370 executed by the joint angle calculation unit 501. The joint angle calculation unit 501 calculates the joint angle θ at the three connected skeletal points 600 to 602. The skeleton information 620 of the skeleton points 600 to 602 is defined as the position vectors O, A, and B with respect to the origin 630, respectively. The joint angle calculation unit 501 calculates a relative vector with the skeleton point 600 as the origin as shown in the following equations (10) and (11), and the following equation (12) is established from the calculated vector, and the following equation (13) is established. The joint angle θ is calculated by calculating the inverted cosine as shown in.

<Example of calculation of movement amount between frames>
FIG. 7 is an explanatory diagram showing an example of a detailed calculation method of the movement amount between frames executed by the movement amount calculation unit 502. The movement amount calculation unit 502 uses the skeleton information 701 of the Nth frame and the skeleton information 702 of the NM frame for the same subject in the calculation of the movement amount between frames. N and M are integers of 1 or more, and N> M. The value of M can be set arbitrarily. As shown in the following formulas (14) to (16), the movement amount calculation unit 502 calculates the distances of the same skeleton points 300 to 317 of the same person shown between the frames. The amount of movement between the frames of the 18 skeleton points 300 to 317 is the amount of movement between the frames for the person.

However, the movement amount between frames executed by the movement amount calculation unit 502 is not limited to this, and as shown in the following equation (17), the movement amount calculation unit 502 is the same person shown between each frame. The distance between the same skeleton points 300 to 317 is calculated, and the total value of the movement amounts between all 18 skeleton points 300 to 317 may be used as the movement amount between frames for the person.

Further, the movement amount calculation unit 502 may use the center of gravity skeleton information 711 and the center of gravity skeleton information 712, which are the centers of gravity, among the skeleton information 701 of the nth frame and the skeleton information 702 of the nth frame. Specifically, for example, the movement amount calculation unit 502 calculates the center of gravity for each person as shown in the following formulas (18) to (19), and with respect to the calculated center of gravity as shown in the following formula (20). , The amount of movement between frames for the person may be calculated.

<Example of normalization>
FIG. 8 is an explanatory diagram showing a detailed method of normalizing the skeleton information 320 executed by the normalization unit 503. First, the normalization unit 503 calculates the center of gravity from (a) all or part of the skeleton information 320, and (b) converts it into relative coordinates with the center of gravity as the origin. After that, the normalization unit 503 divides (d) the position information of each skeleton point of the skeleton information 320 by (c) the length L of the diagonal line of the smallest rectangle surrounding the 18 skeleton points 300 to 317. When the skeleton information 320 obtained in (d) is used as a teacher signal, the position information of the skeleton points 300 to 317 after division is also incorporated.

For example, if the normalization unit 503 is not executed and learning for skeleton detection and behavior classification is executed for an action such as "a person of 180 cm sits at point A", "does not sit at any place other than point A" and "other than 180 cm". There is a possibility that a judgment such as "the person does not sit down" may be made. In order to exclude such restrictions and make the behavior classification versatile, the normalization unit 503 normalizes the skeleton information 320 in order to remove the absolute position information in the image and the information regarding the size of the skeleton. do.

<Teacher signal held by the teacher signal DB 104>
FIG. 9 is an explanatory diagram showing a detailed example of the teacher signal held by the teacher signal DB 104. (A) In the person reflected in the image 900, which is the data to be analyzed, (b) the skeleton information 320A, the joint angle 370 (not shown), and (c) the behavior information 901 (“standing”) associated with the skeleton information 320A. The combination of and becomes a teacher signal. Similarly, in the person (a) reflected in the image 910 which is the analysis target data, (b) the skeleton information 320B, the joint angle 370 (not shown), and (c) the behavior information 911 (“falling down”) associated with the skeleton information 320B. The combination of ") and is the teacher signal.

<Dimension control by dimension control unit 405 and behavior learning by behavior learning unit 406>
FIG. 10 is an explanatory diagram showing an example in which the principal component generated by the principal component analysis unit 404 using the teacher signal as input data is plotted on the principal component space. The legend shows behavioral information 1000-1004 contained in the teacher signal.

In FIG. 10, (a) shows an example in which the first principal component is taken on the X-axis and the second principal component is taken on the Y-axis, and the information up to the second principal component is plotted on a two-dimensional plane. (B) is an example in which the X-axis has the first principal component, the Y-axis has the second principal component, the Z-axis has the third principal component, and the information up to the third principal component is plotted in a three-dimensional space. show.

In (a), it can be seen that the standing 1000, the sitting 1001, and the falling 1004 can be separated even on the two-dimensional plane up to the second main component, but the walking 1002 and the squatting 1003 are 2 up to the second main component. It can be seen that separation is difficult on the dimensional plane. Here, in (b), when the walking 1002 and the squatting 1003 are plotted on the three-dimensional space including the third principal component, the possibility of separation may be expanded.

Therefore, if a large number of principal components generated by the principal component analysis unit 404 are used, there is a possibility of highly accurate behavior classification. However, since the amount of calculation increases when the ordinal number k indicating the dimension of the principal component is increased, it is necessary to consider the amount of the principal component from the accuracy and the amount of calculation and determine how many dimensions of the space the action is represented.

Therefore, the dimension number control unit 405 changes the maximum ordinal number of the principal components used for learning in the behavior learning unit 406, and outputs the principal component group from the first principal component to the principal component of the maximum ordinal number to the behavior learning unit 406. Specifically, for example, the required accuracy of the above-mentioned behavior classification (for example, an ordinal number indicating the minimum required dimension of the main component) or / and the allowable calculation amount are set in advance, and the dimension number control unit 405 performs the action. The learning unit 406 changes the maximum ordinal number of the principal components used for learning, and determines the ordinal number that satisfies the required accuracy and / and the allowable calculation amount to the maximum.

For example, in the case of the condition that the required accuracy is the ordinal number "3" (third principal component) indicating the dimension, the dimension number control unit 405 determines the maximum ordinal number to be "3", and the first principal component to the third principal component. The principal component group of is output to the behavior learning unit 406.

Further, when the permissible calculation amount is set as a condition, the dimension number control unit 405 sequentially acquires the calculation amount in ascending order from the first principal component, and the ordinal number when the maximum ordinal number is exceeded for the first time (the permissible calculation amount). For example, it is determined to have an ordinal number one less than "5") (for example, "4"), and the principal component group from the first principal component to the fourth principal component with the maximum ordinal number k = 4 is output to the behavior learning unit 406. ..

Further, if the required accuracy is the ordinal number "3" (third principal component) or more indicating the dimension and the allowable calculation amount is set as the condition, the cumulative calculation amount up to the third principal component is the allowable calculation amount. If the following, the dimension number control unit 405 changes the maximum ordinal number from "3" to "4". Then, if the cumulative calculation amount up to the fourth main component exceeds the allowable calculation amount, the dimension number control unit 405 determines the maximum ordinal number k to be "3", and the main components from the first main component to the third main component. The group is output to the behavior learning unit 406.

On the other hand, if the cumulative calculation amount up to the third principal component exceeds the allowable calculation amount, the dimension number control unit 405 changes the maximum ordinal number from "3" to "2". Then, if the cumulative calculation amount up to the second principal component is equal to or less than the allowable calculation amount, the dimension number control unit 405 determines the maximum ordinal number k to be "2", and the main principal component to the second principal component is the main component. The component group is output to the behavior learning unit 406.

The principal component group output to the behavior learning unit 406 does not need to be limited in ascending order from the first principal component. For example, the dimension number control unit 405 may take out a specific number of predetermined main component groups. Further, the dimension number control unit 405 may determine the main component group to be output to the behavior learning unit 406 after excluding the specific main component group. As described above, the principal component group output to the behavior learning unit 406 is not limited to the principal component group in ascending order from the first principal component.

Further, also in this case, when the permissible calculation amount is set as a condition, the dimension number control unit 405 calculates the principal component group not limited to the ascending order from the first principal component described above in the ascending order of the ordinal number. The quantities are sequentially acquired, and the principal component group up to the ordinal number one before the ordinal number when the permissible computational complexity is exceeded for the first time is output to the behavior learning unit 406. For example, when the principal component group consists of the second principal component, the third principal component, and the fifth principal component, the allowable calculation amount is not exceeded for the second principal component, and the allowable calculation amount is also applied for the second principal component and the third principal component. When the allowable calculation amount is exceeded for the first time in the second principal component, the third principal component, and the fifth principal component without exceeding the allowable calculation amount, the dimension number control unit 405 is the first from the second principal component to the one immediately before the fifth principal component. Up to 3 principal components may be determined as the principal component group to be output to the behavior learning unit 406.

The behavior learning unit 406 performs behavior learning under a plurality of conditions in advance, generates a behavior classification model, and outputs it to the behavior classification model selection unit 456. By selecting a behavior classification model according to the situation from the plurality of behavior classification models generated in this way, general-purpose and highly accurate behavior recognition is realized.

FIG. 11 is an explanatory diagram showing a detailed method for the behavior learning unit 406 to learn the behavior and the behavior recognition unit 457 to classify the behavior. For each action on the main component space, the action learning unit 406 classifies each action for each area by using (a) the boundary line 1101 and (b) the boundary plane 1102. As a method for learning and classifying behaviors, a k-means method, a support vector machine, a decision tree, a random forest, or the like may be adopted, and the behavior learning method is not limited.

The behavior recognition unit 457 recognizes the behavior using the behavior classification model learned and generated by the behavior learning unit 406. Specifically, for example, the client 102 applies the principal component analysis to the newly input skeleton information 320, and the newly generated principal component is set according to the boundary line 1101 and the boundary plane 1102 set by the behavior classification model. It determines which area it belongs to, and recognizes the action according to the determined area.

FIG. 12 is a graph showing the transition of the cumulative contribution rate used by the dimension number determination unit 455 when determining the dimension number. The cumulative contribution rate is a measure of how much the newly generated principal components represent the amount of information in the original data. Therefore, even if the number of principal components is increased and the number of dimensions in the action classification is increased, if the cumulative contribution rate does not change significantly, a large improvement in accuracy cannot be expected.

Therefore, the dimension number determination unit 455 determines the number of dimensions by using as many main components as necessary to exceed the predetermined cumulative contribution rate threshold value. For example, when the predetermined threshold value of the cumulative contribution rate is "0.8", the condition is satisfied as long as it is up to the second principal component. Therefore, the dimension number k here is set to "2", and the first principal component and the first principal component are satisfied. The two principal components are output to the behavior classification model selection unit 456.

The principal component group output to the behavior classification model selection unit 456 does not need to be limited in ascending order from the first principal component. For example, the dimension number determination unit 455 may determine the combination of the ordinal number k of the principal component that does not exceed the predetermined cumulative contribution rate threshold value and maximizes the cumulative contribution rate. Further, the dimension number determination unit 455 may select such a combination of the ordinal numbers k of the principal components from the principal component group applied to the behavior classification model. As described above, the principal component group output to the behavior classification model selection unit 456 is not limited to the principal component group in ascending order from the first principal component.

<Learning process>
FIG. 13 is a flowchart showing a detailed processing procedure example of the learning process by the server 101 (learning device) according to the first embodiment. The server 101 acquires a single teacher signal or a plurality of teacher signals used for learning about the teacher signal acquired from the teacher signal DB 104 by the teacher signal acquisition unit 401 (step S1300).

The server 101 deletes information from the acquired skeleton information 320 in the teacher signal by the defect information control unit 402, updates the deleted skeleton information 320 as the skeleton information 320 in the teacher signal, and deletes the skeleton. The point name 321 and the position information (x coordinate value 322, y coordinate value 323) are used as missing information (step S1301). The teacher signal executed by the missing information control unit is called an updated teacher signal.

The server 101 executes skeletal information processing for each update teacher signal by the skeletal information processing unit 403 (step S1302). Specifically, for example, the server 101 executes processing by the joint angle calculation unit 501, the movement amount calculation unit 502, and the normalization unit 503.

FIG. 14 is a flowchart showing a detailed processing procedure example of the skeleton information processing according to the first embodiment. The server 101 calculates the joint angle 370 from the skeleton information 320 in the update teacher signal for each update teacher signal by the joint angle calculation unit 501 (step S1401). Next, the server 101 calculates the movement amount between frames from the skeleton information 320 in the update teacher signal for each update teacher signal by the movement amount calculation unit 502 (step S1401).

Then, the server 101 excludes the absolute position information with respect to the skeleton information 320 for each update teacher signal by the normalization unit, and executes normalization in which the size of the skeleton information 320 becomes constant (step S1303). ). As a result, for the update teacher signal, the joint angle 370, the amount of movement between frames, and the normalized skeletal information 320 are obtained. Then, the process proceeds to step S1303 in FIG.

Returning to FIG. 13, the server 101 performs principal component analysis by the principal component analysis unit 404 with the normalized skeleton information 320, the joint angle 370, and the amount of movement between frames as input data, and is unilateral. Alternatively, a plurality of principal components are generated (step S1303).

Next, the server 101 determines, by the dimension number control unit 405, how many dimensions of the generated principal components to be used for learning are to be used in descending order of the variance value, and the determined principal components up to the k-dimension (the first). (1st principal component to kth principal component) are selected in descending order of dispersion value (step S1304).

Then, the server 101 learns based on the selected principal component and the behavior information in the update teacher signal by the behavior learning unit, and as a result of the learning, generates a behavior classification model and associates it with the missing information (step). S1305).

In the principal component analysis (step S1303), it is possible to generate a principal component having the same number of dimensions k as the information before executing the principal component analysis. Therefore, in step S1306, if the server 101 has a dimension of the principal component that has not been determined so far with respect to the dimension number k of the principal component used for learning determined in step S1304 by the dimension number control unit 405 ( Step S1306: No), the process returns to step S1304 to determine the dimension of the principal component that has not been determined so far (step S1304).

On the other hand, if the dimensions of all the principal components used for decidable learning have been determined so far (step S1306: Yes), the process proceeds to step S1307. However, the determination of the process in step S1306 is not limited to the determination of the next process depending on the pros and cons of determining the dimensions of all the principal components used for decidable learning. For example, if the number of repetitions is set in advance and step S1304 is repeated a predetermined number of times, the process may proceed to step S1307.

In step S1307, if there is skeleton information 320 that has not been deleted yet (step S1307: No) with respect to the skeleton information 320 that was deleted in step S1301, the process returns to step S1301 and the server 101 has deleted it so far. Defects for no skeleton (step S1301).

On the other hand, when all the skeleton information 320 is deleted (step S1307: Yes), the process proceeds to step S1308. However, the determination of the process of step S1307 is not limited to this, and the server 101 may determine whether to return to step S1301 or proceed to step S1308 according to a predetermined number of repetitions. Further, the skeleton to be deleted may be predetermined, and the server 101 may determine whether to return to step S1301 or proceed to step S1308 depending on whether or not all the predetermined skeletons are deleted.

In step S1308, if there is a teacher signal that has not yet been selected for the teacher signal selected in step S1300 (step S1308: No), the server 101 selects a teacher signal that has not been selected so far (step S1300). .. On the other hand, when all the teacher signals are selected (step S1308: Yes), the server 101 ends the behavior learning process. However, the determination of the process of step S1308 is not limited to this, and the server 101 may determine whether to return to step S1300 or end the action learning process according to a predetermined number of repetitions.

<Action recognition processing>
FIG. 15 is a flowchart showing an example of an action recognition processing procedure by the client 102 (behavior recognition device) according to the first embodiment. The client 102 detects the human skeleton information 320 reflected in the analysis target data acquired from the sensor 103 by the skeleton detection unit 451 (step S1500). Next, the client 102 determines that, among the detected skeleton information 320, the position information of the skeleton point that could not be detected due to occlusion or the like is the missing information by the missing information determination unit 452 (step S1501).

Next, the client 102 executes skeleton information processing for the skeleton information 320 detected in step S1500 by the skeleton information processing unit 453 in the same manner as the processing in step S1302 (step S1502). Specifically, for example, as shown in FIG. 14, the server 101 executes processing by the joint angle calculation unit 501, the movement amount calculation unit 502, and the normalization unit 503.

Next, the client 102 performs principal component analysis by the principal component analysis unit 454 with the skeleton information 320 normalized in step S1502, the joint angle 370, and the amount of movement between frames as input data, and is singular or plural. The principal component of is generated, and the contribution rate and the cumulative contribution rate are calculated together with the principal component (step S1503).

Next, the client 102 determines from the calculated contribution rate and the cumulative contribution rate by the dimension number determination unit 455, how many principal components are to be used in descending order of dispersion among the generated principal components (step S1504). ..

Next, the client 102 is associated with the same missing information as the missing information detected in step S1501 among the behavior classification models generated by the behavior learning by the behavior classification model selection unit 456, and the main component determined in step S1504. A behavior classification model in which behavior learning is performed with a principal component having the same number of dimensions as the number of dimensions is selected (step S1505).

Next, the client 102 recognizes the behavior of the person reflected in the analysis target data acquired from the sensor 103 based on the behavior classification model selected in step S1505 and the main component by the behavior recognition unit 457 (step S1506). The client 102 may send the recognition result to the server 101, or may use the recognition result to control the device connected to the client 102.

For example, when the analysis environment in which the sensor 103 is deployed is a factory, the behavior recognition system 100 can be applied to work monitoring of workers in the factory, defect inspection of products, and the like by using the recognition result. .. When the analysis environment is a train, the behavior recognition system 100 can be applied to monitoring passengers in a train, monitoring equipment in a car, detecting a disaster such as a fire, etc. by using the recognition result.

As described above, according to the first embodiment, it is possible to recognize a plurality of types of actions to be recognized with high accuracy. In particular, even when the skeletal points 300 to 317 are partially deleted due to occlusion or the like, it is possible to recognize a plurality of types of actions according to the deleted skeletal points with high accuracy.

The second embodiment will be described with a focus on the differences from the first embodiment. The points common to the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 16 is a block diagram showing a functional configuration example of the behavior recognition system 100 according to the second embodiment. In the second embodiment, the missing information control unit 402 is deleted, and the missing information determination unit 452 is changed to the missing information interpolation unit 1652. As a result, when the skeleton cannot be partially measured due to occlusion or the like and the missing information is included in the measurement of the position where the person is moving, the missing information interpolation unit 1652 is missing from the measurable skeleton information 320. Interpolate the information.

Specifically, for example, the missing information interpolation unit 1652 uses the position information of the skeleton point that could not be acquired due to occlusion or the like as the missing information in the skeleton information 320 acquired from the skeleton detecting unit 451 and interpolates the missing information. It is output to the skeleton information processing unit 453. The missing information interpolation unit 1652 may interpolate the missing information from, for example, the connected skeleton points or the skeleton points close to the missing information in the acquired skeleton information 320.

Further, the missing information interpolation unit 1652 may substitute predetermined position information for the missing information. Further, the missing information interpolation unit 1652 may interpolate using the missing information of the skeleton information 320 determined to include the missing information for the skeleton information 320 of another frame acquired so far. As described above, the method of interpolating the missing information is not limited.

<Learning process>
FIG. 17 is a flowchart showing a detailed processing procedure example of the learning process by the server 101 (learning device) according to the second embodiment. In the second embodiment, the missing information control (step S1301) is not executed, and the skeletal information processing (step S1302) is executed for the teacher signal selected in step S1300. That is, in Example 2, the behavior learning unit 406 generates one behavior classification model without distinguishing the skeleton information 320 regardless of the presence or absence of the skeleton point defect.

<Action recognition processing>
FIG. 18 is a flowchart showing an example of an action recognition processing procedure by the client 102 (behavior recognition device) according to the second embodiment. In the second embodiment, the missing information determination (step S1501) is changed to the missing information interpolation (step S1801). The client 102 interpolates the position information of the skeleton point that could not be acquired due to occlusion or the like among the skeleton information 320 detected in the skeleton detection (step S1500) with the missing information, and updates the skeleton information 320 after interpolation (step S1801). ). In the skeleton information processing (step S1502), a teacher signal including the skeleton information 320 after interpolation is used.

As described above, according to the second embodiment, it is not necessary to generate a behavior classification model for each defect information by interpolating the skeleton information 320 having a defect due to occlusion or the like. As a result, it is possible to reduce the processing load of the learning function and speed up the action recognition function.

Example 3 is an example in which Example 1 and Example 2 are combined. Specifically, for example, in the action recognition system 100 of the third embodiment, the first mode for executing the learning process and the action recognition process according to the first embodiment by the user operation, and the learning process and the action recognition according to the second embodiment. It is possible to switch to the second mode for executing the process.

As described above, according to the third embodiment, a highly accurate action recognition result can be obtained by selecting the first mode if the defect information is to be considered, and the second mode is selected if the defect is to be interpolated. By doing so, the behavior recognition result can be obtained efficiently.

Example 4 will be described focusing on the differences from Examples 1 to 3. The points common to Examples 1 to 3 are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 19 is a block diagram showing a functional configuration example of the skeleton information processing unit according to the fourth embodiment. In Example 4, the skeletal information processing units 403 and 453 have a mutual information normalization unit 1904. The mutual information normalization unit 1904 normalizes the range of the skeleton information 320, the joint angle 370, and the amount of movement between frames output to the principal component analysis unit 404 within a certain range.

The range of the skeleton information 320 and the amount of movement between frames depends on the resolution of the data to be analyzed. On the other hand, the range of the joint angle 370 is in the range of 0 to 2π or 0 degrees to 360 degrees. If there is a large difference in the range of the data to be executed for the principal component analysis, the influence of the original data on the principal component may be biased for each data type.

In order to eliminate this bias, the mutual information normalization unit 1904 executes normalization that keeps the range of the data applied to the main component within a certain range. For example, the mutual information normalization unit 1904 unifies the range of the original data from 0 to 2π according to the following equations (21) to (22) for the skeleton information 320 and the following equation (23) for the inter-frame movement amount.

However, the normalization method executed by the mutual information normalization unit 1904 is not limited to this, and the mutual information normalization unit 1904 has, for example, a joint angle 370 according to the magnitude of the resolution of the data to be executed for the principal component analysis. The range of may be normalized to be constant.

FIG. 20 is a flowchart showing a detailed processing procedure example of the skeleton information processing unit according to the fourth embodiment. In the fourth embodiment, in the skeletal information processing (steps S1302 and S1502), the client 102 executes mutual information normalization (step S2004) after the normalization (step S1403). In the mutual information normalization (step S2004), the range that can be taken is constantly normalized with respect to the skeleton information 320 normalized by the normalization unit, the joint angle 370, and the amount of movement between frames.

As described above, according to the fourth embodiment, a wide range can be obtained by uniformly unifying the possible range of the original data (skeleton information 320, joint angle 370, movement amount between frames) for executing the principal component analysis. It is possible to eliminate the bias of the influence of the specific data on the principal component and to discriminate multiple types of actions with high accuracy.

Example 5 will be described focusing on the differences from Examples 1 to 4. The points common to Examples 1 to 4 are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 21 is a block diagram showing a functional configuration example of the behavior recognition system 100 according to the fifth embodiment. In the fifth embodiment, the principal component analysis unit 404 and the principal component analysis unit 445 are changed to the dimension reduction unit 2100 and the dimension reduction unit 2101. Dimensionality reduction is a process of reducing the number of original variables or the number of original dimensions while maintaining the original amount of information as much as possible, such as principal component analysis and independent component analysis of Examples 1 to 4. It is a concept that includes component analysis.

The dimension reduction unit 2100 executes dimension reduction using the normalized skeletal information 320, the joint angle 370, and the movement amount between frames among the teacher signals acquired from the skeletal information processing unit 403 as input data. Generates one or more variables and outputs them to the dimension control unit 405.

As the dimension reduction method performed by the dimension reduction unit 2100, SNE (Stochastic Neighbor Embedding), t-SNE (t-Distributed Stochastic Neibbor Embedding), UMAP (Uniform Management, Lap. There are techniques such as Laplacian Eignmap, LargeVis, and diffusion maps. The dimension reduction unit 2100 may reduce the dimension by combining t-SNE and UMAP with principal component analysis and independent component analysis. Hereinafter, the method of dimension reduction and the method of dimension reduction performed by combining each method will be described.

The processing of SNE will be described using the following equations (24) to (28).

Let the conditional probability p _{j | i} select x _j as the neighborhood when x _i is given, the similarity between the two x coordinate values 322 (input data) of x _i and x _j . The conditional probability p _{j | i} is shown in the above equation (24). At this time, it is assumed that x _j is selected based on a normal distribution centered on x _i . Next, the similarity between the two y-coordinate values 323 (principal component _{) after the dimension reduction is also the same as the similarity between x i and x j before the} dimension reduction, according to the above equation (25). Let the conditional probability q _{j | i} shown. However, the variance of the coordinate values after the dimension reduction is fixed at 1 / √2 to simplify the equation.

If y of the dimension reduction is generated so as to maintain the distance relationship before and after the dimension reduction, it is possible to reduce the dimension while maintaining the amount of information as much as possible. In order to reduce the dimension while suppressing the reduction of the amount of information, the dimension reduction unit 2100 performs processing so that p _{j | i} = q _{j | i} . KL divergence, which is a measure of how similar the two probability distributions are, is used for dimensionality reduction.

The above equation (26) shows an equation to which the probability distribution before and after the dimension reduction is applied with the KL divergence as the loss function. The dimension reduction unit 2100 minimizes the above equation (26), which is a loss function, by a stochastic gradient descent method. This gradient fluctuates y _i using the above equation (27) in which the loss function is differentiated by y _i . The update formula at the time of this fluctuation is shown by the above formula (28).

As described above, the above equation (28) is updated while changing y _i , and the dimension is reduced by obtaining y _i that minimizes the above equation (27), and a new variable is obtained. However, in the case of SNE, unlike principal component analysis, the number of dimensions (variables) after reduction is 2 or 3 types due to the characteristics of processing. Therefore, when the dimension reduction by SNE is carried out, a predetermined number of dimensions (variables) is output to the dimension number control unit 405, and the dimension number control unit 405 uses the number of variables according to the predetermined number of dimensions. Should be decided.

However, in SNE, it is difficult to minimize the loss function, and there is a problem that the skeleton points specified by the x-coordinate value 322 and the y-coordinate value 323 become dense in order to maintain equidistantness at the time of dimension reduction. There is t-SNE as a solution method of this problem.

The processing of t-SNE will be described using the following equations (29) to (33).

Symmetrize the loss function to simplify the loss function minimization. In the loss function symmetrization process, as shown in the above equation (29), the distance between x _i and x _j is represented by the joint probability distribution p _ij . p _{j | i} is the same as the above formula (24) and can be represented by the above formula (30). Further, the distance between y _i and y _j after the dimension reduction is expressed by the joint probability distribution q _i j shown in the above equation (31).

The distance of points after dimensionality reduction assumes a Student's t distribution. Student's t distribution is characterized by a higher probability of existence of values deviating from the mean value compared to the normal distribution, and this feature also allows distribution of long distances with respect to the distance between data after dimension reduction. Is possible.

In t-SNE, the dimension reduction unit 2100 reduces the dimension by minimizing the loss function shown in the above equation (32) by using the pij and q _ij obtained by the above equations (29) to (31). .. The dimension reduction unit 2100 uses the stochastic gradient descent method shown in the above equation (33) to minimize the loss function, as in SNE.

As described above, by obtaining y _i that minimizes the above equation (33), the dimension reduction unit 2100 performs dimension reduction and obtains a new variable. Similar to SNE, t-SNE has 2 or 3 types of dimensions (variables) after contraction due to the characteristics of processing. Therefore, when the dimension reduction by t-SNE is carried out, the predetermined dimension number (variable) is output to the dimension number control unit 405, and the dimension number control unit 405 uses the variable according to the predetermined dimension number. You just have to decide the number of.

t-SNE can accurately reduce dimensions because it captures the global structure as much as possible while maintaining the high-dimensional local structure before dimension reduction, but it depends on the number of dimensions before dimension reduction. There is a problem that the calculation time increases. There is UMAP as a method for solving the problem of calculation time of this dimension reduction. The processing of UMAP will be described using the following equations (34) to (36).

Among the total A of possible values, there is a high-dimensional set X (the above equation (34)). Let μ be a membership function that outputs the degree to which arbitrary data is included in the set X in the range of 0 to 1 when arbitrary data is taken out from A. For the input X shown in the above formula (1), the Y shown in the above formula (2) is prepared. Y is a set of m (<p) points existing in a space having a lower dimension than X, and is a set of data after dimension reduction. Then, with the membership function of Y as ν, the dimension reduction unit 2100 performs dimension reduction by determining Y such that the above equation (36) is minimized, and obtains a new variable.

When implementing dimension reduction by UMAP, the dimension reduction unit 2100 may output a predetermined number of dimensions (variables) to the dimension number control unit 405 as in the case of SNE and t-SNE, or the dimension reduction unit may be used. A dimension number (variable) such that the later membership function ν is equal to or greater than a predetermined value range may be output to the dimension control unit 405 as a required dimension number. At this time, the dimension number control unit 405 may determine the number of dimensions (number of variables) to be used according to the number of dimensions (variables) output by the dimension reduction unit 2100.

The processing of Isomap will be described. The dimension reduction unit 2100 calculates the shortest distance of data in the vicinity of arbitrary data, and reduces the dimension by expressing the calculated distance as a geodetic distance matrix by multidimensional scaling (MDS). Get the variable. When implementing dimension reduction by Isomap, the dimension reduction unit 2100 outputs a predetermined number of dimensions (variables) to the dimension number control unit 405, and determines the number of variables to be used according to the predetermined number of dimensions. do it.

LLE will be described using the following equations (35) to (41).

The points in the vicinity of _xi are approximately represented by the above equation (35) by a linear combination. Here, by minimizing the above equation (37) under the constraint of the above equation (36), the approximate value of _xi before the dimension reduction is determined. Next, with respect to y _i after dimension reduction, the dimension reduction unit 2100 minimizes the above equation (38) in order to maintain the linear adjacency relationship of x _i as much as possible even after dimension reduction. This solution is obtained according to the above equation (40) by extracting the eigenvector of the above equation (39) from the second smallest eigenvalue _vi to the (d + 1) th v _d , and the dimension reduction unit 2100 is described above. As shown in equation (41), y _i after dimension reduction is acquired.

When implementing dimension reduction by LLE, the dimension reduction unit 2100 outputs a predetermined number of dimensions (variables) to the dimension number control unit 405, and the dimension number control unit 405 uses the dimension reduction unit 405 according to the predetermined number of dimensions. You just have to decide the number of variables to do.

The processing of the Laplacian eigenmap will be described using the following equations (42) to (47).

Each side x _i x _j of the neighborhood graph generated by the data before the dimension reduction is assigned to the above equation (42) or the above equation (43). Introduce the graph Laplacian of the above equation (44) to the assigned weight, and extract the eigenvector of the graph Laplacian (the above equation (45)) from the second smallest _vi of the eigenvalue to the (d + 1) th v _d . Is obtained according to the above equation (46), and the dimension reduction unit 2100 acquires the value y _i after the dimension reduction according to the above equation (47).

When the dimension reduction by the Laplacian eigenmap is carried out, the dimension reduction unit 2100 outputs a predetermined number of dimensions (variables) to the dimension number control unit 405, and the dimension number control unit 405 follows the predetermined number of dimensions. , You just have to decide the number of variables to use.

The processing of LargeVis will be described. LargeVis is a method that improves the calculation time of t-SNE. In t-SNE, since the distance between data points is obtained, the calculation time increases according to the number of data. In LargeVis, the dimension reduction unit 2100 divides data from nearby data into regions using a K-NN graph, and performs dimension reduction for each region-divided data model in the same manner as t-SNE.

When implementing dimension reduction by LargeVis, the dimension number control unit 405 outputs a predetermined number of dimensions (variables) to the dimension number control unit 405, and the number of variables to be used is calculated according to the predetermined number of dimensions. You just have to decide.

The diffusion map will be described using the following equations (48) to (53).

A weight W _ij is assigned to each side x _i x _j of a neighborhood graph composed of x _i before dimension reduction and x _j in the neighborhood, and this is normalized to the transition probability of N × N shown in the above equation (48). Make a matrix P. It is assumed that pt (x _i x _j ) represents the probability of starting x _i and reaching x _j after _t steps by a random walk on the graph represented by P. Due to the nature of the transition matrix, pt (x _i x _j ) converges to the steady distribution φ ₀ (x _j ) at _t → ∞. At this time, the diffusion distance of the point x _i x _j is defined by the above equation (49). The eigenvalue of the transition probability matrix P is the above equation (50), and the eigenvector is the above equation (51). At this time, the above equation (52) holds. Since the absolute value of λ _i is 1 or less, the dimension reduction unit 2100 takes an eigenvector up to an appropriate dimension d (t) smaller than N, performs dimension reduction according to the above equation (53), and performs a new variable. To get.

When the dimension reduction by the diffusion map is carried out, a predetermined number of dimensions (variables) is output to the dimension number control unit 405, and the dimension number control unit 405 determines the number of variables to be used according to the predetermined number of dimensions. You just have to decide.

The dimension reduction unit 2100 may be carried out by combining the principal component analysis, the independent component analysis, t-SNE, UMAP, Isomap, LLE, Laplacian eigenmap, LargeVis, diffusion map and the like described above. For example, the dimension reduction unit 2100 performs dimension reduction up to 10 dimensions using principal component analysis for high-dimensional data having 36 dimensions or 36 variables, and then uses UMAP for dimension reduction up to 2 dimensions. , The combination of methods used for dimension reduction is not limited. By combining various methods when reducing dimensions in this way, complex effects can be expected in performance and calculation time.

Further, these dimension reduction methods are not limited to the scope described in the fifth embodiment, and for example, high-dimensional information may be simply calculated, subtracted, multiplied or divided, or convoluted according to a predetermined coefficient. However, the dimension reduction method is not limited as long as it is a method for generating high-dimensional data or multiple variables, lower-dimensional data, or a small number of variables as in the method described in Example 5.

The dimension reduction unit 2101 has the same function as the dimension reduction unit 2100. The dimension reduction unit 2101 executes the same processing as the dimension reduction unit 2100 on the output data from the skeleton information processing unit 453, and generates a smaller number or a plurality of new variables as compared with before the dimension reduction. .. Further, the principal component analysis unit 454 outputs the contribution rate and the cumulative contribution rate generated together with the principal component to the dimension number determination unit 455.

The dimension reduction unit 2101 has the same function as the dimension reduction unit 2100. The dimension reduction unit 2101 executes the same processing as the dimension reduction unit 2100 on the output data from the skeleton information processing unit 453 to generate a single or a plurality of new variables. Further, the dimension reduction unit 2101 outputs the information of the number of dimensions (variables) required by the dimension number determination unit 455 together with the new variable to the dimension number determination unit 455 by the same method as the dimension reduction unit 2100.

The dimension number determination unit 455 determines the dimension number k indicating how many acquired variables are output to the behavior classification model selection unit 456 based on the acquired dimension number (variable), and newly determines the number of dimensions k. The generated variable is output to the behavior classification model selection unit 456.

As described above, according to the fifth embodiment, by changing the dimension reduction method, the dimension can be reduced effectively or by shortening the calculation time according to the data acquired from the skeleton information processing unit 403, which is complicated. It is possible to discriminate various behaviors with high accuracy.

Example 6 will be described focusing on the differences from Examples 1 to 5. The points common to Examples 1 to 5 are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 22 is a block diagram showing a functional configuration example of the behavior recognition system 100 according to the sixth embodiment. In the sixth embodiment, the behavior learning unit 406 and the behavior recognition unit 457 are changed to the behavior learning unit 2200 and the behavior recognition unit 2201. A detailed method for classifying behaviors by the behavior learning unit 2200 and the behavior recognition unit 2201 will be described with reference to FIGS. 23 to 25.

FIG. 23 is an explanatory diagram showing a decision tree, which is a basic method for the behavior learning unit 2200 and the behavior recognition unit 2201 to classify behaviors. The behavior classification method using the decision tree will be explained. In the decision tree, (a) the boundary line 2310 is generated by using the variable 2303 from the variable 2300 given the action type in advance for each action in the variable space newly generated after the dimension reduction.

(A) A method for generating the boundary line 2310 will be described. The decision tree classifies the behavior step by step so that the impureness of the input variable group 2321 is minimized. In the first stage, the behavior is classified into the variable group 2322 and the variable group 2323 on the second variable axis, and in the second stage, the variable group 2322 and the variable group 2323 are divided into the variable groups 2324 to 2327 on the first variable axis. Classify. (A) The boundary line 2310 is generated by using the discriminant obtained in the process of classifying so as to minimize the impureness. It should be noted that there is no limitation on which axis the action is classified at each stage, and there is no limitation such as classifying the action classification at each axis by a specified number of times such as once.

FIG. 24 is an explanatory diagram showing a detailed development method of classification by a decision tree. The decision tree includes a levelwise 2400 that grows a decision tree for each level (depth) and a leafwise 2401 that grows a decision tree for each leaf (data group after branching). Ensemble learning is the process of learning by stacking classifiers like decision trees.

FIG. 25 is an explanatory diagram showing ensemble learning and a method used by the behavior learning unit 2200 and the behavior recognition unit 2201 to classify behaviors. Ensemble learning includes bagging 2401 that uses classification trees such as decision trees in parallel, and boosting 2402 that inherits the previous results and updates the learning results. The random forest of Example 1 is a method of adopting bagging 2401 for a decision tree, and the behavior learning unit 2200 and the behavior recognition unit 2201 of Example 6 are a classification method using boosting 2402.

When the behavior learning unit 2200 learns the behavior and the behavior recognition unit 2201 classifies the behavior, even if each decision tree is grown by levelwise and the variables input by boosting in which multiple decision trees are overlapped are classified. Alternatively, each decision tree may be grown leafwise and the variables input by boosting the multiple decision trees may be classified.

When each decision tree is grown by levelwise and boosting in which a plurality of decision trees are overlapped is adopted as an action classification method, it may be implemented by using the software library xgboost. On the other hand, when each decision tree is grown by leafwise and boosting in which a plurality of decision trees are overlapped is adopted as an action classification method, it may be implemented using the software library LightGBM. However, the implementation method is not limited to these.

As described above, according to the sixth embodiment, by using boosting as the behavior classification method and superimposing a plurality of decision trees, it is possible to discriminate complex behaviors with high accuracy.

Example 7 will be described focusing on the differences from Examples 1 to 6. The points common to Examples 1 to 6 are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 26 is a block diagram showing a functional configuration example of the behavior recognition system 100 according to the seventh embodiment. In the seventh embodiment, the dimension reduction unit 2100, the dimension number control unit 405, the action learning unit 406, and the dimension number determination unit 455 are changed to the dimension reduction unit 2600, the dimension number control unit 2601, the behavior learning unit 2602, and the dimension reduction unit 2603. Will be done.

The dimension reduction unit 2600 performs dimension reduction by the method of any one of Examples 1 to 6 according to a predetermined number of dimensions, and outputs a new variable generated after the dimension reduction to the dimension number control unit 2601. The dimension number control unit 2601 outputs the variable after dimension reduction to the behavior learning unit 2602 according to the acquired dimension number.

The behavior learning unit 2602 generates a boundary line for behavior classification by machine learning from a given behavior type together with the acquired variable after dimension reduction, and generates a behavior classification model. At this time, the behavior classification accuracy of how accurately the behavior can be predicted with respect to the generated behavior classification model is calculated.

The behavior learning unit 2602 may calculate the behavior classification accuracy using the variables used for generating the behavior classification model. The behavior learning unit 2602 may calculate the behavior classification accuracy by using some of the variables acquired from the dimension control unit 2600 for the behavior classification model generation and using the variables not used for the behavior classification generation. However, the method of calculating the behavior classification accuracy is not limited to these. If the calculated behavior classification accuracy is higher than the predetermined accuracy, the behavior learning unit 2602 outputs the generated behavior classification model to the behavior classification model selection unit 456. At this time, the behavior learning unit 2602 outputs to the dimension control unit 2601 that the acquired number of dimensions and the behavior classification accuracy have passed.

On the other hand, if the calculated behavior classification accuracy is lower than the predetermined accuracy, the behavior learning unit 2602 outputs to the dimension control unit 2601 that the behavior classification accuracy has failed. However, if the behavior classification model is generated for all the configurable dimensions (variables) and the behavior classification accuracy fails in all of them, the behavior learning unit 2602 will generate the behavior classification so far. The behavior classification model with the highest behavior classification accuracy among the models is output to the behavior classification model selection unit 456, and the dimension number (variable) used at the time of output is output to the dimension number control unit 2601 together with all learning completion information. ..

When the dimension control unit 2601 acquires pass or all learning completion information according to the pass / fail information and all learning completion information acquired from the behavior learning unit 2602, the dimension control unit 2601 outputs the acquired dimension number information to the dimension reduction unit 2603 and fails. If the result is passed, a dimension reduction instruction is output to the dimension reduction unit 2600 so that the number of dimensions used for dimension reduction is changed and the dimension reduction is performed again.

The dimension reduction unit 2600 sets a dimension number that has not been set so far according to the acquired dimension reduction instruction, performs dimension reduction again, and outputs the generated variable to the dimension number control unit 2601.

The dimension reduction unit 2603 reduces the dimensions of the data acquired from the skeleton information processing unit 452 according to the dimension number (variable) acquired from the dimension number control unit 2601 by using the dimension reduction methods of Examples 1 to 6. , The generated variable is output to the behavior classification model selection unit 456. In addition, the dimension reduction unit 2603 does not determine the behavior classification accuracy for judging pass / fail, learns with all the configurable dimensions, calculates the behavior classification accuracy, and then sets the behavior classification model according to the calculated behavior classification accuracy. The number of dimensions may be determined.

The behavior classification accuracy calculated by the behavior learning unit 2602 may be regarded as the contribution rate described in Example 1. For example, the acquired variable after dimension reduction is associated with the behavior classification accuracy calculated using it, and the calculated behavior classification accuracy is the contribution rate of the variable after dimension reduction used for calculation to the original information. do. The dimension control unit 2601 determines which of the variables after the dimension reduction is used for control according to the contribution ratio thus determined.

<Learning process>
FIG. 27 is a flowchart showing a detailed processing procedure example of the learning process by the server 101 (learning device) according to the seventh embodiment. The server 101 determines the number of dimensions by the dimension number control unit 2601. At this time, when the dimension reduction is carried out for the first time, the predetermined number of dimensions is determined, and in the case of the second and subsequent dimension reductions, the number of dimensions which has not been determined so far is determined (step S2700).

Next, the server 101 performs dimension reduction by the dimension reduction unit 2601 according to the determined number of dimensions, and generates a new variable (S2701). In step S2702, the server 101 makes a pass / fail judgment on the behavior classification accuracy acquired from the behavior learning unit 2602, and if it passes, it proceeds to step S1307, and if it fails, it returns to step S2700.

As described above, according to the seventh embodiment, the complicated behavior can be discriminated with high accuracy by changing the number of dimensions according to the behavior classification accuracy of the target and repeating the dimension reduction.

Further, the behavior recognition device and the learning device of Examples 1 to 7 described above can be configured as described in (1) to (14) below.

(1) An action recognition device (client 102) having a processor 201 for executing a program and a storage device 202 for storing the program is used for component analysis (principal component analysis or component analysis) to generate statistical components by multivariate analysis. The behavior classification model group learned for each component group can be accessed by using the component group obtained from the shape of the learning target (skeleton information 320) by the independent component analysis) and the behavior of the learning target. The processor 201 is based on the detection process of detecting the shape of the recognition target (skeleton information 320) from the analysis target data obtained from the sensor 103 and the shape of the recognition target detected by the detection process by the component analysis. Based on the component analysis process that produces one or more components and the respective contribution rates of the above components, and the cumulative contribution rate obtained from each of the above contribution rates, the order k indicating each dimension of the above one or more. From the behavior classification model group, a determination process for determining the above and a specific behavior classification model learned in the same component group as the specific component group containing one or more components of the order k indicating the dimension determined by the determination process. The selection process for selecting and the action recognition process for outputting the recognition result indicating the action of the recognition target by inputting the specific component group into the specific action classification model selected by the selection process are executed. ..

As a result, since the behavior classification model according to the shape of the learning target is prepared, it is possible to recognize a plurality of types of behaviors to be recognized with high accuracy.

(2) In the behavior recognition device of the above (1), each behavior classification model of the behavior classification model group is obtained from the shape of the learning target and the angles (joint angles 370) of a plurality of apex constituting the shape. Learning is performed for each component group using the component group and the behavior of the learning target, and the processor 201 has a plurality of vertices constituting the shape of the recognition target based on the shape of the recognition target. A calculation process for calculating an angle (joint angle 370) is executed, and in the component analysis process, the processor 201 determines the shape of the recognition target and the angle of the apex of the recognition target calculated by the calculation process. Based on this, the one or more components and the contribution rate are generated.

As a result, it is possible to recognize a plurality of types of actions to be recognized with high accuracy according to the change in shape caused by the angle of the apex.

(3) In the behavior recognition device of the above (1), each behavior classification model of the behavior classification model group has a component group obtained from the shape of the learning target and the movement amount of the learning target, and the behavior of the learning target. The processor 201 executes a calculation process for calculating the movement amount of the recognition target based on a plurality of shapes at different time points of the recognition target, and the processor 201 is learned for each component group. In the component analysis process, the processor 201 generates the one or more components and the contribution rate based on the shape of the recognition target and the movement amount of the recognition target calculated by the calculation process. do.

As a result, it is possible to recognize a plurality of types of behaviors to be recognized with high accuracy according to changes in shape over time due to movement.

(4) In the behavior recognition device of the above (1), the processor 201 executes a first normalization process for normalizing the size of the shape of the recognition target, and in the component analysis process, the processor 201 performs the first normalization process. Based on the shape of the recognition target after the first normalization by the first normalization process, the one or more components and the contribution rate are generated.

As a result, it is possible to suppress misrecognition by improving the versatility of behavior classification.

(5) In the behavior recognition device of the above (2), the processor 201 executes a second normalization process for normalizing a value range in which the shape of the recognition target and the angle of the apex can be taken, and in the component analysis process, the second normalization process is performed. The processor 201 generates the one or more components and the contribution rate based on the shape of the recognition target and the angle of the apex (joint angle 370) after the second normalization by the second normalization process. do.

As a result, it is possible to suppress the bias of the range in different data types such as shape and angle, and it is possible to improve the accuracy of action recognition.

(6) In the action recognition device of (1) above, in the determination process, the processor 201 determines an ordinal number k indicating the dimension of the component required for the cumulative contribution rate to exceed the threshold value.

Since the cumulative contribution rate is a measure of how much the newly generated components represent the amount of information in the original data, the increase in the number of dimensions is suppressed by referring to the cumulative contribution rate. be able to.

(7) In the behavior recognition device of the above (1), each behavior classification model of the behavior classification model group uses a component group obtained from a partially missing shape of the learning target and the behavior of the learning target. The processor 201 is learned for each combination of the partially missing shape and the component group, and the processor 201 is determined in the determination process for determining the partially missing shape of the recognition target, and the component analysis process is the processor 201. Generates one or more components and the contribution ratios of each of the one or more components based on the partially missing shape of the recognition target determined by the determination process, and in the selection process, The processor 201 selects from the behavior classification model group a specific behavior classification model learned by a combination of the same defect shape as the partially missing shape of the recognition target and the same component group as the specific component group.

Even if the shape of the recognition target is partially defective, highly accurate behavior recognition can be performed by using the behavior classification model corresponding to the partial defect.

(8) In the behavior recognition device of the above (1), the processor 201 executes an interpolation process of interpolating if there is a partial defect in the shape of the recognition target, and in the component analysis process, the processor 201 is the processor 201. Based on the shape of the recognition target after interpolation by the interpolation process, the one or more components and the contribution rate are generated.

As a result, it is possible to give an appropriate input to the behavior classification model generated by the learning target having no defect in the shape, and it is possible to suppress the deterioration of the behavior recognition accuracy.

(9) A behavior recognition device (client 102) having a processor 201 for executing a program and a storage device 202 for storing the program is dimensionally reduced (principal component analysis or) to generate statistical components in multivariate analysis. independent component analysis or SNE (Stochastic Neighbor Embedding) or t-SNE (t-Distributed Stochastic Neighbor Embedding) or UMAP (Uniform Manifold Approximation and Projection) or Isomap or LLE (Locally Linear Embedding) or Laplacian specific map (Laplacian Eignmap) or LargeVis Or a behavior classification model group learned for each component group using the component group in ascending order from the first variable obtained from the shape of the learning target (skeleton information 320) by the diffusion map) and the behavior of the learning target. The processor 201 detects the shape of the recognition target (skeleton information 320) from the analysis target data obtained from the sensor 103, and the detection process is detected by the dimension reduction. Based on the shape of the recognition target, one or more components, the contribution rate of each of the components, and the dimension reduction processing for generating, and the cumulative contribution rate obtained from each of the contribution rates, the one or more. A determination process for determining the order number k indicating the dimension of the ascending component from the first variable among the components, and a specific component group from the first variable to the component having the order number k indicating the dimension determined by the determination process. By selecting a specific behavior classification model learned in the same component group from the behavior classification model group and inputting the specific component group into the specific behavior classification model selected by the selection process. The action recognition process for outputting the recognition result indicating the action of the recognition target is executed.

As a result, since the behavior classification model according to the shape of the learning target is prepared, it is possible to recognize a plurality of types of behaviors to be recognized with high accuracy.

(10) In the behavior recognition device of the above (9), each behavior classification model of the behavior classification model group is obtained from the shape of the learning target and the angles (joint angles 370) of a plurality of apex constituting the shape. The learning is performed for each component group using the component group in ascending order from the first variable and the behavior of the learning target, and the processor 201 is the shape of the recognition target based on the shape of the recognition target. The calculation process for calculating the angles (joint angles 370) of a plurality of apex constituting the above is executed, and in the dimension reduction process, the processor 201 has the shape of the recognition target and the recognition target calculated by the calculation process. Based on the angle of the apex of, the one or more components and the contribution ratio are generated.

As a result, it is possible to recognize a plurality of types of actions to be recognized with high accuracy according to the change in shape caused by the angle of the apex.

(11) In the behavior recognition device of the above (9), each behavior classification model of the behavior classification model group is a component group in ascending order from the first variable obtained from the shape of the learning target and the movement amount of the learning target. And the behavior of the learning target are learned for each component group, and the processor 201 calculates the movement amount of the recognition target based on a plurality of shapes of the recognition target at different time points. The calculation process is executed, and in the dimension reduction process, the processor 201 includes the one or more components based on the shape of the recognition target and the movement amount of the recognition target calculated by the calculation process. The contribution rate and the above are generated.

As a result, it is possible to recognize a plurality of types of behaviors to be recognized with high accuracy according to changes in shape over time due to movement.

(12) In the behavior recognition device of the above (9), the processor 201 executes a first normalization process for normalizing the size of the shape of the recognition target, and in the dimension reduction process, the processor 201 performs the first normalization process. Based on the shape of the recognition target after the first normalization by the first normalization process, the one or more components and the contribution rate are generated.

As a result, it is possible to suppress misrecognition by improving the versatility of behavior classification.

(13) In the behavior recognition device of the above (10), the processor 201 executes a second normalization process for normalizing a value range in which the shape of the recognition target and the angle of the apex can be taken, and in the dimension reduction process, the second normalization process is performed. The processor 201 generates the one or more components and the contribution rate based on the shape of the recognition target and the angle of the apex (joint angle 370) after the second normalization by the second normalization process. do.

As a result, it is possible to suppress the bias of the range in different data types such as shape and angle, and it is possible to improve the accuracy of action recognition.

(14) In the behavior recognition device of the above (9), in the determination process, the processor 201 is in ascending order from the first variable required for the cumulative contribution rate from the first variable to exceed the threshold value. The ordinal k indicating the dimension of the component is determined.

Since the cumulative contribution rate is a measure of how much the newly generated components represent the amount of information in the original data, the increase in the number of dimensions is suppressed by referring to the cumulative contribution rate. be able to.

(15) In the behavior recognition device of the above (9), each behavior classification model of the behavior classification model group has a component group in ascending order from the first variable obtained from a partially missing shape of the learning target and the learning. The behavior of the target is learned for each combination of the partially missing shape and the component group, and the processor 201 performs a determination process for determining the partially missing shape of the recognition target and the dimension. In the reduction process, the processor 201 generates the one or more components and the contribution ratio of each of the one or more components based on the partially missing shape of the recognition target determined by the determination process. Then, in the selection process, the processor 201 uses the specific behavior classification model learned by the combination of the partially defective shape and the same defective shape of the recognition target and the same component group as the specific component group. Select from the classification model group.

Even if the shape of the recognition target is partially defective, highly accurate behavior recognition can be performed by using the behavior classification model corresponding to the partial defect.

(16) In the behavior recognition device of the above (9), the processor 201 executes an interpolation process of interpolating if there is a partial defect in the shape of the recognition target, and in the dimension reduction process, the processor 201 is said. Based on the shape of the recognition target after interpolation by the interpolation process, the one or more components and the contribution rate are generated.

As a result, it is possible to give an appropriate input to the behavior classification model generated by the learning target having no defect in the shape, and it is possible to suppress the deterioration of the behavior recognition accuracy.

(17) In a learning device having a processor 201 for executing a program and a storage device 202 for storing the program, the processor 201 includes an acquisition process for acquiring teacher data including a shape and an action of a learning target. A component analysis process that generates one or more components based on the shape of the learning target acquired by the acquisition process by component analysis (principal component analysis or independent component analysis) that generates statistical components by variable analysis. , A control process that controls the order indicating the dimension of each of the one or more components based on the permissible calculation amount, a component group containing one or more components of the order indicating the dimension controlled by the control process, and the learning. Based on the behavior of the target, the behavior learning process of learning the behavior of the learning target and generating a behavior classification model for classifying the behavior of the learning target is executed.

As a result, it is possible to prepare a plurality of types of behavior classification models according to the shape of the learning target, so that it is possible to recognize a plurality of types of behaviors to be recognized with high accuracy.

(18) In the learning device of the above (17), the processor 201 performs a calculation process of calculating angles (joint angles 370) of a plurality of vertices constituting the shape of the learning target based on the shape of the learning target. In the component analysis process, the processor 201 generates one or more components based on the shape of the learning target and the angle of the apex of the learning target calculated by the calculation process.

As a result, multiple types of behavior classification models can be prepared according to the change in shape due to the angle of the apex, so that multiple types of behavior according to the change in shape due to the angle of the apex to be recognized can be prepared. It can be recognized with high accuracy.

(19) In the learning device of the above (17), the processor 201 executes a calculation process for calculating the movement amount of the learning target based on a plurality of shapes of the learning target at different time points, and performs the component analysis process. Then, the processor 201 generates the one or more components based on the shape of the learning target and the movement amount of the learning target calculated by the calculation process.

As a result, it is possible to prepare multiple types of behavior classification models according to the change in shape caused by movement over time. It can be recognized with accuracy.

(20) In the learning device of the above (17), the processor 201 executes a first normalization process for normalizing the size of the shape of the learning target, and in the component analysis process, the processor 201 is the processor 201. Based on the shape of the learning target after the first normalization by the first normalization process, the one or more components are generated.

As a result, it is possible to suppress erroneous learning by improving the versatility of behavior classification learning.

(21) In the learning apparatus of (18), the processor 201 executes a second normalization process for normalizing a value range in which the shape of the learning target and the angle of the apex can be taken, and in the component analysis process, the processor 201 executes the second normalization process. The processor 201 generates the one or more components based on the shape of the learning target and the angle of the apex after the second normalization by the second normalization process.

As a result, it is possible to suppress the bias of the range in different data types such as shape and angle, and it is possible to improve the accuracy of behavior classification learning.

(22) In the learning device of the above (17), the processor 201 executes a defect control process for partially deleting the shape of the learning target, and in the component analysis process, the processor 201 is subjected to the defect control process. Based on the obtained partially missing shape of the learning target, the one or more components are generated, and in the behavior learning process, the processor 201 is based on the component group and the behavior of the learning target. Then, the behavior of the learning target is learned, the behavior classification model is generated, and the behavior is associated with the defect information regarding the partially deleted shape.

By intentionally generating a partially missing shape, it is possible to increase the number of types of behavior classification models. This enables highly accurate action recognition corresponding to various shapes of recognition targets.

(23) In a learning device having a processor 201 for executing a program and a storage device 202 for storing the program, the processor 201 includes an acquisition process for acquiring teacher data including a shape and an action to be learned. Dimensionality reduction (principal component analysis or independent component analysis or SNE (Stochastic Negative Component Analysis) or t-SNE (t-Distributed Statistical Neighbor Embedding) or UMAP (Uniform Application) Alternatively, a dimension reduction process that generates one or more components based on the shape of the learning target acquired by the acquisition process by LLE (Statistical Liner Embedding) or Laplacian Eignmap or LargeVis or diffusion map). A control process for controlling the order number indicating the dimension of the ascending component from the first variable among the one or more components and the dimension controlled by the control process from the first variable are shown based on the allowable calculation amount. Based on the component group up to the component of the order number and the behavior of the learning target, the behavior learning process of learning the behavior of the learning target and generating a behavior classification model for classifying the behavior of the learning target. Execute.

As a result, it is possible to prepare a plurality of types of behavior classification models according to the shape of the learning target, so that it is possible to recognize a plurality of types of behaviors to be recognized with high accuracy.

(24) In the learning device of the above (23), the processor 201 performs a calculation process of calculating angles (joint angles 370) of a plurality of vertices constituting the shape of the learning target based on the shape of the learning target. In the dimension reduction process, the processor 201 generates one or more components based on the shape of the learning target and the angle of the apex of the learning target calculated by the calculation process.

As a result, multiple types of behavior classification models can be prepared according to the change in shape due to the angle of the apex, so that multiple types of behavior according to the change in shape due to the angle of the apex to be recognized can be prepared. It can be recognized with high accuracy.

(25) In the learning device of the above (23), the processor 201 executes a calculation process of calculating the movement amount of the learning target based on a plurality of shapes of the learning target at different time points, and the dimension reduction process. Then, the processor 201 generates the one or more components based on the shape of the learning target and the movement amount of the learning target calculated by the calculation process.

As a result, it is possible to prepare multiple types of behavior classification models according to the change in shape caused by movement over time. It can be recognized with accuracy.

(26) In the learning device of the above (23), the processor 201 executes a first normalization process for normalizing the size of the shape of the learning target, and in the dimension reduction process, the processor 201 is the processor 201. Based on the shape of the learning target after the first normalization by the first normalization process, the one or more components are generated.

As a result, it is possible to suppress erroneous learning by improving the versatility of behavior classification learning.

(27) In the learning apparatus of (24), the processor 201 executes a second normalization process for normalizing a value range in which the shape of the learning target and the angle of the apex can be taken, and in the dimension reduction process, the processor 201 executes the second normalization process. The processor 201 generates the one or more components based on the shape of the learning target and the angle of the apex after the second normalization by the second normalization process.

As a result, it is possible to suppress the bias of the range in different data types such as shape and angle, and it is possible to improve the accuracy of behavior classification learning.

(28) In the learning device of the above (23), the processor 201 executes a defect control process for partially deleting the shape of the learning target, and in the dimension reduction process, the processor 201 is subjected to the defect control process. Based on the obtained partially missing shape of the learning target, the one or more components are generated, and in the behavior learning process, the processor 201 ranges from the first variable to the component of the order indicating the dimension. Based on the component group and the behavior of the learning target, the behavior of the learning target is learned to generate the behavior classification model, and the behavior is associated with the defect information regarding the partially deleted shape.

By intentionally generating a partially missing shape, it is possible to increase the number of types of behavior classification models. This enables highly accurate action recognition corresponding to various shapes of recognition targets.

It should be noted that the present invention is not limited to the above-mentioned examples, but includes various modifications and equivalent configurations within the scope of the attached claims. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to those having all the described configurations. Further, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Further, the configuration of another embodiment may be added to the configuration of one embodiment. In addition, other configurations may be added, deleted, or replaced with respect to a part of the configurations of each embodiment.

Further, each configuration, function, processing unit, processing means, etc. described above may be realized by hardware by designing a part or all of them by, for example, an integrated circuit, and the processor 201 performs each function. It may be realized by software by interpreting and executing the program to be realized.

Information such as programs, tables, and files that realize each function is recorded in a memory, a hard disk, a storage device such as an SSD (Solid State Drive), or an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versail Disc). It can be stored in a medium.

In addition, the control lines and information lines show what is considered necessary for explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In practice, you can think of almost all configurations as interconnected.

100 Behavior recognition system 101 Server 102 Client 103 Sensor 104 Teacher signal DB
201 Processor 202 Storage device 320 Skeletal information 401 Teacher signal acquisition unit 402 Missing information control unit 403,453 Skeletal information processing unit 404,454 Principal component analysis unit 405,2601 Dimension control unit 406,2200,2602 Behavior learning unit 451 Skeletal detection Part 452 Missing information judgment part 455 Dimensional number determination part 456 Behavior classification model selection part 457, 2201 Action recognition part 501 Joint angle calculation part 502 Movement amount calculation part 503 Normalization part 1652 Missing information interpolation part 1904 Mutual information normalization part 2100, 2101, 2600, 2603 Dimensionality reduction department

Claims (20)

An action recognition device having a processor for executing a program and a storage device for storing the program.
Generate statistical components by multivariate analysis Using the component group obtained from the shape of the learning target by component analysis and the behavior of the learning target, it is possible to access the behavior classification model group learned for each component group. And
The processor
Detection processing to detect the shape of the recognition target from the analysis target data,
A component analysis process that produces one or more components and the contribution rate of each of the components based on the shape of the recognition target detected by the component analysis.
A determination process for determining an ordinal number indicating each dimension of the one or more components based on the cumulative contribution rate obtained from each of the contribution rates.
A selection process for selecting a specific behavior classification model learned in the same component group as a specific component group containing one or more components of an ordinal number indicating a dimension determined by the determination process from the behavior classification model group.
Behavior recognition processing that outputs a recognition result indicating the behavior of the recognition target by inputting the specific component group into the specific behavior classification model selected by the selection processing.
An action recognition device characterized by performing. The behavior recognition device according to claim 1.
Each behavior classification model of the behavior classification model group is a combination of the partially defective shape and the component group using the component group obtained from the partially defective shape of the learning target and the behavior of the learning target. It is learned for each
The processor
Judgment processing for determining the partially missing shape of the recognition target, and
In the component analysis process, the processor determines the contribution rate of each of the one or more components and the one or more components based on the partially missing shape of the recognition target determined by the determination process. Generate and
In the selection process, the processor uses the behavior classification model group to obtain a specific behavior classification model learned by combining the same defect shape as the partially missing shape of the recognition target and the same component group as the specific component group. Choose from,
An action recognition device characterized by this. The behavior recognition device according to claim 1.
The processor
If there is a partial defect in the shape to be recognized, interpolation processing is executed to interpolate.
In the component analysis process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target after interpolation by the interpolation process.
An action recognition device characterized by this. An action recognition device having a processor for executing a program and a storage device for storing the program.
It was learned for each component group using the ascending component group from the first variable obtained from the shape of the learning target by the dimension reduction to generate statistical components by multivariate analysis and the behavior of the learning target. Access to behavioral classification models,
The processor
Detection processing to detect the shape of the recognition target from the analysis target data,
A dimension reduction process that generates one or more components and the contribution ratio of each of the components based on the shape of the recognition target detected by the detection process by the dimension reduction process.
A determination process for determining the ordinal number indicating the dimension of the ascending component from the first variable among the one or more components based on each of the contribution rates.
A selection process for selecting from the behavior classification model group a specific behavior classification model learned in the same component group as the specific component group from the first variable to the component of the ordinal number indicating the dimension determined by the determination process. ,
Behavior recognition processing that outputs a recognition result indicating the behavior of the recognition target by inputting the specific component group into the specific behavior classification model selected by the selection processing.
An action recognition device characterized by performing. The action recognition device according to claim 4.
Each behavior classification model of the behavior classification model group includes an ascending component group from the first variable obtained from the shape of the learning target and the angles of a plurality of vertices constituting the shape, and the behavior of the learning target. Is learned for each component group using
The processor
Based on the shape of the recognition target, a calculation process for calculating the angles of a plurality of vertices constituting the shape of the recognition target is executed.
In the dimension reduction process, the processor obtains the one or more components and the contribution rate based on the shape of the recognition target and the angle of the apex of the recognition target calculated by the calculation process. Generate,
An action recognition device characterized by this. The action recognition device according to claim 4.
Each behavior classification model of the behavior classification model group uses an ascending component group from the first variable obtained from the shape of the learning target and the movement amount of the learning target, and the behavior of the learning target. It is learned for each component group,
The processor
A calculation process for calculating the movement amount of the recognition target is executed based on a plurality of shapes of the recognition target at different time points.
In the dimension reduction process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target and the movement amount of the recognition target calculated by the calculation process. do,
An action recognition device characterized by this. The action recognition device according to claim 4.
The processor
The first normalization process for normalizing the size of the shape to be recognized is executed.
In the dimension reduction process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target after the first normalization by the first normalization process.
An action recognition device characterized by this. The behavior recognition device according to claim 5.
The processor
The second normalization process for normalizing the range that the shape of the recognition target and the angle of the apex can take is executed.
In the dimension reduction process, the processor generates the one or more components and the contribution rate based on the shape of the recognition target and the angle of the apex after the second normalization by the second normalization process. do,
An action recognition device characterized by this. The action recognition device according to claim 4.
In the determination process, the processor determines an ordinal number indicating the dimension of the ascending component from the first variable required for the contribution to exceed the threshold.
An action recognition device characterized by this. The action recognition device according to claim 4.
Each behavior classification model of the behavior classification model group uses the component group in ascending order from the first variable obtained from the partially defective shape of the learning target and the behavior of the learning target, and the partial defect. It is learned for each combination of shape and component group,
The processor
Judgment processing for determining the partially missing shape of the recognition target, and
In the dimension reduction processing, the processor determines the contribution rate of each of the one or more components and the one or more components based on the partially missing shape of the recognition target determined by the determination process. Generate and
In the selection process, the processor uses the behavior classification model group to obtain a specific behavior classification model learned by combining the same defect shape as the partially missing shape of the recognition target and the same component group as the specific component group. Choose from,
An action recognition device characterized by this. The action recognition device according to claim 4.
The processor
If there is a partial defect in the shape to be recognized, interpolation processing is executed to interpolate.
In the dimension reduction processing, the processor generates the one or more components and the contribution ratio based on the shape of the recognition target after interpolation by the interpolation processing.
An action recognition device characterized by this. A learning device having a processor for executing a program and a storage device for storing the program.
The processor
Acquisition process to acquire teacher data including the shape and behavior of the learning target,
A component analysis process that generates one or more components based on the shape of the learning target acquired by the acquisition process by component analysis that generates statistical components by multivariate analysis.
Control processing that controls the ordinal number indicating each dimension of the one or more components based on the allowable calculation amount, and
The behavior of the learning target is learned and the behavior of the learning target is classified based on the component group containing one or more components of the order indicating the dimension controlled by the control process and the behavior of the learning target. Behavior learning process to generate behavior classification model,
A learning device characterized by performing. The learning device according to claim 12.
The processor
The defect control process for partially deleting the shape of the learning target is executed, and the defect control process is executed.
In the component analysis process, the processor generates the one or more components based on the partially defective shape of the learning target obtained by the defect control process.
In the behavior learning process, the processor
Based on the component group and the behavior of the learning target, the behavior of the learning target is learned to generate the behavior classification model, which is associated with the defect information regarding the partially deleted shape.
A learning device characterized by that. A learning device having a processor for executing a program and a storage device for storing the program.
The processor
Acquisition process to acquire teacher data including the shape and behavior of the learning target,
Dimension reduction processing that generates one or more components based on the shape of the learning target acquired by the acquisition process by dimension reduction that generates statistical components in multivariate analysis.
A control process that controls the ordinal number indicating the dimension of the ascending component from the first variable among the one or more components based on the allowable calculation amount.
The behavior of the learning target is learned based on the component group from the first variable to the component of the order indicating the dimension controlled by the control process and the behavior of the learning target, and the behavior of the learning target is learned. Behavior learning process that generates behavior classification model to classify
A learning device characterized by performing. The learning device according to claim 14.
The processor
Based on the shape of the learning target, a calculation process for calculating the angles of a plurality of vertices constituting the shape of the learning target is executed.
In the dimension reduction process, the processor generates the one or more components based on the shape of the learning target and the angle of the apex of the learning target calculated by the calculation process.
A learning device characterized by that. The learning device according to claim 14.
The processor
A calculation process for calculating the amount of movement of the learning target is executed based on a plurality of shapes of the learning target at different time points.
In the dimension reduction process, the processor generates one or more components based on the shape of the learning target and the movement amount of the learning target calculated by the calculation process.
A learning device characterized by that. The learning device according to claim 14.
The processor
The first normalization process for normalizing the size of the shape of the learning target is executed.
In the dimension reduction process, the processor generates the one or more components based on the shape of the learning target after the first normalization by the first normalization process.
A learning device characterized by that. The learning device according to claim 15.
The processor
The second normalization process for normalizing the range that the shape of the learning target and the angle of the apex can take is executed.
In the dimension reduction process, the processor generates the one or more components based on the shape and the angle of the apex of the learning target after the second normalization by the second normalization process.
A learning device characterized by that. The learning device according to claim 14.
The processor
The defect control process for partially deleting the shape of the learning target is executed, and the defect control process is executed.
In the dimension reduction processing, the processor generates the one or more components based on the partially defective shape of the learning target obtained by the defect control processing.
In the behavior learning process, the processor
Based on the component group from the first variable to the component of the ordinal number indicating the dimension and the behavior of the learning target, the behavior of the learning target is learned to generate the behavior classification model, and the part thereof. Associate with missing information about the missing shape,
A learning device characterized by that. A behavior recognition method executed by an behavior recognition device having a processor that executes a program and a storage device that stores the program.
Generate statistical components by multivariate analysis Using the component group obtained from the shape of the learning target by component analysis and the behavior of the learning target, it is possible to access the behavior classification model group learned for each component group. And
The behavior recognition method is
The processor
Detection processing to detect the shape of the recognition target from the analysis target data,
A component analysis process that produces one or more components and the contribution rate of each of the components based on the shape of the recognition target detected by the component analysis.
A determination process for determining an ordinal number indicating each dimension of the one or more components based on the cumulative contribution rate obtained from each of the contribution rates.
A selection process for selecting a specific behavior classification model learned in the same component group as a specific component group containing one or more components of an ordinal number indicating a dimension determined by the determination process from the behavior classification model group.
Behavior recognition processing that outputs a recognition result indicating the behavior of the recognition target by inputting the specific component group into the specific behavior classification model selected by the selection processing.
A behavior recognition method characterized by performing.

Images