JP2021191356A - Correction content learning device, operation correction device and program - Google Patents

Abstract

To provide a correction content learning device, operation correction device and program which can grasp the improvement point with the correction content for the operation of a user.SOLUTION: A correction content learning device 100 comprises: moving image data input means 10 which inputs moving image data; feature data acquisition means 20 which acquires a plurality of pieces of feature data from the moving image data; model person feature data storage means 30 which stores model person feature data; feature data comparison means 40 which compares the acquired feature data with the feature data of a model person; teacher data storage means 50 which stores teacher data indicating a correction content for the comparison result; and learning means 60 which has a neural network constitution, in which an input layer takes in the comparison result and the number of output layers corresponds to each feature data. The learning means 60 changes a coefficient between neurons of the input layer and an intermediate layer in correspondence with the comparison result and teacher data.SELECTED DRAWING: Figure 6

Description

The present invention relates to a technique for efficiently correcting movements in exercise, sports, and the like.

In recent years, exercise has been carried out for various purposes such as health purposes and competition purposes. In order to achieve these goals, it is important to carry out with proper operation. For this reason, in the past, the act of imitating a person's movements or seeking guidance was performed. Recently, a learning system that can easily improve proficiency in sports using a computer or the Internet has been proposed (see Patent Document 1).

JP-A-2017-64095

However, in the technique described in Patent Document 1, the captured image is simply displayed, and the user who wants to correct the motion must judge the difference by himself / herself and reflect it in his / her motion. There is also the problem that it is not possible to know the correction details in real time.

Therefore, it is an object of the present invention to provide a correction content learning device, a motion correction device, and a program capable of grasping improvement points in the correction content for the user's movement.

In order to solve the above problems, the present invention
Video data input means for inputting video data of a moving human body,
A feature data acquisition means for acquiring feature data expressing motion features from the input video data as user feature data, and
A modeler feature data storage means that stores the feature data of the modeler as modeler feature data,
A teacher data storage means that stores the feature data indicating the content to be corrected with respect to the comparison result by the feature data comparison means for comparing the user feature data with the model feature data and the feature data comparison means as teacher data. When,
It has a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, the input layer includes means for capturing the comparison result of the feature data comparison means, and the output layer is a number corresponding to the comparison result. Provided is a correction content learning apparatus comprising the above-mentioned neurons and a learning means for changing the coefficient between the neurons in the layers so as to change the value of the neurons corresponding to the teacher data in the output layer.

Further, the correction content learning device of the present invention is
The feature data may include a site in the human body and a correction amount corresponding to the site.

Further, the correction content learning device of the present invention is
The correction amount may be an angle at the site.

Further, the correction content learning device of the present invention is
The angle at the site may be an angle formed by a portion connected to the site in a predetermined state.

Further, the correction content learning device of the present invention is
The angle at the site may be an angle formed at different time points by the connecting sites that are connected to the site.

Further, the present invention
Video data input means for inputting video data of a moving human body,
A feature data acquisition means for acquiring feature data expressing motion features from the input video data as user feature data, and
A modeler feature data storage means that stores the feature data of the modeler as modeler feature data,
It has a feature data comparison means for comparing the user feature data with the model feature data, a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, and the input layer is the feature data comparison means. The output layer is provided with a means for capturing the comparison result, the output layer is provided with a number of neurons corresponding to the comparison result, and the correction amount calculation means for outputting the correction amount in the output layer is provided.
The correction content transmission means for transmitting the correction content to the operating user according to the output correction amount, and the correction content transmission means.
Provided is a motion correction device.

Further, the present invention
Computer,
Feature data acquisition means that acquires feature data that expresses the characteristics of operation as user feature data from video data,
Modeler feature data storage means that stores the feature data of the model as model feature data,
Teacher data storage that stores the feature data indicating the content to be corrected with respect to the feature data comparison means for comparing the user feature data with the model feature data and the comparison result by the feature data comparison means as teacher data. means,
It has a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, the input layer includes means for capturing the comparison result of the feature data comparison means, and the output layer is a number corresponding to the comparison result. Provided is a program for functioning as a learning means for changing a coefficient between neurons between layers so as to change the value of the neuron corresponding to the teacher data in the output layer.

Further, the present invention
Computer,
Feature data acquisition means that acquires feature data that expresses the characteristics of operation as user feature data from video data,
Modeler feature data storage means that stores the feature data of the model as model feature data,
A feature data comparison means for comparing the user feature data with the model feature data,
It has a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, the input layer includes means for capturing the comparison result of the feature data comparison means, and the output layer is a number corresponding to the comparison result. A correction amount calculation means, which is equipped with the above-mentioned neurons and outputs a correction amount in the output layer.
Provided is a program for functioning as a correction content transmission means for transmitting correction content to a user in operation according to the output correction amount.

According to the present invention, it is possible to provide a correction content learning device, a motion correction device, and a program capable of grasping the correction content for the user's movement.

It is a figure which shows the difference of the skeleton data by the difference in body size. It is a figure which shows the difference of the skeleton data by the state of movement of each individual. It is a figure which shows the angle formed by the swing width of an arm. It is a figure which shows the angle formed by the bending degree of a knee. It is a figure which shows the skeleton data of two persons having a physical disparity. It is a functional block diagram of the correction content learning apparatus 100 which concerns on one Embodiment of this invention. It is a hardware block diagram of the main part of the correction content learning device 100 and the motion correction device 200 according to one embodiment of the present invention. It is a flowchart which shows the processing operation of the correction content learning apparatus 100. It is a functional block diagram of the motion correction apparatus 200 which concerns on one Embodiment of this invention. It is a flowchart which shows the processing operation of the motion correction apparatus 200. It is a figure which shows the application example of the correction content learning device 100, and the motion correction device 200. It is a figure which shows the relationship between the user characteristic data, the model person characteristic data, and the teacher data.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.
In the present embodiment, the feature data obtained from the moving image data of the user's movement is compared with the feature data of the model, and the comparison result and the correction data consisting of the correction site and the correction amount determined by the instructor are obtained. It is given to the neural network as teacher data, and the neural network is changed to learn so that the correction content of the optimum operation is obtained.

<1. Feature data required for motion correction>
Here, the feature data necessary for motion correction will be described. In the correction content learning device 100 according to the present embodiment and the motion correction device 200 described later, the correction content is determined using the moving image data obtained by photographing the movement of the human body. Basically, it is conceivable to compare the moving image data of the user who is the correction target and the moving image data of the model. For example, skeleton data can be acquired from each moving image data and the skeleton data can be compared with each other.

However, the simple method of comparing skeletal data has the following two problems. The first problem is that a simple comparison of the skeletal data of the model and the user includes components due to the physical disparity between the model and the user. Therefore, the skeletal data differs greatly depending on the difference in body size. FIG. 1 is a diagram showing a difference in skeletal data due to a difference in body size. FIG. 1A is skeletal data of a person with a small physique, and FIG. 1B is skeletal data of a person with a large physique. In FIG. 1, the right arm and the right leg are shown by a solid line, and the left arm and the left leg are shown by a broken line. The same applies to FIGS. 2 to 5 described later. Since the difference in body size difference as shown in FIG. 1 cannot be advised, it is necessary to acquire a feature that can ignore the difference in body size between the modeler and the user.

The second problem is that the model and the user are moving at different timings. FIG. 2 is a diagram showing differences in skeletal data depending on the state of movement of each individual. 2 (a) to 2 (c) are skeleton data of the model, and FIGS. 2 (d) to 2 (f) are skeleton data of the user. FIG. 2 shows the skeleton data acquired by performing video capture when the time advances from left to right, respectively.

As shown in FIG. 2, the operating state changes depending on the timing of acquiring the skeleton data. In order to make an appropriate comparison, it is necessary to identify the operating states of the model and the user, such as running and walking, and compare them in the same state. The process of adjusting such operating states is extremely difficult, and the processing load is extremely heavy. Therefore, it is difficult to process in real time.

In order to solve the above two problems, in the present embodiment, a part that is a feature of the operation is acquired as feature data from the moving image data, and the correction content is specified using this feature data. The feature data may be of any mode as long as it can express the characteristics of the movement of the human body, but in the present embodiment, when comparing the skeleton data, a portion that becomes a joint between the skeletons is used. It is specified as a predetermined part, and the feature amount due to the movement in that part is used. Further, as the feature amount, an angle formed in relation to a predetermined site is used. Here, the angles formed in relation to a predetermined portion will be described with reference to FIGS. 3 and 4.

FIG. 3 is a diagram showing an angle formed by the swing width of the arm. As shown in FIG. 3, the position of the upper arm differs depending on the timing of each movement. When focusing on, for example, the upper right arm (shown by a solid line in the figure) as one upper arm, FIG. 3 (a) is the most posterior position, and FIG. 3 (c) is the most anterior position. The difference between the front position and the rear position is the swing width. As shown in FIG. 3D, the angle formed by the skeletons of both upper arms indicates the swing width. At this time, since the angle is formed around the shoulder, which is the base of the upper arm, the shoulder is a predetermined portion of interest. At this time, the angle at the site called the shoulder is an angle formed at different time points (timing) by the connecting site (upper arm) which is the site connected to the site (shoulder).

FIG. 4 is a diagram showing an angle formed by the degree of bending of the knee. As shown in FIG. 4, the bending angle of the knee differs depending on the timing of each movement. When focusing on, for example, the right leg as one leg, the model has the minimum angle in FIG. 4 (a) and the maximum angle in FIG. 4 (c). On the other hand, for the user, FIG. 4 (f) is the minimum angle, and FIG. 4 (e) is the maximum angle. In this case, the knee becomes the correction site, and the bending angle of the knee becomes the feature amount. At this time, the angle at the site called the knee is the angle formed by the portion (leg) connected to the site (knee) in a predetermined state.

As described above, by using the angle at a predetermined part formed by the movement as the feature data, it is not necessary to determine the movement state for each part, and a relatively simple comparison becomes possible.

Consider the case where there is a difference in body size between the two to be compared. FIG. 5 is a diagram showing skeletal data of two persons having a physical disparity. As is clear from FIG. 5, FIG. 5A is skeletal data of a person having a relatively small physique, and FIG. 5B is skeletal data of a person having a relatively large physique. When there is a difference in body size between the two to be compared in this way, a large processing load is applied to the comparison between the skeleton data. Even in such a case, for example, if the comparison is made based on the bending angle of the legs centered on the knees, an appropriate comparison can be made regardless of the difference in body size.

<2. Orthodontic content learning device ＞
First, a correction content learning device using a neural network will be described.
Machine learning by the correction content learning device 100 of FIG. 6 using the learning data will be described. When performing machine learning with the correction content learning device 100, for each part of the human body, the correction part and the correction amount considered to be the most appropriate by comparing the characteristic data of the user's movement with the characteristic data of the model of the movement. Is prepared as teacher data. As the correction site, for example, each site such as a shoulder, a knee, an elbow, an ankle, and a spine can be used. Further, the correction amount may be, for example, an angle with another portion or an angle with a predetermined reference such as a horizontal plane or a vertical plane.

FIG. 6 is a functional block diagram of the correction content learning device 100 according to the embodiment of the present invention. In FIG. 6, 10 is a moving image data input means, 20 is a feature data acquisition means, 30 is a model feature data storage means, 40 is a feature data comparison means, 50 is a teacher data storage means, and 60 is a learning means. The moving image data input means 10 is a means for inputting moving image data obtained by photographing a moving human body. Specifically, it can be realized with a camera capable of shooting a moving image. The feature data acquisition means 20 is a means for acquiring feature data expressing the characteristics of the movement of the human body from the moving image data input by the moving image data input means 10. The feature data may be any data as long as it expresses the characteristics of the movement of the human body, but in the present embodiment, a predetermined part in the human body and a feature amount related to the part are included. I'm out. Further, as a feature amount, in this embodiment, an angle obtained in relation to a predetermined site is used.

The modeler feature data storage means 30 is a storage means for storing modeler feature data, which is the modeler's feature data, to be compared with the user feature data, which is the feature data acquired from the human body of the operating user. The model person feature data is also obtained as feature data expressing the characteristics of the model person's movement by photographing the human body of the model person in motion by the same means as the moving image data input means 10 and the feature data acquisition means 20. Therefore, the modeler feature data storage means 30 separately stores the modeler feature data acquired by photographing the modeler. The feature data comparison means 40 is a means for comparing the feature data of the feature data acquisition means 20 and the model feature data storage means 30.

The teacher data storage means 50 is a storage means that stores correction data indicating the content to be corrected for the motion shown in the moving image data as teacher data. The correction data includes the correction part which is the part to be corrected in the human body and the correction amount which is the amount to be corrected in the correction part as the contents to be corrected. The amount of correction can also be expressed as the amount of features to be corrected. Therefore, the correction data is feature data expressing the correction content. In the present embodiment, the content of the correction data is determined by, for example, an instructor of the operation by looking at the moving image data input by the moving image data input means 10. The leader and the model may be the same person. Then, the determined correction data is set and stored in the teacher data storage means 50.

The learning means 60 identifies a correction site to be corrected and a candidate for a correction amount using the comparison result from the feature data comparison means 40, and stores the comparison result from the feature data comparison means 40 and the teacher data storage means 50. It is a means of learning using the correction data. The comparison result from the feature data comparison means 40 and the correction data stored in the teacher data storage means 50 are a learning set of the learning means 60. The learning means 60 has a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, the input layer includes means for capturing each feature data, and the output layer has a number of neurons corresponding to each feature data. The coefficient between the neurons in the layer is changed so that the value of the neuron in the output layer approaches the correction data which is the teacher data.

FIG. 7 is a hardware configuration diagram of a main part of the correction content learning device according to the embodiment of the present invention. The correction content learning device 100 according to the present embodiment can be realized by a camera constituting the moving image data input means 10 and a general-purpose computer. As shown in FIG. 7, the portion other than the moving image data input means 10 stores the CPU (Central Processing Unit) 1, the RAM (Random Access Memory) 2 which is the main memory of the computer, and the programs and data executed by the CPU 1. A large-capacity storage device 3 such as a hard disk and a flash memory, an instruction input I / F (interface) 4 such as a keyboard and a mouse, and a data input / output I for data communication with an external device such as a data storage medium. A / F (interface) 5 and a display unit 6 which is a display device such as a liquid crystal display are provided and are connected to each other via a bus.

The feature data acquisition means 20, the feature data comparison means 40, and the learning means 60 are realized by the CPU 1 executing a program stored in the storage device 3. That is, the computer executes the contents of each means according to a dedicated program. In the present specification, the computer means a device having an arithmetic processing unit such as a CPU and capable of data processing, and is not only a general-purpose computer such as a personal computer but also a portable terminal such as a tablet terminal or a smartphone. Also includes. The moving image data input means 10 is realized by connecting the cameras constituting the moving image data input means 10 to the data input / output I / F (interface) 5. The model person feature data storage means 30 and the teacher data storage means 50 are realized by the storage device 3.

Next, the processing operation of the correction content learning device 100 shown in FIG. 6 will be described. FIG. 8 is a flowchart showing the processing operation of the correction content learning device 100. First, the moving image data is input from the moving image data input means 10 (step S10). Specifically, a camera capable of shooting a moving image captures a user's movement and obtains moving image data. The moving image data is composed of a plurality of frames (still images) having different shooting times. For example, it is possible to obtain moving image data having 30 frames per second.

Next, the feature data acquisition means 20 acquires feature data from the captured moving image data (step S20). In this embodiment, the feature data is acquired in two steps. First, the skeleton data is acquired from the moving image data. This can be done by a known motion capture technique, and can be acquired as skeleton data such as a wire model, a wire frame, and a skeleton model. Skeleton data is created for each frame (still image). Therefore, for moving image data having 30 frames per second, 30 skeleton data can be obtained.

Then, the feature data is acquired from the skeleton data. As feature data, in this embodiment, a site and an angle generated by an operation related to the site are used. Each site can be acquired by designating a predetermined position in the skeletal data. In addition, it is also possible to specify a portion that becomes the axis of operation as a predetermined portion by analysis. In this embodiment, shoulders, knees, elbows, ankles, and spine are extracted as predetermined parts. The shoulders, knees, elbows, and ankles are extracted on the left and right respectively. Therefore, the feature data acquisition means 20 acquires a plurality of feature data according to the number of parts.

Then, the angle generated by the movement related to the specified predetermined portion is obtained. FIG. 3 shows an example of calculating the angle when the shoulder is specified as a predetermined portion. 3 (a) to 3 (c) are skeleton data acquired from each frame of the moving image data. That is, FIGS. 3A to 3C show the postures of the users at different moments. Focusing on the upper right arm shown by the solid line, it is located at the rearmost position (left side in the drawing) in FIG. 3 (a) and at the frontmost position (right side in the drawing) in FIG. 3 (c). In the skeleton data, since the skeleton of each part is expressed, the upper right arm is expressed as a line segment.

The feature data acquisition means 20 extracts the skeleton data of the frame in which the upper right arm is located most rearward with the shoulder as the center and the skeleton data of the frame located in the frontmost position, and positions the upper right arm and the most anterior position located in the rearmost position. Calculate the angle formed by the upper right arm. Specifically, as shown in FIG. 3D, the angle formed by the upper right arm located at the rearmost position and the upper right arm located at the frontmost position is calculated with the shoulder as the center. This angle expresses the swing width of the upper right arm centered on the right shoulder, which is a predetermined part.

In this way, the right shoulder, which is a predetermined part, and its angle are acquired as feature data. Similarly, the angles of the knees, elbows, ankles, and spine are calculated and acquired as feature data. The shoulders, knees, elbows, and ankles can be calculated for each of the left and right sides. Although it depends on the competition in which the movement is performed, it is usually symmetrical. Therefore, in the present embodiment, one of the shoulder, knee, elbow, and ankle is represented, and five characteristic data of the shoulder, knee, elbow, ankle, and spine are obtained. However, since the shoulders, knees, elbows, and ankles may differ on the left and right, it is also possible to obtain the left and right angles for each. Therefore, nine characteristic data of right shoulder, left shoulder, right knee, left knee, right elbow, left elbow, right ankle, left ankle, and spine may be obtained. As described above, in step S20, the user characteristic data expressing the characteristics of the user's movement is acquired from the moving image of the user's movement.

Next, the feature data comparison means 40 compares the model feature data with the user feature data (step S30). Here, the relationship between the user characteristic data, the model characteristic data, and the teacher data will be described. FIG. 12 is a diagram showing the relationship between the user characteristic data, the model person characteristic data, and the teacher data. In this embodiment, there are five parts to be compared: shoulder, knee, elbow, ankle, and spine. Moreover, in this embodiment, the angles of each part are compared. A * is the angle of each part of a certain A user, and M * is the angle of each part of the model. B * is the angle of each part of a certain B user as well. AC * is the result of comparing the angles of each part between the A user and the model. The comparison process is usually a difference process. The comparison result is obtained as a combination of each part and angle. That is, the comparison result, which is the difference between the user feature data and the model feature data, can also be obtained as the feature data which is a combination of each part and the angle. Therefore, in step S30, feature data corresponding to each site can be obtained as a comparison result.

On the other hand, looking at this comparison result, the instructor (trainer) decides one part to be corrected. In addition, the angle to be corrected is determined by looking at the difference angle of the part. For example, in the example shown in FIG. 12, it is determined that the part to be corrected is the knee for the A user, and the angle to be corrected is added by α degree to the comparison result. In addition, for B user, it is judged that the part to be corrected is the ankle, and the angle to be corrected is added by β degree to the comparison result. In this way, the part to be corrected and the angle to be corrected determined by the instructor are used as correction data. The correction data also consists of the part and angle, and has the same format as the feature data. The correction data determined by the instructor is stored in the teacher data storage means 50 as teacher data.

Next, the learning means 60 determines the correction content (step S40). As described above, the learning means 60 includes a deep neural network structure, and the number of input layers is the same as the number of parts (number of feature data) (five in this embodiment). Then, the teacher data is input from the teacher data storage means 50, and the feature data which is the comparison result of the feature data comparison means 40 is also input. Further, the same number of output layers as the number of parts (number of feature data) is prepared, and the angle which is the correction amount for each part is output. Then, the part where the correction amount is maximum is set as the correction part, and the correction amount is output together with the correction amount.

For example, five neurons are prepared in the output layer, each of which corresponds to the shoulder, knee, elbow, ankle, and spine, and the angle to be corrected is output for each. Then, the learning means 60 sets the site corresponding to the neuron that outputs the largest value among the angles output from the output layer as a correction site candidate. Then, in the learning means 60, the correction site candidate and its angle (correction amount) are compared with the teacher data. When the site of the correction site candidate and the site of the teacher data are the same (for example, when both are shoulders), the value of the angle of the neuron corresponding to the correction site candidate (for example, the neuron corresponding to the shoulder) is the correction amount of the teacher data. It changes the coefficient between neurons between layers so that it approaches a value at a certain angle.

On the other hand, as a result of comparison with the teacher data, if the correction site candidate and the teacher data site are different (for example, the correction site candidate is the shoulder and the teacher data is the knee), the neurons corresponding to the correction site candidate (for example, the shoulder). The value of the angle of the part corresponding to the teacher data becomes smaller, the value of the angle of the neuron corresponding to the part indicated by the teacher data (for example, the neuron corresponding to the knee) becomes larger, and the value of the angle corresponds to the part indicated by the teacher data (for example, the knee). The coefficient between the neurons in the layers is changed so that the value of the angle of the corresponding neuron) approaches the value of the angle which is the correction amount of the teacher data. In this way, from the next time onward, it will be possible to output correction site candidates and their angles that are close to the teacher data.

The learning means 60 is made to learn the correction site as the knee and the correction amount ACn + α as teacher data for the comparison result of ACs, ACn, ACe, ACa, and ACb in FIG. Specifically, by changing the coefficient between the neurons in the input layer and the intermediate layer based on this data set, the output result of the learning means 60 is brought closer to the teacher data, and a more accurate correction site / correction amount is calculated. To do. The same applies to B users, and when the instructor determines that the part to be corrected is the ankle, the comparison result of BCs, BCn, BCe, BCa, BCb is compared with BCs, BCn, BCe, BCa, BCb. , The correction site is the ankle, and the teacher data of the correction amount BCa + β is learned as one data set. Further, as the learning means 60, by repeating such learning, the correction amount calculation means 60a (see FIG. 9) in which the neural network is a trained model can be obtained. The correction content learning device 100 according to the present embodiment gives the comparison result of the user feature data and the model feature data obtained from the input moving image and the teacher data to the neural network as a learning set, and the layers of the neural network. Since the coefficient between the neural networks in the above is changed, it becomes possible to efficiently learn the correction contents.

<3. Motion correction device>
Next, the processing operation of the motion correction device having the neural network provided with the trained model will be described. FIG. 9 is a functional block diagram of the motion correction device 200 according to the present embodiment. In FIG. 9, 10 is a moving image data input means, 20 is a feature data acquisition means, 30 is a model feature data storage means, 40 is a feature data comparison means, 60a is a correction amount calculation means, and 70 is a correction content transmission means. The same reference numerals as those of the correction content learning device 100 shown in FIG. 6 are designated by the same reference numerals, and detailed description thereof will be omitted.

The correction amount calculation means 60a includes a learned model obtained by the learning means 60 performing machine learning in the correction content learning device 100. That is, the correction amount calculation means 60a is a neural network in which learning is performed by changing the coefficient between neurons using the comparison result of the feature data comparison means 40 and the teacher data of the teacher data storage means 50 as a learning set. It is equipped with. As a result, the correction amount calculation means 60a calculates the corresponding correction amount for each input portion by using the comparison result of the feature data comparison means 40.

The correction content transmitting means 70 is a means for transmitting the correction content obtained from the correction amount calculation means 60a to the user. As the correction content transmitting means 70, a device capable of outputting (notifying) information in some form can be used. For example, a speaker that transmits the correction content by sound, a display that transmits the correction content by image / video, a printing machine (printer) that prints the correction content by image or text, and the conversion of the correction content into the action of some force. It can be realized by an actuator or the like.

The motion correction device 200 according to the present embodiment can be realized by a camera constituting the moving image data input means 10, a speaker, a display, a printing machine, an actuator and the like constituting the correction content transmitting means 70, and a general-purpose computer. .. The hardware configuration of the main part of the motion correction device 200 is as shown in FIG. 7, similar to the correction content learning device 100.

The feature data acquisition means 20, the model feature data storage means 30, the feature data comparison means 40, and the correction amount calculation means 60a are realized by the CPU 1 executing a program stored in the storage device 3. That is, the computer executes the contents of each means according to a dedicated program. The computer used in the motion correction device 200 also means a device that has an arithmetic processing unit such as a CPU and is capable of data processing, and is not only a general-purpose computer such as a personal computer but also a portable terminal such as a tablet terminal or a smartphone. Can also be used. The moving image data input means 10 is realized by connecting the cameras constituting the moving image data input means 10 to the data input / output I / F (interface) 5. The teacher data storage means 50 is realized by the storage device 3. The correction content transmission means 70 is realized by connecting a speaker, a display, a printing machine, an actuator, or the like to a data input / output I / F (interface) 5.

Next, the processing operation of the motion correction device 200 shown in FIG. 9 will be described. FIG. 10 is a flowchart showing the processing operation of the motion correction device 200. First, the moving image data is input from the moving image data input means 10 (step S11). Specifically, as in step S10 in the correction content learning device 100, the user's motion is photographed by a camera capable of photographing a moving image, and moving image data is obtained.

Next, the feature data acquisition means 20 acquires feature data from the captured moving image data (step S21). Similarly to step S20 in the correction content learning device 100, the motion correction device 200 also acquires the feature data in two steps. Therefore, first, the skeleton data is acquired from the moving image data.

Then, the feature data is acquired from the skeleton data. That is, as in step S20 in the correction content learning device 100, as feature data, in the present embodiment, a portion and an angle generated by an operation related to the portion are acquired. Since the shoulders, knees, elbows, and ankles may differ on the left and right, it is also possible to obtain the left and right angles for each. However, also in the motion correction device 200, the shoulder, knee, elbow, and ankle are represented as either the left or right as predetermined parts, and five parts of the shoulder, knee, elbow, ankle, and spine are extracted. There is. In addition, 9 places of right shoulder, left shoulder, right knee, left knee, right elbow, left elbow, right ankle, left ankle, and spine may be extracted.

Next, specifically, in the same manner as in step S30 in the correction content learning device 100, the feature data comparison means 40 compares the model feature data with the user feature data (step S31). Specifically, the comparison result, which is the difference between the user feature data and the model feature data, is acquired as the feature data which is a combination of each part and the angle. Therefore, even in step S31, feature data corresponding to each site can be obtained as a comparison result.

Next, the correction amount calculation means 60a calculates the correction amount (step S41). As described above, the correction amount calculation means 60a includes a trained model obtained by performing machine learning by the learning means 60 having a deep neural network structure, and the input layer has the number of parts (number of feature data). ) And the same number (5 in this embodiment) are prepared. Then, the angle, which is a feature amount, is input. Further, the same number of output layers as the number of parts (number of feature data) is prepared, and the angle which is the correction amount for each part is output. Then, the part where the correction amount is maximum is set as the correction part, and the correction amount is output together with the correction amount.

Then, the correction content transmission means 70 transmits the correction content to the operating user according to the correction data (step S70). Specifically, using the correction data expressing the correction content, the correction content is transmitted to the user in an embodiment corresponding to the specific device that realizes the correction content transmission means 70.

As described above, the motion correction device 200 acquires the feature data from the moving image data obtained by photographing the user's motion, creates the correction data, and outputs the correction data to the correction content transmitting means 70 to the user. It is possible to convey the content of correction in real time.

<4. Application example>
Next, specific application examples of the correction content learning device 100 and the motion correction device 200 will be described. Since the correction content learning device 100 and the motion correction device 200 have a common part, they can be realized as one system. FIG. 11 is a diagram showing an application example of the correction content learning device 100 and the motion correction device 200. The movements performed by the user occur in various situations such as daily life, exercise, and competition, but the example of FIG. 11 shows an example applied to a treadmill for running.

When the system shown in FIG. 11 is used as the correction content learning device 100, when the user runs on the treadmill, the camera, which is the moving image data input means 10, captures the user and inputs the moving image data. After the feature data is acquired from the moving image data as described above, the correction content is determined. On the other hand, for example, a running coach who is an instructor looks at the running of the user, determines the correction content, and stores it as teacher data in the teacher data storage means 50. Further, as described above, the learning means 60 incorporating the teacher data performs learning by the function as the correction content learning device 100.

When the system shown in FIG. 11 is used as the motion correction device 200, when the user runs on the treadmill, the camera, which is the moving image data input means 10, captures the user and inputs the moving image data. After the feature data is acquired from the moving image data as described above, the correction content is determined. Then, the correction content is transmitted to the user in real time from the speaker, the display, the printing machine, and the actuator, which are the correction content transmission means 70.

In the example of FIG. 11, the speaker outputs a voice "strongly kick the left foot" to the user, the display displays "strongly kick the left foot", and the printing machine shows how the left foot is kicked. Is printed and vibration is transmitted to the actuator mounted on the user's calf. In this way, when the user starts running on the treadmill, the correction contents are transmitted in real time in a plurality of modes, and the correction can be performed on the spot. In the example of FIG. 11, four devices of a speaker, a display, a printing machine, and an actuator are used as the correction content transmitting means 70, but if there is one or more, the number and types can be changed. be.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above embodiment and can be modified in various ways. For example, in the above embodiment, in the computer, the CPU executes the program to realize each of the above means, but each of the above means may be incorporated into the hardware as an arithmetic circuit. In this case, for example, by incorporating it into a display device such as a liquid crystal display, it is possible to display the input original image with high image quality.

1 ... CPU (Central Processing Unit)
2 ... RAM (Random Access Memory)
3 ... Storage device 4 ... Instruction input I / F
5 ... Data input / output I / F
6 ... Display unit 10 ... Video data input means 20 ... Feature data acquisition means 30 ... Modeler feature data storage means 40 ... Feature data comparison means 50 ... Teacher data storage means 60 ...・ ・ Learning means 60a ・ ・ ・ Correction amount calculation means 70 ・ ・ ・ Correction content transmission means 100 ・ ・ ・ Correction content learning device 200 ・ ・ ・ Motion correction device

Claims (8)

Video data input means for inputting video data of a moving human body,
A feature data acquisition means for acquiring feature data expressing motion features from the input video data as user feature data, and
A modeler feature data storage means that stores the feature data of the modeler as modeler feature data,
A teacher data storage means that stores the feature data indicating the content to be corrected with respect to the comparison result by the feature data comparison means for comparing the user feature data with the model feature data and the feature data comparison means as teacher data. When,
It has a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, the input layer includes means for capturing the comparison result of the feature data comparison means, and the output layer is a number corresponding to the comparison result. A corrective content learning device comprising the neurons of the above and comprising a learning means for changing the coefficient between the neurons in the layers so as to change the value of the neurons corresponding to the teacher data in the output layer. The correction content learning device according to claim 1, wherein the feature data includes a part in the human body and a correction amount corresponding to the part. The correction content learning device according to claim 2, wherein the correction amount is an angle at the site. The correction content learning device according to claim 3, wherein the angle at the portion is an angle formed by a portion connected to the portion in a predetermined state. The correction content learning device according to claim 3, wherein the angle at the site is an angle formed at different time points by the connecting sites that are connected to the site. Video data input means for inputting video data of a moving human body,
A feature data acquisition means for acquiring feature data expressing motion features from the input video data as user feature data, and
A modeler feature data storage means that stores the feature data of the modeler as modeler feature data,
It has a feature data comparison means for comparing the user feature data with the model feature data, a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, and the input layer is the feature data comparison means. The output layer is provided with a means for capturing the comparison result, the output layer is provided with a number of neurons corresponding to the comparison result, and the correction amount calculation means for outputting the correction amount in the output layer is provided.
The correction content transmission means for transmitting the correction content to the operating user according to the output correction amount, and the correction content transmission means.
A motion correction device. Computer,
Feature data acquisition means that acquires feature data that expresses the characteristics of operation as user feature data from video data,
Modeler feature data storage means that stores the feature data of the model as model feature data,
Teacher data storage that stores the feature data indicating the content to be corrected with respect to the feature data comparison means for comparing the user feature data with the model feature data and the comparison result by the feature data comparison means as teacher data. means,
It has a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, the input layer includes means for capturing the comparison result of the feature data comparison means, and the output layer is a number corresponding to the comparison result. A program for providing a neuron of the above and functioning as a learning means for changing a coefficient between neurons between layers so as to change the value of the neuron corresponding to the teacher data in the output layer. Computer,
Feature data acquisition means that acquires feature data that expresses the characteristics of operation as user feature data from video data,
Modeler feature data storage means that stores the feature data of the model as model feature data,
A feature data comparison means for comparing the user feature data with the model feature data,
It has a neural network configuration having an input layer, a plurality of intermediate layers, and an output layer, the input layer includes means for capturing the comparison result of the feature data comparison means, and the output layer is a number corresponding to the comparison result. A correction amount calculation means, which is equipped with the above-mentioned neurons and outputs a correction amount in the output layer.
A program for functioning as a correction content transmission means for transmitting correction content to a user in operation according to the output correction amount.

Images