JP2023183624A - Information processing device, information processing method and information processing program - Google Patents

Abstract

To enable an arbitrary exercise video to be generated from an exercise video including movement of a user's body.SOLUTION: An information processing device according to the present application includes: an acquisition part for acquiring a machine learning model including an encoder for generating feature information showing features of time series data on the basis of time series data about a change in a joint angle, and a decoder for generating time series data on the basis of the feature information; and a generation part for generating second time series data about a change in a joint angle of a user from first time series data about the change in the joint angle of the user by using a latent space of the machine learning model.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, in the field of machine learning, techniques related to autoencoders have been known. An autoencoder is comprised of an encoder, which is a neural network that converts target information into a latent representation (also called a feature representation), and a decoder, which is a neural network that restores target information from the latent representation. Further, a technique related to VAE (Variational Autoencoder), which is derived from autoencoder, is known. VAE trains a neural network so that the probability distribution of latent representations follows a normal distribution. For example, using VAE, image data corresponding to handwritten numbers "0" to "9" (hereinafter also referred to as handwritten number images) and label data that are the correct answers to the numbers written in the image are generated. Convert the set of datasets into latent representations and map the latent representations to the latent space. Then, an image is generated while continuously changing the latent variables on the latent space of VAE. There is a known technique for generating a handwritten numeral image in which the numerals drawn in the image are continuously changed.

Diederik P. Kingma, 3 others, “Semi-Supervised Learning with Deep Generative Models”, [online], June 2014, [Retrieved May 31, 2020], Internet <URL: https://arxiv.org /abs/1406.5298v1＞

Furthermore, in recent years, research has been actively conducted on entertainment computing, which is information and communication technology for realizing mental enrichment such as stress relief and mental healing. For example, there is a need for a technology that can generate arbitrary exercise images from exercise images that include the movement of a user's body.

An object of the present application is to provide an information processing device, an information processing method, and an information processing program that can generate an arbitrary exercise image from an exercise image that includes the movement of a user's body.

An information processing device according to the present application includes an encoder that generates feature information indicating characteristics of the time series data based on time series data regarding changes in joint angles, and a decoder that generates the time series data based on the feature information. an acquisition unit that acquires a machine learning model including: a first time series data regarding changes in the user's joint angles from first time series data regarding changes in the user's joint angles using the latent space of the machine learning model; and a generation unit that generates time series data of No. 2.

The acquisition unit acquires the machine learning model trained so that the probability distribution of the feature information follows a normal distribution, and the generation unit maps the first time series data to the latent space and generates the latent space. The latent variable in the space is changed from a value having first feature information corresponding to the first time series data mapped to the latent space to a value having second feature information, and the latent variable after the change is changed. The second time series data is generated based on the second feature information corresponding to the value of the variable.

The time series data includes attribute information corresponding to the time series data, and the acquisition unit is trained such that a probability distribution of the feature information indicating the characteristics of the time series data including the attribute information follows a normal distribution. the machine learning model, and the generation unit converts the latent variable in the latent space from the value having the first feature information corresponding to the first attribute information to the second characteristic information corresponding to the second attribute information. The second time series data is generated based on the second feature information corresponding to the value of the latent variable after the change.

The machine learning model further includes a posture estimation model learned to estimate the posture of the object from an image including the object, and the generation unit uses the posture estimation model to estimate the posture of the user's body. The coordinates of the joint points of the user are estimated from a first motion image including movement, and the first time series data is generated based on the estimated coordinates of the joint points.

The generation unit generates, based on the generated second time-series data, a second exercise image including a movement of the user's body corresponding to the second time-series data.

The machine learning model further includes an image transformation model trained to generate a person image of the subject corresponding to the joint points from a joint image including the joint points of the subject, and the generation unit is configured to perform the image transformation. The second motion image is generated from the second time series data using the model.

The information processing device according to the present application includes: an acquisition unit that acquires time series data regarding changes in joint angles; an encoder that generates feature information indicating characteristics of the time series data based on the time series data; and a model generation unit that generates a machine learning model including a decoder that generates the time series data based on the time series data.

The model generation unit trains the machine learning model so that a degree of similarity between the time series data input to the encoder and the time series data output from the decoder exceeds a predetermined threshold.

The model generation unit trains the machine learning model so that the probability distribution of the feature information follows a normal distribution.

The acquisition unit acquires the time series data including attribute information corresponding to the time series data, and the model generation unit generates a probability distribution of the feature information indicating the characteristics of the time series data including the attribute information. The machine learning model is trained to follow a normal distribution.

The model generation unit classifies the feature information into clusters according to the attribute information.

The attribute information may include the type of body movement of the subject included in the exercise video corresponding to the time series data, the proficiency level of the subject's body movement, the characteristics of the subject's body movement, or This is information indicating the subject's biological information.

The information processing method according to the present application is an information processing method realized by a program executed by an information processing device, and the information processing method generates feature information indicating characteristics of the time series data based on time series data regarding changes in joint angles. an acquisition step of acquiring a machine learning model including an encoder and a decoder that generates the time series data based on the feature information; The method includes a generation step of generating second time-series data regarding changes in joint angles of the user from the first time-series data.

The information processing method according to the present application is an information processing method realized by a program executed by an information processing device, and includes an acquisition step of acquiring time series data regarding changes in joint angles, and a step of acquiring time series data regarding changes in joint angles; The method includes a model generation step of generating a machine learning model including an encoder that generates feature information indicating characteristics of series data, and a decoder that generates the time series data based on the feature information.

The information processing program according to the present application includes an encoder that generates feature information indicating characteristics of the time series data based on time series data regarding changes in joint angles, and a decoder that generates the time series data based on the feature information. , an acquisition procedure for acquiring a machine learning model including: a first time series data regarding changes in the joint angles of the user from first time series data regarding changes in the joint angles of the user using the latent space of the machine learning model; A computer is made to execute the generation procedure for generating time series data in step 2.

The information processing program according to the present application includes an acquisition procedure for acquiring time series data regarding changes in joint angles, an encoder that generates feature information indicating characteristics of the time series data based on the time series data, and an encoder that generates feature information indicating the characteristics of the time series data based on the time series data. a decoder that generates the time-series data based on the data, and a model generation procedure that generates a machine learning model.

According to one aspect of the embodiment, it is possible to generate an arbitrary exercise image from an exercise image that includes the movement of the user's body.

FIG. 1 is a diagram for explaining an overview of information processing according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to an embodiment. FIG. 3 is a diagram illustrating a configuration example of a generation device according to an embodiment. FIG. 4 is a flowchart showing an information processing procedure by the generation device according to the embodiment. FIG. 5 is a diagram illustrating a configuration example of an information processing device according to an embodiment. FIG. 6 is a diagram for explaining an example of the latent space according to the embodiment. FIG. 7 is a flowchart showing an information processing procedure by the information processing apparatus according to the embodiment. FIG. 8 is a diagram for explaining an example of a latent space according to a modification. FIG. 9 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Further, in each of the embodiments below, the same parts are given the same reference numerals, and redundant explanations will be omitted.

(Embodiment)
[1. Overview of information processing]
FIG. 1 is a diagram for explaining an overview of information processing according to an embodiment. In FIG. 1, it is assumed that the information processing apparatus 100 according to the embodiment realizes information processing according to the embodiment. In FIG. 1, the information processing device 100 uses the latent space of the trained machine learning model M1 to create a dance image of the user dancing based on a dance video G1 of the user dancing a jazz dance. A case will be described in which a dance video G2 in which the genre is changed from jazz dance to hip-hop dance is generated.

Specifically, the information processing device 100 uses the posture estimation model to estimate the coordinates of the joint points of the user captured in the dance video G1 for each frame. Next, the information processing device 100 calculates the amount of change in the joint angle of each joint for each frame (hereinafter also referred to as first time series data of the amount of change in joint angle) from the estimated coordinates of each joint point. . Here, the posture estimation model is a machine learning model trained to estimate the posture of a target object from an image including the target object. Subsequently, the information processing device 100 inputs the first time series data of the initial joint angle of each joint and the amount of change in the joint angle of each joint as input information to the machine learning model M1, and inputs the first characteristic information. generate. The information processing device 100 maps the generated first feature information to the latent space of the machine learning model M1.

The machine learning model M1 in this embodiment is a machine learning model trained in advance to map feature information corresponding to time-series data onto a latent space. Specifically, the machine learning model M1 may be a neural network trained in advance so that the probability distribution of feature information corresponding to time-series data follows a normal distribution. For example, the machine learning model M1 may be VRAE (Variational Recurrent Autoencoders), which is a machine learning model that applies RNN (Recurrent Neural Network) to VAE (Variational Autoencoder). VARIATIONAL RECURRENT AUTO-ENCODERS”, [online], December 2014, [searched on May 31, 2020], Internet <URL: https://arxiv.org/abs/1412.6581v1>). In FIG. 1, a case will be described in which the machine learning model M1 is VRAE.

Here, VAE (Variational Autoencoder), which is the basis of VRAE, will be explained in detail. VAE is known as a type of generative model that generates images. For example, a data set includes image data corresponding to handwritten numbers "0" to "9" (hereinafter also referred to as handwritten number images) and label data that is the correct answer to the number written in the image. The VAE is trained using the data as learning data (also referred to as training data). Specifically, we transform the dataset into latent representations and train the VAE to map the latent representations to the latent space. Here, VAE is characterized in that a neural network is trained so that the probability distribution of latent expressions follows a normal distribution. Therefore, in the latent space of VAE, in the case of similar images, that is, handwritten numeric images, latent expressions corresponding to handwritten numeric images in which the same number is drawn tend to be mapped to close positions on the latent space. Furthermore, the fact that latent representations of handwritten digit images with the same digits are mapped to close positions in the latent space means that clusters of latent representations corresponding to handwritten digit images with each digit are generated. corresponds to For example, a cluster of latent expressions corresponding to an image of handwritten numerals with the number "1" drawn on them (hereinafter referred to as a cluster of "1"), a cluster of latent expressions corresponding to an image of handwritten numerals with the number "2" drawn on them. (hereinafter referred to as a cluster of "2"), and a cluster of latent expressions corresponding to a handwritten digit image with the number "3" drawn (hereinafter referred to as a cluster of "3"), corresponding to each number as in... Each cluster of latent representations is generated. For example, assume that cluster "2" is mapped to the closest distance from cluster "1" in the latent space. Further, it is assumed that the cluster "3" is mapped next to the cluster "2" at a distance close to the cluster "1". At this time, the latent variable in the latent space of VAE is changed continuously, for example, from the average value of the cluster "1" to the average value of the cluster "3" in the order of "1" → "2" → "3". Generate images while As a result, it is possible to generate a handwritten number image in which the numbers drawn in the image are changed continuously from "1" to "3", such as "1" → "2" → "3".

Returning to the explanation of FIG. In the example shown in Figure 1, a data set of a dance video and label data indicating the type of dance included in the dance video (for example, a genre such as jazz dance, ballet dance, hip-hop dance, etc.) is used as learning data in advance. Train the machine learning model M1. Specifically, the data set is converted into feature information (corresponding to the latent expression described above), and the machine learning model M1 is trained to map the feature information to the latent space. Here, the machine learning model M1, which is VRAE, trains the neural network so that the probability distribution of feature information follows a normal distribution, similarly to VAE. Therefore, in the latent space of the machine learning model M1, feature information corresponding to similar dance videos, that is, dance videos including the same type of dance, tends to be mapped to close positions on the latent space. Further, the fact that feature information of dance videos including the same type of dance is mapped to close positions in the latent space corresponds to the generation of clusters of latent expressions corresponding to each type of dance video. In FIG. 1, a cluster of feature information corresponding to a dance video of jazz dance (hereinafter referred to as a cluster of jazz dance), a cluster of feature information corresponding to a dance video of ballet dance (hereinafter referred to as a cluster of ballet dance), and a cluster of feature information corresponding to a dance video of ballet dance (hereinafter referred to as a cluster of hip-hop dance) are shown. Clusters of feature information corresponding to each type of dance video are generated, such as a cluster of feature information corresponding to a dance video (hereinafter referred to as a cluster of hip-hop dance), and so on. It also shows how clusters of feature information corresponding to each type of dance video are mapped onto the latent space. In addition, the information processing device 100 uses a known clustering technique to calculate the type of dance included in the dance video (for example, jazz dance, ballet dance, hip-hop dance etc.) may be classified into clusters according to the following. In addition, for example, when the dance video G1 is a jazz dance video, the information processing device 100 maps the first feature information to the position of the jazz dance cluster in the latent space. A point P1 shown in FIG. 1 indicates the position of the first feature information mapped to the latent space.

Subsequently, the information processing device 100 changes the latent variable in the latent space from a value having the first feature information to a value having the second feature information. For example, the information processing device 100 changes the latent variable from a value having first feature information belonging to the jazz dance cluster to a value having second feature information belonging to the hip-hop cluster. Point P2 shown in FIG. 1 indicates the position of the second feature information mapped to the latent space. For example, a ballet dance cluster is mapped to the closest distance from a jazz dance cluster on the latent space. Furthermore, the hip-hop dance cluster is mapped next to the ballet dance cluster and the closest distance from the jazz dance cluster. At this time, the information processing device 100 changes the latent variable from a value having first feature information belonging to the jazz dance cluster to a value having second feature information located within a predetermined range from the average value of the hip hop dance cluster. The value may be changed continuously. For example, the information processing device 100 may select a value having the first characteristic information belonging to the jazz dance cluster→a value having the average value of the ballet dance cluster→a value located within a predetermined range from the average value of the hip hop dance cluster. The latent variable may be changed continuously, such as the value having characteristic information of 2. As a result, the information processing device 100 displays a dance video in which the genre of dance performed by the user is changed continuously from jazz dance to hip hop dance, for example, from jazz dance to ballet dance to hip hop dance. can be generated.

Next, the information processing device 100 calculates the amount of change in the joint angle of each joint (hereinafter referred to as the amount of change in the joint angle) for each frame based on the second feature information corresponding to the value of the latent variable after the change. (also referred to as second time series data). For example, the information processing device 100 outputs second time series data of the amount of change in the joint angle of each joint as output information of the machine learning model M1, and outputs second time series data of the amount of change in the joint angle of each joint. Generate data. Subsequently, the information processing device 100 calculates each joint for each frame based on the second time series data of the amount of change in the joint angle of each joint output from the machine learning model M1 and the initial angle of the joint angle of each joint. Calculate the joint angle of. Subsequently, the information processing device 100 calculates the coordinates of each joint point for each frame from the joint angle of each joint for each frame.

Subsequently, the information processing device 100 uses the image conversion model to convert each frame including the joint points corresponding to the calculated coordinates of each joint point into a dance video G2 including the user dancing. Here, the image conversion model is a machine learning model trained to generate a human image of the subject corresponding to the joint points from a joint image including the subject's joint points. Here, the second feature information is mapped to the position of the hip-hop dance cluster in the latent space, so the dance video G2 corresponding to the second feature information shows the user dancing the hip-hop dance. Compatible with video.

As described above, the information processing device 100 uses the trained machine learning model M1 to generate the first feature information from the dance video G1, and maps the first feature information to the latent space. Subsequently, the information processing device 100 changes the latent variable in the latent space from a value having the first feature information to a value having the second feature information. Subsequently, the information processing device 100 generates the dance video G2 based on the second feature information corresponding to the value of the latent variable after being changed. In this way, by using the latent space of the machine learning model M1, the information processing device 100 can change the dance video G1 of the user into the dance video G2 corresponding to an arbitrary value on the latent space. That is, the information processing device 100 uses the latent space of the machine learning model M1 to realize morphing from the user's dance video G1 to the dance video G2 obtained by processing the dance video G1. For example, the information processing device 100 can realize morphing from a jazz dance dance video G1 to a hip-hop dance video G2 by using latent spaces classified according to the type of dance video. That is, the information processing device 100 can generate a new dance video based on the user's dance video according to the attribute of the dance video (for example, the type of dance) desired by the user. Therefore, the information processing device 100 can generate any dance video from the user's dance video. Further, the information processing device 100 can provide the user with a new dance video according to the dance video attribute (for example, type of dance) desired by the user. That is, the information processing device 100 can provide new entertainment to users. Therefore, the information processing device 100 can provide spiritual enrichment to the user.

[2. Information processing system configuration]
FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes a generation device 20 and an information processing device 100. The generation device 20 and the information processing device 100 are connected to be able to communicate with each other by wire or wirelessly via various communication networks. Note that the information processing system 1 shown in FIG. 2 may include any number of generation devices 20 and any number of information processing devices 100.

The generation device 20 is a server device that generates the machine learning model M1 described in FIG. 1. When generating the machine learning model M1, the generation device 20 distributes information regarding the generated machine learning model M1 to the information processing device 100 of each user.

The information processing device 100 is an information processing device that implements the information processing described in FIG. 1. Specifically, the information processing device 100 may be a terminal device such as a smartphone used by a user. The information processing device 100 acquires the machine learning model M1 from the generation device 20 and implements the information processing described in FIG. 1.

[3. Configuration of generation device]
FIG. 3 is a diagram showing a configuration example of the generation device 20 according to the embodiment. The generation device 20 includes a communication section 21, a storage section 22, and a control section 23.

(Communication Department 21)
The communication unit 21 is realized by a NIC (Network Interface Card), an antenna, or the like. The communication unit 21 is connected to various networks by wire or wirelessly, and transmits and receives information to and from the information processing device 100, for example.

(Storage unit 22)
The storage unit 22 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 22 stores various programs (an example of an information processing program). Furthermore, the storage unit 22 stores information regarding the machine learning model M1 generated by the model generation unit 232.

(Control unit 23)
The control unit 23 is a controller, and includes various programs (an example of an information processing program) stored in a storage device inside the generation device 20 by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). (equivalent to ) is realized by executing the RAM as a work area. Further, the control unit 23 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 23 has an acquisition unit 231, a model generation unit 232, and a distribution unit 233 as functional units, and may realize or execute the information processing operation described below. Note that the internal configuration of the control unit 23 is not limited to the configuration shown in FIG. 3, and may be any other configuration as long as it performs information processing to be described later. Further, each functional unit indicates a function of the control unit 23, and does not necessarily have to be physically distinct.

(Acquisition unit 231)
The acquisition unit 231 acquires learning data used for learning the machine learning model M1 by the model generation unit 232. Specifically, the acquisition unit 231 may acquire, as learning data, information in which attribute information indicating the attributes of the dance video and the dance video are linked manually in advance. The attribute information may be, for example, the type of dance performed by a person (hereinafter also referred to as a target person) included in the dance video. More specifically, the acquisition unit 231 acquires, as learning data, a label indicating the type of dance included in the dance video (for example, a label indicating the dance genre such as jazz dance, ballet dance, hip-hop dance, etc.) and the dance video. You may obtain a dataset consisting of a combination of For example, the acquisition unit 231 may acquire the learning data from a terminal device used by the creator who created the learning data.

The acquisition unit 231 also acquires time-series data of the amount of change in joint angle. Specifically, the acquisition unit 231 may estimate the coordinates of the joint points of the subject included in the dance video of the learning data for each frame using the posture estimation model. Subsequently, the acquisition unit 231 may calculate the amount of change in the joint angle of each joint for each frame from the estimated coordinates of each joint point. For example, the acquisition unit 231 may calculate the amount of change in the joint angles of nine joint points (neck, both shoulders, both elbows, both hips, and both knees) for each frame from the estimated coordinates of each joint point.

For example, the acquisition unit 231 may calculate the joint angle of each joint for each frame from the coordinates of the three joint points using the second cosine theorem and the inverse triangle theorem. As an example, a case will be described in which the acquisition unit 231 calculates the joint angle of the right knee. The acquisition unit 231 obtains the coordinates (r_hip(x), r_hip(y)) of the right hip joint point (hereinafter referred to as point B) and the right knee joint point (hereinafter referred to as point C) in the subject's skeletal model. From the coordinates (r_knee (x), r_knee (y)) of the joint point of the right ankle (hereinafter referred to as point A) (r_ankle (x), r_ankle (y)), The joint angle of the right knee may be calculated. For example, the acquisition unit 231 calculates the square of the length of each side of triangle ABC as follows: (BC) ² = (r_knee(x) - r_hip(x)) ² + (r_knee(y) - r_hip(y)) ² , (CA) ² = (r_knee (x) - r_ankle (x)) ² + (r_knee (y) - r_ankle (y)) ² , (AB) ² = (r_ankle (x) - r_hip (x)) ² +(r_ankle(y)−r_hip(y)) Calculated by ² . Subsequently, the acquisition unit 231 calculates that if the angle of the vertex C of the triangle ABC (that is, the joint angle of the right knee) is expressed by θ, then (AB) ² = (BC) ² + (CA) ² from the second cosine theorem. -2(BC)*(CA)cosθ holds, so using the inverse trigonometric theorem, θ=cos^(-1)((BC) ² +(CA) ² -(AB) ² )/2(BC) *Calculated using (CA). The acquisition unit 231 may calculate the joint angle of each joint for each frame from the coordinates of the three joint points in the same manner as when calculating the joint angle of the right knee. Next, the acquisition unit 231 obtains the joint angle of each joint for each frame by calculating the difference between the joint angle of each joint in a predetermined frame and the joint angle of each joint in the frame following the predetermined frame. The amount of change (hereinafter also referred to as time series data of the amount of change in joint angle) may be calculated. In the above example, the acquisition unit 231 selects the joint points of three consecutive parts of the skeletal model, such as the right hip, right knee, and right ankle, as the three joint points, and calculates the angle formed by these joint points as the joint angle. Although a case has been described in which the calculation is performed as follows, the present embodiment is not limited to this. That is, in one embodiment, the acquisition unit 231 may calculate an angle formed by arbitrary three parts in the skeletal model as a joint angle.

(Model generation unit 232)
The model generation unit 232 is an encoder that generates feature information indicating the characteristics of the time series data based on the time series data (hereinafter also referred to as time series data) of the amount of change in the joint angle of each joint acquired by the acquisition unit 231. A machine learning model M1 is generated that includes: and a decoder that generates time series data based on the feature information. Specifically, the machine learning model M1 may be VRAE, which is a machine learning model in which RNN is applied to VAE.

More specifically, the model generation unit 232 may generate feature information from time-series data using the encoder of the machine learning model M1. Here, the feature information may be a vector with a lower dimension than the time series data. The model generation unit 232 uses the encoder of the machine learning model M1 to dimensionally compress the time series data into feature information. Subsequently, the model generation unit 232 may generate time series data from the feature information using the decoder of the machine learning model M1. Subsequently, the model generation unit 232 may cause the machine learning model M1 to learn such that the degree of similarity between the time series data input to the encoder and the time series data output from the decoder exceeds a predetermined threshold. For example, the model generation unit 232 uses backpropagation or the like to generate the machine learning model M1 until the similarity between the time series data input to the encoder and the time series data output from the decoder exceeds a predetermined threshold. The encoder and decoder may be trained separately. Furthermore, the model generation unit 232 may cause the machine learning model M1 to learn so that the probability distribution of the feature information follows a normal distribution. For example, the model generation unit 232 may make the encoder learn to output the mean μ and variance σ of the normal distribution, assuming that the probability distribution of the feature information follows a normal distribution. The model generation unit 232 also samples feature information according to a normal distribution N(μ, σ) based on the mean μ and variance σ output from the encoder, and restores time series data from the sampled feature information. You can train the decoder. In this way, the model generation unit 232 may generate the trained machine learning model M1.

Further, the acquisition unit 231 may acquire time series data including attribute information corresponding to the time series data. For example, the acquisition unit 231 may acquire a data set of a dance video and a label indicating the type of dance corresponding to the dance video as time series data including attribute information corresponding to the time series data. Furthermore, the model generation unit 232 may cause the machine learning model M1 to learn to map feature information indicating features of time-series data including attribute information to the latent space. For example, the model generation unit 232 may cause the machine learning model M1 to learn to map feature information indicating the characteristics of the dataset acquired by the acquisition unit 231 to the latent space. Further, the model generation unit 232 may cause the machine learning model M1 to learn such that the probability distribution of feature information indicating the characteristics of time series data including attribute information follows a normal distribution. For example, the model generation unit 232 may cause the machine learning model M1 to learn such that the probability distribution of feature information indicating the characteristics of the dataset acquired by the acquisition unit 231 follows a normal distribution. Subsequently, the model generation unit 232 may classify the feature information mapped into the latent space of the learned machine learning model M1 into clusters according to the attribute information. For example, the model generation unit 232 may perform clustering by calculating distances between feature information mapped in the latent space using the k-means method. In addition to the k-means method, the model generation unit 232 may use a known clustering technique to classify the feature information mapped in the latent space into clusters according to attributes. For example, the model generation unit 232 may classify the feature information mapped in the latent space into clusters according to the type of dance indicated by the attribute information (for example, types of jazz dance, ballet dance, hip-hop dance, etc.). .

Furthermore, the machine learning model M1 may include a posture estimation model trained to estimate the posture of the target object from an image including the target object. For example, the model generation unit 232 may use a known posture estimation technique to generate a posture estimation model trained to estimate the posture of the subject included in the dance video from the dance video.

Furthermore, the machine learning model M1 may include an image conversion model learned to generate a person image of the subject corresponding to the joint points from a joint image including the subject's joint points. For example, the model generation unit 232 uses a known image conversion model such as Pix2Pix, CYcleGAN, DiscoGAN, UNIT, etc. to generate a human image of the target person corresponding to the joint points from the joint image including the joint points of the target person. An image transformation model may be trained.

(Distribution Department 233)
The distribution unit 233 distributes information regarding the machine learning model M1 generated by the model generation unit 232 to each user's information processing device 100.

[4. Procedure of information processing by generation device]
FIG. 4 is a flowchart showing an information processing procedure by the generation device 20 according to the embodiment. As shown in FIG. 4, the acquisition unit 231 estimates the coordinates of the joint points of the subject included in the dance video using the posture estimation model (step S11). Subsequently, the acquisition unit 231 generates time-series data of the amount of change in joint angle based on the coordinates of each joint point (step S12). Subsequently, the model generation unit 232 generates feature information from the time series data using the encoder of the machine learning model M1 (step S13). Subsequently, the model generation unit 232 generates time series data from the feature information using the decoder of the machine learning model M1 (step S14). Next, the model generation unit 232 trains the machine learning model M1 so that the degree of similarity between the time series data input to the encoder and the time series data output from the decoder exceeds a predetermined threshold (step S15). .

[5. Configuration of information processing device]
FIG. 5 is a diagram illustrating a configuration example of the information processing device 100 according to the embodiment. The information processing device 100 includes a communication section 110, a storage section 120, an input section 130, an output section 140, and a control section 150.

(Communication Department 110)
The communication unit 110 is realized by a NIC, an antenna, or the like. The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from the generation device 20, for example.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various programs (an example of an information processing program). Furthermore, the storage unit 120 stores information regarding the machine learning model M1.

(Input section 130)
The input unit 130 receives various operations from the user. For example, the input unit 130 may receive various operations from the user via a display screen (for example, the output unit 140) using a touch panel function. Further, the input unit 130 may accept various operations from buttons provided on the information processing device 100 or a keyboard or mouse connected to the information processing device 100. For example, the input unit 130 may receive an operation from the user to process the user's characteristic information displayed on the screen.

(Output section 140)
The output unit 140 is a display screen realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. The output unit 140 displays various information under the control of the control unit 150. For example, the output unit 140 may display an image of the feature information mapped to the latent space under the control of the providing unit 153. Note that when a touch panel is employed in the information processing apparatus 100, the input section 130 and the output section 140 are integrated. Furthermore, in the following description, the output unit 140 may be referred to as a screen.

(Control unit 150)
The control unit 150 is a controller, and for example, various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 are executed by a CPU, an MPU, etc. using the RAM as a work area. This is achieved by Further, the control unit 150 is a controller, and is realized by, for example, an integrated circuit such as an ASIC or an FPGA.

The control unit 150 has an acquisition unit 151, a generation unit 152, a provision unit 153, and a reception unit 154 as functional units, and may realize or execute the information processing operation described below. Note that the internal configuration of the control unit 150 is not limited to the configuration shown in FIG. 5, and may be any other configuration as long as it performs information processing to be described later. Further, each functional unit indicates a function of the control unit 150, and does not necessarily have to be physically distinct.

(Acquisition unit 151)
The acquisition unit 151 is a machine learning system that includes an encoder that generates feature information indicating the characteristics of the time series data based on the time series data of the amount of change in joint angle, and a decoder that generates time series data based on the feature information. Obtain model M1. Further, the acquisition unit 151 acquires a machine learning model M1 trained so that the probability distribution of feature information follows a normal distribution. For example, the acquisition unit 151 may acquire the machine learning model M1 that has been trained so that the probability distribution of feature information indicating characteristics of time series data including attribute information follows a normal distribution. Specifically, the acquisition unit 151 may acquire information regarding the learned machine learning model M1 from the generation device 20.

(Generation unit 152)
The generation unit 152 receives dance videos from users. Specifically, the generation unit 152 may receive the user's own dance video (hereinafter also referred to as the user's dance video). For example, the generation unit 152 may receive a user's dance video (hereinafter also referred to as a first dance video) from the user via the input unit 130. For example, the generation unit 152 may receive a video of a user performing a jazz dance as the first dance video.

Subsequently, when receiving the first dance video, the generation unit 152 may use the posture estimation model to estimate the coordinates of each joint point of the user captured in the first dance video for each frame. . Subsequently, the generation unit 152 may calculate the amount of change in the joint angle of each joint for each frame (hereinafter also referred to as first time series data) from the estimated coordinates of each joint point. In this way, the generation unit 152 estimates the coordinates of each joint point of the user from the first dance video using the posture estimation model, and generates the first time series based on the estimated coordinates of the joint points. May generate data.

Subsequently, when generating the first time series data, the generation unit 152 may map the first time series data to the latent space of the machine learning model using the encoder of the machine learning model M1. For example, the generation unit 152 inputs first time series data of the initial joint angle of each joint and the amount of change in the joint angle of each joint as input information to the encoder of the machine learning model M1, and generates first feature information. may be generated. Subsequently, the generation unit 152 may map the generated first feature information onto the latent space of the machine learning model M1. FIG. 6 is a diagram for explaining an example of the latent space according to the embodiment. A point P1 in the left diagram of FIG. 6 indicates the position of the first feature information mapped to the latent space by the generation unit 152. The diagram on the left side of FIG. 6 shows how the first feature information is mapped to the position of the jazz dance cluster by the generation unit 152, since the first dance video received from the user is a jazz dance video.

(Providing Department 153)
The providing unit 153 provides information regarding the latent space of the machine learning model M1 to the user. For example, the providing unit 153 may control the output unit 140 to display information regarding the latent space to which the learned feature information is mapped. In the example shown in the diagram on the left side of FIG. 6, the providing unit 153 generates a latent space indicating how the first feature information generated by the generating unit 152 is mapped to the position of point P1 together with the learned feature information. The output unit 140 may be controlled to display the image G3.

(Reception Department 154)
The reception unit 154 receives operations on the latent space from the user. Specifically, the reception unit 154 may receive an operation for changing a latent variable in the latent space from the user. In the example shown in the diagram on the right side of FIG. 6, the reception unit 154 changes the latent variable in the latent space from the position of the point P1 indicating the value having the first feature information to the point P2 indicating the value having the second feature information. An operation to change the position may be accepted from the user. For example, the receiving unit 154 may accept, via the input unit 130, an operation on the latent space image G3 displayed by the providing unit 153. The reception unit 154 converts the latent variable from the position of a point P1 indicating a value having first characteristic information belonging to the cluster of jazz dance to the position of point P2 indicating a value having second characteristic information belonging to the cluster of hip-hop dance. You may accept an operation from the user to change it to . Further, the receiving unit 154 may receive second feature information corresponding to the value of the latent variable after being changed by the user.

Further, the generation unit 152 may generate second time series data of the amount of change in the joint angle of each joint based on the second feature information received by the reception unit 154. For example, the generation unit 152 changes the latent variable in the latent space from a value having the first feature information to a value having the second feature information based on the second feature information received by the reception unit 154. For example, the generation unit 152 changes the latent variable in the latent space from a value having first feature information corresponding to first attribute information to a value having second feature information corresponding to second attribute information. good. Specifically, for example, the generation unit 152 converts the latent variable into a label indicating that the dance type is hip-hop dance from a value having feature information corresponding to a label indicating that the dance type is jazz dance. may be changed up to a value having characteristic information corresponding to . Subsequently, the generation unit 152 may generate the second time series data using the machine learning model M1 decoder based on the second feature information corresponding to the value of the latent variable after being changed.

Subsequently, the generation unit 152 may calculate the joint angle of each joint for each frame based on the second time series data output from the machine learning model M1 and the initial angle of the joint angle of each joint. Subsequently, the generation unit 152 may calculate the coordinates of each joint point for each frame from the joint angle of each joint for each frame. In this way, the generation unit 152 may generate an image (hereinafter referred to as a joint image) including movement of a joint corresponding to the second time-series data of the amount of change in joint angle of each joint of the user.

Subsequently, the generation unit 152 may generate a second dance video corresponding to the second time series data based on the generated second time series data. Specifically, the generation unit 152 may generate the second dance video from the second time series data using an image conversion model. For example, the generation unit 152 may use an image conversion model to convert each frame of the joint video corresponding to the second time series data into a second dance video including a dancing person. As shown in FIG. 6, the point P2 indicating the second feature information is mapped to the position of the hip-hop dance cluster in the latent space, so the second dance video corresponding to the second feature information is This is a video of hip hop dance. In this way, the generation unit 152 may generate a dance video of the user's hip-hop dance (second dance video) from a dance video of the user's jazz dance (first dance video).

[6. Information processing procedure by information processing device]
FIG. 7 is a flowchart showing an information processing procedure by the information processing apparatus 100 according to the embodiment. As shown in FIG. 7, the acquisition unit 151 acquires a pre-trained machine learning model M1 (step S101). The generation unit 152 uses the machine learning model M1 acquired by the acquisition unit 151 to map the first feature information corresponding to the user's dance video onto the latent space (step S102). The providing unit 153 provides the user with information on the latent space in which the first feature information is mapped (step S103). The reception unit 154 receives an operation for changing a latent variable in the latent space from the user (step S104). The generation unit 152 generates a new dance video based on the second feature information corresponding to the changed latent variable value accepted by the reception unit 154 (step S105). The providing unit 153 provides the new dance video generated by the generating unit 152 to the user (step S106).

[7. Modified example]
The processing according to the embodiment described above may be implemented in various different forms other than the embodiment described above.

[7-1. About latent space]
In the embodiment described above, a case has been described in which the attribute information is the type of dance performed by the person included in the dance video, but the attribute information is not limited to the type of dance. For example, the attribute information may be information indicating the dancing proficiency level of the person included in the dance video corresponding to the time-series data, the characteristics of the dance, or the biological information of the dancing person. For example, the attribute information may be a score indicating dance proficiency. For example, to obtain a score that indicates dance proficiency, professional dancers evaluate the skill of the dance included in a dance video on a five-point scale from "1" to "5."5" (for example, the better the dancer is, the higher the value is assigned). Further, the attribute information may be a score indicating the characteristics of the dance. For example, similar to scores indicating dance proficiency, scores indicating dance characteristics are obtained by professional dancers numerically evaluating the dance characteristics (for example, presence or absence of sharpness in the dance) included in the dance video. may be given a numerical value corresponding to the evaluation. Further, the attribute information may be a numerical value indicating biological information of the person dancing. For example, biometric information (for example, muscle mass) of a dancing person is acquired using a biosensor before (or even during) a dance video is captured. Then, numerical values of biological information acquired from a biological sensor may be added to the dance video.

FIG. 8 is a diagram for explaining an example of a latent space according to a modification. The diagram on the left side of FIG. 8 shows a case where the attribute information associated with the feature information is dance proficiency. For example, the generation unit 152 converts the latent variable in the latent space from a value having first feature information corresponding to first attribute information (for example, a label indicating that one is bad at dancing) to a value having second attribute information (for example, The value may be changed to a value having second characteristic information corresponding to a label indicating that the dancer is good at dancing. Thereby, the information processing device 100 can generate a dance video in which the user's dancing is better than the user's dance included in the original dance video. Furthermore, the information processing device 100 can provide the user with a dance video in which the user's dancing becomes better.

Furthermore, the center diagram in FIG. 8 shows a case where the attribute information associated with the feature information is whether or not the dance is sharp. For example, the generation unit 152 converts the latent variable in the latent space from a value having first characteristic information corresponding to first attribute information (for example, a label indicating lack of sharpness in dancing) to second attribute information (for example, , a label indicating that the dance is sharp). Thereby, the information processing device 100 can generate a dance video in which the user's dance is sharper than the user's dance included in the original dance video. Furthermore, the information processing device 100 can provide the user with a dance video in which the user's dance becomes more sharp.

Furthermore, the diagram on the right side of FIG. 8 shows a case where the attribute information associated with the feature information is the muscle mass of the person dancing. For example, the generation unit 152 converts the latent variable in the latent space from a value having first feature information corresponding to first attribute information (for example, a label indicating that muscle mass is small) to a value having second attribute information (for example, It may be changed to a value having second characteristic information corresponding to a label indicating that the muscle mass is large. Thereby, the information processing device 100 can generate a dance video in which the user's muscle mass is greater than the user's muscle mass included in the original dance video. Furthermore, the information processing device 100 can provide the user with a dance video in which the user has more muscle mass.

[7-2. About the user's body movements]
In the embodiment described above, a case has been described in which the exercise video including the movement of the user's body is a dance video, but the exercise video is not limited to a dance video. For example, the user's body movements included in the exercise video may be movements in rehabilitation, sports (eg, figure skating, etc.), or acting, in addition to dancing.

[8. effect〕
As described above, the information processing device according to the embodiment (the information processing device 100 in the embodiment) includes an acquisition unit (the acquisition unit 151 in the embodiment) and a generation unit (the generation unit 152 in the embodiment). The acquisition unit uses a machine learning model that includes an encoder that generates feature information indicating the characteristics of the time series data based on the time series data regarding changes in joint angles, and a decoder that generates time series data based on the feature information. get. The generation unit generates second time-series data regarding changes in the user's joint angles from first time-series data regarding changes in the user's joint angles using the latent space of the machine learning model.

As a result, the information processing device uses the latent space of the machine learning model to generate data in the latent space from the first time series data corresponding to the user's first body movement (also referred to as the first movement). can be changed to second time series data corresponding to an arbitrary value. Here, the second time-series data corresponds to a second body movement (hereinafter referred to as second movement) of the user that is different from the first body movement. That is, by using the latent space of the machine learning model, the information processing device calculates the user's first movement obtained by processing the user's first movement from the first time series data corresponding to the user's first movement. Morphing to the second time series data corresponding to the second movement can be realized. For example, by using a latent space classified according to the type of movement, the information processing device can convert first time series data corresponding to a first movement to second time series data corresponding to a second movement. make it possible to morph into Further, the information processing device can generate first time-series data from a first exercise video including a first exercise of the user. Further, the information processing device can generate a second exercise video including a second exercise of the user from the second time series data. That is, the information processing device creates a new exercise image (for example, a second exercise image) according to the attribute of the exercise image desired by the user (for example, the type of exercise) based on the user's first exercise image. ) can be generated. Therefore, the information processing device can generate an arbitrary exercise image from an exercise image that includes the movement of the user's body. In addition, information processing devices can generate arbitrary exercise images from exercise images that include the user's body movements. We can contribute to the achievement of "Let's create something." Further, the information processing device can provide the user with a new exercise video according to the attributes of the exercise video desired by the user. That is, the information processing device can provide new entertainment to users. Therefore, the information processing device can provide spiritual enrichment to the user.

The acquisition unit also acquires a machine learning model trained so that the probability distribution of the feature information follows a normal distribution. The generation unit maps the first time series data to a latent space, and converts a latent variable in the latent space from a value having first feature information corresponding to the first time series data mapped to the latent space to a second value. The latent variable is changed to a value having characteristic information, and second time series data is generated based on the second characteristic information corresponding to the value of the latent variable after the change.

Thereby, the information processing device can generate clusters of feature information according to the attributes of the exercise video on the latent space. The information processing device also maps the first exercise image of the user to the position of a cluster of feature information according to the first attribute of the first exercise image (hereinafter also referred to as a cluster of the first attribute). be able to. Further, the information processing device can change the latent variable from a value having first feature information belonging to a cluster of a first attribute to a value having second feature information belonging to a cluster of a second attribute. . Further, the information processing device can generate a second exercise image based on second feature information belonging to a cluster with a second attribute.

Further, the time series data includes attribute information corresponding to the time series data. The acquisition unit acquires a machine learning model trained such that a probability distribution of feature information indicating characteristics of time-series data including attribute information follows a normal distribution. The generation unit changes the latent variable in the latent space from a value having first feature information corresponding to the first attribute information to a value having second feature information corresponding to the second attribute information. Second time-series data is generated based on second feature information corresponding to the subsequent value of the latent variable.

Thereby, the information processing device can change the latent variable from a value having the first feature information belonging to the cluster of the first attribute to a value having the second feature information belonging to the cluster of the second attribute. can. Further, the information processing device can generate a second exercise image based on second feature information belonging to a cluster of a second attribute.

The machine learning model further includes a posture estimation model trained to estimate the posture of the target object from an image including the target object. The generation unit estimates the coordinates of the user's joint points from the first movement image including the user's body movements using the posture estimation model, and calculates the coordinates of the user's joint points based on the estimated coordinates of the joint points. Generate series data.

Thereby, the information processing device can appropriately estimate the user's posture included in the first exercise video, and therefore can appropriately generate information indicating the user's body movement.

Further, the generation unit generates, based on the generated second time-series data, a second exercise image including the movement of the user's body corresponding to the second time-series data.

Thereby, the information processing device can generate a second exercise image including any body movement obtained by processing the first exercise image including the user's body movement.

The machine learning model further includes an image transformation model learned to generate a person image of the subject corresponding to the joint points from a joint image including the joint points of the subject. The generation unit generates a second exercise image from the second time series data using the image conversion model.

Thereby, the information processing device can generate a second exercise image that includes the user's body movements by fleshing out the user's skeletal model.

As described above, the information processing device according to the embodiment (generation device 20 in the embodiment) includes an acquisition unit (acquisition unit 231 in the embodiment) and a model generation unit (model generation unit 232 in the embodiment). The acquisition unit acquires time-series data regarding changes in joint angles. The model generation unit generates a machine learning model including an encoder that generates feature information indicating characteristics of the time series data based on the time series data, and a decoder that generates time series data based on the feature information.

As a result, the information processing device uses the latent space of the machine learning model to generate data in the latent space from the first time series data corresponding to the user's first body movement (also referred to as the first movement). can be changed to second time series data corresponding to an arbitrary value. That is, by using the latent space of the machine learning model, the information processing device calculates the user's first movement obtained by processing the user's first movement from the first time series data corresponding to the user's first movement. Morphing to second time series data representing second body movement (also referred to as second movement) is made possible. For example, by using a latent space classified according to the type of movement, the information processing device can convert first time series data corresponding to a first movement to second time series data corresponding to a second movement. make it possible to morph into Further, the information processing device can generate the first time-series data from the first exercise video including the user's first exercise. Further, the information processing device can generate a second exercise image including the second exercise of the user from the second time series data. That is, the information processing device creates a new exercise image (for example, a second exercise image) according to the attribute of the exercise image desired by the user (for example, the type of exercise) based on the user's first exercise image. ) can be generated. Therefore, the information processing device can generate an arbitrary exercise image from an exercise image that includes the movement of the user's body. In addition, information processing devices can generate arbitrary exercise images from exercise images that include the user's body movements. We can contribute to the achievement of "Let's create something." Further, the information processing device can provide the user with a new exercise video according to the attributes of the exercise video desired by the user. That is, the information processing device can provide new entertainment to users. Therefore, the information processing device can provide spiritual enrichment to the user.

The model generation unit also trains the machine learning model so that the degree of similarity between the time series data input to the encoder and the time series data output from the decoder exceeds a predetermined threshold.

Thereby, the information processing device can improve the accuracy of the machine learning model.

Further, the model generation unit trains the machine learning model so that the probability distribution of the feature information follows a normal distribution.

Thereby, the information processing device can generate clusters of feature information according to the attributes of the exercise video on the latent space.

The acquisition unit also acquires time series data including attribute information corresponding to the time series data. The model generation unit trains the machine learning model so that the probability distribution of feature information indicating characteristics of time series data including attribute information follows a normal distribution.

Thereby, the information processing device can generate clusters of feature information according to the attributes of the exercise video on the latent space.

Furthermore, the model generation unit classifies the feature information into clusters according to the attribute information.

Thereby, the information processing device can improve usability when providing information to the user regarding the attributes of the exercise video desired by the user (for example, the type of exercise).

In addition, the attribute information includes the type of the subject's body movement included in the exercise video corresponding to the time-series data, the subject's proficiency level of the subject's body movement, the characteristics of the subject's body movement, or the subject's This is information indicating biological information.

As a result, the information processing device can generate a new exercise image according to the type of body movement desired by the user, the proficiency level of body movement, the characteristics of body movement, or biological information. can.

[9. Hardware configuration]
Further, information devices such as the generation device 20 and the information processing device 100 according to the embodiments described above are realized by, for example, a computer 1000 having a configuration as shown in FIG. The information processing device 100 will be described below as an example. FIG. 9 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I/F) 1500, an input/output interface (I/F) 1600, and a media interface (I/F) 1700.

CPU 1100 operates based on a program stored in ROM 1300 or HDD 1400, and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, programs depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. Communication interface 1500 receives data from other devices via a predetermined communication network and sends it to CPU 1100, and transmits data generated by CPU 1100 to other devices via a predetermined communication network.

The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and mouse via an input/output interface 1600. CPU 1100 obtains data from an input device via input/output interface 1600. Further, CPU 1100 outputs the generated data to an output device via input/output interface 1600.

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200. CPU 1100 loads this program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 150 by executing a program loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be acquired from another device via a predetermined communication network.

Some of the embodiments of the present application have been described above in detail based on the drawings, but these are merely examples, and various modifications and variations may be made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure section of the invention. It is possible to carry out the invention in other forms with modifications.

[10. others〕
Furthermore, among the processes described in the above embodiments and modified examples, all or part of the processes described as being performed automatically can be performed manually, or may be described as being performed manually. All or part of this processing can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured. For example, in the embodiment described above, a case has been described in which the generation device 20 and the information processing device 100 are separate devices, but the generation device 20 and the information processing device 100 may be an integrated device. When the generation device 20 and the information processing device 100 are integrated, the information processing device 100 may have the functions of the generation device 20.

Furthermore, the above-described embodiments and modifications can be combined as appropriate within a range that does not conflict with the processing contents.

1 Information processing system 20 Generation device 21 Communication unit 22 Storage unit 23 Control unit 231 Acquisition unit 232 Model generation unit 233 Distribution unit 100 Information processing device 110 Communication unit 120 Storage unit 130 Input unit 140 Output unit 150 Control unit 151 Acquisition unit 152 Generation Department 153 Providing Department 154 Reception Department

Claims (16)

Obtaining a machine learning model that includes an encoder that generates feature information indicating characteristics of the time series data based on time series data regarding changes in joint angles, and a decoder that generates the time series data based on the feature information. an acquisition unit to
a generation unit that uses the latent space of the machine learning model to generate second time-series data regarding changes in the user's joint angles from first time-series data regarding changes in the user's joint angles;
An information processing device comprising: The acquisition unit includes:
obtaining the machine learning model trained such that the probability distribution of the feature information follows a normal distribution;
The generation unit is
The first time series data is mapped onto the latent space, and a latent variable in the latent space is changed from a value having first feature information corresponding to the first time series data mapped onto the latent space to a second value. and generating the second time series data based on the second characteristic information corresponding to the value of the latent variable after the change.
The information processing device according to claim 1. The time series data includes attribute information corresponding to the time series data,
The acquisition unit includes:
obtaining the machine learning model trained so that the probability distribution of the feature information indicating the characteristics of the time series data including the attribute information follows a normal distribution;
The generation unit is
After changing the latent variable in the latent space from a value having the first characteristic information corresponding to the first attribute information to a value having the second characteristic information corresponding to the second attribute information, generating the second time series data based on the second feature information corresponding to the value of the latent variable;
The information processing device according to claim 2. The machine learning model further includes a pose estimation model trained to estimate a pose of the target object from an image including the target object,
The generation unit is
Using the posture estimation model, the coordinates of the joint points of the user are estimated from a first movement image including the body movements of the user, and the coordinates of the joint points of the user are estimated based on the estimated coordinates of the joint points. generate series data,
The information processing device according to claim 1. The generation unit is
Based on the generated second time-series data, a second exercise image including the movement of the user's body corresponding to the second time-series data is generated;
The information processing device according to claim 1. The machine learning model further includes an image transformation model learned to generate a person image of the subject corresponding to the joint points from a joint image including the joint points of the subject,
The generation unit is
generating the second motion image from the second time series data using the image conversion model;
The information processing device according to claim 5. an acquisition unit that acquires time series data regarding changes in joint angles;
A model generation unit that generates a machine learning model, including an encoder that generates feature information indicating characteristics of the time series data based on the time series data, and a decoder that generates the time series data based on the feature information. and,
An information processing device comprising: The model generation unit is
learning the machine learning model so that the degree of similarity between the time series data input to the encoder and the time series data output from the decoder exceeds a predetermined threshold;
The information processing device according to claim 7. The model generation unit is
learning the machine learning model so that the probability distribution of the feature information follows a normal distribution;
The information processing device according to claim 7. The acquisition unit includes:
obtaining the time series data including attribute information corresponding to the time series data;
The model generation unit is
learning the machine learning model so that the probability distribution of the feature information indicating the characteristics of the time series data including the attribute information follows a normal distribution;
The information processing device according to claim 9. The model generation unit is
classifying the feature information into clusters according to the attribute information;
The information processing device according to claim 10. The attribute information may include the type of body movement of the subject included in the exercise video corresponding to the time series data, the proficiency level of the subject's body movement, the characteristics of the subject's body movement, or Information indicating the subject's biological information,
The information processing device according to claim 3 or 10. An information processing method realized by a program executed by an information processing device, the method comprising:
Obtaining a machine learning model that includes an encoder that generates feature information indicating characteristics of the time series data based on time series data regarding changes in joint angles, and a decoder that generates the time series data based on the feature information. an acquisition process to
a generation step of generating second time-series data regarding changes in the user's joint angles from first time-series data regarding changes in the user's joint angles using the latent space of the machine learning model;
Information processing methods including. An information processing method realized by a program executed by an information processing device, the method comprising:
an acquisition step of acquiring time series data regarding changes in joint angle;
A model generation step of generating a machine learning model including an encoder that generates feature information indicating characteristics of the time series data based on the time series data, and a decoder that generates the time series data based on the feature information. and,
Information processing methods including. Obtaining a machine learning model that includes an encoder that generates feature information indicating characteristics of the time series data based on time series data regarding changes in joint angles, and a decoder that generates the time series data based on the feature information. the acquisition procedure,
a generation procedure of generating second time series data regarding changes in the user's joint angles from first time series data regarding changes in the user's joint angles using the latent space of the machine learning model;
An information processing program that causes a computer to execute. an acquisition procedure for acquiring time series data regarding changes in joint angle;
A model generation procedure for generating a machine learning model including an encoder that generates feature information indicating characteristics of the time series data based on the time series data, and a decoder that generates the time series data based on the feature information. and,
An information processing program that causes a computer to execute.

Images