WO2023026784A1

WO2023026784A1 - Program, information processing device, and information processing method

Info

Publication number: WO2023026784A1
Application number: PCT/JP2022/029578
Authority: WO
Inventors: 一裕寺嶋
Original assignee: 株式会社CaTe
Priority date: 2021-08-26
Filing date: 2022-08-02
Publication date: 2023-03-02
Also published as: JP7333537B2; JP2023133487A; JPWO2023026784A1

Abstract

A program according to one aspect of the present disclosure causes a computer to function as: a means for acquiring a user video captured of the user exercising; and a means for estimating the user's exercise tolerance on the basis of the user video.

Description

Program, information processing device, and information processing method

The present disclosure relates to a program, an information processing device, and an information processing method.

Cardiac rehabilitation is a comprehensive activity program that includes exercise therapy to help patients with heart disease regain strength and self-confidence, return to a comfortable home and social life, and prevent recurrence or readmission of heart disease. It aims to. The core of exercise therapy is aerobic exercise such as walking, jogging, cycling, and aerobics. In order to perform aerobic exercise more safely and effectively, it is preferable for the patient to exercise at an intensity near his/her own anaerobic threshold (AT).

The anaerobic metabolic threshold is an example of an exercise tolerance evaluation index, and corresponds to the change point of the cardiopulmonary function status, that is, the exercise intensity near the boundary between aerobic and anaerobic exercise. The anaerobic metabolic threshold is generally determined by a CPX test (cardiopulmonary exercise test) in which exhaled gas is collected and analyzed while the test subject is gradually given an exercise load (see Non-Patent Document 1). . The CPX test uses results measured by exhaled gas analysis (e.g., oxygen uptake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof) to determine the anaerobic metabolic threshold. is determined. Besides the anaerobic metabolic threshold, the CPX test can also determine the maximal oxygen uptake, which corresponds to an exercise intensity near the maximum exercise tolerance.

The CPX examination has problems such as the fact that it imposes a large physical burden on the person to be examined, that the examination equipment is expensive, and the facilities that can carry out the examination are limited.
In addition, it is not comfortable for the test subject to wear an exhalation mask, which is required for exhaled gas analysis, which is essential for the CPX examination.

Forcing the wearer to wear a special device, such as an expiratory mask, during exercise can be uncomfortable, inconvenient, or cumbersome to test subjects, and can affect exercise tolerance endpoints (e.g., anaerobic threshold, or maximal oxygen uptake). (quantity)).

The purpose of the present disclosure is to evaluate the user's exercise tolerance without imposing a burden on the user.

A program according to one aspect of the present disclosure causes a computer to function as means for acquiring a user moving image showing a user exercising and means for estimating the user's exercise tolerance based on the user moving image.

According to the present disclosure, it is possible to evaluate the user's exercise tolerance without imposing a burden on the user.

1 is a block diagram showing the configuration of an information processing system according to an embodiment; FIG. It is a block diagram showing the configuration of the client device of the present embodiment. It is a block diagram which shows the structure of the server of this embodiment. It is a block diagram showing the configuration of the wearable device of the present embodiment. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an explanatory diagram of the outline of this embodiment; It is a figure which shows the data structure of the teacher data set of this embodiment. 4 is a flowchart of information processing according to the embodiment; It is a figure which shows the example of a screen displayed in the information processing of this embodiment. It is a figure which shows the example of a screen displayed in the information processing of this embodiment. It is a figure which shows the example of a screen displayed in the information processing of this embodiment. FIG. 10 is a diagram showing the data structure of a teacher data set of modification 1;

Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In the drawings for describing the embodiments, in principle, the same constituent elements are denoted by the same reference numerals, and repeated description thereof will be omitted.

(1) Configuration of information processing system The configuration of the information processing system will be described. FIG. 1 is a block diagram showing the configuration of the information processing system of this embodiment.

As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a wearable device 50. FIG.
The client device 10 and server 30 are connected via a network (for example, the Internet or an intranet) NW.
The client device 10 and the wearable device 50 are connected via a wireless channel using Bluetooth (registered trademark) technology, for example.

The client device 10 is an example of an information processing device that transmits requests to the server 30 . The client device 10 is, for example, a smart phone, a tablet terminal, or a personal computer.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request sent from the client device 10 . The server 30 is, for example, a web server.

The wearable device 50 is an example of an information processing device that can be worn on a user's body (for example, an arm).

(1-1) Configuration of Client Apparatus The configuration of the client apparatus will be described. FIG. 2 is a block diagram showing the configuration of the client device of this embodiment.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, and a communication interface . Client device 10 is connected to display 15 , camera 16 and depth sensor 17 .

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of ROM (Read Only Memory), RAM (Random Access Memory), and storage (eg, flash memory or hard disk).

Programs include, for example, the following programs.
・Program of OS (Operating System) ・Program of application that executes information processing (for example, web browser, rehabilitation application, or fitness application)

The data includes, for example, the following data.
・Databases referenced in information processing ・Data obtained by executing information processing (that is, execution results of information processing)

The processor 12 is a computer that implements the functions of the client device 10 by activating programs stored in the storage device 11 . Processor 12 is, for example, at least one of the following:
・CPU (Central Processing Unit)
・GPU (Graphic Processing Unit)
・ASIC (Application Specific Integrated Circuit)
・FPGA (Field Programmable Gate Array)

The input/output interface 13 acquires information (e.g., user instructions, images, sounds) from input devices connected to the client device 10, and outputs information (e.g., images, sounds) to output devices connected to the client device 10. command).
The input device is, for example, camera 16, depth sensor 17, microphone, keyboard, pointing device, touch panel, sensor, or a combination thereof.
Output devices are, for example, display 15, speakers, or a combination thereof.

Communication interface 14 is configured to control communication between client device 10 and external devices (eg, server 30 and wearable device 50).
Specifically, communication interface 14 may include a module for communication with server 30 (eg, a WiFi module, a mobile communication module, or a combination thereof). Communication interface 14 may include a module (eg, a Bluetooth module) for communication with wearable device 50 .

The display 15 is configured to display images (still images or moving images). The display 15 is, for example, a liquid crystal display or an organic EL display.

The camera 16 is configured to take pictures and generate image signals.

The depth sensor 17 is, for example, LIDAR (Light Detection And Ranging). The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to surrounding objects (eg, a user).

(1-2) Server Configuration The server configuration will be described. FIG. 3 is a block diagram showing the configuration of the server of this embodiment.

As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface .

The storage device 31 is configured to store programs and data. Storage device 31 is, for example, a combination of ROM, RAM, and storage.

Programs include, for example, the following programs.
・OS program ・Application program that executes information processing

The data includes, for example, the following data.
・Databases referenced in information processing ・Execution results of information processing

The processor 32 is a computer that implements the functions of the server 30 by activating programs stored in the storage device 31 . Processor 32 is, for example, at least one of the following:
・CPU
・GPU
・ASICs
・FPGA

The input/output interface 33 is configured to obtain information (eg, user instructions) from input devices connected to the server 30 and output information to output devices connected to the server 30 .
Input devices are, for example, keyboards, pointing devices, touch panels, or combinations thereof.
An output device is, for example, a display.

The communication interface 34 is configured to control communication between the server 30 and an external device (eg, client device 10).

(1-3) Configuration of Wearable Device The configuration of the wearable device will be described. FIG. 4 is a block diagram showing the configuration of the wearable device of this embodiment.

As shown in FIG. 4, the wearable device 50 includes a storage device 51, a processor 52, an input/output interface 53, and a communication interface . Wearable device 50 is connected to display 55 and heart rate sensor 56 .

The storage device 51 is configured to store programs and data. Storage device 51 is, for example, a combination of ROM, RAM, and storage.

Programs include, for example, the following programs.
・Program of OS ・Program of application (for example, rehabilitation application or fitness application) that executes information processing

The processor 52 is a computer that implements the functions of the wearable device 50 by activating programs stored in the storage device 51 . Processor 52 is, for example, at least one of the following:
・CPU
・GPU
・ASICs
・FPGA

The input/output interface 53 acquires information (e.g., user instructions, sensing results) from input devices connected to the wearable device 50, and outputs information (e.g., images, commands) to output devices connected to the wearable device 50. ).
The input device is, for example, heart rate sensor 56, keyboard, pointing device, touch panel, or a combination thereof.
Output devices are, for example, display 55, speakers, or a combination thereof.

Communication interface 54 is configured to control communication between wearable device 50 and an external device (eg, client device 10).
Specifically, communication interface 54 may include a module (eg, a Bluetooth module) for communication with client device 10 .

The display 55 is configured to display images (still images or moving images). The display 55 is, for example, a liquid crystal display or an organic EL display.

The heartbeat sensor 56 is configured to measure heartbeats and generate sensing signals. As an example, heart rate sensor 56 measures heart rate using optical measurement techniques.

(2) Outline of Embodiment An outline of the present embodiment will be described. FIG. 5 is an explanatory diagram of the outline of this embodiment.

As shown in FIG. 5, the camera 16 of the client device 10 captures the appearance (for example, the whole body) of the user US1 during exercise. Although the example of FIG. 5 shows an example in which the user US1 exercises by bicycle, the user US1 can perform any exercise (aerobic exercise or anaerobic exercise).

As an example, the camera 16 captures the appearance of the user US1 from the front or obliquely in front. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. Note that it is also possible to generate three-dimensional video data by combining, for example, moving image data (two-dimensional) generated by the camera 16 and depth data generated by, for example, the depth sensor 17 .

The heartbeat sensor 56 of the wearable device 50 measures the heartbeat of the user US1 and transmits the measurement result to the client device 10.

The client device 10 at least refers to the video data acquired from the camera 16 and analyzes the user's physical condition during exercise. The client device 10 may further refer to depth data acquired from the depth sensor 17 in order to analyze the user's physical condition during exercise. The client device 10 provides data related to the physical condition of the user US1 during exercise (hereinafter referred to as , “user data”) to the server 30 .

The server 30 estimates the exercise tolerance of the user US1 by applying the learned model LM1 (an example of an "estimation model") to the acquired user data. The server 30 transmits the estimation result (for example, the numerical value indicating the anaerobic threshold, the maximum oxygen uptake, or the real-time cardiopulmonary exercise load of the user US1) to the client device 10 .

In this way, the information processing system 1 estimates the exercise tolerance of the user US1 based on the moving image (or moving image and depth) of the user US1 during exercise and the heart rate. Therefore, according to the information processing system 1, it is possible to evaluate the exercise tolerance of the user without imposing a burden on the user such as wearing special equipment.

(3) Teacher data set The teacher data set of this embodiment will be described. FIG. 6 is a diagram showing the data structure of the teacher data set of this embodiment.

As shown in FIG. 6, the teacher data set includes multiple teacher data. Teacher data is used to train or evaluate a target model. Teacher data includes a sample ID, input data, and correct answer data.

A sample ID is information that identifies teacher data.

Input data is the data that is input to the target model during training or evaluation. The input data correspond to the examples used when training or evaluating the target model. As an example, the input data is data regarding the subject's physical condition during exercise. At least part of the data regarding the subject's physical condition is obtained by analyzing the subject's physical condition with reference to subject moving image data (or subject moving image data and subject depth data).

　Subject video data is data related to subject videos showing subjects in motion. The subject video data is, for example, a camera (for example, a camera mounted on a smartphone) from the front or obliquely forward (for example, 45 degrees forward) of the subject's appearance (for example, the whole body) during a test on expired gas (for example, a CPX test) ) can be obtained by shooting with

Subject depth data is data related to the distance (depth) from the depth sensor to each part of the subject during exercise. The subject depth data can be obtained by operating the depth sensor when capturing the subject moving image.

The subject may be the same person as the user whose exercise tolerance is estimated during operation of the information processing system 1, or may be a different person. By setting the subject and the user to be the same person, the target model may learn the user's individuality and the estimation accuracy may be improved. On the other hand, allowing the subject to be a different person from the user has the advantage of facilitating enrichment of the teacher data set. Also, the subjects may be composed of multiple people including the user or multiple people not including the user.

In the example of FIG. 6, the input data includes skeletal data, facial expression data, skin color data, respiration data, and heart rate data.

Skeletal data is data (for example, feature values) related to the subject's skeleton during exercise. Skeletal data includes, for example, data on velocity or acceleration of each part of the subject (which may include data on changes in muscle parts used by the subject or data on bodily sensation of the subject). Skeletal data can be obtained by analyzing the skeleton of a exercising subject with reference to subject moving image data (or subject moving image data and subject depth data). As an example, the iOS ® 14 SDK, Vision, or other skeleton detection algorithms are available for skeleton analysis. Alternatively, skeletal data for the teacher data set can be obtained, for example, by having the subject perform exercise while wearing motion sensors on each part of the subject.

Facial expression data is data (for example, feature values) related to the subject's facial expression during exercise. Facial expression data can be analyzed by applying algorithms or trained models to subject video data. Alternatively, the facial expression data for the teacher data set can be obtained by labeling by a person who has watched the subject moving image, for example.

The skin color data is data (for example, feature values) related to the subject's skin color during exercise. Skin color data can be analyzed by applying algorithms or trained models to subject video data. Alternatively, the skin color data for the training data set can be obtained by labeling by a person who has watched the subject video, for example.

Breathing data is data (for example, feature quantity) relating to breathing of a subject during exercise. Respiration data relates, for example, to the number of breaths per unit time or the breathing pattern. Breathing modalities can include at least one of the following.
・Ventilation frequency ・Ventilation volume ・Ventilation rate (that is, ventilation volume per unit time, or ventilation rate)
・Ventilation acceleration (that is, the time derivative of the ventilation rate)
・Carbon dioxide emission concentration ・Carbon dioxide emission (VCO2)
・Oxygen uptake concentration ・Oxygen uptake (VO2)
Data relating to breathing patterns may include data that can be calculated based on a combination of the above data, such as the gas exchange ratio R (=VCO2/VO2).

Respiration data can be obtained, for example, by analyzing the skeleton data. As an example, the following items can be analyzed from skeleton data.
Movement (spreading) of the shoulders, chest (which may include the lateral chest), abdomen, or a combination thereof
・Inspiratory time ・Expiratory time ・Use of accessory muscles

Respiratory data for the teacher data set can be obtained, for example, from the results of breath gas tests performed on subjects during exercise. Details of a breath gas test that can be performed on an exercising subject are described below. Alternatively, ventilation rate, ventilation volume, ventilation rate, or ventilation acceleration among the respiratory data for the teacher dataset may be obtained from, for example, a respiratory function test (e.g., pulmonary function test, or spirometry test) performed on the subject during exercise. ) can also be obtained from the results of In this case, the respiratory function test is not limited to medical equipment, and commercially available test equipment may be used.

Heart rate data is data (for example, feature values) related to the subject's heart rate during exercise. Heart rate data can be acquired by analyzing, for example, subject moving image data or its analysis results (for example, skin color data). Alternatively, the heart rate data for the training data set may be obtained from the results of a test for exhaled gas, for example together with the respiration data described below. Subject heart rate data for the training data set can also be obtained by having the subject perform the above exercise while wearing a heart rate sensor or electrodes for an electrocardiogram monitor.

Correct data is data corresponding to the correct answer for the corresponding input data (example). The target model is trained (supervised learning) to produce an output that is closer to the correct data for the input data. As an example, the correct data is at least an exercise tolerance evaluation index, an index showing the relationship between the evaluation index and the amount of exercise load, or an index that is a material for determining the exercise tolerance evaluation index. including one. Anaerobic threshold (AT) and maximum oxygen uptake (Peak VO2) are examples of exercise tolerance evaluation indices. The cardiopulmonary exercise load is an example of an index that indicates the relationship between the exercise tolerance evaluation index and the exercise load. For example, the cardiopulmonary exercise load can be calculated as a ratio of the (real-time) exercise load to the maximum oxygen uptake.

Correct data can be obtained, for example, from the results of breath gas tests conducted on subjects who are exercising. A first example of a breath gas test is a test (typically a CPX test) performed while a subject wearing a breath gas analyzer is performing progressive resistance exercise (eg, ergometer). A second example of an expired gas test is performed while a subject wearing an expired gas analyzer is exercising with a constant or variable load (e.g., body weight exercise, gymnastics, strength training). It is an inspection that is used.

Alternatively, the correct data can also be obtained from the results of tests other than breath gas performed on exercising subjects. Specifically, correct data can also be obtained from the results of a cardiopulmonary exercise load prediction test based on the measurement of lactate concentration in sweat or blood of a subject during exercise. A wearable lactate sensor may be utilized to measure a subject's lactate concentration.

The exercise load is an index for quantitatively evaluating the exercise load. Exercise load can be expressed numerically using at least one of the following:
・Energy (calorie) consumption ・Oxygen consumption ・Heart rate

(4) Estimation model The estimation model used by the server 30 corresponds to a trained model created by supervised learning using the teacher data set (FIG. 6), or a derived model or distilled model of the trained model.

(5) Information processing Information processing according to the present embodiment will be described. FIG. 7 is a flow chart of information processing in this embodiment. FIG. 8 is a diagram showing an example of a screen displayed during information processing according to this embodiment. FIG. 9 is a diagram showing an example of a screen displayed during information processing according to this embodiment. FIG. 10 is a diagram showing an example of a screen displayed during information processing according to this embodiment.

Information processing is started, for example, when one of the following start conditions is met.
・The information processing was called by another process.
- The user performed an operation to call up information processing.
- The client device 10 has entered a predetermined state (for example, a predetermined application has been activated).
・The specified date and time has arrived.
- A predetermined time has passed since a predetermined event.

As shown in FIG. 7, the client device 10 performs sensing (S110).
Specifically, the client device 10 enables the operation of the camera 16 to start shooting a video of the user exercising (hereinafter referred to as “user video”). In addition, by enabling the operation of the depth sensor 17, the client device 10 starts measuring the distance from the depth sensor 17 to each part of the user during exercise (hereinafter referred to as "user depth"). Further, the client device 10 causes the wearable device 50 to start measuring the heart rate (hereinafter referred to as “user heart rate”) by the heart rate sensor 56 .

After step S110, the client device 10 executes data acquisition (S111).
Specifically, the client device 10 acquires sensing results generated by various sensors enabled in step S110. For example, the client device 10 acquires user video data from the camera 16 , user depth data from the depth sensor 17 , and user heart rate data from the wearable device 50 .

After step S111, the client device 10 executes the request (S112).
Specifically, the client device 10 refers to the data acquired in step S111 and generates a request. The client device 10 transmits the generated request to the server 30 . The request can include, for example, at least one of the following.
- Data acquired in step S111 (for example, user video data, user depth data, or user heart rate data)
・Data obtained by processing the data acquired in step S111 ・User data (for example, skeleton data, facial expression data, skin color) acquired by analyzing the user video data (or user video data and user depth data) acquired in step S111 data, respiratory data, or a combination thereof)

After step S112, the server 30 performs exercise tolerance estimation (S130).
Specifically, the server 30 acquires the input data of the estimation model based on the request acquired from the client device 10 . The input data includes user data (skeletal data, facial expression data, skin color data, respiration data, heart rate data, or a combination thereof) as well as teacher data. The server 30 estimates exercise tolerance by applying an estimation model to the input data. As an example, the server 30 estimates at least one of the following.
・Evaluation index of exercise tolerance (e.g., anaerobic threshold, or maximal oxygen uptake)
- The relationship between the user's (real-time) exercise load and the estimated evaluation index (for example, magnitude relationship, difference, ratio (that is, cardiopulmonary exercise load), or a combination thereof)

After step S130, the server 30 executes a response (S131).
Specifically, the server 30 generates a response based on the estimation result in step S130. The server 30 transmits the generated response to the client device 10 . As an example, the response can include at least one of the following.
・Data corresponding to the result of estimation regarding exercise tolerance ・Data obtained by processing the result of estimation regarding exercise tolerance (for example, screen data to be displayed on the display 15 of the client device 10, or data for generating the screen) referenced data)

The client device 10 executes information presentation (S113) after step S131.
Specifically, the client device 10 causes the display 15 to display information based on the response acquired from the server 30 (that is, the result of estimation regarding the user's exercise tolerance).
However, the information may be presented to the user's mentor (eg, medical personnel or trainer) on the terminal used by the trainer instead of, or in addition to, the user.

In a first example of information presentation (S113), the client device 10 causes the display 15 to display a screen P10 (FIG. 8). The screen P10 includes display objects A10a to A10b.
The display object A10a displays information indicating the user's exercise tolerance evaluation index. In the example of FIG. 8, the display object A10a displays the result of converting the user's anaerobic threshold into METS.
The display object A10b displays information on recommended activities for the user. The activity recommended to the user is an activity different from the exercise performed by the user and having a load corresponding to the user's exercise tolerance evaluation index. The load amount according to the exercise tolerance evaluation index of the user is, for example, the same load amount as the exercise tolerance evaluation index, or a load amount slightly above/below the exercise tolerance evaluation index. Activities recommended for the user may be selected based on user attributes (eg, preferences). In the example of FIG. 8, the display object A10b displays three selected from the activities with loads corresponding to the user's anaerobic threshold.

In a second example of information presentation (S113), the client device 10 causes the display 15 to display a screen P11 (FIG. 9). The screen P11 includes display objects A11a to A11b.
The display object A11a displays information indicating the user's exercise tolerance evaluation index. In the example of FIG. 9, the display object A11a displays the result of converting the user's maximum oxygen uptake into METS and the change in the result from the previous time.
The display object A11b displays changes over time in the result of converting the user's exercise tolerance evaluation index into METS. In the example of FIG. 9, the display object A11b displays a graph showing changes over time in the results of converting the user's maximum oxygen uptake into METS.

In a third example of information presentation (S113), the client device 10 causes the display 15 to display screens P12a to P12c (FIG. 10). The client device 10 displays the screens P12a to P12c on the display 15 while the user is exercising.
Screens P12a-P12c include display objects A12a-A12e.
The display object A12a is an object that displays the remaining time of exercise or the duration of exercise. The length of exercise time may be fixed or variable. The length of exercise time may be set by the user, may be set by the user's instructor, or may be set by the operator of the application executed by the client device 10 . In the example of FIG. 10, the display object A12a displays a timer indicating the remaining time of exercise.
The display object A12b displays a score that evaluates the quality of the user's exercise. As an example, the score is an accumulated value of points given so that the smaller the difference between the user's exercise load and the exercise tolerance evaluation index, the higher the score.
The display object A12c displays information about the relationship between the user's exercise load and the evaluation index of the user's exercise tolerance. In the example of FIG. 10, the display object A12c displays whether the user's exercise load exceeds the anaerobic metabolism threshold or falls below the anaerobic metabolism threshold by animation production of the state of the balance.
The display object A12d displays information about the relationship between the user's exercise load and the evaluation index of the user's exercise tolerance. In the example of FIG. 10, the display object A12d indicates whether the user's exercise load exceeds the anaerobic metabolism threshold or falls below the anaerobic metabolism threshold by the expression of the icon.
The display object A12e displays information about the user's exercise load amount adjustment guideline. In the example of FIG. 10, the display object A12e displays a comment telling whether the exercise load amount of the user should be increased, maintained, or decreased.

After step S113, the client device 10 ends the information processing (FIG. 7). However, if the user's exercise tolerance is estimated in real time while the user is exercising, the client device 10 may return to data acquisition (S111) after step S113.

(6) Summary As described above, the information processing system 1 of the embodiment estimates the user's exercise tolerance based on the user's moving image (or moving image and depth) during exercise and heart rate. . This makes it possible to evaluate the user's exercise tolerance without placing a burden on the user, such as wearing a special device.

The information processing system 1 may estimate the user's exercise tolerance by applying an estimation model to input data based on the user's moving image (or moving image and depth) during exercise and heart rate. . This allows a statistical estimation of the exercise tolerance of the user to be made in a short period of time. Furthermore, the estimated model may correspond to a trained model created by supervised learning using the aforementioned teacher data set (FIG. 6), or a derived or distilled model of the trained model. Thereby, an estimation model can be constructed efficiently. Additionally, the input data to which the estimation model is applied may include user data regarding the user's physical state during exercise. This makes it possible to improve the accuracy of the estimation model. The user data may include data regarding at least one of the user's anatomy, facial expression, skin tone, respiration, or heart rate during exercise. This makes it possible to improve the accuracy of the estimation model. Furthermore, the subject may be the same person as the user. As a result, highly accurate estimation can be performed using a model that has learned the user's individuality.

The information processing system 1 may estimate at least one of the user's anaerobic threshold, maximum oxygen uptake, or real-time cardiopulmonary exercise load. This makes it possible to appropriately evaluate the user's exercise tolerance or the user's real-time exercise load.

The information processing system 1 may present information based on the results of estimation regarding the user's exercise tolerance. Thereby, it is possible to give advice to the user or the instructor according to the user's exercise tolerance. As a first example, the information processing system 1 may present an evaluation index of the user's exercise tolerance. This allows the recipient of the information to grasp the user's current exercise tolerance. As a second example, the information processing system 1 may present information regarding an activity different from the exercise performed by the user, having a load amount according to the user's exercise tolerance evaluation index. As a result, the degree of freedom of the user's activities can be increased, and the motivation for continuing rehabilitation or fitness can be improved. As a third example, the information processing system 1 may present information about changes over time in the user's exercise tolerance evaluation index. This allows the user to feel the growth or decline of his or her own exercise tolerance, and improves motivation to continue rehabilitation or fitness. As a fourth example, the information processing system 1 may present information regarding an adjustment guideline for the user's exercise load amount based on the result of estimation regarding exercise tolerance. This allows the user to optimize load by following instructions presented during exercise. As a fifth example, information about the relationship between the user's exercise load and the estimation result of the user's exercise tolerance evaluation index may be presented. This allows the user to self-adjust and optimize the load during exercise.

(7) Modification 1
Modification 1 will be described. Modification 1 is an example of modifying the input data for the estimation model.

(7-1) Outline of modification 1

An overview of Modification 1 will be explained. In this embodiment, an example in which an estimation model is applied to input data based on user moving images has been described. In Modification 1, by applying an estimation model to input data based on both the user's video and the user's health condition, it is also possible to estimate the user's exercise tolerance.

Health conditions include at least one of the following:
・Age, gender, height, weight, body fat percentage, muscle mass, bone density, history of current illness, medical history, oral medication history, surgery history, lifestyle history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty) score, etc.)
・Family history ・Respiratory function test results ・Test results other than respiratory function tests (e.g., blood test, urine test, electrocardiogram (including Holter electrocardiogram), echocardiography, X-ray, CT scan (cardiac morphology) including CT and coronary artery CT), MRI examination, nuclear medicine examination, PET examination, etc.)
・Data acquired during cardiac rehabilitation (including Borg index)

(7-2) Teacher data set The teacher data set of Modification 1 will be described. FIG. 11 is a diagram showing the data structure of the teacher data set of Modification 1. As shown in FIG.

As shown in FIG. 11, the teacher data set of Modification 1 includes a plurality of teacher data. Teacher data is used to train or evaluate a target model. Teacher data includes a sample ID, input data, and correct answer data.

The sample ID and correct answer data are as described in this embodiment.

Input data is the data that is input to the target model during training or evaluation. The input data correspond to the examples used when training or evaluating the target model. By way of example, the input data are data regarding the subject's physical condition during exercise (ie, relatively dynamic data) and data regarding the subject's health state (ie, relatively static data). The data regarding the subject's physical condition are as described in this embodiment.

Data on the health status of subjects can be obtained in various ways. The data regarding the health condition of the subject may be obtained before, during, or after the subject's exercise. Data on the subject's health condition may be obtained based on a report from the subject or the attending physician, or may be obtained by extracting information linked to the subject in the medical information system. , may be obtained via the subject's app (eg, health care app).

(7-3) Estimation Model In Modification 1, the estimation model used by the server 30 is a trained model created by supervised learning using a teacher data set (FIG. 11), or a derived model of the trained model. Or it corresponds to the distillation model.

(7-4) Information Processing The information processing of Modification 1 will be described with reference to FIG.

In Modification 1, the client device 10 performs sensing (S110) in the same manner as in FIG.

Furthermore, the client device 10 acquires data regarding the user's health condition (hereinafter referred to as "user health condition data"). For example, the client device 10 may acquire user health condition data based on an operation (report) by the user or the doctor in charge, or extract information linked to the user in the medical information system. User health data may be acquired, or user health data may be acquired via a user's app (for example, a healthcare app). However, the client device 10 may acquire user health condition data at a timing different from step S111 (for example, before step S110, at the same timing as step S110, or after step S111).

After step S111, the client device 10 executes the request (S112).
Specifically, the client device 10 refers to the data acquired in step S111 and generates a request. The client device 10 transmits the generated request to the server 30 . The request can include, for example, at least one of the following.
- Data acquired in step S111 (for example, user video data, user depth data, user heart rate data, or user health condition data)
・Data obtained by processing the data acquired in step S111 ・User data (for example, skeleton data, facial expression data, skin color) acquired by analyzing the user video data (or user video data and user depth data) acquired in step S111 data, respiratory data, or a combination thereof)

After step S112, the server 30 performs exercise tolerance estimation (S130).
Specifically, the server 30 acquires the input data of the estimation model based on the request acquired from the client device 10 . Input data includes user data (skeletal structure data, facial expression data, skin color data, respiration data, heart rate data, or a combination thereof, and health condition data) as well as teacher data. The server 30 estimates exercise tolerance by applying an estimation model to the input data. As an example, the server 30 estimates at least one of the following.
・Evaluation index of exercise tolerance (e.g., anaerobic threshold, or maximal oxygen uptake)
- The relationship between the user's (real-time) exercise load and the estimated evaluation index (for example, magnitude relationship, difference, ratio (that is, cardiopulmonary exercise load), or a combination thereof)

After step S130, the server 30 executes a response (S131) as in FIG.

After step S131, the client device 10 executes information presentation (S113) in the same manner as in FIG.

(7-5) Summary As described above, the information processing system 1 of Modification 1 applies an estimation model to the input data based on both the user's video and the user's health condition, thereby determining the exercise tolerance of the user. make inferences about ability. This makes it possible to perform highly accurate estimation by further considering the health condition of the user. For example, even if there is a difference between the user's health condition and the subject's health condition from which the training data is based, a reasonable estimation can be made.

(8) Other Modifications The storage device 11 may be connected to the client device 10 via the network NW. The display 15 may be built into the client device 10 . Storage device 31 may be connected to server 30 via network NW.

An example of implementing the information processing system of the embodiment and modification 1 by a client/server system has been shown. However, the information processing system of the embodiment and modification 1 can also be implemented by a stand-alone computer. As an example, the client device 10 alone may use the estimation model to estimate exercise tolerance.

Each step of the above information processing can be executed by either the client device 10 or the server 30. As an example, instead of the client device 10, the server 30 may acquire at least part of the user data by analyzing the user moving image (or the user moving image and the user depth).

In the above explanation, an example of estimating the anaerobic threshold or maximal oxygen uptake as an exercise tolerance evaluation index and presenting information on the results of the estimation was shown. However, instead of or in addition to these evaluation indices, it is also possible to make estimations about evaluation indices of other types of exercise tolerance, or to present information about the results of the estimations.

In the above description, an example of displaying a screen based on the response from the server 30 on the display 15 of the client device 10 was shown. A screen based on the response from server 30 may be displayed on display 55 of wearable device 50 .

In the above description, an example of shooting a user video using the camera 16 of the client device 10 has been shown. However, the user moving image may be shot using a camera other than camera 16 . An example of measuring the user depth using the depth sensor 17 of the client device 10 has been shown. However, the user depth may be measured using a depth sensor other than depth sensor 17 .

In the above description, an example of measuring the user's heart rate with the wearable device 50 was shown. However, the heart rate can be obtained by analyzing (eg, rPPG (Remote Photo-plethysmography) analysis) video data or its analysis results (eg, skin color data). Heart rate analysis may be performed by a trained model built using machine learning techniques. Alternatively, the electrocardiogram monitor may measure the heart rate of the user by exercising while wearing the electrodes for the electrocardiogram monitor. In these variations, the wearable device 50 need not be worn by the user.

Wearable device 50 may include a sensor for measuring at least one of the following items instead of heartbeat sensor 56 or in addition to heartbeat sensor 56 .
Acceleration/Blood Sugar Level/Oxygen Saturation Measurement results obtained by each sensor can be appropriately used in generating input data, estimating exercise tolerance, presenting information based on the results of estimation, or in other situations. As an example, the measurement result of blood glucose level can be referred to, for example, to evaluate energy consumption or exercise load converted into oxygen consumption. As another example, the acceleration measurement result can be used, for example, to determine the score of the user's exercise (eg gymnastics).

It is also possible to use acceleration data as part of the input data for the estimation model. Alternatively, the user's skeleton may be analyzed with reference to acceleration data. Acceleration data may be obtained by, for example, having the user carry the client device 10 having an acceleration sensor when shooting a user moving image.

It is also possible to use oxygen saturation data as part of the input data for the estimation model. The oxygen saturation data may be obtained, for example, by having the user wear a pulse oximeter when shooting a user moving image. Alternatively, oxygen saturation data may be estimated, for example, by performing rPPG analysis on user video data.

In the above explanation, an example of presenting information on recommended activities to the user was shown. However, instead of or in addition to such information, information regarding the amount of exercise load recommended to the user may be presented. As an example, for the same activity as the exercise performed by the user, information indicating the load amount according to the user's exercise tolerance evaluation index (for example, target speed, target rotation speed, etc.) may be presented.

The information processing system 1 of the present embodiment and Modification 1 can also be applied to a video game in which the progress of the game is controlled according to the body movements of the player. As an example, the information processing system 1 may estimate the user's exercise tolerance during game play, and determine any one of the following according to the result of the estimation. As a result, it is possible to enhance the effect of the video game on promoting the user's health.
- Quality (e.g., difficulty) or quantity of video game-related tasks (e.g., stages, missions, quests) given to users - Video game-related benefits (e.g., in-game currency, items, bonuses) given to users quality (e.g. type) or quantity of game parameters related to video game progression (e.g. score, damage)

A microphone mounted on the client device 10 or a microphone connected to the client device 10 receives sound waves emitted by a user who is exercising (for example, sounds caused by breathing or vocalization) and generates sound data. good too. Sound data, together with user data, may constitute input data for the estimation model.

In the above explanation, the CPX test was exemplified as a test related to exhaled gas. In the CPX examination, an exercise load is gradually applied to the examinee. However, it is not necessary to gradually increase the exercise load given to the user when shooting the user moving image. Specifically, the real-time cardiopulmonary exercise load can be estimated even when the user is given a constant or variable exercise load. For example, the exercise performed by the user may be body weight exercise, gymnastics, or strength training.

In Modified Example 1, an example of applying an estimation model to input data based on health conditions was shown. However, it is also possible to build multiple estimation models based on (at least in part) the subject's health status. In this case, (at least part of) the user's health may be referenced to select the estimation model. In this further variation, the input data for the estimation model may be data not based on the user's health, or data based on the user's health and user videos.

Although the embodiments and modifications of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments and modifications. Also, the above embodiments and modifications can be modified and modified in various ways without departing from the scope of the present invention. Also, the above embodiments and modifications can be combined.

1: information processing system 10: client device 11: storage device 12: processor 13: input/output interface 14: communication interface 15: display 16: camera 17: depth sensor 30: server 31: storage device 32: processor 33: input/output interface 34: communication interface 50: wearable device 51: storage device 52: processor 53: input/output interface 54: communication interface 55: display 56: heart rate sensor

Claims

the computer,
a means for acquiring a user video showing a user exercising;
A program that functions as means for estimating exercise tolerance of the user based on the user moving image.
The means for estimating the exercise tolerance estimates the user's exercise tolerance by applying an estimation model to the input data based on the user moving image.
A program according to claim 1.
The estimation model is learning created by supervised learning using a supervised learning using a teacher data set including input data including data related to a subject video in which the subject is exercising and correct data associated with each of the input data. a pre-trained model, or equivalent to a derived or distilled model of such pre-trained model;
3. A program according to claim 2.
The subject is the same person as the user,
4. A program according to claim 3.
the input data includes user data relating to the user's physical condition during the exercise;
4. The program according to claim 2 or 3.
the user data includes data relating to at least one of the user's anatomy, facial expression, skin tone, respiration, or heart rate during the exercise;
6. A program according to claim 5.
the means for making an estimate makes an estimate regarding at least one of the user's anaerobic threshold, maximal oxygen uptake, or real-time cardiopulmonary exercise load;
A program according to any one of claims 1 to 6.
further causing the computer to serve as a means for presenting information based on the results of the estimation regarding exercise tolerance of the user;
A program according to any one of claims 1 to 7.
the means for presenting presents an evaluation index of exercise tolerance of the user;
9. A program according to claim 8.
The means for presenting comprises a load amount according to an evaluation index of the user's exercise tolerance, and presents information on an activity different from the exercise performed by the user.
10. The program according to claim 8 or 9.
the means for presenting presents information on changes over time in the user's exercise tolerance evaluation index;
11. The program according to any one of claims 8 to 10.
The means for presenting presents information regarding an adjustment guideline for the user's exercise load amount based on the result of the estimation of the exercise tolerance of the user.
A program according to any one of claims 8 to 11.
The means for presenting presents information about the relationship between the exercise load amount of the user and the evaluation index of the exercise tolerance of the user.
A program according to any one of claims 8 to 12.
The acquiring means acquires a user moving image showing a user playing a video game in which the progress of the game is controlled according to the movement of the player's body,
the means for estimating estimates the user's exercise tolerance during play of the video game;
further causing the computer to act as a means for determining at least one of a challenge or reward associated with the video game to be presented to the user as a function of the results of the exercise tolerance estimation of the user;
14. The program according to any one of claims 1 to 13.
a means for acquiring a user video showing a user exercising;
and means for estimating exercise tolerance of the user based on the user moving image.
the computer
Acquiring a user video in which the user is exercising;
making an estimate regarding the user's exercise tolerance based on the user animation.