WO2023026784A1 - プログラム、情報処理装置、および情報処理方法 - Google Patents

プログラム、情報処理装置、および情報処理方法 Download PDF

Info

Publication number
WO2023026784A1
WO2023026784A1 PCT/JP2022/029578 JP2022029578W WO2023026784A1 WO 2023026784 A1 WO2023026784 A1 WO 2023026784A1 JP 2022029578 W JP2022029578 W JP 2022029578W WO 2023026784 A1 WO2023026784 A1 WO 2023026784A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
data
exercise
exercise tolerance
program according
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/029578
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
一裕 寺嶋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cate Inc
Original Assignee
Cate Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cate Inc filed Critical Cate Inc
Priority to JP2023509555A priority Critical patent/JP7333537B2/ja
Publication of WO2023026784A1 publication Critical patent/WO2023026784A1/ja
Priority to JP2023125910A priority patent/JP2023133487A/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G16H20/30—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising

Definitions

  • the present disclosure relates to a program, an information processing device, and an information processing method.
  • Cardiac rehabilitation is a comprehensive activity program that includes exercise therapy to help patients with heart disease regain strength and self-confidence, return to a comfortable home and social life, and prevent recurrence or readmission of heart disease. It aims to.
  • the core of exercise therapy is aerobic exercise such as walking, jogging, cycling, and aerobics. In order to perform aerobic exercise more safely and effectively, it is preferable for the patient to exercise at an intensity near his/her own anaerobic threshold (AT).
  • AT anaerobic threshold
  • the anaerobic metabolic threshold is an example of an exercise tolerance evaluation index, and corresponds to the change point of the cardiopulmonary function status, that is, the exercise intensity near the boundary between aerobic and anaerobic exercise.
  • the anaerobic metabolic threshold is generally determined by a CPX test (cardiopulmonary exercise test) in which exhaled gas is collected and analyzed while the test subject is gradually given an exercise load (see Non-Patent Document 1). .
  • the CPX test uses results measured by exhaled gas analysis (e.g., oxygen uptake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof) to determine the anaerobic metabolic threshold. is determined.
  • the CPX test can also determine the maximal oxygen uptake, which corresponds to an exercise intensity near the maximum exercise tolerance.
  • the CPX examination has problems such as the fact that it imposes a large physical burden on the person to be examined, that the examination equipment is expensive, and the facilities that can carry out the examination are limited. In addition, it is not comfortable for the test subject to wear an exhalation mask, which is required for exhaled gas analysis, which is essential for the CPX examination.
  • Forcing the wearer to wear a special device, such as an expiratory mask, during exercise can be uncomfortable, inconvenient, or cumbersome to test subjects, and can affect exercise tolerance endpoints (e.g., anaerobic threshold, or maximal oxygen uptake). (quantity)).
  • exercise tolerance endpoints e.g., anaerobic threshold, or maximal oxygen uptake. (quantity)
  • the purpose of the present disclosure is to evaluate the user's exercise tolerance without imposing a burden on the user.
  • a program causes a computer to function as means for acquiring a user moving image showing a user exercising and means for estimating the user's exercise tolerance based on the user moving image.
  • FIG. 1 is a block diagram showing the configuration of an information processing system according to an embodiment
  • FIG. It is a block diagram showing the configuration of the client device of the present embodiment. It is a block diagram which shows the structure of the server of this embodiment. It is a block diagram showing the configuration of the wearable device of the present embodiment.
  • FIG. 1 is an explanatory diagram of the outline of this embodiment; It is a figure which shows the data structure of the teacher data set of this embodiment. 4 is a flowchart of information processing according to the embodiment; It is a figure which shows the example of a screen displayed in the information processing of this embodiment. It is a figure which shows the example of a screen displayed in the information processing of this embodiment. It is a figure which shows the example of a screen displayed in the information processing of this embodiment. It is a figure which shows the example of a screen displayed in the information processing of this embodiment.
  • FIG. 10 is a diagram showing the data structure of a teacher data set of modification 1;
  • FIG. 1 is a block diagram showing the configuration of the information processing system of this embodiment.
  • the information processing system 1 includes a client device 10, a server 30, and a wearable device 50.
  • FIG. The client device 10 and server 30 are connected via a network (for example, the Internet or an intranet) NW.
  • the client device 10 and the wearable device 50 are connected via a wireless channel using Bluetooth (registered trademark) technology, for example.
  • the client device 10 is an example of an information processing device that transmits requests to the server 30 .
  • the client device 10 is, for example, a smart phone, a tablet terminal, or a personal computer.
  • the server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request sent from the client device 10 .
  • the server 30 is, for example, a web server.
  • the wearable device 50 is an example of an information processing device that can be worn on a user's body (for example, an arm).
  • FIG. 2 is a block diagram showing the configuration of the client device of this embodiment.
  • the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, and a communication interface .
  • Client device 10 is connected to display 15 , camera 16 and depth sensor 17 .
  • the storage device 11 is configured to store programs and data.
  • the storage device 11 is, for example, a combination of ROM (Read Only Memory), RAM (Random Access Memory), and storage (eg, flash memory or hard disk).
  • Programs include, for example, the following programs. ⁇ Program of OS (Operating System) ⁇ Program of application that executes information processing (for example, web browser, rehabilitation application, or fitness application)
  • the data includes, for example, the following data. ⁇ Databases referenced in information processing ⁇ Data obtained by executing information processing (that is, execution results of information processing)
  • the processor 12 is a computer that implements the functions of the client device 10 by activating programs stored in the storage device 11 .
  • Processor 12 is, for example, at least one of the following: ⁇ CPU (Central Processing Unit) ⁇ GPU (Graphic Processing Unit) ⁇ ASIC (Application Specific Integrated Circuit) ⁇ FPGA (Field Programmable Gate Array)
  • the input/output interface 13 acquires information (e.g., user instructions, images, sounds) from input devices connected to the client device 10, and outputs information (e.g., images, sounds) to output devices connected to the client device 10. command).
  • the input device is, for example, camera 16, depth sensor 17, microphone, keyboard, pointing device, touch panel, sensor, or a combination thereof.
  • Output devices are, for example, display 15, speakers, or a combination thereof.
  • Communication interface 14 is configured to control communication between client device 10 and external devices (eg, server 30 and wearable device 50).
  • communication interface 14 may include a module for communication with server 30 (eg, a WiFi module, a mobile communication module, or a combination thereof).
  • Communication interface 14 may include a module (eg, a Bluetooth module) for communication with wearable device 50 .
  • the display 15 is configured to display images (still images or moving images).
  • the display 15 is, for example, a liquid crystal display or an organic EL display.
  • the camera 16 is configured to take pictures and generate image signals.
  • the depth sensor 17 is, for example, LIDAR (Light Detection And Ranging).
  • the depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to surrounding objects (eg, a user).
  • FIG. 3 is a block diagram showing the configuration of the server of this embodiment.
  • the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface .
  • the storage device 31 is configured to store programs and data.
  • Storage device 31 is, for example, a combination of ROM, RAM, and storage.
  • Programs include, for example, the following programs. ⁇ OS program ⁇ Application program that executes information processing
  • the data includes, for example, the following data. ⁇ Databases referenced in information processing ⁇ Execution results of information processing
  • the processor 32 is a computer that implements the functions of the server 30 by activating programs stored in the storage device 31 .
  • Processor 32 is, for example, at least one of the following: ⁇ CPU ⁇ GPU ⁇ ASICs ⁇ FPGA
  • the input/output interface 33 is configured to obtain information (eg, user instructions) from input devices connected to the server 30 and output information to output devices connected to the server 30 .
  • Input devices are, for example, keyboards, pointing devices, touch panels, or combinations thereof.
  • An output device is, for example, a display.
  • the communication interface 34 is configured to control communication between the server 30 and an external device (eg, client device 10).
  • an external device eg, client device 10
  • FIG. 4 is a block diagram showing the configuration of the wearable device of this embodiment.
  • the wearable device 50 includes a storage device 51, a processor 52, an input/output interface 53, and a communication interface . Wearable device 50 is connected to display 55 and heart rate sensor 56 .
  • the storage device 51 is configured to store programs and data.
  • Storage device 51 is, for example, a combination of ROM, RAM, and storage.
  • Programs include, for example, the following programs. ⁇ Program of OS ⁇ Program of application (for example, rehabilitation application or fitness application) that executes information processing
  • the data includes, for example, the following data. ⁇ Databases referenced in information processing ⁇ Execution results of information processing
  • the processor 52 is a computer that implements the functions of the wearable device 50 by activating programs stored in the storage device 51 .
  • Processor 52 is, for example, at least one of the following: ⁇ CPU ⁇ GPU ⁇ ASICs ⁇ FPGA
  • the input/output interface 53 acquires information (e.g., user instructions, sensing results) from input devices connected to the wearable device 50, and outputs information (e.g., images, commands) to output devices connected to the wearable device 50. ).
  • the input device is, for example, heart rate sensor 56, keyboard, pointing device, touch panel, or a combination thereof.
  • Output devices are, for example, display 55, speakers, or a combination thereof.
  • Communication interface 54 is configured to control communication between wearable device 50 and an external device (eg, client device 10).
  • communication interface 54 may include a module (eg, a Bluetooth module) for communication with client device 10 .
  • the display 55 is configured to display images (still images or moving images).
  • the display 55 is, for example, a liquid crystal display or an organic EL display.
  • the heartbeat sensor 56 is configured to measure heartbeats and generate sensing signals.
  • heart rate sensor 56 measures heart rate using optical measurement techniques.
  • FIG. 5 is an explanatory diagram of the outline of this embodiment.
  • the camera 16 of the client device 10 captures the appearance (for example, the whole body) of the user US1 during exercise.
  • the example of FIG. 5 shows an example in which the user US1 exercises by bicycle, the user US1 can perform any exercise (aerobic exercise or anaerobic exercise).
  • the camera 16 captures the appearance of the user US1 from the front or obliquely in front.
  • the depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. Note that it is also possible to generate three-dimensional video data by combining, for example, moving image data (two-dimensional) generated by the camera 16 and depth data generated by, for example, the depth sensor 17 .
  • the heartbeat sensor 56 of the wearable device 50 measures the heartbeat of the user US1 and transmits the measurement result to the client device 10.
  • the client device 10 at least refers to the video data acquired from the camera 16 and analyzes the user's physical condition during exercise.
  • the client device 10 may further refer to depth data acquired from the depth sensor 17 in order to analyze the user's physical condition during exercise.
  • the client device 10 provides data related to the physical condition of the user US1 during exercise (hereinafter referred to as , “user data”) to the server 30 .
  • the server 30 estimates the exercise tolerance of the user US1 by applying the learned model LM1 (an example of an "estimation model") to the acquired user data.
  • the server 30 transmits the estimation result (for example, the numerical value indicating the anaerobic threshold, the maximum oxygen uptake, or the real-time cardiopulmonary exercise load of the user US1) to the client device 10 .
  • the information processing system 1 estimates the exercise tolerance of the user US1 based on the moving image (or moving image and depth) of the user US1 during exercise and the heart rate. Therefore, according to the information processing system 1, it is possible to evaluate the exercise tolerance of the user without imposing a burden on the user such as wearing special equipment.
  • FIG. 6 is a diagram showing the data structure of the teacher data set of this embodiment.
  • the teacher data set includes multiple teacher data.
  • Teacher data is used to train or evaluate a target model.
  • Teacher data includes a sample ID, input data, and correct answer data.
  • a sample ID is information that identifies teacher data.
  • Input data is the data that is input to the target model during training or evaluation.
  • the input data correspond to the examples used when training or evaluating the target model.
  • the input data is data regarding the subject's physical condition during exercise. At least part of the data regarding the subject's physical condition is obtained by analyzing the subject's physical condition with reference to subject moving image data (or subject moving image data and subject depth data).
  • Subject video data is data related to subject videos showing subjects in motion.
  • the subject video data is, for example, a camera (for example, a camera mounted on a smartphone) from the front or obliquely forward (for example, 45 degrees forward) of the subject's appearance (for example, the whole body) during a test on expired gas (for example, a CPX test) ) can be obtained by shooting with
  • Subject depth data is data related to the distance (depth) from the depth sensor to each part of the subject during exercise.
  • the subject depth data can be obtained by operating the depth sensor when capturing the subject moving image.
  • the subject may be the same person as the user whose exercise tolerance is estimated during operation of the information processing system 1, or may be a different person.
  • the target model may learn the user's individuality and the estimation accuracy may be improved.
  • allowing the subject to be a different person from the user has the advantage of facilitating enrichment of the teacher data set.
  • the subjects may be composed of multiple people including the user or multiple people not including the user.
  • the input data includes skeletal data, facial expression data, skin color data, respiration data, and heart rate data.
  • Skeletal data is data (for example, feature values) related to the subject's skeleton during exercise.
  • Skeletal data includes, for example, data on velocity or acceleration of each part of the subject (which may include data on changes in muscle parts used by the subject or data on bodily sensation of the subject).
  • Skeletal data can be obtained by analyzing the skeleton of a exercising subject with reference to subject moving image data (or subject moving image data and subject depth data).
  • subject moving image data or subject moving image data and subject depth data.
  • the iOS ® 14 SDK, Vision, or other skeleton detection algorithms are available for skeleton analysis.
  • skeletal data for the teacher data set can be obtained, for example, by having the subject perform exercise while wearing motion sensors on each part of the subject.
  • Facial expression data is data (for example, feature values) related to the subject's facial expression during exercise. Facial expression data can be analyzed by applying algorithms or trained models to subject video data. Alternatively, the facial expression data for the teacher data set can be obtained by labeling by a person who has watched the subject moving image, for example.
  • the skin color data is data (for example, feature values) related to the subject's skin color during exercise.
  • Skin color data can be analyzed by applying algorithms or trained models to subject video data.
  • the skin color data for the training data set can be obtained by labeling by a person who has watched the subject video, for example.
  • Breathing data is data (for example, feature quantity) relating to breathing of a subject during exercise.
  • Respiration data relates, for example, to the number of breaths per unit time or the breathing pattern.
  • Breathing modalities can include at least one of the following. ⁇ Ventilation frequency ⁇ Ventilation volume ⁇ Ventilation rate (that is, ventilation volume per unit time, or ventilation rate) ⁇ Ventilation acceleration (that is, the time derivative of the ventilation rate) ⁇ Carbon dioxide emission concentration ⁇ Carbon dioxide emission (VCO2) ⁇ Oxygen uptake concentration ⁇ Oxygen uptake (VO2)
  • Respiration data can be obtained, for example, by analyzing the skeleton data.
  • the following items can be analyzed from skeleton data. Movement (spreading) of the shoulders, chest (which may include the lateral chest), abdomen, or a combination thereof ⁇ Inspiratory time ⁇ Expiratory time ⁇ Use of accessory muscles
  • Respiratory data for the teacher data set can be obtained, for example, from the results of breath gas tests performed on subjects during exercise. Details of a breath gas test that can be performed on an exercising subject are described below.
  • ventilation rate, ventilation volume, ventilation rate, or ventilation acceleration among the respiratory data for the teacher dataset may be obtained from, for example, a respiratory function test (e.g., pulmonary function test, or spirometry test) performed on the subject during exercise. ) can also be obtained from the results of In this case, the respiratory function test is not limited to medical equipment, and commercially available test equipment may be used.
  • Heart rate data is data (for example, feature values) related to the subject's heart rate during exercise.
  • Heart rate data can be acquired by analyzing, for example, subject moving image data or its analysis results (for example, skin color data).
  • the heart rate data for the training data set may be obtained from the results of a test for exhaled gas, for example together with the respiration data described below.
  • Subject heart rate data for the training data set can also be obtained by having the subject perform the above exercise while wearing a heart rate sensor or electrodes for an electrocardiogram monitor.
  • Correct data is data corresponding to the correct answer for the corresponding input data (example).
  • the target model is trained (supervised learning) to produce an output that is closer to the correct data for the input data.
  • the correct data is at least an exercise tolerance evaluation index, an index showing the relationship between the evaluation index and the amount of exercise load, or an index that is a material for determining the exercise tolerance evaluation index. including one.
  • Anaerobic threshold (AT) and maximum oxygen uptake (Peak VO2) are examples of exercise tolerance evaluation indices.
  • the cardiopulmonary exercise load is an example of an index that indicates the relationship between the exercise tolerance evaluation index and the exercise load.
  • the cardiopulmonary exercise load can be calculated as a ratio of the (real-time) exercise load to the maximum oxygen uptake.
  • Correct data can be obtained, for example, from the results of breath gas tests conducted on subjects who are exercising.
  • a first example of a breath gas test is a test (typically a CPX test) performed while a subject wearing a breath gas analyzer is performing progressive resistance exercise (eg, ergometer).
  • a second example of an expired gas test is performed while a subject wearing an expired gas analyzer is exercising with a constant or variable load (e.g., body weight exercise, gymnastics, strength training). It is an inspection that is used.
  • correct data can also be obtained from the results of tests other than breath gas performed on exercising subjects.
  • correct data can also be obtained from the results of a cardiopulmonary exercise load prediction test based on the measurement of lactate concentration in sweat or blood of a subject during exercise.
  • a wearable lactate sensor may be utilized to measure a subject's lactate concentration.
  • the exercise load is an index for quantitatively evaluating the exercise load.
  • Exercise load can be expressed numerically using at least one of the following: ⁇ Energy (calorie) consumption ⁇ Oxygen consumption ⁇ Heart rate
  • the estimation model used by the server 30 corresponds to a trained model created by supervised learning using the teacher data set (FIG. 6), or a derived model or distilled model of the trained model.
  • FIG. 7 is a flow chart of information processing in this embodiment.
  • FIG. 8 is a diagram showing an example of a screen displayed during information processing according to this embodiment.
  • FIG. 9 is a diagram showing an example of a screen displayed during information processing according to this embodiment.
  • FIG. 10 is a diagram showing an example of a screen displayed during information processing according to this embodiment.
  • Information processing is started, for example, when one of the following start conditions is met. ⁇ The information processing was called by another process. - The user performed an operation to call up information processing. - The client device 10 has entered a predetermined state (for example, a predetermined application has been activated). ⁇ The specified date and time has arrived. - A predetermined time has passed since a predetermined event.
  • the client device 10 performs sensing (S110). Specifically, the client device 10 enables the operation of the camera 16 to start shooting a video of the user exercising (hereinafter referred to as “user video”). In addition, by enabling the operation of the depth sensor 17, the client device 10 starts measuring the distance from the depth sensor 17 to each part of the user during exercise (hereinafter referred to as “user depth”). Further, the client device 10 causes the wearable device 50 to start measuring the heart rate (hereinafter referred to as “user heart rate”) by the heart rate sensor 56 .
  • user heart rate the heart rate sensor 56 .
  • the client device 10 executes data acquisition (S111). Specifically, the client device 10 acquires sensing results generated by various sensors enabled in step S110. For example, the client device 10 acquires user video data from the camera 16 , user depth data from the depth sensor 17 , and user heart rate data from the wearable device 50 .
  • the client device 10 executes the request (S112). Specifically, the client device 10 refers to the data acquired in step S111 and generates a request. The client device 10 transmits the generated request to the server 30 .
  • the request can include, for example, at least one of the following.
  • - Data acquired in step S111 for example, user video data, user depth data, or user heart rate data
  • User data for example, skeleton data, facial expression data, skin color
  • the server 30 performs exercise tolerance estimation (S130). Specifically, the server 30 acquires the input data of the estimation model based on the request acquired from the client device 10 .
  • the input data includes user data (skeletal data, facial expression data, skin color data, respiration data, heart rate data, or a combination thereof) as well as teacher data.
  • the server 30 estimates exercise tolerance by applying an estimation model to the input data. As an example, the server 30 estimates at least one of the following. ⁇ Evaluation index of exercise tolerance (e.g., anaerobic threshold, or maximal oxygen uptake) - The relationship between the user's (real-time) exercise load and the estimated evaluation index (for example, magnitude relationship, difference, ratio (that is, cardiopulmonary exercise load), or a combination thereof)
  • ⁇ Evaluation index of exercise tolerance e.g., anaerobic threshold, or maximal oxygen uptake
  • the server 30 executes a response (S131). Specifically, the server 30 generates a response based on the estimation result in step S130.
  • the server 30 transmits the generated response to the client device 10 .
  • the response can include at least one of the following. ⁇ Data corresponding to the result of estimation regarding exercise tolerance ⁇ Data obtained by processing the result of estimation regarding exercise tolerance (for example, screen data to be displayed on the display 15 of the client device 10, or data for generating the screen) referenced data)
  • the client device 10 executes information presentation (S113) after step S131. Specifically, the client device 10 causes the display 15 to display information based on the response acquired from the server 30 (that is, the result of estimation regarding the user's exercise tolerance). However, the information may be presented to the user's mentor (eg, medical personnel or trainer) on the terminal used by the trainer instead of, or in addition to, the user.
  • the user's mentor eg, medical personnel or trainer
  • the client device 10 causes the display 15 to display a screen P10 (FIG. 8).
  • the screen P10 includes display objects A10a to A10b.
  • the display object A10a displays information indicating the user's exercise tolerance evaluation index.
  • the display object A10a displays the result of converting the user's anaerobic threshold into METS.
  • the display object A10b displays information on recommended activities for the user.
  • the activity recommended to the user is an activity different from the exercise performed by the user and having a load corresponding to the user's exercise tolerance evaluation index.
  • the load amount according to the exercise tolerance evaluation index of the user is, for example, the same load amount as the exercise tolerance evaluation index, or a load amount slightly above/below the exercise tolerance evaluation index.
  • Activities recommended for the user may be selected based on user attributes (eg, preferences). In the example of FIG. 8, the display object A10b displays three selected from the activities with loads corresponding to the user's anaerobic threshold.
  • the client device 10 causes the display 15 to display a screen P11 (FIG. 9).
  • the screen P11 includes display objects A11a to A11b.
  • the display object A11a displays information indicating the user's exercise tolerance evaluation index.
  • the display object A11a displays the result of converting the user's maximum oxygen uptake into METS and the change in the result from the previous time.
  • the display object A11b displays changes over time in the result of converting the user's exercise tolerance evaluation index into METS.
  • the display object A11b displays a graph showing changes over time in the results of converting the user's maximum oxygen uptake into METS.
  • the client device 10 causes the display 15 to display screens P12a to P12c (FIG. 10).
  • the client device 10 displays the screens P12a to P12c on the display 15 while the user is exercising.
  • Screens P12a-P12c include display objects A12a-A12e.
  • the display object A12a is an object that displays the remaining time of exercise or the duration of exercise.
  • the length of exercise time may be fixed or variable.
  • the length of exercise time may be set by the user, may be set by the user's instructor, or may be set by the operator of the application executed by the client device 10 .
  • the display object A12a displays a timer indicating the remaining time of exercise.
  • the display object A12b displays a score that evaluates the quality of the user's exercise.
  • the score is an accumulated value of points given so that the smaller the difference between the user's exercise load and the exercise tolerance evaluation index, the higher the score.
  • the display object A12c displays information about the relationship between the user's exercise load and the evaluation index of the user's exercise tolerance. In the example of FIG. 10, the display object A12c displays whether the user's exercise load exceeds the anaerobic metabolism threshold or falls below the anaerobic metabolism threshold by animation production of the state of the balance.
  • the display object A12d displays information about the relationship between the user's exercise load and the evaluation index of the user's exercise tolerance. In the example of FIG.
  • the display object A12d indicates whether the user's exercise load exceeds the anaerobic metabolism threshold or falls below the anaerobic metabolism threshold by the expression of the icon.
  • the display object A12e displays information about the user's exercise load amount adjustment guideline. In the example of FIG. 10, the display object A12e displays a comment telling whether the exercise load amount of the user should be increased, maintained, or decreased.
  • the client device 10 After step S113, the client device 10 ends the information processing (FIG. 7). However, if the user's exercise tolerance is estimated in real time while the user is exercising, the client device 10 may return to data acquisition (S111) after step S113.
  • the information processing system 1 of the embodiment estimates the user's exercise tolerance based on the user's moving image (or moving image and depth) during exercise and heart rate. . This makes it possible to evaluate the user's exercise tolerance without placing a burden on the user, such as wearing a special device.
  • the information processing system 1 may estimate the user's exercise tolerance by applying an estimation model to input data based on the user's moving image (or moving image and depth) during exercise and heart rate. .
  • This allows a statistical estimation of the exercise tolerance of the user to be made in a short period of time.
  • the estimated model may correspond to a trained model created by supervised learning using the aforementioned teacher data set (FIG. 6), or a derived or distilled model of the trained model. Thereby, an estimation model can be constructed efficiently.
  • the input data to which the estimation model is applied may include user data regarding the user's physical state during exercise. This makes it possible to improve the accuracy of the estimation model.
  • the user data may include data regarding at least one of the user's anatomy, facial expression, skin tone, respiration, or heart rate during exercise. This makes it possible to improve the accuracy of the estimation model. Furthermore, the subject may be the same person as the user. As a result, highly accurate estimation can be performed using a model that has learned the user's individuality.
  • the information processing system 1 may estimate at least one of the user's anaerobic threshold, maximum oxygen uptake, or real-time cardiopulmonary exercise load. This makes it possible to appropriately evaluate the user's exercise tolerance or the user's real-time exercise load.
  • the information processing system 1 may present information based on the results of estimation regarding the user's exercise tolerance. Thereby, it is possible to give advice to the user or the instructor according to the user's exercise tolerance.
  • the information processing system 1 may present an evaluation index of the user's exercise tolerance. This allows the recipient of the information to grasp the user's current exercise tolerance.
  • the information processing system 1 may present information regarding an activity different from the exercise performed by the user, having a load amount according to the user's exercise tolerance evaluation index. As a result, the degree of freedom of the user's activities can be increased, and the motivation for continuing rehabilitation or fitness can be improved.
  • the information processing system 1 may present information about changes over time in the user's exercise tolerance evaluation index.
  • the information processing system 1 may present information regarding an adjustment guideline for the user's exercise load amount based on the result of estimation regarding exercise tolerance. This allows the user to optimize load by following instructions presented during exercise.
  • information about the relationship between the user's exercise load and the estimation result of the user's exercise tolerance evaluation index may be presented. This allows the user to self-adjust and optimize the load during exercise.
  • Modification 1 will be described. Modification 1 is an example of modifying the input data for the estimation model.
  • Modification 1 An overview of Modification 1 will be explained. In this embodiment, an example in which an estimation model is applied to input data based on user moving images has been described. In Modification 1, by applying an estimation model to input data based on both the user's video and the user's health condition, it is also possible to estimate the user's exercise tolerance.
  • Health conditions include at least one of the following: ⁇ Age, gender, height, weight, body fat percentage, muscle mass, bone density, history of current illness, medical history, oral medication history, surgery history, lifestyle history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty) score, etc.) ⁇ Family history ⁇ Respiratory function test results ⁇ Test results other than respiratory function tests (e.g., blood test, urine test, electrocardiogram (including Holter electrocardiogram), echocardiography, X-ray, CT scan (cardiac morphology) including CT and coronary artery CT), MRI examination, nuclear medicine examination, PET examination, etc.) ⁇ Data acquired during cardiac rehabilitation (including Borg index)
  • FIG. 11 is a diagram showing the data structure of the teacher data set of Modification 1. As shown in FIG.
  • the teacher data set of Modification 1 includes a plurality of teacher data.
  • Teacher data is used to train or evaluate a target model.
  • Teacher data includes a sample ID, input data, and correct answer data.
  • the sample ID and correct answer data are as described in this embodiment.
  • Input data is the data that is input to the target model during training or evaluation.
  • the input data correspond to the examples used when training or evaluating the target model.
  • the input data are data regarding the subject's physical condition during exercise (ie, relatively dynamic data) and data regarding the subject's health state (ie, relatively static data).
  • the data regarding the subject's physical condition are as described in this embodiment.
  • Data on the health status of subjects can be obtained in various ways.
  • the data regarding the health condition of the subject may be obtained before, during, or after the subject's exercise.
  • Data on the subject's health condition may be obtained based on a report from the subject or the attending physician, or may be obtained by extracting information linked to the subject in the medical information system. , may be obtained via the subject's app (eg, health care app).
  • the estimation model used by the server 30 is a trained model created by supervised learning using a teacher data set (FIG. 11), or a derived model of the trained model. Or it corresponds to the distillation model.
  • the client device 10 performs sensing (S110) in the same manner as in FIG.
  • the client device 10 executes data acquisition (S111). Specifically, the client device 10 acquires sensing results generated by various sensors enabled in step S110. For example, the client device 10 acquires user video data from the camera 16 , user depth data from the depth sensor 17 , and user heart rate data from the wearable device 50 .
  • the client device 10 acquires data regarding the user's health condition (hereinafter referred to as "user health condition data").
  • user health condition data data regarding the user's health condition
  • the client device 10 may acquire user health condition data based on an operation (report) by the user or the doctor in charge, or extract information linked to the user in the medical information system.
  • User health data may be acquired, or user health data may be acquired via a user's app (for example, a healthcare app).
  • the client device 10 may acquire user health condition data at a timing different from step S111 (for example, before step S110, at the same timing as step S110, or after step S111).
  • the client device 10 executes the request (S112). Specifically, the client device 10 refers to the data acquired in step S111 and generates a request. The client device 10 transmits the generated request to the server 30 .
  • the request can include, for example, at least one of the following.
  • - Data acquired in step S111 for example, user video data, user depth data, user heart rate data, or user health condition data
  • ⁇ Data obtained by processing the data acquired in step S111
  • User data for example, skeleton data, facial expression data, skin color
  • the server 30 performs exercise tolerance estimation (S130). Specifically, the server 30 acquires the input data of the estimation model based on the request acquired from the client device 10 .
  • Input data includes user data (skeletal structure data, facial expression data, skin color data, respiration data, heart rate data, or a combination thereof, and health condition data) as well as teacher data.
  • the server 30 estimates exercise tolerance by applying an estimation model to the input data. As an example, the server 30 estimates at least one of the following.
  • ⁇ Evaluation index of exercise tolerance e.g., anaerobic threshold, or maximal oxygen uptake
  • the relationship between the user's (real-time) exercise load and the estimated evaluation index for example, magnitude relationship, difference, ratio (that is, cardiopulmonary exercise load), or a combination thereof)
  • step S130 the server 30 executes a response (S131) as in FIG.
  • step S131 the client device 10 executes information presentation (S113) in the same manner as in FIG.
  • the information processing system 1 of Modification 1 applies an estimation model to the input data based on both the user's video and the user's health condition, thereby determining the exercise tolerance of the user. make inferences about ability. This makes it possible to perform highly accurate estimation by further considering the health condition of the user. For example, even if there is a difference between the user's health condition and the subject's health condition from which the training data is based, a reasonable estimation can be made.
  • the storage device 11 may be connected to the client device 10 via the network NW.
  • the display 15 may be built into the client device 10 .
  • Storage device 31 may be connected to server 30 via network NW.
  • the information processing system of the embodiment and modification 1 can also be implemented by a stand-alone computer.
  • the client device 10 alone may use the estimation model to estimate exercise tolerance.
  • Each step of the above information processing can be executed by either the client device 10 or the server 30.
  • the server 30 may acquire at least part of the user data by analyzing the user moving image (or the user moving image and the user depth).
  • a screen based on the response from server 30 may be displayed on display 55 of wearable device 50 .
  • an example of shooting a user video using the camera 16 of the client device 10 has been shown.
  • the user moving image may be shot using a camera other than camera 16 .
  • An example of measuring the user depth using the depth sensor 17 of the client device 10 has been shown.
  • the user depth may be measured using a depth sensor other than depth sensor 17 .
  • the heart rate can be obtained by analyzing (eg, rPPG (Remote Photo-plethysmography) analysis) video data or its analysis results (eg, skin color data).
  • Heart rate analysis may be performed by a trained model built using machine learning techniques.
  • the electrocardiogram monitor may measure the heart rate of the user by exercising while wearing the electrodes for the electrocardiogram monitor. In these variations, the wearable device 50 need not be worn by the user.
  • Wearable device 50 may include a sensor for measuring at least one of the following items instead of heartbeat sensor 56 or in addition to heartbeat sensor 56 .
  • Acceleration/Blood Sugar Level/Oxygen Saturation Measurement results obtained by each sensor can be appropriately used in generating input data, estimating exercise tolerance, presenting information based on the results of estimation, or in other situations.
  • the measurement result of blood glucose level can be referred to, for example, to evaluate energy consumption or exercise load converted into oxygen consumption.
  • the acceleration measurement result can be used, for example, to determine the score of the user's exercise (eg gymnastics).
  • acceleration data may be used as part of the input data for the estimation model.
  • the user's skeleton may be analyzed with reference to acceleration data.
  • Acceleration data may be obtained by, for example, having the user carry the client device 10 having an acceleration sensor when shooting a user moving image.
  • oxygen saturation data may be obtained, for example, by having the user wear a pulse oximeter when shooting a user moving image.
  • oxygen saturation data may be estimated, for example, by performing rPPG analysis on user video data.
  • the information processing system 1 of the present embodiment and Modification 1 can also be applied to a video game in which the progress of the game is controlled according to the body movements of the player.
  • the information processing system 1 may estimate the user's exercise tolerance during game play, and determine any one of the following according to the result of the estimation. As a result, it is possible to enhance the effect of the video game on promoting the user's health.
  • Quality e.g., difficulty
  • quantity of video game-related tasks e.g., stages, missions, quests
  • Video game-related benefits e.g., in-game currency, items, bonuses
  • quality e.g. type
  • game parameters related to video game progression e.g. score, damage
  • a microphone mounted on the client device 10 or a microphone connected to the client device 10 receives sound waves emitted by a user who is exercising (for example, sounds caused by breathing or vocalization) and generates sound data. good too. Sound data, together with user data, may constitute input data for the estimation model.
  • the CPX test was exemplified as a test related to exhaled gas.
  • an exercise load is gradually applied to the examinee.
  • the real-time cardiopulmonary exercise load can be estimated even when the user is given a constant or variable exercise load.
  • the exercise performed by the user may be body weight exercise, gymnastics, or strength training.
  • Modified Example 1 an example of applying an estimation model to input data based on health conditions was shown. However, it is also possible to build multiple estimation models based on (at least in part) the subject's health status. In this case, (at least part of) the user's health may be referenced to select the estimation model. In this further variation, the input data for the estimation model may be data not based on the user's health, or data based on the user's health and user videos.
  • information processing system 10 client device 11: storage device 12: processor 13: input/output interface 14: communication interface 15: display 16: camera 17: depth sensor 30: server 31: storage device 32: processor 33: input/output interface 34: communication interface 50: wearable device 51: storage device 52: processor 53: input/output interface 54: communication interface 55: display 56: heart rate sensor

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Public Health (AREA)
  • Medical Informatics (AREA)
  • Biophysics (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Dentistry (AREA)
  • Molecular Biology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Pathology (AREA)
  • Physiology (AREA)
  • Physics & Mathematics (AREA)
  • Veterinary Medicine (AREA)
  • Physical Education & Sports Medicine (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
PCT/JP2022/029578 2021-08-26 2022-08-02 プログラム、情報処理装置、および情報処理方法 Ceased WO2023026784A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2023509555A JP7333537B2 (ja) 2021-08-26 2022-08-02 プログラム、情報処理装置、および情報処理方法
JP2023125910A JP2023133487A (ja) 2021-08-26 2023-08-02 プログラム、情報処理装置、および情報処理方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021137934 2021-08-26
JP2021-137934 2021-08-26

Publications (1)

Publication Number Publication Date
WO2023026784A1 true WO2023026784A1 (ja) 2023-03-02

Family

ID=85323083

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/029578 Ceased WO2023026784A1 (ja) 2021-08-26 2022-08-02 プログラム、情報処理装置、および情報処理方法

Country Status (2)

Country Link
JP (2) JP7333537B2 (https=)
WO (1) WO2023026784A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7627985B1 (ja) * 2024-09-18 2025-02-07 株式会社CaTe 情報処理プログラム、情報処理装置、及び情報処理方法

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025159150A1 (ja) * 2024-01-26 2025-07-31 株式会社CaTe 情報処理装置、方法、プログラム、およびシステム
JP7662239B1 (ja) * 2024-01-26 2025-04-15 株式会社CaTe 情報処理装置、方法、プログラム、およびシステム

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021114021A (ja) * 2020-01-16 2021-08-05 コニカミノルタ株式会社 リハビリテーションを支援する情報の提供の方法およびリハビリテーション支援システム

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7373788B2 (ja) * 2019-09-20 2023-11-06 パナソニックＩｐマネジメント株式会社 リハビリ支援装置、リハビリ支援システム及びリハビリ支援方法

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021114021A (ja) * 2020-01-16 2021-08-05 コニカミノルタ株式会社 リハビリテーションを支援する情報の提供の方法およびリハビリテーション支援システム

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7627985B1 (ja) * 2024-09-18 2025-02-07 株式会社CaTe 情報処理プログラム、情報処理装置、及び情報処理方法

Also Published As

Publication number Publication date
JPWO2023026784A1 (https=) 2023-03-02
JP2023133487A (ja) 2023-09-22
JP7333537B2 (ja) 2023-08-25

Similar Documents

Publication Publication Date Title
Al Osman et al. Ubiquitous biofeedback serious game for stress management
JP7333537B2 (ja) プログラム、情報処理装置、および情報処理方法
KR100466665B1 (ko) 게임을 이용한 자동체력평가운동방법
JP7303595B1 (ja) プログラム、情報処理装置、および情報処理方法
US20070027369A1 (en) Apparatus and methods for assessing human physical performance
Cardona et al. Modulation of physiological responses and activity levels during exergame experiences
JP2023133487A5 (https=)
Montes Validation and reliability of the Hexoskin and Fitbit Flex wearable BIO collection devices
JP7356666B2 (ja) プログラム、情報処理装置、および情報処理方法
JP7689782B2 (ja) 情報処理装置、方法、プログラム、およびシステム
JP7662239B1 (ja) 情報処理装置、方法、プログラム、およびシステム
Park et al. Youth athletes and wearable technology
JP7411945B2 (ja) プログラム、情報処理装置、および情報処理方法
JP7688940B2 (ja) 情報処理装置、方法、プログラム、およびシステム
JP7662240B1 (ja) 情報処理装置、方法、プログラム、およびシステム
WO2025159149A1 (ja) 情報処理装置、方法、プログラム、およびシステム
Portanti Development of a VR scenario for controlled box breathing exercises with heart rate and respiratory pattern biofeedback using embedded headset sensors
Bota et al. Case Study: COPD Pulmonary
WO2025159150A1 (ja) 情報処理装置、方法、プログラム、およびシステム
Arcidiacono Predicting cardiovascular fitness in ethnic minority youth: A comparison of demographic, body composition, and physical activity variables

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2023509555

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22861075

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22861075

Country of ref document: EP

Kind code of ref document: A1