WO2023106382A1 - 情報処理装置、電子機器、情報処理システム、情報処理方法及びプログラム - Google Patents

情報処理装置、電子機器、情報処理システム、情報処理方法及びプログラム Download PDF

Info

Publication number
WO2023106382A1
WO2023106382A1 PCT/JP2022/045370 JP2022045370W WO2023106382A1 WO 2023106382 A1 WO2023106382 A1 WO 2023106382A1 JP 2022045370 W JP2022045370 W JP 2022045370W WO 2023106382 A1 WO2023106382 A1 WO 2023106382A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
sensor data
movement
sensor
information processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/045370
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
マルティン クリンキグト
秀行 金原
エドワード 村上
貴志 鈴木
雅之 岸
崇 長友
尚樹 西田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kyocera Corp
Original Assignee
Kyocera Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kyocera Corp filed Critical Kyocera Corp
Priority to EP22904309.6A priority Critical patent/EP4446698A4/en
Priority to JP2023566372A priority patent/JP7818622B2/ja
Priority to US18/716,374 priority patent/US20250032885A1/en
Publication of WO2023106382A1 publication Critical patent/WO2023106382A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B71/00—Games or sports accessories not covered in groups A63B1/00 - A63B69/00
    • A63B71/06—Indicating or scoring devices for games or players, or for other sports activities
    • A63B71/0619—Displays, user interfaces and indicating devices, specially adapted for sport equipment, e.g. display mounted on treadmills
    • A63B71/0622—Visual, audio or audio-visual systems for entertaining, instructing or motivating the user
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B21/00—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
    • G01B21/22—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring angles or tapers; for testing the alignment of axes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • A61B5/1113—Local tracking of patients, e.g. in a hospital or private home
    • A61B5/1114—Tracking parts of the body
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • A61B5/1116—Determining posture transitions
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • A61B5/112—Gait analysis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • A61B5/1121—Determining geometric values, e.g. centre of rotation or angular range of movement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
    • A63B24/0003—Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
    • A63B24/0006—Computerised comparison for qualitative assessment of motion sequences or the course of a movement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G16H20/30—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2505/00—Evaluating, monitoring or diagnosing in the context of a particular type of medical care
    • A61B2505/09—Rehabilitation or training
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2562/00—Details of sensors; Constructional details of sensor housings or probes; Accessories for sensors
    • A61B2562/02—Details of sensors specially adapted for in-vivo measurements
    • A61B2562/0219—Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2208/00—Characteristics or parameters related to the user or player
    • A63B2208/02—Characteristics or parameters related to the user or player posture
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/80—Special sensors, transducers or devices therefor
    • A63B2220/83—Special sensors, transducers or devices therefor characterised by the position of the sensor
    • A63B2220/836—Sensors arranged on the body of the user
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2230/00—Measuring physiological parameters of the user
    • A63B2230/62—Measuring physiological parameters of the user posture

Definitions

  • the present disclosure relates to an information processing device, an electronic device, an information processing system, an information processing method, and a program.
  • Patent Literature 1 describes a motion capture system that estimates the motion of an object based on images captured by a plurality of cameras.
  • An information processing device a controller that obtains an estimated posture angle of at least one of a plurality of body parts of the user based on sensor data indicating movement of at least a part of the user's body part and a learning model;
  • the learning model is learned to output an estimated value of the attitude angle when the sensor data is input.
  • An electronic device includes: An output unit for outputting data of the walking model generated by the information processing device is provided.
  • An information processing system includes: an information processing device that obtains an estimated value of a posture angle of at least one of a plurality of body parts of the user based on sensor data indicating movement of at least part of the body part of the user and a learning model; The learning model is learned to output an estimated value of the attitude angle when the sensor data is input.
  • An information processing method includes obtaining an estimated posture angle of at least one of a plurality of body parts of the user from sensor data indicating movement of at least a part of the user's body part and a learning model;
  • the learning model is learned to output an estimated value of the attitude angle when the sensor data is input.
  • a program is to the computer, Acquiring an estimated posture angle of at least one of a plurality of body parts of the user based on sensor data indicating movement of at least a part of the user's body part and a learning model;
  • the learning model is learned to output an estimated value of the attitude angle when the sensor data is input.
  • FIG. 1 is a diagram showing a schematic configuration of an information processing system according to an embodiment of the present disclosure
  • FIG. 2 is a diagram for explaining a local coordinate system and a global coordinate system
  • FIG. 2 is a block diagram showing the configuration of the information processing system shown in FIG. 1
  • FIG. 2 is a block diagram showing the configuration of a transformer
  • FIG. It is a block diagram which shows the structure of "Multi-Head Attention”. It is a block diagram which shows the structure of "Scaled Dot-Product Attention". It is a figure which shows an example of the combination of sensor data. It is a graph of evaluation results.
  • FIG. 10 shows a subject; It is a graph of the posture angle of a test subject's neck.
  • Fig. 10 is a graph of posture angles of the left lower leg of a subject; Fig. 10 is a graph of the posture angle of the right foot of the subject; Fig. 3 is a graph of the posture angle of the left foot of a subject; It is a graph of the posture angle of the right thigh of subjects with high center-of-gravity shift evaluations. It is a graph of the posture angle of the right thigh of subjects with high center-of-gravity shift evaluations. It is a graph of the posture angle of the right thigh of subjects with high center-of-gravity shift evaluations. It is a graph of the posture angle of the right thigh of subjects with high center-of-gravity shift evaluations. It is a graph of the posture angle of the right thigh of subjects with high center-of-gravity shift evaluations.
  • FIG. 1 It is a graph of the posture angle of the upper arm on the right side of subjects with high center-of-gravity shift evaluations. It is a graph of the posture angle of the upper arm on the right side of subjects with high center-of-gravity shift evaluations. It is a graph of the posture angle of the right upper arm of subjects with low center-of-gravity shift evaluations. It is a graph of the posture angle of the right upper arm of subjects with low center-of-gravity shift evaluations. It is a graph of the posture angle of the right upper arm of subjects with low center-of-gravity shift evaluations. It is a graph of the posture angle of the right upper arm of subjects with low center-of-gravity shift evaluations. It is a graph of the posture angle of the right upper arm of subjects with low center-of-gravity shift evaluations. FIG.
  • FIG. 2 is a flowchart showing operations of posture angle estimation processing executed by the electronic device shown in FIG. 1 ;
  • FIG. FIG. 11 is a block diagram showing the configuration of an information processing system according to another embodiment of the present disclosure;
  • FIG. FIG. 40 is a sequence diagram showing operations of an estimation process executed by the information processing system shown in FIG. 39;
  • An information processing system 1 as shown in FIG. 1 can estimate the posture angle of any body part of a user who is exercising periodically.
  • the information processing system 1 can generate a model, such as a three-dimensional animation, showing the state of the user performing periodic exercise, for example, by estimating the posture angles of body parts over the user's whole body.
  • Periodic motion can be any motion.
  • periodic motion is walking, running, or pedaling a bicycle.
  • it is assumed that the periodic exercise is walking. That is, in the present embodiment, the information processing system 1 estimates the posture angles of the user's body parts during walking.
  • a user for example, walks as an exercise in daily life.
  • the information processing system 1 includes a sensor device 10A, a sensor device 10B, a sensor device 10C, sensor devices 10D-1 and 10D-2, sensor devices 10E-1 and 10E-2, and sensor devices 10F-1 and 10F. - 2 and the electronic device 20 . However, the information processing system 1 does not have to include all of the sensor devices 10A, 10B, 10C, 10D-1, 10D-2, 10E-1, 10E-2, 10F-1 and 10F-1. The information processing system 1 may include at least one of the sensor devices 10A, 10B, 10C, 10D-1, 10D-2, 10E-1, 10E-2, 10F-1 and 10F-1.
  • sensor devices 10D-1 and 10D-2 are not particularly distinguished from each other, they are also collectively referred to as “sensor device 10D".
  • sensor devices 10E-1 and 10E-2 are not particularly distinguished from each other, they are also collectively described as the “sensor device 10E.”
  • sensor devices 10F-1 and 10F-2 are not particularly distinguished from each other, they are collectively referred to as the “sensor device 10F.”
  • sensor devices 10A to 10D are not particularly distinguished from each other, they are also collectively described as the “sensor device 10”.
  • the sensor device 10 and the electronic device 20 can communicate via a communication line.
  • the communication line includes at least one of wired and wireless.
  • the local coordinate system is a coordinate system based on the position of the sensor device 10, as shown in FIG. In FIG. 2, as an example of the position of the sensor device 10, the position of the sensor device 10A is indicated by a dashed line.
  • the local coordinate system is composed of, for example, x-, y-, and z-axes.
  • the x-axis, y-axis, and z-axis are orthogonal to each other.
  • the x-axis is parallel to the front-rear direction as seen from the sensor device 10 .
  • the y-axis is parallel to the horizontal direction as seen from the sensor device 10 .
  • the z-axis is parallel to the vertical direction as seen from the sensor device 10 .
  • the positive and negative directions of the x-axis, y-axis, and z-axis may be set according to the configuration of the information processing system 1 and the like.
  • the global coordinate system is a coordinate system based on the position in the space where the user walks, as shown in FIG.
  • the global coordinate system is composed of, for example, X, Y and Z axes.
  • the X-axis, Y-axis, and Z-axis are orthogonal to each other.
  • the X-axis is parallel to the front-rear direction as viewed by the user.
  • the positive direction of the X-axis is the direction from the back side of the user toward the front side of the user.
  • the negative direction of the X-axis is the direction from the user's front side to the user's rear side.
  • the Y-axis is parallel to the vertical direction viewed from the user.
  • the positive direction of the Y-axis is the direction from the bottom of the user to the top of the user. It is assumed that the negative direction of the Y-axis is the direction from the user's upper side to the user's lower side.
  • the Z-axis is parallel to the left-right direction viewed from the user. In this embodiment, the positive direction of the Z axis is the direction from the user's left side to the right side. It is assumed that the negative direction of the Z-axis is the direction from the user's right side to the left side.
  • the positive and negative directions of the X-axis, Y-axis, and Z-axis may be set according to the configuration of the information processing system 1 or the like.
  • the sensor device 10 is worn on at least a part of the user's body.
  • the sensor device 10 detects sensor data indicating the movement of the body part on which the sensor device 10 is worn among the user's body parts.
  • the sensor data are data in the local coordinate system.
  • the sensor data is data indicating motion of at least a part of the user's body.
  • the sensor device 10A is worn on the user's head.
  • the sensor device 10A is worn on the user's ear.
  • the sensor device 10A may be a wearable device.
  • the sensor device 10A may be an earphone or may be included in an earphone.
  • the sensor device 10A may be a device that can be retrofitted to existing glasses, earphones, or the like.
  • the sensor device 10A may be worn on the user's head by any method.
  • the sensor device 10A may be attached to the user's head by being attached to a hair accessory such as a hair band, a hairpin, an earring, a helmet, a hat, a hearing aid, false teeth, an implant, or the like.
  • the x-axis of the local coordinate system based on the position of the sensor device 10A is parallel to the front-rear direction of the head viewed from the user, and the y-axis of the local coordinate system is aligned with the head viewed from the user. It may be mounted on the user's head so that it is parallel to the horizontal direction and the z-axis of the local coordinate system is parallel to the vertical direction of the head as seen from the user.
  • the x-axis, y-axis, and z-axis of the local coordinate system based on the position of the sensor device 10A do not necessarily correspond to the front-rear direction, left-right direction, and up-down direction of the head as seen from the user.
  • the orientation of the sensor device 10A relative to the user's head may be initialized or known as appropriate.
  • the relative orientation is initialized or known using information on the shape of a jig for attaching the sensor device 10A to the user's head or image information generated by imaging the user's head on which the sensor device 10A is mounted. It may be done by
  • the sensor device 10A detects sensor data indicating the movement of the user's head.
  • the sensor data detected by the sensor device 10A includes, for example, the velocity of the user's head, the acceleration of the user's head, the angle of the user's head, the angular velocity of the user's head, the temperature of the user's head, and the user's head including at least one data of the geomagnetic field at the location of the part.
  • the sensor device 10B is worn on the user's forearm.
  • the sensor device 10B is worn on the user's wrist.
  • the sensor device 10B may be worn on the user's left forearm or may be worn on the user's right forearm.
  • the sensor device 10B may be a wristwatch-type wearable device.
  • the sensor device 10B may be worn on the user's forearm by any method.
  • the sensor device 10B may be worn on the user's forearm by being attached to a band, bracelet, misanga, glove, ring, false nail, artificial hand, or the like.
  • the bracelet may be worn by the user for decorative purposes, or may be used to attach a key to a locker or the like to the wrist.
  • the x-axis of the local coordinate system based on the position of the sensor device 10B is parallel to the front-rear direction of the wrist as seen from the user, and the y-axis of the local coordinate system is the left-right direction of the wrist as seen from the user. and the z-axis of the local coordinate system is parallel to the direction of rotation of the wrist as seen by the user.
  • the rotation direction of the wrist is, for example, the direction in which the wrist twists and rotates.
  • the sensor device 10B detects sensor data indicating the movement of the user's forearm. For example, the sensor device 10B detects sensor data indicating wrist movement.
  • the sensor data detected by the sensor device 10B includes, for example, the velocity of the user's forearm, the acceleration of the user's forearm, the angle of the user's forearm, the angular velocity of the user's forearm, the temperature of the user's forearm, and the user's forearm. including at least one data of the geomagnetic field at the location of the part.
  • the sensor device 10C is worn on the waist of the user.
  • the sensor device 10C may be a wearable device.
  • the sensor device 10C may be attached to the waist of the user with a belt, clip, or the like.
  • the x-axis of the local coordinate system based on the position of the sensor device 10C matches the front-rear direction of the waist as seen from the user, and the y-axis of the local coordinate system runs in the left-right direction of the waist as seen from the user. It may be worn on the user's waist such that it matches and the z-axis of the local coordinate system matches the direction of rotation of the waist as seen by the user.
  • the rotation direction of the waist is, for example, the direction in which the waist twists and rotates.
  • the sensor device 10C detects sensor data indicating the movement of the user's lower back.
  • the sensor data detected by the sensor device 10C includes, for example, the velocity of the user's waist, the acceleration of the user's waist, the angle of the user's waist, the angular velocity of the user's waist, the temperature of the user's waist, and the geomagnetism at the position of the user's waist. Contains at least some data.
  • the sensor device 10D-1 is worn on the user's left thigh.
  • the sensor device 10D-2 is worn on the user's right thigh.
  • Sensor device 10D may be a wearable device.
  • Sensor device 10D may be worn on the user's thigh by any method.
  • the sensor device 10D may be attached to the user's thigh by a belt, clip, or the like.
  • the sensor device 10D may be worn on the thigh by being placed in a pocket of pants worn by the user near the thigh.
  • the sensor device 10D may be worn on the user's thigh by being installed on pants, underwear, shorts, a supporter, an artificial leg, an implant, or the like.
  • the x-axis of the local coordinate system based on the position of the sensor device 10D is parallel to the front-rear direction of the thigh viewed from the user, and the y-axis of the local coordinate system is aligned with the thigh viewed from the user. It may be worn on the user's thigh so that it is parallel to the left-right direction of the body and the z-axis of the local coordinate system is parallel to the rotation direction of the thigh viewed from the user.
  • the rotation direction of the thigh is, for example, the direction in which the thigh is twisted and rotated.
  • the sensor device 10D-1 detects sensor data indicating the movement of the user's left thigh.
  • Sensor device 10D-2 detects sensor data indicative of movement of the user's right thigh.
  • the sensor data detected by the sensor device 10D includes, for example, the velocity of the user's thigh, the acceleration of the user's thigh, the angle of the user's thigh, the angular velocity of the user's thigh, and the Includes temperature and/or geomagnetism data at the user's thigh.
  • the sensor device 10E-1 is worn on the user's left ankle.
  • the sensor device 10E-2 is worn on the user's right ankle.
  • the sensor device 10E may be a wearable device.
  • Sensor device 10E may be worn on the user's ankle by any method.
  • the sensor device 10E may be attached to the user's ankle by a belt, clip, or the like.
  • the sensor device 10E may be worn on the user's ankle by being placed on an anklet, band, misanga, tattoo sticker, supporter, cast, sock, artificial leg or implant, or the like.
  • the x-axis of the local coordinate system based on the position of the sensor device 10E coincides with the front-rear direction of the ankle as seen from the user, and the y-axis of the local coordinate system runs with the left-right direction of the ankle as seen from the user. It may be worn on the user's ankle such that it coincides and the z-axis of the local coordinate system coincides with the direction of rotation of the ankle as seen by the user.
  • the rotation direction of the ankle is, for example, the direction in which the ankle twists and rotates.
  • the sensor device 10E-1 detects sensor data indicating movement of the user's left ankle.
  • the sensor device 10E-2 detects sensor data indicative of movement of the user's right ankle.
  • the sensor data detected by the sensor device 10E includes, for example, the velocity of the user's ankle, the acceleration of the user's ankle, the angle of the user's ankle, the angular velocity of the user's ankle, the temperature of the user's ankle, and the geomagnetism at the position of the user's ankle. Contains at least some data.
  • the sensor device 10F-1 is worn on the user's left foot.
  • the sensor device 10F-2 is worn on the user's right foot.
  • the foot is the portion from the user's ankle to the toe.
  • the sensor device 10F may be a shoe last wearable device.
  • the sensor device 10F may be provided on the shoe.
  • the sensor device 10F may be worn on the user's foot by any method.
  • the sensor device 10F may be attached to the user's foot by being attached to an anklet, band, misanga, false nail, tattoo sticker, supporter, cast, sock, insole, artificial leg, ring, implant, or the like.
  • the x-axis of the local coordinate system based on the position of the sensor device 10F is parallel to the front-rear direction of the foot viewed from the user, and the y-axis of the local coordinate system is aligned with the foot viewed from the user. It may be worn on the foot of the user so that it is parallel to the left-right direction and the z-axis of the local coordinate system is parallel to the up-down direction of the foot viewed from the user.
  • the sensor device 10F-1 detects sensor data indicating the movement of the user's left foot.
  • the sensor device 10F-2 detects sensor data indicative of movement of the user's right ankle.
  • the sensor data detected by the sensor device 10F includes, for example, the velocity of the user's foot, the acceleration of the user's foot, the angle of the user's foot, the angular velocity of the user's foot, the temperature of the user's foot, and the user's foot. including at least one data of the geomagnetic field at the location of the part.
  • the electronic device 20 is carried by, for example, a walking user.
  • the electronic device 20 is, for example, a mobile device such as a mobile phone, a smart phone, or a tablet.
  • the electronic device 20 functions as an information processing device, and acquires the estimated values of the posture angles of the body parts of the user based on the sensor data detected by the sensor device 10 and a learning model described later.
  • the posture angle of the body part is the angle of the body part in the global coordinate system.
  • the angle at which the body part rotates about the X axis is also referred to as "posture angle ⁇ X”.
  • the angle by which the body part rotates around the Y-axis is also described as the “posture angle ⁇ Y”.
  • the angle by which the body part rotates around the Z-axis is also described as the "posture angle ⁇ Z”.
  • the positive direction of the posture angle ⁇ X is assumed to be the direction of clockwise rotation of the X-axis when viewed from the negative direction of the X-axis. It is assumed that the negative direction of the posture angle ⁇ X is the direction in which the X-axis rotates counterclockwise when viewed from the negative direction of the X-axis. Also, the positive direction of the posture angle ⁇ Y is assumed to be the direction in which the Y-axis rotates clockwise when viewed from the negative direction of the Y-axis. The negative direction of the attitude angle ⁇ Y is assumed to be the direction in which the Y-axis is rotated counterclockwise when viewed from the negative direction of the Y-axis.
  • the positive direction of the posture angle ⁇ Z is assumed to be the direction in which the Z-axis rotates clockwise when viewed from the negative direction of the Z-axis. It is assumed that the negative direction of the posture angle ⁇ Z is the direction in which the Z-axis rotates counterclockwise when viewed from the negative direction of the Z-axis.
  • the sensor device 10 includes a communication section 11, a sensor section 12, a notification section 13, a storage section 15, and a control section 16. As shown in FIG. However, the sensor devices 10C to 10F do not have to include the notification unit 13. FIG.
  • the communication unit 11 includes at least one communication module capable of communicating with the electronic device 20 via a communication line.
  • the communication module is a communication module conforming to the communication line standard.
  • the communication line standard is, for example, a short-range wireless communication standard including Bluetooth (registered trademark), infrared rays, and NFC (Near Field Communication).
  • the sensor unit 12 is configured including arbitrary sensors corresponding to sensor data to be detected by the sensor device 10 .
  • the sensor unit 12 includes, for example, a 3-axis motion sensor, a 3-axis acceleration sensor, a 3-axis velocity sensor, a 3-axis gyro sensor, a 3-axis geomagnetic sensor, a temperature sensor, and an inertial measurement unit (IMU). and at least one of a camera and the like.
  • the sensor unit 12 includes a camera, the movement of the user's body part can be detected by analyzing the image generated by the camera.
  • the data detected by the acceleration sensor and the geomagnetic sensor are used to calculate the initial angle of the body part to be detected by the sensor device 10. good. Further, data detected by each of the acceleration sensor and the geomagnetic sensor may be used to correct angle data detected by the sensor device 10 .
  • the angle of the body part to be detected by the sensor device 10 may be calculated by time-integrating the angular velocity detected by the gyro sensor.
  • the notification unit 13 notifies information.
  • the notification unit 13 includes an output unit 14 .
  • the notification unit 13 is not limited to the output unit 14 .
  • the reporting unit 13 may include any component capable of reporting information.
  • the output unit 14 can output data.
  • the output unit 14 includes at least one output interface capable of outputting data.
  • the output interface is, for example, a display or speaker.
  • the display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.
  • the output unit 14 may include a speaker when included in the sensor device 10A. Moreover, the output unit 14 may include a display when included in the sensor device 10B.
  • the storage unit 15 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them.
  • the semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory).
  • the RAM is, for example, SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory).
  • the ROM is, for example, EEPROM (Electrically Erasable Programmable Read Only Memory) or the like.
  • the storage unit 15 may function as a main storage device, an auxiliary storage device, or a cache memory.
  • the storage unit 15 stores data used for the operation of the sensor device 10 and data obtained by the operation of the sensor device 10 .
  • the storage unit 15 stores system programs, application programs, embedded software, and the like.
  • the control unit 16 includes at least one processor, at least one dedicated circuit, or a combination thereof.
  • the processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing.
  • the dedicated circuit is, for example, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit).
  • the control unit 16 executes processing related to the operation of the sensor device 10 while controlling each unit of the sensor device 10 .
  • the control unit 16 receives a signal instructing the start of data detection from the electronic device 20 by the communication unit 11 . Upon receiving this signal, the control section 16 starts data detection. For example, the control unit 16 acquires data detected by the sensor unit 12 from the sensor unit 12 . The control unit 16 transmits the acquired data as sensor data to the electronic device 20 through the communication unit 11 . A signal instructing the start of data detection is transmitted from the electronic device 20 to the plurality of sensor devices 10 as a broadcast signal. By transmitting a signal instructing the start of data detection as a broadcast signal to the plurality of sensor devices 10, the plurality of sensor devices 10 can simultaneously start data detection.
  • the control unit 16 acquires data from the sensor unit 12 at preset time intervals, and transmits the acquired data as sensor data through the communication unit 11 .
  • This time interval may be set based on a typical user's walking speed or the like. This time interval may be the same for each of the plurality of sensor devices 10 . Since the time intervals are the same for the plurality of sensor devices 10, the timings at which the plurality of sensor devices 10 detect data can be synchronized.
  • the electronic device 20 includes a communication section 21 , an input section 22 , a notification section 23 , a storage section 26 and a control section 27 .
  • the communication unit 21 includes at least one communication module capable of communicating with the sensor device 10 via a communication line.
  • the communication module is at least one communication module compatible with the communication line standard.
  • the communication line standard is, for example, a short-range wireless communication standard including Bluetooth (registered trademark), infrared rays, NFC, and the like.
  • the communication unit 21 may further include at least one communication module connectable to the network 2 as shown in FIG. 39 to be described later.
  • the communication module is, for example, a communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation).
  • the input unit 22 can accept input from the user.
  • the input unit 22 includes at least one input interface capable of accepting input from the user.
  • the input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with the display, a microphone, or the like.
  • the notification unit 23 notifies information.
  • the notification unit 23 includes an output unit 24 and a vibration unit 25 .
  • the notification unit 23 is not limited to the output unit 24 and the vibration unit 25 .
  • the reporting unit 23 may include any component capable of reporting information.
  • the output unit 24 and the vibration unit 25 may be mounted on the electronic device 20, or may be arranged near any one of the sensor devices 10B to 10F.
  • the output unit 24 can output data.
  • the output unit 24 includes at least one output interface capable of outputting data.
  • the output interface is, for example, a display or speaker.
  • the display is, for example, an LCD or an organic EL display.
  • the vibrating section 25 can vibrate the electronic device 20 .
  • the vibrating section 25 is configured including a vibrating element.
  • the vibrating element is, for example, a piezoelectric element or the like.
  • the storage unit 26 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them.
  • a semiconductor memory is, for example, a RAM or a ROM.
  • RAM is, for example, SRAM or DRAM.
  • ROM is, for example, EEPROM or the like.
  • the storage unit 26 may function as a main storage device, an auxiliary storage device, or a cache memory.
  • the storage unit 26 stores data used for the operation of the electronic device 20 and data obtained by the operation of the electronic device 20 .
  • the storage unit 26 stores system programs, application programs, embedded software, and the like.
  • the storage unit 26 stores data of the transformer 30 and data used in the transformer 30 as shown in FIG. 4 which will be described later.
  • the control unit 27 includes at least one processor, at least one dedicated circuit, or a combination thereof.
  • a processor may be a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a particular process.
  • the dedicated circuit is, for example, FPGA or ASIC.
  • the control unit 27 executes processing related to the operation of the electronic device 20 while controlling each unit of the electronic device 20 .
  • the control unit 27 may execute processing executed by the transformer 30 as shown in FIG. 4 which will be described later.
  • the control unit 27 receives an input instructing execution of posture angle estimation processing through the input unit 22 .
  • This input is an input that causes the electronic device 20 to execute a process of estimating the posture angle of the user's body part.
  • This input is input from the input unit 22 by a user wearing the sensor device 10, for example.
  • the user inputs this input from the input unit 22, for example, before starting walking.
  • the control unit 27 may receive an input indicating the height of the user through the input unit 22 together with an input instructing execution of this estimation process. When an input indicating the user's height is received by the input unit 22 , the control unit 27 may cause the storage unit 26 to store the received data of the user's height.
  • the control unit 27 transmits a signal instructing the start of data detection as a broadcast signal to the plurality of sensor devices 10 by the communication unit 21 . After the signal instructing the start of data detection is transmitted to the plurality of sensor devices 10 , sensor data is transmitted from at least one sensor device 10 to the electronic device 20 .
  • the control unit 27 receives sensor data from at least one sensor device 10 via the communication unit 21 .
  • the control unit 27 acquires sensor data from the sensor device 10 by receiving the sensor data from the sensor device 10 .
  • the control unit 27 acquires an estimated value of the posture angle of at least one of the plurality of body parts of the user based on the sensor data and the learning model.
  • the control unit 27 may acquire the sensor data in the global coordinate system by executing coordinate transformation on the sensor data in the local coordinate system acquired from the sensor device 10 .
  • the learning model is machine-learned so that when sensor data is input, an estimated value of the posture angle of at least one of the user's body parts is output.
  • the body part for which the learning model outputs the estimated value may be appropriately set according to the application.
  • the control unit 27 uses the transformer ( Transformer). Transformers can process time series data. The transformer will be described later with reference to FIG. However, learning models are not limited to transformers.
  • the control unit 27 may use a learning model generated by machine learning based on any machine learning algorithm.
  • the control unit 27 may acquire the time-series data of the estimated values of the posture angles of the body parts over the user's whole body by using the learning model.
  • the control unit 27 may generate a gait model from time-series data of estimated posture angles of body parts of the user and time-series data of movement speed of the user's lower back.
  • the moving speed of the user's waist is the speed of the global coordinate system.
  • the control unit 27 may acquire the time-series data of the moving speed of the user's waist by converting the sensor data detected by the sensor device 10C into data of the global coordinate system. Alternatively, the control unit 27 may acquire time-series data of the movement speed of the user's lower back using a learning model.
  • the control unit 27 When using a transformer, which will be described later, as a learning model, the control unit 27 obtains a normalized speed, which will be described later, of the user's waist from the transformer, and multiplies the obtained normalized speed by the height of the user to obtain the moving speed of the user's waist. Data may be calculated.
  • the generated walking model is a model showing how the user walks.
  • the control unit 27 may generate a walking model as a three-dimensional animation.
  • the control unit 27 may generate a walking model by scaling a humanoid model whose size is determined in advance according to the user's height.
  • the user's body parts that are used to generate the walking model may be appropriately set around the waist.
  • the user's body parts that are used to generate the walking model include the user's head, neck, chest, lumbar spine, pelvis, right and left thighs, right and left sides, as shown in FIG. lower legs, right and left feet, right and left upper arms, and right and left forearms.
  • body parts covering the whole body of the user may be set as appropriate.
  • the control unit 27 may cause the output unit 24 to output the generated walking model data. If the output unit 24 includes a display, the control unit 27 may cause the display of the output unit 24 to display the generated walking model. With such a configuration, the user can grasp how the user is walking.
  • the control unit 27 may display the generated walking model on the display of the output unit 24 as a three-dimensional animation.
  • the control unit 27 may cause the display of the output unit 24 to display the three-dimensional animation of the walking model as a free-viewpoint image based on the user's input received by the input unit 22 .
  • the user can grasp in detail how he or she walks.
  • the control unit 27 transmits the estimated value data of the user's body part acquired by the learning model or the generated walking model data to the external device via the network 2 shown in FIG.
  • a user may be being instructed by an instructor about his or her walking.
  • the estimated value data of the user's body part or the data of the walking model is transmitted to the external device, so that the instructor can grasp the user's walking state via the external device and instruct the user on walking. can be done.
  • the user may be an instructor.
  • the data of the estimated values of the body parts of the user who is the instructor or the data of the walking model is transmitted to the external device, so that the students can see the instructor walking as a model through the external device.
  • Transformer 30 as shown in FIG. 4 outputs time-series data of estimated values of posture angles of predetermined body parts of a user when a plurality of time-series sensor data are input. can learn.
  • the transformer 30 can also be trained to output time-series data of the normalized velocity of the waist in addition to the time-series data of the estimated posture angles of the user's body parts.
  • the normalized velocity of the user's waist is normalized by dividing the movement velocity of the user's waist by the height of the user.
  • the time range and time interval of the sensor data along the time series input to the transformer 30 may be set according to desired estimation accuracy and the like.
  • the transformer 30 includes an encoder 40 and a decoder 50.
  • the encoder 40 includes a functional section 41 , a functional section 42 and N stages of layers 43 .
  • Layer 43 includes functional section 44 , functional section 45 , functional section 46 , and functional section 47 .
  • the decoder 50 includes a functional section 51 , a functional section 52 , N stages of layers 53 , a functional section 60 , and a functional section 61 .
  • Layer 53 includes functional portion 54 , functional portion 55 , functional portion 56 , functional portion 57 , functional portion 58 , and functional portion 59 .
  • the number of stages of the layers 43 included in the encoder 40 and the number of stages of the layers 53 included in the decoder 50 are also N stages (N is a natural number).
  • the functional unit 41 is also described as "Input Embedding".
  • the functional unit 41 receives a plurality of time-series arrays of sensor data. For example, if the sensor data at time ti (0 ⁇ i ⁇ n) is described as “D ti ”, the array of sensor data input to the function unit 41 is expressed as (D t0 , D t1 , . . . , D tn ). be done. An array in which multiple types of sensor data are combined may be input to the function unit 41 .
  • the array of sensor data input to the function unit 41 is (Da t0 , Da t1 , . . . , Da tn , Db t0 , Db t1 , .
  • the functional unit 41 converts each element of the array of input sensor data into a multidimensional vector to generate a distributed vector.
  • the number of dimensions of the multidimensional vector may be preset.
  • the functional unit 42 is also described as "Positional Encoding".
  • the functional unit 42 gives position information to the distributed vector.
  • the functional unit 42 calculates and adds the position information to each element of the distributed vector.
  • the position information indicates the position of each element of the distributed vector in the array of sensor data input to the function unit 41 and the position in the element array of the distributed vector.
  • the functional unit 42 calculates the position information PE of the (2 ⁇ i)-th element in the array of elements of the vector expressed in a distributed manner, using Equation (1).
  • the functional unit 42 calculates the position information PE of the (2 ⁇ i+1)-th element in the array of the elements of the vector represented by the distributed representation, using Equation (2).
  • pos is the position of the element of the distributed vector in the sensor data array input to the function unit 41 .
  • dmodel is the number of dimensions of the distributed vector.
  • the first-stage layer 43 receives from the functional unit 42 a vector to which position information is added and which is expressed in a distributed manner.
  • a vector from the preceding layer 43 is input to the second and subsequent layers 43 .
  • the functional unit 44 is also described as "Multi-Head Attention".
  • a Q (Query) vector, a K (Key) vector, and a V (Value) vector are input to the functional unit 44 .
  • the Q vector is the vector input to layer 43 multiplied by the weight matrix WQ.
  • the K vector is the vector input to the layer 43 multiplied by the weight matrix WK.
  • the V vector is obtained by multiplying the vector input to the layer 43 by the weight matrix WV.
  • the transformer 30 learns the weight matrix WQ, the weight matrix WK and the weight matrix WV.
  • the functional unit 44 includes h functional units 70 and "Linear” and “Contact” functional units.
  • the functional unit 70 is also described as "Scaled Dot-Product Attention".
  • the functional unit 70 receives the Q vector, the K vector, and the V vector divided into h pieces.
  • the function section 70 includes the function sections "MatMul”, “Scale”, “Mask (opt.)” and “Softmax” as shown in FIG.
  • the functional unit 70 calculates Scaled Dot-Product Attention using the Q vector, the K vector, the V vector, and Equation (3).
  • dk is the dimensionality of the Q and K vectors.
  • Equation (4) dk is the dimensionality of the Q and K vectors. dv is the number of dimensions of the V vector.
  • the multi-head attention calculated by the functional unit 44 is input to the functional unit 45 as shown in FIG.
  • the functional unit 45 is also described as "Add & Norm".
  • the functional unit 45 normalizes the vector input to the layer 43 by adding the multi-head attention calculated by the functional unit 44 .
  • the functional unit 45 inputs the normalized vector to the functional unit 46 .
  • the functional unit 46 is also described as "Position-wise Feed-Forward Networks".
  • the functional unit 46 generates an output using an activation function such as ReLU (Rectified Linear Unit) and a vector input from the functional unit 45 .
  • the function unit 46 generates different FFNs (Feed-Forward Networks) for each position of the element array of the sensor data along the time series before vectorization, that is, the sensor data along the time series input to the function unit 41. use. Denoting the vector input from the functional unit 45 to the functional unit 46 as "x”, the functional unit 46 generates the output FFN(x) according to equation (5).
  • W1 and W2 are coefficients.
  • b1 and b2 are biases. W1 and W2 and b1 and b2 may differ for each position of the element array of the sensor data along the time series before vectorization.
  • the functional unit 47 is also described as "Add & Norm".
  • the functional unit 47 normalizes the vector output by the functional unit 45 by adding the output generated by the functional unit 46 to the vector.
  • the function unit 51 is also described as "Input Embedding".
  • the functional unit 51 receives time-series data such as the estimated values of the posture angles of the body parts output by the decoder 50 in the previous process.
  • preset data such as dummy data may be input to the function unit 51 .
  • the functional unit 51 generates a distributed vector by converting each element of the input time-series data into a multidimensional vector in the same or similar manner as the functional unit 41 .
  • the number of dimensions of the multidimensional vector may be preset in the same or similar manner as the functional unit 41 .
  • the functional unit 52 is also described as "Positional Encoding".
  • the functional unit 52 in the same or similar manner as the functional unit 42 , gives position information to the vector expressed in a distributed manner. In other words, the functional unit 52 calculates and adds the position information to each element of the distributed vector.
  • the position information indicates the position of each element of the distributed representation of the vector in the array of the time-series data input to the function unit 51 and the position of the distributed representation of the vector in the element array.
  • the first-stage layer 53 is supplied with a vector to which position information is added and which is expressed in a distributed manner from the functional unit 52 .
  • a vector from the preceding layer 53 is input to the second and subsequent layers 53 .
  • the function unit 54 is also described as "Masked Multi-Head Attention".
  • the Q vector, the K vector, and the V vector are input to the functional unit 54 in the same or similar manner as the functional unit 44 .
  • the Q vector, K vector and V vector are the vectors input to layer 53 multiplied by the same or different weight matrices, respectively.
  • Transformer 30 learns these weight matrices during training.
  • Functional unit 54 identical or similar to functional unit 44, calculates multi-head attention from the input Q vector, K vector and V vector.
  • time-series data such as correct posture angles of body parts are input to the function unit 54 at a time when the transformer 30 learns.
  • the functional unit 54 masks the time data after the time data to be estimated by the decoder 50 in the time-series data such as the posture angle of the body part.
  • the functional unit 55 is also described as "Add & Norm".
  • the functional unit 55 normalizes the vector input to the layer 53 by adding the multi-head attention calculated by the functional unit 54 .
  • the functional unit 56 is also described as "Multi-Head Attention".
  • a Q vector, a K vector and a V vector are input to the functional unit 56 .
  • the Q vector is a normalized vector that the functional unit 55 inputs to the functional unit 56 .
  • the K vector and the V vector are obtained by multiplying the vectors output from the final stage layer 43 of the encoder 40 by the same or different weight matrices.
  • Functional unit 56 identical or similar to functional unit 44 , calculates multi-head attention from the input Q vector, K vector and V vector.
  • the functional unit 57 is also described as "Add & Norm".
  • the functional unit 57 normalizes the vector output by the functional unit 55 by adding the multi-head attention calculated by the functional unit 56 .
  • the functional unit 58 is also described as "Position-wise Feed-Forward Networks".
  • Functional unit 58 identical or similar to functional unit 46 , generates an output using an activation function such as ReLU and a vector input from functional unit 57 .
  • the functional unit 59 is also described as "Add & Norm".
  • the functional unit 59 normalizes the vector output by the functional unit 57 by adding the output generated by the functional unit 58 to the vector.
  • the functional unit 60 is also described as "Linear”.
  • the functional unit 61 is also described as “SoftMax”.
  • the output of the final stage layer 53 is normalized by the functional unit 60 and the functional unit 61, and then output from the decoder 50 as data of estimated values such as posture angles of body parts.
  • the walking speed differs depending on the user. That is, the walking cycle differs depending on the user.
  • a walking cycle is a period from when one foot of the user's two feet lands on the ground or the like until it lands on the ground or the like again.
  • the transformer 30 can learn which part of the walking cycle the sensor data corresponds to, even if the time-series data of the input sensor data is part of the walking cycle. Therefore, the transformer 30 can be learned to output the time-series data of the estimated value of the attitude angle when the time-series data of the sensor data of about half the average value of the walking cycle is input.
  • the control unit 27 may use a transformer that has learned one type of sensor data, or may use a transformer that has learned a combination of multiple types of sensor data. Combinations of multiple types of sensor data are, for example, cases C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, and C13 as shown in FIG.
  • Fig. 7 shows an example of a combination of sensor data.
  • Cases C1 to C13 are examples of combinations of sensor data.
  • the control unit 27 may select one of the cases C1 to C13 according to the type of the sensor device 10 that has transmitted the sensor data to the electronic device 20.
  • the data of the transformer 30 used in each of the cases C1 to C13 may be stored in the storage unit 26 in association with each of the cases C1 to C13.
  • the control unit 27 inputs the sensor data of one of the selected cases C1 to C13 to the transformer 30 corresponding to one of the selected cases C1 to C13, thereby obtaining an estimated value of the posture angle of the body part of the user. get.
  • the control unit 27 may select case C1.
  • sensor data indicating the movement of the user's head is used.
  • sensor data D10AG and sensor data D10AL are used.
  • the sensor data D10AG is sensor data indicating the movement of the user's head in the global coordinate system.
  • the sensor data D10AG is the velocity data and acceleration data of the user's head on the X-axis, the velocity data and acceleration data of the user's head on the Y-axis, and the velocity data of the user's head on the Z-axis of the global coordinate system. and acceleration data.
  • the control unit 27 acquires sensor data D10AG by executing coordinate transformation on the sensor data in the local coordinate system acquired from the sensor device 10A.
  • the sensor data D10AL is sensor data indicating the movement of the user's head in the local coordinate system based on the position of the sensor device 10A.
  • the sensor data D10AL is the velocity data and acceleration data of the user's head on the x-axis, the velocity data and acceleration data of the user's head on the y-axis, and the velocity of the user's head on the z-axis of the local coordinate system. data and acceleration data.
  • the control unit 27 acquires the sensor data D10AL from the sensor device 10A.
  • the control unit 27 may select case C2.
  • sensor data indicating movement of the user's head and sensor data indicating movement of one of the user's two ankles are used.
  • sensor data D10AG, sensor data D10AL, and sensor data D10EL-1 or sensor data D10EL-2 are used.
  • the sensor data D10EL-1 is sensor data indicating the movement of the user's left ankle in the local coordinate system based on the position of the sensor device 10E-1.
  • the sensor data D10EL-1 is the velocity data and acceleration data of the user's left ankle on the x-axis, the velocity data and acceleration data of the user's left ankle on the y-axis, and the user's left ankle on the z-axis of the local coordinate system. Includes velocity and acceleration data for the left ankle.
  • the control unit 27 acquires the sensor data D10EL-1 from the sensor device 10E-1.
  • the sensor data D10EL-2 is sensor data indicating the movement of the user's right ankle in the local coordinate system based on the position of the sensor device 10E-2.
  • the sensor data D10EL-2 is the velocity data and acceleration data of the user's right ankle on the x-axis, the velocity data and acceleration data of the user's right ankle on the y-axis, and the user's right ankle on the z-axis of the local coordinate system. Includes velocity and acceleration data for the right ankle.
  • the control unit 27 acquires the sensor data D10EL-2 from the sensor device 10E-2.
  • the control unit 27 may select case C3.
  • sensor data indicating movement of the user's head and sensor data indicating movement of one of the user's two feet are used.
  • sensor data D10AG, sensor data D10AL, and sensor data D10FL-1 or sensor data D10FL-2 are used.
  • the sensor data D10FL-1 is sensor data indicating the movement of the user's left foot in the local coordinate system based on the position of the sensor device 10F-1.
  • the sensor data D10FL-1 is the velocity data and acceleration data of the user's left foot on the x-axis, the velocity data and acceleration data of the user's left foot on the y-axis, and the z-axis on the local coordinate system. Includes velocity and acceleration data for the user's left foot.
  • the control unit 27 acquires the sensor data D10FL-1 from the sensor device 10F-1.
  • the sensor data D10FL-2 is sensor data indicating the movement of the user's right foot in the local coordinate system based on the position of the sensor device 10F-2.
  • the sensor data D10FL-2 is the velocity data and acceleration data of the user's right foot on the x-axis, the velocity data and acceleration data of the user's right foot on the y-axis, and the z-axis on the local coordinate system. Includes velocity and acceleration data for the user's right foot.
  • the control unit 27 acquires the sensor data D10FL-2 from the sensor device 10F-2.
  • the control unit 27 may select case C4.
  • sensor data indicating movement of the user's head and sensor data indicating movement of one of the user's two thighs are used.
  • sensor data D10AG, sensor data D10AL, and sensor data D10DL-1 or sensor data D10DL-2 are used.
  • the sensor data D10DL-1 is sensor data indicating the movement of the user's left thigh in the local coordinate system based on the position of the sensor device 10D-1.
  • the sensor data D10DL-1 is the velocity data and acceleration data of the user's left thigh on the x-axis, the velocity data and acceleration data of the user's left thigh on the y-axis, and z Includes velocity and acceleration data for the user's left thigh on axis.
  • the control unit 27 acquires sensor data D10DL-1 from the sensor device 10D-1.
  • the sensor data D10DL-2 is sensor data indicating the movement of the user's right thigh in the local coordinate system based on the position of the sensor device 10D-2.
  • the sensor data D10DL-2 is the velocity data and acceleration data of the user's right thigh on the x-axis, the velocity data and acceleration data of the user's right thigh on the y-axis, and z Includes velocity and acceleration data for the user's right thigh on axis.
  • the control unit 27 acquires the sensor data D10DL-2 from the sensor device 10D-2.
  • the control unit 27 may select case C5.
  • sensor data indicating the movement of the user's head and sensor data indicating the movement of one of the user's two wrists are used.
  • sensor data D10AG, sensor data D10AL, and sensor data D10BL are used.
  • the sensor data D10BL is sensor data indicating the movement of the user's wrist in the local coordinate system based on the position of the sensor device 10B.
  • the sensor data D10BL is assumed to be sensor data indicating the movement of the user's left wrist.
  • the sensor data D10BL may be sensor data indicating the movement of the user's right wrist.
  • the sensor data D10BL is the velocity data and acceleration data of the user's wrist on the x-axis, the velocity data and acceleration data of the user's wrist on the y-axis, and the velocity data and acceleration of the user's wrist on the z-axis of the local coordinate system. including data and
  • the control unit 27 acquires the sensor data D10BL from the sensor device 10B.
  • the control unit 27 may select case C6.
  • sensor data indicating movement of the user's head sensor data indicating movement of one of the user's two wrists, and one of the user's two ankles. sensor data indicating the movement of the
  • sensor data D10AG, sensor data D10AL, sensor data D10BL, and sensor data D10EL-1 or sensor data D10EL-2 are used.
  • the control unit 27 may select case C7.
  • sensor data indicating movement of the user's head sensor data indicating movement of one of the two wrists of the user, and sensor data indicating movement of one of the two wrists of the user.
  • Sensor data indicative of foot movement is used.
  • sensor data D10AG, sensor data D10AL, sensor data D10BL, and sensor data D10FL-1 or sensor data D10FL-2 are used.
  • the control unit 27 may select case C8.
  • the sensor data indicating the movement of the user's head the sensor data indicating the movement of one of the user's two wrists, and the movement of each of the user's two feet. is used.
  • sensor data D10AG, sensor data D10AL, sensor data D10BL, sensor data D10FL-1, and sensor data D10FL-2 are used.
  • the control unit 27 may select case C9.
  • sensor data indicating the movement of each of the user's two feet is used.
  • sensor data D10FL-1 and sensor data D10FL-2 are used.
  • the control unit 27 may select case C10.
  • sensor data indicating the movement of each of the user's two thighs is used.
  • sensor data D10DL-1 and sensor data D10DL-2 are used.
  • the sensor data D10DL-1 is sensor data indicating the movement of the user's left thigh in the local coordinate system based on the position of the sensor device 10D-1.
  • the sensor data D10DL-1 is the velocity data and acceleration data of the user's left thigh on the x-axis, the velocity data and acceleration data of the user's left thigh on the y-axis, and z Includes velocity and acceleration data for the user's left thigh on axis.
  • the control unit 27 acquires sensor data D10DL-1 from the sensor device 10D-1.
  • the sensor data D10DL-2 is sensor data indicating the movement of the user's right thigh in the local coordinate system based on the position of the sensor device 10D-2.
  • the sensor data D10DL-2 is the velocity data and acceleration data of the user's right thigh on the x-axis, the velocity data and acceleration data of the user's right thigh on the y-axis, and z Includes velocity and acceleration data for the user's right thigh on axis.
  • the control unit 27 acquires the sensor data D10DL-2 from the sensor device 10D-2.
  • the control unit 27 may select case C11.
  • sensor data indicating the movement of the user's lower back is used.
  • sensor data D10CG and sensor data D10CL are used.
  • the sensor data D10CG is sensor data indicating the movement of the user's lower back in the global coordinate system.
  • the sensor data D10CG is the user's waist velocity data and acceleration data on the X-axis, the user's waist velocity data and acceleration data on the Y-axis, and the user's waist velocity data and acceleration data on the Z-axis of the global coordinate system.
  • the control unit 27 may acquire the sensor data D10CG by executing coordinate transformation on the sensor data in the local coordinate system acquired from the sensor device 10C.
  • the sensor data D10CL is sensor data indicating the movement of the user's lower back in the local coordinate system based on the position of the sensor device 10C.
  • the sensor data D10CL includes velocity data and acceleration data of the user's waist on the x-axis, velocity data and acceleration data of the user's waist on the y-axis, and velocity data and acceleration of the user's waist on the z-axis of the local coordinate system. including data and
  • the control unit 27 acquires the sensor data D10CL from the sensor device 10C.
  • the control unit 27 may select case C12.
  • sensor data indicating movement of one of the user's two wrists and sensor data indicating movement of the user's lower back are used.
  • sensor data D10BL, sensor data D10CG, and sensor data D10CL are used.
  • the control unit 27 may select case C13.
  • sensor data indicating movement of one of the user's two wrists sensor data indicating movement of each of the user's two legs, and sensor data indicating movement of the user's lower back. and are used.
  • sensor data D10BL, sensor data D10FL-1, sensor data D10FL-2, sensor data D10CG, and sensor data D10CL are used.
  • Transformer generation and evaluation is described below.
  • the inventors have generated and evaluated a transformer that outputs estimates of the normalized velocities of the waist in addition to estimates of the posture angles of the body parts over the whole body of the user.
  • the user's whole body parts include the user's head, neck, chest, lumbar spine, pelvis, right and left thighs, right and left lower legs, right and left feet, right and left sides. Includes upper arm and right and left forearms.
  • the subject's gait database was used to generate the transformer.
  • As a walking database of subjects "Yoshiyuki Kobayashi, Naoto Hida, Kanako Nakajima, Masahiro Fujimoto, Masaaki Mochimaru, '2019: AIST Walking Database 2019', [Online], [Searched on November 11, 2021], Internet ⁇ The data provided at https://unit.aist.go.jp/harc/ExPART/GDB2019_e.html> were used.
  • the gait data of a plurality of subjects are registered in this gait database.
  • the walking data of the subject includes data indicating the movement of the subject during walking and data of the floor reaction force applied to the subject during walking. Data representing the subject's motion while walking was detected by a motion capture system. The data of the floor reaction force applied to the subject while walking was detected by a floor reaction force meter.
  • the sensor data D10EL-1 representing the movement of the user's left ankle was used out of the sensor data D10EL-1 and sensor data D10EL-2.
  • the sensor data D10FL-1 indicating the movement of the left foot was used out of the sensor data D10FL-1 and sensor data D10FL-2.
  • sensor data D10DL-1 indicating movement of the left thigh was used.
  • Transformer training was performed using the generated dataset. In training the transformer, noise of about 10% was added to the dataset to prevent overfitting.
  • the inventors evaluated the trained transformer using a data set that was not used for transformer training.
  • the inventors obtained evaluation results for cases C1 to C13 described above with reference to FIG.
  • FIG. 8 shows a graph of evaluation results.
  • FIG. 8 shows a bar graph of the mean squared error (MSE) of the estimated values of the posture angles of the body parts over the whole body of the subject in each of the cases C1 to C13 as the evaluation results.
  • the mean squared error data shown in FIG. 8 are data obtained from subjects shown in FIG. 9 described later.
  • the mean squared error was calculated from the transformer estimates of posture angle and hip velocity and the measured posture angle and hip velocity from the dataset.
  • the mean squared error was calculated by Equation (6) below.
  • Equation (6) j corresponds to the X, Y and Z axes of the Grohl coordinate system.
  • d is 3, the number of dimensions of the global coordinate system.
  • ai ,j is the measured value of the posture angle of the body part or the measured value of the moving speed of the waist.
  • b i,j is the estimated body part pose angle where a i,j is the measured body part pose angle.
  • b i,j is the estimate of the waist movement speed where a i,j is the measured waist movement speed.
  • n is the number of attitude angle samples.
  • case C1 the mean squared error was 6.328 [(deg) 2 ].
  • case C1 only sensor data indicating movement of the user's head is used. From the result of case C1, it can be seen that the posture angles of the body parts of the user's whole body can be estimated with a certain degree of accuracy using only the sensor data indicating the movement of the user's head. The reason for this is presumed to be that the movement of the user's head while walking is reflected in the movement of the user's head.
  • the mean squared errors of cases C2 to C8 are smaller than the mean squared error of case C1. That is, in cases C2 to C8, the accuracy of estimating the posture angle of the user's body part is improved as compared to case C1.
  • cases C2 to C8 in addition to sensor data indicating movement of the user's head, sensor data indicating movement of at least one of the user's wrists, ankles, feet, and thighs is used. .
  • the sensor data shown is used in cases C2 to C8 in addition to the sensor data indicating the movement of the trunk including the user's head.
  • the sensor data indicating the movement of the user's limbs and the sensor data indicating the movement of the user's trunk have significantly different patterns in one walking cycle. For example, due to the left-right symmetry of the user's body, sensor data indicating the movement of the user's limbs has one pattern in one walking cycle. On the other hand, sensor data indicating the movement of the trunk of the user has two patterns in one walking cycle. In Cases C2 to C8, it is presumed that the accuracy of estimating the posture angle of the body part is improved as compared to Case C1 by using sensor data having different patterns in one walking cycle.
  • the mean squared error was 4.173 [(deg) 2 ]. Also, in case C10, the mean squared error was 2.544 [(deg) 2 ].
  • sensor data indicating movement of the user's leg and sensor data indicating movement of the user's thigh are used, respectively.
  • the feet and thighs are body parts that are highly relevant to walking among the user's body parts.
  • sensor data indicating movements of the legs or thighs, which are body parts highly relevant to walking are used, so it is presumed that the posture angles of the body parts could be estimated with a certain degree of accuracy. .
  • case C11 the mean squared error was 2.527 [(deg) 2 ].
  • case C11 only sensor data indicating the movement of the waist is used. From the result of case C11, it can be seen that the posture angles of the user's body parts can be estimated with a certain degree of accuracy even with only the sensor data indicating the movement of the user's waist. The reason for this is presumed to be that the vertical movement of the user while walking is reflected in the movement of the trunk including the waist.
  • the mean square errors of cases C12 and C13 are smaller than the mean square error of case C11. That is, in cases C12 and C13, the accuracy of estimating the posture angle of the body part is improved as compared to case C11.
  • sensor data indicating movement of the user's lower back in addition to sensor data indicating movement of the user's lower back, sensor data indicating movement of at least one of the user's wrist and/or ankle is used. That is, in cases C12 and C13, in addition to the sensor data indicating the movement of the user's trunk including the waist, the sensor data indicating the movement of the user's limbs including at least one of the wrist and ankle are used.
  • the sensor data indicating the movement of the user's limbs and the sensor data indicating the movement of the trunk of the user have significantly different patterns in one walking cycle.
  • cases C12 and C13 it is presumed that the accuracy of estimating the posture angle of the body part is improved more than in case C11 by using sensor data having different patterns in one walking cycle.
  • the mean square error was less than 1.0 [(deg) 2 ].
  • the accuracy of posture angle estimation for C06, C07, C08, and C13 was high among cases C1 to C13.
  • Fig. 9 shows an example of a subject. Subjects have a wide variety of physical characteristics.
  • Subject SU1 is male, 33 years old, 171 cm tall, and weighs 100 kg.
  • a physical characteristic of subject SU1 was that he was a heavy male.
  • Subject SU2 is female, age 70, height 151 [cm], and weight 39 [kg]. Subject SU2's physical characteristics were that she was a light weight female.
  • Subject SU3 is female, 38 years old, 155 cm tall, and weighs 41 kg. The physical characteristics of subject SU3 were that she was light in weight and young in age.
  • Test subject SU4 is female, age 65, height 149 [cm], and weight 70 [kg]. The physical characteristics of subject SU4 were that she was a heavy female.
  • Subject SU5 is male, 22 years old, 163 cm tall, and weighs 65 kg. The physical characteristics of subject SU5 were that he was a man of average height and weight.
  • Subject SU6 is female, age 66, height 149 [cm], and weight 47 [kg].
  • a physical characteristic of subject SU6 was that she was a short woman.
  • Test subject SU7 is female, age 65, height 148 [cm], and weight 47 [kg].
  • a physical characteristic of subject SU7 was that she was a short woman.
  • Subject SU8 is male, 57 years old, 178 cm tall, and weighs 81 kg. The physical characteristics of subject SU8 were that he was a tall man.
  • FIGS. 10 to 21 show graphs of measured values and estimated values of posture angles of body parts of subject SU6.
  • the horizontal axis in FIGS. 10 to 21 is time [s].
  • the vertical axis in FIGS. 10 to 21 is the attitude angle [deg].
  • FIG. 10 to 15 are graphs of posture angles of body parts of the upper body of subject SU6.
  • FIG. 10 is a graph of the posture angle of the neck of subject SU6.
  • FIG. 11 is a graph of posture angles of the chest of subject SU6.
  • FIG. 12 is a graph of the posture angle of the right upper arm of subject SU6.
  • FIG. 13 is a graph of posture angles of the left upper arm of subject SU6.
  • FIG. 14 is a graph of the posture angle of the right forearm of subject SU6.
  • FIG. 15 is a graph of the posture angle of the left forearm of subject SU6.
  • FIG. 16 to 21 are graphs of posture angles of body parts of the lower body of subject SU6. Specifically, FIG. 16 is a graph of the posture angle of the right thigh of subject SU6.
  • FIG. 17 is a graph of the posture angle of the left thigh of subject SU6.
  • FIG. 18 is a graph of the posture angle of the right lower leg of subject SU6.
  • FIG. 19 is a graph of the posture angle of the left lower leg of subject SU6.
  • FIG. 20 is a graph of the posture angle of the right foot of subject SU6.
  • FIG. 21 is a graph of the posture angle of the left foot of subject SU6.
  • the posture angle ⁇ Xr is the measured value of the posture angle ⁇ X described above.
  • the posture angle ⁇ Yr is the measured value of the posture angle ⁇ Y described above.
  • the posture angle ⁇ Zr is the measured value of the posture angle ⁇ Z described above.
  • the attitude angle ⁇ Xe is the estimated value of the attitude angle ⁇ X described above.
  • the posture angle ⁇ Ye is the estimated value of the posture angle ⁇ Y described above.
  • the attitude angle ⁇ Ze is the estimated value of the attitude angle ⁇ Z described above.
  • the estimated values of the posture angles of the body parts of the upper body of subject SU6 agreed relatively well with the measured values.
  • the sensor data indicating the movement of the subject's left wrist was used as the sensor data D10BL for case C6 shown in FIG.
  • sensor data indicating the movement of the subject's right wrist is not used.
  • the estimated posture angle of the right upper arm of subject SU6 agreed with the measured value with the same or similar accuracy as the left upper arm shown in FIG.
  • the estimated value of the posture angle of the right forearm of subject SU6 agreed with the measured value with the same or similar accuracy as that of the left forearm shown in FIG.
  • the estimated values of the posture angles of the body parts of the lower body of subject SU6 agreed relatively well with the measured values. As described above, in case C6, no sensor data indicating movement of the subject's right wrist is used. Nevertheless, as shown in FIG. 16, the estimated value of the posture angle of the right thigh of subject SU6 matched the measured value with the same or similar accuracy as the left thigh shown in FIG. . In addition, as shown in FIG. 18, the estimated value of the posture angle of the right lower leg of subject SU6 matched the measured value with the same or similar accuracy as that of the left lower leg shown in FIG. As shown in FIG. 20, the estimated value of the posture angle of the right foot of subject SU6 matched the measured value with the same or similar accuracy as the left foot shown in FIG.
  • a subject with a high center-of-gravity movement evaluation means a subject whose center-of-gravity movement in the vertical direction is large during walking.
  • a subject with a low center-of-gravity shift evaluation means a subject whose center-of-gravity shift in the vertical direction is small during walking.
  • FIG. 22 is a graph of the posture angle of the right thigh of subject SU7 with a high evaluation of center of gravity movement.
  • FIG. 23 is a graph of the posture angle of the right thigh of subject SU1 with a high evaluation of center-of-gravity movement.
  • FIG. 24 is a graph of the posture angle of the right thigh of subject SU3 with a high evaluation of center-of-gravity movement.
  • FIG. 25 is a graph of the posture angle of the right thigh of subject SU6 with a high evaluation of the movement of the center of gravity.
  • FIG. 26 is a graph of the posture angle of the right thigh of subject SU5 with a low evaluation of center-of-gravity movement.
  • FIG. 27 is a graph of the posture angle of the right thigh of subject SU2 with a low evaluation of center-of-gravity movement.
  • FIG. 28 is a graph of the posture angle of the right thigh of subject SU4 with a low center of gravity shift evaluation.
  • FIG. 29 is a graph of the posture angle of the right thigh of subject SU8 with a low center of gravity shift evaluation.
  • the horizontal axis in FIGS. 22 to 29 is time [s].
  • the vertical axis in FIGS. 22 to 29 is the posture angle [deg].
  • Subjects with low center-of-gravity movement evaluations have less center-of-gravity movement in the vertical direction and less movement in the vertical direction than subjects with high center-of-gravity movement evaluations. Therefore, for a subject with a low evaluation of the movement of the center of gravity, the movement of the subject in the vertical direction during walking is less likely to be reflected in the sensor data than a subject with a high evaluation of the movement of the center of gravity. Nevertheless, as shown in FIGS. 22 to 29, the right thigh posture angle estimates in subjects with low center-of-gravity-shift evaluations were obtained with the same or similar accuracy as subjects with high center-of-mass-shift evaluations. , agreed with the measured values.
  • FIG. 30 is a graph of the posture angle of the right upper arm of subject SU7 with a high evaluation of center of gravity movement.
  • FIG. 31 is a graph of the posture angle of the upper arm on the right side of subject SU1 with a high evaluation of center-of-gravity movement.
  • FIG. 32 is a graph of the posture angle of the right upper arm of subject SU3 with a high evaluation of center of gravity movement.
  • FIG. 33 is a graph of the posture angle of the right upper arm of subject SU6 with a high evaluation of center of gravity movement.
  • FIG. 34 is a graph of the posture angle of the right upper arm of subject SU5 with a low evaluation of the movement of the center of gravity.
  • FIG. 35 is a graph of the posture angle of the right upper arm of subject SU2 with a low evaluation of center of gravity movement.
  • FIG. 36 is a graph of the posture angle of the right upper arm of subject SU4 with a low evaluation of center of gravity movement.
  • FIG. 37 is a graph of the posture angle of the right upper arm of subject SU8 with a low evaluation of center of gravity movement.
  • the horizontal axis in FIGS. 30 to 37 is time [s].
  • the vertical axis in FIGS. 30 to 37 is the posture angle [deg].
  • the subject's vertical movement is small, and the vertical movement of the subject during walking is reflected in the sensor data more than in subjects with high center-of-gravity shift evaluations. Hateful. Nevertheless, as shown in FIGS. 30 to 37 , the estimated value of the posture angle of the right upper arm in subjects with low center-of-gravity movement evaluations is the same or similar to that of subjects with high center-of-gravity movement evaluations, It agrees with the measured value.
  • the control unit 27 may determine the evaluation of the walking of the user based on the estimated values of the posture angles of the body parts of the user. As an example, the control unit 27 may estimate the amount of movement of the user's center of gravity in the vertical direction based on the estimated values of the posture angles of the user's body parts, and determine the evaluation of the movement of the user's center of gravity.
  • the control unit 27 may notify the user of the determined evaluation by the notification unit 23 .
  • the control unit 27 may cause the determined evaluation information to be displayed on the display of the output unit 24, or may output the determined evaluation information to the speaker of the output unit 24 as voice.
  • the control unit 27 may vibrate the vibration unit 25 in a vibration pattern according to the determined evaluation.
  • the control unit 27 may generate an evaluation signal indicating the determined evaluation.
  • the control unit 27 may transmit the generated evaluation signal to any external device through the communication unit 21 .
  • the control unit 27 may transmit the evaluation signal through the communication unit 21 to any sensor device 10 having the notification unit 13 as an external device.
  • the control section 16 receives the evaluation signal through the communication section 11 .
  • the control unit 16 causes the notification unit 13 to notify the information indicated by the evaluation signal.
  • the control unit 16 causes the output unit 14 to output information indicated by the evaluation signal.
  • the control unit 27 may transmit the evaluation signal to the earphone as an external device through the communication unit 21.
  • the control unit 16 may cause the speaker of the output unit 14 to output information indicated by the evaluation signal as voice.
  • FIG. 38 is a flowchart showing operations of posture angle estimation processing executed by electronic device 20 shown in FIG. This operation corresponds to an example of the information processing method according to this embodiment. For example, when the user inputs an input instructing execution of posture angle estimation processing from the input unit 22, the control unit 27 starts the processing of step S1.
  • the control unit 27 receives an input instructing execution of posture angle estimation processing through the input unit 22 (step S1).
  • the control unit 27 transmits a signal instructing the start of data detection as a broadcast signal to the plurality of sensor devices 10 through the communication unit 21 (step S2). After the process of step S ⁇ b>2 is executed, sensor data is transmitted from at least one sensor device 10 to the electronic device 20 .
  • the control unit 27 receives sensor data from at least one sensor device 10 through the communication unit 21 (step S3).
  • the control unit 27 selects one of the cases C1 to C13 according to the type of the sensor device 10 that has transmitted the sensor data to the electronic device 20 (step S4).
  • the control unit 27 acquires the data of the transformer 30 used in the cases C1 to C13 selected in the process of step S4 from the storage unit 26 (step S5).
  • the control unit 27 inputs the sensor data of cases C1 to C13 selected in the process of step S4 to the transformer from which the data was acquired in the process of step S5.
  • the control unit 27 inputs the sensor data to the transformer, and acquires from the transformer time-series data of the estimated values of the posture angles of the body parts of the user and time-series data of the movement speed of the user's lower back (step S6).
  • the control unit 27 generates a gait model based on the time-series data of the estimated posture angles of the user's whole body and the time-series data of the movement speed of the user's lower back acquired in step S6 (step S7).
  • the control unit 27 causes the output unit 24 to output the walking model data generated in the process of step S7 (step S8).
  • the control unit 27 After executing the process of step S8, the control unit 27 terminates the estimation process. After ending the estimation process, the control unit 27 may perform the estimation process again. In the estimation process to be executed again, the control unit 27 may start from the process of step S3. The control unit 27 may repeatedly execute the estimation process until an input instructing to end the estimation process is received from the input unit 22 .
  • An input instructing to end the estimation process is input from the input unit 22 by the user, for example. For example, when the user finishes walking, the user inputs an instruction to finish the estimation process from the input unit 22 .
  • the control unit 27 may transmit a signal instructing the end of data detection as a broadcast signal to the plurality of sensor devices 10 by the communication unit 21 . In the sensor device 10 , the control unit 16 may terminate data detection when the communication unit 11 receives a signal instructing termination of data detection.
  • the control unit 27 acquires the estimated values of the posture angles of the body parts of the user based on the sensor data and the learning model. By obtaining an estimate of the pose angles of the user's body parts, the motion of the user can be detected. Further, from the results shown in FIGS. 8 to 37, the estimated value of the posture angle of one of the left and right body parts of the user is obtained by using the sensor data indicating the movement of one of the body parts and the learning model. It turns out that it is obtainable.
  • the data captured by the camera is analyzed, and the user's movement is detected.
  • it is required to install a camera. Setting up the camera is time consuming.
  • it is possible to photograph the user's walking only at the place where the camera is installed.
  • the movement of the user's body may be obscured by the clothing. If the movement of the user's body cannot be seen from the outside, the camera may not be able to capture the movement of the user's body.
  • the present embodiment does not need to install a camera, so the user's movement can be detected more easily.
  • the sensor device 10 is worn by the user, the sensor data can be detected, so the movement of the user can be detected.
  • the sensor data can be detected as long as the sensor device 10 is worn by the user, so the movement of the user can be detected.
  • the sensor device 10 is worn by the user, it is possible to detect the movement of the user regardless of where the user walks and how long the user walks.
  • the transformer may have learned to output an estimated value of the posture angle of the user's body part when the sensor data of case C1 is input.
  • the sensor data of case C1 is detected by the sensor device 10A.
  • the transformer may be learned to output an estimated value of the posture angle of the user's body part when sensor data of any one of cases C2 to C5 is input.
  • the sensor data of case C2 is detected by the sensor device 10A and the sensor device 10E-1 or sensor device 10E-2, that is, by the two sensor devices 10.
  • FIG. The sensor data of case C3 is detected by the sensor device 10A and the sensor device 10F-1 or sensor device 10F-2, that is, by the two sensor devices 10.
  • the sensor data of case C4 is detected by the sensor device 10A and the sensor device 10D-1 or sensor device 10D-2, that is, by the two sensor devices 10.
  • the sensor data of case C5 is detected by the sensor device 10A and the sensor device 10B, that is, by the two sensor devices 10.
  • FIG. In cases C2 to C5, the sensor data is detected by the two sensor devices 10 as described above, so the user only has to wear the two sensor devices 10. FIG. Therefore, user convenience can be improved.
  • the accuracy of estimating the posture angle of the body part of the user is improved as compared to case C1. Therefore, by using the sensor data of cases C2 to C5, it is possible to accurately estimate the posture angles of the user's body parts.
  • the transformer may be learned to output the estimated value of the posture angle of the user's body part when the sensor data of either case C6 or C7 is input.
  • the sensor data of case C6 is detected by the sensor device 10A, the sensor device 10B, and the sensor device 10E-1 or the sensor device 10E-2, that is, by the three sensor devices 10.
  • the sensor data of case C7 is detected by the sensor device 10A, the sensor device 10B, and the sensor device 10F-1 or sensor device 10F-2, that is, by three sensor devices 10.
  • the sensor data is detected by the three sensor devices 10, so the user only has to wear the three sensor devices 10. FIG. Therefore, user convenience can be improved.
  • FIG. 39 is a functional block diagram showing the configuration of an information processing system 101 according to another embodiment of the present disclosure.
  • the information processing system 101 includes a sensor device 10, an electronic device 20, and a server 80.
  • the server 80 functions as an information processing device, and acquires the estimated values of the posture angles of the body parts of the user based on the sensor data detected by the sensor device 10 and the learning model.
  • the electronic device 20 and the server 80 can communicate via the network 2.
  • the network 2 may be any network including mobile communication networks, the Internet, and the like.
  • the control unit 27 of the electronic device 20 receives sensor data from the sensor device 10 via the communication unit 21 in the same or similar manner as the information processing system 1 .
  • the control unit 27 transmits sensor data to the server 80 via the network 2 using the communication unit 21 .
  • the server 80 is, for example, a server belonging to a cloud computing system or other computing system.
  • the server 80 includes a communication section 81 , a storage section 82 and a control section 83 .
  • the communication unit 81 includes at least one communication module connectable to the network 2.
  • the communication module is, for example, a communication module conforming to a standard such as wired LAN (Local Area Network) or wireless LAN.
  • the communication unit 81 is connected to the network 2 via a wired LAN or wireless LAN by a communication module.
  • the storage unit 82 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them.
  • a semiconductor memory is, for example, a RAM or a ROM.
  • RAM is, for example, SRAM or DRAM.
  • ROM is, for example, EEPROM or the like.
  • the storage unit 82 may function as a main storage device, an auxiliary storage device, or a cache memory.
  • the storage unit 82 stores data used for the operation of the server 80 and data obtained by the operation of the server 80 .
  • the storage unit 82 stores system programs, application programs, embedded software, and the like.
  • the storage unit 82 stores data of the transformer 30 and data used in the transformer 30 as shown in FIG.
  • the control unit 83 includes at least one processor, at least one dedicated circuit, or a combination thereof.
  • a processor may be a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a particular process.
  • the dedicated circuit is, for example, FPGA or ASIC.
  • the control unit 83 executes processing related to the operation of the server 80 while controlling each unit of the server 80 .
  • the control unit 83 may execute the processing executed by the transformer 30 as shown in FIG.
  • the control unit 83 receives sensor data from the electronic device 20 via the network 2 using the communication unit 81 .
  • the control unit 83 acquires the estimated values of the posture angles of the body parts of the user based on the sensor data and the learning model by executing processing that is the same as or similar to the processing by the control unit 27 of the electronic device 20 described above.
  • FIG. 40 is a sequence diagram showing operations of estimation processing executed by the information processing system 101 shown in FIG. This operation corresponds to an example of the information processing method according to this embodiment. For example, when the user inputs an instruction to execute the posture angle estimation process from the input unit 22 of the electronic device 20, the information processing system 101 starts the process of step S1.
  • control unit 27 receives an input instructing execution of posture angle estimation processing through the input unit 22 (step S11).
  • the control unit 27 transmits a signal instructing the start of data detection as a broadcast signal to the plurality of sensor devices 10 through the communication unit 21 (step S12).
  • control unit 16 receives the signal instructing the start of data detection from the electronic device 20 by the communication unit 11 (step S13). Upon receiving this signal, the control section 16 starts data detection. The control unit 16 acquires data detected by the sensor unit 12 from the sensor unit 12 . The control unit 16 transmits the acquired data as sensor data to the electronic device 20 through the communication unit 11 (step S14).
  • control unit 27 receives the sensor data from the sensor device 10 through the communication unit 21 (step S15).
  • the control unit 27 transmits the sensor data to the server 80 via the network 2 using the communication unit 21 (step S16).
  • the control unit 83 receives the sensor data from the electronic device 20 via the network 2 by the communication unit 81 (step S17).
  • the control unit 83 selects one of the cases C1 to C13 according to the type of the sensor device 10 that has transmitted the sensor data to the server 80 via the electronic device 20 (step S18).
  • the control unit 83 acquires the data of the transformer 30 used in the cases C1 to C13 selected in the process of step S18 from the storage unit 82 (step S19).
  • the control unit 83 inputs the sensor data of cases C1 to C13 selected in the process of step S18 to the transformer from which the data was acquired in the process of step S19.
  • the control unit 83 inputs the sensor data to the transformer, and acquires from the transformer time-series data of the estimated values of the posture angles of the body parts of the user and time-series data of the movement speed of the user's lower back (step S20).
  • the control unit 83 generates a gait model based on the time-series data of the estimated values of the posture angles of the body parts of the user's whole body and the time-series data of the movement speed of the user's lower back acquired in the process of step S20 ( step S21).
  • the control unit 83 transmits the walking model data generated in the process of step S21 to the electronic device 20 via the network 2 by the communication unit 81 (step S22).
  • control unit 27 receives the walking model data from the server 80 via the network 2 by the communication unit 21 (step S23).
  • the control unit 27 causes the output unit 24 to output the received walking model data (step S24).
  • the information processing system 101 After executing the process of step S24, the information processing system 101 terminates the estimation process. After completing the estimation process, the information processing system 101 may perform the estimation process again. In the estimation process to be executed again, the information processing system 101 may start from the process of step S14. The information processing system 101 may repeatedly execute the estimation process until the electronic device 20 receives from the input unit 22 an input instructing to end the estimation process. As described above, when the electronic device 20 receives an input instructing to end the estimation process, the electronic device 20 may transmit a signal instructing to end data detection to the plurality of sensor devices 10 as a broadcast signal. As described above, the sensor device 10 may terminate data detection upon receiving a signal instructing it to terminate data detection.
  • the information processing system 101 can achieve the same or similar effects as the information processing system 1 .
  • each functional unit, each means, each step, etc. may be added to another embodiment so as not to be logically inconsistent, or each functional unit, each means, each step, etc. of another embodiment may be added.
  • each functional unit, each means, each step, etc. of another embodiment may be added.
  • the above-described embodiments of the present disclosure are not limited to faithful implementation of the respective described embodiments, and may be implemented by combining features or omitting some of them as appropriate. can also
  • the electronic device 20 or the server 80 may include a filter that can be applied to data output from the learning model.
  • the filter is, for example, a Butterworth filter.
  • the periodic exercise is walking.
  • the learning model has been explained as learning so as to output the estimated values of the posture angles of the user's body parts during walking.
  • periodic exercise is not limited to walking.
  • the information processing system of the present disclosure can obtain an estimate of the posture angle of the body part of the user when performing any periodic exercise. That is, the learning model can be trained to output an estimate of the posture angle of the body part when the user is performing any periodic exercise.
  • the information processing systems 1 and 101 have been described as estimating the posture angle of the user who walks as exercise in daily life.
  • the application of the information processing system of the present disclosure is not limited to this.
  • the information processing system of the present disclosure may be used to allow other customers to see how customers walk at an event venue.
  • the controller 27 of the electronic device 20 communicates the generated walking model data to the projection device at the event venue as an external device via the network 2 or short-range wireless communication. may be transmitted by unit 21;
  • the control unit 83 of the server 80 may transmit the generated walking model data to the projection device at the event venue as an external device via the network 2 by the communication unit 21 .
  • the projection device at the event venue can project a walking model showing how the customer walks on a screen or the like.
  • the information processing system of the present disclosure may be used to generate an image of a character walking using the generated walking model in a movie, game, or the like.
  • various walking models can be generated.
  • the communication unit 11 of the sensor device 10 may further include at least one communication module connectable to the network 2 as shown in FIG.
  • the communication module is, for example, a communication module compatible with mobile communication standards such as LTE, 4G, or 5G.
  • the control unit 16 of the sensor device 10 may directly transmit the data detected by the sensor device 10 to the server 80 via the network 2 using the communication unit 11. .
  • cases C5 to C8, C12, and C13 are described as including sensor data indicating movement of the user's wrist.
  • sensor data indicating the movement of the user's wrist instead of sensor data indicating the movement of the user's wrist, sensor data indicating the movement of the user's forearm other than the wrist may be used.
  • the sensor device 10 is configured to include the communication unit 11 as shown in FIGS. 3 and 39.
  • the sensor device 10 does not have to include the communication unit 11 .
  • the sensor data detected by the sensor device 10 may be transferred to a device such as the electronic device 20 or the server 80 that estimates the posture angle via a storage medium such as an SD (Secure Digital) memory card.
  • the SD memory card is also called "SD card”.
  • the sensor device 10 may be configured such that a storage medium such as an SD memory card can be inserted.
  • a general-purpose computer functions as the electronic device 20 according to this embodiment.
  • a program describing processing details for realizing each function of the electronic device 20 according to this embodiment is stored in the memory of a general-purpose computer, and the program is read and executed by the processor. Therefore, the configuration according to this embodiment can also be implemented as a program executable by a processor or a non-transitory computer-readable medium that stores the program.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Public Health (AREA)
  • Physics & Mathematics (AREA)
  • Pathology (AREA)
  • Biophysics (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • Veterinary Medicine (AREA)
  • Molecular Biology (AREA)
  • Physiology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Dentistry (AREA)
  • Physical Education & Sports Medicine (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Fuzzy Systems (AREA)
  • Psychiatry (AREA)
  • Signal Processing (AREA)
  • Geometry (AREA)
PCT/JP2022/045370 2021-12-10 2022-12-08 情報処理装置、電子機器、情報処理システム、情報処理方法及びプログラム Ceased WO2023106382A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP22904309.6A EP4446698A4 (en) 2021-12-10 2022-12-08 INFORMATION PROCESSING DEVICE, ELECTRONIC APPARATUS, INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, AND PROGRAM
JP2023566372A JP7818622B2 (ja) 2021-12-10 2022-12-08 情報処理装置、電子機器、情報処理システム、情報処理方法及びプログラム
US18/716,374 US20250032885A1 (en) 2021-12-10 2022-12-08 Information processing device, electronic device, information processing system, information processing method, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021201275 2021-12-10
JP2021-201275 2021-12-10

Publications (1)

Publication Number Publication Date
WO2023106382A1 true WO2023106382A1 (ja) 2023-06-15

Family

ID=86730519

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/045370 Ceased WO2023106382A1 (ja) 2021-12-10 2022-12-08 情報処理装置、電子機器、情報処理システム、情報処理方法及びプログラム

Country Status (4)

Country Link
US (1) US20250032885A1 (https=)
EP (1) EP4446698A4 (https=)
JP (1) JP7818622B2 (https=)
WO (1) WO2023106382A1 (https=)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09229667A (ja) * 1996-02-28 1997-09-05 Imeeji Joho Kagaku Kenkyusho 回転関節構造物の動作計測装置および方法
JP2008289866A (ja) * 2007-03-15 2008-12-04 Xsens Technologies Bv 較正ユニットを使った動作追跡のためのシステムと方法
WO2019203188A1 (ja) * 2018-04-17 2019-10-24 ソニー株式会社 プログラム、情報処理装置、及び情報処理方法
JP2020201183A (ja) 2019-06-12 2020-12-17 株式会社ソニー・インタラクティブエンタテインメント カメラ位置調整方法
JP2021100453A (ja) * 2019-12-24 2021-07-08 ブリヂストンスポーツ株式会社 情報処理装置、情報処理方法、プログラム

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6288706B2 (ja) 2014-03-26 2018-03-07 本田技研工業株式会社 上体運動計測システム及び上体運動計測方法
WO2019175899A1 (en) * 2018-03-15 2019-09-19 On My Own Technology Pvt Ltd Wearable device for gait analysis
CN110537922B (zh) * 2019-09-09 2020-09-04 北京航空航天大学 基于深度学习的人体行走过程下肢运动识别方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09229667A (ja) * 1996-02-28 1997-09-05 Imeeji Joho Kagaku Kenkyusho 回転関節構造物の動作計測装置および方法
JP2008289866A (ja) * 2007-03-15 2008-12-04 Xsens Technologies Bv 較正ユニットを使った動作追跡のためのシステムと方法
WO2019203188A1 (ja) * 2018-04-17 2019-10-24 ソニー株式会社 プログラム、情報処理装置、及び情報処理方法
JP2020201183A (ja) 2019-06-12 2020-12-17 株式会社ソニー・インタラクティブエンタテインメント カメラ位置調整方法
JP2021100453A (ja) * 2019-12-24 2021-07-08 ブリヂストンスポーツ株式会社 情報処理装置、情報処理方法、プログラム

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ASHISH VASWANI ET AL.: "Attention Is All You Need", ARXIV:1706.03762V5 [CS.CL, 12 June 2017 (2017-06-12)
See also references of EP4446698A4

Also Published As

Publication number Publication date
EP4446698A4 (en) 2025-10-15
EP4446698A1 (en) 2024-10-16
US20250032885A1 (en) 2025-01-30
JPWO2023106382A1 (https=) 2023-06-15
JP7818622B2 (ja) 2026-02-20

Similar Documents

Publication Publication Date Title
US11455909B2 (en) Identification and analysis of movement using sensor devices
US11194386B1 (en) Artificial reality wearable magnetic sensor system for body pose tracking
CN111460875B (zh) 图像处理方法及装置、图像设备及存储介质
CN113632176B (zh) 用于基于神经肌肉数据的低等待时间身体状态预测的方法和装置
CN105688396A (zh) 运动信息显示系统和运动信息显示方法
US12221728B2 (en) Knitted textile structures formed by altering knit patterns to accommodate external mediums, and manufacturing processes associated therewith
Salehi et al. Body-IMU autocalibration for inertial hip and knee joint tracking.
WO2021094775A1 (en) Method performed by an electronics arrangement for a wearable article
KR20230166319A (ko) 대상자의 3차원 전신 자세를 식별하기 위한 장치, 방법 및 프로그램 기록 매체
US12346493B2 (en) Information processing apparatus, electronic device, information processing system, method for processing information, and program
US11983320B2 (en) Techniques for incorporating stretchable conductive textile traces and textile-based sensors into knit structures
JP2009125229A (ja) 動き測定装置
Zhou et al. IMUCoCo: Enabling Flexible On-Body IMU Placement for Human Pose Estimation and Activity Recognition
KR20210040671A (ko) 동적으로 변화하는 인체 무게 중심 궤적 추정 장치 및 그 방법
JP7818622B2 (ja) 情報処理装置、電子機器、情報処理システム、情報処理方法及びプログラム
US12562043B2 (en) Method for detecting fall of user and electronic device supporting the same
US20240374166A1 (en) Information processing device, electronic device, information processing system, information processing method, and program
TW202430257A (zh) 任何姿勢動作捕捉校準方法及設備
WO2022250098A1 (ja) 情報処理装置、電子機器、情報処理システム、情報処理方法及びプログラム
US10549426B2 (en) Method for estimating movement of a poly-articulated mass object
Salehi Mourkani IMU-based Suit for Strength Exercises: Design, Calibration and Tracking
WO2021094777A1 (en) Method and electronics arrangement for a wearable article
EP4464245A1 (en) Posture estimating device, posture estimating system, and posture estimating method
Zhang et al. A Human Motion Capture System Based on MEMS Inertial Measurement
WO2025143202A1 (ja) 歩行姿勢状態推定システム、歩行姿勢状態推定方法、歩行姿勢状態推定装置及び制御プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22904309

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18716374

Country of ref document: US

Ref document number: 2023566372

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2022904309

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022904309

Country of ref document: EP

Effective date: 20240710