WO2022123800A1 - 情報処理装置、情報処理方法およびプログラム - Google Patents

情報処理装置、情報処理方法およびプログラム Download PDF

Info

Publication number
WO2022123800A1
WO2022123800A1 PCT/JP2021/006290 JP2021006290W WO2022123800A1 WO 2022123800 A1 WO2022123800 A1 WO 2022123800A1 JP 2021006290 W JP2021006290 W JP 2021006290W WO 2022123800 A1 WO2022123800 A1 WO 2022123800A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature amount
data
information processing
time
motion data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/006290
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
敬太 望月
佑樹 田中
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/253,933 priority Critical patent/US20240013410A1/en
Priority to EP21902908.9A priority patent/EP4261709A4/en
Priority to JP2022568034A priority patent/JP7521602B2/ja
Priority to KR1020237017611A priority patent/KR20230112636A/ko
Priority to CN202180080531.4A priority patent/CN116528759A/zh
Publication of WO2022123800A1 publication Critical patent/WO2022123800A1/ja
Anticipated expiration legal-status Critical
Priority to JP2024111415A priority patent/JP7747131B2/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7847—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content
    • G06F16/786—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content using motion, e.g. object motion or camera motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/73—Querying
    • G06F16/732—Query formulation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—Three-dimensional [3D] animation
    • G06T13/40—Three-dimensional [3D] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30221—Sports video; Sports image

Definitions

  • This disclosure relates to information processing devices, information processing methods and programs.
  • animation production and distribution using motion capture to acquire motion information indicating the user's motion has been actively performed.
  • motion data imitating a user's motion is generated using the acquired motion information, and an avatar image based on the motion data is distributed.
  • Patent Document 1 discloses a technique for creating animation data by concatenating a plurality of motion data.
  • a weight parameter prepared for each time or each part is added to the pre-processing feature amount, which is the feature amount for each time or part of the object, calculated from the time-series data of the movement of the object.
  • Information processing including an acquisition unit that acquires a post-processing feature amount that is an applied and calculated feature amount, and a search unit that searches for motion data using the post-processing feature amount acquired by the acquisition unit.
  • Equipment is provided.
  • weights prepared for each time or for each part are added to the pre-processing feature amount, which is the feature amount for each part of the object, calculated from the time-series data of the movement of the object.
  • Information executed by a computer including acquiring a post-process feature that is a feature calculated by applying parameters and searching for motion data using the acquired post-process feature.
  • the computer prepares the pre-processing feature amount, which is the feature amount for each target part, for each time or for each part, calculated from the time-series data of the movement of the target.
  • a program is provided to realize.
  • skeleton data represented by a skeleton structure showing the structure of the body is used in order to visualize information on the movement of a moving body such as a human or an animal.
  • the skeleton data includes information such as the position and posture of the part.
  • the part in the skeleton structure corresponds to, for example, a terminal part of the body, a joint part, or the like.
  • the skeleton data may include bones that are line segments connecting the parts. Bones in a skeleton structure can correspond to, for example, human bones, but the positions and numbers of bones do not necessarily have to be consistent with the actual human skeleton.
  • the position and posture of each part in the skeleton data can be acquired by various motion capture techniques.
  • camera-type technology that attaches markers to each part of the body and acquires the position of the marker using an external camera or the like, or time-series data acquired by attaching a motion sensor to a part of the body and acquiring the motion sensor.
  • sensor method technology that acquires the position information of the motion sensor based on the above.
  • time-series data of skeleton data is used for form improvement in sports, or used for applications such as VR (Virtual Reality) or AR (Augmented Reality).
  • VR Virtual Reality
  • AR Augmented Reality
  • time-series data of skeleton data an avatar video that imitates the movement of a user is generated, and the avatar video is also distributed.
  • the feature amount of the skeleton data calculated from the time-series data of the movement of the whole body of the user or the feature amount of each part of the skeleton data is acquired, and the motion data is used using the feature amount.
  • An example of a configuration of an information processing system for searching for data will be described.
  • human beings will be mainly described below as an example of moving objects, the embodiments of the present disclosure can be similarly applied to other moving objects such as animals and robots.
  • FIG. 1 is an explanatory diagram showing an information processing system according to an embodiment of the present disclosure.
  • the information processing system according to the embodiment of the present disclosure includes six sensor devices S1 to S6, an information processing terminal 10 and a server 20 worn by the user U.
  • the information processing terminal 10 and the server 20 are connected via the network 1.
  • the network 1 is a wired or wireless transmission path for information transmitted from a device connected to the network 1.
  • the network 1 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Network) including Ethernet (registered trademark), and a WAN (Wide Area Network).
  • the network 1 may include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).
  • the sensor device S detects the movement of the user U.
  • the sensor device S includes, for example, an inertial sensor (IMU: Initial Measurement Unit) such as an acceleration sensor for acquiring acceleration (Acceleration) and a gyro sensor (angular velocity sensor) for acquiring angular velocity (Angular velocity).
  • IMU Initial Measurement Unit
  • the sensor device S may be various sensor devices equipped with a sensor that detects the movement of the user U, such as an image pickup sensor, a ToF (Time of Flight) sensor, a magnetic sensor, or an ultrasonic sensor.
  • a sensor that detects the movement of the user U
  • an image pickup sensor such as an image pickup sensor, a ToF (Time of Flight) sensor, a magnetic sensor, or an ultrasonic sensor.
  • the sensor devices S1 to S6 are attached to a joint part that serves as a reference for the body (for example, the waist or head) or near the end of the body (wrist, ankle, head, etc.).
  • the sensor device S1 is attached to the waist of the user U
  • the sensor devices S2 and S5 are attached to both wrists
  • the sensor devices S3 and S4 are attached to both ankles
  • the sensor device S5 is attached to the head. It is installed.
  • the part of the body to which the sensor device S is attached may also be referred to as the attachment portion.
  • the number of sensor devices S and the mounting position are not limited to the example shown in FIG. 1, and the number of sensor devices S mounted on the user U may be larger or smaller.
  • Such a sensor device S acquires the acceleration or angular velocity of the mounting portion as time-series data, and transmits the time-series data to the information processing terminal 10.
  • the information processing terminal 10 may detect the movement of the user U by using various sensors (for example, an image pickup sensor or a ToF sensor) included in the information processing terminal 10.
  • various sensors for example, an image pickup sensor or a ToF sensor
  • the information processing terminal 10 is an example of an information processing device.
  • the information processing terminal 10 calculates the feature amount of the movement of the user U from the time-series data received from the sensor device S, and searches the motion data using the calculated feature amount.
  • the information processing terminal 10 transmits the processed feature amount to the server 20 as a search request. Then, the information processing terminal 10 receives the motion data searched by the server 20 in response to the search request from the server 20.
  • the information processing terminal 10 shows a smartphone in FIG. 1, the information processing terminal 10 may be another information processing device such as a notebook PC (Personal Computer) and a desktop PC.
  • a notebook PC Personal Computer
  • desktop PC Personal Computer
  • the server 20 holds a plurality of motion data and features of each of the plurality of motion data. Further, the server 20 evaluates the similarity between each feature amount of the plurality of motion data and the processed feature amount received from the information processing terminal 10, and outputs the motion data according to the result of the similarity evaluation to the information processing terminal 10. Send to.
  • FIG. 2 is an explanatory diagram for explaining a functional configuration example of the information processing terminal 10 according to the present disclosure.
  • the information processing terminal 10 includes an operation display unit 110, a communication unit 120, and a control unit 130.
  • the operation display unit 110 has a function as a display unit that displays the search results transmitted by the server 20. Further, the operation display unit 110 has a function as an operation unit for the user to input an operation.
  • the function as a display unit is realized by, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, and an OLED (Organic Light Emitting Diode) device.
  • CTR Cathode Ray Tube
  • LCD liquid crystal display
  • OLED Organic Light Emitting Diode
  • the function as an operation unit is realized by, for example, a touch panel, a keyboard or a mouse.
  • the information processing terminal 10 has a configuration in which the functions of the display unit and the operation unit are integrated in FIG. 1, the functions of the display unit and the operation unit may be separated.
  • the communication unit 120 communicates various information with the server 20 via the network 1. For example, the communication unit 120 transmits the processed skeleton data calculated from the time-series data of the user's movement to the server 20. Further, the communication unit 120 receives the motion data searched by the server 20 according to the transmitted processed feature amount.
  • Control unit 130 The control unit 130 controls the overall operation of the information processing terminal 10. As shown in FIG. 2, the control unit 130 includes a posture estimation unit 131, a feature amount calculation unit 135, a search request unit 139, and a correction unit 143.
  • the posture estimation unit 131 estimates the mounting site information indicating the position and posture of each mounting site based on the time-series data such as the acceleration or speed of the mounting site acquired from the sensor device S.
  • the position and posture of each mounting portion may be a two-dimensional position or a three-dimensional position.
  • the posture estimation unit 131 generates skeleton data including position information and posture information of each part in the skeleton structure based on the mounting part information. Further, the posture estimation unit 131 may convert the generated skeleton data into reference skeleton data. Details related to posture estimation will be described later.
  • the feature amount calculation unit 135 is an example of the acquisition unit, and calculates the feature amount of the whole body of the skeleton data or the feature amount before processing, which is the feature amount of each part, from the time series data of the skeleton data. Then, the feature amount calculation unit 135 applies a weight parameter to the feature amount before processing and calculates the feature amount after processing. Details of the pre-processed features, weight parameters and post-processed features will be described later.
  • the search request unit 139 is an example of the search unit, and the processed feature amount calculated by the feature amount calculation unit 135 is transmitted to the communication unit 120 as a search request.
  • the correction unit 143 corrects the feature amount of the motion data by mixing the processed feature amount with the feature amount of the motion data received as a search result from the server 20 at a set ratio. Details of the amendment will be described later.
  • FIG. 3 is an explanatory diagram for explaining a functional configuration example of the server 20 according to the present disclosure.
  • the server 20 includes a communication unit 210, a storage unit 220, and a control unit 230.
  • the communication unit 210 communicates various information with the information processing terminal 10 via the network 1. For example, the communication unit 210 receives the processed feature amount of the whole body or each part of the skeleton data calculated from the time-series data of the user's movement from the information processing terminal 10. Further, the communication unit 210 transmits the motion data searched according to the processed feature amount received from the information processing terminal 10 to the information processing terminal 10.
  • the storage unit 220 holds software and various data. As shown in FIG. 3, the storage unit 220 includes a motion data storage unit 221 and a motion feature amount storage unit 225.
  • the motion data storage unit 221 holds a plurality of motion data.
  • the motion feature amount storage unit 225 holds the feature amounts of each of the plurality of motion data held in the motion data storage unit 221. More specifically, the motion feature amount storage unit 225 holds the feature amount of the reference motion data, which is the motion data in which each skeleton data included in the motion data is converted into the reference skeleton data.
  • Control unit 230 controls the operation of the server 20 in general. As shown in FIG. 3, the control unit 230 includes a reference skeleton conversion unit 231, a feature amount calculation unit 235, a similarity evaluation unit 239, a learning unit 243, and an estimator 247.
  • the reference skeleton conversion unit 231 converts the skeleton data included in each of the plurality of motion data into the reference skeleton data. More specifically, the skeleton of each site of each of the skeleton data is converted into a reference skeleton containing predetermined skeleton information.
  • the feature amount calculation unit 235 calculates the feature amount of the motion data converted into the reference skeleton data, and outputs the calculation result of the feature amount to the motion feature amount storage unit 225.
  • the motion data converted into the reference skeleton data is an example of the reference motion data.
  • the similarity evaluation unit 239 evaluates the similarity between the processed feature amount received from the information processing terminal 10 and the feature amount of each of the plurality of motion data held in the motion feature amount storage unit 225. The details of the similarity evaluation will be described later.
  • the learning unit 243 generates learning data by a machine learning technique that uses a set of time-series data for each part of skeleton data and feature quantities for each part of motion data as teacher data.
  • the learning unit 243 uses attention for machine learning technology that uses a set of time-series data of skeleton data and feature quantities of each part of motion data as teacher data, so that the weight parameter for each part and each time can be used. You may get the weight parameter.
  • the estimator 247 estimates the pre-processing feature amount of each part from the user's skeleton data.
  • the function of the estimator 247 is obtained from the learning data generated by the learning unit 243.
  • the user searches for motion data or modifies existing animation data by performing an operation on the display screen of the operation display unit 110.
  • search of motion data an example of concatenating a plurality of searched motion data according to the movement of a user to generate one animation data will be described.
  • modification of the animation data an example of modifying a part of the section included in the existing animation data to the motion data searched according to the weight parameter will be described.
  • FIG. 4 is an explanatory diagram for explaining an example of a GUI (Graphical User Interface) that connects a plurality of search results.
  • GUI Graphic User Interface
  • the GUI that concatenates a plurality of search results may include skeleton data s, search buttons s1, sections A1 to A3, correction sections d2, and seek bar b1.
  • the search button s1 is a button for turning on or off the search function for acquiring the user's movement information.
  • the sections A1 to A3 are sections for inserting the motion data searched according to the movement of the user, and the correction section d2 is a section connecting the sections into which the two motion data are inserted.
  • the seek bar b1 is an instruction bar for displaying the skeleton data s at the timing specified by the cursor.
  • the operations and processes performed in the GUI are as follows. (1) First, the user selects the search button s1 based on a predetermined operation, and turns on the search function. (2) Next, the user performs an operation including information that the user wants to search as motion data. (3) Subsequently, the user selects the search button s1 again to turn off the search function. (4) Then, the operation display unit 110 displays the motion data searched according to the movement of the user. (5) When a plurality of searched motion data are displayed according to the movement of the user, the user selects one of the displayed motion data. (6) Further, the user selects any of the sections A1 to A3 as the insertion section. (7) Then, the operation display unit 110 inserts motion data into a section selected by the user.
  • animation data in which a plurality of motion data are concatenated is generated.
  • correction section d2 it is a correction section d2, and the correction section d2 may be filled by using an arbitrary correction method, or the correction section d2 may be eliminated and a plurality of insertion sections may be connected to generate animation data.
  • the user may be able to confirm the animation data generated by concatenating the motion data.
  • the user does not have to specify the insertion section.
  • motion data may be inserted in order from the timing before the time.
  • the motion data selected by the user in (5) may be inserted in order from the section A1.
  • the information processing terminal 10 may connect the motion data of each section to the correction section d2 between the sections A1 and A2 and between the sections A2 and A3 by using an arbitrary correction method.
  • three sections A1 to A3 are shown as sections for inserting motion data, but the number of sections to be inserted does not have to be three sections.
  • the number of sections into which motion data is inserted may be determined according to the number of operations and processes of (1) to (5).
  • the operation display unit 110 may display various parameter setting fields such as various weight parameters, post-process feature amounts, and motion data feature amount setting ratios.
  • FIG. 5 is an explanatory diagram for explaining an example of modifying a part of the section included in the existing animation data into motion data.
  • existing animation data A a part of the section included in the animation data obtained by motion capture or manual operation
  • the user selects the section A2 as the correction section from the plurality of sections A1 to A3 included in the existing animation data.
  • the operation display unit 110 may replace the motion data B searched based on the processed feature amount of the time-series data of the skeleton data included in the section A2 with the section A2 of the existing animation data and display it.
  • the operation display unit 110 replaces the left figure of the motion data B shown in FIG. 5 with the existing animation section A2 and displays the section A2.
  • FIG. 6 is an explanatory diagram for explaining an example of a GUI that modifies an existing animation.
  • the GUI for modifying the existing animation includes skeleton data s, a weight parameter setting field w1 for each part, a weight parameter setting field w2 for each time, and a setting ratio setting field qb.
  • the button s2, the section A2, the seek bar b2, and the reproduction command c1 may be included.
  • the weight parameter setting column w1 for each part is a setting column for setting the weight parameter applied to the pre-processing feature amount calculated for each part.
  • the weight parameter setting column w2 for each time is a setting column for setting the weight parameter to be applied to the pre-processing feature amount calculated for each time.
  • the setting ratio setting column qb is a setting column for setting a ratio of mixing the processed feature amount with the feature amount of the motion data for each part.
  • the user can confirm the corrected animation data by operating the playback command c1.
  • the user may confirm the corrected animation data by the operation by the seek bar b2.
  • the user selects the section A2 as the correction section. Subsequently, the user sets various parameters in each setting field of the weight parameter setting field w1 for each part, the weight parameter setting field w2 for each time, and the setting ratio setting field qb, and selects the search button s2.
  • the operation display unit 110 displays at least one or more motion data searched according to the user's operation.
  • the motion data of 1 is displayed as a search result
  • the operation display unit 110 replaces the motion data with the section A2 and inserts the motion data.
  • a plurality of motion data are displayed as search results, the user selects one motion data from the plurality of motion data, and the operation display unit 110 replaces the one motion data selected by the user with the section A2. And insert.
  • the information processing terminal 10 may present the modification candidate section to the user.
  • the operation display unit 110 may present the correction candidate section to the user together with displaying the existing animation data. In this case, the user may perform an operation of changing the presented correction candidate section.
  • the correction candidate section presented by the operation display unit 110 may be, for example, a section having a relatively large movement among all sections of existing animation data, or a machine learning technique such as DNN (Deep Neural Network). It may be a section estimated to be particularly important.
  • DNN Deep Neural Network
  • FIG. 7 is an explanatory diagram showing a specific example of a method for generating skeleton data.
  • the posture estimation unit 131 acquires the mounting site information PD including the position information and the posture information of the mounting site on which the sensor devices S1 to S6 are mounted, as shown in the left figure of FIG. 7, based on the time series data.
  • the posture estimation unit 131 acquires the skeleton data SD including the position information and the posture information of each part in the skeleton structure as shown in the right figure of FIG. 7 based on the attachment part information PD of the attachment part.
  • the skeleton data SD includes not only the information of the mounting portion SP1 corresponding to the mounting portion of the sensor device S1 and the mounting portion SP2 corresponding to the mounting portion of the sensor device S2, but also the information of the non-mounting portion SP7.
  • the skeleton data SD may include bone information (position information, posture information, etc.) in addition to site information.
  • the skeleton data SD may include information on the bone SB1.
  • the posture estimation unit can specify the bone information between the parts based on the position information and the posture information of the parts in the skeleton structure.
  • the user's movement may be detected by using an image pickup sensor or a ToF sensor provided in the information processing terminal 10.
  • the posture estimation unit 131 uses, for example, an estimator obtained by a machine learning technique using a set of skeleton data and time-series data of an image acquired by photographing a person as teacher data.
  • the skeleton data SD may be generated.
  • the features of the post-processed features calculated from the time-series data of the skeleton data SD generated based on the mounting site information and the features of each of the plurality of motion data held by the motion data storage unit 221.
  • it may be better to convert each skeleton data into the same skeletal information bone length, bone thickness, etc.
  • the posture estimation unit 131 may convert the skeleton of each part of the skeleton data SD into a reference skeleton and convert the skeleton data SD into reference skeleton data.
  • the posture estimation unit 131 does not have to convert the skeleton data SD into the reference skeleton data.
  • the feature amount that does not depend on the skeleton includes the posture information of each part.
  • the posture estimation unit 131 may convert the skeleton data SD into the reference skeleton data by using, for example, any method.
  • the arbitrary method includes, for example, copying the posture of each joint, scaling the root position according to the height, and adjusting the terminal position of each site using IK (Inverse Kinematics).
  • the learning unit 243 included in the server 20 may use DNN to perform learning to separate the skeleton information and the motion information of the skeleton data.
  • the attitude estimation unit 131 may omit the process of converting the skeleton data SD into the reference skeleton data.
  • the reference skeleton data may be simply referred to as skeleton data.
  • the feature amount will be described by dividing it into two types, a pre-process feature amount and a post-process feature amount to which a weight parameter described later is applied to the pre-process feature amount.
  • the feature amount calculation unit 135 calculates the pre-processing feature amount from the time-series data of the skeleton data estimated by the attitude estimation unit 131.
  • the pre-process feature amount may be the speed, position or posture (rotation, etc.) of each joint, or may be ground contact information.
  • the learning unit 243 may learn the relationship between the time-series data of the skeleton data and the pre-processing feature amount by using a machine learning technique such as DNN.
  • the feature amount calculation unit 135 calculates the pre-processing feature amount using the estimator 247 obtained by learning.
  • FIG. 8 an example of a method of learning the relationship between the time-series data of the skeleton data and the pre-processing feature amount by using the machine learning technique will be described.
  • FIG. 8 is an explanatory diagram illustrating an example of a method of learning the relationship between time-series data of skeleton data and pre-processed features using machine learning technology.
  • the learning unit 243 may use the Encoder-Decoder Model to learn the relationship between the time-series data of the skeleton data and the pre-processing feature amount.
  • the learning unit 243 estimates the pre-processing feature amount using CNN (Convolutional Neural Network) as the encoder. Then, the learning unit 243 outputs the posture of the whole body of the skeleton data in the time interval t to t + T using CNN as a decoder for the estimated pre-processing feature amount.
  • CNN Convolutional Neural Network
  • FIG. 8 shows an example in which the posture of the whole body is input as time-series data of the skeleton data, but for example, the input may be information related to other movements such as the position and speed of the joint. You may enter multiple pieces of information.
  • the Encoder-Decoder Model according to the present disclosure may have a more multi-layered or complicated structure, or may use other machine learning techniques such as RNN (Recurrent Neural Network).
  • the learning unit 243 may learn the relationship between the time-series data of the skeleton data and the pre-processing feature amount by using Deep Metal Learning. For example, the learning unit 243 may use Triplet Ross to learn the relationship between the time-series data of the skeleton data and the pre-processing feature amount.
  • time-series data may be artificially prepared, or similarity evaluation of time-series data may be performed.
  • the method may be used.
  • data that are close in time may be considered to be similar, and data that are distant in time may be considered to be similar.
  • the similarity evaluation method for time-series data includes, for example, DTW (Dynamic Time Warping).
  • the data set to be trained may be given information on a class label (for example, kick, punch, etc.).
  • the intermediate feature amount for classifying may be used as the pre-processing feature amount.
  • the machine learning technique by semi-teacher learning in which Encoder-Decoder Model and Triplet Less are combined may be used for learning.
  • FIG. 9 is an explanatory diagram for explaining an example of a method of calculating the pre-treatment feature amount for each part according to the present disclosure.
  • Part 243 may learn the relationship between the time-series data of each part of the skeleton data and each pre-processing feature amount by using DNN for each part of the skeleton data.
  • the learning unit 243 inputs the posture of the body of the skeleton data in the time interval t to t + T, and uses DNN as the Encoder to estimate the pre-processed feature amount of the body of the skeleton data.
  • the feature amount calculation unit 135 integrates the pre-processing feature amount of each part with the calculated pre-processing feature amount of each part by using DNN as a posture, so that the skeleton data in the time interval t to t + T can be obtained. Output the posture of the whole body.
  • the learning unit 243 may learn the relationship between the input and the pre-processing feature amount by combining the above-mentioned learning methods of the plurality of pre-processing feature amounts.
  • the feature amount calculation unit 135 features the features for each predetermined time interval from the time series data of the skeleton data indicating the user's movement. Calculate the amount.
  • the feature amount calculation unit 135 calculates the pre-processing feature amount of each part of the skeleton data indicating the movement of the user. For example, when the user performs a kick operation, the feature amount calculation unit 135 calculates not only the pre-processing feature amount of the foot kicked up by the user, but also the pre-processing feature amount for each part such as the head or hand. do.
  • the feature amount calculation unit 135 has a weight parameter prepared for each time or part for the pre-processing feature amount calculated from the time-series data of the movement of the skeleton data for each part. Is applied to calculate the processed features.
  • FIG. 10 is an explanatory diagram for explaining an example of a method of calculating post-processing parameters by applying weight parameters to pre-processing features.
  • the feature amount calculation unit 135 applies the weight parameter wm to each dimension or each time of the pre-processed feature amount bm of one part j to obtain the post-processed feature amount am. calculate.
  • the unprocessed feature amount bm of the part j is expressed by the determinant of bm j ⁇ RM ⁇ T.
  • M indicates the number of dimensions in the feature quantity direction
  • T indicates the number of time intervals divided into predetermined time intervals in the time direction. That is, FIG. 10 shows an example in which the number of dimensions M in the feature quantity direction and the number of time intervals T in the time direction are 5.
  • the number of dimensions M in the feature quantity direction may be singular or plural.
  • the weight parameter wm and the post-processing feature amount am are also expressed by the same number of rows and columns as the pre-processing feature amount bm.
  • the degree of magnitude of each feature amount included in the pre-processed feature amount, each parameter included in the weight parameter, and each feature amount included in the post-processed feature amount is expressed by shades of color.
  • the degree of shading of each feature included in the pre-processed feature amount bm is represented by one value
  • the degree of shading of each parameter included in the weight parameter wm and each feature amount included in the post-processed feature amount am is expressed by two values, but various values can be included.
  • the weight parameter wm is expressed by a determinant of wm ⁇ R (M ⁇ N) ⁇ T.
  • the weight parameter wm may be set by the user on the GUI, or may be determined by using the estimator 247 obtained by the machine learning technique. First, with reference to FIG. 11, an example in which the weight parameter is set by the user will be described.
  • FIG. 11 is an explanatory diagram for explaining an example of weight parameters prepared for each time.
  • FIG. 11 shows an example in which the time-series data of the acceleration of the foot acquired by the sensor device S attached to the user's foot is converted into the time-series data of the velocity v of the foot.
  • the sensor device S acquires time-series data before, during, and after the kick. If the kick movement is characteristic in the search for motion data, the user may set the weight parameter in the pre-kick and post-kick time intervals to be small or zero.
  • the user may set the weight parameter wm for each time by using the operation display unit 110 included in the information processing terminal 10. For example, when the section with the hatching shown in FIG. 11 is the time section in which the kick operation is performed by the user, the user sets the weight parameter wm for acquiring the feature amount of the hatched section every hour. May be set to.
  • the weight parameter wmt for each time may be set by using the following formula 1.
  • wm t 1 / L (adopted section)
  • wm t 0 (non-adopted section)
  • wm t 1 ...
  • L in Equation 1 is the time length of the adopted section.
  • the feature amount calculation unit 135 uses the formula 1 as the weight parameter wmt set for each time for the pre-processing feature amount for each time, for example, to obtain the feature amount in the time interval in which the user performs the kick operation. It can be calculated as a feature amount after processing.
  • the user may set the weight parameter wm Leg for the kicked foot larger than the weight parameter wm j of another portion.
  • the weight parameter wm may be set by the user using the operation display unit 110, or may be automatically set by the feature amount calculation unit 135.
  • the feature quantity calculation unit 135 increases the part weight parameter wm j in which the magnitude of the velocity or the amount of change in the velocity is equal to or more than a predetermined value, and determines the magnitude or velocity of the velocity.
  • the weight parameter wm j of the portion where the amount of change is less than a predetermined value may be set small.
  • the learning unit 243 may learn the relationship between the pre-processing feature amount and the weight parameter wm in addition to learning the relationship between the time-series data of the skeleton data and the pre-processing feature amount.
  • FIG. 12 is an explanatory diagram for explaining an example of a learning method of weight parameters.
  • the learning unit 243 learns the relationship between the posture of each part of the skeleton data and the pre-processing feature amount of each part in the time interval t to t + T by using the method of calculating the pre-processing feature amount described with reference to FIG. do.
  • the learning unit 243 inputs the posture of the whole body of the skeleton data and the posture of each part in the time interval t to t + T, and uses the attention of DNN to input the pre-processing feature amount for each part and the weight parameter for each part. You may learn the relationship with.
  • the learning unit 243 inputs the posture of the whole body and the posture of each part of the skeleton data, and learns the relationship between the pre-processing feature amount for each time and the weight parameter for each time by using the attention of DNN. May be good.
  • the feature amount calculation unit 235 determines the weight parameter for each time and the weight parameter for each part by using the estimator 247 obtained by learning.
  • Similarity evaluation> The information processing terminal 10 transmits information on the processed feature amount to the server 20. Then, the similarity evaluation unit 239 included in the server 20 evaluates the similarity between the received processed feature amount and the feature amount of the motion data held in the motion feature amount storage unit 225.
  • the similarity evaluation unit 239 may perform similarity evaluation using, for example, a square error. For example, the pre-processed features of the time interval t and the dimension m in the part j are set to qery f j t and m , the feature amounts of the motion data are set set f j t and m, and the weight parameters are set to w j t and m . Let s be the degree. In this case, the similarity evaluation unit 239 evaluates the similarity between the processed feature amount and the feature amount of the motion data by using the mathematical formula 2.
  • the similarity evaluation unit 239 may perform similarity evaluation using, for example, a correlation coefficient. More specifically, the similarity evaluation unit 239 evaluates the similarity between the processed feature amount and the feature amount of the motion data by using the mathematical formula 3.
  • the server 20 transmits motion data according to the result of the similarity evaluation by the similarity evaluation unit 239 to the information processing terminal 10.
  • the similarity evaluation unit 239 calculates the similarity between the received post-processing feature amount and the feature amount of each of the plurality of motion data, and the server 20 uses a predetermined number of motion data as the search result from the one with the highest similarity. It may be transmitted to the information processing terminal 10.
  • the user may perform an operation of excluding motion data having a high degree of similarity from the search results.
  • the motion data whose similarity is equal to or higher than the predetermined value is excluded from the search results from the similarity evaluation unit 239.
  • the movement of the whole body of the user or the movement of the portion where the weight parameter is increased can be the movement particularly required by the user.
  • the movements of all parts of the motion data do not always match or resemble the movements required by the user.
  • the correction unit 143 may execute a process of correcting the feature amount of the motion data for at least one or more parts of the motion data acquired as the search result.
  • a process of correcting the feature amount of the motion data for at least one or more parts of the motion data acquired as the search result.
  • FIG. 13 is an explanatory diagram for explaining an example of processing for correcting a feature amount of motion data.
  • the skeleton data indicating the movement of the user acquired by the sensor device S is referred to as the query Q (t)
  • the skeleton data of the motion data acquired as the search result is referred to as the search result R (t).
  • the correction unit 143 searches based on the setting ratio set by the user as described above. You may execute the process of correcting the result.
  • the correction unit 143 corrects the feature amount of the motion data by mixing the feature amount of the motion data with the feature amount after processing for at least one part of the motion data received as a search result from the server 20. To execute. As a result, the correction unit 143 acquires the corrected search result R'(t), which is a mixture of the query Q (t) and the search result R (t).
  • correction unit 143 may correct the portion designated as the correction target by the user so as to be the same as the position of the query Q (t).
  • the correction unit 143 uses the posture of the search result R (t) as the initial value, and corrects using IK so that the position of the terminal portion of the search result R (t) matches the position of the query Q (t). The process may be executed. Since there is a possibility that the position of the waist of the query Q (t) and the search result R (t) may deviate when correcting the position of the portion, for example, the correction unit 143 executes the correction process based on the relative position from the waist. You may.
  • the portion to be corrected may be designated by the user using, for example, the operation display unit 110, or may be automatically designated by the correction unit 143.
  • the correction unit 143 may determine the part to be corrected based on the weight parameter prepared for each part. For example, the correction unit 143 adopts the feature amount of the search result R (t) in the portion where the weight parameter satisfies the predetermined criterion, and the post-processing feature of the query Q (t) in the portion where the weight parameter does not satisfy the predetermined criterion. A process of correcting the amount may be executed.
  • the correction unit 143 is based on the set ratio. It may not be necessary to execute the correction process. For example, when the whole body of the motion data cannot be balanced by following the set ratio, the correction unit 143 may execute a process of correcting the feature amount of the part and other parts according to the positional relationship of each part.
  • FIG. 14 is an explanatory diagram for explaining an operation processing example related to a search for motion data of the information processing terminal 10 according to the present disclosure.
  • the information processing terminal 10 acquires time-series data of the target movement from the sensor device S (S101).
  • the posture estimation unit 131 generates skeleton data from the acquired time-series data of the movement of the target (S105).
  • the posture estimation unit 131 converts the skeleton of each part of the generated skeleton data into a reference skeleton and generates the reference skeleton data (S109).
  • the feature amount calculation unit 135 calculates the pre-processing feature amount of each part of the reference skeleton data from the time-series data of the reference skeleton data (S113).
  • the feature amount calculation unit 135 applies the weight parameter set for each time or each part to the pre-processing feature amount to calculate the post-processing feature amount (S117).
  • the communication unit 120 transmits a signal including the calculated processed feature amount information to the server 20 under the control of the search request unit 139 (S121).
  • the communication unit 120 receives a signal including the motion data information searched by the server 20 according to the transmitted processed feature amount information (S125).
  • the correction unit 143 corrects the feature amount of the motion data based on the setting ratio of the feature amount after processing and the feature amount of the acquired motion data (S129).
  • the operation display unit 110 displays the corrected motion data generated based on the corrected feature amount of the motion data (S133), and the information processing terminal 10 ends the operation process related to the motion data search.
  • FIG. 15 is an explanatory diagram for explaining an operation processing example related to a search for motion data of the server 20 according to the present disclosure.
  • the communication unit 210 receives the processed feature amount from the information processing terminal 10 (S201).
  • the similarity evaluation unit 239 calculates the similarity between the received processed feature amount and the feature amount of each of the plurality of motion data held in the motion feature amount conversion unit (S205).
  • the similarity evaluation unit 239 acquires a predetermined number of motion data as a search result from the one with the highest similarity (S209).
  • the communication unit 210 transmits a predetermined number of motion data acquired by S209 to the information processing terminal 10 as a search result (S213), and the server 20 ends the operation process related to the motion data search.
  • the feature amount calculation unit 135 calculates the post-processing feature amount by applying the weight parameter prepared for each part to the pre-processing feature amount calculated from the time-series data of the user's movement. This makes it possible to search for more important parts in the motion data search.
  • the feature amount calculation unit 135 calculates the post-processing feature amount by applying the weight parameter prepared for each time to the pre-processing feature amount calculated from the time-series data of the user's movement. As a result, in the search of motion data, it may be possible to perform a search focusing on a more important time interval.
  • the weight parameter is determined by using the estimator 247 obtained by the machine learning technique, it is not necessary for the user to manually input the weight parameter, and the convenience of the user can be improved.
  • the information processing terminal 10 has a predetermined number of motion data from the one having the highest degree of similarity between the processed feature amount calculated from the time-series data of the skeleton data indicating the user's movement and the feature amount of each of the plurality of motion data. As a search result. As a result, the user can select motion data including particularly desired motion information from a plurality of presented motion data.
  • each of the skeleton data showing the movement of the user and the skeleton data of the motion data are converted into the reference skeleton data, and the feature quantities of the reference skeleton data are compared with each other. This can reduce the possibility of search error due to the difference between the skeleton of the user and the skeleton of the motion data.
  • the correction unit 143 corrects the feature amount of the motion data by mixing the feature amount of the motion data with the feature amount after processing at a set ratio for at least one part. As a result, the movement of the part of the motion data can be corrected to the movement of the part more required by the user, and the convenience of the user can be further improved.
  • Hardware configuration example >> The embodiments of the present disclosure have been described above. Information processing such as the generation of the skeleton data and the extraction of the feature amount described above is realized by the cooperation between the software and the hardware of the information processing terminal 10 described below. The hardware configuration described below can also be applied to the server 20.
  • FIG. 16 is a block diagram showing the hardware configuration of the information processing terminal 10.
  • the information processing terminal 10 includes a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, and a host bus 1004. Further, the information processing terminal 10 includes a bridge 1005, an external bus 1006, an interface 1007, an input device 1008, an output device 1010, a storage device (HDD) 1011 and a drive 1012, and a communication device 1015.
  • a CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the CPU 1001 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing terminal 10 according to various programs. Further, the CPU 1001 may be a microprocessor.
  • the ROM 1002 stores programs, calculation parameters, and the like used by the CPU 1001.
  • the RAM 1003 temporarily stores a program used in the execution of the CPU 1001 and parameters that appropriately change in the execution. These are connected to each other by a host bus 1004 composed of a CPU bus or the like.
  • the host bus 1004 is connected to an external bus 1006 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 1005. It is not always necessary to separately configure the host bus 1004, the bridge 1005, and the external bus 1006, and these functions may be implemented in one bus.
  • PCI Peripheral Component Interconnect / Interface
  • the input device 1008 includes input means for the user to input information such as a mouse, keyboard, touch panel, buttons, microphones, switches and levers, and an input control circuit that generates an input signal based on the input by the user and outputs the input signal to the CPU 1001. It is composed of such things.
  • the user of the information processing terminal 10 can input various data to the information processing terminal 10 and instruct the processing operation.
  • the output device 1010 includes, for example, a liquid crystal display device, an OLED device, and a display device such as a lamp. Further, the output device 1010 includes an audio output device such as a speaker and headphones. The output device 1010 outputs, for example, the reproduced content. Specifically, the display device displays various information such as reproduced video data as text or an image. On the other hand, the voice output device converts the reproduced voice data or the like into voice and outputs it.
  • the storage device 1011 is a device for storing data.
  • the storage device 1011 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded on the storage medium, and the like.
  • the storage device 1011 is composed of, for example, an HDD (Hard Disk Drive).
  • the storage device 1011 drives a hard disk and stores programs and various data executed by the CPU 1001.
  • the drive 1012 is a reader / writer for a storage medium, and is built in or externally attached to the information processing terminal 10.
  • the drive 1012 reads out the information recorded in the removable storage medium 30 such as the mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 1003.
  • the drive 1012 can also write information to the removable storage medium 30.
  • the communication device 1015 is, for example, a communication interface composed of a communication device or the like for connecting to the network 12. Further, the communication device 1015 may be a wireless LAN compatible communication device, an LTE (Long Term Evolution) compatible communication device, or a wire communication device that performs wired communication.
  • LTE Long Term Evolution
  • the information processing terminal 10 may further include all or part of the functional configuration of the server 20 according to the present disclosure.
  • the information processing terminal 10 can execute a series of processes related to the search without communication via the network 1.
  • the information processing terminal 10 is provided with a partial functional configuration of the server 20 according to the present disclosure, for example, the information processing terminal 10 previously uses communication via the network 1 to transmit a plurality of motion data from the server 20. You may receive it. Then, the information processing terminal 10 evaluates the similarity between the processed feature amount calculated by the feature amount calculation unit 135 and the plurality of motion data received in advance from the server 20, and moves according to the evaluation result of the similarity. You may search the data.
  • each step in the processing of the information processing terminal 10 and the server 20 of the present specification does not necessarily have to be processed in chronological order in the order described as a flowchart.
  • each step in the processing of the information processing terminal 10 and the server 20 may be processed in an order different from the order described in the flowchart.
  • the following configurations also belong to the technical scope of the present disclosure.
  • (1) It was calculated by applying the weight parameter prepared for each time or each part to the pre-processing feature amount which is the feature amount for each time or for each part of the object calculated from the time series data of the movement of the object.
  • the acquisition unit that acquires the post-processing feature amount, which is the feature amount
  • a search unit that searches for motion data using the processed features acquired by the acquisition unit, and a search unit.
  • An information processing device equipped with An information processing device equipped with.
  • the weight parameter applied to the pre-process feature amount is determined by an estimator obtained by learning the relationship between the feature amount for each part and the weight parameter for each part.
  • the weight parameter applied to the pre-process feature amount is determined by an estimator obtained by learning the relationship between the time-by-time feature amount and the time-by-time weight parameter.
  • the information processing apparatus according to (1) or (2) above.
  • the search unit The degree of similarity between the processed feature amount of the target and the feature amount of each of the plurality of motion data acquired by the acquisition unit is calculated, and the motion data is searched based on the calculation result of the similarity.
  • the search unit Based on the calculation result of the similarity, a predetermined number of motion data are acquired as a search result from the one having the highest similarity with the processed feature amount.
  • the information processing apparatus according to (4) above.
  • the acquisition unit The post-processed feature amount calculated from the time-series data of the movement of the reference target obtained by converting the skeleton of the target object into the reference skeleton, and the feature amount calculated from the reference motion data obtained by converting the skeleton data skeleton into the reference skeleton. Search for the motion data by comparison, The information processing apparatus according to (4) or (5) above.
  • the information processing device is A correction unit that corrects the feature amount of the motion data by mixing the feature amount of the processed motion with the feature amount of the motion data at a set ratio for at least one portion.
  • the information processing apparatus according to any one of the above (1) to (6).
  • the feature amount for each target part includes at least one of speed, position, and posture.
  • the information processing apparatus according to any one of the above (1) to (7). (9) It was calculated by applying the weight parameter prepared for each time or each part to the pre-processing feature amount which is the feature amount for each time or for each part of the object calculated from the time series data of the movement of the object. Acquiring the processed feature amount, which is the feature amount, Searching for motion data using the acquired post-processing features, Information processing methods performed by a computer, including. (10) On the computer It was calculated by applying the weight parameter prepared for each time or each part to the pre-processing feature amount which is the feature amount for each time or for each part of the object calculated from the time series data of the movement of the object.
  • the acquisition function to acquire the feature amount after processing, which is the feature amount
  • Information processing terminal 20 Server 110 Operation display unit 120 Communication unit 130 Control unit 131 Attitude estimation unit 135 Feature amount calculation unit 139 Search request unit 143 Correction unit 210 Communication unit 220 Storage unit 221 Motion data storage unit 225 Motion feature amount storage unit 230 Control unit 231 Reference skeleton conversion unit 235 Feature calculation unit 239 Similarity evaluation unit 243 Learning unit 247 Estimator

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Library & Information Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Dentistry (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Physiology (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Psychiatry (AREA)
  • Processing Or Creating Images (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Image Analysis (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
PCT/JP2021/006290 2020-12-08 2021-02-19 情報処理装置、情報処理方法およびプログラム Ceased WO2022123800A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
US18/253,933 US20240013410A1 (en) 2020-12-08 2021-02-19 Information processing apparatus, information processing method, and program
EP21902908.9A EP4261709A4 (en) 2020-12-08 2021-02-19 Information processing device, information processing method, and program
JP2022568034A JP7521602B2 (ja) 2020-12-08 2021-02-19 情報処理装置、情報処理方法およびプログラム
KR1020237017611A KR20230112636A (ko) 2020-12-08 2021-02-19 정보 처리 장치, 정보 처리 방법 및 프로그램
CN202180080531.4A CN116528759A (zh) 2020-12-08 2021-02-19 信息处理装置、信息处理方法和程序
JP2024111415A JP7747131B2 (ja) 2020-12-08 2024-07-11 情報処理装置、情報処理方法およびプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063122509P 2020-12-08 2020-12-08
US63/122,509 2020-12-08

Publications (1)

Publication Number Publication Date
WO2022123800A1 true WO2022123800A1 (ja) 2022-06-16

Family

ID=81974333

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/006290 Ceased WO2022123800A1 (ja) 2020-12-08 2021-02-19 情報処理装置、情報処理方法およびプログラム

Country Status (6)

Country Link
US (1) US20240013410A1 (https=)
EP (1) EP4261709A4 (https=)
JP (2) JP7521602B2 (https=)
KR (1) KR20230112636A (https=)
CN (1) CN116528759A (https=)
WO (1) WO2022123800A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024122203A1 (ja) * 2022-12-07 2024-06-13 ソニーグループ株式会社 情報処理装置、および情報処理方法
JP2024154850A (ja) * 2023-04-20 2024-10-31 グリー株式会社 情報処理システム、情報処理方法、及びプログラム

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240280687A1 (en) * 2021-06-28 2024-08-22 Google Llc System and method for motion capture
TWI900829B (zh) * 2023-03-07 2025-10-11 緯創資通股份有限公司 識別方法及無線動作捕捉系統

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010033163A (ja) * 2008-07-25 2010-02-12 Kddi Corp 動きデータ検索装置及びコンピュータプログラム
US20120038628A1 (en) 2010-04-28 2012-02-16 Mixamo, Inc. Real-time automatic concatenation of 3d animation sequences

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE69421404T2 (de) * 1994-12-30 2000-02-10 Daewoo Electronics Co., Ltd. Blockübereinstimmungsbewegungsauswertung mit einer gewichteten Funktion
US6414684B1 (en) * 1996-04-25 2002-07-02 Matsushita Electric Industrial Co., Ltd. Method for communicating and generating computer graphics animation data, and recording media
JP2953413B2 (ja) * 1996-12-09 1999-09-27 日本電気株式会社 動作修正方法および動作修正装置
US8542928B2 (en) * 2005-09-26 2013-09-24 Canon Kabushiki Kaisha Information processing apparatus and control method therefor
CN101515371B (zh) * 2009-03-26 2011-01-19 浙江大学 人体运动数据片段提取方法
US8751191B2 (en) * 2009-12-22 2014-06-10 Panasonic Corporation Action analysis device and action analysis method
JP2015011480A (ja) * 2013-06-27 2015-01-19 カシオ計算機株式会社 画像生成装置、画像生成方法及びプログラム
KR101765208B1 (ko) * 2014-01-07 2017-08-04 한국전자통신연구원 시간 및 자세 제약 조건 아래에서 무용 동작을 생성하는 방법 및 장치
KR102013577B1 (ko) * 2015-09-14 2019-08-23 한국전자통신연구원 안무 구상 지원 장치 및 방법
US10019629B2 (en) * 2016-05-31 2018-07-10 Microsoft Technology Licensing, Llc Skeleton-based action detection using recurrent neural network
US10368087B2 (en) * 2016-09-20 2019-07-30 Ati Technologies Ulc Dynamic reload of video encoder motion estimation search window under performance/power constraints
EP3730196A4 (en) * 2017-12-19 2021-06-09 Sony Corporation INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD AND PROGRAM
WO2019203188A1 (ja) * 2018-04-17 2019-10-24 ソニー株式会社 プログラム、情報処理装置、及び情報処理方法
WO2019243899A1 (en) * 2018-06-18 2019-12-26 Unity IPR ApS Method and system for real-time animation generation using machine learning
JP7262937B2 (ja) * 2018-06-29 2023-04-24 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム
KR102543650B1 (ko) * 2018-07-30 2023-06-15 주식회사 엔씨소프트 모션 합성 장치 및 모션 합성 방법
CN111444889B (zh) * 2020-04-30 2023-07-25 南京大学 基于多级条件影响的卷积神经网络的细粒度动作检测方法
WO2022249278A1 (ja) * 2021-05-25 2022-12-01 日本電気株式会社 画像処理装置、画像処理方法、およびプログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010033163A (ja) * 2008-07-25 2010-02-12 Kddi Corp 動きデータ検索装置及びコンピュータプログラム
US20120038628A1 (en) 2010-04-28 2012-02-16 Mixamo, Inc. Real-time automatic concatenation of 3d animation sequences

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
KENSUKE TAKAHARA, PANTUWONG NATAPON, TAKESHI YOSHIKAWA, HIDETOSHI NONAKA, MASANORI SUGIMOTO: "A Rapid Motion Retrieval Technique using Simple and Discrete Representation of Feature Vector", IPSJ INTERACTION 2015, 26 February 2015 (2015-02-26), XP055947770, Retrieved from the Internet <URL:https://www.interaction-ipsj.org/proceedings/2015/data/20150226/A62.pdf> *
See also references of EP4261709A4
XIAO QINKUN; CHU CHAOQIN: "Human motion retrieval based on deep learning and dynamic time warping", 2017 2ND INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION ENGINEERING (ICRAE), IEEE, 29 December 2017 (2017-12-29), pages 426 - 430, XP033320852, DOI: 10.1109/ICRAE.2017.8291423 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024122203A1 (ja) * 2022-12-07 2024-06-13 ソニーグループ株式会社 情報処理装置、および情報処理方法
JP2024154850A (ja) * 2023-04-20 2024-10-31 グリー株式会社 情報処理システム、情報処理方法、及びプログラム

Also Published As

Publication number Publication date
JP7521602B2 (ja) 2024-07-24
JP7747131B2 (ja) 2025-10-01
CN116528759A (zh) 2023-08-01
KR20230112636A (ko) 2023-07-27
EP4261709A4 (en) 2024-04-24
EP4261709A1 (en) 2023-10-18
US20240013410A1 (en) 2024-01-11
JPWO2022123800A1 (https=) 2022-06-16
JP2024133181A (ja) 2024-10-01

Similar Documents

Publication Publication Date Title
JP7407919B2 (ja) 動画処理方法、動画処理装置、コンピュータプログラム及び電子機器
US11586276B2 (en) Systems and methods for generating complementary data for visual display
JP7747131B2 (ja) 情報処理装置、情報処理方法およびプログラム
KR102438488B1 (ko) 3d 마커리스 모션 캡처 기반 3d 아바타 생성 장치 및 방법
Ikeuchi et al. Describing upper-body motions based on labanotation for learning-from-observation robots
CN110599573A (zh) 一种基于单目相机的人脸实时交互动画的实现方法
Cong et al. Laserhuman: Language-guided scene-aware human motion generation in free environment
Kitsikidis et al. Multi-sensor technology and fuzzy logic for dancer’s motion analysis and performance evaluation within a 3D virtual environment
Zhang et al. Freemotion: Mocap-free human motion synthesis with multimodal large language models
US20240135618A1 (en) Generating artificial agents for realistic motion simulation using broadcast videos
CN115018962B (zh) 一种基于虚拟人物模型的人体运动姿态数据集生成方法
CN110853131A (zh) 一种用于行为识别的虚拟视频数据生成方法
KR102342760B1 (ko) 인공지능 기반의 골프 영상 학습장치, 그 방법 및 그 기록매체
CN111507920B (zh) 基于Kinect的骨骼运动数据增强方法及系统
US20250258537A1 (en) Information processing apparatus, information processing method, and program
Gao Intelligent Dance Notation: A Dance Movement Quantification Framework Based on Digital Human
WO2022196059A1 (ja) 情報処理装置、情報処理方法およびプログラム
Zhang et al. Optimizing Motion Capture Data in Animation Sequences Using Machine Learning Techniques
Aslanyan Development of intelligent workout environment for vr devices
Huang et al. Interactive demonstration of pointing gestures for virtual trainers
Weiss Cohen et al. Generating 3D cad art from human gestures using kinect depth sensor
Ferreira et al. UnMimic: An Automatic Pipeline for Transferring Body Movements to a Photorealistic Unreal Avatar
CN119810268A (zh) 姿态过渡动画生成方法、装置、电子设备和存储介质
Dong Three-Dimensional Animation Capture Driver Technology for Digital Media.
CN119987557A (zh) 一种人机共融系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21902908

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022568034

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18253933

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 202180080531.4

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021902908

Country of ref document: EP

Effective date: 20230710