WO2023188216A1 - 情報処理プログラム、情報処理方法、および情報処理装置 - Google Patents

情報処理プログラム、情報処理方法、および情報処理装置 Download PDF

Info

Publication number
WO2023188216A1
WO2023188216A1 PCT/JP2022/016363 JP2022016363W WO2023188216A1 WO 2023188216 A1 WO2023188216 A1 WO 2023188216A1 JP 2022016363 W JP2022016363 W JP 2022016363W WO 2023188216 A1 WO2023188216 A1 WO 2023188216A1
Authority
WO
WIPO (PCT)
Prior art keywords
information processing
series data
information
time
subject
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/016363
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
成幸 小田嶋
創輔 山尾
達也 鈴木
帆 楊
昇一 桝井
山 姜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujitsu Ltd
Original Assignee
Fujitsu Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujitsu Ltd filed Critical Fujitsu Ltd
Priority to JP2024510981A priority Critical patent/JP7684622B2/ja
Priority to CN202280094183.0A priority patent/CN119343695A/zh
Priority to PCT/JP2022/016363 priority patent/WO2023188216A1/ja
Priority to EP22935361.0A priority patent/EP4502924A4/en
Publication of WO2023188216A1 publication Critical patent/WO2023188216A1/ja
Priority to US18/887,451 priority patent/US20250014215A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/29—Graphical models, e.g. Bayesian networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20036—Morphological image processing
    • G06T2207/20044—Skeletonization; Medial axis transform
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20072—Graph-based image processing

Definitions

  • the present invention relates to an information processing program, an information processing method, and an information processing device.
  • the result of the first process the likelihood of the result of the second process is calculated based on the likelihood of the result of the first process, the likelihood of the result of the second process, and the likelihood of the result of the third process.
  • some of the results of the third process are output as the object skeleton recognition results.
  • there is a technique for recognizing a heat map image in which the likelihood of a plurality of joint positions of a subject is projected from a plurality of directions from a distance image of the subject.
  • there is a behavior detection technique that uses a recurrent neural network.
  • the conventional technology it may be difficult to accurately specify the three-dimensional coordinates of each joint of a person.
  • the three-dimensional coordinates of a joint in a person's right hand may be mistaken as the three-dimensional coordinates of a joint in a person's left hand.
  • the three-dimensional coordinates of a part of an object other than a person in a multi-view image may be mistaken as the three-dimensional coordinates of a human joint.
  • the present invention aims to enable accurate identification of the location of a subject's body part.
  • time-series data of skeletal information including the position of each of a plurality of body parts of a subject is acquired, and based on the feature amount related to the skeletal information in the acquired time-series data, the acquired Regarding the skeletal information at the first point in time in the time-series data, any part that is in an abnormal state with respect to position is identified, and based on the feature amount related to the skeletal information in the acquired time-series data, the skeleton at the first point in time is determined.
  • a probability distribution model is determined that constrains the position of any of the specified parts according to the size of the probability that any of the specified parts is in an abnormal state, and
  • the model includes a node indicating the position of a part and a first edge connecting nodes indicating positions of different biologically connected parts at each time point, and the model is determined to be a node indicating the position of any of the parts.
  • FIG. 1 is an explanatory diagram showing an example of an information processing method according to an embodiment.
  • FIG. 2 is an explanatory diagram showing an example of the information processing system 200.
  • FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing device 100.
  • FIG. 4 is a block diagram showing an example of the hardware configuration of the image capturing device 201.
  • FIG. 5 is a block diagram showing an example of the functional configuration of the information processing device 100.
  • FIG. 6 is an explanatory diagram showing the flow of operations of the information processing device 100.
  • FIG. 7 is an explanatory diagram (part 1) showing a specific example of identifying an abnormal joint.
  • FIG. 8 is an explanatory diagram (part 2) showing a specific example of identifying an abnormal joint.
  • FIG. 1 is an explanatory diagram showing an example of an information processing method according to an embodiment.
  • FIG. 2 is an explanatory diagram showing an example of the information processing system 200.
  • FIG. 3 is a block diagram showing an example of the hardware configuration
  • FIG. 9 is an explanatory diagram showing a specific example of generating a Factor Graph 900.
  • FIG. 10 is an explanatory diagram showing a specific example of modifying the 3D skeleton inference result 602.
  • FIG. 11 is an explanatory diagram (part 1) showing a specific example of the flow of data processing in the operation example.
  • FIG. 12 is an explanatory diagram (part 2) showing a specific example of the flow of data processing in the operation example.
  • FIG. 13 is a flowchart showing an example of the overall processing procedure.
  • FIG. 1 is an explanatory diagram showing an example of an information processing method according to an embodiment.
  • the information processing device 100 is a computer that enables the position of a subject's joints to be specified with high accuracy.
  • the target person is, for example, a person.
  • the position is, for example, a three-dimensional coordinate.
  • the two-dimensional coordinates of each joint of the person are determined based on the detected area, and the identified two-dimensional coordinates are determined by taking into account the angle. It is conceivable to specify the three-dimensional coordinates of each joint of a person based on the following. Specifically, a model learned through deep learning is used to identify the three-dimensional coordinates of each joint of a person. For an example of this technique, reference can be made specifically to Reference 1 and Reference 2 below.
  • V2v-posenet Voxel-to-voxel prediction network for accurate 3D hand and human pose estimation "From a single depth map.” Procedures of the IEEE conference on computer vision and pattern Recognition. 2018.
  • the information processing device 100 acquires time-series data of the skeleton information 101.
  • the skeletal information 101 includes, for example, the position of each of a plurality of body parts of the subject. Sites include, for example, the neck, head, right and left shoulders, right and left elbows, right and left hands, right and left knees, right and left feet, and the like. Specifically, the site is a joint. In the example of FIG. 1, the parts are specifically joint 1, joint 2, joint 3, etc. The position is, for example, a three-dimensional coordinate.
  • the time series data includes, for example, skeletal information 101 for each time point. In the example of FIG. 1, the time-series data specifically includes skeletal information 101 at time T, skeletal information 101 at time T-1, and the like.
  • the information processing device 100 determines whether the skeletal information 101 at the first point in time in the acquired time series data is in an abnormal state in terms of position, based on the feature amount regarding the skeletal information 101 in the acquired time series data. Identify that part.
  • the feature amount may be, for example, the position of each part of the subject indicated by the skeletal information 101.
  • the feature amount may be, for example, a deviation in the position of each part of the subject indicated by the skeletal information 101 at different times.
  • the feature amount may be, for example, the distance between the positions of different parts of the subject indicated by the skeletal information 101.
  • the information processing device 100 includes, for example, a first model for identifying one of a plurality of body parts of a subject that is in an abnormal position.
  • the first model calculates the probability that the position of each of a plurality of body parts of the subject is in an abnormal state in accordance with the input of the feature amount related to the skeletal information 101, and calculates the probability that the position of each body part of the subject is in an abnormal state. It has a function that makes it possible to determine whether a position is in an abnormal state.
  • the information processing device 100 uses the first model to identify any part of the skeleton information 101 at the first point in time in the acquired time series data that is in an abnormal state in terms of position.
  • the information processing device 100 determines the second model based on the feature amount related to the skeleton information 101 in the acquired time series data. For example, the information processing device 100 generates a second probability distribution model that constrains the position of any of the specified parts in the skeletal information 101 at the first time point according to the probability that the part is in an abnormal state. Determine. The probability is calculated by the first model, for example, based on the feature amount regarding the skeleton information 101 in the time series data. In the example of FIG. 1, the information processing device 100 specifically uses a probability distribution that constrains the position of the joint 1 according to the magnitude of the probability that the joint 1 is in an abnormal state in the skeletal information 101 at the time T. A second model is determined.
  • the information processing device 100 has nodes 111 indicating the positions of respective parts at each time point, and first edges 112 connecting nodes indicating the positions of different biologically connected parts at each time point.
  • a graph 110 containing the information is generated.
  • the information processing device 100 associates the determined second model with a node 111 indicating the position of one of the identified parts.
  • the information processing device 100 specifically associates the determined second model with a node 111 indicating the position of the subject's joint 1 at time T, and generates the graph 110.
  • the information processing device 100 corrects the skeleton information 101 at the first point in time in the time series data based on the generated graph 110. For example, the information processing device 100 corrects the position of the subject's joint 1 included in the skeletal information 101 at time T in the time series data. Thereby, the information processing device 100 can accurately identify the position of each joint of the subject. The information processing device 100 can accurately identify temporal changes in the positions of each joint of the subject.
  • the information processing apparatus 100 uses the first model to identify any part of the skeleton information 101 at the first point in time series data that is in an abnormal state with respect to position. It is not limited to this.
  • the information processing device 100 may identify any part that is in an abnormal state with respect to the position of the skeletal information 101 at the first point in time series data without using the first model. .
  • a plurality of computers may cooperate to realize the functions of the information processing device 100.
  • a computer identifies any part that is in an abnormal state in terms of position, a computer generates a graph 110, and based on the graph 110, the skeletal information 101 at the first point in time series data is corrected.
  • computers may also be cases in which computers cooperate with each other.
  • FIG. 2 is an explanatory diagram showing an example of the information processing system 200.
  • an information processing system 200 includes an information processing device 100, one or more image capturing devices 201, and one or more client devices 202.
  • the information processing device 100 and the image capturing device 201 are connected via a wired or wireless network 210.
  • the network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like. Further, in the information processing system 200, the information processing device 100 and the client device 202 are connected via a wired or wireless network 210.
  • the information processing device 100 acquires a plurality of images of the subject from different angles at different times from one or more image capturing devices 201.
  • the information processing device 100 identifies the distribution of the existence probability of each part of the subject in a three-dimensional space based on the plurality of acquired images at each point in time, and identifies the three-dimensional coordinates of each part of the subject. do.
  • the information processing device 100 calculates the degree of abnormality of each part of the subject in terms of three-dimensional coordinates based on the plurality of acquired images at each time point, and determines whether any part of the subject is abnormal in terms of three-dimensional coordinates. It is identified as an abnormal area that is in a certain state.
  • the information processing device 100 determines a probability distribution model that constrains the position of the identified abnormal region.
  • the information processing device 100 includes nodes indicating the three-dimensional coordinates of each part of the subject at each specified point in time and edges connecting the nodes, and determines the node to indicate the three-dimensional coordinates of the abnormal part. Generate a graph that associates models. The information processing device 100 refers to the graph and corrects the three-dimensional coordinates of each part of the identified subject.
  • the information processing device 100 outputs the corrected three-dimensional coordinates of each part of the subject.
  • the output format includes, for example, displaying on a display, printing out to a printer, transmitting to another computer, or storing in a storage area.
  • the information processing device 100 transmits the corrected three-dimensional coordinates of each part of the subject to the client device 202.
  • the information processing device 100 is, for example, a server, a PC (Personal Computer), or the like.
  • the image capturing device 201 is a computer that captures an image of a subject.
  • the image capturing device 201 includes a camera having a plurality of image sensors, and captures an image of the subject using the camera.
  • the image capturing device 201 generates an image of the subject and transmits it to the information processing device 100.
  • the image capturing device 201 is, for example, a smartphone.
  • the image capturing device 201 may be, for example, a fixed point camera.
  • the image capturing device 201 may be, for example, a drone.
  • the client device 202 receives the three-dimensional coordinates of each part of the subject from the information processing device 100.
  • the client device 202 outputs the received three-dimensional coordinates of each body part of the subject so that the user can refer to them.
  • the client device 202 displays, for example, the received three-dimensional coordinates of each part of the subject on a display.
  • the client device 202 is, for example, a PC, a tablet terminal, or a smartphone.
  • the information processing device 100 is a device different from the image capturing device 201, but the invention is not limited to this.
  • the information processing device 100 may have a function as the image capturing device 201 and may also operate as the image capturing device 201.
  • the information processing device 100 may have a function as the client device 202 and may also operate as the client device 202.
  • FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing device 100.
  • the information processing apparatus 100 includes a CPU (Central Processing Unit) 301, a memory 302, a network I/F (Interface) 303, a recording medium I/F 304, and a recording medium 305.
  • Information processing device 100 further includes a display 306 and an input device 307. Further, each component is connected to each other by a bus 300.
  • the CPU 301 controls the entire information processing device 100.
  • the memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and a RAM is used as a work area for the CPU 301.
  • the program stored in the memory 302 is loaded into the CPU 301 and causes the CPU 301 to execute the coded processing.
  • the network I/F 303 is connected to a network 210 through a communication line, and is connected to other computers via the network 210.
  • the network I/F 303 serves as an internal interface with the network 210, and controls data input/output from other computers.
  • the network I/F 303 is, for example, a modem or a LAN adapter.
  • the recording medium I/F 304 controls reading/writing of data to/from the recording medium 305 under the control of the CPU 301.
  • the recording medium I/F 304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, or the like.
  • the recording medium 305 is a nonvolatile memory that stores data written under the control of the recording medium I/F 304.
  • the recording medium 305 is, for example, a disk, a semiconductor memory, a USB memory, or the like.
  • the recording medium 305 may be removable from the information processing apparatus 100.
  • the display 306 displays data such as a cursor, icon, toolbox, document, image, or functional information.
  • the display 306 is, for example, a CRT (Cathode Ray Tube), a liquid crystal display, or an organic EL (Electroluminescence) display.
  • the input device 307 has keys for inputting characters, numbers, various instructions, etc., and inputs data.
  • the input device 307 is, for example, a keyboard or a mouse.
  • the input device 307 may be, for example, a touch panel type input pad or a numeric keypad.
  • the information processing device 100 may include, for example, a camera. Further, the information processing device 100 may include, for example, a printer, a scanner, a microphone, a speaker, or the like in addition to the components described above. Further, the information processing apparatus 100 may include a plurality of recording medium I/Fs 304 and recording media 305. Furthermore, the information processing apparatus 100 does not need to have the display 306 or the input device 307. Further, the information processing apparatus 100 does not need to have the recording medium I/F 304 and the recording medium 305.
  • FIG. 4 is a block diagram showing an example of the hardware configuration of the image capturing device 201.
  • the image capturing device 201 includes a CPU 401, a memory 402, a network I/F 403, a recording medium I/F 404, a recording medium 405, and a camera 406. Further, each component is connected to each other by a bus 400.
  • Memory 402 includes, for example, ROM, RAM, flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and a RAM is used as a work area for the CPU 401. The program stored in the memory 402 is loaded into the CPU 401 and causes the CPU 401 to execute the coded processing.
  • the network I/F 403 is connected to a network 210 through a communication line, and is connected to other computers via the network 210.
  • the network I/F 403 serves as an internal interface with the network 210, and controls data input/output from other computers.
  • the network I/F 403 is, for example, a modem or a LAN adapter.
  • the recording medium I/F 404 controls data read/write to the recording medium 405 under the control of the CPU 401.
  • the recording medium I/F 404 is, for example, a disk drive, an SSD, a USB port, or the like.
  • the recording medium 405 is a nonvolatile memory that stores data written under the control of the recording medium I/F 404.
  • the recording medium 405 is, for example, a disk, a semiconductor memory, a USB memory, or the like.
  • the recording medium 405 may be removable from the image capturing device 201.
  • the camera 406 has a plurality of image sensors, and generates an image of a target object captured by the plurality of image sensors. Camera 406 is, for example, a competition camera. Camera 406 is, for example, a surveillance camera.
  • the image capturing device 201 may include, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, and the like. Further, the image capturing device 201 may include a plurality of recording medium I/Fs 404 and recording media 405. Further, the image capturing device 201 does not need to have the recording medium I/F 404 or the recording medium 405.
  • Example of hardware configuration of client device 202 The example hardware configuration of the client device 202 is specifically the same as the example hardware configuration of the information processing device 100 shown in FIG. 3, so the description thereof will be omitted.
  • FIG. 5 is a block diagram showing an example of the functional configuration of the information processing device 100.
  • the information processing device 100 includes a storage unit 500, an acquisition unit 501, an analysis unit 502, a learning unit 503, a specification unit 504, a determination unit 505, a generation unit 506, a correction unit 507, and an output unit 508. including.
  • the storage unit 500 is realized, for example, by a storage area such as the memory 302 or the recording medium 305 shown in FIG. 3. Although a case will be described below in which the storage unit 500 is included in the information processing device 100, the present invention is not limited to this. For example, there may be a case where the storage unit 500 is included in a device different from the information processing device 100, and the storage contents of the storage unit 500 can be referenced from the information processing device 100.
  • the acquisition unit 501 to output unit 508 function as an example of a control unit. Specifically, the acquisition unit 501 to the output unit 508 execute the program by causing the CPU 301 to execute a program stored in a storage area such as the memory 302 or the recording medium 305 shown in FIG. This function is realized by The processing results of each functional unit are stored in a storage area such as the memory 302 or the recording medium 305 shown in FIG. 3, for example.
  • the storage unit 500 stores various information that is referenced or updated in the processing of each functional unit.
  • the storage unit 500 stores, for example, a plurality of images taken of a specific person from different angles at each of a plurality of consecutive time points. The angle indicates the imaging position.
  • the image is acquired by the acquisition unit 501, for example.
  • the storage unit 500 stores, for example, time-series data of skeletal information.
  • the time series data includes skeletal information at each of a plurality of consecutive time points.
  • the skeletal information includes the position of each of a plurality of body parts of a specific person.
  • the site is, for example, a joint. Sites include, for example, the neck, head, right and left shoulders, right and left elbows, right and left hands, right and left knees, right and left feet, and the like.
  • the position is, for example, a three-dimensional coordinate.
  • the time series data is acquired by the acquisition unit 501.
  • the time series data may be generated by the analysis unit 502, for example.
  • the acquisition unit 501 acquires various information used in the processing of each functional unit.
  • the acquisition unit 501 stores the acquired various information in the storage unit 500 or outputs it to each functional unit. Further, the acquisition unit 501 may output various information stored in the storage unit 500 to each functional unit.
  • the acquisition unit 501 acquires various information based on, for example, a user's operation input.
  • the acquisition unit 501 may receive various information from a device different from the information processing device 100, for example.
  • the acquisition unit 501 acquires, for example, time-series data of the subject's skeletal information.
  • the skeletal information of the subject includes, for example, the position of each of a plurality of body parts of the subject.
  • the acquisition unit 501 acquires time-series data of the skeletal information of the subject by accepting input of time-series data of the skeletal information of the subject based on the user's operation input.
  • the acquisition unit 501 may acquire time-series data of the subject's skeletal information by receiving it from another computer.
  • the acquisition unit 501 may, for example, acquire time-series data of skeletal information of past subjects.
  • the subject may be, for example, the same as the subject.
  • the subject's skeletal information includes, for example, the position of each of a plurality of body parts of the subject.
  • the acquisition unit 501 acquires time-series data of the skeletal information of the subject by accepting input of time-series data of the skeletal information of the subject based on the user's operation input.
  • the acquisition unit 501 may acquire time-series data of the subject's skeletal information by receiving it from another computer.
  • the acquisition unit 501 acquires, for example, a plurality of images of the subject taken from different angles at each of a plurality of consecutive time points.
  • the acquisition unit 501 acquires a plurality of images when the analysis unit 502 generates time-series data of the subject's skeletal information without acquiring it. Thereby, the acquisition unit 501 can enable the analysis unit 502 to generate time-series data of the subject's skeletal information.
  • the acquisition unit 501 may acquire a plurality of images of the subject taken from different angles at each of a plurality of consecutive time points.
  • the acquisition unit 501 acquires a plurality of images. Thereby, the acquisition unit 501 can enable the analysis unit 502 to generate time-series data of the subject's skeletal information.
  • the acquisition unit 501 may receive a start trigger to start processing of any functional unit.
  • the start trigger is, for example, a predetermined operation input by the user.
  • the start trigger may be, for example, receiving predetermined information from another computer.
  • the start trigger may be, for example, that any functional unit outputs predetermined information.
  • the acquisition unit 501 may accept the acquisition of a plurality of images as a start trigger for starting the processing of the analysis unit 502.
  • the acquisition unit 501 may receive, for example, the acquisition of time-series data of the subject's skeletal information as a start trigger for starting the processing of the learning unit 503.
  • the acquisition unit 501 may receive the acquisition of time-series data of the subject's skeletal information as a start trigger for starting the processing of the identification unit 504, the determination unit 505, the generation unit 506, and the modification unit 507. .
  • the analysis unit 502 generates time-series data of skeletal information of a predetermined person.
  • the analysis unit 502 generates, for example, time-series data of the subject's skeletal information. Specifically, the analysis unit 502 estimates the position of each part of the subject at each time point based on a plurality of images taken of the subject from different angles at each of a plurality of time points, and performs estimation. Generates skeletal information of the subject including the location. Specifically, the analysis unit 502 generates time-series data of the skeletal information of the subject based on the generated skeletal information of the subject. Thereby, the analysis unit 502 can provisionally specify the position of each part of the subject at each point in time, and can obtain correction targets.
  • the analysis unit 502 may generate time-series data of the subject's skeletal information. Specifically, the analysis unit 502 generates skeletal information of the subject at each time point based on a plurality of images taken of the subject from different angles at each of a plurality of time points, and generates skeletal information of the subject at each time point. Generate series data. The analysis unit 502 may add noise to the generated time series data of the subject's skeletal information. The analysis unit 502 sets the subject's skeletal information as teacher information for generating a learning model. Thereby, the analysis unit 502 can obtain teacher information for generating a learning model.
  • the learning unit 503 learns a learning model based on teacher information including the position of each of a plurality of body parts of the subject.
  • the learning model makes it possible to identify any part of a given person that is in an abnormal position among multiple parts of the given person, according to features related to skeletal information in time-series data of the given person's skeletal information.
  • the learning model has, for example, a function that makes it possible to determine whether or not each body part of a predetermined person is in an abnormal state in terms of position.
  • the learning model has a function of calculating an index value indicating the probability that each part of a predetermined person is in an abnormal state with respect to position. More specifically, the learning model is an index that indicates the probability that each part of a given person is in an abnormal state in terms of position, in response to input of feature amounts related to skeletal information. Output the value.
  • the learning model is a neural network. Thereby, the learning unit 503 can specify one of the plurality of parts of the subject whose position is abnormal.
  • the identification unit 504 identifies abnormalities regarding the position of the skeletal information at the first point in time in the acquired time series data of the skeletal information of the target person, based on the feature amount related to the skeletal information in the acquired time series data of the skeletal information of the target person. Identify any part of the condition. For example, the identifying unit 504 uses the learned learning model to identify an abnormal state regarding the position of the skeletal information at the first time point based on the feature amount regarding the skeletal information in the acquired time series data of the skeletal information of the subject. Identify any part that is.
  • the identifying unit 504 inputs into the learning model the feature amounts related to the skeletal information in the time-series data of the skeletal information of the subject, thereby determining the skeletal information of each part of the subject at the first time point.
  • An index value indicating the probability of an abnormal state is calculated.
  • the specifying unit 504 specifies, based on the calculated index value, any part that is in an abnormal state with respect to the position of the skeletal information at the first time point. More specifically, the specifying unit 504 specifies, among the plurality of parts of the subject, any part whose calculated index value is equal to or greater than a threshold value as a part that is in an abnormal state with respect to position.
  • the identifying unit 504 can obtain guidelines for correcting the position of each of the plurality of body parts of the subject.
  • the specifying unit 504 can make it possible to determine which part of the subject's position is preferable to be corrected.
  • the identifying unit 504 refers to a predetermined rule and determines whether the skeletal information at the first time point is in an abnormal state regarding the position based on the feature amount regarding the skeletal information in the acquired time series data of the skeletal information of the subject. Any particular site may be specified.
  • the predetermined rule may be such that it is possible to identify one of a plurality of body parts of a subject that is in an abnormal state in terms of position, according to the feature amount related to skeletal information in time-series data of the subject's skeletal information.
  • the predetermined rule may include a rule that allows calculation of an index value indicating the probability that each body part of the predetermined person is in an abnormal state with respect to position.
  • the identifying unit 504 refers to a predetermined rule and identifies each of the target persons with respect to the skeletal information at the first time point based on the feature amount regarding the skeletal information in the time series data of the skeletal information of the target person. An index value indicating the probability that the part is in an abnormal state is calculated. Specifically, the specifying unit 504 specifies, based on the calculated index value, any part that is in an abnormal state with respect to the position of the skeletal information at the first time point. More specifically, the specifying unit 504 specifies, among the plurality of parts of the subject, any part whose calculated index value is equal to or greater than a threshold value as a part that is in an abnormal state with respect to position. Thereby, the identifying unit 504 can obtain guidelines for correcting the position of each of the plurality of body parts of the subject. The specifying unit 504 can make it possible to determine which part of the subject's position is preferable to be corrected.
  • the determining unit 505 determines a distribution model of a probability distribution that constrains the position of the specified part in the skeletal information at the first time point based on the feature amount related to the skeletal information in the acquired time series data of the skeletal information of the subject.
  • the distribution model is, for example, a model for constraining the position of any part according to an index value indicating the probability that the part is in an abnormal state. Thereby, the determining unit 505 can obtain guidelines for correcting the position of the region identified by the identifying unit 504.
  • the generation unit 506 includes a node indicating the position of each part at each time point and a first edge connecting nodes indicating the positions of different biologically connected parts at each time point, and generates the position of the identified part.
  • a graph is generated in which the determined distribution model is associated with the indicated node. Thereby, the generation unit 506 can modify the skeletal information at the first point in time in the time series data of the skeletal information of the subject.
  • the generation unit 506 includes a node, a first edge, and a second edge that connects nodes indicating the position of any part at different times, and applies the determined distribution model to the node indicating the position of the specified part.
  • a corresponding graph may also be generated. Thereby, the generation unit 506 can modify the skeletal information at the first point in time in the time series data of the skeletal information of the subject.
  • the modification unit 507 modifies the skeletal information at the first point in time in the time series data of the skeletal information of the subject based on the generated graph.
  • the modification unit 507 modifies the skeletal information at the first point in time in the time series data of the skeletal information of the subject, for example, by optimizing the generated graph. Thereby, the modification unit 507 can accurately specify the position of each part of the subject, taking into consideration the probability that each part of the subject is in an abnormal state.
  • the output unit 508 outputs the processing result of at least one of the functional units.
  • the output format is, for example, displaying on a display, printing out to a printer, transmitting to an external device via network I/F 303, or storing in a storage area such as memory 302 or recording medium 305. Thereby, the output unit 508 can notify the user of the processing results of at least one of the functional units, thereby improving the usability of the information processing apparatus 100.
  • the output unit 508 outputs, for example, the skeleton information at the first time point corrected by the correction unit 507. Specifically, the output unit 508 transmits the skeleton information at the first time point modified by the modification unit 507 to the client device 202. Specifically, the output unit 508 displays the skeleton information at the first time point corrected by the correction unit 507 on the display. Thereby, the output unit 508 can make available the position of each part of the subject.
  • FIG. 6 is an explanatory diagram showing the flow of operations of the information processing device 100.
  • the information processing apparatus 100 acquires a plurality of multi-view images 600 obtained by capturing the subject at different times and from different angles.
  • the information processing apparatus 100 detects an area in which a target person appears in each multi-view image 600 by performing a person detection process on each of the plurality of multi-view images 600.
  • the information processing device 100 performs 2D (Dimension) pose estimation processing on each multi-view image 600 at each time point.
  • the information processing device 100 performs 2D pose estimation processing on each multi-view image 600 at each time point, thereby generating a 2-D pose estimation process that indicates the distribution of the existence probability of each joint of the subject in each multi-view image 600.
  • a heat map 601 is generated.
  • the 2D heat map 601 includes, for example, joint likelihoods indicating the probability of existence of any joint of the subject at each point in the 2D space corresponding to the multi-view image 600.
  • the information processing device 100 determines the joint of the subject in the multi-view image 600 based on the 2D heat map 601 that indicates the distribution of the existence probability of each joint of the subject in the multi-view image 600 at each point in time. Identify the 2D coordinates of.
  • the variance of the joint likelihood in the 2D heat map 601, which indicates the probability of the existence of a joint of the subject, can be treated as an index value indicating the accuracy of the specified 2D coordinates.
  • the information processing device 100 acquires placement information indicating the angle of each multi-view image 600 at each time point.
  • the information processing device 100 performs a 3D pose estimation process based on the placement information and the 2D coordinates of each joint of the subject in each multi-view image 600 at each point in time, thereby estimating the subject in the 3D space. Identify the 3D coordinates of each joint of the person.
  • the information processing device 100 generates a 3D skeleton inference result 602 including 3D coordinates of each joint of the identified subject at each time point, and generates time series data of the 3D skeleton inference result 602.
  • the information processing device 100 corrects the 3D skeleton inference result 602 by performing correction processing on the time series data of the 3D skeleton inference result 602.
  • the information processing apparatus 100 outputs the time series data of the corrected 3D skeleton inference result 603 in a usable manner. For example, the information processing apparatus 100 outputs time-series data of the corrected 3D skeleton inference result 603 so that the user can refer to it.
  • the user performs a predetermined analysis process based on the time series data of the corrected 3D skeleton inference result 603.
  • the analysis process is, for example, scoring participants in an athletic competition.
  • the user performs an analysis process to score the participants based on the time series data of the corrected 3D skeleton inference results 603.
  • the target person is a medical institution visitor undergoing rehabilitation, or a medical institution visitor receiving a diagnosis of athletic ability such as walking ability.
  • the analysis process is, for example, determining the effectiveness of rehabilitation, or diagnosing athletic ability or health status.
  • the user determines the effectiveness of rehabilitation of the medical institution visitor, or diagnoses the motor ability or health condition of the medical institution visitor. do.
  • the information processing device 100 may perform the above-described analysis process based on the time-series data of the corrected 3D skeleton inference result 603.
  • the information processing device 100 outputs the results of the analysis process so that the user can refer to them.
  • the information processing apparatus 100 may output the time-series data of the corrected 3D skeleton inference result 603 to the analysis unit 502 that performs the above-described analysis process.
  • the analysis unit 502 is included in a computer other than the information processing device 100, for example. Thereby, the information processing apparatus 100 can perform analysis processing with high accuracy.
  • FIGS. 7 to 9 a specific example of the correction process will be described using FIGS. 7 to 9. Specifically, first, using FIG. 7 and FIG. 8, a specific example in which the information processing device 100 identifies an abnormal joint that is determined to be in an abnormal state with respect to 3D coordinates among a plurality of joints of a subject. I will explain about it.
  • FIGS. 7 and 8 are explanatory diagrams showing specific examples of identifying abnormal joints.
  • the information processing apparatus 100 acquires time-series data of a plurality of original data 700.
  • Original data 700 shows the subject's skeletal information.
  • Original data 700 shows 3D coordinates of each of a plurality of joints of the subject.
  • the 3D coordinates of the joints are, for example, ⁇ in the figure.
  • the information processing device 100 generates processed data 701 by adding noise to the original data 700.
  • the information processing device 100 changes the processed data 701 by changing the 3D coordinates of at least one of the plurality of joints of the subject indicated by the original data 700 to 3D coordinates that are determined to be in an abnormal state.
  • the abnormal condition corresponds to, for example, a condition in which the 3D coordinates of a joint are incorrectly estimated.
  • the abnormal state is jitter, inversion, swap, or miss.
  • the information processing apparatus 100 can acquire time-series data of the processed data 701.
  • the information processing device 100 uses time-series data of the processed data 701 to learn an abnormality determination DNN (Deep Neural Network) 710.
  • the abnormality determination DNN 710 detects an abnormality in each joint of the subject in at least one of the 3D skeleton inference results 602 in accordance with the input of the feature amount of the 3D skeleton inference result 602 in the time series data of the 3D skeleton inference result 602. It has a function to output probability.
  • the abnormality probability indicates the probability that the 3D coordinates of the joints of the subject are in a positionally abnormal state.
  • the abnormality determination DNN 710 outputs the abnormality probability of each joint of the subject in the entire time series data in response to input of the feature amount of the 3D skeleton inference result 602 in the time series data of the 3D skeleton inference result 602. It may have a function. Next, the explanation will move on to FIG.
  • the information processing device 100 inputs the feature amount of the 3D skeleton inference result 602 in the time series data of the 3D skeleton inference result 602 to the abnormality determination DNN 710.
  • the information processing device 100 acquires the abnormality probability of each joint of the subject in the entire time series data of the 3D skeleton inference result 602 output by the abnormality determination DNN 710 in response to the input.
  • the information processing device 100 identifies abnormal joints based on the acquired abnormality probability of each joint of the subject. For example, the information processing device 100 specifies, as an abnormal joint, any joint for which the obtained abnormality probability is equal to or greater than a threshold value among the plurality of joints of the subject.
  • the information processing device 100 uses the abnormality determination DNN 710 to identify an abnormal joint, but the present invention is not limited to this.
  • the information processing device 100 may identify abnormal joints based on rules. Specifically, in the 3D skeleton inference result 602, the information processing device 100 stores a rule for calculating the abnormality probability of each joint according to the magnitude of the difference between the feature amount and the threshold value regarding each joint. You can stay there. Specifically, the information processing device 100 calculates the abnormality probability of each joint with reference to the stored rules, and identifies any joint for which the calculated abnormality probability is greater than or equal to a threshold value as an abnormal joint. It is possible to do so. Next, a specific example in which the information processing apparatus 100 generates the Factor Graph 900 will be described using FIG. 9 .
  • FIG. 9 is an explanatory diagram showing a specific example of generating a Factor Graph 900.
  • the information processing device 100 generates a Factor Graph 900.
  • the Factor Graph 900 includes, for example, nodes indicating the positions of each joint of the subject at each time point.
  • Factor Graph 900 analyzes the subject's head, upper cervical vertebrae, lower cervical vertebrae, thoracic vertebrae, lumbar vertebrae, left and right hip joints, left and right knee joints, left and right leg joints, left and right feet, left and right shoulder joints, and left and right elbows at each time point. Contains nodes that indicate the positions of joints, left and right wrists, and left and right hands.
  • the information processing device 100 may generate the Factor Graph 901 by connecting nodes indicating positions of predetermined joints at different times in the Factor Graph 900 with second edges.
  • the second edge may be associated with a pairwise term indicating a time series constraint determined for each type of joint.
  • the time series constraints are defined, for example, by probability distributions corresponding to uniform motion, uniform linear motion, uniform acceleration motion, or the like.
  • Pairwise terms indicating time series constraints corresponding to uniform linear motion are, for example, g t (x j,t-1 , x j,t ) ⁇ N(
  • characters with ⁇ attached to the top may be written as "characters ⁇ ".
  • x j,t-1 is the estimated position of the joint at time t-1.
  • x j,t is the estimated position of the joint at time t.
  • v j ⁇ is the average velocity of the joint.
  • ⁇ t is a unit time width.
  • ⁇ vj ⁇ is the velocity dispersion of the joint.
  • a Pairwise Term distribution model indicating a time series constraint corresponding to equipositional motion is associated.
  • the Pairwise Term is, for example, g t (x j,t-1 , x j,t ) to N (
  • ⁇ xj ⁇ is the joint position variance.
  • the information processing device 100 generates a Factor Graph 901 by connecting leaf nodes with only one connected edge indicating the positions of the same joint at different times in the Factor Graph 900 with a third edge. Good too.
  • a leaf node is, for example, a node to which only one first edge is connected and no second edge is connected.
  • the information processing device 100 determines whether a leaf node in the Factor Graph 900 to which only one first edge is connected and to which no second edge is connected is a node indicating the position of the identified abnormal joint. . If the leaf node indicates the position of the identified abnormal joint, the information processing apparatus 100 connects the leaf nodes at different times with the third edge 910.
  • the third edge 910 may be associated with a pairwise term indicating a predetermined time series constraint. Thereby, the information processing device 100 can accurately correct the position of the abnormal joint.
  • FIG. 10 is an explanatory diagram showing a specific example of correcting the 3D skeleton inference result 602.
  • the information processing apparatus 100 uses the generated Factor Graph 901 to modify the 3D skeleton inference result 602.
  • the Factor Graph 901 includes a node group 1010 corresponding to time t-1, a node group 1020 corresponding to time t, and so on.
  • the node group 1010 includes nodes 1011 to 1013 and the like.
  • the node group 1020 includes nodes 1021 to 1023 and the like.
  • the node 1011 and the node 1012 are connected by a first edge 1031.
  • node 1012 and node 1013 are connected by first edge 1032.
  • node 1021 and node 1022 are connected by first edge 1041.
  • node 1022 and node 1023 are connected by first edge 1042.
  • the first edge 1042 that connects the node 1022 and the node 1023 may be associated with a Pairwise term indicating a bone length constraint.
  • the information processing device 100 may further connect the node 1012 and the node 1022 with a second edge 1051, for example.
  • the second edge 1051 may be associated with, for example, a pairwise term indicating a time series constraint determined for each type of joint.
  • the node 1011 and the node 1021 are connected by a third edge 1061.
  • the third edge 1061 may be associated with a pairwise term indicating a predetermined time series constraint.
  • the information processing device 100 associates a node indicating the position of at least one joint of the Factor Graph 901 with a Unary Term indicating a constraint on the abnormal joint that acts to constrain the position of the joint according to the abnormality probability of the joint. .
  • the information processing device 100 associates, for example, a node 1021 indicating the position of the joint 1 among the node group 1020 with an Unary Term that includes the abnormality probability of the joint 1.
  • the Unary Term is, for example, f(x j ) to N(x j
  • p(x j ) is the abnormality probability.
  • the information processing device 100 corrects the position of each joint at each time point based on the Unary Term and Pairwise Term in the Factor Graph 901.
  • the information processing device 100 corrects the position of each joint at each time point, for example, by optimizing the Factor Graph 901.
  • the information processing device 100 can accurately correct the 3D skeleton inference result 602.
  • the information processing device 100 can accurately identify the position of each joint at each point in time. For example, the information processing device 100 can determine the position of each joint of the subject at each point in time with relatively high accuracy even when the subject performs relatively high-speed or relatively complex movements such as gymnastics. can be identified.
  • x j ⁇ is a weighted sum of joint likelihoods of a 3D heat map that integrates joint likelihoods of a plurality of 2D heat maps.
  • ⁇ 3D j ⁇ is the variance of the joint likelihood of a 3D heat map that integrates the joint likelihoods of a plurality of 2D heat maps.
  • the Unary Term acts to correct the 3D coordinates of any joint with a relatively low joint likelihood based on the 3D coordinates of another joint with a relatively high likelihood.
  • it may be difficult to accurately specify the 3D coordinates of each joint of the subject and it may be difficult to accurately correct temporal changes in the 3D coordinates of each joint of the person.
  • it is not always preferable to modify the 3D coordinates of a joint with a relatively low joint likelihood but it may be preferable to use the 3D coordinates of a joint with a relatively high joint likelihood as a reference. Not exclusively. Therefore, in the comparative example, it may not be possible to appropriately correct the 3D coordinates of each joint of the subject.
  • the information processing device 100 adopts f ( x j ) to N(x j
  • f ( x j ) to N(x j
  • FIG. 11 and FIG. 12 are explanatory diagrams showing a specific example of the flow of data processing in the operation example.
  • the information processing apparatus 100 acquires a plurality of camera images 1101 at each time point.
  • the information processing device 100 stores a 2D skeleton inference model 1110.
  • the information processing device 100 stores, for example, weight parameters that define a neural network that becomes the 2D skeleton inference model 1110.
  • the information processing apparatus 100 refers to the 2D skeleton inference model 1110 at each point in time and performs 2D skeleton inference processing on each of the plurality of camera images 1101, thereby obtaining a 2D skeleton inference result 1102. generate.
  • the 2D skeleton inference result 1102 includes, for example, 2D coordinates (x[pixel], y[pixel]) indicating the joint position and a likelihood indicating the certainty of the joint position.
  • the information processing device 100 stores a 3D skeleton inference model 1120.
  • the information processing device 100 stores, for example, weight parameters that define a neural network that becomes the 3D skeleton inference model 1120.
  • the information processing device 100 generates 3D skeleton inference results 1103 by referring to the 3D skeleton inference model 1120 and performing 3D skeleton inference processing on the plurality of 2D skeleton inference results 1102 at each time point.
  • the 3D skeleton inference result 1103 includes, for example, 3D coordinates (x [mm], y [mm], z [mm]) indicating the positions of joints.
  • the information processing apparatus 100 generates time series data 1104 that summarizes the 3D skeleton inference results 1103 for each time point. Next, the explanation will move on to FIG. 12.
  • the information processing device 100 stores an anomaly detection model 1210.
  • the information processing device 100 stores, for example, weight parameters that define a neural network serving as the anomaly detection model 1210.
  • the information processing device 100 performs anomaly detection processing on the time series data 1104 with reference to the anomaly detection model 1210 to calculate the anomaly probability for each joint, and includes the computed anomaly probability for each joint.
  • a skeletal abnormality detection result 1201 is generated.
  • the information processing device 100 stores a bone length model 1220 and a time-series motion model 1230.
  • Bone length model 1220 includes parameters that define Pairwise Term, which represents bone length constraints. The parameters are, for example, the average and variance of bone lengths.
  • the time-series motion model 1230 includes Pairwise Term indicating time-series constraints for each joint.
  • the information processing device 100 generates a Factor Graph by referring to the skeletal abnormality detection result 1201, the bone length model 1220, and the time-series motion model 1230 and performing a Factor Graph generation process.
  • the information processing device 100 corrects the position of each joint by performing optimization processing on the generated Factor Graph.
  • the information processing apparatus 100 generates a corrected 3D skeleton inference result 1202 that includes the corrected positions of each joint.
  • the overall processing is realized by, for example, the CPU 301 shown in FIG. 3, storage areas such as the memory 302 and the recording medium 305, and the network I/F 303.
  • FIG. 13 is a flowchart illustrating an example of the overall processing procedure.
  • the information processing apparatus 100 acquires time-series data of the three-dimensional skeleton inference results of the subject (step S1301).
  • the information processing device 100 calculates the degree of abnormality of each part of the subject based on the acquired time series data of the three-dimensional skeleton inference results of the subject, and calculates the degree of abnormality of each part of the subject based on the calculated degree of abnormality.
  • An abnormal region is identified among the plurality of regions (step S1302).
  • the information processing apparatus 100 then calculates the likelihood of each part of the subject based on the degree of abnormality of each part of the subject (step S1303).
  • the information processing device 100 outputs time-series data of the corrected three-dimensional skeleton inference results of the subject (step S1306). Then, the information processing device 100 ends the overall processing. Thereby, the information processing device 100 can accurately correct the three-dimensional skeleton inference result of the subject. Therefore, the information processing device 100 can improve the usefulness of the three-dimensional skeleton inference result of the subject. The information processing device 100 can, for example, improve the accuracy of analysis processing based on the 3D skeleton inference results of the subject.
  • time-series data of skeletal information including the position of each of a plurality of body parts of a subject can be acquired.
  • the information processing device 100 based on the feature amount related to the skeletal information in the acquired time-series data, one of the parts that is in an abnormal state with respect to the position of the skeletal information at the first time point in the acquired time-series data is identified. can do.
  • a probability distribution is created that constrains the position of any identified part according to the probability that the identified part is in an abnormal state. model can be determined.
  • the information processing apparatus 100 it is possible to generate a graph including nodes indicating the positions of respective parts at each time point. According to the information processing apparatus 100, it is possible to provide a first edge that connects nodes indicating the positions of different biologically connected parts at each time point in the graph. According to the information processing apparatus 100, it is possible to associate the determined model with a node indicating the position of one of the specified parts in the graph. According to the information processing device 100, the skeleton information at the first point in time in the time series data can be corrected based on the generated graph. Thereby, the information processing device 100 can accurately correct the skeleton information at the first time point.
  • the information processing device 100 it is possible to identify which part of a plurality of parts of a subject is in an abnormal state in terms of position, based on the teacher information including the position of each of the plurality of parts of the subject. It is possible to learn a model to According to the information processing device 100, the learned model is used to detect an abnormal state regarding the position of the skeletal information at the first point in time in the acquired time series data, based on the feature amount regarding the skeletal information in the acquired time series data. It is possible to specify any part where . Thereby, the information processing apparatus 100 can accurately identify any part that is in an abnormal state with respect to position.
  • a rule is stored that allows identifying one of a plurality of body parts of a subject that is in an abnormal state in terms of position, according to a feature amount related to skeletal information in time-series data. be able to.
  • the information processing device 100 with reference to the rules, it is determined that the skeletal information at the first point in time in the acquired time series data is in an abnormal state in terms of position, based on the feature amount regarding the skeletal information in the acquired time series data. Any part can be specified. Thereby, the information processing apparatus 100 can accurately identify any part that is in an abnormal state with respect to position.
  • the information processing device 100 it is possible to generate a graph that includes nodes, first edges, and second edges that connect nodes indicating the positions of any parts at different times. Thereby, the information processing apparatus 100 can easily correct the skeleton information at the first time point with high accuracy.
  • the information processing method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a PC or a workstation.
  • the information processing program described in this embodiment is recorded on a computer-readable recording medium, and executed by being read from the recording medium by the computer.
  • the recording medium may be a hard disk, a flexible disk, a CD (Compact Disc)-ROM, an MO (Magneto Optical Disc), a DVD (Digital Versatile Disc), or the like.
  • the information processing program described in this embodiment may be distributed via a network such as the Internet.
  • Information processing device 101 Skeletal information 110 Graph 111, 1011 to 1013, 1021 to 1023 Node 112, 1031, 1032, 1041, 1042 First edge 200 Information processing system 201 Image capturing device 202 Client device 210 Network 300, 400 Bus 301, 401 CPU 302,402 Memory 303,403 Network I/F 304, 404 Recording medium I/F 305,405 Recording medium 306 Display 307 Input device 406 Camera 500 Storage unit 501 Acquisition unit 502 Analysis unit 503 Learning unit 504 Specification unit 505 Determination unit 506 Generation unit 507 Correction unit 508 Output unit 600 Multi-view image 601 2D heat map 602,603 ,1103,1202 3D skeleton inference result 700 Original data 701 Processed data 710 Abnormality determination DNN 900,901 Factor Graph 910, 1061 Third edge 1010, 1020 Node group 1051 Second edge 1101 Camera image 1102 2D skeleton inference result 1104 Time series data 1110 2D skeleton inference

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Image Analysis (AREA)
PCT/JP2022/016363 2022-03-30 2022-03-30 情報処理プログラム、情報処理方法、および情報処理装置 Ceased WO2023188216A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
JP2024510981A JP7684622B2 (ja) 2022-03-30 2022-03-30 情報処理プログラム、情報処理方法、および情報処理装置
CN202280094183.0A CN119343695A (zh) 2022-03-30 2022-03-30 信息处理程序、信息处理方法、以及信息处理装置
PCT/JP2022/016363 WO2023188216A1 (ja) 2022-03-30 2022-03-30 情報処理プログラム、情報処理方法、および情報処理装置
EP22935361.0A EP4502924A4 (en) 2022-03-30 2022-03-30 Information processing program, information processing method, and information processing device
US18/887,451 US20250014215A1 (en) 2022-03-30 2024-09-17 Non-transitory computer-readable recording medium storing information processing program, information processing method, and information processing device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/016363 WO2023188216A1 (ja) 2022-03-30 2022-03-30 情報処理プログラム、情報処理方法、および情報処理装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/887,451 Continuation US20250014215A1 (en) 2022-03-30 2024-09-17 Non-transitory computer-readable recording medium storing information processing program, information processing method, and information processing device

Publications (1)

Publication Number Publication Date
WO2023188216A1 true WO2023188216A1 (ja) 2023-10-05

Family

ID=88199817

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/016363 Ceased WO2023188216A1 (ja) 2022-03-30 2022-03-30 情報処理プログラム、情報処理方法、および情報処理装置

Country Status (5)

Country Link
US (1) US20250014215A1 (https=)
EP (1) EP4502924A4 (https=)
JP (1) JP7684622B2 (https=)
CN (1) CN119343695A (https=)
WO (1) WO2023188216A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118965236A (zh) * 2024-10-10 2024-11-15 河北华通科技股份有限公司 一种基于边缘智控的能源设备数据监测方法及系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170344829A1 (en) 2016-05-31 2017-11-30 Microsoft Technology Licensing, Llc Skeleton -based action detection using recurrent neural network
JP2020042476A (ja) 2018-09-10 2020-03-19 国立大学法人 東京大学 関節位置の取得方法及び装置、動作の取得方法及び装置
WO2021002025A1 (ja) 2019-07-04 2021-01-07 富士通株式会社 骨格認識方法、骨格認識プログラム、骨格認識システム、学習方法、学習プログラムおよび学習装置
WO2021064942A1 (ja) 2019-10-03 2021-04-08 富士通株式会社 評価方法、評価プログラムおよび情報処理システム
CN112991656A (zh) * 2021-02-04 2021-06-18 北京工业大学 基于姿态估计的全景监控下人体异常行为识别报警系统及方法
JP2021105887A (ja) * 2019-12-26 2021-07-26 国立大学法人 東京大学 ３ｄポーズ取得方法及び装置
CN113191230A (zh) * 2021-04-20 2021-07-30 内蒙古工业大学 一种基于步态时空特征分解的步态识别方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170344829A1 (en) 2016-05-31 2017-11-30 Microsoft Technology Licensing, Llc Skeleton -based action detection using recurrent neural network
JP2020042476A (ja) 2018-09-10 2020-03-19 国立大学法人 東京大学 関節位置の取得方法及び装置、動作の取得方法及び装置
WO2021002025A1 (ja) 2019-07-04 2021-01-07 富士通株式会社 骨格認識方法、骨格認識プログラム、骨格認識システム、学習方法、学習プログラムおよび学習装置
WO2021064942A1 (ja) 2019-10-03 2021-04-08 富士通株式会社 評価方法、評価プログラムおよび情報処理システム
JP2021105887A (ja) * 2019-12-26 2021-07-26 国立大学法人 東京大学 ３ｄポーズ取得方法及び装置
CN112991656A (zh) * 2021-02-04 2021-06-18 北京工业大学 基于姿态估计的全景监控下人体异常行为识别报警系统及方法
CN113191230A (zh) * 2021-04-20 2021-07-30 内蒙古工业大学 一种基于步态时空特征分解的步态识别方法

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ISKAKOV, KARIM ET AL.: "Learnable triangulation of human pose", PROCEEDINGS OF THE IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION, 2019
MOON, GYEONGSIKJU YONG CHANGKYOUNG MU LEE: "V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map", PROCEEDINGS OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2018
See also references of EP4502924A4
WEI SHIH-EN; RAMAKRISHNA VARUN; KANADE TAKEO; SHEIKH YASER: "Convolutional Pose Machines", 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 27 June 2016 (2016-06-27), pages 4724 - 4732, XP033021664, DOI: 10.1109/CVPR.2016.511 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118965236A (zh) * 2024-10-10 2024-11-15 河北华通科技股份有限公司 一种基于边缘智控的能源设备数据监测方法及系统

Also Published As

Publication number Publication date
US20250014215A1 (en) 2025-01-09
JP7684622B2 (ja) 2025-05-28
CN119343695A (zh) 2025-01-21
JPWO2023188216A1 (https=) 2023-10-05
EP4502924A1 (en) 2025-02-05
EP4502924A4 (en) 2025-04-16

Similar Documents

Publication Publication Date Title
US10817795B2 (en) Handstate reconstruction based on multiple inputs
CN104035557B (zh) 一种基于关节活跃度的Kinect动作识别方法
US11759126B2 (en) Scoring metric for physical activity performance and tracking
JPWO2018087933A1 (ja) 情報処理装置、情報処理方法、およびプログラム
US20220284652A1 (en) System and method for matching a test frame sequence with a reference frame sequence
Varshney et al. RETRACTED ARTICLE: Rule-based multi-view human activity recognition system in real time using skeleton data from RGB-D sensor: N. Varshney et al.
WO2022018811A1 (ja) 被写体の３次元姿勢推定装置、３次元姿勢推定方法、及びプログラム
KR20230112636A (ko) 정보 처리 장치, 정보 처리 방법 및 프로그램
Wu et al. An unsupervised real-time framework of human pose tracking from range image sequences
JP2023527627A (ja) 逆運動学に基づいた関節の回転の推測
Jatesiktat et al. Personalized markerless upper-body tracking with a depth camera and wrist-worn inertial measurement units
US20250014215A1 (en) Non-transitory computer-readable recording medium storing information processing program, information processing method, and information processing device
JPWO2023188216A5 (https=)
WO2025054192A1 (en) Unsupervised depth features for three-dimensional pose estimation
JP7727242B2 (ja) 情報処理プログラム、情報処理方法、および情報処理装置
Klein et al. Assessing the reliability of AI-based angle detection for shoulder and elbow rehabilitation
CN117612251A (zh) 一种人体姿态识别方法及系统
Slupczynski et al. Analyzing Exercise Repetitions: YOLOv8-Enhanced Dynamic Time Warping Approach on InfiniteRep Dataset
US20260120315A1 (en) Approaches to generating semi-synthetic training data for real-time estimation of pose and systems for implementing the same
JP7419993B2 (ja) 信頼度推定プログラム、信頼度推定方法、および信頼度推定装置
Ang et al. Markerless Human Motion Analysis for Telerehabilitation: A Case Study on Squat
Rishabh et al. Enhancing Exercise Form: A Pose Estimation Approach with Body Landmark Detection
Agatha et al. Human pose estimation: A systematic literature review
Shraddha et al. AI-Driven 3D Motion Capture and Human Body Detection
Pavlikov et al. On the Application of Human Pose Estimation in a Driver Condition Monitoring Task

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22935361

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2024510981

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202280094183.0

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2022935361

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022935361

Country of ref document: EP

Effective date: 20241030

WWP Wipo information: published in national office

Ref document number: 202280094183.0

Country of ref document: CN