WO2021117165A1 - 生成方法、生成プログラム及び情報処理システム - Google Patents

生成方法、生成プログラム及び情報処理システム Download PDF

Info

Publication number
WO2021117165A1
WO2021117165A1 PCT/JP2019/048515 JP2019048515W WO2021117165A1 WO 2021117165 A1 WO2021117165 A1 WO 2021117165A1 JP 2019048515 W JP2019048515 W JP 2019048515W WO 2021117165 A1 WO2021117165 A1 WO 2021117165A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
cylindrical model
noise
dimensional point
influence
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/048515
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
村上　亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujitsu Ltd
Original Assignee
Fujitsu Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujitsu Ltd filed Critical Fujitsu Ltd
Priority to EP19955580.6A priority Critical patent/EP4075377A4/en
Priority to PCT/JP2019/048515 priority patent/WO2021117165A1/ja
Priority to CN201980102506.4A priority patent/CN114730497A/zh
Priority to JP2021563516A priority patent/JP7375829B2/ja
Publication of WO2021117165A1 publication Critical patent/WO2021117165A1/ja
Priority to US17/741,824 priority patent/US20220270408A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/149—Segmentation; Edge detection involving deformable models, e.g. active contour models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/30—Noise filtering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images
    • G06V2201/033—Recognition of patterns in medical or anatomical images of skeletal patterns

Definitions

  • the present invention relates to a generation method and the like.
  • Skeleton recognition determines the position of each joint of the subject.
  • FIG. 14 is a diagram for explaining a fitting process.
  • the cylindrical model data 1b that models the body shape of the subject is applied to the point cloud data 1a obtained from the measurement result of the depth sensor.
  • the skeleton model 1c is obtained by connecting the axes of each cylinder of the cylinder model data 1b that is the result of the fitting process, and the position of each joint of the subject can be estimated using the skeleton model 1c.
  • the cylindrical model data may be applied to the noise-derived point cloud instead of the subject-derived point cloud.
  • FIG. 15 is a diagram for explaining a problem of the prior art.
  • noise 2 is included in the point cloud data 1a.
  • a part of the cylinder 3 of the cylinder model data 1b may be applied to the noise 2.
  • the cylinder 3 is eventually moved to the noise 2 side. It applies.
  • the cylinder 3 is applied to the noise 2 in this way, the accuracy of the skeleton recognition result is lowered.
  • an object of the present invention is to provide a generation method, a generation program, and an information processing system capable of preventing a decrease in accuracy of a skeleton recognition result due to incorrect fitting due to noise.
  • the first plan is to have the computer perform the following processing.
  • the computer acquires 3D point cloud data based on the measurement result of the 3D sensor that measures the subject in 3D.
  • the computer evaluates the influence of noise on the measurement result for each part based on the result of applying a cylindrical model in which each part of the human body is represented by a plurality of cylinders to the three-dimensional point cloud data.
  • the computer excludes the point cloud around the cylindrical model corresponding to the part where the influence of noise is judged to be more than a predetermined value from the 3D point cloud data, and applies the cylindrical model to the 3D point cloud data excluding the point cloud. Repeat the process of reapplying.
  • the computer generates the skeleton recognition result of the subject based on the result of applying the cylindrical model to the 3D point cloud data when the influence of noise for each part is less than the predetermined value, and outputs the skeleton recognition result. To do.
  • FIG. 1 is a diagram for explaining an example of processing of the information processing system according to the present embodiment.
  • FIG. 2 is a diagram showing an example of an information processing system according to this embodiment.
  • FIG. 3 is a functional block diagram showing the configuration of the information processing apparatus according to the present embodiment.
  • FIG. 4 is a diagram showing an example of a data structure of cylindrical model data.
  • FIG. 5 is a diagram showing a correspondence relationship between the cylinder and the site identification information.
  • FIG. 6 is a diagram showing an example of a cylindrical parameter.
  • FIG. 7 is a diagram for explaining a process for evaluating the influence of noise.
  • FIG. 8 is a diagram for explaining an example of processing of the fitting processing unit.
  • FIG. 9 is a diagram showing an example of screen information.
  • FIG. 1 is a diagram for explaining an example of processing of the information processing system according to the present embodiment.
  • FIG. 2 is a diagram showing an example of an information processing system according to this embodiment.
  • FIG. 3 is a functional block diagram showing
  • FIG. 10 is a flowchart showing a processing procedure of the information processing apparatus according to the present embodiment.
  • FIG. 11 is a flowchart showing a processing procedure of noise evaluation processing.
  • FIG. 12 is a flowchart showing a processing procedure of the fitting process.
  • FIG. 13 is a diagram showing an example of a hardware configuration of a computer that realizes a function similar to that of an information processing device.
  • FIG. 14 is a diagram for explaining a fitting process.
  • FIG. 15 is a diagram for explaining a problem of the prior art.
  • FIG. 1 is a diagram for explaining an example of processing of the information processing system according to this embodiment.
  • the information processing system according to this embodiment uses the sensor 10 to measure the distance image data of the subject 5.
  • the distance image data is data including the distance between each point (pixel) on the surface of the subject 5 and the sensor 10.
  • the distance image data is converted into three-dimensional point cloud data 5a by the information processing device.
  • the three-dimensional point cloud data 5a is data indicating the three-dimensional coordinates of each point for the point cloud on the surface of the subject 5.
  • the information processing device executes a fitting process that applies the cylindrical model data 40 of the subject 5 to the three-dimensional point cloud data 5a.
  • the cylinder model data 40 is a model in which the length and thickness of each part of the subject 5 are approximated by the radii and lengths of a plurality of cylinders.
  • the information processing device evaluates the influence of noise based on the result of the fitting process. For example, in the process of evaluating the influence of noise, the number of points included in the point cloud around the cylinder is counted for each cylinder of the cylinder model data 40, and the number of counted points is less than expected (less than a predetermined number). It is a process to specify whether or not. The information processing device evaluates that the influence of noise is above a certain level when the number of points around any of the cylinders is less than expected.
  • the predetermined number is a value set for each part. For example, a predetermined number is set based on the surface area and resolution of the cylinder of each part.
  • the information processing apparatus evaluates that the influence of noise is above a certain level.
  • the information processing apparatus evaluates that the influence of noise is equal to or higher than a certain level, the information processing apparatus excludes the point cloud around the cylinder affected by noise and executes the fitting process again.
  • the information processing apparatus excludes the point cloud around the cylinder 7 and executes the fitting process again.
  • the information processing device repeatedly executes the fitting process by excluding the point cloud around the cylinder affected by the noise until the influence of the noise becomes less than a certain value, and specifies the posture of the final cylinder model data 40. Then, the information processing device specifies the skeleton model data 6 based on the posture of the final cylindrical model data 40.
  • the skeleton model data 6 corresponds to the skeleton recognition result of the subject 5.
  • edge noise is a result of the laser beam fading on the contour of the subject, and it seems that the points that should be observed at the original contour position of the subject fluttered to a position opposite to the sensor from the point cloud corresponding to the subject. In addition, it is the noise generated by being observed.
  • the noise including edge noise is a point cloud compared to the density of the point cloud (point cloud derived from the subject) corresponding to the observation result with the laser beam radiated to the subject. Density is low.
  • the number of points around the cylinder is smaller than that of the case where the points are applied to the subject-derived point group. Therefore, when the number of points around the cylinder is less than the predetermined number, it can be seen that the point cloud derived from noise has been fitted, and by further removing the point cloud derived from the noise, the noise is generated. It can be updated to 3D point cloud data with reduced influence. Further, by repeating the update of the three-dimensional point cloud data and the fitting process using the updated three-dimensional point cloud data, the point cloud derived from noise can be sequentially removed in this implementation. Further, in this embodiment, it is possible to reduce the influence of noise by removing the point cloud derived from noise, and finally to prevent the accuracy of the skeleton recognition result from being lowered due to erroneous fitting due to noise.
  • FIG. 2 is a diagram showing an example of an information processing system according to this embodiment. As shown in FIG. 2, this information processing system includes a sensor 10 and an information processing device 100. The sensor 10 is connected to the information processing device 100 by wire or wirelessly.
  • the sensor 10 is a depth sensor that measures the distance image data of the subject 5.
  • the sensor 10 outputs the distance image data to the information processing device 100 by a predetermined FPS (Frames Per Second).
  • the information processing apparatus 100 repeatedly executes the fitting process by excluding the point cloud around the cylinder affected by the noise until the influence of the noise becomes less than a certain level, and finally fitting. Identify the skeletal model data from the processing results.
  • the information processing device 100 repeatedly executes a process of specifying the skeleton model data every time the distance image data is acquired from the sensor 10.
  • the information processing device 100 recognizes the movement of the subject 5 based on the time-series skeleton model data. For example, as a movement, a technique included in a performance in a scoring competition is recognized, and screen information including a recognition result and a performance score is generated and displayed.
  • the subject 5 is described as an athlete performing gymnastics, but the present invention is not limited to this, and can be applied to other scoring competitions.
  • Other scoring competitions include trampoline, swimming dive, figure skating, karate kata, social dance, snowboarding, skateboarding, ski aerial and surfing. It may also be applied to classical ballet, ski jumping, mogul air, turns, baseball, basketball form checks, and the like. It may also be applied to competitions such as kendo, judo, wrestling, and sumo. Furthermore, it can be used to evaluate whether or not the weightlifting barbell has been raised.
  • FIG. 3 is a functional block diagram showing the configuration of the information processing device according to the present embodiment.
  • the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.
  • the communication unit 110 is a processing unit that executes data communication with an external device such as a sensor 10 via a network.
  • the control unit 150 which will be described later, exchanges data with an external device via the communication unit 110.
  • the communication unit 110 is an example of a communication device.
  • the input unit 120 is an input device that inputs various information to the control unit 150 of the information processing device 100.
  • the input unit 120 corresponds to a keyboard, a mouse, a touch panel, and the like.
  • the user operates the input unit 120 to request the display of screen information.
  • the display unit 130 is a display device that displays information output from the control unit 150.
  • the display unit 130 displays screen information such as skill certification and scoring results of various sports.
  • the display unit 130 corresponds to a liquid crystal display, an organic EL (Electro-Luminescence) display, a touch panel, and the like.
  • the storage unit 140 has a cylindrical model data 40, a distance image table 141, and a skeleton model table 142.
  • the storage unit 140 corresponds to semiconductor memory elements such as RAM (Random Access Memory) and flash memory (Flash Memory), and storage devices such as HDD (Hard Disk Drive).
  • the cylindrical model data 40 is model data in which a part of the human body (subject) is represented by a cylinder.
  • FIG. 4 is a diagram showing an example of a data structure of cylindrical model data.
  • the cylindrical model data 40 associates the site identification information with the parameters.
  • the site identification information is information for identifying each cylinder included in the cylinder model.
  • the parameters indicate the parameters of the cylinder identified by the site identification information. Cylinder parameters include the length of the cylinder, the radius of the cylinder, and so on. Different cylindrical model data may be prepared for each subject to be measured.
  • FIG. 5 is a diagram showing the correspondence between the cylinder and the part identification information. As shown in FIG. 5, each part identification information Md1 to Md14 is associated with each part of the cylindrical model data 40.
  • the site identification information Md1 to Md14 shown in FIG. 4 corresponds to the site identification information Md1 to Md14 shown in FIG.
  • FIG. 6 is a diagram showing an example of cylindrical parameters.
  • FIG. 6 shows one of a plurality of cylinders Md1 to Md11.
  • the parameters of the cylinder include the height l of the cylinder and the radius r of the cylinder.
  • the radius r of the cylinder corresponds to the thickness of the cylinder.
  • the distance image table 141 is a table that stores the distance image data measured by the sensor 10 in time series.
  • the skeleton model table 142 is a table that stores the skeleton model data described in FIG. 1 in chronological order.
  • the control unit 150 includes an acquisition unit 151, a fitting processing unit 152, a generation unit 153, and a technique recognition unit 154.
  • the control unit 150 is realized by hard-wired logic such as CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), and the like.
  • the acquisition unit 151 is a processing unit that acquires distance image data from the sensor 10.
  • the acquisition unit 151 registers the distance image data from the sensor 10 in the distance image table 141.
  • the acquisition unit 151 converts the distance image data into three-dimensional point cloud data based on the conversion table (not shown).
  • the conversion table is a table that defines the relationship between the pixel positions and distances of the distance image data and the three-dimensional coordinates.
  • the acquisition unit 151 outputs the three-dimensional point cloud data to the fitting processing unit 152.
  • the acquisition unit 151 repeatedly executes the above process every time the distance image data is acquired from the sensor 10.
  • the fitting processing unit 152 evaluates the influence of noise based on the result of fitting the cylindrical model data 40 to the three-dimensional point cloud data.
  • the fitting processing unit 152 excludes the point cloud around the cylindrical model data 40 from the three-dimensional point cloud data until the influence of noise becomes less than a certain value, and the cylindrical model with respect to the three-dimensional point cloud data excluding the point cloud.
  • the process of reapplying the data 40 is repeatedly executed.
  • the fitting processing unit 152 outputs the data of the final fitting result when the influence of noise becomes less than a certain value to the generation unit 153.
  • the fitting processing unit 152 executes a process of specifying the initial value of the parameter, a process of calculating the parameter using the EM (Expectation Maximization) algorithm, and a process of evaluating the influence of noise.
  • the "parameter” indicates each joint angle connecting each cylinder of the cylinder model data 40.
  • the fitting processing unit 152 will explain an example of processing for specifying the initial value of the parameter.
  • the fitting processing unit 152 uses the result of fitting processing for the n-1st 3D point cloud data as the initial value of the parameter of the nth 3D point cloud data.
  • the fitting processing unit 152 executes the following processing when the result of the fitting processing for the n-1st three-dimensional point cloud data does not exist.
  • the fitting processing unit 152 aligns the center of gravity of the cylindrical model data 40 with the center of gravity of the three-dimensional point cloud data, and has a degree of coincidence between the cylindrical model data 40 and the three-dimensional point cloud data for a plurality of predetermined parameters. May be calculated and the parameter with the highest degree of agreement may be used as the initial value.
  • the fitting processing unit 152 may input the three-dimensional point cloud data into the geometric model and calculate the initial value of the parameter.
  • the method disclosed in International Publication No. 2018/207292 may be used.
  • the fitting processing unit 152 compares the cylindrical model data 40 of the initial posture with the three-dimensional point cloud data, and calculates the posterior distribution p nm of the point cloud.
  • the initial value of the cylindrical model data 40 of the initial posture is set in the cylindrical model data 40.
  • ⁇ m included in the equation (1) indicates a surface residual, and it is assumed that it has a Gaussian distribution.
  • the surface residual indicates the difference in the vertical direction between the point cloud and the surface of the m-th cylinder.
  • the first cylinder indicates the cylinder having the site identification number Md1.
  • the surface residual ⁇ m is specified by x n and ⁇ 2.
  • x n indicates the three-dimensional coordinates of the nth point.
  • ⁇ indicates the variance of the three-dimensional coordinates of the point cloud included in the point cloud frame.
  • the fitting processing unit 152 updates the evaluation function Q defined by the equation (2) after calculating the posterior distribution p nm.
  • P represents the sum of p nm.
  • the fitting processing unit 152 calculates the amount of change ⁇ of the cylindrical parameter so that the likelihood of the evaluation function Q is minimized by using the Levenberg-Marquardt (LM) method. For example, the fitting processing unit 152 calculates the amount of change ⁇ based on the equations (3) and (4).
  • the fitting processing unit 152 is not limited to the LM method, and may randomly generate a change amount ⁇ .
  • the fitting processing unit 152 repeatedly executes the process of generating the amount of change ⁇ , updating the parameters, and calculating the likelihood of the evaluation function Q until a predetermined convergence condition is satisfied. For example, the fitting processing unit 152 may determine that the convergence condition is satisfied when the amount of change ⁇ is less than the predetermined amount of change and the likelihood of the evaluation function Q is less than the threshold value.
  • the cylindrical model data 40 of the parameter (posture) satisfying the convergence condition is referred to as "cylindrical model data 40 of the fitting processing result".
  • the fitting processing unit 152 selects a portion of the cylindrical model data 40 of the fitting processing result, and evaluates the influence of noise on the selected portion.
  • FIG. 7 is a diagram for explaining a process for evaluating the influence of noise.
  • a cylinder of the site identification information “Md13” will be used for explanation.
  • the fitting processing unit 152 counts the number of points included in the point group around the cylinder A13.
  • the periphery of the cylinder of a certain cylinder indicates a predetermined range from the surface of the cylinder.
  • the predetermined range is from the surface of the cylinder to r / 10.
  • r indicates the radius of the cylinder.
  • the cylinder periphery A13 of the cylinder Md13 indicates a range from the surface of the cylinder Md13 to r Md13 / 10.
  • r Md13 indicates the radius of the cylinder Md13.
  • the fitting processing unit 152 evaluates that the influence of noise is equal to or more than a certain number. When the number of counted points is less than a predetermined number, the fitting processing unit 152 treats the point cloud around the cylinder as noise.
  • the fitting processing unit 152 evaluates that the influence of noise is less than a certain value.
  • the fitting processing unit 152 treats the point cloud around the cylinder as a point cloud derived from the subject 5.
  • the fitting processing unit 152 evaluates that the influence of noise is above a certain level, it excludes the point cloud included in the cylinder periphery A13.
  • the fitting processing unit 152 also evaluates the influence of noise on the portion other than the cylinder Md13, and excludes the point cloud around the cylinder when the influence of noise is equal to or higher than a certain level. On the other hand, when the influence of noise is less than a certain value, the fitting processing unit 152 does not exclude the point cloud from the periphery of the selected cylinder.
  • the fitting processing unit 152 updates the three-dimensional point cloud data that is the target of the fitting processing by executing the above processing.
  • the fitting processing unit 152 performs processing for specifying the initial value, processing for calculating parameters, and processing for evaluating the influence of noise on the updated 3D point cloud data, and the influence of noise is less than a certain value for all parts. Repeat until.
  • FIG. 8 is a diagram for explaining an example of processing of the fitting processing unit.
  • a cylinder of the site identification information “Md9, Md13” will be used for explanation.
  • the fitting processing unit 152 evaluates that the influence of noise on the cylinder Md9 is equal to or higher than a certain level, and excludes the point cloud around the cylinder of the cylinder Md9.
  • the fitting processing unit 152 evaluates that the influence of noise on the cylinder Md13 is equal to or higher than a certain level, and excludes the point cloud around the cylinder of the cylinder Md13.
  • the point cloud around the cylinders of the cylinders Md9 and Md13 is excluded from the three-dimensional point cloud data (step S10).
  • step S10 the fitting processing unit 152 executes a process of specifying an initial value and calculating a parameter for the three-dimensional point cloud data in which the point cloud around the cylinders of the cylinders Md9 and Md13 is excluded, and the fitting process is performed.
  • the resulting cylindrical model data 40 is obtained.
  • the fitting processing unit 152 evaluates that the influence of noise on the cylinder Md9 is equal to or higher than a certain level, and excludes the point cloud around the cylinder of the cylinder Md9.
  • the fitting processing unit 152 evaluates that the influence of noise on the cylinder Md13 is equal to or higher than a certain level, and excludes the point cloud around the cylinder of the cylinder Md13.
  • the point cloud around the cylinders of the cylinders Md9 and Md13 is excluded from the three-dimensional point cloud data (step S11).
  • step S11 the fitting processing unit 152 executes a process of specifying an initial value and calculating a parameter for the three-dimensional point cloud data in which the point cloud around the cylinders of the cylinders Md9 and Md13 is excluded, and the fitting process is performed.
  • the resulting cylindrical model data 40 is obtained.
  • the fitting processing unit 152 evaluates that the influence of noise on the cylinder Md9 is less than a certain level.
  • the fitting processing unit 152 evaluates that the influence of noise on the cylinder Md13 is less than a certain level.
  • the final parameters of the cylindrical model data 40 are specified (step S12).
  • the fitting processing unit 152 specifies the final parameter of the cylindrical model data 40 by executing the above processing until the influence of noise on the fitting processing result previously executed becomes less than a certain level.
  • the fitting processing unit 152 outputs the final parameter to the generation unit 153.
  • the generation unit 153 is a processing unit that generates skeleton model data based on the final parameters of the cylindrical model data 40.
  • the generation unit 153 specifies the posture of the final cylindrical model data 40 by setting the final parameters acquired from the fitting processing unit 152 in the cylindrical model data 40.
  • the generation unit 153 identifies the skeleton model data in which the axes of each cylinder of the cylinder model data 40 are connected.
  • the skeleton model data specified by the generation unit 153 corresponds to the skeleton model data 6 described with reference to FIG.
  • the skeletal model data includes data on the coordinates of each joint.
  • the generation unit 153 stores the skeleton model data in the skeleton model table 142.
  • the generation unit 153 repeatedly executes the above processing every time the final parameter is acquired from the fitting processing unit 152.
  • the technique recognition unit 154 acquires each skeleton model data of the skeleton model table 142, and identifies the time-series change of each joint coordinate based on the continuous skeleton model data. The technique recognition unit 154 compares the time-series change of each joint position with the technique recognition table (not shown) to specify the type of technique. In addition, the technique recognition unit 154 compares the combination of technique types with the technique recognition table to calculate the performance score of the subject 5.
  • the technique recognition table is a table that associates the time-series changes of each joint position included in each skeleton recognition result with the type of technique.
  • the technique recognition table associates the combination of technique types with the score.
  • the score is calculated as the sum of the D (Difficulty) score and the E (Execution) score.
  • the D score is a score calculated based on the success or failure of the technique.
  • the E score is a score calculated by the deduction method according to the degree of completion of the technique.
  • the technique recognition unit 154 outputs the type of technique included in the performance, the score of the performance, the skeleton model data from the start to the end of the performance, and the like to the screen information output control unit 155.
  • the screen information output control unit 155 generates screen information based on the performance score and the skeleton model data from the start to the end of the performance.
  • the screen information output control unit 155 outputs the generated screen information to the display unit 130 and displays it.
  • FIG. 9 is a diagram showing an example of screen information.
  • the screen information 60 includes the areas 60a, 60b, and 60c.
  • the area 60a is an area for displaying the type of the technique recognized in the performance performed by the subject 5. In addition to the type of technique, the difficulty of the technique may also be displayed.
  • the area 60b is an area for displaying the performance score.
  • the area 60a is an area for displaying an animation of a three-dimensional model based on the skeleton model data from the start to the end of the performance.
  • the user operates the input unit 120 to instruct playback, stop, and the like of the animation.
  • FIG. 10 is a flowchart showing a processing procedure of the information processing apparatus according to the present embodiment.
  • the acquisition unit 151 of the information processing apparatus 100 acquires the distance image data from the sensor 10 and stores it in the distance image table 141 (step S101).
  • the acquisition unit 151 converts the distance image data into three-dimensional point cloud data (step S102).
  • the fitting processing unit 152 of the information processing device 100 executes noise evaluation processing (step S103).
  • the generation unit 153 of the information processing apparatus 100 generates skeleton model data based on the result of the fitting process and stores it in the skeleton model table 142 (step S104).
  • step S105 If the performance is not completed (steps S105, No), the information processing device 100 shifts to step S101 again. On the other hand, when the performance is completed (steps S105, Yes), the information processing apparatus 100 shifts to step S106.
  • the technique recognition unit 154 of the information processing device 100 refers to the skeleton model table 142, executes technique recognition, and specifies the type of technique and the score of the performance (step S106).
  • the screen information output control unit 155 of the information processing device 100 generates screen information (step S107).
  • the screen information output control unit 155 causes the display unit 130 to display the screen information (step S108).
  • the technique recognition is performed when the performance is completed, but the technique is not limited to this.
  • the information processing apparatus 100 sequentially performs skeleton recognition based on the output from the sensor. Then, the information processing apparatus 100 may sequentially perform the technique recognition by using the skeleton recognition result of a set unit among the skeleton recognition results executed sequentially. Further, independently of the technique recognition, the information processing apparatus 100 may generate an animation of a three-dimensional model based on the corresponding skeleton model data and display it on the screen each time the skeleton recognition result is acquired. ..
  • FIG. 11 is a flowchart showing a processing procedure of noise evaluation processing.
  • the fitting processing unit 152 of the information processing apparatus 100 executes the fitting process (step S201). Details of the fitting process will be described later.
  • the fitting processing unit 152 sets i to 1 (step S202).
  • the fitting processing unit 152 counts the point cloud around the cylinder of the part Mdi (step S203).
  • the fitting processing unit 152 shifts to step S206.
  • the predetermined number is a value set for each cylinder.
  • step S204 When the number of point groups is less than a predetermined number (step S204, Yes), the fitting processing unit 152 shifts to step S205.
  • the fitting processing unit 152 excludes the peripheral point cloud of the site Mdi (step S205).
  • step S208 the fitting processing unit 152 shifts to step S208 (step S206, Yes). In this embodiment, an example in which the number of sites is 14 is shown. If the value of i is not 14, the fitting processing unit 152 proceeds to step S207. The fitting processing unit 152 adds 1 to the value of i (step S207), and proceeds to step S203.
  • the fitting processing unit 152 determines whether or not there is a portion where the number of point groups around the cylinder is less than a predetermined number (step S208). If the portion exists (step S209, Yes), the fitting processing unit 152 shifts to step S201. On the other hand, when the portion does not exist (step S209, No), the fitting processing unit 152 shifts to step S210. In this way, the fitting processing unit 152 repeatedly executes the fitting process while updating the three-dimensional point cloud data to be processed until the influence of noise becomes less than a certain value for all the parts.
  • the fitting processing unit 152 when the number of point clouds around the cylinder corresponding to a certain part is less than a predetermined number in the fitting result performed earlier, the fitting performed earlier is affected by noise. Judge. That is, since the fitting performed earlier is affected by noise for a certain amount or more, it is determined that the fitting needs to be performed again. Then, the fitting processing unit 152 executes the fitting process again on the updated three-dimensional point cloud data in which some point clouds have been deleted by the previous noise evaluation process. As described above, the fitting processing unit 152 determines that the influence of noise on the previous fitting processing result is less than a certain value when there is no portion where the point cloud around the cylinder of each portion is less than a predetermined number.
  • the fitting processing unit 152 outputs the result of the fitting processing to the generation unit 153 (step S210).
  • FIG. 12 is a flowchart showing a processing procedure of the fitting process.
  • the fitting processing unit 152 of the information processing apparatus 100 acquires the three-dimensional point cloud data (step S301).
  • the fitting processing unit 152 specifies the initial value of the cylindrical model data 40 (step S302).
  • the fitting processing unit 152 calculates the prior distribution p nm (step S303).
  • the fitting processing unit 152 calculates the parameter change amount ⁇ (step S304).
  • the fitting processing unit 152 calculates the likelihood using the evaluation function Q (step S306). If the fitting processing unit 152 does not satisfy the convergence condition (step S307, No), the fitting processing unit 152 proceeds to step S303. On the other hand, when the convergence condition is satisfied (step S307, Yes), the fitting processing unit 152 shifts to step S308.
  • the fitting processing unit 152 determines whether or not the fitting is successful (step S308). For example, the fitting processing unit 152 determines that the fitting is successful if the sum of the prior distribution p nm is equal to or greater than a preset threshold value.
  • step S308 If the fitting processing unit 152 has not succeeded in fitting (steps S308, No), the fitting processing unit 152 proceeds to step S302. On the other hand, if the fitting is successful (steps S308, Yes), the fitting processing unit 152 proceeds to step S309. The fitting processing unit 152 outputs the fitting result (step S309).
  • the information processing apparatus 100 repeatedly executes the fitting process by excluding the point cloud around the cylinder affected by the noise until the influence of the noise becomes less than a certain value, and specifies the posture of the final cylinder model data 40. .. Then, the information processing device specifies the skeleton model data 6a based on the posture of the final cylindrical model data 40. This makes it possible to prevent a decrease in accuracy of the skeleton model data due to incorrect fitting due to noise.
  • noise appeared at a constant rate in the opposite direction of the sensor, and the error due to noise was about 10 cm.
  • the information processing apparatus 100 executes the above processing, noise is removed and the error due to noise is reduced to about 1 cm.
  • the information processing device 100 counts the number of points included in the point group around the cylinder, and repeatedly executes the process of reapplying the cylinder model data 40 until the number of points becomes a predetermined number or more. Thereby, the point cloud derived from the noise included in the three-dimensional point cloud data can be excluded. Further, by applying the cylindrical model data to the three-dimensional point cloud data excluding the point cloud derived from the noise, it is possible to suppress that a part of the cylindrical model data 40 is assigned to the noise.
  • the processing of the information processing device 100 described above is an example, and the information processing device 100 may execute other processing.
  • the fitting processing unit 152 of the information processing apparatus 100 fixes the range around the cylinder to "the range from the surface of the cylinder to r / 10", and the number of point clouds included around the cylinder (points included in the point cloud). The number of points) was counted, and when the influence of noise was above a certain level, the point cloud included around the cylinder was excluded, but the present invention is not limited to this.
  • the fitting processing unit 152 may gradually increase the range around the cylinder.
  • the range around the cylinder is set to r / 10 from the surface of the cylinder.
  • the fitting processing unit 152 sets the range around the cylinder to r / 9 from the surface of the cylinder.
  • the fitting processing unit 152 sets the range around the cylinder to r / 8 from the surface of the cylinder.
  • the fitting processing unit 152 can gradually increase the number of point clouds to be excluded from the three-dimensional point cloud data by gradually increasing the range around the cylinder.
  • FIG. 13 is a diagram showing an example of a hardware configuration of a computer that realizes a function similar to that of an information processing device.
  • the computer 200 includes a CPU 201 that executes various arithmetic processes, an input device 202 that receives data input from a user, and a display 203. Further, the computer 200 has a communication device 204 for receiving the measurement result from the sensor, and an interface device 205 for connecting to various devices. The computer 200 has a RAM 206 for temporarily storing various information and a hard disk device 207. Then, each device 201 to 207 is connected to the bus 208.
  • the hard disk device 207 has an acquisition program 207a, a fitting processing program 207b, a generation program 207c, a technique recognition program 207d, and a screen information output control program 207e.
  • the CPU 201 reads the acquisition program 207a, the fitting processing program 207b, the generation program 207c, the technique recognition program 207d, and the screen information output control program 207e and deploys them in the RAM 206.
  • the acquisition program 207a functions as the acquisition process 206a.
  • the fitting processing program 207b functions as the fitting processing process 206b.
  • the generation program 207c functions as the generation process 206c.
  • the technique recognition program 207d functions as a technique recognition process 206d.
  • the screen information output control program 207e functions as the screen information output control process 206e.
  • the processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151.
  • the processing of the fitting processing process 206b corresponds to the processing of the fitting processing unit 152.
  • the processing of the generation process corresponds to the processing of the generation unit 153.
  • the process of the technique recognition process 206a corresponds to the process of the technique recognition unit 154.
  • the processing of the screen information output control process 206e corresponds to the processing of the screen information output control unit 155.
  • each program does not necessarily have to be stored in the hard disk device 207 from the beginning.
  • each program is stored in a "portable physical medium" such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into the computer 200.
  • the computer 200 may read and execute each of the programs 207a to 207e.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
PCT/JP2019/048515 2019-12-11 2019-12-11 生成方法、生成プログラム及び情報処理システム Ceased WO2021117165A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
EP19955580.6A EP4075377A4 (en) 2019-12-11 2019-12-11 GENERATION METHOD, GENERATION PROGRAM AND INFORMATION PROCESSING SYSTEM
PCT/JP2019/048515 WO2021117165A1 (ja) 2019-12-11 2019-12-11 生成方法、生成プログラム及び情報処理システム
CN201980102506.4A CN114730497A (zh) 2019-12-11 2019-12-11 生成方法、生成程序以及信息处理系统
JP2021563516A JP7375829B2 (ja) 2019-12-11 2019-12-11 生成方法、生成プログラム及び情報処理システム
US17/741,824 US20220270408A1 (en) 2019-12-11 2022-05-11 Generation method, non-transitory computer-readable storage medium for storing generation program, and information processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/048515 WO2021117165A1 (ja) 2019-12-11 2019-12-11 生成方法、生成プログラム及び情報処理システム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/741,824 Continuation US20220270408A1 (en) 2019-12-11 2022-05-11 Generation method, non-transitory computer-readable storage medium for storing generation program, and information processing system

Publications (1)

Publication Number Publication Date
WO2021117165A1 true WO2021117165A1 (ja) 2021-06-17

Family

ID=76330012

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/048515 Ceased WO2021117165A1 (ja) 2019-12-11 2019-12-11 生成方法、生成プログラム及び情報処理システム

Country Status (5)

Country Link
US (1) US20220270408A1 (https=)
EP (1) EP4075377A4 (https=)
JP (1) JP7375829B2 (https=)
CN (1) CN114730497A (https=)
WO (1) WO2021117165A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023030622A (ja) * 2021-08-23 2023-03-08 株式会社日立産機システム 地図データ作成装置、地図データ評価装置及び地図データ作成方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121392992A (zh) * 2025-12-24 2026-01-23 深圳初量科技有限公司 基于nfc射频及体感交互的高空探险打卡计分系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015102913A (ja) * 2013-11-21 2015-06-04 キヤノン株式会社 姿勢推定装置及び姿勢推定方法
WO2018207292A1 (ja) 2017-05-10 2018-11-15 富士通株式会社 対象物認識方法、装置、システム、プログラム
WO2019030794A1 (ja) * 2017-08-07 2019-02-14 富士通株式会社 情報処理装置、モデルデータ作成プログラム、モデルデータ作成方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4930854B2 (ja) * 2005-07-19 2012-05-16 日本電気株式会社 関節物体位置姿勢推定装置及びその方法ならびにプログラム
CA2814839C (en) * 2010-10-25 2018-12-04 Christian H. Debrunner Detecting structural changes to underwater structures
US9235928B2 (en) * 2012-01-24 2016-01-12 University Of Southern California 3D body modeling, from a single or multiple 3D cameras, in the presence of motion
US10033979B2 (en) * 2012-03-23 2018-07-24 Avigilon Fortress Corporation Video surveillance systems, devices and methods with improved 3D human pose and shape modeling
CN109147038B (zh) * 2018-08-21 2023-02-07 北京工业大学 基于三维点云处理的管道三维建模方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015102913A (ja) * 2013-11-21 2015-06-04 キヤノン株式会社 姿勢推定装置及び姿勢推定方法
WO2018207292A1 (ja) 2017-05-10 2018-11-15 富士通株式会社 対象物認識方法、装置、システム、プログラム
WO2019030794A1 (ja) * 2017-08-07 2019-02-14 富士通株式会社 情報処理装置、モデルデータ作成プログラム、モデルデータ作成方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
M. YER. YANG: "Real-time Simultaneous Pose and Shape Estimation for Articulated Objects Using a Single Depth Camera", CVPR, 2014

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023030622A (ja) * 2021-08-23 2023-03-08 株式会社日立産機システム 地図データ作成装置、地図データ評価装置及び地図データ作成方法
JP7614052B2 (ja) 2021-08-23 2025-01-15 株式会社日立産機システム 地図データ作成装置、地図データ評価装置及び地図データ作成方法

Also Published As

Publication number Publication date
US20220270408A1 (en) 2022-08-25
CN114730497A (zh) 2022-07-08
EP4075377A4 (en) 2022-12-28
JP7375829B2 (ja) 2023-11-08
JPWO2021117165A1 (https=) 2021-06-17
EP4075377A1 (en) 2022-10-19

Similar Documents

Publication Publication Date Title
CN105229666B (zh) 3d图像中的运动分析
JP6733738B2 (ja) 運動認識装置、運動認識プログラムおよび運動認識方法
CN105228709B (zh) 用于重复检测和分析的信号分析
CN105209136B (zh) 用于分析3d图像中的用户动作的质心状态矢量
US20200193866A1 (en) Non-transitory computer readable recording medium, practice assist method, and practice assist system
US11776134B2 (en) Non-transitory computer readable recording medium, element recognition method, and element recognition system
JP7327494B2 (ja) 補正方法、補正プログラムおよび情報処理システム
JP2020031406A (ja) 判定システム、及び判定方法
JP2020030190A (ja) 位置追跡システム、及び位置追跡方法
US20220270408A1 (en) Generation method, non-transitory computer-readable storage medium for storing generation program, and information processing system
JP7272470B2 (ja) 運動認識方法、運動認識プログラムおよび情報処理装置
JP7409390B2 (ja) 運動認識方法、運動認識プログラムおよび情報処理装置
JP2023179239A (ja) 情報処理プログラム、情報処理方法および情報処理装置
US20220222974A1 (en) Evaluation method, non-transitory computer readable recording medium, and information processing apparatus
JP7054950B1 (ja) 位置計測システム、及び位置計測方法
JP2020030189A (ja) 位置計測システム、及び位置計測方法
US20210187374A1 (en) Augmented extended realm system
CN111353345B (zh) 提供训练反馈的方法、装置、系统、电子设备、存储介质
JP2022138007A (ja) 骨格認識方法、骨格認識プログラムおよび体操採点支援システム
JP7188612B2 (ja) 生成方法、生成プログラムおよび情報処理装置
JP7331927B2 (ja) 生成方法、生成プログラムおよび情報処理装置
CN113705536B (zh) 连续动作打分方法、装置及存储介质
CN118823274A (zh) 基于ar设备的健身信息显示方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19955580

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021563516

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019955580

Country of ref document: EP

Effective date: 20220711