WO2023100246A1

WO2023100246A1 - Technique recognition program, technique recognition method, and information processing device

Info

Publication number: WO2023100246A1
Application number: PCT/JP2021/043871
Authority: WO
Inventors: 卓也佐藤
Original assignee: 富士通株式会社
Priority date: 2021-11-30
Filing date: 2021-11-30
Publication date: 2023-06-08

Abstract

This information processing device acquires sensing data obtained by sensing competitors. The information processing device generates time-series 3D skeletal information of a competitor on the basis of the sensing data. When identifying the motion performed by the competitor on the basis of the skeletal information, the computer identifies the competitor's grip on the bar at the end of a second motion, on the basis of a combination of the competitor's grip on the bar at the end of a first motion, and the second motion following the first motion.

Description

Technique recognition program, technique recognition method, and information processing device

The present invention relates to a technique recognition program and the like.

In the past, multiple referees visually scored gymnastics competitions, but as techniques become more sophisticated with more complex movements, it is sometimes difficult to visually recognize techniques.

On the other hand, there is a conventional technology that uses a learning model such as DL (Deep Learning) to score performances performed by competitors. For example, in the conventional technology, sensors, cameras, and the like are used to obtain sensing data of an athlete, input the sensing data into a trained learning model, and calculate skeletal information of the athlete. Then, in the conventional technology, a feature amount indicating the characteristics of the posture corresponding to the "skill" is calculated from the time-series data of the skeleton information, and the technique performed by the athlete is calculated based on the time-series data of the skeleton information and the feature amount. automatically recognizes and outputs the performance score.

The performance score is calculated as the sum of the D (Difficulty) score and the E (Execution) score. For example, the D score is a score calculated based on whether or not a technique is established. The E-score is a score calculated by a deduction method according to the degree of perfection of the technique.

Japanese Patent Application Laid-Open No. 2020-038440 Japanese Patent Publication No. 2012-518856 WO2020/121500 U.S. Patent Application Publication No. 2020/0074247

In gymnastics, even the same movement can be scored differently depending on how you grip the stick. The difference in grip means the difference in how the arm is twisted, but with the conventional technology, it is difficult to accurately recognize even the twist of the arm from the player's sensing data, and there are cases where the technique cannot be accurately recognized.

It should be noted that even in the case of machine learning a learning model that can output differences in grip, there are many frames in which features for identifying differences in grip cannot be obtained for the reasons described above, and machine learning alone is not sufficient for accurate recognition. is difficult.

In one aspect, it is an object of the present invention to provide a technique recognition program, a technique recognition method, and an information processing apparatus that can improve the accuracy of technique recognition by distinguishing between different ways of gripping a stick.

In the first plan, the computer executes the following processing. The computer obtains sensing data obtained by sensing the athlete. Based on the sensing data, the computer generates time-series three-dimensional skeletal information of the athlete. When specifying the exercise performed by the athlete based on the skeletal information, the computer determines the combination of the athlete's grip on the stick at the end of the first exercise and the second exercise following the first exercise. Based on this, the player's grip on the bar at the end of the second movement is determined.

It is possible to improve the accuracy of skill recognition by distinguishing between different ways of gripping the stick.

FIG. 1 is a diagram showing an example of a technique recognition system according to this embodiment. FIG. 2 is a diagram (1) for explaining the effect of the information processing apparatus according to the embodiment. FIG. 3 is a diagram (2) for explaining the effect of the information processing apparatus according to the embodiment. FIG. 4 is a functional block diagram showing the configuration of the information processing apparatus according to this embodiment. FIG. 5 is a diagram showing an example of the data structure of joint definition data according to this embodiment. FIG. 6 is a diagram showing an example of the data structure of the joint position DB according to this embodiment. FIG. 7 is a diagram showing an example of the data structure of a skeleton information DB. FIG. 8 is a diagram showing an example of the data structure of motion dictionary data. FIG. 9 is a diagram showing an example of the data structure of a grip determination table. FIG. 10 is a diagram showing an example of the data structure of a grip determination result table. FIG. 11 is a diagram showing an example of the data structure of technique dictionary data. FIG. 12 is a diagram showing an example of the data structure of the skill determination result table. FIG. 13 is a diagram (1) for explaining the processing of the correction unit; FIG. 14 is a diagram (2) for explaining the processing of the correction unit; FIG. 15 is a flow chart showing the processing procedure of the information processing apparatus according to this embodiment. FIG. 16 is a flow chart showing the procedure of grip determination processing. FIG. 17 is a flow chart showing the processing procedure of grip correction processing. FIG. 18 is a diagram for supplementary explanation of the processing of the specifying unit. FIG. 19 is a diagram illustrating an example of a hardware configuration of a computer that implements functions similar to those of an information processing apparatus.

Hereinafter, embodiments of the technique recognition program, technique recognition method, and information processing apparatus disclosed in the present application will be described in detail based on the drawings. In addition, this invention is not limited by this Example.

FIG. 1 is a diagram showing an example of a technique recognition system according to this embodiment. As shown in FIG. 1, this technique recognition system has 3D (3-dimension) laser sensors 10a and 10b and an information processing device 100. As shown in FIG. The information processing device 100 and the 3D laser sensors 10a and 10b are interconnected.

The 3D laser sensors 10a and 10b are sensors that perform 3D sensing on the player 5. The 3D laser sensors 10 a and 10 b output distance image data as sensing results to the information processing device 100 . In the following description, the 3D laser sensors 10a and 10b are collectively referred to as "3D laser sensor 10".

For example, the distance image data includes a plurality of distance image frames, and each distance image frame is assigned a frame number that uniquely identifies the frame in ascending order. One distance image frame includes distance information to each point on the athlete 5 sensed by the 3D laser sensor 10 at a certain timing.

The contestant 5 performs a predetermined performance to be scored in front of the 3D laser sensor 10 . In this embodiment, as an example, a case where the player 5 performs a horizontal bar performance will be described, but the same can be applied to other scoring competitions in which techniques differ depending on how the player holds the bar.

For example, in a horizontal bar performance, the way to hold the bar includes the right hand, the reverse hand, and the overhand hand. Shuntetsu is a normal grip with the back of the hand up. The reverse hand is a grip that is twisted outward by 180° from the normal hand. Ogyakute is a grip that is twisted 180 degrees inward from the forward hand.

Based on the distance image data acquired from the 3D laser sensor 10, the information processing device 100 generates time-series skeletal information of the athlete 5. Based on the combination of the grip style of the athlete 5 at the end of the current exercise and the type of the current exercise, the grip style of the athlete 5 at the end of the current exercise is specified.

The information processing device 100 identifies the current exercise (skill) based on the combination of the type of the current exercise and the grip style at the end of the current exercise. For example, information processing apparatus 100 identifies the technique name as "front wheel" when the current exercise is "front wheel" and the grip style for the current exercise is "right hand" or "reverse hand". On the other hand, the information processing apparatus 100 specifies the name of the technique as "large reverse wheel" when the current exercise is "forward system wheel" and the grip style of the current exercise is "high reverse wheel."

Here, the effect of the information processing device 100 compared with the conventional technology will be described. 2 and 3 are diagrams for explaining the effect of the information processing apparatus according to this embodiment. FIG. 2 will be described. In technique T1, competitor 5 performs exercises in order of T1-1, T1-2, and T1-3. Technique T1 is a C-difficulty technique called "Endo 1 Twist Kata Dai Gakute". On the other hand, in technique T2, the competitor 5 performs exercises in order of T2-1, T2-2, and T2-3. Technique T2 is a D-difficulty technique called "End-One Twisting Ogyakute". Comparing the technique T1 and the technique T2, the grip A1 in T1-3 is different from the grip A2 in T2-3.

In the conventional technology, since it is not possible to recognize the grip A1 and the grip A2 explained in FIG. On the other hand, since the information processing apparatus 100 can recognize the grip A1 and the grip A2, it recognizes the "end-single-twist-single-upside-down hand" and the "end-single-twist-single-upside-down hand" respectively. can be accurately identified.

Fig. 3 will be explained. Technique T3 is an A-difficulty technique called "front wheel". Technique T4 is an A-difficulty technique called "Ogyakute Wheel". Technique T3 is an A-difficulty technique called "front wheel". Technique T4 is a B-difficulty technique called "Ogyakute Wheel". In FIG. 3, for the sake of convenience, the exercise movements of the technique T3 are summarized in one picture, and the exercise movements of the technique T4 are arranged in chronological order. The difference between technique T3 and technique T4 is the way the stick is gripped.

With the conventional technology, it is not possible to recognize the difference in how the stick is gripped as described in FIG. On the other hand, since the information processing apparatus 100 can recognize the difference in gripping the stick, it can correctly identify the "front wheel" and the "large reverse wheel".

Next, a configuration example of the information processing apparatus 100 will be described. FIG. 4 is a functional block diagram showing the configuration of the information processing apparatus according to this embodiment. As shown in FIG. 4 , the information processing apparatus 100 has a communication section 110 , an input section 120 , a display section 130 , a storage section 140 and a control section 150 .

The communication unit 110 is connected to the 3D laser sensor 10. The communication unit 110 acquires distance image data from the 3D laser sensor 10 and outputs the acquired distance image data to the control unit 150 . The communication unit 110 may be connected to a camera (not shown) that captures an image of the competitor 5 and acquire image data from this camera.

The input unit 120 is an input device for inputting various types of information to the information processing device 100 . The input unit 120 corresponds to a keyboard, mouse, touch panel, and the like.

The display unit 130 is a display device that displays information on the display screen that is output from the control unit 150 . The display unit 130 corresponds to a liquid crystal display, a touch panel, or the like.

The storage unit 140 includes a distance image DB (Data Base) 141, joint definition data 142, joint position DB 143, skeleton information DB 144, motion dictionary data 145, a grip determination table 146, a grip determination result table 147, technique dictionary data 148, and technique determination. It has a result table 149 . The storage unit 140 corresponds to semiconductor memory devices such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, and storage devices such as HDD (Hard Disk Drive).

The distance image DB 141 is a DB that stores distance image data acquired from the 3D laser sensor 10 . For example, the distance image DB 141 associates frame numbers with distance image frames. The frame number is a number that uniquely identifies each distance image frame, and is assigned in ascending order. A distance image frame is a frame included in distance image data sensed by the 3D laser sensor 10 .

The joint definition data 142 defines each joint position of the athlete (athlete 5). FIG. 5 is a diagram showing an example of the data structure of joint definition data according to this embodiment. As shown in FIG. 5, this joint definition data 142 stores information in which each joint identified by a known skeleton model is numbered. For example, as shown in FIG. 5, the right shoulder joint (SHOULDER_RIGHT) is assigned number 7, the left elbow joint (ELBOW_LEFT) is assigned number 5, and the left knee joint (KNEE_LEFT) is assigned number 11. , and the right hip joint (HIP_RIGHT) is given number 14. Here, in this embodiment, the X coordinate of the right elbow joint No. 8 may be described as X8, the Y coordinate as Y8, and the Z coordinate as Z8.

The joint position DB 143 is a DB that stores the position data of each joint of the player 5 generated based on the distance image data of the 3D laser sensor 10. FIG. 6 is a diagram showing an example of the data structure of the joint position DB according to this embodiment. As shown in FIG. 6, this joint position DB 143 associates a frame number with three-dimensional coordinates "X0, Y0, Z0, . . . , X17, Y17, Z17" of each joint.

Fig. 6 shows changes in the time series of each joint in the range image data. For example, frame number "1" indicates that the joint positions are "X0=100, Y0=20, Z0=0, . . . , X17=200, Y17=40, Z17=5". Frame number “2” indicates that the positions of the joints have moved to “X0=101, Y0=25, Z0=5, . . . , X17=202, Y17=39, Z17=15”.

The skeleton information DB 144 is a DB that stores the athlete's skeleton information generated based on the distance image data. FIG. 7 is a diagram showing an example of the data structure of a skeleton information DB. As shown in FIG. 7, the skeleton information DB 144 associates frame numbers with skeleton information. The explanation about the frame number is the same as the explanation given for the distance image DB 141 . The skeletal information is data indicating the skeletal structure of the player 5 estimated by connecting joint positions.

The exercise dictionary data 145 is dictionary data that defines exercise. The exercise may correspond to, for example, a technique narrowed down based on a feature amount other than how to grip. In that case, the technique corresponding to exercise includes a plurality of technique candidates. For example, the exercise ``front wheel'' includes techniques ``front wheel'', ``big reverse wheel'', and the like. By specifying the way of gripping in the process described later, it is specified whether the motion "forward system wheel" is "front wheel", "big reverse wheel", or other tricks.

FIG. 8 is a diagram showing an example of the data structure of exercise dictionary data. As shown in FIG. 8, the motion dictionary data 145 associates each motion with a plurality of basic motions. A basic motion and a feature amount are set for each basic motion. The feature amount is a value indicating the feature of the movement of the player 5 calculated from the time-series skeleton information. For example, the feature amount is time-series changes in joint positions. A movement is specified by a combination of basic techniques. Exercises are not limited to the above examples, and for example, a specific action may be defined as an exercise regardless of the technique defined by the scoring rules, and a grip may be recognized for each exercise.

The grip determination table 146 is a table for specifying a module corresponding to the combination of the grip at the end of the previous exercise and the current exercise. The specified module specifies the grip corresponding to the current exercise.

FIG. 9 is a diagram showing an example of the data structure of the grip determination table. As shown in FIG. 9, the grip determination table 146 associates conditions, modules, and priorities. The conditions include the previous grip and exercise. Here, the previous grip indicates the grip at the end of the previous exercise. If the previous gripping method is "any", it indicates that any gripping method is acceptable. The module defines a determination policy for each grip style. For example, if the grip at the end of the previous exercise was "any" and the current exercise was "Adler", the module would be "first module". Although FIG. 9 shows the first to fourth modules, the present invention is not limited to this.

　Priority indicates the priority of the module, and the smaller the number, the higher the priority. Modules with higher priority are more likely to output values than modules with lower priority.

The first module outputs the preset likelihoods of the left and right hand grips. For example, the hand grip after Adler is always a large reverse hand due to the characteristics of the movement. In this case, the first module outputs (0, 0, 1) as the likelihood of the left hand grip in the order of reverse hand, forward hand, and major reverse hand. The first module outputs (0, 0, 1) in the order of reverse hand, forward hand, and large reverse hand as the likelihood of the right hand grip.

The second module (holding change judgment) will be explained. The second module determines whether or not there has been a grip change based on the movement of the wrist and arms estimated from the feature quantity corresponding to the current exercise and the balance of the whole body. When the second module determines that there is a grip change from a large reverse hand to a reverse hand, the likelihood of the grip is set to (0.7, 0.1, 0.2) in the order of reverse hand, straight hand, and large reverse hand. to output Since the previous grip was "(both hands) large reverse hand", the likelihood of the reverse hand is set high.

On the other hand, when the second module determines that there was no change of grip while still in the great reverse hand, the likelihood of the grip is set to (0.2, 0.1, 0.7) in the order of the reverse hand, the straight hand, and the great reverse hand. to output Since the previous grip was "(both hands) Ogyakute", the likelihood of Ogyakute is set high.

The third module (previous likelihood inheritance) is a module that outputs the same likelihood as the previous motion likelihood. For example, if (0.7, 0.1, 0.2) is output by the previous exercise module, the likelihood for this exercise is (0.7, 0.1, 0.2 ).

The fourth module (inverted twist determination) will be explained. In the case of a single twist, the way of grasping the handle is determined by the motion and constraints of the human body structure, so the likelihood of determination is output. For example, the fourth module outputs (0, 0, 1) as the likelihood of the left hand grip in the order of reverse hand, forward hand, and major reverse hand.

In addition, the fourth module evaluates the likelihood of the hand that is not the axis, based on characteristics such as the trajectory of the arm, which side the stick is gripped from, and whether the grip is made by rubbing the stick after gripping. Guess which way you grabbed it and output the likelihood. The DL may be used to learn the relationship between motion features and the likelihood of the non-axis hand and used as a fourth module.

Return to the description of Fig. 4. The grip determination result table 147 is a table that holds determination results of how to grip. FIG. 10 is a diagram showing an example of the data structure of a grip determination result table. As shown in FIG. 10, this grip determination result table 147 has order, motion, left hand likelihood, right hand likelihood, left hand grip, and right hand grip.

The order indicates the order of the exercises performed. Exercise, as described above, is a technique that represents a plurality of technique candidates. The left-hand likelihood includes each likelihood of a reverse hand, a forward hand, and a major reverse hand regarding how to hold the left hand. The right hand likelihood includes the likelihoods of each of reverse hand, forward hand, and large reverse hand regarding how to hold the right hand. The left hand grip is a way of gripping the left hand identified from the likelihood. The right hand grip is a way of gripping the right hand identified from the likelihood.

In FIG. 10, in the order "kth", the likelihood of the left hand at the end of the exercise "Adler" is (0, 0, 1) in the order of the reverse move, the forward move, and the major reverse move, and the "great reverse move" having the maximum likelihood. is set to left hand grip. In the order "k-th", the likelihood of the right hand at the end of the exercise "Adler" is (0, 0, 1) in the order of the reverse move, the forward move, and the great reverse move. It is set in a grip.

In FIG. 10, the left hand grip and right hand grip of the grip determination result table 147 are the grips at the end of the exercise, but both the grips at the start and the grips at the end may be retained. .

Return to the description of Fig. 4. The technique dictionary data 148 is dictionary data that defines techniques corresponding to exercise and how to grip at the end of the exercise. FIG. 11 is a diagram showing an example of the data structure of technique dictionary data. As shown in FIG. 11, this technique dictionary data 148 associates exercises with left hand grips, right hand grips, and techniques. The explanations regarding exercise, left hand grip, and right hand grip are the same as those described with reference to FIG. 10 . The technique shown in FIG. 11 is the name of the technique that is finally specified.

In FIG. 11, the technique corresponding to the exercise "front wheel", left hand grip "reverse hand", and right hand hand grip "reverse hand" is "front wheel".

Return to the description of Fig. 4. The technique determination result table 149 is a table that holds the determination results of techniques performed by the player 5 . FIG. 12 is a diagram showing an example of the data structure of the skill determination result table. As shown in FIG. 12, the technique determination result table 149 associates the order of techniques with the techniques.

Return to the description of Fig. 4. The control unit 150 has an acquisition unit 151 , a generation unit 152 , a calculation unit 153 , a specification unit 154 , a correction unit 155 , a technique determination unit 156 and an evaluation unit 157 . The control unit 150 can be realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. The control unit 160 can also be realized by hardwired logic such as ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array).

The acquisition unit 151 acquires distance image data from the 3D laser sensor 10 via the communication unit 110, and registers the acquired distance image data (frame number, distance image frame) in the distance image DB 141.

The generation unit 152 generates skeleton information by performing processing for extracting the position data of each joint of the player 5 in time series and processing for generating skeleton information in time series.

An example of a process in which the generation unit 152 extracts the position data of each joint of the player 5 in time series will be described. The generation unit 152 compares the distance image frame of the distance image DB 141 and the joint definition data 142 to identify the type of each joint included in the distance image frame and the three-dimensional coordinates of the joint. The generation unit 152 registers information in which the frame number and the three-dimensional coordinates of each joint type are associated with each other in the joint position DB 143 . The generation unit 152 repeatedly executes the above process for each frame number.

An example of a process in which the generation unit 152 generates skeleton information in time series will be described. The generator 152 generates skeleton information corresponding to each frame number based on the joint position DB 143 . Generation unit 152 stores the generated skeleton information in skeleton information DB 144 in association with the frame number.

For example, the generation unit 152 generates skeleton information by connecting the three-dimensional coordinates of each joint stored in the joint position DB 143 based on the connection relationship defined in the joint definition data 142 .

The calculation unit 153 calculates feature amounts based on the time-series skeleton information stored in the skeleton information DB 144 . For example, the feature amount calculated by the calculator 153 is a time-series change in each joint position, and indicates the feature of the movement of the player 5 .

The calculation unit 153 calculates the feature quantity for each break in the movement at which the basic movement is switched. For example, the calculation unit 153 determines the timing at which the change in each joint position is less than the threshold for a certain period of time as a discontinuity. The calculation unit 153 may identify a break in exercise using the technology disclosed in “International Publication No. 2019/116496”. The calculation unit 153 outputs the feature amount for each break to the identification unit 154 . In the following description, the feature amount for each break is simply referred to as "feature amount".

The specifying unit 154 is a processing unit that specifies how the player 5 grips the stick. For example, the identifying unit 154 executes a process of identifying an exercise and a process of identifying a grip style.

The process of identifying exercise performed by the identifying unit 154 will be described. The specifying unit 154 compares each feature quantity with the feature quantity of the basic motion in the motion dictionary data 145 to specify the basic motion corresponding to each feature quantity. The identifying unit 154 compares the specified combinations of basic motions with the basic motions in the motion dictionary data 145 to specify motions corresponding to the basic motions. The specifying unit 154 repeats the above process to specify motions in order.

A description will be given of the process by which the identifying unit 154 identifies the way of gripping. Here, as an example, the k-1th exercise is referred to as "previous exercise", and the kth exercise is referred to as "current exercise". The identification unit 154 compares the combination of the gripping manner of the previous exercise (the gripping manner at the end) and the current exercise with the conditions of the grip determination table 146, and determines the module to be used for identifying the gripping manner. If this is the first time (when k=1), the specifying unit 154 may set the gripping manner of the previous exercise to a predetermined gripping manner (for example, smooth).

The identifying unit 154 uses the determined module to identify the likelihood of each gripping style, and identifies the gripping style with the highest likelihood as the gripping style corresponding to the current exercise. The identifying unit 154 registers information on the current exercise, left hand likelihood, right hand likelihood, left hand grip, and right hand grip in the grip determination result table 147 for the k-th. The specifying unit 154 registers the gripping manner at the start and the gripping manner at the end for left hand grip and right hand grip.

The specifying unit 154 sequentially registers each piece of information in the grip determination result table 147 by repeatedly executing the above process for the following k+1st, k+2nd, . . . , k+nth.

The correction unit 155 refers to the grip determination result table 147 registered by the processing of the identification unit 154 described above, and determines whether there is a contradiction in the human body structure in the state transition from the previous gripping method to the current gripping method. However, if there is a contradiction, the likelihood of each grip is calculated again.

13 and 14 are diagrams for explaining the processing of the correction unit. FIG. 13 will be described. For convenience of explanation, the grip determination result table 147A shown in FIG. (illustration omitted).

The correction unit 155 scans two consecutive records in the grip determination result table 147A and identifies records in which the grip at the end of the previous exercise is different from the grip at the start of the subsequent exercise. Correction unit 155 identifies the module that has identified the left-hand likelihood and right-hand likelihood set in the identified record, and selects the module with the higher priority value. The correcting unit 155 uses the module with the higher priority to re-specify the likelihood while changing the conditions.

For example, in the example shown in FIG. 13, the left hand grip "upside down hand" at the end of the "k+1th" record and the left hand grip "upside down" at the start of the "k+2th" record are different. Therefore, the correcting unit 155 identifies the “k+1st” and “k+2nd” records.

Based on the grip determination table 146, the correction unit 155 identifies the module "second module" used when identifying the "k+1" left hand likelihood and right hand likelihood. Based on the grip determination table 146, the correction unit 155 identifies the module “fourth module” used when identifying the “k+2” left hand likelihood and right hand likelihood.

The correction unit 155 selects the second module because the priority of the "second module" is "3" and the priority of the "fourth module" is "1".

Move to the description of FIG. The correction unit 155 changes the conditions for using the second module, and identifies the left-hand likelihood and the right-hand likelihood again. For example, it is assumed that each likelihood "0.2, 0.1, 0.7" was output because the condition of the previous second module was "there was no grip change". In this case, the correction unit 155 changes the condition to "there was a grip change" and specifies each likelihood as "0.7, 0.1, 0.2". Also, by this, the left hand grip and right hand grip are updated to "upside down". For example, by the processing of the correction unit 155, the grip determination result table 147B is updated to the grip determination result table 147C, and the contradiction is resolved.

Further, when the selected module (second module in the above example) is a module that outputs a plurality of output results, the correction unit 155 may select an output result that eliminates contradiction. For example, in FIG. 13, the way of gripping at the end of the previous time is "Ogyakute" and the way of gripping at the start of this time is "Inverse hand", which contradicts each other. In this case, among the output results of the selected modules, the output result that maximizes the likelihood of oganete is selected.

It should be noted that the correcting unit 155 skips the above processing when there is no contradiction in the grip determination result table 147 .

Return to the description of Fig. 4. The technique determination unit 156 identifies the technique performed by the player 5 based on the combination of the exercise and the grip method (left hand grip, right hand grip) registered in the grip determination result table 147 and the technique dictionary data 148. Then, the identified results are registered in the skill determination result table 149 in order.

The evaluation unit 157 grades the score of the competitor 5 based on the skill determination result table 149. For example, the evaluation unit 157 calculates the D score of the competitor 5 by totaling the points corresponding to the difficulty levels of the techniques (successful techniques) registered in the technique determination result table 149 . Further, the evaluation unit 157 may further evaluate the degree of perfection of the technique, etc., based on the skeleton information registered in the skeleton information DB 144, and further calculate the E-score based on the deduction method. The evaluation unit 157 outputs the evaluation result to the display unit 130 for display.

Next, an example of the processing procedure of the information processing apparatus 100 according to this embodiment will be described. FIG. 15 is a flow chart showing the processing procedure of the information processing apparatus according to this embodiment. As shown in FIG. 15, the acquisition unit 151 of the information processing device 100 acquires distance image data from the 3D laser sensor 10 and registers it in the distance image DB 141 (step S101).

The generation unit 152 of the information processing device 100 generates time-series skeleton information based on the distance image DB 141 and the joint definition data 142, and registers it in the skeleton information DB 144 (step S102). The calculation unit 153 of the information processing apparatus 100 calculates feature amounts based on the skeleton information registered in the skeleton information DB 144 (step S103).

The identifying unit 154 of the information processing device 100 identifies exercise based on the feature amount and the exercise dictionary data 145 (step S104). The identifying unit 154 executes grip determination processing (step S105). The correction unit 155 of the information processing device 100 executes grip correction processing (step S106).

The technique determination unit 156 of the information processing device 100 determines a technique based on the grip determination result table 147 and the technique dictionary data 148, and registers it in the technique determination result table 149 (step S107). The evaluation unit 157 of the information processing device 100 grades the score based on the skill determination result table 149 (step S108), and causes the display unit 130 to display the score (step S109).

Next, an example of the grip determination process shown in step S105 of FIG. 15 will be described. FIG. 16 is a flow chart showing the procedure of grip determination processing. As shown in FIG. 16, the specifying unit 154 of the information processing device 100 sets k to an initial value (k=1) (step S201).

Based on the grip determination table 146, the identifying unit 154 identifies the module corresponding to the k-th exercise and the previous grip method (step S202). The identifying unit 154 reads out the identified module (step S203).

The identifying unit 154 identifies the gripping style corresponding to the maximum likelihood among the likelihoods of the gripping styles output by the module as the gripping style of the k-th exercise, and registers it in the grip determination result table 147 (step S204).

If the k-th movement is the last movement (step S205, Yes), the identification unit 154 ends the grip determination process.

If the k-th motion is not the last exercise (step S205, No), the specifying unit 154 adds 1 to k (step S206), and proceeds to step S202.

Next, an example of the grip correction process shown in step S106 of FIG. 15 will be described. FIG. 17 is a flow chart showing the processing procedure of grip correction processing. As shown in FIG. 17, the correcting unit 155 of the information processing device 100 refers to the grip determination result table 147 and identifies a record in which the grip at the start and the grip at the end contradict each other (step S301).

The correction unit 155 extracts the module with the higher priority value from among the modules used to calculate the likelihood contained in the contradictory record (step S302).

The correction unit 155 sets k to an initial value (k=1) (step S303). The correction unit 155 selects the module with the k-th highest priority value from among the plurality of extracted modules (step S304).

The correction unit 155 selects, from among the results that can be output by the selected module, an output that eliminates the contradiction between the way of gripping at the start and the way of gripping at the end (step S305). The correction unit 155 updates the grip determination result table 147 (step S306).

If the k-th module is the last extracted module (step S307, Yes), the specifying unit 154 ends the grip correction process.

If the k-th module is not the last extracted module (step S307, No), the specifying unit 154 adds 1 to k (step S308), and proceeds to step S304.

Next, the effects of the information processing apparatus 100 according to this embodiment will be described. Based on the distance image data acquired from the 3D laser sensor 10, the information processing device 100 generates time-series skeletal information of the athlete 5. Based on the combination of the grip style of the athlete 5 at the end of the current exercise and the type of the current exercise, the grip style of the athlete 5 at the end of the current exercise is specified. As described above with reference to FIGS. 2 and 3, this makes it possible to distinguish between different ways of gripping the stick and improve the accuracy of technique recognition.

The information processing device 100 identifies a module based on the combination of how the athlete 5 gripped the bar at the end of the previous exercise and the current exercise, and identifies the likelihoods of the plurality of grips. The information processing device 100 identifies how to grip the stick in the current exercise based on the identified likelihood. As a result, it is possible to improve the recognition accuracy of the grip of the player 5 .

When there is a contradiction in the transition from how the athlete 5 grips the bar at the end of the previous exercise to how the athlete 5 grips the bar in the current exercise, the information processing device 100 determines whether the grips related to the plurality of grips are inconsistent. Correct the likelihood. As a result, it is possible to further improve the recognition accuracy of the grip of the player 5 .

The information processing device 100 identifies the technique performed by the athlete 5 based on the exercise and the grip and combination of the exercise, and scores a score for evaluating the gymnastics performance based on the identified technique. do. This makes it possible to assist the referee's judgment in gymnastics.

By the way, the embodiment described above is an example, and the information processing apparatus 100 may further execute other processes.

Other processes

1 and 2 of the information processing apparatus 100 will be described below.

Other processing 1 will be explained. Although the information processing apparatus 100 acquires distance image data from the 3D laser sensor 10 and generates skeleton information, it is not limited to this, and skeleton information is generated based on image data captured by an RGB camera. may be generated. In this case, the information processing apparatus 100 uses a trained learning model that receives image data and outputs skeleton information.

Other processing 2 will be explained. The identification unit 154 of the information processing device 100 basically executes the above-described processing (processing according to the processing procedure described with reference to FIG. 16) to identify the grip of the player 5. From this point of view, an example of the processing of the identification unit 154 will be described.

The elements that change the grip of competitor 5 include "twisting the body", "twisting the arm by putting the leg between the arms", and "changing the grip". In addition, as for the elements that twist the body, there are those that can be grasped by the amount of twisting the shaft, and those that change the grip depending on how the hand is released.

The identifying unit 154 identifies the gripping method by motion recognition for the element that twists the body that can be grasped by the amount of twisting of the shaft. The identification unit 154 identifies the gripping method based on the gripping method recognition for the element that twists the body and the gripping method changes depending on the gripping method of the separated hand.

The specifying unit 154 specifies how to grip an element in which the arm is twisted by passing the leg between the arms through motion recognition.

The specifying unit 154 specifies how to grip the change-of-grip element by motion recognition or gripping method recognition.

An example of motion recognition performed by the identifying unit 154 will be described. When the player 5 twists in an inverted position, the grip differs depending on the initial grip, the direction of twisting the body, and the amount of twisting. For example, the identification unit 154 indicates that when the player 5 grabs a stick and twists his or her body once (360°) around the gripped hand, the change from an overhand to a backhand is made, or from a backhand to a backhand. identify as changed.

For example, when the athlete 5 grabs a stick and twists his or her body 1/2 times (180°) around the gripping hand, the identifying part 154 changes from a reverse hand to a straight hand or from a straight hand to a reverse hand. , to identify a change from right hand to big hand hand, or from high hand hand to right hand.

The specifying unit 154 specifies that when the arm is twisted by putting the leg between the arms, it changes from the reverse hand to the overhand hand.

An example of grip recognition performed by the identification unit 154 will be described. In the case where the player 5 twists in an inverted position, etc., the grip of the released hand changes depending on how it is gripped. For example, since the direction of twisting the arm is opposite in ogakute and gakute, the player's 5 elbow tends to open outward when grasping with ogakute, and the elbow tends to move inward when grasping with gakute. Based on the rule based on this characteristic, the specifying unit 154 may specify how to grasp. Further, the identifying unit 154 may perform grasp recognition using a learning model trained based on the above characteristics.

An example of motion recognition or grip recognition performed by the identification unit 154 will be described. If player 5 changed grips, it is highly likely that he changed his grip if he made a movement to release his hand from the bar. For example, in most cases, after performing the two-handed ogakute technique, the two hands are released from the stick and the grip is changed to the opposite hand. For example, the identification unit 154 detects that a grip has been changed by motion recognition and grip recognition, and when a specific hand is released in a specific state, the grip changes , changed the grip with both hands upside down).

FIG. 18 is a diagram for supplementary explanation of the processing of the identification unit. The trick shown in Figure 18 is an end-one twist. Athlete 5 exercises in order of T3-1, T3-2, and T3-3. When the specifying unit 154 specifies the grip A3, the following processing is performed for the “hand” and “hand in the air”.

The identification unit 154 identifies the grip after twisting based on "which hand is the left or right hand?", "which side is it twisted around?", and "what is the original grip". The specifying unit 154 specifies “which hand is left or right?” from the skeleton information. The identifying unit 154 identifies “which way the body is twisted” based on motion recognition. The identification unit 154 uses the determination result of the above module or the like for "what is the original nigiri", but there is a possibility that it is wrong.

The identifying unit 154 determines the grip after grasping the hand in the air based on the shape of the arm, the trajectory (whether the elbow is inside or outside, which side of the bar to grasp from when grasping, etc.). Be specific, but you may be wrong.

For "what is the original grip" and "hand in the air" specified by the specifying unit 154, the correct way of gripping is specified by the processing based on the correction unit 155 described above.

Next, an example of the hardware configuration of a computer that implements the same functions as the information processing apparatus 100 described in this embodiment will be described. FIG. 19 is a diagram illustrating an example of a hardware configuration of a computer that implements functions similar to those of an information processing apparatus.

As shown in FIG. 19, the computer 200 has a CPU 201 that executes various arithmetic processes, an input device 202 that receives data input from the user, and a display 203 . The computer 200 also has a communication device 204 and an interface device 205 for exchanging data with other computers via a wired or wireless network. The computer 200 also has a RAM 206 that temporarily stores various information, and a hard disk device 207 . Each device 201 - 207 is then connected to a bus 208 .

The hard disk device 207 has an acquisition program 207a, a generation program 207b, a calculation program 207c, a specific program 207d, a correction program 207e, a technique determination program 207f, and an evaluation program 207g. The CPU 201 reads out the acquisition program 207a, the generation program 207b, the calculation program 207c, the specific program 207d, the correction program 207e, the technique determination program 207f, and the evaluation program 207g, and develops them in the RAM 206. FIG.

The acquisition program 207a functions as an acquisition process 206a. Generation program 207b functions as generation process 206b. The calculation program 207c functions as a calculation process 206c. The specific program 207d functions as the specific process 206d. The correction program 207e functions as a correction process 206e. The technique determination program 207f functions as a technique determination process 206f. Evaluation program 207g functions as evaluation process 206g.

The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the generation process 206 b corresponds to the processing of the generation unit 152 . Processing of the calculation process 206 c corresponds to processing of the calculation unit 153 . The processing of the identification process 206 d corresponds to the processing of the identification unit 154 . Processing of the correction process 206 e corresponds to processing of the correction unit 155 . The processing of the technique determination process 206 f corresponds to the processing of the technique determination section 156 . Processing of the evaluation process 206 g corresponds to processing of the evaluation unit 157 .

It should be noted that the programs 207a to 207g do not necessarily have to be stored in the hard disk device 207 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, IC card, etc., inserted into the computer 200 . Then, the computer 200 may read and execute each program 207a to 207g.

100 information processing device 110 communication unit 120 input unit 130 display unit 140 storage unit 141 distance image DB
142 joint definition data 143 joint position DB
144 skeleton information DB
145 Exercise dictionary data 146 Grip determination table 147 Grip determination result table 148 Technique dictionary data 149 Technique determination result table 150 Control unit 151 Acquisition unit 152 Generation unit 153 Calculation unit 154 Identification unit 155 Correction unit 156 Technique determination unit 157 Evaluation unit

Claims

Acquire the sensing data that sensed the athlete,
generating time-series 3D skeletal information of the athlete based on the sensing data;
When identifying the exercise performed by the athlete based on the skeleton information,
The player's grip on the bar at the end of the second movement based on the combination of the player's grip on the bar at the end of the first movement and the second movement following the first movement. A technique recognition program characterized by causing a computer to execute a process of specifying a technique.
The process of specifying the gripping method is characterized by specifying likelihoods related to a plurality of gripping methods, and specifying the gripping method for the rod at the end of the second movement based on the specified likelihoods. The technique recognition program according to claim 1.
When there is an anatomical contradiction in the transition from the player's grip on the bar at the end of the first exercise to the player's grip on the bar in the second exercise, a plurality of grips 3. The technique recognition program according to claim 2, further causing the computer to execute a process of correcting the likelihood.
　The technique recognition program according to claim 1, characterized in that the process of specifying the gripping method specifies any one of a straight hand, a reverse hand, and a large reverse hand as the gripping method of the player with respect to the stick.
The identifying process identifies the exercise performed by the athlete based on how the athlete grips the stick,
2. The technique recognition program according to claim 1, further causing a computer to execute a process of evaluating gymnastics performances of said competitor based on the identification result of said identifying process.
Acquire the sensing data that sensed the athlete,
generating time-series 3D skeletal information of the athlete based on the sensing data;
When identifying the exercise performed by the athlete based on the skeleton information,
The player's grip on the bar at the end of the second movement based on the combination of the player's grip on the bar at the end of the first movement and the second movement following the first movement. A technique recognition method characterized in that a computer executes a process of identifying a technique.
The process of specifying the gripping method is characterized by specifying likelihoods related to a plurality of gripping methods, and specifying the gripping method for the rod at the end of the second movement based on the specified likelihoods. The technique recognition method according to claim 6.
When there is an anatomical contradiction in the transition from the player's grip on the bar at the end of the first exercise to the player's grip on the bar in the second exercise, a plurality of grips 8. The technique recognition method according to claim 7, wherein the computer further executes a process of modifying the likelihood.
The technique recognition method according to claim 6, characterized in that the process of specifying the way of gripping the stick specifies any one of straight hand, reverse hand, and overhand hand grip as the way of gripping the stick by the player.
The identifying process identifies the exercise performed by the athlete based on how the athlete grips the stick,
7. The technique recognition method according to claim 6, wherein the computer further executes a process of evaluating the gymnastics performance of the competitor based on the identification result of the identifying process.
an acquisition unit that acquires sensing data obtained by sensing an athlete;
generating time-series 3D skeletal information of the athlete based on the sensing data;
When identifying the exercise performed by the athlete based on the skeletal information, a combination of how the athlete grips the bar at the end of the first exercise and the second exercise following the first exercise. and a specifying unit that specifies how the player grips the stick at the end of the second exercise based on the above.
12. The method according to claim 11, wherein the identifying unit identifies likelihoods related to a plurality of gripping manners, and identifies the gripping manner for the stick at the end of the second movement based on the identified likelihoods. The information processing device described.
When there is an anatomical contradiction in the transition from the player's grip on the bar at the end of the first exercise to the player's grip on the bar in the second exercise, a plurality of grips 13. The information processing apparatus according to claim 12, further comprising a corrector for correcting likelihood.
12. The information processing apparatus according to claim 11, wherein the identifying unit identifies any one of a straight hand, a reverse hand, and a large reverse hand as the way of gripping the stick by the player.
A technique determination unit that identifies the exercise performed by the athlete based on how the athlete grips the bar, and evaluates the gymnastics performance of the athlete based on the identification result of the technique determination unit. 12. The information processing apparatus according to claim 11, further comprising an evaluation section.