JP2023179239A - Information processing program, information processing method, and information processing apparatus - Google Patents

Abstract

To provide an information processing program, method, and apparatus capable of improving the accuracy of a model for inferring skeletal data of a person.SOLUTION: In a skeleton recognition system 30, an information processing apparatus 100 is configured to: infer a plurality of pieces of skeletal data of a person included in a plurality of pieces of image data, based on the result of inputting the plurality of pieces of image data in which the areas of the person are cut out from a plurality of pieces of training image data into a learning model; determine whether or not the image data corresponding to abnormal skeletal data is the image data in which the area of the person is abnormal; if it is the image data in which the area of the person is abnormal, identify similar training image data from the plurality of pieces of training image data, wherein the similar training image data has joint position characteristics of a person that is similar to the joint position characteristics of the person identified from abnormal training image data; adjust the area of the person identified from the similar training image data; and train the learning model based on the image data obtained by cutting out the adjusted area of the person from the similar training image data.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing program and the like.

The need for skeletal recognition is increasing in the fields of sports such as gymnastics, healthcare, and entertainment. Furthermore, with the improvement of deep learning technology, the accuracy of two-dimensional (2D) or three-dimensional (3D) skeleton recognition is also improving depending on the image method.

FIG. 16 is a diagram for explaining a conventional skeleton recognition system. For example, in the conventional skeleton recognition system 1, a learning phase process and an inference phase process are executed.

In the learning phase, the joint position learning unit 10 of the skeleton recognition system 1 trains the joint position estimation model 11 based on the training data set 5. The training data set 5 includes a set of image data and a correct label. The image data of the training data set 5 is image data obtained by cutting out a human region included in the image frame of the camera using a bounding box. The correct label of training data set 5 is a label indicating the correct joint position of the person.

In the inference phase, the human detection unit 13 of the skeleton recognition system 1 detects a human region in the image frame photographed by the camera 12, and generates image data 6 by cutting out the human region from the image frame with a bounding box. The joint position estimating unit 14 of the skeleton recognition system 1 obtains the joint position estimation result 7 of the person by inputting the image data 6 to the trained joint position estimation model 11 .

For example, the person detection unit 13 shown in FIG. 1 uses a machine learning model when detecting a person. The person detection unit 13 may not be able to accurately detect a human area if the machine learning model is insufficiently trained. Further, if the joint position estimating unit 14 inputs image data in which a human region cannot be accurately detected to the joint position estimation model 11, the joint positions of the person may not be accurately estimated.

FIG. 17 is a diagram showing an example of joint position estimation results according to image data. In FIG. 17, when image data 6a is input to the joint position estimation model 11, a joint position estimation result 7a is obtained. Since the image data 6a is normal image data obtained by cutting out the human region with a bounding box, the joint position estimation result 7a is a normal estimation result.

When the image data 6b is input to the joint position estimation model 11, a joint position estimation result 7b is obtained. Since the image data 6b is image data in which the left leg is cut off, the joint position estimation result 7b is not a normal estimation result. For example, due to the effect of the left foot being cut off, the joint positions of the right foot and the joint positions of the left foot overlap in the joint position estimation result 7b.

When the image data 6c is input to the joint position estimation model 11, a joint position estimation result 7c is obtained. The image data 6c includes gymnastics equipment in addition to the person, and the extracted area is larger than the human area, so the joint position estimation result 7c is not a normal estimation result. For example, due to the fact that the image data 6c includes gymnastics equipment, in the joint position estimation result 7c, a part of the person's joint position (left ankle) is placed on the gymnastics equipment.

Next, an example of a conventional 3D skeleton recognition system using multiple cameras will be described. FIG. 18 is a diagram showing an example of a 3D skeleton recognition system. As shown in FIG. 18, this 3D skeleton recognition system 2 includes cameras 20a, 20b, 20c, 20d, human detection units 21a, 21b, 21c, 21d, joint position estimation units 22a, 22b, 22c, 22d, It has a 3D joint estimation unit 23.

The cameras 20a to 20d take images of people from different directions, and output the taken image frames to the corresponding person detection units 21a, 21b, 21c, and 21d. In the following description, the cameras 20a to 20d will be collectively referred to as the camera 20 unless otherwise distinguished.

The human detection units 21a to 21d detect human regions from the image frames captured by the camera 20 using trained machine learning models, and output image data in which the detected human regions are cut out using bounding boxes. In the example shown in FIG. 18, the person detection unit 21a outputs image data 8a based on the image frame of the camera 20a. The person detection unit 21b outputs image data 8b based on the image frame of the camera 20b. The person detection unit 21c outputs image data 8c based on the image frame of the camera 20c. The person detection unit 21d outputs image data 8d based on the image frame of the camera 20d.

In the example shown in FIG. 18, image data 8a and 8d are normal image data. On the other hand, image data 8b is image data in which the left ankle is cut off. Image data 8c is image data in which the right ankle is not visible.

The joint position estimation units 22a to 22d generate heat maps 9a, 9b, 9c, and 9d by inputting the image data 8a to 8d, respectively, to the trained joint position estimation model 11 described in FIG. 16. The heat maps 9a to 9d are information indicating the positions of each joint of the person.

For example, the heat maps 9a to 9d include a right ankle joint heat map, a left ankle joint heat map, and joint heat maps of other joints. In a joint heat map, coordinates and likelihoods are associated with each other, and the likelihood of the most probable coordinates of the corresponding joint position is increased.

The heat maps 9a and 9d are heat maps estimated based on normal image data 8a and 8d, and the joint positions shown in each joint heat map are appropriate.

The heat map 9b is a heat map estimated based on the image data 8b in which the left ankle is not visible, and the joint positions shown in the joint heat map of the left ankle are misrecognized. The heat map 9c is a heat map estimated based on the image data 8c in which the right ankle is not visible, and the joint positions shown in the joint heat map of the right ankle are misrecognized.

The 3D joint estimation unit 23 generates 3D skeletal data 24 based on the heat maps 9a to 9d. The 3D skeleton data 24 includes three-dimensional joint positions of a person. For example, when joint positions are incorrectly recognized as in the heat maps 9b and 9c, disturbances occur in the three-dimensional joint positions of the 3D skeleton data 24.

The 3D skeleton recognition system 2 described above repeatedly executes the above processing every time image frames are inputted in time series from the camera 20, and generates a plurality of 3D skeleton data in time series. Time-series 3D skeletal data is used when scoring gymnastics competitions.

Here, as conventional techniques for suppressing disturbances in the three-dimensional joint positions of the 3D skeleton data 24 shown in FIG. 18, there are conventional techniques 1 and 2.

In Prior Art 1, image data of various sizes are prepared as training data sets when training machine learning models used by the human detection units 21a to 21d. For example, in Prior Art 1, the size of the bounding box is not only changed to the normal size, but also intentionally changed at random to artificially create image data with cut-offs, overly large image data that includes objects other than human areas, etc. A machine learning model is trained using such image data. Thereby, the human region is appropriately included in the image data output from the human detection units 21a to 21d, and disturbances in the three-dimensional joint positions of the 3D skeleton data 24 are suppressed.

In conventional technology 2, 3D skeletal data in which disturbances have occurred in the 3D joint positions are identified from a plurality of 3D skeletal data output from the 3D joint estimating unit 23, and an image is used as the basis for generating the identified 3D skeletal data. Extract data (image data with abnormal bounding box size). For example, if a disturbance occurs in the 3D skeleton data 24 in FIG. 18, image data 8b and 8c are extracted. In prior art 2, the joint position estimation model 11 is retrained based on the extracted image data 8b and 8c, thereby suppressing disturbances in the three-dimensional joint positions of the 3D skeleton data 24.

JP2021-056922A JP 2021-174059 Publication

In Prior Art 1 described above, the randomly generated image data is not necessarily the same as the image data corresponding to the actual human area detection error, so the machine learning model cannot be trained appropriately and the machine learning model cannot be trained properly. can't get any effect. Furthermore, if a machine learning model is retrained using image data that does not correspond to actual human area detection errors, the accuracy of the machine learning model may decrease.

Conventional technology 2 uses image data that corresponds to skeletal data in which disturbances in three-dimensional coordinates have actually occurred, so it is difficult to perform retraining using only image data in which abnormalities have actually occurred. It is difficult to secure sufficient training data.

That is, in order to improve the accuracy of a model that infers human skeletal data, it is necessary to secure useful training data and train the model.

In one aspect, the present invention aims to provide an information processing program, an information processing method, and an information processing device that can improve the accuracy of a model for inferring skeletal data of a person.

In the first plan, the computer executes the following process. The computer infers the plurality of skeletal data of the person included in the plurality of image data based on the result of inputting the plurality of image data obtained by cutting out the region of the person from the plurality of training image data into the learning model. The computer detects abnormal skeletal data based on a plurality of skeletal data. The computer performs a search based on a region of the person identified from the abnormal training image data corresponding to the abnormal skeletal data and a region of the person in the image data corresponding to the abnormal skeletal data among the plurality of training image data. Then, it is determined whether or not the image data corresponding to the abnormal skeletal data has an abnormal human area. When the image data corresponding to abnormal skeletal data is abnormal image data, the computer performs similar training that has joint position characteristics of a person similar to the joint position characteristics of the person identified from the abnormal training image data. training image data is identified from a plurality of training image data. The computer adjusts the region of the person specified from the similar training image data based on the region of the person specified from the abnormal training image data. The computer trains the learning model based on the image data obtained by cutting out the adjusted human region from the similar training image data.

The accuracy of a model that infers human skeletal data can be improved.

FIG. 1 is a diagram showing an example of a skeleton recognition system according to this embodiment. FIG. 2 is a diagram for explaining the processing of the information processing apparatus according to this embodiment. FIG. 3 is a diagram for explaining the processing of the person detection section. FIG. 4 is a diagram illustrating an example of segmentation. FIG. 5 is a diagram for explaining the processing of the abnormal image data detection section. FIG. 6 is a diagram (1) for explaining the processing of the similar posture detection section. FIG. 7 is a diagram (2) for explaining the processing of the similar posture detection section. FIG. 8 is a diagram for explaining the processing of the training image data generation section. FIG. 9 is a functional block diagram showing the configuration of the information processing device according to this embodiment. FIG. 10 is a flowchart showing the processing procedure of the information processing apparatus according to this embodiment. FIG. 11 is a flowchart showing the processing procedure of abnormal skeletal data detection processing. FIG. 12 is a flowchart showing the processing procedure of abnormal image data detection processing. FIG. 13 is a flowchart showing the processing procedure of similar posture detection processing. FIG. 14 is a flowchart showing the processing procedure of training image data generation processing. FIG. 15 is a diagram illustrating an example of the hardware configuration of a computer that implements the same functions as the information processing device of the embodiment. FIG. 16 is a diagram for explaining a conventional skeleton recognition system. FIG. 17 is a diagram showing an example of joint position estimation results according to image data. FIG. 18 is a diagram showing an example of a 3D skeleton recognition system.

Embodiments of an information processing program, an information processing method, and an information processing apparatus disclosed in the present application will be described in detail below based on the drawings. Note that the present invention is not limited to this example.

FIG. 1 is a diagram showing an example of a skeleton recognition system according to this embodiment. As shown in FIG. 1, this skeleton recognition system 30 includes cameras 31a, 31b, 31c, and 31d, and an information processing device 100. Cameras 31a to 31d are connected to information processing device 100.

The cameras 31a to 31d are installed at different positions and take images of the contestants (RGB <Red Green Blue> images). The cameras 31a to 31d transmit data of captured images to the information processing device 100. The data of images taken by the cameras 31a to 31d will be referred to as "image frames." The cameras 31a to 31d transmit a plurality of image frames to the information processing device 100 in time series. Frame numbers are assigned to each image frame in ascending order. In the following description, the cameras 31a to 31d will be collectively referred to as "camera 31" as appropriate.

The information processing device 100 trains the skeletal inference model 40 in advance based on the training image data and correct label set stored in the training data set 50. For example, the skeletal inference model 40 is a model that inputs image data obtained by cutting out a human region using a bounding box, and outputs 3D skeletal data.

The information processing device 100 infers the athlete's skeletal data based on the image frame received from the camera 31 and the trained skeletal inference model 40.

Here, a process when the information processing apparatus 100 retrains the skeletal inference model 40 will be described. For example, the information processing device 100 inputs a plurality of image data obtained by cutting out a human region from a plurality of training image data stored in the training data set 50 to the skeletal inference model 40, and infers a plurality of skeletal data.

The information processing device 100 identifies abnormal skeletal data from a plurality of pieces of skeletal data, and determines whether image data from which the abnormal skeletal data is inferred is abnormal image data. When the image data is abnormal, the information processing apparatus 100 identifies other training image data having human characteristics similar to the human characteristics included in the training image data corresponding to the abnormal image data. The information processing device 100 adjusts the human region of the other training image data that has been identified, and uses the image data obtained by cutting out the adjusted human region at the time of retraining.

As a result, the skeletal inference model 40 can be retrained using other image data similar to the human features of the image data corresponding to the actual human area detection error. Furthermore, the skeletal inference model 40 can be retrained using not only the image data corresponding to the skeletal data in which the three-dimensional coordinates are actually disturbed, but also the other image data mentioned above. That is, the skeletal inference model 40 can be retrained to accurately estimate the skeletal data of a person.

FIG. 2 is a diagram for explaining the processing of the information processing apparatus according to this embodiment. As shown in FIG. 2, the information processing device 100 includes a person detection section 151, a skeleton inference section 152, and a segmentation section 153. The information processing device 100 also includes an abnormal skeleton data detection section 154, an abnormal image data detection section 155, a similar posture detection section 156, a training image data generation section 157, and a machine learning execution section 158.

The person detection unit 151 detects a human region from the training image data 60 and generates image data 61 in which the human region is cut out using a bounding box. FIG. 3 is a diagram for explaining the processing of the person detection section. The person detection unit 151 uses a trained machine learning model 151a. The machine learning model 151a includes YOLO (YOU Only Look Once), SSD (Single Shot Multibox Detector), and RCNN (Region Based Convolutional Neural Networks).

As shown in FIG. 3, the person detection unit 151 inputs the training image data 60 into the machine learning model 70, detects a person area A1, and cuts out the detected person area A1 using a bounding box. Output. For example, the training image data 60 is assigned a frame number, and the image data 61 is assigned the same frame number as the training image data 60.

The person detection unit 151 outputs a plurality of image data 61 by repeatedly performing the above processing on a plurality of training image data 60. The person detection section 151 outputs the plurality of image data 61 to the skeleton inference section 152 and the abnormal image data detection section 155.

Returning to the explanation of FIG. 2. The skeletal inference unit 152 infers skeletal data 62 by inputting the image data 61 to the trained skeletal inference model 40. In the skeleton data 62, three-dimensional joint position data is set for each joint of the person. The same frame number as the inference source image data 61 is assigned to the skeleton data 62. As a result, the frame number assigned to the skeleton data 62 becomes the same as the frame number of the image data 61 that is the inference source.

The skeleton inference unit 152 generates a plurality of skeleton data 62 by repeatedly performing the above processing on a plurality of image data 61. The skeletal inference section 152 outputs the plurality of skeletal data 62 to the abnormal skeletal data detection section 154.

The segmentation unit 153 performs segmentation on the training image data 60 to generate segmentation data 63 in which each part of the person included in the training image data 60 is extracted. For example, the segmentation unit 153 uses BodyPix or the like to perform segmentation. The segmentation data 63 is given the same frame number as the training image data 60.

FIG. 4 is a diagram illustrating an example of segmentation. In the example shown in FIG. 4, segmentation data 63 is obtained by performing segmentation on the training image data 60. For example, a person h1 in the training image data 60 is divided into multiple segments p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13, p14, p15 in the segmentation data 63. It is divided. A body part of a person is assigned to each segment p1 to p15. When the training image data 60 includes multiple people, each person is divided into multiple segments.

The segmentation unit 153 generates a plurality of segmentation data 63 by repeatedly performing the above processing on a plurality of training image data 60. The segmentation unit 153 outputs the plurality of segmentation data 63 to the abnormal image data detection unit 155, the similar posture detection unit 156, and the training image data generation unit 157.

The abnormal skeletal data detection unit 154 detects an abnormality from a plurality of skeletal data 62 based on the distance between joint positions (bone length) of the skeletal data 62, the joint angle, and the movement distance of the joint positions of the previous and subsequent skeletal data. Detect skeletal data. In the following description, abnormal skeletal data will be referred to as "abnormal skeletal data."

An example of a process in which the abnormal skeletal data detection unit 154 detects abnormal skeletal data based on bone length will be described. The abnormal skeletal data detection unit 154 selects a first joint position and a second joint position adjacent to the first joint position from among the plurality of joint positions included in the target skeletal data, and selects the first joint position and the second joint position adjacent to the first joint position. Calculate the distance to the second joint position. The abnormal skeletal data detection unit 154 repeatedly executes the above process while changing the first joint position, and calculates the distance (bone length) between each joint position.

The abnormal skeletal data detection unit 154 detects the skeletal data of the target as abnormal skeletal data when the average value of the distances between the joint positions calculated from the skeletal data of the target is equal to or greater than the threshold Th1.

An example of a process in which the abnormal skeletal data detection unit 154 detects abnormal skeletal data based on joint angles will be described. The abnormal skeletal data detection unit 154 selects a first joint and a second joint adjacent to the first joint from among the plurality of joints included in the target skeletal data, and detects the joint between the first joint and the second joint. Calculate the angle. The abnormal skeletal data detection unit 154 detects the target skeletal data as abnormal skeletal data when the angle between the first joint and the second joint is equal to or greater than a threshold value Th2 corresponding to the maximum joint range of motion of the human body. The abnormal skeletal data detection unit 154 repeatedly executes the above process while changing the first joint.

An example of a process in which the abnormal skeletal data detection unit 154 detects abnormal skeletal data based on movement distances of joint positions of previous and subsequent skeletal data will be described. The abnormal skeletal data detection unit 154 selects the first joint position (N) of the first joint of the skeletal data of frame number N, and selects the first joint position (N+1) of the first joint of the skeletal data of frame number N+1. do. N is a natural number. The abnormal skeleton data detection unit 154 calculates the distance (movement distance) between the first joint position (N) and the first joint position (N+1), and when the movement distance is equal to or greater than the threshold Th3, the abnormal skeleton data detection unit 154 detects the target skeleton. The data is detected as abnormal skeletal data. The abnormal skeletal data detection unit 154 repeatedly executes the above process while changing the first joint.

The abnormal skeletal data detection unit 154 detects abnormal skeletal data by performing the above processing on a plurality of skeletal data, and outputs the frame number corresponding to the abnormal skeletal data to the abnormal image data detection unit 155. .

The abnormal image data detection unit 155 obtains image data 61 corresponding to the frame number of the abnormal skeleton data and segmentation data 63 corresponding to the frame number of the abnormal skeleton data. In the following description of the abnormal image data detection unit 155, the image data 61 and segmentation data 63 corresponding to the frame number of the abnormal skeleton data are simply expressed as image data 61 and segmentation data 63.

The abnormal image data detection unit 155 determines whether the image data 61 is abnormal image data 61 based on the image data 61 and the segmentation data 63. In the following description, abnormal image data will be referred to as "abnormal image data."

FIG. 5 is a diagram for explaining the processing of the abnormal image data detection section. The processing of the abnormal image data detection unit 155 will be explained separately in case 1 and case 2.

Case 1 will be explained. The abnormal image data detection unit 155 detects a circumscribed BBOX 75a of a human region made up of a plurality of parts of the segmentation data 63. The abnormal image data detection unit 155 sets a rectangle 75b of the image data 61 on the segmentation data 63. Since the image data 61 is image data corresponding to the BBOX of the human region detected from the training image data 60, the abnormal image data detection unit 155 detects the position of the BBOX of the human region detected from the training image data 60. The rectangle 75b of the image data 61 may be set on the segmentation data 63 using the information.

The abnormal image data detection unit 155 compares the circumscribed BBOX 75a and the rectangle 75b, and determines that the image data 61 is normal image data if the first normal condition and the second normal condition are satisfied. The first normal condition is that the sides of the circumscribed BBOX 75a and the sides of the rectangle 75b do not intersect. The second normal condition is that the distance between the side of the circumscribed BBOX 75a and the side of the rectangle 75b corresponding to the side is less than the threshold Th4.

As a result of comparing the circumscribed BBOX 75a shown in case 1 with the rectangle 75b, the abnormal image data detection unit 155 determines that the image data 61 is normal image data because the first normal condition and the second normal condition are not satisfied.

Case 2 will be explained. The abnormal image data detection unit 155 detects a circumscribed BBOX 75a of a human region made up of a plurality of parts of the segmentation data 63. The abnormal image data detection unit 155 sets a rectangle 75c of the image data 61 on the segmentation data 63.

When the abnormal image data detection unit 155 compares the circumscribed BBOX 75a and the rectangle 75b, the sides of the circumscribed BBOX 75a intersect with the sides of the rectangle 75b, and the first normal condition is not satisfied. Therefore, the abnormal image data detection unit 155 determines the image data 61 to be abnormal image data.

Note that even if the side of the circumscribed BBOX 75a and the side of the rectangle 75b intersect, the abnormal image data detection unit 155 determines that the first normal condition is satisfied if the distance between each side is less than the threshold Th5. You may.

The abnormal image data detection unit 155 executes the above processing, and if it is determined that the image data 61 is abnormal image data, the abnormal image data detection unit 155 sends the frame number of the abnormal image data (image data 61) to the similar posture detection unit 156. Output.

Returning to the explanation of FIG. 2. The similar posture detection unit 156 acquires segmentation data 63 (hereinafter referred to as reference data) corresponding to the frame number of the abnormal image data (image data 61). The similar posture detection unit 156 compares the reference data and the segmentation data 63, and detects segmentation data 63 that is similar to the person's posture of the reference data and the camera angle of the reference data.

6 and 7 are diagrams for explaining the processing of the similar posture detection section. FIG. 6 will be explained. Similar posture detection section 156 generates reference vector information 76a by identifying joint positions based on the boundaries of each part of reference data 76 and repeating the process of obtaining partial vectors of two adjacent joint positions. In the example shown in FIG. 6, the number of joints of the person is "15". Each joint is defined as J0 to J14, and the joint position (joint coordinates) of the joint is Ji (x, y). For example, assume that the position of the pixel at the upper left corner of the reference data 76 is (0, 0). x is the pixel position in the width direction, and y is the pixel position in the height direction.

For example, the similar posture detection unit 156 calculates the partial vector V0 by "V0=J1-J0". The similar posture detection unit 156 calculates the partial vector V1 by “V1=J2−J1”. The similar posture detection unit 156 calculates the partial vector V2 by "V2=J3-J1". The similar posture detection unit 156 calculates the partial vector V3 by "V3=J4-J3". The similar posture detection unit 156 calculates the partial vector V4 by "V4=J5-J4". The similar posture detection unit 156 calculates the partial vector V5 by "V5=J6-J1". The similar posture detection unit 156 calculates the partial vector V6 by "V6=J7-J6". The similar posture detection unit 156 calculates the partial vector V7 by "V7=J8-J7".

The similar posture detection unit 156 calculates the partial vector V8 by "V8=J9-J0". The similar posture detection unit 156 calculates the partial vector V9 by "V9=J10-J9". The similar posture detection unit 156 calculates the partial vector V10 by "V10=J11-J10". The similar posture detection unit 156 calculates the partial vector V11 by "V11=J12-J0". The similar posture detection unit 156 calculates the partial vector V12 by "V12=J13-J12". The similar posture detection unit 156 calculates the partial vector V13 by "V13=J14-J13".

The similar posture detecting unit 156 calculates each partial vector V0 to V13 by executing the above processing, and generates the reference vector information 76a.

The similar posture detection unit 156 normalizes each partial vector based on equation (1). Let the partial vector after normalization be V' _i . By normalizing, in the case of the same posture, the magnitude and direction of the partial vectors become the same regardless of the front and rear positions (far or near) of the person in the image.

Moving on to the explanation of FIG. The similar posture detection unit 156 generates reference vector information 76a from the reference data 76 by executing the process described in FIG. The similar posture detection unit 156 calculates each partial vector and generates vector information 77 by performing processing similar to the processing described in FIG. 6 on the segmentation data 63.

The similar posture detection unit 156 determines whether the reference vector information 76a and the vector information 77 are similar based on the comparison results between each partial vector of the reference vector information 76a and each partial vector of the vector information 77. judge. For example, the similar posture detection unit 156 performs a process of determining whether the difference between the partial vector V' _i of the reference vector information 76a and the partial vector V' _i of the vector information 77 is less than the threshold Th6. Execute each for 0 to 13. The similar posture detection unit 156 determines that the reference vector information 76a and the vector information 77 are similar when there is no partial vector with a difference less than the threshold Th6.

When determining that the reference vector information 76a and the vector information 77 are similar, the similar posture detection unit 156 identifies the frame number of the segmentation data 63 from which the vector information 77 is generated. In the following description, the frame number of the segmentation data 63 from which the reference vector information 76a is generated will be referred to as a reference frame number. The frame number of the segmentation data 63 from which the vector information 77 similar to the reference vector information 76a is generated is referred to as a similar frame number. Similar posture detection section 156 outputs the reference frame number and similar frame number to training image data generation section 157.

The similar posture detection unit 156 identifies similar frame numbers by repeatedly performing the above processing on the plurality of segmentation data 63, and uses the reference frame number and the similar frame number as the training image data generation unit 157. Output to.

Returning to the explanation of FIG. 2. The training image data generation unit 157 generates image data 64 based on the reference frame number and similar frame numbers.

FIG. 8 is a diagram for explaining the processing of the training image data generation section. The training image data generation unit 157 acquires segmentation data (reference data) corresponding to the reference frame number. The training image data generation unit 157 acquires segmentation data (hereinafter referred to as similar segmentation data 80) corresponding to the similar frame number.

The training image data generation unit 157 detects a circumscribed BBOX 77a of a human region made up of a plurality of parts of the similar segmentation data 80. The training image data generation unit 157 detects a circumscribed BBOX 77b of a human region made up of a plurality of parts of the reference data. The training image data generation unit 157 generates a BBOX 77c in which the aspect ratio of the circumscribed BBOX 77a is adjusted to match the aspect ratio of the circumscribed BBOX 77b.

The training image data generation unit 157 acquires the training image data 60 corresponding to the similar frame number, and generates the image data 64 by cutting out the training image data 60 using the BBOX 77c. The training image data generation unit 157 generates a plurality of image data 64 by performing the above processing on each set of a reference frame number and a similar frame number. The training image data generation unit 157 generates a training data set by setting the correct label of the training image data 60 corresponding to the similar frame number as the correct label corresponding to the image data 64.

The machine learning execution unit 158 regenerates the skeletal inference model 40 using the training data set and the training image data 60 (the image data 61 cut out by the bounding box and the correct label) generated by the training image data generation unit 157. train. For example, the machine learning execution unit 158 trains the parameters of the skeletal inference model 40 so that the output when image data is input to the skeletal inference model 40 approaches the correct label.

Next, a configuration example of the information processing device 100 according to the present embodiment will be described. FIG. 9 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in FIG. 9, the information processing device 100 includes a communication section 110, an input section 120, a display section 130, a storage section 140, and a control section 150.

The communication unit 110 receives image frames from the camera 31. The communication unit 110 may perform data communication with an external device and receive the training data set 50 and the like.

The input unit 120 is realized using an input device such as a keyboard or a mouse, and inputs various information to the control unit 150 in response to input operations by an operator.

The display unit 130 is realized by a display device such as a liquid crystal display.

The storage unit 140 includes a skeletal inference model 40 and a training data set 50. Although not shown, the storage unit 140 also stores various data used by the control unit 150. The storage unit 140 is realized by, for example, a semiconductor memory element such as a flash memory, or a storage device such as a hard disk or an optical disk.

The skeletal inference model 40 is a model that inputs image data obtained by cutting out a human region using a bounding box, and outputs 3D skeletal data. The skeleton inference model 40 is a neural network (NN) or the like.

The training data set 50 stores a plurality of pairs of training image data and correct labels. The training data set 50 is used when training the skeletal inference model 40.

The technique recognition table 141 is a table that associates time-series changes in the positions of each joint included in each skeleton data with the type of technique. Further, the technique recognition table 141 associates combinations of technique types with scores. The score is calculated as the sum of the D (Difficulty) score and the E (Execution) score. For example, the D score is a score calculated based on the difficulty level of the technique. The E score is a score calculated by a point deduction method according to the degree of perfection of the technique.

The control unit 150 includes a person detection unit 151, a skeleton inference unit 152, and a segmentation unit 153. The control unit 150 includes an abnormal skeleton data detection unit 154, an abnormal image data detection unit 155, a similar posture detection unit 156, a training image data generation unit 157, a machine learning execution unit 158, and a technique recognition unit 159. The control unit 150 is realized by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Further, the control unit 150 may be executed by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

Each of the processing units 151 to 159 of the control unit 150 executes different processing in the inference phase and the learning phase.

The inference phase processing of the control unit 150 will be explained. In the inference phase, the person detection unit 151, the skeleton inference unit 152, and the technique recognition unit 159 execute processing.

The human detection unit 151 receives an image frame transmitted from the camera 31, detects a human region from the image frame, and generates image data by cutting out the human region using a bounding box. The person detection unit 151 outputs the generated image data to the skeleton inference unit 152. The person detection unit 151 repeatedly performs the above processing on time-series image frames.

The skeletal inference unit 152 infers the athlete's skeletal data by inputting the image data acquired from the person detection unit 151 into the trained skeletal inference model 40. The skeleton inference unit 152 outputs the inferred skeleton data to the technique recognition unit 159. The skeleton estimation unit 152 repeatedly performs the above processing on time-series image data.

The technique recognition unit 159 identifies time-series changes in the positions of each joint based on the time-series skeletal data. The technique recognition unit 159 compares the time-series changes in the positions of each joint with the technique recognition table to identify the type of technique. The technique recognition unit 159 also compares the combination of technique types with the technique recognition table 141 to calculate the score of the contestant's performance.

The technique recognition unit 159 generates screen information based on the performance score and skeletal data from the start to the end of the performance. The technique recognition unit 159 outputs the generated screen information to the display unit 130 for display.

Next, the learning phase processing of the control unit 150 will be described. In the learning phase, a person detection unit 151, a skeleton inference unit 152, a segmentation unit 153, an abnormal skeleton data detection unit 154, an abnormal image data detection unit 155, a similar posture detection unit 156, a training image data generation unit 157, and a machine learning execution unit 158 executes the process. The learning phase process of the control unit 150 corresponds to the process described with reference to FIG.

The person detection unit 151 acquires training image data 60 stored in the training data set 50, detects a human region from the training image data 60, and generates image data 61 in which the human region is cut out using a bounding box. . The person detection section 151 outputs the plurality of image data 61 to the skeleton inference section 152 and the abnormal image data detection section 155. Other processes of the person detection unit 151 in the learning phase are similar to those of the person detection unit 151 described with reference to FIG.

The skeleton estimation unit 152 infers skeleton data 62 by inputting the image data 61 to the trained skeleton inference model 40. The skeletal inference section 152 outputs the plurality of skeletal data 62 to the abnormal skeletal data detection section 154. Other processes related to the skeleton estimating section 152 are similar to those of the skeleton estimating section 152 described with reference to FIG.

The segmentation unit 153 acquires training image data 60 stored in the training data set 50. The segmentation unit 153 performs segmentation on the training image data 60 to generate segmentation data 63 in which each part of the person included in the training image data 60 is extracted. The segmentation unit 153 outputs the plurality of segmentation data 63 to the abnormal image data detection unit 155, the similar posture detection unit 156, and the training image data generation unit 157. Other processes related to the segmentation unit 153 are similar to those of the segmentation unit 153 described with reference to FIG.

The abnormal skeletal data detection unit 154 detects an abnormality from a plurality of skeletal data 62 based on the distance between joint positions (bone length) of the skeletal data 62, the joint angle, and the movement distance of the joint positions of the previous and subsequent skeletal data. Detect abnormal skeletal data (abnormal skeletal data). The abnormal skeleton data detection unit 154 outputs the frame number corresponding to the abnormal skeleton data to the abnormal image data detection unit 155. Other processes regarding the abnormal skeletal data detection section 154 are similar to those of the abnormal skeletal data detection section 154 described with reference to FIG.

The abnormal image data detection unit 155 acquires image data 61 corresponding to the frame number of the abnormal skeleton data and segmentation data 63 corresponding to the frame number of the abnormal skeleton data, and determines whether the image data 61 is abnormal image data. Determine whether The abnormal image data detection unit 155 outputs the frame number of the abnormal image data (image data 61) to the similar posture detection unit 156. Other processes related to the abnormal image data detection section 155 are similar to those of the abnormal image data detection section 155 described with reference to FIG.

The similar posture detection unit 156 acquires segmentation data 63 (hereinafter referred to as reference data) corresponding to the frame number of the abnormal image data (image data 61). The similar posture detection unit 156 compares the reference data and the segmentation data 63, and detects segmentation data 63 that is similar to the person's posture of the reference data and the camera angle of the reference data. The similar posture detection unit 156 outputs the reference frame number of the reference data and the similar frame number of the segmentation data 63 that is similar to the camera angle of the reference data to the training image data generation unit 157. Other processing related to the similar posture detection section 156 is the same as the processing of the similar posture detection section 156 described with reference to FIG.

The training image data generation unit 157 generates image data 64 based on the reference frame number and similar frame numbers. The training image data generation unit 157 generates a training data set by setting the correct label of the training image data 60 corresponding to the similar frame number as the correct label corresponding to the image data 64. The training image data generation unit 157 additionally registers the generated training data set data in the training data set 50. Other processing related to the training image data generation section 157 is similar to the processing of the training image data generation section 157 described with reference to FIG.

The machine learning execution unit 158 retrains the skeletal inference model 40 using the training data set 50. For example, the machine learning execution unit 158 trains the parameters of the skeletal inference model 40 so that the output when image data is input to the skeletal inference model 40 approaches the correct label.

Next, an example of a processing procedure of the information processing apparatus 100 according to the present embodiment will be described. FIG. 10 is a flowchart showing the processing procedure of the information processing apparatus according to this embodiment. As shown in FIG. 10, the control unit 150 of the information processing device 100 acquires training image data from the training data set 50 (step S101).

The person detection unit 151 of the information processing device 100 performs person detection on the training image data to generate image data (step S102). The skeleton inference unit 152 of the information processing device 100 inputs the image data to the skeleton inference model 40 and infers the skeleton data (step S103). The segmentation unit 153 of the information processing device 100 performs segmentation on the training image data to generate segmentation data (step S104).

The abnormal skeletal data detection unit 154 of the information processing device 100 executes abnormal skeletal data detection processing (step S105). The abnormal image data detection unit 155 of the information processing device 100 executes abnormal image data detection processing (step S106).

The similar posture detection unit 156 of the information processing device 100 executes similar posture detection processing (step S107). The training image data generation unit 157 of the information processing device 100 executes training image data generation processing (step S108).

The training image data generation unit 157 registers the generated training image data and correct label in the training data set 50 (step S109). The machine learning execution unit 158 of the information processing device 100 trains a skeletal inference model using the training data set 50 (step S110).

If the information processing apparatus 100 continues the process (step S111, Yes), the process moves to step S101. If the information processing apparatus 100 does not continue the process (step S111, No), the information processing apparatus 100 ends the process.

Next, an example of the processing procedure of the abnormal skeleton data detection process shown in step S105 of FIG. 10 will be described. FIG. 11 is a flowchart showing the processing procedure of abnormal skeletal data detection processing. The abnormal skeletal data detection unit 154 of the information processing device 100 acquires skeletal data (step S201).

The abnormal skeletal data detection unit 154 calculates the length of the bone based on the skeletal data (step S202). If the length of the bone is equal to or greater than the threshold Th1 (step S203, Yes), the abnormal skeleton data detection unit 154 moves to step S209.

On the other hand, if the bone length is not equal to or greater than the threshold Th1 (step S203, No), the abnormal skeletal data detection unit 154 calculates the joint angle based on the skeletal data (step S204). If the joint angle is equal to or greater than the threshold Th2 (step S205, Yes), the abnormal skeleton data detection unit 154 moves to step S209.

If the joint angle is not equal to or greater than the threshold Th2 (step S205, No), the abnormal skeleton data detection unit 154 calculates the movement distance based on the skeleton data (step S206). If the moving distance is not equal to or greater than the threshold Th2 (Step S207, No), the abnormal skeleton data detection unit 154 determines that the skeleton data is normal (Step S208).

On the other hand, if the moving distance is equal to or greater than the threshold Th2 (Step S207, Yes), the abnormal skeleton data detection unit 154 determines that the skeleton data is abnormal (Step S209). The abnormal skeleton data detection unit 154 outputs the frame number of the abnormal skeleton data (step S210).

Next, an example of the processing procedure of the abnormal image data detection process shown in step S106 of FIG. 10 will be described. FIG. 12 is a flowchart showing the processing procedure of abnormal image data detection processing. As shown in FIG. 12, the abnormal image data detection unit 155 of the information processing device 100 acquires image data corresponding to the frame number of the abnormal skeleton data and segmentation data (step S301).

The abnormal image data detection unit 155 detects the circumscribed BBOX of the human region of the segmentation data (step S302). The abnormal image data detection unit 155 compares the rectangle of the image data and the circumscribed BBOX (step S303).

If the relationship between the rectangle of the image data and the circumscribed BBOX satisfies the first normal condition (step S304, Yes), the abnormal image data detection unit 155 moves to step S306. On the other hand, if the first normal condition is not satisfied (step S304, No), the abnormal image data detection unit 155 determines that the image data is abnormal (cut off) (step S305), and proceeds to step S308.

If the relationship between the rectangle of the image data and the circumscribed BBOX satisfies the second normal condition (Step S306, Yes), the abnormal image data detection unit 155 determines that the image data is normal (Step S309).

On the other hand, if the second normal condition is not satisfied (step S306, No), the abnormal image data detection unit 155 determines that the image data is abnormal (too large) (step S307). The abnormal image data detection unit 155 outputs the frame number of the abnormal image data (step S308).

Next, an example of the processing procedure of the similar posture detection processing shown in step S107 in FIG. 10 will be described. FIG. 13 is a flowchart showing the processing procedure of similar posture detection processing. The similar posture detection unit 156 of the information processing device 100 acquires segmentation data (reference data) corresponding to the frame number of the abnormal image data (step S401). The similar posture detection unit 156 calculates and normalizes each partial vector of the reference data (step S402).

The similar posture detection unit 156 acquires unselected segmentation data (step S403). The similar posture detection unit 156 calculates and normalizes each partial vector of the obtained segmentation data (step S404).

The similar posture detection unit 156 compares each partial vector of the reference data with each partial vector of the segmentation data (step S405). If the difference in angle and size of all partial vectors is not less than the threshold Th6 (step S406, No), the similar posture detection unit 156 determines the segmentation data as a dissimilar frame (step S407), and moves to step S410. do.

On the other hand, if the difference in angle and size of all partial vectors is less than the threshold Th6 (Step S406, Yes), the similar posture detection unit 156 determines that the segmentation data is a similar frame (Step S408). The similar posture detection unit 156 outputs the similar frame number of the similar frame (step S409).

If the similar posture detection unit 156 has not selected all the segmentation data (step S410, No), the process proceeds to step S403. When the similar posture detection unit 156 selects all the segmentation data (step S410, Yes), the process ends.

Next, an example of the processing procedure of the training image data generation process shown in step S108 of FIG. 10 will be described. FIG. 14 is a flowchart showing the processing procedure of training image data generation processing. As shown in FIG. 14, the training image data generation unit 157 of the information processing device 100 acquires similar segmentation data corresponding to a similar frame number (step S501).

The training image data generation unit 157 acquires segmentation data (reference data) corresponding to the reference frame number (step S502). The training image data generation unit 157 detects the circumscribed BBOX of the similar segmentation data (step S503).

The training image data generation unit 157 detects the circumscribed BBOX of the reference data (step S504). The training image data generation unit 157 adjusts the circumscribed BBOX of the similar segmentation data based on the aspect ratio of the circumscribed BBOX of the reference data (step S505).

The training image data generation unit 157 generates image data by cutting out the training image data using the adjusted circumscribed BBOX (step S506).

Next, the effects of the information processing device 100 according to this embodiment will be explained. The information processing device 100 identifies abnormal skeletal data from a plurality of pieces of skeletal data, and determines whether image data from which the abnormal skeletal data is inferred is abnormal image data. When the image data is abnormal, the information processing apparatus 100 identifies other training image data having human characteristics similar to the human characteristics included in the training image data corresponding to the abnormal image data. The information processing device 100 adjusts the human region of the other training image data that has been identified, and uses the image data obtained by cutting out the adjusted human region at the time of retraining.

As a result, the skeletal inference model 40 can be retrained using other image data similar to the human features of the image data corresponding to the actual human area detection error. Furthermore, the skeletal inference model 40 can be retrained using not only the image data corresponding to the skeletal data in which the three-dimensional coordinates are actually disturbed, but also the other image data mentioned above. That is, the skeletal inference model 40 can be retrained to accurately estimate the skeletal data of a person.

The information processing device 100 detects abnormal skeletal data based on the distance between joint positions and joint angles included in the skeletal data, and the movement distance of the same joint position in consecutive skeletal data. This makes it possible to detect skeletal data in which three-dimensional coordinate disturbances have actually occurred.

The information processing apparatus 100 performs segmentation on the training image data to identify multiple body parts of the person and identify circumscribed rectangles that circumscribe the multiple body parts. Accordingly, in addition to the human area specified by the detection unit 151, it is possible to specify the circumscribed rectangle of the human area based on the segmentation result.

The information processing device 100 detects abnormal image data based on the image data 61 and a circumscribed rectangle based on the segmentation data. This makes it possible to specify image data that can be used as candidates for retraining.

The information processing apparatus 100 detects image data having a vector based on a joint position of a person similar to a vector based on a joint position of a person included in the abnormal image data. Thereby, it is possible to detect image data that is a candidate that can be used at the time of retraining and that is similar to the abnormal image data.

Next, an example of the hardware configuration of a computer that implements the same functions as the information processing apparatus 100 shown in the above embodiment will be described. FIG. 15 is a diagram illustrating an example of the hardware configuration of a computer that implements the same functions as the information processing device of the embodiment.

As shown in FIG. 15, the computer 200 includes a CPU 201 that executes various calculation processes, an input device 202 that receives data input from a user, and a display 203. The computer 200 also includes a communication device 204 and an interface device 205 that exchange data with an external device or the like via a wired or wireless network. The computer 200 also includes a RAM 206 that temporarily stores various information and a hard disk device 207. Each device 201-207 is then connected to a bus 208.

The hard disk device 207 includes a person detection program 207a, a skeleton inference program 207b, a segmentation program 207c, an abnormal skeleton data detection program 207d, and an abnormal image data detection program 207e. The hard disk device 207 includes a similar posture detection program 207f, a training image data generation program 207g, a machine learning execution program 207h, and a technique recognition program 207i. Further, the CPU 201 reads each program 207a to 207i and expands it in the RAM 206.

The person detection program 207a functions as a person detection process 206a. The skeleton inference program 207b functions as a skeleton inference process 206b. Segmentation program 207c functions as segmentation process 206c. The abnormal skeleton data detection program 207d functions as an abnormal skeleton data detection process 206d. The abnormal image data detection program 207e functions as an abnormal image data detection process 206e. The similar posture detection program 207f functions as a similar posture detection process 206f. The training image data generation program 207g functions as a training image data generation process 207g. The machine learning execution program 207h functions as a machine learning execution process 206h. The technique recognition program 207i functions as a technique recognition process 206i.

The processing of the person detection process 206a corresponds to the processing of the person detection unit 151. The processing of the skeleton inference process 206b corresponds to the processing of the skeleton estimation unit 152. The processing of the segmentation process 206c corresponds to the processing of the segmentation unit 153. The processing of the abnormal skeletal data detection process 206d corresponds to the processing of the abnormal skeletal data detection unit 154. The processing of the abnormal image data detection process 206e corresponds to the processing of the abnormal image data detection section 155. The processing of the similar posture detection process 206f corresponds to the processing of the similar posture detection section 156. The processing of the training image data generation process 207g corresponds to the processing of the training image data generation unit 157. The processing of the machine learning execution process 206h corresponds to the processing of the machine learning execution unit 158. The processing of the technique recognition process 206i corresponds to the processing of the technique recognition section 159.

Note that each of the programs 207a to 207i does not necessarily have to be stored in the hard disk device 307 from the beginning. For example, each program is stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each program 207a to 207i.

Regarding the embodiments including each of the above examples, the following additional notes are further disclosed.

(Appendix 1) Based on the results of inputting a plurality of image data into a learning model into which a region of a person is extracted from a plurality of training image data, multiple pieces of skeletal data of a person included in the plurality of image data are inferred. death,
Detecting abnormal skeletal data based on the plurality of skeletal data,
Based on the region of the person identified from the abnormal training image data corresponding to the abnormal skeletal data among the plurality of training image data and the region of the person in the image data corresponding to the abnormal skeletal data. and determining whether or not the image data corresponding to the abnormal skeletal data has an abnormal human area,
When image data corresponding to the abnormal skeletal data is abnormal image data, similarity having characteristics of joint positions of the person similar to characteristics of joint positions of the person identified from the abnormal training image data. identifying training image data from the plurality of training image data;
adjusting the region of the person specified from the similar training image data based on the region of the person specified from the abnormal training image data;
An information processing program that causes a computer to execute a process of training the learning model based on image data obtained by cutting out a region of a person after adjustment from the similar training image data.

(Supplementary note 2) The process of detecting the abnormal skeletal data is based on the distance between joint positions included in the skeletal data, the joint angle, and the movement distance of the same joint position in consecutive skeletal data. The information processing program according to supplementary note 1, which detects abnormal skeletal data.

(Additional note 3) A process of performing segmentation on the plurality of training image data to identify a plurality of parts of the person, and specifying a circumscribed rectangle that circumscribes the plurality of parts as an area of the person. The information processing program according to supplementary note 1, further being caused to be executed by a computer.

(Additional note 4) The process of determining whether the image data is abnormal includes a circumscribed rectangle obtained from the segmentation result of the abnormal training image data and a region of a person in the image data corresponding to the abnormal skeletal data. The information processing program according to appendix 3, characterized in that the abnormal image data is identified based on a comparison result with the above.

(Additional Note 5) The process of identifying the similar training image data is based on the segmentation result, and the first joint position of the person included in the abnormal training image data and the training image data to be compared is determined based on the segmentation result. If the vector based on the first joint position and the vector based on the second joint position are similar, the training image data to be compared is used for the similar training. The information processing program according to appendix 4, characterized in that the information processing program is specified as image data for use.

(Additional Note 6) Based on the results of inputting a plurality of image data into a learning model into which a region of a person is extracted from a plurality of training image data, multiple pieces of skeletal data of a person included in the plurality of image data are inferred. death,
Detecting abnormal skeletal data based on the plurality of skeletal data,
Based on the region of the person identified from the abnormal training image data corresponding to the abnormal skeletal data among the plurality of training image data and the region of the person in the image data corresponding to the abnormal skeletal data. and determining whether or not the image data corresponding to the abnormal skeletal data has an abnormal human area,
When image data corresponding to the abnormal skeletal data is abnormal image data, similarity having characteristics of joint positions of the person similar to characteristics of joint positions of the person identified from the abnormal training image data. identifying training image data from the plurality of training image data;
adjusting the region of the person specified from the similar training image data based on the region of the person specified from the abnormal training image data;
An information processing method characterized in that a computer executes a process of training the learning model based on image data obtained by cutting out a region of a person after adjustment from the similar training image data.

(Additional Note 7) The process of detecting the abnormal skeletal data is based on the distance between joint positions included in the skeletal data, the joint angle, and the movement distance of the same joint position in consecutive skeletal data. The information processing method according to appendix 6, characterized in that abnormal skeletal data is detected.

(Additional note 8) A process of performing segmentation on the plurality of training image data to identify a plurality of parts of the person, and specifying a circumscribed rectangle that circumscribes the plurality of parts as an area of the person. The information processing method according to appendix 6, further comprising causing a computer to execute the information processing method.

(Additional Note 9) The process of determining whether the image data is abnormal includes a circumscribing rectangle obtained from the segmentation result of the abnormal training image data and a region of a person in the image data corresponding to the abnormal skeletal data. The information processing method according to appendix 8, characterized in that the abnormal image data is identified based on a comparison result between the image data and the image data.

(Additional note 10) The process of identifying the similar training image data is based on the segmentation result, and the first joint position of the person included in the abnormal training image data and the training image data to be compared is determined based on the segmentation result. If the vector based on the first joint position and the vector based on the second joint position are similar, the training image data to be compared is used for the similar training. The information processing method according to appendix 9, characterized in that the information processing method is specified as image data for use.

(Additional Note 11) Based on the results of inputting a plurality of image data into a learning model into which a region of a person is extracted from a plurality of training image data, multiple pieces of skeletal data of a person included in the plurality of image data are inferred. death,
Detecting abnormal skeletal data based on the plurality of skeletal data,
Based on the region of the person identified from the abnormal training image data corresponding to the abnormal skeletal data among the plurality of training image data and the region of the person in the image data corresponding to the abnormal skeletal data. and determining whether or not the image data corresponding to the abnormal skeletal data has an abnormal human area,
When image data corresponding to the abnormal skeletal data is abnormal image data, similarity having characteristics of joint positions of the person similar to characteristics of joint positions of the person identified from the abnormal training image data. identifying training image data from the plurality of training image data;
adjusting the region of the person specified from the similar training image data based on the region of the person specified from the abnormal training image data;
An information processing apparatus comprising a control unit that executes a process of training the learning model based on image data obtained by cutting out a region of a person after adjustment from the similar training image data.

(Additional Note 12) The process of detecting the abnormal skeletal data executed by the control unit includes the distance between joint positions included in the skeletal data, the joint angle, and the movement distance of the same joint position in consecutive skeletal data. The information processing device according to appendix 11, wherein the abnormal skeleton data is detected based on.

(Supplementary note 13) The control unit executes segmentation on the plurality of training image data to identify a plurality of parts of the person, and defines a circumscribing rectangle that circumscribes the plurality of parts as an area of the person. The information processing apparatus according to supplementary note 11, further executing a process specified as .

(Additional Note 14) The process of determining whether the image data is abnormal, which is executed by the control unit, is based on a circumscribed rectangle obtained from the segmentation result of the abnormal training image data and corresponding to the abnormal skeletal data. The information processing device according to appendix 13, wherein the abnormal image data is identified based on a comparison result of the image data with a region of a person.

(Additional Note 15) The process of identifying the similar training image data executed by the control unit is based on the segmentation result, and the first joint position of the person included in the abnormal training image data and the comparison target. A second joint position of the person included in the training image data is specified, and if a vector based on the first joint position and a vector based on the second joint position are similar, the comparison target training image The information processing device according to appendix 14, characterized in that the data is specified as the similar training image data.

100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Technique recognition table 150 Control unit 151 Person detection unit 152 Skeleton inference unit 153 Segmentation unit 154 Abnormal skeleton data detection unit 155 Abnormal image data detection unit 156 Similar posture detection Section 157 Training image data generation section 158 Machine learning execution section 159 Technique recognition section

Claims (7)

Inferring a plurality of skeletal data of a person included in the plurality of image data based on the result of inputting a plurality of image data in which a region of a person is cut out from a plurality of training image data into a learning model,
Detecting abnormal skeletal data based on the plurality of skeletal data,
Based on the region of the person identified from the abnormal training image data corresponding to the abnormal skeletal data among the plurality of training image data and the region of the person in the image data corresponding to the abnormal skeletal data. and determining whether or not the image data corresponding to the abnormal skeletal data has an abnormal human area,
When image data corresponding to the abnormal skeletal data is abnormal image data, similarity having characteristics of joint positions of the person similar to characteristics of joint positions of the person identified from the abnormal training image data. identifying training image data from the plurality of training image data;
adjusting the region of the person specified from the similar training image data based on the region of the person specified from the abnormal training image data;
An information processing program that causes a computer to execute a process of training the learning model based on image data obtained by cutting out a region of a person after adjustment from the similar training image data. The process of detecting the abnormal skeletal data includes detecting the abnormal skeletal data based on the distance between joint positions and joint angles included in the skeletal data, and the movement distance of the same joint position in consecutive skeletal data. The information processing program according to claim 1, wherein the information processing program detects. The computer further executes a process of performing segmentation on the plurality of training image data to specify a plurality of parts of the person, and specifying a circumscribed rectangle that circumscribes the plurality of parts as an area of the person. 2. The information processing program according to claim 1, wherein the information processing program performs the following operations. The process of determining whether the image data is abnormal includes the comparison result between the circumscribed rectangle obtained from the segmentation result of the abnormal training image data and the region of the person in the image data corresponding to the abnormal skeletal data. 4. The information processing program according to claim 3, wherein the abnormal image data is identified based on. The process of identifying the similar training image data includes identifying the first joint position of the person included in the abnormal training image data and the first joint position of the person included in the comparison target training image data based on the segmentation result. If the vector based on the first joint position and the vector based on the second joint position are similar, the training image data to be compared is used as the similar training image data. 5. The information processing program according to claim 4, wherein the information processing program specifies the information processing program. Inferring a plurality of skeletal data of a person included in the plurality of image data based on the result of inputting a plurality of image data in which a region of a person is cut out from a plurality of training image data into a learning model,
Detecting abnormal skeletal data based on the plurality of skeletal data,
Based on the region of the person identified from the abnormal training image data corresponding to the abnormal skeletal data among the plurality of training image data and the region of the person in the image data corresponding to the abnormal skeletal data. and determining whether or not the image data corresponding to the abnormal skeletal data has an abnormal human area,
When image data corresponding to the abnormal skeletal data is abnormal image data, similarity having characteristics of joint positions of the person similar to characteristics of joint positions of the person identified from the abnormal training image data. identifying training image data from the plurality of training image data;
adjusting the region of the person specified from the similar training image data based on the region of the person specified from the abnormal training image data;
An information processing method characterized in that a computer executes a process of training the learning model based on image data obtained by cutting out a region of a person after adjustment from the similar training image data. Inferring a plurality of skeletal data of a person included in the plurality of image data based on the result of inputting a plurality of image data in which a region of a person is cut out from a plurality of training image data into a learning model,
Detecting abnormal skeletal data based on the plurality of skeletal data,
Based on the region of the person identified from the abnormal training image data corresponding to the abnormal skeletal data among the plurality of training image data and the region of the person in the image data corresponding to the abnormal skeletal data. and determining whether or not the image data corresponding to the abnormal skeletal data has an abnormal human area,
When image data corresponding to the abnormal skeletal data is abnormal image data, similarity having characteristics of joint positions of the person similar to characteristics of joint positions of the person identified from the abnormal training image data. identifying training image data from the plurality of training image data;
adjusting the region of the person specified from the similar training image data based on the region of the person specified from the abnormal training image data;
An information processing apparatus comprising a control unit that executes a process of training the learning model based on image data obtained by cutting out a region of a person after adjustment from the similar training image data.

Images