WO2022044399A1

WO2022044399A1 - Terminal device and support method for improving form

Info

Publication number: WO2022044399A1
Application number: PCT/JP2021/010525
Authority: WO
Inventors: 英和鈴木; 洋介芳賀; 知恵子遠藤; 裕明梨本; 恵児西巻
Original assignee: 株式会社Ｊｖｃケンウッド
Priority date: 2020-08-27
Filing date: 2021-03-16
Publication date: 2022-03-03
Also published as: JP2022039120A

Abstract

An image data acquiring part 11 acquires, from an imaging part 40, data of an image in which a user is captured. A skeleton data generating part 12 generates, on the basis of the acquired data of the image, skeleton data of a person in which multiple characteristic points including joint positions in a person region in which the user is captured in the image are connected by a line. A skeleton data transmitting/receiving part 13 acquires skeleton data of an instructor generated by a terminal device 3 used by the instructor. A normalizing part 15 normalizes at least one set of skeleton data such that the size of the skeleton data of the instructor acquired by the skeleton data acquiring part corresponds to the size of the skeleton data of a user, who is being instructed by the instructor, generated by the skeleton data generating part 12. A display controlling part 16 performs control such that the skeleton data of the user and the skeleton data of the instructor having been normalized are overlapped on the image in which the user is captured by the imaging part 40, and the same is displayed on a display part 60.

Description

Terminal device and form improvement support method

The present invention relates to a terminal device for improving a form of sports or fitness, and a form improvement support method.

In sports such as baseball, tennis and golf, as well as fitness such as yoga and aerobics, it is important to obtain the correct form. For the purpose of acquiring a correct form, a method has been proposed in which a subject is equipped with a motion sensor, locus information of a subject model is generated, and the locus information of a reference model is compared with the locus information (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2012-12579

The above method requires multiple motion sensors and a dedicated data acquisition system. Further, in the above-mentioned method, it is necessary to give feedback to the subject after analyzing the form data acquired from a plurality of motion sensors, and it is difficult to give feedback in real time during the practice of the form.

This embodiment was made in view of such a situation, and the purpose is to provide a technique capable of providing support for improving a form in real time with a simple system configuration.

In order to solve the above-mentioned problems, the terminal device of an embodiment of the present embodiment has an image data acquisition unit that acquires image data of a user reflected from an imaging unit, and an image in an image based on the acquired image data. A skeletal data generation unit that generates skeletal data of a person connecting a plurality of feature points including joint positions in the person area in which the user is reflected, and a skeletal structure of the instructor generated by a terminal device used by the instructor. The size of the skeletal data acquisition unit for acquiring data, the skeletal data of the instructor acquired by the skeletal data acquisition unit, and the skeletal data of the user instructed by the instructor generated by the skeletal data generation unit Correspondingly, the normalization unit that normalizes at least one of the skeletal data, and the image of the user captured by the imaging unit are normalized to the skeletal data of the instructor and the skeletal data of the user. Is provided with a display control unit for controlling the display on the display unit by superimposing the data on the display unit.

Another aspect of this embodiment is a form improvement support method. In this method, a step of acquiring image data of a user imaged from an imaging unit and a plurality of feature points including joint positions of a person area in the image in which the user is imaged are obtained based on the acquired image data. The step of generating the skeleton data of the person connected by the line, the step of acquiring the skeleton data of the instructor generated by the terminal device used by the instructor, the acquired skeleton data of the instructor, and the generated relevant The step of normalizing at least one of the skeletal data and the image of the user captured by the image pickup unit were normalized so that the size of the skeletal data of the user being instructed by the instructor corresponds. It has a step of superimposing the skeleton data of the instructor and the skeleton data of the user and controlling the display unit to display the skeleton data.

Yet another aspect of this embodiment is a form improvement support program. This program acquires image data of the user from the image pickup unit, and based on the acquired image data, a plurality of feature points including joint positions of the person area in which the user is reflected in the image. The process of generating the skeletal data of the person connected by the line, the process of acquiring the skeletal data of the instructor generated by the terminal device used by the instructor, the acquired skeletal data of the instructor, and the generated corresponding The process of normalizing at least one of the skeletal data and the image of the user captured by the image pickup unit are normalized so that the size of the skeletal data of the user instructed by the instructor corresponds to the size of the skeletal data. The computer is made to execute a process of superimposing the skeletal data of the instructor and the skeletal data of the user and controlling the display on the display unit.

It should be noted that any combination of the above components and the conversion of the expression of the present embodiment between methods, devices, systems, recording media, computer programs, etc. are also effective as aspects of the present embodiment.

According to this embodiment, it is possible to support the improvement of the form in real time with a simple system configuration.

It is a figure which shows the structure of the form improvement support system which concerns on embodiment. It is a figure which shows an example of the selection screen displayed on the display part of the terminal apparatus used by an instructor. 3 (a)-(b) are diagrams showing an image of the instructor displayed on the display unit of the terminal device used by the instructor and an example of the skeleton data of the instructor. FIGS. 4 (a)-(c) show the normalization of the received instructor's skeleton data, the image of the student displayed on the display of the terminal device used by the student, and the student's skeleton data. It is a figure which shows an example of comparison with the skeleton data of a leader. It is a flowchart which shows the flow of the form improvement support process using the terminal apparatus which concerns on embodiment.

FIG. 1 is a diagram showing a configuration of a form improvement support system 1 according to an embodiment. The form improvement support system 1 includes a plurality of terminal devices 3, and the plurality of terminal devices 3 are connected via the network 2. One of the plurality of terminal devices 3 is used by the instructor, and the other is used by the student who receives the instruction from the instructor. As used herein, an instructor is a person who can instruct a student in the correct exercise form in a particular discipline, including instructors, coaches, and school teachers.

When the instructor and the student are in the same place (studio, etc.), the plurality of terminal devices 3 are connected by a LAN (Local Area Network). If the student and the instructor are in the same place, the student can receive direct instruction from the instructor, but if there are many students, the instructor's instruction may not be sufficient. Also, it may be difficult to see the correct form of the instructor from behind. By using the terminal device 3, students can intuitively and accurately grasp the difference between their own form and the form of the instructor who should be a model, and efficiently acquire the correct form. Can be done.

When the instructor and the student are geographically separated, the plurality of terminal devices 3 are connected by a wide area network such as the Internet or a dedicated line. In this case, the student can receive the guidance of the instructor remotely while staying at home or in the office. A popular instructor at the headquarters can also remotely instruct students in a satellite studio.

FIG. 1 is a diagram assuming an example in which the instructor and the student are 1: N (N is an integer of 2 or more), but the instructor and the student may have an N: N relationship. In that case, the leaders may be geographically separated from each other. In the case of individual instruction, the instructor and the student have a 1: 1 relationship.

The terminal device 3 may be a dedicated terminal device for supporting the improvement of the exercise form, or may be a general-purpose terminal device such as a tablet, a notebook PC, or a smartphone. In the following explanation, a dedicated machine is assumed. For example, when used in a sports club, the terminal device 3 may be rented out to the student from the sports club.

The terminal device 3 includes a processing unit 10, a storage unit 20, a communication unit 30, an imaging unit 40, an operation unit 50, a display unit 60, an audio output unit 70, and a vibration device 80. The processing unit 10 includes an image data acquisition unit 11, a skeleton data generation unit 12, a skeleton data transmission / reception unit 13, an operation reception unit 14, a normalization unit 15, a display control unit 16, an alignment unit 17, a deviation amount calculation unit 18, and a message. The generation unit 19 and the notification control unit 110 are included.

The function of the processing unit 10 can be realized only by the collaboration of hardware resources and software resources, or by hardware resources only. As hardware resources, CPU, GPU, ROM, RAM, DSP (Digital Signal Processor), ISP (Image Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SoC (System on a Chip) , Other LSIs can be used. Programs such as operating systems, middleware, and applications can be used as software resources.

The storage unit 20 has a non-volatile recording medium (for example, NAND flash memory, HDD (Hard Disk Drive)), and includes a dictionary data holding unit 21. A removable recording medium (for example, an SD card) may be attached to the storage unit 20.

The communication unit 30 performs signal processing for connecting to the network 2. Specifically, it performs signal processing based on communication protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol) and Ethernet (registered trademark). The communication unit 30 can be connected to the network 2 via a router (not shown). The router and the terminal device 3 can be connected by LAN cable or Wi-Fi (registered trademark). The mobile phone network may be used to connect to the network 2.

The communication unit 30 also performs signal processing for short-range wireless communication with a device existing at a short distance (vibration device 80 in this embodiment). As short-range wireless communication, Bluetooth (registered trademark), Wi-Fi, infrared communication, etc. can be used.

The imaging unit 40 is a camera for photographing a student or an instructor who is exercising (hereinafter, both are collectively referred to as a user). It is desirable that the image pickup unit 40 is arranged at a position designated by the user and away from the user. Therefore, the image pickup unit 40 and the main body of the terminal device 3 may be connected by a cable (for example, a USB cable) or wirelessly (for example, Wi-Fi). Further, an external Web camera may be used as the image pickup unit 40.

The image pickup unit 40 includes a solid-state image sensor and a signal processing circuit. For example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) can be used as the solid-state image sensor. The solid-state image sensor converts the incident light into an electrical image signal and outputs it to a signal processing circuit. The signal processing circuit performs signal processing such as A / D conversion and noise removal on the image signal input from the solid-state image sensor, and outputs the image data as digital value to the processing unit 10.

The operation unit 50 has a touch panel and physical buttons, receives a user's operation, and outputs the operation to the processing unit 10. The display unit 60 has a display such as a liquid crystal display, an organic EL display, and a micro LED display, and displays images and characters supplied from the processing unit 10. In this embodiment, it is assumed that a touch panel display integrated with the touch panel function is used.

It is preferable that the size of the display is large so that the user can see his / her own image while exercising. Therefore, a large-screen external display may be connected to the main body of the terminal device 3. For example, a large screen TV or PC monitor may be connected to the main body of the terminal device 3 by using a cable (for example, HDMI (registered trademark) cable, USB cable) or wireless (for example, Wi-Fi). ..

The voice output unit 70 has a speaker and outputs the voice supplied from the processing unit 10. In this embodiment, a voice message for improving the exercise form is output. The voice message may be notified to the user by using a wireless earphone wirelessly (for example, Bluetooth) connected to the main body of the terminal device 3.

The vibration device 80 is a band-type vibration device worn on a user's arm or foot (for example, both wrists and both ankles), and vibrates in response to a signal supplied from the processing unit 10. The main body of the terminal device 3 and the vibration device 80 are wirelessly (for example, Bluetooth, Wi-Fi) connected. The vibration device 80 is optional and can be omitted.

The image data acquisition unit 11 acquires image data showing the entire body of the user from the image pickup unit 40. Based on the acquired image data, the skeleton data generation unit 12 is a person who connects a plurality of feature points (also referred to as key points) including a plurality of joint positions in a person area in which the user is shown in the image with a line. Generate skeleton data for. The skeleton data generation unit 12 can generate skeleton data using an existing posture estimation model of a person (for example, OpenPose). The skeleton data generation unit 12 generates skeleton data of a person with a plurality of feature points extracted from the person image and coordinate data defining the relationship between the feature points.

The operation reception unit 14 accepts the user's operation. The operation reception unit 14 receives input and change of various setting information. For example, as setting information, selection of instructor mode or student mode and setting information of a communication destination (for example, IP address, port number) are accepted. It also accepts inputs such as name, gender, age, identification number uniquely assigned to each user, and body shape information. Accepts height and weight inputs as body shape information. Further, as auxiliary information of body shape information, input such as shoulder width, waist width, arm length, and leg length may be accepted. This makes it possible to estimate the length and width of each part of the person reflected in the image with high accuracy.

The operation reception unit 14 accepts the selection of the exercise item, the selection of the form to be acquired in the selected item, and the selection of the joint position to be the reference position based on the operation of the instructor.

FIG. 2 is a diagram showing an example of a selection screen displayed on the display unit 60 of the terminal device 3 used by the instructor. Prior to or during instruction to the student, the instructor selects an item from the item selection field 61, selects a form from the form selection field 62, and selects a joint from the reference position selection field 63.

In the form selection field 62, a plurality of forms of the item selected in the item selection field 61 are displayed as selection candidates. In the example shown in FIG. 2, since yoga is selected in the item selection field 61, the form selection field 62 includes a cat pose, a cow face pose, ..., a board pose, a warrior pose A, and so on. Warrior pose B, ... is displayed as a selection candidate. In the example shown in FIG. 2, the warrior pose A is selected in the form selection field 62. The instructor selects a new pose each time the pose changes.

For example, when baseball is selected in the item selection field 61, a plurality of patterns of pitching form, a plurality of patterns of batting form, a plurality of patterns of catching form, ... Are displayed as selection candidates in the form selection field 62.

The instructor selects a joint to be the reference position of the form selected in the form selection field 62 from the reference position selection field 63. In the example shown in FIG. 2, the hip joint is selected. The selected reference position is used as a reference point when collating the skeletal data of the instructor with the skeletal data of the student.

In the case of a stationary form (pose), the joint closest to the center of gravity of the body may be selected as the joint that should be the reference position for each form. In the case of a moving form, a joint that serves as a fulcrum of the moving part may be selected. For example, in the case of a baseball pitching form, the shoulder joint may be selected. The instructor can select the joint that is judged to be optimal, regardless of the center of gravity of the body or the fulcrum of movement, in consideration of the visibility when collating the skeletal data.

The joint that should be the reference position for each form may be preset. In this case, when a form is selected from the form selection field 62 by the instructor, the joint associated with the selected form is automatically determined. In this case, the instructor does not have to select the joint that should be the reference position.

The dictionary data holding unit 21 in the storage unit 20 uses a classifier generated by machine learning a large number of skeletal data based on a large number of images showing the posture and movement of the instructor as dictionary data for each form of each item. Hold. In the dictionary data of each form of each item, the joint position to be the reference position of the form may be registered in advance. In this case, the above-mentioned reference position can be automatically determined.

In the case of the instructor mode, the skeleton data transmission / reception unit 13 transmits the skeleton data and the reference position information of the instructor generated by the skeleton data generation unit 12 to the student terminal device 3 via the network 2. In the student mode, the skeleton data transmission / reception unit 13 receives the skeleton data and the reference position information of the instructor transmitted from the instructor's terminal device 3 via the network 2.

The display control unit 16 superimposes the skeleton data of the user on the image of the user captured by the image pickup unit 40 and displays it on the display unit 60.

3 (a)-(b) are diagrams showing an image showing the instructor Pi displayed on the display unit 60 of the terminal device 3 used by the instructor, and an example of the skeleton data Si of the instructor Pi. Is. As shown in FIG. 3A, the display control unit 16 superimposes and displays the skeleton data Si generated based on the image data of the area on the image area in which the instructor Pi is reflected in the image. The skeleton data Si changes in real time following the movement of the instructor Pi in the image.

The above-mentioned instructor selects a joint to be a reference position by touching a specific feature point of the skeleton data Si displayed on the touch panel display that integrally constitutes the display unit 60 and the operation unit 50. It may be done by. For example, the hip joint can be selected by touching the point A in FIG.

Return to Fig. 1. The process of the processing unit 10 described below is a process that is executed in the student mode and not executed in the instructor mode. The normalization unit 15 generates at least one skeleton data so that the size of the skeleton data of the instructor received by the skeleton data transmission / reception unit 13 and the size of the student's own skeleton data generated by the skeleton data generation unit 12 correspond to each other. Normalize. In this embodiment, the normalization unit 15 normalizes the instructor's skeleton data so that the received instructor's skeleton data fits the size of the student's own skeleton data. In this case, it is possible to prevent the fitting of the student's image displayed on the display unit 60 of the terminal device 3 used by the student and the student's skeleton data from being misaligned.

The scale factor when normalizing the skeletal data of the instructor may be determined based on the ratio of the height of the instructor to the height of the student, or the size of the image area in which the instructor is shown and the student are shown. It may be determined based on the ratio of the size of the image area. The skeleton data of the standard body shape may be prepared in advance, and the normalization unit 15 may normalize the skeleton data of the instructor and the skeleton data of the student according to the skeleton data of the standard body shape.

The alignment unit 17 aligns the normalized skeleton data of the instructor with the skeleton data of the student. Specifically, the alignment unit 17 determines the reference position of the student's skeleton data based on the reference position information received from the instructor's terminal device 3, and sets the instructor's skeleton data reference position at the reference position. Place the leader's skeletal data to match.

The reference position of the skeleton data may be specified by image recognition. In the terminal device 3 of the instructor, the image recognition unit (not shown) of the processing unit 10 uses the classifier of each form held in the dictionary data holding unit 21 in the image area in which the instructor is reflected, and is used by the instructor. Detects the exercise form that is being performed. For the detection of the motion form, for example, a Haar-like feature amount, a HOG (Histogram of Gradients) feature amount, an LBP (Local Binary Patterns) feature amount, or the like can be used. The image recognition unit identifies the reference position associated with the specified form. The same process is executed on the student terminal device 3 side.

In this case, it is not necessary to transmit the reference position information from the instructor's terminal device 3 to the student's terminal device 3. Further, in addition to the work of the instructor selecting the joint to be the reference position from the operation reception unit 14, the work of selecting the form can be omitted. Further, if the detection accuracy of each form by image recognition is high, the work of selecting an item can be omitted.

The deviation amount calculation unit 18 calculates the deviation amount between the instructor's skeleton data and the student's skeleton data for each body part. Based on the calculated deviation amount, the message generation unit 19 generates a message for encouraging the student to improve the form in order to bring the student's form closer to the instructor's form. Specifically, the deviation amount calculation unit 18 calculates the direction and distance in which the student should move each joint in order to make the deviation amount of each joint between the instructor's skeleton data and the student's skeleton data zero. .. The message generation unit 19 generates a message including the direction and distance in which the student should move each joint. For example, it generates messages such as "wrist 10 cm back" and "right knee 10 cm forward".

The notification control unit 110 controls to notify the student of the generated message by at least one of video, audio, and vibration. When notifying by video, the notification control unit 110 displays a message telop in the screen of the display unit 60. When notifying by voice, the notification control unit 110 causes the voice output unit 70 to output a voice message. When notifying by vibration, the notification control unit 110 notifies the student of the magnitude and direction of the deviation from the instructor's form by the vibration interval or strength of the vibration device 80. For example, the vibration may be increased when the distance from the instructor's form is increased, and the vibration may be decreased when the distance from the instructor's form is increased. In addition, the vibration interval may be shortened as the distance from the instructor's form increases, and the vibration interval may increase as the distance from the instructor's form increases.

FIGS. 4A-(c) show normalization of the received skeleton data of the instructor, an image showing the student Ps displayed on the display unit 60 of the terminal device 3 used by the student, and the student Ps. It is a figure which shows an example of the comparison between the skeleton data of the leader Pi, and the skeleton data Si of the instructor Pi.

FIGS. 3 (a)-(b) and 4 (a)-(c) show an example in which the instructor Pi is taller than the student Ps. Therefore, as shown in FIG. 4A, the normalization unit 15 of the student terminal device 3 reduces the skeleton data received from the instructor's terminal device 3 so as to match the size of the student skeleton data.

As shown in FIG. 4B, the display control unit 16 superimposes and displays the skeleton data Ss of the student Ps generated based on the image data of the area on the image area in which the student Ps is reflected in the image. do. Further, the display control unit 16 has the skeleton data of the instructor Pi normalized so that the reference point A of the skeleton data Ss of the student Ps and the reference point A of the skeleton data Sin of the normalized instructor Pi coincide with each other. Sin is superimposed and displayed on the screen. Further, the display control unit 16 superimposes message telops 64a-64d on the screen in order to encourage the student to improve the form according to the control by the notification control unit 110.

FIG. 5 is a flowchart showing the flow of form improvement support processing using the terminal device 3 according to the embodiment. The user activates the terminal device 3 and sets and registers the user information from the operation unit 50 (S10). The information to be set and registered is whether it is an instructor or a student, a name, an identification number, a height, a weight, and the like. When registered with the instructor authority (S11: Yes), the user sets the transmission destination of the skeleton data from the operation unit 50 (S12). Multiple destinations can be set. The information to be set is the IP address and port number, the student's name, and so on. The user sets the form of the item to be learned and the joint to be the reference position from the operation unit 50 (S13). When registering with student authority in step S11 (S11: No), the processes of steps S12 and S13 are skipped.

Next, the image pickup unit 40 actually photographs one's own figure, and a photography test is performed to see if the skeleton can be estimated normally (S14). When the test is completed, the actual shooting is started (S15). During shooting (S16: Yes), the skeleton data generation unit 12 generates user skeleton data from the shot image (S17). The display control unit 16 superimposes and displays the generated skeleton data on the image area of the image being photographed by the user (S18).

When logged in with the instructor authority (S19: Yes), the skeleton data transmission / reception unit 13 transmits the skeleton data to the set destination terminal device 3 (S20). Transition to step S16.

When the instructor's skeleton data is received from the instructor's terminal device 3 (S21: Yes) when logged in with student authority (S19: No), the normalization unit 15 receives the instructor's skeleton data. , Normalize according to your height (S22). The alignment unit 17 specifies a reference point for the skeleton data of the instructor (S23). Further, the alignment unit 17 specifies a reference point of its own skeleton data (S24). The display control unit 16 superimposes and displays the skeleton data of the instructor so that the reference points match (S25).

The skeleton data generation unit 12 detects the orientation of the student's head from the image data in the image area in which the student is reflected (S26). The deviation amount calculation unit 18 calculates the deviation amount of each joint position between the skeleton data of the instructor and the skeleton data of the student (S27). The message generation unit 19 generates a message for encouraging the student to improve the form based on the amount of deviation of each joint position (S28). The notification control unit 110 notifies the student of the generated message by at least one of video, audio, and vibration (S29). When displaying on the screen, the direction and distance (in cm) in which the joint should be moved are displayed in the vicinity of the joint. The direction in which the joint should be moved is defined by the up, down, left, right, front and back with respect to the direction of the head. Transition to step S16. When the shooting is completed (S16: No), the form improvement support process is completed.

As described above, according to this embodiment, a motion sensor and a dedicated data acquisition system are not required to acquire skeleton data. Since there is no need to attach sensors to the body, both teachers and students can teach or learn the correct form in a more natural way. Since the system can be configured with only a simple device using a camera, the cost of the entire system can be kept low.

Also, while practicing the form, the skeleton data of the instructor and the skeleton data of the student can be compared and fed back to the student in real time. This allows students to quickly master the correct form. By superimposing the skeletal data of the student and the instructor on the form image of the actual student, the difference between the actual body movement and the correct body movement can be easily and intuitively confirmed.

Further, in the terminal device 3 according to the present embodiment, the reference position of the form to be compared can be changed according to the type of exercise. For example, if a baseball pitcher wants to focus on the shoulder-to-shoulder movement of the pitching form, by setting the starting point for comparison to the shoulder, he can concentrate on the shoulder movement without being disturbed by the movement of the entire body. You can learn the form.

Further, in the form improvement support system 1 according to the present embodiment, the skeleton data of the instructor is transmitted to the student terminal device 3 in real time, and the skeleton data comparison processing and display processing are performed by the student terminal device 3. As a result, a plurality of students can smoothly learn the form at the same time without concentrating the load on the instructor's terminal device 3.

Further, in the terminal device 3 according to the present embodiment, both a dynamic form with movement and a static form without movement can be learned. In addition, since the difference in posture between the student and the instructor can be fed back to the student in terms of the part and the quantitative value, the student can easily learn the posture. Feedback to the student can be given via voice or vibration as well as screen display, so the student does not have to stare at the screen and can concentrate on learning posture. Further, by using the terminal device 3 according to the present embodiment, it is possible to instruct students in the same place or a remote place in real time.

In Patent Document 1, the locus information of the motion sensor can be normalized and compared and displayed, but it is difficult for the student to actually imagine the movement of his / her body. Further, in Patent Document 1, it is possible to superimpose and display 3D graphics animation, but it is difficult to compare forms after fixing the position of the superimposing reference to a specific joint part. .. On the other hand, in the present embodiment, in addition to the normalization of the physique, the forms can be compared more accurately by selecting the reference position for comparing the forms according to the type of exercise. Further, in Patent Document 1, it is not considered to compare the form of one instructor with the form of a plurality of students, and it is difficult to realize it due to the system configuration. On the other hand, in the present embodiment, the form of one instructor and the form of a plurality of students can be easily compared.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that this embodiment is an example, and that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. be.

In the above-described embodiment, an example of learning the instructor's form in real time was described. In this regard, the skeleton data of the instructor may be recorded as a moving image and the recorded data may be distributed to the student terminal device 3. In this case, the student can play the instructor's video at his / her favorite time. At that time, it is also possible to specify a specific period of the video and play it. As a method of designation, the student may input the time, or the reproduction period may be designated by automatic detection. For example, the sound of kicking the ball may be automatically detected and the skeleton data for several seconds before and after that may be compared. In addition, the playback period may be automatically set by automatically detecting a specific movement of the instructor (hand gooper, opening the mouth wide, etc.). If the form matches, you can also notify the student. In addition, the playback speed of the instructor's video may be variable. In that case, for example, it is possible to adjust the form in detail by slow playback.

In the above-described embodiment, an example is assumed in which the instructor and the student use the same terminal device 3. In this respect, the instructor may use the terminal device 3 having high specifications. In that case, it is also possible to intensively compare the skeleton data with the instructor's terminal device 3. Further, the skeleton data generation process and the skeleton data comparison process may be performed on the cloud server.

The present invention can be used when instructing sports and fitness form improvement.

1 form improvement support system, 2 network, 3 terminal device, 10 processing unit, 11 image data acquisition unit, 12 skeleton data generation unit, 13 skeleton data transmission / reception unit, 14 operation reception unit, 15 normalization unit, 16 display control unit, 17 alignment unit, 18 deviation amount calculation unit, 19 message generation unit, 110 notification control unit, 20 storage unit, 21 dictionary data retention unit, 30 communication unit, 40 image pickup unit, 50 operation unit, 60 display unit, 70 voice output. Department, 80 vibration device.

Claims

An image data acquisition unit that acquires image data of the user from the image pickup unit, and an image data acquisition unit.
Based on the acquired image data, a skeleton data generation unit that generates skeletal data of a person who connects a plurality of feature points including joint positions in the person area in which the user is shown in the image with a line, and a skeleton data generation unit.
The skeleton data acquisition unit that acquires the skeleton data of the instructor generated by the terminal device used by the instructor,
At least one skeleton data so that the size of the skeleton data of the instructor acquired by the skeleton data acquisition unit and the skeleton data of the user instructed by the instructor generated by the skeleton data generation unit correspond to each other. And the normalization part that normalizes
A display control unit that controls the display unit to superimpose the normalized skeleton data of the instructor and the skeleton data of the user on the image of the user captured by the image pickup unit.
A terminal device.
Further provided with an alignment unit that aligns the normalized skeleton data of the leader with the skeleton data of the user.
The alignment unit aligns the skeleton data of the instructor with the skeleton data of the user with a specific joint position as a reference position.
The terminal device according to claim 1.
Further equipped with an operation reception unit that accepts the selection of an exercise item based on the user's operation.
The alignment unit selects the reference position according to the selected exercise item.
The terminal device according to claim 2.
A message generation unit that generates a message for bringing the user's form closer to the instructor's form based on the amount of deviation between the instructor's skeleton data and the user's skeleton data.
A notification control unit that controls the user to be notified of the generated message by at least one of video, audio, and vibration.
The terminal device according to any one of claims 1 to 3, further comprising.
The step of acquiring the image data of the user from the image pickup unit,
Based on the acquired image data, a step of generating skeleton data of a person connecting a plurality of feature points including joint positions in the person area in which the user is shown in the image with a line, and
Steps to acquire the skeleton data of the instructor generated by the terminal device used by the instructor,
A step to normalize at least one of the skeleton data so that the size of the acquired skeleton data of the instructor and the generated skeleton data of the user instructed by the instructor correspond to each other.
A step of superimposing the normalized skeleton data of the instructor and the skeleton data of the user on the image of the user captured by the image pickup unit, and controlling the display unit to display the normalized skeleton data.
Form improvement support method.