KR20240018998A - Method and system for body pose guiding based on video contents selected by user - Google Patents

Abstract

The present invention relates to a technology for measuring and analyzing the posture of an object (body) including the human body, and also provides a guide that can improve the matching rate of imitation posture compared to model posture through comparison between the posture information. It's about the system. A method of providing a user posture guide using a terminal device according to an embodiment of the present invention for solving the above-described technical problem includes inputting image selection information, and selecting a first video in which the first motion is captured based on the image selection information. Obtaining a first motion associated with the first video, displaying the first video, obtaining a second video in which a second motion imitating the first motion is captured, It may include comparing the first motion and the second motion, and displaying posture guide information based on the comparison.

Description

Method and system for providing object pose guide based on user-selected video {METHOD AND SYSTEM FOR BODY POSE GUIDING BASED ON VIDEO CONTENTS SELECTED BY USER}

The present invention relates to a technology for measuring and analyzing the posture of an object (body) including the human body, and also provides a guide that can improve the matching rate of imitation posture compared to model posture through comparison between the posture information. It's about the system.

So-called home training services, which provide exercise instruction to individuals in remote locations without face-to-face contact, have been operated in the past. This home training service uses experts to film exemplary exercise postures and programs in advance, and then provides a function to encourage service users to develop good exercise habits by imitating the postures on personal information and communication devices such as smartphones. This means that it is provided based on

The most basic home training service method is through demonstration broadcasts showing exercise postures. We utilize television broadcasts, VOD services, streaming services, Internet video sharing services, and other specialized systems to provide exemplary posture and exercise programs from exercise experts (e.g., health trainers, sports trainers, Pilates instructors, yoga instructors, etc.) It provides video content, and allows users watching the video content to learn exercises from a remote location by imitating the video.

However, the user imitates the movements while watching the instructor's exemplary exercise video, but in a situation where the video content is unilaterally provided through a broadcast, the user cannot know whether the user is accurately imitating the movements. Therefore, many of the excellent exercise programs created by the above experts are not properly delivered to users, and sometimes even lead to injuries due to incorrect imitation. In addition, since the user's faithful imitation of the exemplary exercise videos provided by the above service is based on the premise that the user is not lazy, there is a limitation that it is difficult to judge whether it has actually helped the user's health beyond simply confirming the video viewing. .

Background technology for improving these home training services includes technology related to measuring human body posture using computer vision technology. Using posture measurement technology, it is possible to extract information about the three-dimensional shape and posture of the human body from images taken with a general camera without using a special scanner. Therefore, home training services are used by comparing human body posture information extracted from expert's exemplary movements with human body posture information extracted from movements imitated by users. It has the advantage of quantitatively scoring the consistency between postures and providing feedback through automatic or manual means based on this.

However, in conventionally known home training services, providing exemplary posture becomes the sole responsibility of the service provider. That is, after recruiting experts suitable for the user to imitate, such as health trainers, Pilates instructors, or yoga instructors, model movement images are obtained from the experts, and human body posture information is provided for each of the model movement images. After extraction, it must be processed as operation information to function as a service program that users can follow, and must be stored in advance in the service server's storage, etc.

The present invention is intended to overcome the limitations of conventional home training service technology. In the conventionally known home training service described above, the service provider must continuously plan exercise programs, recruit experts in the corresponding exercise program, and film content containing exemplary postures, thereby continuously actively supplying new content. . On the other hand, an unspecified number of service users can only select and perform this.

However, there is a limit to the speed at which content is supplied by the service provider. Therefore, it has the limitation of imposing a heavy operating burden on the service provider. Even if the demand for the service increases and more and more exercise experts try to provide various programs to the users of the service provider, there is a high possibility that a bottleneck phenomenon will occur in which the service provider cannot meet the supply. These limitations inevitably exist as long as conventional home training service technology and service systems utilizing the same are built as a closed service ecosystem with clear distinctions between content providers, service providers, and service users.

In order to overcome the above-mentioned limitations, the present invention provides a novel method for providing a home training service in an open manner by providing a home training service based on a video selected by the user, and an embodiment of implementing the method. begins.

A method of providing a user posture guide using a terminal device according to an embodiment of the present invention for solving the above-described technical problem includes inputting image selection information, and selecting a first video in which the first motion is captured based on the image selection information. Obtaining a first motion associated with the first video, displaying the first video, obtaining a second video in which a second motion imitating the first motion is captured, It may include comparing the first motion and the second motion, and displaying posture guide information based on the comparison.

Obtaining the first video based on the video selection information includes connecting to a server device, receiving interface information from the server device, displaying interface information for inputting the video selection information, Inputting the image selection information according to the interface information and transmitting it to the server, and receiving the first video corresponding to the image selection information from the server.

The video selection information may be included in the interface information and may be information for selecting one of at least one video selection provided from the server device.

The video selection information may include at least one piece of information used by the server device to acquire the first video from a content provider of the first video.

The video selection information includes communication information that identifies the content provider of the first video in a communication network, identification information used by the content provider to identify the first video, and information used to obtain the first video. Communication authentication information including communication protocol information and at least one of an ID, password, and authentication key required to obtain the first video from the content provider; It may be characterized as including at least one of.

Obtaining the first video based on the image selection information includes displaying interface information for selecting at least one image stored in a storage, inputting the image selection information according to the interface information, and It may be characterized by including the step of acquiring the first video corresponding to the video selection information from storage.

It further includes extracting the first motion from the first video by a first motion extraction unit, and extracting the second motion from the second video by a second motion extraction unit, wherein the first The motion and the second motion may be characterized as information related to a motion indicating a change in the posture of an object in chronological order.

At least one of the first motion extraction unit and the second motion extraction unit operates according to a posture extraction algorithm, and the posture extraction algorithm receives a video as an input, outputs a motion, and extracts at least one image frame from the video. generating at least one object joint information based on the at least one image frame, generating at least one object skeleton information based on the at least one image frame, the at least one object joint information Generating at least one object posture information by combining the at least one object skeleton information, and extracting a motion by continuously combining the at least one object posture information.

The pose extraction algorithm further includes normalizing the object pose information, wherein the normalization includes at least one of enlarging, reducing, rotating, inverting, and tilting at least a portion of the object pose information by at least one vector. This may mean standardizing the object posture information by applying a geometric transformation that belongs to one category.

The posture extraction algorithm may be characterized in that at least one step operates by an artificial neural network.

The displaying the first video includes converting object posture information included in the first motion into a posture guide graphic element in the form of a reproduced object, and combining the posture guide graphic element with the first video. It may include a display step.

The first motion extraction unit may operate in a server device.

The step of comparing the first motion and the second motion includes comparing at least one object posture information included in the first motion and at least one object posture information included in the second motion by using a comparison algorithm to obtain at least Obtaining one posture comparison information, wherein the at least one posture comparison information is information representing at least one of a difference vector and a degree of matching of the second motion with respect to the first motion, the posture Guide information may be generated based on the at least one posture comparison information.

The comparison algorithm applies a geometric transformation belonging to at least one of enlargement, reduction, rotation, inversion, and tilt by at least one vector to at least a portion of the object posture information included in the second operation. A normalization step may be included to standardize object posture information.

The step of displaying the posture guide information may include visualizing the posture guide information, overlaying it on at least one of the first video and the second video, and displaying the posture guide information through a display unit of the terminal device. .

The posture guide information may include the step of converting the posture guide information into audio and displaying it through a speaker unit of the terminal device.

In the first video, a plurality of first motions are recorded, and at least one motion including information distinguishing a time when the motion appears in the first video and information distinguishing an object that performed the motion in the first video. It further includes the step of selecting classification information, and the step of obtaining a first motion associated with the first video includes selecting a first motion identified based on the motion classification information among a plurality of first motions associated with the first video. It can be characterized by acquiring only.

A method of providing a user posture guide using a terminal device according to an embodiment of the present invention for solving the above-described technical problem includes inputting image selection information, and selecting a first video in which the first motion is captured based on the image selection information. Obtaining, extracting at least one image frame from the first video, generating at least one object joint information based on the at least one image frame, at least one object joint information based on the at least one image frame Generating one object skeleton information, combining the at least one object joint information and the at least one object skeleton information to generate at least one object posture information, continuously combining the at least one object posture information extracting a first motion, converting object posture information included in the first motion into a posture guide graphic element in the form of a reproduced object, and displaying the posture guide graphic element together with the first video. It may include steps.

A method of providing a user posture guide using a server device according to an embodiment of the present invention for solving the above-described technical problem includes receiving image selection information for a first video from a terminal device, and based on the image selection information. requesting the first video from a content provider, obtaining the first video from the content provider, obtaining a first motion associated with the first video by a posture extraction algorithm, and the first video and It may include transmitting the first operation to the terminal device.

A terminal device that provides a user posture guide according to an embodiment of the present invention for solving the above-described technical problem includes a first input unit that receives image selection information, and an image that acquires the first video based on the image selection information. An acquisition unit, a first processing unit for acquiring a first motion associated with the first video, a second input unit for acquiring a second video in which a second motion imitating the first motion was filmed, a second video associated with the second video. A second processing unit for acquiring a motion, a third processing unit for generating posture guide information by comparing the first motion and the second motion, the first video, the first motion, the second video, the second motion, the It may include a display unit that displays at least one of the posture guide information, a processor that controls the operation of each part, and a memory connected to the processor.

According to the embodiments described below and implementation methods that can be freely modified within the spirit and scope of the present invention, the present invention is not limited by the embodiments and embodiments described below, and when the user uses the home training service, the user is solely responsible for the service provider. It has the effect of providing an open service (so-called “open platform service”) that allows content providers to be freely selected without dependence.

The open service is provided through the Internet, etc., and when the user directly selects and inputs an image to be used for home training, motion extraction based on artificial intelligence, etc. is performed by the open service, and based on the motion extraction, the user is provided with a home training video. It provides an effect that allows training to take place.

Therefore, when a freelance exercise instructor or a professional creator of exercise videos freely uploads an exercise training video, the user can directly select the training video and exercise without any planning or interference from the service provider, thereby providing a variety of exercise content. There are beneficial effects that can be provided by a technical method and such a system that allows suppliers and consumers to be more directly connected, thereby resolving the bottleneck in the content supply speed described above.

Furthermore, the types of videos that the user can point to are not limited, and by using the home training service, not only exercise training videos but also videos that require imitation of choreography, such as K-POP dance videos, can be provided to the user in the same service. It has the effect of providing practice opportunities through motion recognition and comparison.

1 is a conceptual diagram showing a user posture guide providing service using a terminal device according to an embodiment of the present invention;
Figure 2 is a conceptual diagram of the operation of a service for providing a user posture guide according to the first embodiment of the present invention;
3 is an example of an interface that can be displayed on a terminal device based on interface information according to the first embodiment of the present invention;
Figure 4 is a flowchart of object posture information extraction by the motion extraction unit according to an embodiment of the present invention;
5 is an example of a first action selection interface that can be displayed on a terminal device based on interface information according to an embodiment of the present invention;
Figure 6 is a flowchart of service operations according to the first embodiment of the present invention;
Figure 7 is an example of a service interface that can be displayed on a terminal device based on interface information according to the first embodiment of the present invention;
Figure 8 is a conceptual diagram of the operation of a service for providing user posture guide according to the second embodiment of the present invention;
9 is an example of an interface that can be displayed on a terminal device based on interface information according to the second embodiment of the present invention;
Figure 10 is a conceptual diagram of the operation of a service for providing user posture guide according to the third embodiment of the present invention;
11 is an example of an interface that can be displayed on a terminal device based on interface information according to the third embodiment of the present invention;
Figure 12 is a conceptual diagram of the operation of a service for providing user posture guide according to the fourth embodiment of the present invention, and
Figure 13 is a block diagram of a terminal device for providing a user posture guide according to the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the present invention. The term “and/or” includes any combination of a plurality of related stated items or any of a plurality of related stated items and, unless otherwise indicated, is non-exclusive. When items are listed in the present application, it is merely an exemplary description to easily explain the spirit and possible implementation methods of the invention of the present application, and therefore, it is not intended to limit the scope of the embodiments of the present invention.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present application. No.

In describing the invention in this application, embodiments may be described or illustrated in terms of unit blocks that perform the described function or functions. In the present application, the blocks may be expressed as one or more devices, units, modules, units, etc. The blocks may be implemented in hardware using one or more logic gates, integrated circuits, processors, controllers, memories, electronic components, or information processing hardware implementation methods that are not limited thereto. Alternatively, the blocks may be implemented in software by an implementation method of application software, operating system software, firmware, or information processing software that is not limited thereto. One block may be implemented separately into a plurality of blocks that perform the same function, or conversely, one block may be implemented to simultaneously perform the functions of a plurality of blocks. The blocks may also be implemented physically separated or combined by any criteria. The blocks may be implemented to operate in an environment where their physical locations are not specified and are spaced apart from each other by a communication network, the Internet, a cloud service, or a communication method that is not limited thereto. All of the above implementation methods are within the scope of various embodiments that can be taken by a person skilled in the art of information and communication technology to implement the same technical idea, so any detailed implementation methods are all included in the technical idea scope of the invention in this application. It should be interpreted as

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate overall understanding when describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted. Additionally, it is assumed that the plurality of embodiments are not mutually exclusive, and that some embodiments may be combined with one or more other embodiments to form new embodiments.

basic concepts

1 is a conceptual diagram showing a user posture guide providing service using a terminal device according to an embodiment of the present invention.

In the service 100 of an embodiment shown in FIG. 1, the user 110 of the service may be a person who wishes to perform a program consisting of a certain movement with a certain posture continuously from a remote location, for example, a home training service. It could be a person who uses it to exercise.

A model video 120 may be provided (190) to the user (110) of the service. The exemplary video 120 is a film of exemplary movements performed by an expert 125, and is capable of identifying fitness exercise, strength training, yoga, pilates, dance, golf swing, and other postures and following exemplary postures. This may be an exercise training video filmed by an expert (125) demonstrating the necessary physical movements. Therefore, the user 110 using the service is expected to watch 190 the model image 120 through a display device, etc. and exercise by imitating the movements taken by the expert 125 of the model image 120. do.

Model motion information 130 may be generated for the model image 120. The model motion information 130 may include model posture information 135. The exemplary posture information 135 may represent the posture taken by the expert 125 in a computer-readable form.

In a preferred embodiment of the present invention, the model posture information 135 may be composed of absolute or relative position information of at least one specific joint of the human body and directional vector information connecting the joints. In this specification, the position information of the joints is referred to as human joint information, and the directional vector information is referred to as human skeleton information. Additionally, information representing the posture of the human body by combining the human joint information and human skeleton information will be referred to as human body posture information. That is, the exemplary posture information 135 may be in the form of human body posture information.

The exemplary posture information 135 can be obtained by inputting the exemplary image 120 into the first posture extraction algorithm 137. The first posture extraction algorithm is an algorithm for extracting the human body posture information from a given image such as the model image 120, and its implementation method will be described later.

Meanwhile, information on the posture taken by the user may be photographed (115). According to a preferred embodiment of the present invention, the photographing 115 may be performed in real time, but this may not necessarily be the case. As a result of the shooting 115, a user image 140 may be obtained. Additionally, user motion information 150 may be generated for the user image 140. The user motion information 150 may include user posture information 155 obtained by inputting the posture assumed by the user 110 into the second posture extraction algorithm 157.

Depending on the embodiment of the present invention, the first posture extraction algorithm 137 and the second posture extraction algorithm 157 may be the same in content, but may be different as long as the purpose is maintained. Likewise, whether the implementation location and time of the two algorithms 137 and 157 are the same or different does not affect achieving the effect of the present invention.

The exemplary motion information 130 and the user motion information 140 may be compared with each other (160). That is, the user posture information 145 included in the user motion information 140 may be evaluated based on the exemplary posture information 135 included in the exemplary motion information 130. The method of the above comparison will be described later. Through the above comparison, it can be easily seen that the degree of agreement between the posture taken by the expert 125 and the posture taken by the user 110 can be compared.

As a result of the comparison (160), posture guide information may be generated and displayed (165) to the user. The posture guide information may be derived based on the result of calculating the change that must be made to the user motion information 140 in order to conform to the model motion information 130. For example, when the rotation angle of the upper arm of the user 110 is insufficient compared to that of the expert 125, this may be a display method that can be recognized by a person requesting supplementation of the rotation angle of the upper arm, and is not limited to this example.

The user 110, who has received the display 165 of the posture guide information, can recognize what actions to take in order to better imitate the movements of the expert 125 while watching the model video 120 (190). Therefore, the service 100 can be used for the purpose of a home training service according to an embodiment of the present invention.

According to the present invention, a function is provided that allows the user 110 to directly select the exemplary image 120. More specifically, it allows the user to provide image selection information 192 to specify and select the exemplary image. As a result, the present invention can provide the effect of allowing the user 110 to use the service 100 in a manner in which the content provider of the exemplary video 120 is not limited.

First embodiment and its applied embodiment

Hereinafter, a first embodiment, which corresponds to a preferred method of implementing the present invention but does not limit the method of implementing the present invention, will be described. In addition, in implementing the first embodiment, embodiments that can be applied at the discretion of the person skilled in the art will also be described.

Figure 2 is a conceptual diagram of the operation of a service for providing a user posture guide according to the first embodiment of the present invention. The service system 200 shown in FIG. 2 is, for example, for providing a home training service by providing a posture guide according to the present invention, and includes a terminal device 210, a server device 220, and a content provider ( 230).

The terminal device 210 may be an information and communication terminal device. According to the implementation method of the present invention, the terminal device 210 is a terminal device belonging to any one of personal information and communication terminal devices including a smartphone, a tablet computer, a personal computer (PC), a laptop computer, and a smart TV. You can. Additionally, it may be a terminal device capable of communicating information by connecting to the server device 220 through communication means such as IMT-2000, LTE, 5G, Wi-fi, LAN, or short-distance communication.

In the following description of the first embodiment, which does not limit the implementation method of the present invention, the terminal device 210 is a portable information and communication device such as a smartphone operated by a user who wants to use a home training service (hereinafter referred to as service). It is assumed that The user can be considered the same as, for example, the user 110 shown in FIG. 1, and in the description of this embodiment, the user acquires the first video, which is an exercise image of an exercise expert that the user wishes to imitate, and provides the service. Assume you want to use it.

In the following first embodiment, which does not limit the implementation method of the present invention, the server device 220 is a server installed to provide the service, and obtains the first video by receiving a request from the terminal device 210. And, it may be a server configured to provide posture guide information based on the first video. The server device 220 supplies the first video to the terminal device 210 via the World Wide Web as a single file, streaming packet, or a similar digital data exchange method. It may be configured to include. In addition, the server device 220 may be configured to include a function of a first motion extraction unit that extracts motion information from the first video. In relation to the function of the first motion extraction unit, reference numeral 400 of FIG. 2 and FIG. This will be described later in the explanation of 4.

The server device 220 may be an information and communication service server. The server device 220 may be implemented as a single server computer device according to the implementation method of the present invention. However, depending on other implementation methods, there is no problem in achieving the purpose of the present invention even if it is implemented by a plurality of server devices, a cloud server, or a processing process distributed over at least one server and at least one client.

The content provider 230 may be a content provider that mainly focuses on videos. According to the implementation method of the present invention, the content provider 230 is a person who supplies digital video information through a single file, streaming packet, or similar digital data exchange method through the World Wide Web. In particular, it may be regarded as the provider's storage server device installed by such provider to supply the digital video. Of course, depending on the implementation method of the present invention, it is obvious that any implementation method may be replaced as long as it is a means of supplying moving images for the purpose of being used in the core configuration of the present invention in addition to the above-mentioned method.

In the following description of the first embodiment, which does not limit the implementation method of the present invention, the content provider 230 provides video content of an exercise expert used in the service, for example, by the expert 125 shown in FIG. 1. The description will be made assuming that it is a storage server device that holds and supplies content that can be used as a model video 120.

When the user starts the service on the terminal device 210, the terminal device 210 can connect to the server device 220 providing the service (S251). In response to the connection (S251), the server device 220 may provide interface information to the terminal device 210 (S252).

The method of configuring the connection (S251) and the type of interface information provided (S252) are not limited. According to the implementation method of the present invention, the terminal device 210 connects to the server device 220 based on an Internet protocol (IP), for example, by using built-in web browser software (S251), The server device 220 may provide information (S252) related to a web service interface including hypertext language (HTML) that can be displayed in the web browser software in response to the connection (S251). For another example, the terminal device 210 connects to the server device 220 using a unique communication method (S251) using built-in application software, and the server device (S220) connects to the connection (S251). In response, information indicating display of a user interface included in the application software may be provided (S252). In addition, in implementing information and communication services using typical terminal-server application software, various applied implementation methods known or newly invented can be applied.

According to reception of the interface information (S252), the corresponding interface 300 may be displayed on the terminal device 210.

Hereinafter, the description will be made with further reference to FIG. 3. Figure 3 is an example diagram of an interface that can be displayed on a terminal device based on interface information according to the first embodiment of the present invention. In the description of this first embodiment, which does not limit the method of implementing the present invention, the interface 300 may have the purpose of allowing the user to designate a first video to be used for the home training service.

The interface 300 may be displayed through the display 310 of the terminal device 305. The interface 300 includes a function 320 for inputting image selection information for acquisition of the first video, and a function 330 for instructing the server device 220 to execute acquisition of the first video. It can be included. The interface may further include a display item 315 indicating the purpose of the interface, for example, a display item indicating the name of the service. However, the function of the interface 300 is illustrative, and the function of the interface 300 may be added, changed, or removed as long as the technical purpose of the present invention is maintained.

The video selection information may include at least one piece of information used by the server device 220 to obtain the first video from the content provider 230 of the first video. In the description of this first embodiment, which does not limit the implementation method of the present invention, the video selection information is a URL (uniform) used to acquire the first video content held by the content provider 230 by Internet protocol. resource locator).

More specifically, the video selection information may include communication information such as an IP address and domain name used to identify the content provider 230 of the first video in a communication network. The video selection information may include identification information such as a web page address, database ID, and other service identification codes used by the content provider 230 to identify the first video. The video selection information may include information indicating a communication protocol such as http (hypertext markup language) or ftp (file transfer protocol) used to acquire the first video. The video selection information may include communication authentication information including at least one of an ID, password, and authentication key required to obtain the first video from the content provider. The communication authentication information can be used when certain qualifications, such as login or API authentication, are required to acquire the first video from the content provider 230. Therefore, it is clear that even if the communication authentication information is transformed into a set of any conventionally known or newly invented communication authentication information used for the purpose of determining eligibility to receive data using an information and communication network, there will be no problem in achieving the purpose of the present invention. do.

As an example that is easy for those skilled in the art to understand, the image selection information may include an http URL such as "http://www.*******.com/12345678". The communication information is a domain address ("www.*******.com") that points to the content provider in the URL, and the identification code is an additional address ("/12345678") that points to a specific video in the URL. , the information on the communication protocol can be seen as exemplified by the protocol notation (“http://”) of the URL. Additionally, the video selection information may further include a unique authentication key of the video download API permitted by the content provider (i.e., “*******.com”), separately from the URL.

Referring again to FIG. 2, the user inputs the image selection information through the input function 320 of the interface 300 and transmits the image selection information to the server device using the instruction function 330. (S253) You can.

The server device 220 can connect to the server of the content provider 230 (S254) based on the video selection information received (S253) and obtain the first video that the user wanted (S255). . Accordingly, the server device 220 may provide the first video to the terminal device 210 (S256).

In the first video, at least one object, and in some cases, multiple objects, may be captured. In the description of the first embodiment, which does not limit the method of implementing the present invention, the object may be a human body, and therefore the first video is an exemplary video by the expert 125 shown in FIG. 1 as described above. It may be (120).

Using the first video, the server device 220 may be configured to extract (400) a first motion from the first video by a first motion extraction unit. According to one embodiment of the present invention, the first motion extraction unit may include at least one artificial intelligence model, such as machine learning or artificial neural network, previously trained through supervised learning or unsupervised learning. According to an embodiment of the present invention, the artificial intelligence model may be implemented as a convolutional neural network (CNN). The first motion extraction unit may operate depending entirely or partially on the at least one artificial intelligence model.

Hereinafter, the description will be made with reference to FIG. 4. Figure 4 is a flowchart of object posture information extraction by the motion extraction unit according to an embodiment of the present invention. The motion extraction unit receives a video such as the first video as an input (S410), separates the video into frames to obtain at least one video frame (S420), and extracts at least one human body from each video frame. Identify the object (body) including (S430), analyze the identified object to generate at least one object joint information (S440), and analyze the identified object to generate at least one object joint information based on the at least one image frame. One object skeleton information may be generated (S450), and object posture information may be generated by combining the at least one object joint information and the at least one object skeleton information (S460).

In a more applied embodiment 495 of the present invention, the object pose information may be generated and then normalized (S490). The normalization may mean standardizing the object posture information by applying a geometric transformation of at least one of enlargement, reduction, rotation, inversion, and tilt to at least a portion of the object posture information by at least one vector.

The normalization may be intended to correct the variability due to the size of the object, that is, the photographed human body, and the variability due to the photographing method when the object is photographed in the input video. For example, the normalization may have the purpose of offsetting so-called Rotate, Scaling, and Transform (RST) changes in the object posture information, such as rotation, enlargement, reduction, and angle change. Additionally, the normalization may include a process of geometrically transforming the object posture information and converting it into human body posture information with standardized arm lengths and leg lengths. For another example, the normalization calculates the deformation in the By doing so, it may include a process of modifying the alignment to a reference point according to at least one of the X, Y, and Z axes.

The normalization may be performed by including at least one geometric transformation function including a rigid transform, an affine transform, and a projection transform to achieve the above purpose.

In addition to the above-described implementation method, the normalization may be configured to include any information modification process to standardize the object posture information and process it so that it can be easily used for the purpose of the present invention.

The object posture information may be repeatedly generated for each successive frame (S465). At least one object posture information generated for the object may be continuously combined over time and used to generate object motion information for the object (S470).

The object motion information may be output (S480) as a result of processing by the motion extraction unit. Accordingly, the object motion information may be regarded as the first motion extracted by the first motion extractor from the first video.

Referring again to FIG. 2, the first motion extracted by the first motion extraction unit 400 is, as described above, the object appearing in the first video, that is, the motion representing the change in posture of the expert in chronological order. May contain relevant information. If the first action is extracted, the first action may be transmitted from the server 230 to the terminal device 210 (S259).

The service system 200 according to the first embodiment of the present invention can be configured so that two or more objects can be captured in the first video, and thus two or more first actions can be extracted from the first video. . In this case, operations 201 corresponding to the first applied embodiment of the present invention may be further included.

In the first applied embodiment of the present invention, which does not limit the implementation method of the present invention, the first motion extraction unit 400 may be configured to extract the first motion for each of the plurality of objects. . In order to generate a plurality of first operations for the plurality of objects, any step in the processing flowchart of the first motion extraction unit 400 shown in FIG. 4 may be configured to be repeatedly executed on an object basis. , It may also be configured to extract a plurality of first movements from a plurality of objects using other implementation methods.

The plurality of first motions may be distinguished by motion classification information including information distinguishing the time at which the motion appears and information distinguishing the object that performed the motion in the first video. For example, the motion classification information may be information configured to point out a specific exercise expert that appears at a specific time among several exercise experts that appear in the first video.

In the first applied embodiment of the present invention, which does not limit the implementation method of the present invention, the service 200 may allow the user to select one of the plurality of first operations. Accordingly, the server device 220 may provide the terminal device with at least one motion classification information that distinguishes a plurality of first motions included in the first video (S257).

This will be explained further with reference to FIG. 5 . Figure 5 is an exemplary diagram of a first action selection interface that can be displayed on a terminal device based on interface information according to an embodiment of the present invention. The interface 500 may have the purpose of allowing the user to select motion classification information that points out a specific first motion to be taken as an object of imitation among the at least one motion classification information provided (S527).

The interface 500 may be displayed through the display 310 of the terminal device 305. According to one embodiment of the present invention, the interface includes a function 520 for displaying the first video, a play bar 525 or a time indicator 526 for searching the first video in the time direction, and The same function, a selection cursor 527 function provided to select one of a plurality of objects shown in the first video, and a function to confirm the selection of motion classification information according to the time and object selection by the above functions It may include (530). The interface may further include an indication 515 indicating the purpose of the interface. However, the function of the interface 500 is illustrative, and the function of the interface 500 may be added, changed, or removed as long as the technical purpose of the present invention is maintained.

According to the interface 500 shown in FIG. 5, the user checks the appearance of an exercise expert performing the desired first exercise while viewing the first video through the play bar 525, and then clicks on the exercise expert. By doing so, the object identified for the exercise expert can be selected, and as a result, one motion classification information according to the time and object selection can be selected to select the first exercise to be imitated.

However, the above-described embodiment does not limit the implementation method of the present invention, and the implementation method of the interface 500 may be changed in various ways. For example, even if the interface 500 is implemented to provide a list listing the plurality of operation classification information provided (S527) in the form of a scroll list or drop-down list, It's okay. Additionally, it is clear that no matter what other interface is provided, there is no problem in achieving the purpose of the present invention as long as it has the function of selecting operation classification information.

Information related to selecting the operation classification information in the terminal device 210 may be transmitted to the server device 220 (S258). Depending on the embodiment, the transmitted information (S258) may be an index that identifies one of the plurality of operation classification information, or may be the selected operation classification information itself. The server device 220 will be configured to select only the first operation identified based on the selected operation classification information (S270) and transmit information related to the selected first operation to the terminal device 210 (S259). You can.

By the operation procedure shown in FIG. 2, the terminal device 210 can acquire the first video (S256) and obtain information about the first operation (S259). Accordingly, the terminal device 210 may be configured to perform the service operation 600 based on this.

Figure 6 is a flowchart of service operations according to the first embodiment of the present invention. Additionally, Figure 7 is an example diagram of a service interface that can be displayed on a terminal device based on interface information according to the first embodiment of the present invention. The following description will be made with cross-reference to the two drawings.

In the description of the first embodiment, which does not limit the implementation method of the present invention, the service interface 700 includes the home training service operation to help the user successfully imitate the first movement appearing in the first video. It may have the purpose of implementing (600).

The interface 700 may be displayed through the display 310 of the terminal device 305. The interface 700 includes a function for displaying the first video (710), a function for displaying the first motion (720), a function for displaying the second video (730), and a function for displaying the second motion (740). ), and may include a function 750 to display a posture guide. In a preferred embodiment of the present invention, the user takes a second action imitating the first action while observing the display 710 of the first video and the display 720 of the first action, and further 2 You can check your own appearance in the second motion through the user's shape 735 and the display 740 of the second motion shown in the display 730 of the second video, and the display 750 of the posture guide. Through this, information that can more successfully imitate the first motion can be obtained.

However, the function of the interface 700 is illustrative, and the function of the interface 700 may be added, changed, or removed as long as the technical purpose of the present invention is maintained. Additionally, at least one photographing means 760 may be installed in the terminal device 305 that displays the interface 700.

The acquired first video may be displayed to the user. Additionally, according to an embodiment of the present invention, the first video may be displayed together with the first motion (S610). The first video may be displayed through the first video display function 710 of the interface 700, and the first motion may be displayed through the first motion display function 720 of the interface 700.

The display of the first motion may be accomplished by a procedure including converting object posture information included in the first motion into a graphic element in the form of a reproduced object. For example, when the object posture information of the present invention consists of object joint information and object skeleton information as in the above-described embodiment, as shown in the first motion display function 720 of FIG. 7, the object joint Information and the object skeleton information may be visualized (725) and provided to the user.

The first video display function 710 and the first motion display function 720 may be separated from each other on the interface 700, or, depending on the embodiment, some or all of them may overlap with each other. For example, in a modified embodiment of the present invention, the first motion display function 720 may be displayed in an overlay form on the first video display function 710.

According to the display of the first video and the first motion, the user can observe and attempt to imitate the first motion of the expert 715 captured in the first video. Displaying the first motion together has the beneficial effect of helping the user more smoothly imitate the first motion appearing in the first video.

In this specification, the action taken by the user by observing and imitating the first action will be referred to as the second action. The second motion of the user imitating the first motion may be captured by the photographing means 760 to generate a second video (S620). Depending on the embodiment, the photographing means 760 may mean a camera attached to the terminal device 305. However, depending on other embodiments, any photographing means that can be connected internally or externally to the terminal device 305 and wired or wirelessly may be used without affecting the achievement of the purpose of the present invention.

The second motion may be extracted from the second video by a second motion extraction unit (S630). As described above, the second motion extracted by the second motion extractor may include information related to the motion representing the change in posture of the object appearing in the second video, that is, the user 735, in chronological order. . In the description of this first embodiment, which does not limit the implementation method of the present invention, the second motion extraction unit may operate in the terminal device 305. According to this embodiment, the second video captured by the photographing means 760 may be input to the second motion extraction unit within the terminal device 305.

According to a preferred embodiment of the present invention, the operating method of the second motion extraction unit may be the same as that of the first motion extraction unit. Accordingly, all embodiments of the first motion extraction unit described with reference to FIG. 4 can be equally used in the second motion extraction unit. However, according to another embodiment of the present invention, the second motion extraction unit may be implemented in a similar but different structure from the first motion extraction unit. For example, when the first motion extraction unit is run in a server device, it may be difficult to implement the same motion extraction function in the terminal device, so the second motion cannot be extracted in the same data format as the first motion. Different implementation methods may be applied to the second motion extraction unit.

Additionally, according to another embodiment of the present invention, the second motion extraction unit may not operate in the terminal device, but may be configured to operate in a remote location such as the server device. In this case, the steps of transmitting the second video from the terminal device to the server device to extract the second motion from the second video and receiving information related to the second motion from the server device are: It can be added to the results of the invention. Furthermore, depending on the embodiment, the first motion extraction unit and the second motion extraction unit may mean one functional unit that operates with different inputs.

When the first motion and the second motion are obtained, the two motions can be compared to generate posture guide information (S640). As described above in the description with reference to FIG. 1, the comparison of the two motions is based on the object posture information included in the first motion corresponding to the exemplary motion of the expert 715, and the imitation motion of the user 735. This can be done by evaluating object posture information included in the corresponding second motion.

In the description of this first embodiment, which does not limit the implementation method of the present invention, the comparison may be made by comparing the first operation and the second operation on a frame-by-frame basis. More specifically, in each frame, object joint information and object skeleton information constituting the object posture information of the first motion are expressed as first vector information, and object joints constituting the object posture information of the second motion are expressed as first vector information. This can be done by representing the information and the object skeleton information as second vector information and then calculating the difference between the second vector information based on the first vector information.

In obtaining the difference between the first vector information and the second vector information, the difference can be obtained by dividing the information into joint units or skeleton units. For example, by comparing a first detail vector representing skeletal information corresponding to the upper arm among the object posture information of the first motion and a second detail vector representing the same information in the second motion, the user It can be derived how much the second action of 735 differs from the first action of the expert 715.

The calculation of the difference between the vector information may be performed in a pre-specified form by an algorithm, or may be performed in a way that obtains the effect of performing such calculation as a result through the application of advanced information processing functions such as artificial neural networks, In addition, there is no problem in achieving the purpose of the present invention even if any previously known or newly developed calculation technology for calculating the difference between object posture information is applied.

Before the difference calculation is performed, the object pose information included in the second operation may be normalized. The normalization may mean standardizing the object posture information by applying a geometric transformation of at least one of enlargement, reduction, rotation, inversion, and tilt to at least a portion of the object posture information by at least one vector.

The normalization may be intended to correct the variability due to the size of the object, that is, the photographed user's human body, and the variability due to the photographing method when the object is photographed in the second video. In particular, since the environment in which the first video is filmed and the environment in which the second video is filmed are different, the second motion imitating the first motion by correcting this difference is caused by external factors, such as the user and the expert. (715) To suppress unexpected differences between the two due to differences in body size, height, joint range of motion, distance from the imaging means, lens angle of the imaging means, resolution of the imaging means, etc. You can have a purpose.

For example, the normalization may have the purpose of offsetting so-called Rotate, Scaling, and Transform (RST) changes in the object posture information, such as rotation, enlargement, reduction, and angle change. Additionally, the normalization may include a process of geometrically transforming the object posture information and converting it into human body posture information with standardized arm lengths and leg lengths. For another example, the normalization calculates the deformation in the By doing so, it may include a process of modifying the alignment to a reference point according to at least one of the X, Y, and Z axes.

The normalization may be performed by including at least one geometric transformation function including a rigid transform, an affine transform, and a projection transform to achieve the above purpose.

In addition to the above-described implementation method, the normalization may be configured to include any information modification process to standardize the object posture information and process it so that it can be easily used for the purpose of the present invention.

The difference between the first operation and the second operation is, according to an embodiment of the present invention, the degree of coincidence between the first vector and the second vector and the difference (差分) of the second vector based on the first vector. ) can be derived as posture comparison information including at least one of the vectors. Based on the posture comparison information, posture guide information can be generated (S660).

In the description of the first embodiment, which does not limit the method of implementing the present invention, the posture guide information will be helpful in guiding the user's second motion to be closer to the expert's 715's first motion. It can be derived in various forms of information. For example, the posture guide information may include information indicating in which direction and how a specific body part should be further moved when the user takes the second motion based on the difference vector. For another example, the posture guide information may include information visualizing the difference vector between the first motion and the second motion using an indicating means such as an arrow. For another example, the posture guide information may include information evaluating the movement rate of a specific body part of the user. As another example, the posture guide information indicates that the degree to which a specific body part of the user matches the first movement is different compared to other body parts, especially symmetrical body parts (for example, the left arm and the right arm). It may contain information that it represents. For another example, the posture guide information may include statistical information that the degree of agreement between the user and the second movement imitating a specific type of first movement is low. In addition, it is obvious that various guide information that can be obtained through comparison of the first and second movements can be used as posture guide information within the scope of the present invention by conventionally known or newly developed exercise assistance methods. do.

The posture guide information can be displayed through the interface 700, and according to a preferred embodiment of the present invention, it can be displayed together with the second video and the second motion (S670). The second video is displayed through the second video display function 730 of the interface 700, and the second motion is displayed through the second motion display function 740 of the interface 700. The posture guide information Can be displayed through the posture guide display function 750 of the interface 700.

The display of the second motion is, similar to the first motion, by a procedure comprising converting the object posture information included in the second motion into a graphic element in the form of a reproduced object, for example This can be done by visualizing (745). An embodiment of the visualization may use the visualization 725 for displaying the first operation.

The second video display function 730 and the second motion display function 740 may be separated from each other on the interface 700, or, depending on the embodiment, some or all of them may overlap with each other. For example, in a modified embodiment of the present invention, the second motion display function 740 may be displayed in an overlay form on the second video display function 730.

The posture guide display function 750 may be displayed at any position on the interface 700. For example, the posture guide display function 750 may be displayed adjacent to or overlaps with the second motion display function 740. However, in a modified embodiment of the present invention, the posture guide display function 750 may be displayed in the form of a splash message that is temporarily overlaid on the entire interface 700, and may also be displayed in the form of a splash message that is temporarily overlaid on the entire interface 700. Guide information may be displayed to the user in various forms in order to explain detailed implementation methods and improve the user's service experience.

In a modified embodiment of the present invention, the posture guide display function 750 may be implemented in a form that does not occupy the display 310. For example, the posture guide display function 750 may be included in the voice effect of the interface 700 and configured to be displayed through a speaker unit of the terminal device.

A function for displaying the first video through the interface 700 (710), a function for displaying the first motion (720), a function for displaying the second video (730), a function for displaying the second motion ( According to a preferred implementation method of the present invention, the functions 740) and the function 750 for displaying the posture guide can be substantially simultaneously combined into one display image and displayed on the display 305. After the first video is displayed (710), the time interval until an image of the user observing the first video and making a second motion imitating the first motion is captured and displayed as a second video (730) Additionally, the time interval required to make a comparison between the first operation and the second operation may be neglected, depending on the performance of the information and communication device used to implement the present invention and other implementation method factors. . In addition, it is obvious that time delays occurring during computation or communication can be responded to using commonly known implementation technologies for information and communication applications.

Second embodiment

Hereinafter, a description will be given regarding the second embodiment of the present invention, which is derived by changing the implementation method from the first embodiment.

Figure 8 is a conceptual diagram of the operation of a service for providing a user posture guide according to the second embodiment of the present invention. The service system 800 shown in FIG. 8 is for providing a home training service by providing a posture guide according to the present invention, for example, and is configured to include a terminal device 810 and a server device 820. You can.

In the description of FIG. 8, items sharing the same symbols as those of FIG. 2 may refer to the first embodiment.

In the following second embodiment, which does not limit the implementation method of the present invention, the content provider 230 in the first embodiment may not be separately provided. The server device 820 is a server installed to provide the service, and also has a content storage that stores digital video information that can be used as at least one first video, and stores the digital video information in a single file via the World Wide Web. It may be configured to include a function to supply by file or streaming packet or similar digital data exchange method. Additionally, the server device 820 may be configured to include the function of the first motion extraction unit of the first embodiment.

When the user starts the service on the terminal device 810, the terminal device 810 can connect to the server device 820 providing the service (S251). In response to the connection (S251), the server device 820 may provide interface information to the terminal device 810 (S852). According to reception of the interface information (S852), the corresponding interface 900 may be displayed on the terminal device 810.

Hereinafter, the description will be made with further reference to FIG. 9. Figure 9 is an example diagram of an interface that can be displayed on a terminal device based on interface information according to the second embodiment of the present invention. In the description of the second embodiment, which does not limit the implementation method of the present invention, the interface 900 may have the purpose of allowing the user to designate a first video to be used for the home training service.

The interface 900 may be displayed through the display 310 of the terminal device 305. The interface 900 includes a function 920 for providing a list of candidate videos stored in the server device 820 and permitted to be used by the user, and selecting at least one video from among the candidate videos as the first video. It may include a function 930 that instructs the server device 820 to transmit image selection information, which is information about what has been done. The interface may further include a display item 315 indicating the purpose of the interface, for example, a display item indicating the name of the service. However, the function of the interface 900 is illustrative, and the function of the interface 900 may be added, changed, or removed as long as the technical purpose of the present invention is maintained.

The list 920 of the candidate videos may be provided in the form of a scroll list or drop-down list with a scroll function 922 within the interface 900. The user can search for candidate videos of the first video available through the server device 820 in the service 800 through the list 920, and select at least one of them (921). You can enter video selection information.

The user can use the instruction function 930 to cause the terminal 220 to transmit the video selection information to the server device 820 (S853), and the server device 820 responds accordingly. 1 A video can be provided to the terminal device 810 (S256).

In this second embodiment, the implementation method after providing the first video (S256) can use the same contents described in the first embodiment and its modifications. The embodiment of the service operation 600 described above with reference to FIG. 6 may also be used in the same manner. Additionally, the embodiment 201 applied to correspond to a plurality of objects in the first video from the first embodiment can also be combined with the second embodiment in the same way.

Third embodiment

Hereinafter, a description will be given regarding the second embodiment of the present invention, which is derived by changing the implementation method from the above second embodiment.

Figure 10 is a conceptual diagram of the operation of a service for providing user posture guide according to the third embodiment of the present invention. The service system 1000 shown in FIG. 10 is for providing a home training service by providing a posture guide according to the present invention, for example, and is configured to include a terminal device 1010 and a server device 1020. You can.

In the description of FIG. 10, items sharing the same symbols as those of FIG. 2 may refer to the first embodiment.

In the third embodiment below, which does not limit the implementation method of the present invention, the first video can be provided from the terminal device 1010. The server device 1020 is a server installed to provide the service, and may be configured to include the function of the first motion extraction unit of the first embodiment.

When the user starts the service on the terminal device 1010, the terminal device 1010 can connect to the server device 1020 providing the service (S251). In response to the connection (S251), the server device 1020 may provide interface information to the terminal device 1010 (S1052). According to reception of the interface information (S1052), the corresponding interface 1100 may be displayed on the terminal device 1020.

Hereinafter, the description will be made with further reference to FIG. 11. Figure 11 is an example diagram of an interface that can be displayed on a terminal device based on interface information according to the third embodiment of the present invention. In the description of this third embodiment, which does not limit the implementation method of the present invention, the interface 1100 may have the purpose of allowing the user to directly provide the first video that the user wishes to use for the home training service.

The interface 1100 may be displayed through the display 310 of the terminal device 305. The interface 900 includes a function 1120 for providing a list of videos stored in the terminal device 1010, and selecting an image including at least one video information determined to be used as the first video among the videos. It may include a function 1130 that instructs information to be transmitted to the server device 1020. The interface may further include a display item 315 indicating the purpose of the interface, for example, a display item indicating the name of the service. However, the function of the interface 1100 is illustrative, and the function of the interface 1100 may be added, changed, or removed as long as the technical purpose of the present invention is maintained.

The user can search for candidate videos of the first video to be used to use the service 1000 through the list 1120 and select at least one of them (1121). The user can use the instruction function 1130 to cause the terminal device 1010 to transmit the image selection information to the server device 1120 (S1053), and the server device 1020 selects the image. The first video can be received from the terminal device 1010 through information (S1053). Therefore, unlike other embodiments described above, the server device 1020 of this third embodiment does not require the step of providing the first video to the terminal device 1010, and the server device 1020 does not require the step of providing the first video to the terminal device 1010, and It may be configured to extract 1 action (400).

In this third embodiment, the implementation method after the first operation extraction 400 can use the same contents described in the first embodiment and its modifications. The embodiment of the service operation 600 described above with reference to FIG. 6 may also be used in the same manner. Additionally, the embodiment 201 applied to correspond to a plurality of objects in the first video from the first embodiment can also be combined with the third embodiment in the same way.

Embodiment 4

Hereinafter, a description will be given regarding the fourth embodiment of the present invention, which is derived by changing the implementation method from the first embodiment. In addition, in addition to each of the main embodiments described above, some modified embodiments that can be applied or modified at the discretion of a person skilled in the art will also be described.

Figure 12 is a conceptual diagram of the operation of a service for providing user posture guide according to the fourth embodiment of the present invention. The service system 1200 shown in FIG. 12 is, for example, for providing a home training service by providing a posture guide according to the present invention, and includes a terminal device 1210, a server device 1220, and a content provider ( 1230).

In the description of FIG. 12, items sharing the same symbols as those of FIG. 2 may refer to the first embodiment.

In the following fourth embodiment, which does not limit the implementation method of the present invention, the present invention is performed in a manner that does not require acquisition of the second moving image and comparison with the second operation shown in the first, second, and third embodiments described above. A method of carrying out is provided.

In this fourth embodiment, the implementation method until the first operation is extracted (400) and transmitted (S259) from the server device 1220 to the terminal device 1210 is the same as the first embodiment and its modification. The contents described in can be used in the same way.

That is, according to one implementation method of the present invention for providing a user posture guide according to the fourth embodiment, the terminal device 1210 connects to the server device 820 providing the service (S251), and the connection (S251) In response to S251), the server device 1220 may provide interface information to the terminal device 1210 (S252). According to reception of the interface information (S252), the corresponding interface 300 may be displayed on the terminal device 252.

Video selection information can be input through the interface 300, and the video selection information can be transmitted to the server device 1120 (S253) and then to the content provider 1230 (S254). Based on the video selection information, the content provider 1230 provides the first video to the server device 1230 (S255), and the terminal device 1210 receives the first video from the server device 1230. The first video in which the motion was filmed can be obtained.

The server device 1220 includes a first motion extraction unit, and can extract the first motion from the first video using the first motion extraction unit. Accordingly, the first motion extraction unit extracts at least one image frame from the first video, generates at least one object joint information based on the at least one image frame, and generates at least one object joint information based on the at least one image frame. Generate one object skeleton information, generate at least one object posture information by combining the at least one object joint information and the at least one object skeleton information, and continuously combine the at least one object posture information to create a first It can be configured to extract 1 action.

As a result of the extraction, information about the first operation may be provided from the server device 1220 to the terminal device (S259). Since the first video and the first operation have been secured, the terminal device 1210 can be configured to perform the service operation 1260 based on them.

In the description of this fourth embodiment, which does not limit the implementation method of the present invention, an interface that implements the home training service operation 1260 to help the user successfully imitate the first movement appearing in the first video. can be implemented.

The acquired first video may be displayed to the user. Additionally, according to an embodiment of the present invention, the first video may be displayed together with the first motion. The first video may be displayed through the first video display function of the interface, and the first motion may be displayed through the first motion display function of the interface.

The display of the first motion may be accomplished by a procedure including converting object posture information included in the first motion into a graphic posture guide element in the form of a reproduced object. For example, when the object posture information of the present invention consists of object joint information and object skeleton information as in the above-described embodiment, the object joint information and object skeleton information may be visualized and provided to the user as a posture guide. there is.

Additionally, the embodiment 201 applied to correspond to a plurality of objects in the first video from the first embodiment can be combined with the fourth embodiment in the same way as in the first embodiment.

In the above-described fourth embodiment, the second motion in which the user observes and imitates the first motion picture is not required, and comparison of the first motion and the second motion can be omitted. The posture guide in the fourth embodiment has the form of the first motion analyzed from the first video. Therefore, the posture guide in the fourth embodiment is beneficial to users who want to obtain only data related to a sample of an exercise movement or the first movement by analyzing the posture information appearing in the first video through the motion extraction unit according to the present invention. It can be provided.

Embodiment of terminal device

Hereinafter, an embodiment of a terminal device used to implement the present invention will be described.

Figure 13 is a block diagram of a terminal device for providing a user posture guide according to the present invention. In the following embodiment, which does not limit the implementation method of the present invention, the terminal device 1300 includes a first input unit 1310 that receives image selection information 1315, and a first video based on the image selection information. An image acquisition unit 1320 that acquires, a first processing unit 1330 that acquires a first motion associated with the first video, and a second video 1345 in which a second motion imitating the first motion is captured. A second input unit 1340, a second processing unit 1350 that acquires a second motion associated with the second video, and a third processing unit 1360 that generates posture guide information by comparing the first motion and the second motion. , a display unit 1370 that displays at least one of the first video, the first motion, the second video, the second motion, and the posture guide information, a processor 1380 that controls the operation of each part, and the processor It may include a memory 1390 connected to .

Depending on the embodiment, the display unit 1370 may be connected to the display unit 1375 to display a visual display. Additionally, according to another modification, it may be connected to the speaker unit 1378 and configured to perform an audio display.

The terminal device 1300 may be a device that implements the terminal devices 210, 810, and 1010 shown in the first, second, and fourth embodiments of the present invention described above. In the terminal devices 210, 810, and 1010 of the first, second, and fourth embodiments, the terminal device 1300 may further include a communication unit 1335, and the communication unit may communicate with the outside world through a communication network. It may be configured to connect to the server 1338, that is, the corresponding server devices 220, 820, and 1020 in each of the above embodiments to transmit and receive necessary information.

In some embodiments of the present invention, the communication unit 1335 functions as at least one of the image acquisition unit 1320, the first processing unit 1330, the second processing unit 1350, and the third processing unit 1360. Communication for execution can be performed. When the first video needs to be received from the external server 1338, when the first motion extraction unit operates in the external server 1338, when the second motion extraction unit operates in the external server 1338, and When the first operation and the second operation are compared in the external server 1338, the communication unit 1335 may need to process information transmission and reception for each corresponding functional unit.

In addition, when the terminal device 1300 is used as a device that implements the terminal device 1010 shown in the fourth embodiment of the present invention described above, the second input unit 1340 and the second input unit 1340 among the above-described functional units. The processing unit 1350 and the third processing unit 1360 may not be used, and therefore may be omitted in the implementation of the fourth embodiment to the extent that there is no problem in achieving the purpose of the present invention.

Possibility of other modifications

Although the description has been made with reference to the drawings and examples, this does not mean that the scope of protection of the present invention is limited by the drawings or examples, and those skilled in the art will understand the spirit and scope of the present invention as set forth in the following patent claims. It will be understood that the present invention can be modified and changed in various ways without departing from the scope of the present invention. Hereinafter, some modified embodiments of the present invention will be illustratively described, and the possibility of modified implementation of the present invention is not limited to the modified embodiments described later.

In a modified embodiment of the present invention, an object identified in at least one of the first video and the second video of the present invention may include something other than the human body. Therefore, the object taking the first action and the object taking the second action are not limited to the human body, and the posture identification and action according to the present invention includes any living thing or inanimate object in a form capable of posture analysis, or a visual representation of such an object. Even if it is the subject of comparison, it is a fact that those skilled in the art can easily see that the structure and various implementation methods of the present invention described above can be used in the same manner without major modification.

As an example that does not limit the scope of application of the present invention, the first video may include an image of a first motion shown by a virtual human body visualized using computer graphics. As another example, the second video may be a film of a second motion of an articulated robot that imitates the first motion.

The first video does not have data in a completed form in advance from the content provider, and like the second video, it will be supplied in the form of relaying digital video data that is shot in real time or in delayed real time using the shooting means of the terminal device. You can.

As shown in the third embodiment described above, when the user directly provides an exemplary video corresponding to the first video, the first video is provided by the user in the form of a video file, or represents continuous frames. It may be provided in the form of multiple photos. The plurality of photos may be converted into the first video, and a conversion process including frame interpolation may be applied as needed.

In the various embodiments described above, the operation of the first motion extraction unit may be performed in real time or non-real time according to execution instructions through the user's terminal device. When operating in non-real time, the server device shown in the above embodiments notifies the terminal device for execution of a service using the first action when extraction of the first action is completed by the first action extraction unit. It may be configured to transmit and perform an operation that prompts the user to operate the terminal device.

The first video and the second video can be compared by different speeds. For example, the user can take the second action by imitating the first video by playing it slowly at 0.5 times the speed, or the user can take the second action by imitating the first video by playing it quickly at 2 times the speed.

The display of the first video can be paused, and when the display is paused, the display of the second motion can also be paused. Additionally, the comparison between the first and second motions and the generation and display of posture guide information based on the comparison information may be paused together.

The first video may be displayed in a loop playback format in which the first video is repeatedly played from the beginning even after playback is terminated until instructed to do so by a terminal device operated by the user. Repetition by loop playback may be a predetermined number of times or indefinitely.

In the case of the embodiment in which the terminal device receives and uses the first video from the server device as in the first, second, and fourth embodiments, the first video is stored in whole or in part as a cache in the terminal device. It can be stored and used to efficiently communicate with the server device. Likewise, the first motion extracted from the first video is stored as a cache corresponding to the first video in the terminal device or server device where the first motion extraction unit is located, so that computational resources resulting from the repetitive operation of the first motion extraction unit are stored in the cache corresponding to the first video. Waste can be suppressed.

Although the present invention has been described above with reference to the drawings and examples, as already described above, this does not mean that the scope of protection of the present invention is limited by the drawings or examples presented above, and those skilled in the art will It will be understood that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims.

Claims (20)

In a method of providing a user posture guide using a terminal device,
Inputting image selection information and acquiring a first video in which a first motion is captured based on the image selection information;
Obtaining a first motion associated with the first video;
displaying the first video;
Obtaining a second video in which a second motion imitating the first motion is captured;
Comparing the first operation and the second operation; and
A method comprising displaying posture guide information based on the comparison.
According to claim 1,
The step of acquiring the first video based on the video selection information,
Connecting to a server device;
Receiving interface information from the server device;
displaying interface information for inputting the image selection information;
inputting the video selection information according to the interface information and transmitting it to the server; and
Characterized in that it comprises the step of receiving the first video corresponding to the video selection information from the server.
According to claim 2,
The method, characterized in that the video selection information is included in the interface information and is information for selecting one of at least one video selection provided from the server device.
According to claim 2,
The video selection information includes at least one piece of information used by the server device to acquire the first video from a content provider of the first video.
According to claim 4,
The video selection information is,
Communication information identifying the content provider of the first video in a communication network;
Identification information used by the content provider to identify the first video;
Communication protocol information used to obtain the first video; and
Communication authentication information including at least one of an ID, password, and authentication key required to obtain the first video from the content provider; A method comprising at least one of the following:
According to claim 1,
The step of acquiring the first video based on the video selection information,
Displaying interface information for selecting at least one image stored in the storage;
inputting the image selection information according to the interface information; and
Characterized in that it comprises the step of obtaining the first video corresponding to the video selection information from storage.
According to claim 1,
extracting the first motion from the first video by a first motion extraction unit; and
Further comprising extracting the second motion from the second video by a second motion extraction unit,
The method is characterized in that the first operation and the second operation are information related to an operation that indicates a change in the posture of an object in chronological order.
According to claim 7,
At least one of the first motion extraction unit and the second motion extraction unit operates according to a posture extraction algorithm,
The posture extraction algorithm receives a video as input and outputs a motion,
Extracting at least one image frame from a video;
generating at least one object joint information based on at least one image frame;
generating at least one object skeleton information based on the at least one image frame;
generating at least one object posture information by combining the at least one object joint information and the at least one object skeleton information; and
A method operating including extracting a motion by continuously combining the at least one object posture information.
According to claim 8,
The pose extraction algorithm further includes normalizing the object pose information,
The normalization means standardizing the object posture information by applying a geometric transformation belonging to at least one of enlargement, reduction, rotation, inversion, and tilt by at least one vector to at least a portion of the object posture information.
According to claim 8,
The pose extraction algorithm is characterized in that at least one step operates by an artificial neural network.
According to claim 8,
The step of displaying the first video is,
Converting the object posture information included in the first operation into a posture guide graphic element in the form of a reproduced object; and
A method comprising displaying the posture guide graphic element together with the first video.
According to claim 8,
A method, characterized in that the first motion extraction unit operates in a server device.
According to claim 1,
The step of comparing the first operation and the second operation is,
Comprising at least one object posture information included in the first operation and at least one object posture information included in the second operation using a comparison algorithm to obtain at least one posture comparison information,
The at least one posture comparison information is information representing at least one of a degree of agreement and a difference vector of the second motion based on the first motion,
The method, characterized in that the posture guide information is generated based on the at least one posture comparison information.
According to claim 13,
The comparison algorithm applies a geometric transformation belonging to at least one of enlargement, reduction, rotation, inversion, and tilt by at least one vector to at least a portion of the object posture information included in the second operation. A method comprising a normalization step of standardizing object pose information.
According to claim 1,
The displaying of the posture guide information includes visualizing the posture guide information, overlaying it on at least one of the first video and the second video, and displaying the posture guide information through a display unit of the terminal device. .
According to claim 1,
The method includes the step of converting the posture guide information into audio and displaying the posture guide information through a speaker unit of the terminal device.
According to claim 1,
In the first video, a plurality of first movements are filmed,
Further comprising the step of selecting at least one motion classification information including information distinguishing a time when a motion appears in the first video and information distinguishing an object that took a motion in the first video,
The step of acquiring the first motion associated with the first video is characterized in that acquiring only the first motion identified based on the motion classification information among the plurality of first motions associated with the first video.
In a method of providing a user posture guide using a terminal device,
Inputting image selection information and acquiring a first video in which a first motion is captured based on the image selection information;
extracting at least one video frame from the first video;
generating at least one object joint information based on at least one image frame;
generating at least one object skeleton information based on the at least one image frame;
generating at least one object posture information by combining the at least one object joint information and the at least one object skeleton information;
extracting a first motion by continuously combining the at least one object posture information;
Converting the object posture information included in the first operation into a posture guide graphic element in the form of a reproduced object; and
A method comprising displaying the posture guide graphic element together with the first video.
In a method of providing a user posture guide using a server device,
Receiving video selection information about a first video from a terminal device;
requesting the first video from a content provider based on the video selection information;
Obtaining the first video from the content provider;
Obtaining a first motion associated with the first video using a posture extraction algorithm; and
A method comprising transmitting the first video and the first motion to the terminal device.
In a terminal device that provides a user posture guide,
A first input unit that receives image selection information;
An image acquisition unit that acquires a first video based on the image selection information,
a first processing unit that acquires a first motion associated with the first video;
a second input unit that acquires a second video in which a second motion imitating the first motion is captured;
a second processing unit that acquires a second motion associated with the second video;
a third processing unit that compares the first motion and the second motion to generate posture guide information;
a display unit that displays at least one of the first video, the first motion, the second video, the second motion, and the posture guide information;
A processor that controls the operation of each part; and
A device comprising memory coupled to the processor.

Images