KR20200087436A - Method and apparatus for providing exercise information based on video data and sensing data - Google Patents

Abstract

Disclosed is a method for providing exercise information based on image data and sensing data, comprising the following steps: acquiring image data and sensing data in accordance with a user′s movement; updating a characteristic data matrix based on the sensing data; generating cumulative data distribution based on the characteristic data matrix; and displaying exercise information of the user including the image data and the cumulative data distribution.

Description

Method and apparatus for providing exercise information based on image data and sensing data {METHOD AND APPARATUS FOR PROVIDING EXERCISE INFORMATION BASED ON VIDEO DATA AND SENSING DATA}

The present invention relates to a method and apparatus for providing exercise information based on image data and sensing data, and more specifically, a method for providing exercise information based on first person image data and Global Positioning System-Inertial Measurement Unit (GPS-IMU) sensing data, and It is about the device.

As action cams such as gopro have become more common, the shooting of first person videos has become easier, and in the sports industry, it is used as a monitoring method for coaches to immerse themselves in the position of sports players to help sports players.

In addition, in recent years, the convergence of high technology and sports science has been activated in the sports industry, and the method of analyzing sports data by extracting data from sports players through various sensors is also being advanced.

However, the current method of analyzing sports data is difficult to understand if the results are not data experts. As a result, problems that are difficult to use by data non-professionals such as sports coaches, coaches, athletes, and the general public who want to utilize them have emerged, and to solve these problems, it is easy to understand and intuitively use the data analysis results. It is being studied as.

An object of the present invention for solving the above problems is to provide a method for providing exercise information based on image data and sensing data.

Another object of the present invention for solving the above problems is to provide an apparatus for providing exercise information based on image data and sensing data.

A method for providing exercise information according to an embodiment of the present invention for achieving the above object includes: obtaining image data and sensing data according to a user's movement, updating a characteristic data matrix based on the sensing data, and characteristic data The method may include generating a cumulative data distribution based on a matrix and displaying user motion information including image data and cumulative data distribution.

Here, the sensing data may include Global Positioning System (GPS) data and Inertial Measurement Unit (IMU) data.

Here, it may further include the step of generating the integrated sensing data by pre-processing the GPS data and the IMU data.

Here, updating the characteristic data matrix based on the sensing data includes: obtaining coordinate information corresponding to the sensing data in a preset region of the cumulative data distribution, and of the characteristic data matrix corresponding to the coordinate information based on the sensing data. And updating the value.

Here, the step of updating the characteristic data matrix based on the sensing data includes: obtaining peripheral coordinate information located within a certain distance from the coordinates and corresponding to the peripheral coordinate information based on the distance information and sensing data between the coordinates and the peripheral coordinates. The method may further include updating a value of the characteristic data matrix.

Here, the step of displaying the user's motion information including the image data and the cumulative data distribution, the color of the position corresponding to the data accumulated on the cumulative data distribution based on the number of data accumulated according to the update of the characteristic data matrix. And adjusting.

Here, the step of analyzing the time included in each zone on the preset table based on the cumulative data distribution may further include extracting the user's exercise intensity information and fatigue information.

Here, it may further include the step of extracting the user's exercise load information based on the cumulative data distribution.

Here, the displaying of the user's exercise information including the image data and the cumulative data distribution may include synchronizing the image data and the cumulative data distribution based on the visual information of the image data and the visual information of the sensing data. .

An apparatus for providing exercise information according to an embodiment of the present invention for achieving the other object includes a processor and a memory in which at least one instruction executed through the processor is stored, and at least one instruction is , Is executed to acquire image data and sensing data according to a user's movement, and is executed to update the characteristic data matrix based on the sensed data, and is executed to generate a cumulative data distribution based on the characteristic data matrix, and accumulated with the image data It can be executed to display the user's exercise information including the data distribution.

Here, the sensing data may include Global Positioning System (GPS) data and Intelligent Measurement Unit (IMU) data.

Here, at least one command may be executed to preprocess GPS data and IMU data to generate integrated sensing data.

Here, the at least one command is executed to obtain coordinate information corresponding to sensing data in a preset area of the cumulative data distribution, and may be executed to update the value of the characteristic data matrix corresponding to the coordinate information based on the sensing data. .

Here, the at least one command is executed to obtain information about the coordinates located within a certain distance from the coordinates, and updates the value of the characteristic data matrix corresponding to the information about the surrounding coordinates based on the distance information and sensing data between the coordinates and the surrounding coordinates. Can be executed.

Here, the at least one command may be executed to adjust the color of the position corresponding to the accumulated data on the accumulated data distribution based on the number of accumulated data according to the update of the characteristic data matrix.

Here, the at least one command may be executed to analyze the time included in each zone on the preset table based on the cumulative data distribution to extract exercise intensity information and fatigue information of the user.

Here, at least one command may be executed to extract the user's exercise load information based on the cumulative data distribution.

Here, the at least one command may be executed to synchronize the image data and the cumulative data distribution based on the time information of the image data and the time information of the sensing data.

According to the present invention, it is possible to improve the efficiency of sports coaching by comparing and analyzing image data and sensing data simultaneously.

According to the present invention, non-experts related to data can intuitively utilize data analysis results by simultaneously checking image data and sensing data.

1 is a view showing a state of a GPS-IMU sensor according to an embodiment of the present invention.
2 is a view showing sensing data of a GPS-IMU sensor according to an embodiment of the present invention.
3 is a view showing a part of programming code for a pre-processing process according to an embodiment of the present invention.
4 is a diagram illustrating a part of programming code for displaying a heat map according to an embodiment of the present invention.
5 is a diagram showing a part of programming code for a Cdata matrix according to an embodiment of the present invention.
6 is a view showing a heat map UI according to an embodiment of the present invention.
7 is a block diagram of an exercise information providing apparatus according to an embodiment of the present invention.
8 is a flowchart of a method for providing exercise information according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term “and/or” includes a combination of a plurality of related described items or any one of a plurality of related described items.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but there may be other components in between. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

The terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms, such as those defined in a commonly used dictionary, should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the overall understanding in describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a state of the GPS7-IMU sensor according to an embodiment of the present invention.

The method for providing exercise information according to an embodiment of the present invention uses sensing data obtained by measuring a user's movement through a sensor, and the sensing data may include Global Positioning System (GPS) data and Intelligent Measurement Unit (IMU) data. have.

Here, GPS may refer to a system that receives a signal from a satellite and calculates a user's location, and IMU may refer to an inertial measurement device that measures an object's speed, direction, gravity, and acceleration.

Referring to FIG. 1, a black box-shaped sensor is a GPS-IMU sensor device capable of measuring GPS data and IMU data together, and is attached or mounted on the back of a user's body to acquire sensing data for the user's movement However, the shape, color, and location of the body to which the sensor is attached are not limited thereto. In addition, since the GPS data and the IMU data may be respectively obtained through the GPS sensor device and the IMU sensor device, the scope of the present invention is not limited to the integrated sensor.

Here, the GPS sensor can have a position error within 2.5 m, a speed error within 0.1 m/s, and a performance of 10 Hz, and the IMU sensor has a maximum acceleration range of ±16 g and an angle measurement speed of ± 2000 yen. However, since this is an embodiment, the performance of each sensor is not limited thereto.

2 is a view showing sensing data of a GPS-IMU sensor according to an embodiment of the present invention.

The GPS-IMU sensor according to an embodiment of the present invention can measure a user's movement in real time, and transmit the measured GPS data and IMU data to an exercise information providing device.

The GPS data and the IMU data can each appear as text files. Here, the GPS data may include at least one of measurement time information, latitude information, longitude information, and speed information (m/s) in order, and the IMU data includes measurement time information, x-axis acceleration information, and x-axis angular velocity information, At least one of y-axis acceleration information, y-axis angular velocity information, z-axis acceleration information, and z-axis angular velocity information may be sequentially included, but is not limited thereto. Here, the standard of each information may be in accordance with a normal standard, but may be set in advance.

Referring to FIG. 2, values included in one row of GPS data may indicate GPS data at a specific time. In other words, the values in the first row of GPS data can mean one GPS data, including measurement time information of 2018.07.18.21.23.0.0, latitude information of 36.0177278333333, longitude information of 129.324110333333 and speed information of 5.650452 can do.

Values included in one row of IMU data may indicate IMU data at a specific time. In other words, the values of the first row of IMU data may mean one IMU data, measurement time information of 2018.07.18.21.23.0.0, x-axis acceleration information of 1298, x-axis angular velocity information of -456,- 768 y-axis acceleration information, -94 y-axis angular velocity information, -35 z-axis acceleration information, and -264 z-axis angular velocity information.

3 is a view showing a part of programming code for a pre-processing process according to an embodiment of the present invention.

The apparatus for providing exercise information according to an embodiment of the present invention may generate integrated sensing data by integrating GPS data obtained from a GPS-IMU sensor and IMU data.

In other words, since GPS data and IMU data are measured at different frequencies and constitute separate data, data parsing and preprocessing may be performed using specific programming codes for data analysis.

Referring to Figure 3, the exercise information providing apparatus according to an embodiment of the present invention can use a Python (python) code, '1_och.txt' may mean GPS data, '1_imu.txt' is IMU It can mean data. In addition,'output2.txt' may mean integrated sensing data in which GPS data and IMU data are integrated. However, since the Python code of FIG. 3 may be changed and other programming codes may be used, the scope of the present invention is not limited thereto.

4 is a diagram illustrating a part of programming code for displaying a heat map according to an embodiment of the present invention.

The apparatus for providing exercise information according to an embodiment of the present invention may generate cumulative data distribution using integrated sensing data incorporating GPS data and IMU data, and may perform display setting in advance to display the cumulative data distribution. have. Here, the cumulative data distribution may be referred to as a heat map.

Referring to FIG. 4, the apparatus for providing exercise information according to an embodiment of the present invention may set the resolution of a heat map to 200x150, separately set a thread for processing data and a thread for rendering heat wrap, , Accordingly, data processing and display may be performed using a data processing thread. However, the size and/or resolution may be determined according to the latitude information and longitude information of the sensing data for the heat map display, and may be changed to different resolutions for the user's movement, sports, and display effect. It is not limited.

According to an embodiment of the present invention, an interactive data display library may be used to display a heat map, and codes may be changed and integrated according to a data format and a rendering method used in the library. However, an embodiment of the present invention may be directly displayed using a datagridview in consideration of the degree of customization or processing speed, so the scope of the present invention is not limited thereto.

5 is a diagram showing a part of programming code for a Cdata matrix according to an embodiment of the present invention.

The apparatus for providing exercise information according to an embodiment of the present invention may generate a cumulative data distribution by using integrated sensing data incorporating GPS data and IMU data to display a heat map according to the settings shown in FIG. 4.

In more detail, the apparatus for providing exercise information according to an embodiment of the present invention may update a characteristic data matrix based on the integrated sensing data, and render a characteristic data matrix to represent a heat map.

In other words, according to an embodiment of the present invention, when the data processing thread and the rendering thread are separately set, the characteristic data matrix may be updated in real time based on the integrated sensing data acquired in real time by the data processing thread, and the rendering thread may As updated, a heatmap can be generated based on the characteristic data matrix.

Here, the apparatus for providing exercise information may acquire coordinate information corresponding to sensing data in an area according to the setting of the heat map, and may update the value of the characteristic data matrix corresponding to the coordinate information. Also, the apparatus for providing motion information may acquire peripheral coordinate information located within a certain distance from coordinates of the acquired coordinate information, and may also update values of a characteristic data matrix corresponding to all peripheral coordinate information located within a certain distance. have. Here, the value of the characteristic data matrix corresponding to the surrounding coordinate information may be updated differently according to the distance from the coordinate corresponding to the sensing data, and the value may be changed in proportion to the distance, but is not limited thereto.

6 is a view showing a heat map UI according to an embodiment of the present invention.

Referring to FIG. 6, a heat map user interface (UI) according to an embodiment of the present invention may simultaneously display a heat map based on image data and integrated sensing data according to a user's movement, and the image data and heat map are time It may be synchronized, and operations related to normal video playback such as forward, backward and pause may be performed by the user.

The apparatus for providing exercise information according to an embodiment of the present invention may acquire image data according to a user's movement corresponding to sensing data through a photographing device in order to simultaneously provide image data and a heat map. Time synchronization. In other words, the apparatus for providing exercise information may synchronize based on the visual information of the image data and the visual information of the sensing data that is the basis of the heat map.

In addition, an embodiment of the present invention may represent a pre-photographed image and a heat map corresponding thereto, or a real-time image and a real-time heat map corresponding thereto. Here, when representing a pre-photographed image and a heat map corresponding thereto, the heat map may be generated in advance based on sensing data corresponding to the corresponding image.

Here, the exercise information providing device may adjust the color of the position corresponding to the data accumulated on the heat map based on the number of data accumulated according to the update of the characteristic data matrix. In other words, the exercise information providing apparatus may make it easy for the user to check the color difference between the high and low data accumulation locations on the heat map, as shown in FIG. 6. The color may be set to gradually change according to the cumulative number, but each color may be changed according to the setting.

In addition, the exercise information providing apparatus according to an embodiment of the present invention may extract the user's exercise intensity information and fatigue information based on the cumulative data distribution. In other words, the exercise information providing device may analyze the time included in each zone on the preset table, and extract exercise intensity information and fatigue information based on the times included in each zone. Here, the preset table may be as shown in Table 1, but is not limited thereto.

Referring to Table 1, the exercise information providing apparatus may determine a corresponding zone based on the speed information included in the sensing data, and based on this, may analyze the time at which the user's movement corresponds to each zone.

The apparatus for providing exercise information according to an embodiment of the present invention may also extract the user's exercise load information based on the cumulative data distribution. In other words, the exercise information providing apparatus may calculate the exercise load information of the user based on the acceleration information included in the sensing data for a predetermined time according to a preset method. Here, the preset method may be the same as Equation 1, but is not limited thereto.

Referring to Equation 1, PlayerLoad may mean motion load information or motion load values, and fwd, side and up may mean x-axis acceleration information, y-axis acceleration information, and z-axis acceleration information, respectively, t May denote time information according to a sensing data acquisition time unit. In other words, the exercise information providing device may calculate the exercise load information of the user for a predetermined time based on the acceleration information for each axis included in the sensing data.

7 is a block diagram of an exercise information providing apparatus according to an embodiment of the present invention.

Referring to FIG. 7, an apparatus 700 for providing exercise information according to an embodiment of the present invention may include at least one processor 710, a memory 720, and a storage device 730.

The processor 710 may execute program commands stored in the memory 720 and/or the storage device 730. The processor 710 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to the present invention are performed. The memory 720 and the storage device 730 may be composed of volatile storage media and/or non-volatile storage media. For example, the memory 720 may be composed of read only memory (ROM) and/or random access memory (RAM).

The memory 720 may store at least one instruction executed through the processor 710. At least one command includes a command for acquiring image data and sensing data according to a user's movement, a command for updating a characteristic data matrix based on the sensed data, a command for generating a cumulative data distribution based on the characteristic data matrix, and an image data and It may include a command for displaying the user's exercise information including the cumulative data distribution.

Here, the sensing data may include Global Positioning System (GPS) data and Inertial Measurement Unit (IMU) data.

Here, the at least one command may further include a command to pre-process GPS data and IMU data to generate integrated sensing data.

Here, the command for updating the characteristic data matrix based on the sensing data includes: an instruction for obtaining coordinate information corresponding to the sensing data in a preset region of the cumulative data distribution and a characteristic data matrix corresponding to the coordinate information based on the sensing data. It can include instructions to update values.

Here, the command for updating the characteristic data matrix based on the sensing data corresponds to a command for obtaining peripheral coordinate information located within a certain distance from the coordinates, and corresponding to the peripheral coordinate information based on the distance information and sensing data between the coordinates and the peripheral coordinates. It may further include an instruction to update the value of the characteristic data matrix.

Here, the command to display the user's motion information including the image data and the cumulative data distribution, the color of the position corresponding to the data accumulated on the cumulative data distribution based on the number of data accumulated according to the update of the characteristic data matrix. It may include an order to coordinate.

Here, it may further include a command for analyzing the time included in each zone on the preset table based on the cumulative data distribution to extract the user's exercise intensity information and fatigue information.

Here, a command for extracting the user's exercise load information based on the cumulative data distribution may be further included.

Here, the command for displaying the user's motion information including the image data and the cumulative data distribution may include a command for synchronizing the image data and the cumulative data distribution based on the visual information of the image data and the visual information of the sensing data. .

8 is a flowchart of a method for providing exercise information according to an embodiment of the present invention.

Referring to FIG. 8, an apparatus for providing exercise information according to an embodiment of the present invention may first obtain image data and sensing data according to a user's movement (S810). Here, image data may be obtained from a connected photographing device, and sensing data may be obtained from a connected GPS-IMU sensor.

The apparatus for providing exercise information may update the characteristic data matrix based on the acquired sensing data (S820 ). Here, the sensing data may mean data combining GPS data and IMU data, and the characteristic data matrix may be updated based on the integrated data. In addition, the exercise information providing device may first perform a setting for displaying the cumulative data distribution based on the sensing data.

Also, the apparatus for providing exercise information may generate a cumulative data distribution based on the updated characteristic data matrix (S830). Here, the cumulative data distribution may be referred to as a heat map, and may be generated by rendering in a characteristic data matrix. Also, a thread that generates a cumulative data distribution from the characteristic data matrix may be set separately from a thread that updates the characteristic data matrix from the sensing data.

Thereafter, the apparatus for providing exercise information may synchronize the image data and the cumulative data distribution based on the visual information of the image data and the visual information of the sensing data (S840), and display the synchronized image data and the cumulative data distribution together. (S850). Here, the cumulative data distribution may indicate a position accumulated on the cumulative data distribution using different colors according to the cumulative amount of data. In addition, the exercise information providing apparatus may extract exercise intensity information, fatigue, and exercise load information based on the sensing data, and may provide the information together to the user.

The operation according to the embodiment of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. The computer-readable recording medium can also be distributed over network coupled computer systems so that computer-readable programs or codes are stored and executed in a distributed fashion.

In addition, the computer-readable recording medium may include a hardware device specially configured to store and execute program instructions, such as a ROM, a RAM, and a flash memory. Program instructions may include high-level language code that can be executed by a computer using an interpreter or the like, as well as machine code such as those produced by a compiler.

While some aspects of the invention have been described in the context of an apparatus, it can also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method can also be characterized by a corresponding block or item or characteristic of a corresponding device. Some or all of the method steps may be performed by (or using) a hardware device, such as, for example, a microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In embodiments, the field programmable gate array may work with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by some hardware device.

Although described above with reference to preferred embodiments of the present invention, those skilled in the art variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You can understand that you can.

Claims (18)

Obtaining image data and sensing data according to a user's movement;
Updating a characteristic data matrix based on the sensing data;
Generating a cumulative data distribution based on the characteristic data matrix; And
And displaying exercise information of the user including the image data and the cumulative data distribution. The method according to claim 1,
The sensing data,
A method of providing exercise information, including Global Positioning System (GPS) data and IMU (Inertial Measurement Unit) data. The method according to claim 2,
And preprocessing the GPS data and the IMU data to generate integrated sensing data. The method according to claim 1,
The updating of the characteristic data matrix based on the sensing data may include:
Obtaining coordinate information corresponding to the sensing data in a preset area of the cumulative data distribution; And
And updating a value of the characteristic data matrix corresponding to the coordinate information based on the sensing data. The method according to claim 4,
The updating of the characteristic data matrix based on the sensing data may include:
Obtaining peripheral coordinate information located within a certain distance from the coordinates; And
And updating a value of the characteristic data matrix corresponding to the peripheral coordinate information based on the distance information between the coordinates and the peripheral coordinates and the sensing data. The method according to claim 1,
The displaying of motion information of the user including the image data and the cumulative data distribution may include:
And adjusting a color of a position corresponding to the accumulated data on the accumulated data distribution based on the number of accumulated data according to the update of the characteristic data matrix. The method according to claim 1,
And analyzing the time included in each zone on a preset table based on the cumulative data distribution to extract the exercise intensity information and fatigue information of the user. The method according to claim 1,
And extracting exercise load information of the user based on the cumulative data distribution. The method according to claim 1,
The displaying of motion information of the user including the image data and the cumulative data distribution may include:
And synchronizing the image data and the cumulative data distribution based on the visual information of the image data and the visual information of the sensing data. A processor; And
And a memory in which at least one instruction executed through the processor is stored,
The at least one command,
It is executed to acquire image data and sensing data according to the user's movement,
Is executed to update a characteristic data matrix based on the sensing data,
Is executed to generate a cumulative data distribution based on the characteristic data matrix,
And an exercise information providing device configured to display exercise information of the user including the image data and the cumulative data distribution. The method according to claim 10,
The sensing data,
An apparatus for providing exercise information, including Global Positioning System (GPS) data and Inertial Measurement Unit (IMU) data. The method according to claim 11,
The at least one command,
Exercise information providing device, which is executed to pre-process the GPS data and the IMU data to generate integrated sensing data. The method according to claim 10,
The at least one command,
Is executed to obtain coordinate information corresponding to the sensing data in a preset area of the cumulative data distribution,
The exercise information providing apparatus is executed to update a value of the characteristic data matrix corresponding to the coordinate information based on the sensing data. The method according to claim 13,
The at least one command,
It is executed to obtain the surrounding coordinate information located within a certain distance from the coordinate,
Exercise information providing device that is executed to update a value of the characteristic data matrix corresponding to the peripheral coordinate information based on the distance information between the coordinates and the peripheral coordinates and the sensing data. The method according to claim 10,
The at least one command,
And adjusting the color of a position corresponding to the accumulated data on the accumulated data distribution based on the number of accumulated data according to the update of the characteristic data matrix. The method according to claim 10,
The at least one command,
The exercise information providing apparatus is executed to extract the exercise intensity information and the fatigue information of the user by analyzing the time included in each zone on a preset table based on the cumulative data distribution. The method according to claim 10,
The at least one command,
Exercise information providing device, which is executed to extract the exercise load information of the user based on the cumulative data distribution. The method according to claim 10,
The at least one command,
The apparatus for providing exercise information, which is executed to synchronize the image data and the cumulative data distribution based on time information of the image data and time information of the sensing data.

Images