KR102584540B1 - Subject movement information generation device using artificial intelligence and method thereof - Google Patents

Abstract

The present invention relates to an apparatus and method for automatically generating information about subject movement in an image, and more specifically to an apparatus and method for automatically generating detailed information about subject movement in an image using artificial intelligence. will be. The motion information generating device according to an embodiment of the present invention includes a memory and a processor in which an artificial intelligence algorithm and a target image are stored, and the memory stores target motion information about the movement of a subject in the target image that is executable by the processor. You can store program instructions that are automatically generated using artificial intelligence algorithms. At this time, the artificial intelligence algorithm is learned from a data set virtually created by a preset method, and the data set is automatically set learning movement information and learning images automatically generated corresponding to the learning movement information. may include.

Description

Subject movement information generation device using artificial intelligence and method thereof}

The present invention relates to an apparatus and method for automatically generating information about subject movement in an image, and more specifically to an apparatus and method for automatically generating detailed information about subject movement in an image using artificial intelligence. will be.

Due to the global spread of COVID-19, golf, which can be enjoyed outdoors with a small number of people, is gaining attention. When practicing at an indoor golf driving range, golfers want to know exactly detailed information about the golf ball they hit. I am curious about how far the golf ball I hit flew and what the amount of backspin/sidespin was. This may also be the case when enjoying a round at an outdoor golf course.

In order to resolve these questions of golfers, a 'smart golf ball' with sensors such as a gyro sensor, GPS sensor, and acceleration sensor, memory, and a wireless rechargeable battery formed inside has been disclosed (see Figure 1). These smart golf balls are expensive and have inconvenient problems in use, such as the need for charging.

In addition, a method of photographing a golf ball using a high-speed camera and sensing the movement of the golf ball through changes in the position and shape of the photographed golf ball is widely used in indoor golf driving ranges, etc. A high-speed camera captures a moving golf ball at preset intervals, and a computer connected to the high-speed camera analyzes the captured images to measure the speed and rotation of the golf ball, and then uses a preset method based on this to measure the golf ball. You can estimate the flying distance, falling point, etc.

Figure 2 shows an example in which the high-speed camera 210 photographs a golf ball for q period. The first golf ball 220-1 is a golf ball photographed in the first cycle, the second golf ball 220-2 is a golf ball photographed in the second cycle, and the q golf ball 220-q is a golf ball photographed in the second cycle. It may be a golf ball photographed in period q. In addition, the first shadow 230-1 is a shadow of a golf ball photographed in the first cycle, the second shadow 230-2 is a shadow of a golf ball photographed in the second cycle, and the q-th shadow 230- q) may be the shadow of a golf ball taken at the qth period. A computer (not shown) connected to the high-speed camera 210 uses information such as the position and size of the golf balls (220-1 to 220-q) in the image and the shutter speed of the high-speed camera 210 to determine the speed and speed of the golf ball. The left and right movement angle can be measured. Additionally, a computer (not shown) can measure the vertical angle of the golf ball using the size of the golf ball's shadow (230-1 to 230-q) in the image. A computer (not shown) can simulate the distance and flight trajectory of a golf ball using measured information.

Meanwhile, in order for the high-speed camera 210 to measure the rotation speed of the golf ball, a characteristic shape (four large dots in FIG. 2) must be formed on the golf ball. A computer (not shown) can recognize the rotation of a golf ball by recognizing changes in its characteristic shape. Of course, in theory, a computer (not shown) can recognize the rotation speed of a golf ball through the brand, line, logo, dimple, etc. printed on the surface of the golf ball even if a characteristic shape such as a large dot is not formed. They say it can be done, but there is a problem with the accuracy being significantly lower.

Other golf ball tracking methods using lidar are also widely used, but there is a problem in that they are difficult to use widely because the necessary equipment is expensive.

The present invention seeks to provide a device and method that can accurately generate detailed information on subject movement in an image in real time using artificial intelligence.

According to one embodiment of the present invention, a memory in which an artificial intelligence algorithm and a target image are stored; And a processor; wherein the memory stores program instructions that are executable by the processor and automatically generate target motion information about the movement of the subject of the target image using the artificial intelligence algorithm, wherein the artificial intelligence The algorithm is learned by a data set virtually created by a preset method, and the data set includes automatically set learning exercise information and a learning image automatically generated corresponding to the learning exercise information. , an exercise information generating device is disclosed.

According to another embodiment of the present invention, a method of generating information about the movement of a subject in an image, which is performed in a movement information generating device storing an artificial intelligence algorithm, includes the steps of inputting a target image; And automatically generating target motion information about the movement of the subject of the target image using the artificial intelligence algorithm, wherein the artificial intelligence algorithm is a data set virtually created by a preset method. ), and the dataset includes automatically set learning exercise information and a learning image automatically generated corresponding to the learning exercise information. A method of generating exercise information is disclosed.

Depending on the embodiment, the dataset may be automatically generated using n virtually created subjects (where n is a natural number).

Depending on the embodiment, the learning exercise information includes automatically set speed information, side spin information, back spin information, up and down angle information, and left and right angle information, and the learning image includes any one of the n subjects in the learning exercise. It may be about performing an exercise corresponding to the information.

Depending on the embodiment, the learning image may be about a case where an exercise corresponding to the learning exercise information is performed after the initial position of any one of the n subjects is automatically set.

Depending on the embodiment, the learning images may be plural and may include two or more that differ only in initial positions.

Depending on the embodiment, the learning image includes a plurality of images in which rendering information of any one of the n subjects is converted, and the rendering information may include one or more of lighting information, shooting angle information, and lens information. there is.

Depending on the embodiment, the learning image may include a 3D image of one of the n subjects and one or more 2D images corresponding to the 3D image.

According to the present invention, detailed information on the subject's movement within an input image can be quickly and accurately generated using an artificial intelligence algorithm learned through the movement of a virtually created subject and detailed information about the movement.

In order to more fully understand the drawings cited in the detailed description of the present invention, a brief description of each drawing is provided.
Figures 1 and 2 are example photos to explain a conventional technology for recognizing the movement of a subject (golf ball).
Figure 3 is a configuration diagram of an exercise information generation system according to an embodiment of the present invention.
Figure 4 is a block diagram of an exercise information generating device according to an embodiment of the present invention.
Figure 5 is a flowchart for explaining the operation of the exercise information generating device according to an embodiment of the present invention.
Figure 6 is a flowchart of the operation of generating a data set for artificial intelligence learning stored in an exercise information generating device according to an embodiment of the present invention.
Figure 7 is a diagram illustrating the colors and shapes of various types of golf balls.
Figure 8 is an example diagram for explaining various initial positions of an arbitrary golf ball.
Figure 9 is a diagram for explaining a learning dataset according to an embodiment of the present invention.
Figure 10 is a diagram for explaining a target image analysis operation according to an embodiment of the present invention.

Since the technical idea of the present disclosure can be subject to various changes and can have various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technical idea of the present disclosure to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the scope of the technical idea of the present disclosure.

In explaining the technical idea of the present disclosure, if it is determined that a detailed description of related known technology may unnecessarily obscure the gist of the technical idea of the present disclosure, the detailed description will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present application are merely identification symbols to distinguish one component from another component.

In addition, herein, when a component is referred to as "connected" or "connected" to another component, the component may be directly connected or directly connected to the other component, but in particular, the contrary It should be understood that unless a base material exists, it may be connected or connected through another component in the middle.

In addition, terms such as “unit”, “unit”, “unit”, etc. used herein refer to a unit that processes at least one function or operation, such as a processor, microprocessor, Micro Controller, CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerate Processor Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) It may be implemented through hardware, software, or a combination of hardware and software.

In addition, we would like to clarify that the division of components in this institution is merely a division according to the main function each component is responsible for. That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions. In addition to the main functions it is responsible for, each of the components described below may additionally perform some or all of the functions handled by other components, and some of the main functions handled by each component may be performed by other components. Of course, it can also be carried out exclusively by .

Hereinafter, various embodiments according to the technical idea of the present disclosure will be described in detail one by one.

Figure 3 is a configuration diagram of an exercise information generation system according to an embodiment of the present invention.

Referring to FIG. 3, the exercise information generating system 300 according to an embodiment of the present invention may include an exercise information generating device 310 and a camera 320.

The camera 320 may generate image data by capturing the subject 330 for a preset time and then transmit it to the exercise information generating device 310. For example, the camera 320 may photograph the subject 330 at a shutter speed of 0.01 [ms] for 1 [sec] and then transmit it to the exercise information generating device 310. Hereinafter, video data and/or image data captured by the camera 320 are referred to as target images.

When a target image is input from the camera 320, the motion information generating device 310 can automatically generate information about the subject's motion (hereinafter referred to as 'target motion information') using a stored artificial intelligence algorithm. . That is, the motion information generating device 310 can automatically generate target motion information of the target image subject using an artificial intelligence algorithm that has been learned and stored in advance.

Hereinafter, the operation of the exercise information generating device 310 to generate target exercise information will be described with a focus on learning the artificial intelligence algorithm.

Figure 4 is a block diagram of an exercise information generating device according to an embodiment of the present invention.

Referring to FIG. 4 , the exercise information generating device 310 may include a memory 410 and a processor 420 .

The memory 410 may be a data storage medium such as a magnetic disk (HDD) or solid state drive (SSD). Program instructions executable by the processor 420 may be stored in the memory 410 . In particular, an artificial intelligence algorithm that can generate a target image and target motion information corresponding to the target image may be stored in the memory 410.

The processor 420 is a central processing unit (CPU), a graphics processing unit (GPU), and/or a corresponding computing device capable of controlling the overall operation of the exercise information generating device 310, and is particularly stored in the memory 310. Target motion information corresponding to the target image subject can be automatically generated using program commands. Hereinafter, the operation of generating target motion information under the control of the processor 420 will be described.

Figure 5 is a flowchart for explaining the operation of the exercise information generating device according to an embodiment of the present invention.

Referring to FIG. 5, it can be seen that an artificial intelligence algorithm (AI) stored in the memory 410 is used to automatically generate target motion information by the processor 420. At this time, the artificial intelligence algorithm stored in the memory 410 may have been learned in advance through a specified learning data set.

That is, first, a dataset that is learning material for learning the artificial intelligence algorithm is created (Learning Data Generating, 510), then the artificial intelligence algorithm is learned using the generated dataset (Artificial Intelligence Learning, 520), and analysis and judgment are made. A target image is generated (Target data Generating, 530), and the target image can be analyzed and judged using a learned artificial intelligence algorithm (Target Data Analysis, 540).

The learning material creation operation 510 and the artificial intelligence algorithm learning operation 520 may be performed in an external device (not shown) and then stored in the memory 410. In other words, operations for learning the artificial intelligence algorithm may be performed in a device other than the exercise information generating device 310. Hereinafter, each operation in FIG. 5 will be described in more detail.

Figure 6 is a flowchart of the operation of generating a data set for artificial intelligence learning stored in an exercise information generating device according to an embodiment of the present invention, and Figure 7 illustrates the colors and shapes of various types of golf balls. It is a diagram, FIG. 8 is an example diagram for explaining various initial positions of an arbitrary golf ball, and FIG. 9 is a diagram for explaining a learning dataset according to an embodiment of the present invention.

Hereinafter, the dataset creation operation 510 for learning the artificial intelligence algorithm of FIG. 5 will be described in detail with reference to FIGS. 6 to 9. Here, the dataset is data for training the artificial intelligence algorithm stored in the memory 410 and may include virtual image data (2-dimensional and/or 3-dimensional image data, hereinafter referred to as 'learning image'). The virtual image data may not be an image of an actual subject, but may be an image of a virtual subject created according to a preset method.

Additionally, since each step in FIG. 6 may be performed in the exercise information generating device 410 or another device, the device in which each step in FIG. 6 is performed will be described without further limitation.

In step S610, information on n subjects virtually created to create a dataset for learning an artificial intelligence algorithm may be input to the server device, or may be generated within the server device according to the administrator's operation (where n is natural number). Here, the information about the subject may be information about a virtual three-dimensional object, not a real object. The virtual subject may be any object that has no joints and does not undergo significant self-deformation, such as a golf ball, baseball ball, or table tennis ball, and can be approximated as a rigid body. In Figure 6, for convenience of understanding and explanation, it is assumed that the subject is a virtual ball (first virtual ball to nth virtual ball). Since the method for creating a virtual 3D ball is very similar to the published method, detailed description thereof may be omitted. For example, n virtual balls may be implemented as golf balls currently in use.

Referring to Figure 7, three virtual balls are illustrated. The first virtual ball is a virtual golf ball of the first brand, and may have a main brand on the front (710-1) and a sub-brand on the back (710-2). The second virtual ball is a virtual golf ball of a second brand, and may have a main brand on the front (720-1) and a sub-brand on the back (720-2). Likewise, the third virtual ball is a virtual golf ball of a third brand, and may have a main brand on the front (730-1) and a sub-brand on the back (730-2). Meanwhile, the color of the front (730-1) and the back (730-2) of the third virtual ball may be different. In addition, although not shown in FIG. 7, it is obvious that virtual golf balls can be formed in more diverse colors.

In step S620, the first initial position of the first virtual ball may be set. Even if it is the same virtual ball, the learning image generated may be different depending on the initial position. Therefore, the server device can select the first virtual ball, which is one of the n virtual balls, and then set the first initial position of the first virtual ball. Figure 8 shows examples of various initial positions of three virtual balls 810 to 830. That is, the initial position may be relative to the position of the front main mark when any virtual ball is at rest.

In step S630, based on the first initial position of the first virtual ball, speed information a [m/sec], side spin information b [rpm], back spin information c [rpm], vertical angle information d [°], and left and right angles. Information e[°] (i.e., learning exercise information) may be set. Here, a, b, c, d, and e may be real numbers. Here, the speed information may be information about the speed at the moment when the first virtual ball starts moving from a stationary state (hereinafter referred to as 'at the time of departure'). Additionally, the side spin information may be information about the side spin at the start of the first virtual ball. Additionally, the backspin information may be information about the backspin at the time of departure of the first virtual ball. Additionally, the vertical angle information may be information about the vertical angle at the time of departure of the first virtual ball. Additionally, the left and right angle information may be information about the left and right angles at the time of departure of the first virtual ball.

The server device can set a, b, c, d, and e arbitrarily or randomly. However, the range of each value may be set in advance depending on the type of information. For example, the absolute values of side spin, back spin, and left and right angle information may be set to maximum values. For another example, maximum values may be set for speed information and vertical angle information.

In addition, the server device can generate a virtual 1-1-1 learning image in which the first virtual ball performs movements corresponding to the set a, b, c, d and e values based on the set first initial position. there is. For example, the server device uses the provided game engine to automatically create a 3D video of the initial movement of the first virtual ball at a speed, b side spin, c back spin, d up and down angle, and e left and right angle. It can be created with Here, the game engine is an application used when developing a 3D computer game or mobile game. The game engine automatically generates images (3D and/or 2D) in which virtual objects implement movements corresponding to given motion information. This can shorten the video development process. The first virtual ball performs movements of set speed information a [m/sec], side spin information b [rpm], back spin information c [rpm], up and down angle information d [°], and left and right angle information e [°]. A game engine that automatically generates 3D video can be easily implemented by a person skilled in the art, so detailed description thereof may be omitted.

In step S640, it may be determined whether the learning image (i.e., 1-1-p learning image) generated at the first initial position of the first virtual ball corresponds to a preset number (Pth) (where p is a natural acceptance). For example, assume that Pth is set to 10,000. At this time, if the number of learning images generated for the first virtual ball at the first initial position is less than 10,000, the server device changes one or more of speed information, side spin information, back spin information, up and down angle information, and left and right angle information. You can do it. At this time, the server device may change this information arbitrarily or randomly (step S645).

Afterwards, the server device may generate a learning image of the first virtual ball at the first initial position using the game engine based on the information changed in step S645.

In step S650, when the training image (i.e., 1-1-p learning image) generated at the first initial position of the first virtual ball reaches a preset number (Pth), the initial position of the first virtual ball is changed to the first position. 2 Can be changed to the initial position. The second initial position may be different from the first initial position. When the initial position of the first virtual ball is changed, the server device can arbitrarily or randomly set the speed information, side spin information, back spin information, up and down angle information, and left and right angle information, and use the game engine based on the set values. Thus, a learning video (i.e., a 1-2-1 learning video) can be generated. The server device may perform this operation until the number of learning images (i.e., 1-2-p learning images) generated at the second initial position of the first virtual ball reaches a preset number (Pth). .

In addition, the server device may change the initial position of the first virtual ball to the third initial position when the number of learning images generated at the second initial position of the first virtual ball reaches a preset number (Pth). The server device performs this operation when the initial position of the first virtual ball is changed mth times, and the number of learning images (i.e., 1-mth-p learning images) generated at the mth initial position is adjusted to the preset number (Pth). You can do this until you reach it.

In step S660, when the learning image (i.e., 1-mth-p learning image) generated at the mth initial position of the first virtual ball reaches a preset number (Pth), the learning image for the second virtual ball is can be created. The learning image corresponding to the second virtual ball may also be generated in a completely similar manner to the operation of generating the learning image corresponding to the first virtual ball. Additionally, the server device may repeat the operations of steps S620 to S660 for generating learning images until all learning images for the nth virtual ball are generated.

In this way, countless learning images can be automatically generated, and among the learning images, two or more may be included that differ only in initial positions. In other words, there may be two or more learning images where the virtual ball is the same and the detailed motion information of the virtual ball is the same, but only the initial position is different.

In the above, it has been explained that the learning image for the second virtual ball is generated after the learning image for the first virtual ball is generated, but these may be generated simultaneously. Therefore, the order in which learning videos are generated cannot limit the scope of the present invention. In addition, although the above description has shown that the learning images are generated sequentially one by one in the game engine, multiple learning images may be generated simultaneously.

In step S670, the rendering information of each of the generated plurality of training images may be changed to generate a plurality of data sets. Referring to FIG. 9, the dataset includes learning images of the first to nth virtual balls and the correct answers to each learning image, including speed information (a), side spin information (b), back spin information (c), and vertical angle. Information (d) and/or left and right angle information (e) may be labeled. Here, the training images of the dataset may include 3D videos and/or one or more 2D videos, and the training images may include 3D videos and 2D videos viewed from various angles (shooting angles) change).

Additionally, various datasets in which the lighting information of individual learning images are changed can be created. Even if the subject is the same, different images may be generated and recognized depending on the location, illuminance, and type of shooting lighting. Accordingly, the server device can generate various data sets by changing the location information, illuminance information, and type information (hereinafter collectively referred to as lighting information) of virtual lighting corresponding to the learning image.

Additionally, various datasets can be created by changing the lens information of the virtual camera corresponding to the training image. Even if it is the same subject, different images may be generated and recognized depending on the type of camera lens that captures the subject. Accordingly, the server device can generate various data sets by changing the type information (hereinafter collectively referred to as lens information) of the virtual camera lens that photographed the virtual ball.

Referring again to FIG. 5, the artificial intelligence algorithm learned through the dataset generated by the above-described method may be stored in the memory 410 (Artificial Intelligence Learning, 520). Therefore, when a 2-dimensional image and/or a 3-dimensional image (i.e., target image) is input to the artificial intelligence algorithm, the artificial intelligence algorithm uses movement information (i.e., target movement information) corresponding to the input image as the result. You will be able to print it. This is because datasets for learning artificial intelligence algorithms are labeled with 2-dimensional and/or 3-dimensional images for learning, and learning motion information corresponding to each image.

Therefore, when a target image (target data) is generated from the camera 320 (530), the processor 420 can input the target image into an artificial intelligence algorithm to generate target motion information corresponding to the target image (540).

Figure 10 is a diagram for explaining a target image analysis operation according to an embodiment of the present invention.

Referring to FIG. 10 , a case in which a target image (2D image, 1010) generated by the camera 320 is transmitted to the exercise information generating device 310 is illustrated. The processor 420 of the exercise information generating device 310 may process the target image 1010 as an input to an artificial intelligence algorithm (540). Through this, the processor 420 will be able to generate target motion information 1020 corresponding to the target image 1210.

As described above, high-quality datasets for learning artificial intelligence algorithms can be quickly created from virtual data through 3D graphics technology such as game engines at low cost (data augmentation technology). Therefore, according to the present invention, an artificial intelligence algorithm is learned through a learning image about the movement of a virtual subject generated according to the above-described data augmentation technology, and through this artificial intelligence algorithm, a learning image about the movement of the subject within the target image is learned. Information (i.e., target exercise information) can be accurately generated in real time.

The exercise information generating operation according to the present invention described above can be implemented methodically by being implemented as computer-readable code in the memory 120, which is a computer-readable recording medium. Computer-readable recording media include all types of recording media storing data that can be deciphered by a computer system. For example, there may be Read Only Memory (ROM), Random Access Memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, etc. Additionally, the computer-readable recording medium can be distributed to computer systems connected through a computer communication network, and stored and executed as code that can be read in a distributed manner. Accordingly, the processor 130 may execute the program instructions (code) stored in the memory 120 so that the above-described method of generating exercise information is performed step by step.

Above, the present invention has been described in detail with preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and changes can be made by those skilled in the art within the technical spirit and scope of the present invention. This is possible.

300: Exercise information generation system
310: Exercise information generating device
320: camera
410: memory
420: processor

Claims (14)

Memory where artificial intelligence algorithms and target images are stored; and
processor;
Including,
The memory is executable by the processor,
Store program instructions that automatically generate target motion information about the movement of the subject of the target image using the artificial intelligence algorithm,
The artificial intelligence algorithm is learned from a data set virtually created by a preset method,
The dataset includes automatically set learning exercise information and automatically generated learning images corresponding to the learning exercise information,
The dataset was automatically created using n virtually created subjects (where n is a natural number),
The learning exercise information includes one or more of automatically set speed information, side spin information, back spin information, up and down angle information, and left and right angle information,
The learning video is for a case where one of the n subjects performs a movement corresponding to the learning movement information,
The training image includes a plurality of images in which rendering information of one of the n subjects is converted.
delete delete According to paragraph 1,
The learning video is for a case where an exercise corresponding to the learning exercise information is performed after the initial position of any one of the n subjects is automatically set.
According to paragraph 4,
The exercise information generating device includes a plurality of learning images and includes two or more images that differ only in initial positions.
According to paragraph 4,
The rendering information includes one or more of lighting information, shooting angle information, and lens information.
According to any one of claims 1, 4 to 6,
The training image includes a 3D image of one of the n subjects and one or more 2D images corresponding to the 3D image.
In a method of generating information about the movement of a subject in an image, which is performed in a movement information generating device in which an artificial intelligence algorithm is stored,
Inputting a target image; and
Automatically generating target motion information about the movement of the subject of the target image using the artificial intelligence algorithm;
Including,
The artificial intelligence algorithm is learned from a data set virtually created by a preset method,
The dataset includes automatically set learning exercise information and automatically generated learning images corresponding to the learning exercise information,
The dataset was automatically created using n virtually created subjects (where n is a natural number),
The learning exercise information includes one or more of automatically set speed information, side spin information, back spin information, up and down angle information, and left and right angle information,
The learning video is for a case where one of the n subjects performs a movement corresponding to the learning movement information,
The training image includes a plurality of images in which rendering information of one of the n subjects is converted.
delete delete According to clause 8,
The learning video is for a case where an exercise corresponding to the learning exercise information is performed after the initial position of any one of the n subjects is automatically set.
According to clause 11,
A method of generating exercise information in which the learning images are plural and include two or more images that differ only in initial positions.
According to clause 11,
The rendering information includes one or more of lighting information, shooting angle information, and lens information.
According to any one of claims 8, 11 to 13,
The training image includes a 3D image of one of the n subjects and one or more 2D images corresponding to the 3D image.