KR102619792B1 - System and method for judging martial arts using image information based on artificial intelligence - Google Patents

Abstract

A martial arts artificial intelligence judgment system and method using video information is provided. A martial arts artificial intelligence judgment system using image information according to various embodiments of the present invention uses a plurality of sensor modules that generate sensor data and a previously learned artificial intelligence model by scanning a martial arts game taking place in a predetermined space. , and a server that calculates scores for players participating in the martial arts game by analyzing a plurality of sensor data generated from each of the plurality of sensor modules.

Description

Martial arts artificial intelligence judgment system and method using video information {SYSTEM AND METHOD FOR JUDGING MARTIAL ARTS USING IMAGE INFORMATION BASED ON ARTIFICIAL INTELLIGENCE}

Various embodiments of the present invention relate to a martial arts artificial intelligence judgment system and method using image information.

Taekwondo is a modern martial art created and developed in Korea. It is a martial art that attacks or defends using bare hands and feet, and is a traditional martial art formed in modern times characterized by fast and powerful kicking techniques. Taekwondo was adopted as a demonstration sport at the 1988 Summer Olympics, and was adopted as an official sport starting at the 2000 Summer Olympics.

Taekwondo can be largely divided into poomsae (form), sparring (sparring), and breaking (training). Taekwondo, which was adopted as an Olympic sport, is a sparring sport in which players strike against each other and the winner and loser are determined by the principle of multiple points. .

In this type of sparring, attacks and defenses can be made against opposing players according to the established game rules. The allowable range of attacks and defenses is the torso, where strikes using fists and foot techniques are permitted, and the head area, where only strikes using foot techniques are permitted. , points can be scored by hitting within this acceptable range.

In order to make accurate judgments due to the nature of the sparring sport, conventionally, players attach sensors to the hands and feet that are hitting, the protective gear on the torso (e.g., protector) and the protective gear on the head (e.g., headgear) that are hit. , a game was conducted in which points were awarded based on whether the sensor was operating when a hit was made.

However, according to the conventional method, the body armor and protective gear worn by the players are heavy, and since the sensors must be directly attached and mounted on the hands and feet, there is a problem that great discomfort is felt during the game. .

In addition, because the sensor must be attached to the area where the blow is actually made, sensor failure frequently occurs due to repeated blows, and as a result, it is difficult to make a clear decision due to errors in the judgment even though the blow is applied. There is.

It has been clearly shown that these problems occur frequently in many games, including domestic competitions as well as international competitions, and many players participating in the games are being harmed by these problems. Therefore, it can be seen that the damage caused by these problems can occur to anyone, regardless of whether at home or abroad, and it is an urgent task that must be clearly improved.

As a sport before the introduction of the electronic body protector system, Taekwondo matches were played with torso protectors (body protectors) and chest protectors to prevent injuries and score points. Players used their hands to defend themselves, hit the torso, and use their feet to defend themselves. It was an exciting style of play, with players using the body and face to strike or defend themselves.

However, in Taekwondo matches before the introduction of the electronic body protector system, the referee and assistant referee decided the win or loss by visually judging whether points were scored or deducted, so biased decisions due to misjudgment sometimes led to controversy over fairness, and this was adopted to erase the controversy over fairness. This is the electronic protector system.

Meanwhile, in the case of the electronic body protector system, the strength of the blow is not constant, and the game is played with the top priority given to operating so that the sensor can accurately recognize it rather than hitting, so the players in the game are competing in Taekwondo in order to win the game. There is a problem that there is no choice but to choose a game management method in which the technical elements of the game are significantly deteriorating.

As a result, not only the players who directly manage the game, the leaders who provide tactical guidance, and the parents who watch, but also the spectators who watch and hope for an exciting Taekwondo game as a sport will gradually turn away from Taekwondo games, and furthermore, the international There is even a sense of crisis that one could face the risk of being expelled from competitions (Asian Games, Olympics) if one is not careful.

The problem to be solved by the present invention is to solve the above-mentioned conventional problems, by scanning a martial arts game through a plurality of sensors installed in the stadium, a plurality of sensor data is generated, and this is analyzed to analyze the combat sports. By conducting evaluations (games, judging) on martial arts games, such as calculating scores for each player, the inconvenience felt by athletes can be alleviated in that they do not have to attach sensors directly, and the sensors can be damaged by blows. The aim is to provide a martial arts artificial intelligence judgment system and method using image information that can prevent incorrect judgments from being made.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

In order to solve the above-described problem, a martial arts artificial intelligence judgment system using image information according to an embodiment of the present invention is a martial arts artificial intelligence judgment system using image information, which scans a martial arts match taking place in a predetermined space. Using a plurality of sensor modules that generate sensor data and a previously learned artificial intelligence model, scores for players who participated in the martial arts game are calculated by analyzing the plurality of sensor data generated from each of the plurality of sensor modules. It can include servers that do.

In various embodiments, the plurality of sensor modules include a plurality of camera modules installed at different positions within a predetermined space, and the plurality of sensor data is transmitted to each other across the predetermined space through the plurality of camera modules. It includes a plurality of image data generated by shooting from different viewpoints, wherein the server extracts a plurality of two-dimensional skeleton information about players participating in the martial arts game from each of the plurality of image data, and the extracted plurality 3D coordinate information is generated using 2D skeleton information, and the 3D coordinate information generated using image data collected over a predetermined period is analyzed to extract motion information of players participating in the martial arts game, Based on the extracted motion information, scores to be awarded to players participating in the martial arts game can be calculated.

In various embodiments, the server determines whether a blow has occurred between players participating in the martial arts game, and when it is determined that a blow has occurred between players participating in the martial arts game, the server analyzes the plurality of sensor data and An indicator regarding the generated hit can be extracted, and a score for the generated hit can be calculated using the extracted indicator.

In various embodiments, when the server determines that the first player participating in the martial arts game has struck the second player, the server based on the motion information of the first player extracted from the plurality of sensor data, A hitting technique may be determined as an indicator of a hit applied by a first player to the second player, and a score to be awarded to the first player may be calculated based on the determined hitting technique.

In various embodiments, when the server determines that the first player participating in the martial arts game has struck the second player, the server based on the motion information of the first player extracted from the plurality of sensor data, The hitting area may be determined as an indicator of the hit applied by the first player to the second player, and a score to be awarded to the first player may be calculated based on the determined hitting area.

In various embodiments, when it is determined that the first athlete participating in the martial arts game has struck the second athlete, the server determines the striking speed based on the motion information of the first athlete extracted from the plurality of sensor data. and calculate the striking angle, and calculate the amount of impact as an indicator of the hit applied by the first athlete to the first athlete based on the calculated striking speed and angle and the weight class of the martial arts game, and calculate the amount of impact as an indicator of the hit applied by the first athlete to the first athlete, and the calculated amount of impact Based on this, the score to be awarded to the first player can be calculated.

In various embodiments, when the server determines that the first player participating in the martial arts game has struck the second player, the server based on the motion information of the first player extracted from the plurality of sensor data, The number of rotations may be calculated as an indicator of the blow applied by the first player to the second player, and the score to be awarded to the first player may be calculated based on the calculated number of rotations.

In various embodiments, when the server determines that the first player participating in the martial arts game has struck the second player, the server based on the motion information of the second player extracted from the plurality of sensor data, To determine the movement of the second player corresponding to the blow inflicted by the first player to the second player, and to verify the blow inflicted by the first player to the second player based on the determined movement of the second player. You can.

In various embodiments, when it is determined that a blow has occurred between players participating in the martial arts game, the server uses the terminal of an assistant referee registered in advance for the martial arts game to select the generated data among the plurality of sensor data. Provides at least one sensor data containing information about a hit and information about a score calculated for the generated hit, and provides the generated information based on feedback obtained from the assistant referee's terminal in response to the provided information. Verification of the score calculated for hitting can be performed.

In order to solve the above-described problem, the martial arts artificial intelligence decision method using image information according to another embodiment of the present invention utilizes image information including a plurality of sensor modules and servers installed at different locations within a predetermined space. A method performed through a martial arts artificial intelligence judgment system, generating a plurality of sensor data by scanning a martial arts game taking place in the predetermined space through the plurality of sensor modules, and using the server to learn It may include calculating scores for players who participated in the martial arts game by analyzing the plurality of sensor data generated using the artificial intelligence model.

Other specific details of the invention are included in the detailed description and drawings.

According to various embodiments of the present invention, a plurality of sensor data is generated as a martial arts game is scanned through a plurality of sensors installed in the stadium, and this is analyzed to calculate scores for fighters participating in the martial arts game. By conducting a review, there is an advantage in that the inconvenience felt by the players can be alleviated in that they do not have to attach the sensors themselves, and it is possible to prevent incorrect decisions due to sensors being damaged by hitting.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram illustrating a martial arts artificial intelligence judgment system using image information according to an embodiment of the present invention.
Figure 2 is a diagram showing the hardware configuration of a server that performs a martial arts artificial intelligence decision method using image information according to another embodiment of the present invention.
Figure 3 is a flowchart of a martial arts artificial intelligence determination method using image information according to another embodiment of the present invention.
FIG. 4 is a diagram illustrating a plurality of sensor modules installed in a stadium where a martial arts game is held in various embodiments.
Figure 5 is a flowchart illustrating a method of calculating a score using a plurality of image data in various embodiments.
FIG. 6 is a diagram illustrating two-dimensional skeleton information extracted from image data in various embodiments.
FIG. 7 is a diagram illustrating 3D coordinate information generated using a plurality of 2D skeleton information in various embodiments.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

As used in the specification, the term “unit” or “module” refers to a hardware component such as software, FPGA, or ASIC, and the “unit” or “module” performs certain roles. However, “part” or “module” is not limited to software or hardware. A “unit” or “module” may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a “part” or “module” refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, Includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within components and “parts” or “modules” can be combined into smaller components and “parts” or “modules” or into additional components and “parts” or “modules”. Could be further separated.

Spatially relative terms such as “below”, “beneath”, “lower”, “above”, “upper”, etc. are used as a single term as shown in the drawing. It can be used to easily describe the correlation between a component and other components. Spatially relative terms should be understood as terms that include different directions of components during use or operation in addition to the directions shown in the drawings. For example, if a component shown in a drawing is flipped over, a component described as “below” or “beneath” another component will be placed “above” the other component. You can. Accordingly, the illustrative term “down” may include both downward and upward directions. Components can also be oriented in other directions, so spatially relative terms can be interpreted according to orientation.

In this specification, a computer refers to all types of hardware devices including at least one processor, and depending on the embodiment, it may be understood as encompassing software configurations that operate on the hardware device. For example, a computer can be understood to include, but is not limited to, a smartphone, tablet PC, desktop, laptop, and user clients and applications running on each device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Each step described in this specification is described as being performed by a computer, but the subject of each step is not limited thereto, and depending on the embodiment, at least part of each step may be performed in a different device.

Figure 1 is a diagram illustrating a martial arts artificial intelligence judgment system using image information according to an embodiment of the present invention.

Referring to FIG. 1, the martial arts artificial intelligence judgment system using image information according to an embodiment of the present invention includes a server 100, a plurality of sensor modules 200, a user terminal 300, an external server 400, and It may include a network 500.

Here, the martial arts artificial intelligence judgment system using the image information shown in FIG. 1 is according to one embodiment, and its components are not limited to the embodiment shown in FIG. 1, and can be added, changed, or deleted as necessary. It can be.

In one embodiment, the server 100 analyzes sensor data generated by scanning a martial arts game taking place in a predetermined space (e.g., a stadium where a martial arts game is held) to screen athletes participating in the martial arts game. You can proceed.

In various embodiments, the server 100 proceeds with a martial arts game by analyzing a plurality of sensor data generated by scanning a predetermined space using a plurality of sensor modules 200 using a previously learned artificial intelligence model. You can calculate scores for players who do this.

Here, the pre-trained artificial intelligence model is a model learned using as learning data a plurality of sensor data containing objects and labeled with coordinate information for the objects, and using specific sensor data as input data to identify objects included in specific sensor data. It may be a model that extracts coordinate information corresponding to , that is, a skeleton extraction model. However, it is not limited to this, and the previously learned artificial intelligence model includes a skeleton extraction model, a model that models 3D coordinate information based on 2D skeleton information extracted through the skeleton extraction model, and a model that models 3D coordinate information based on 3D coordinate information. It may further include a model for determining posture and a rule-base model for awarding scores based on the determined motion and posture.

An artificial intelligence model (e.g. a neural network) consists of one or more network functions, and one or more network functions may consist of a set of interconnected computational units, which can generally be referred to as ‘nodes’. These ‘nodes’ may also be referred to as ‘neurons’. One or more network functions are composed of at least one or more nodes. Nodes (or neurons) that make up one or more network functions may be interconnected by one or more ‘links’.

Within an artificial intelligence model, one or more nodes connected through a link can form a relative input node and output node relationship. The concepts of input node and output node are relative, and any node in an output node relationship with one node may be in an input node relationship with another node, and vice versa. As described above, input node to output node relationships can be created around links. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, the nodes connecting the input node and the output node may have a weight. Weights may be variable and may be varied by the user or algorithm in order for the artificial intelligence model to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. The output node value can be determined based on the weight.

As described above, in an artificial intelligence model, one or more nodes are interconnected through one or more links to form an input node and output node relationship within the artificial intelligence model. The characteristics of the artificial intelligence model can be determined according to the number of nodes and links within the artificial intelligence model, the correlation between nodes and links, and the weight value assigned to each link. For example, if there are two artificial intelligence models with the same number of nodes and links and different weight values between the links, the two artificial intelligence models may be recognized as different from each other.

Some of the nodes constituting the artificial intelligence model may form one layer based on the distances from the initial input node. For example, a set of nodes with a distance n from the initial input node may constitute n layers. The distance from the initial input node can be defined by the minimum number of links that must be passed to reach the node from the initial input node. However, this definition of the layer is arbitrary for explanation purposes, and the order of the layer within the artificial intelligence model may be defined in a different way than described above. For example, a layer of nodes may be defined by distance from the final output node.

The initial input node may refer to one or more nodes in the artificial intelligence model into which data is directly input without going through links in relationships with other nodes. Alternatively, in an artificial intelligence model network, in the relationship between nodes based on links, it may mean nodes that do not have other input nodes connected by links. Similarly, the final output node may refer to one or more nodes that do not have an output node in their relationship with other nodes among the nodes in the artificial intelligence model. Additionally, hidden nodes may refer to nodes that constitute an artificial intelligence model rather than the first input node and the last output node. The artificial intelligence model according to an embodiment of the present invention may have more nodes in the input layer than the nodes in the hidden layer close to the output layer, and is an artificial intelligence model in which the number of nodes decreases as it progresses from the input layer to the hidden layer. You can.

An artificial intelligence model may include one or more hidden layers. The hidden node of the hidden layer can take the output of the previous layer and the output of surrounding hidden nodes as input. The number of hidden nodes for each hidden layer may be the same or different. The number of nodes in the input layer may be determined based on the number of data fields of the input data and may be the same as or different from the number of hidden nodes. Input data input to the input layer can be operated by the hidden node of the hidden layer and output by the fully connected layer (FCL), which is the output layer.

In various embodiments, the artificial intelligence model may be a deep learning model.

A deep learning model (e.g., deep neural network (DNN)) may refer to an artificial intelligence model that includes multiple hidden layers in addition to the input layer and output layer. Using a deep neural network, data Latent structures can be identified, that is, the latent structures of photos, text, video, voice, and music (e.g., what objects are in the photo, what the content and emotion of the text are, and what the sound of the voice is like). content, emotions, etc.).

Deep neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), auto encoders, generative adversarial networks (GAN), and restricted Boltzmann machines (RBMs). Boltzmann machine), deep belief network (DBN), Q network, U network, Siamese network, etc., but are not limited to these.

In various embodiments, the server 100 may be connected to the user terminal 300 through the network 500, and the results derived by performing the martial arts artificial intelligence determination method using image information according to various embodiments of the present invention. The results may be provided to the user terminal 300.

For example, the user terminal 300 may be the terminal of a referee and an assistant referee for a martial arts game held in a predetermined space, and the server 100 may be a player calculated by performing a martial arts artificial intelligence judgment method using video information. By providing information about the scores of players to the user terminal 300, it is possible to assist referees and assistant referees of martial arts games to more conveniently and accurately perform evaluations of martial arts games.

As another example, the user terminal 300 is placed in a predetermined space and may be a display device that outputs information about a martial arts game taking place in a predetermined space, and the server 100 makes a martial arts artificial intelligence decision using video information. By providing information about the players' scores calculated by performing the method to the user terminal 300, spectators watching a martial arts game can check the players' scores in real time.

Here, the user terminal 300 may refer to any type of entity(s) in the system that has a mechanism for communication with the server 100. For example, the user terminal 300 includes a personal computer (PC), a note book, a mobile terminal, a smart phone, a tablet PC, and a wearable device. etc., and may include all types of terminals that can access wired/wireless networks. Additionally, the user terminal 300 may include any computing device implemented by at least one of an agent, an application programming interface (API), and a plug-in. Additionally, the user terminal 300 may include an application source and/or a client application.

Also, here, the network 500 may mean a connection structure that allows information exchange between nodes, such as a plurality of terminals and servers. For example, the network 500 includes a local area network (LAN), a wide area network (WAN), the World Wide Web (WWW), a wired and wireless data communication network, a telephone network, and a wired and wireless television communication network. can do.

Wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, and Internet. (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC (Near- Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc., but is not limited thereto.

In one embodiment, the plurality of sensor modules 200 may be installed at different locations within a predetermined space and may generate sensor data by scanning a martial arts game taking place in a predetermined space.

Here, referring to FIG. 1, the plurality of sensor modules 200 are a plurality of camera modules installed at different positions within a predetermined space, and the plurality of sensor data is transmitted from different viewpoints in the predetermined space through the plurality of camera modules. It may be a plurality of image data generated by shooting at a (point of view), but is not limited to this, and various types of sensors such as a plurality of point cloud data generated by scanning a predetermined space through a lidar sensor module Sensor data may be applied.

In addition, here, as shown in FIG. 4, eight sensor modules 200 may be installed in a predetermined space, but the installation location and number of sensor modules 200 are not limited to this.

In various embodiments, the plurality of sensor modules 200 may be installed at a height of about 1 m to 1.5 m relative to the ground, but the installation is not limited thereto.

In one embodiment, the external server 400 may be connected to the server 100 through the network 500, and various information and data necessary for the server 100 to perform a martial arts artificial intelligence determination method using image information. can be stored and managed, or the results obtained as the server 100 performs a martial arts artificial intelligence judgment method using video information can be provided, stored and managed. For example, the external server 400 may be a storage server separately provided outside the server 100, but is not limited thereto. Hereinafter, with reference to FIG. 2, the hardware configuration of the server 100 that performs the martial arts artificial intelligence decision method using image information will be described.

Figure 2 is a diagram showing the hardware configuration of a server that performs a martial arts artificial intelligence decision method using image information according to another embodiment of the present invention.

Referring to FIG. 2, the server 100 includes one or more processors 110, a memory 120 that loads a computer program 151 executed by the processor 110, a bus 130, and a communication interface ( 140) and a storage 150 for storing a computer program 151. Here, only components related to the embodiment of the present invention are shown in Figure 2. Accordingly, anyone skilled in the art to which the present invention pertains will know that other general-purpose components may be included in addition to the components shown in FIG. 2.

The processor 110 controls the overall operation of each component of the server 100. The processor 110 includes a Central Processing Unit (CPU), Micro Processor Unit (MPU), Micro Controller Unit (MCU), Graphic Processing Unit (GPU), or any other type of processor well known in the art of the present invention. It can be.

Additionally, the processor 110 may perform operations on at least one application or program for executing methods according to embodiments of the present invention, and the server 100 may include one or more processors.

In various embodiments, the processor 110 includes random access memory (RAM) (not shown) and read memory (ROM) that temporarily and/or permanently store signals (or data) processed within the processor 110. -Only Memory, not shown) may be further included. Additionally, the processor 110 may be implemented in the form of a system on chip (SoC) that includes at least one of a graphics processing unit, RAM, and ROM.

Memory 120 stores various data, commands and/or information. Memory 120 may load a computer program 151 from storage 150 to execute methods/operations according to various embodiments of the present invention. When the computer program 151 is loaded into the memory 120, the processor 110 can perform the method/operation by executing one or more instructions constituting the computer program 151. The memory 120 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

The bus 130 provides communication functions between components of the server 100. The bus 130 may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

The communication interface 140 supports wired and wireless Internet communication of the server 100. Additionally, the communication interface 140 may support various communication methods other than Internet communication. To this end, the communication interface 140 may be configured to include a communication module well known in the technical field of the present invention. In some embodiments, communication interface 140 may be omitted.

Storage 150 may store the computer program 151 non-temporarily. When performing a martial arts artificial intelligence decision process using video information through the server 100, the storage 150 can store various information necessary to provide a martial arts artificial intelligence judgment process using video information.

The storage 150 is a non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or a device well known in the art to which the present invention pertains. It may be configured to include any known type of computer-readable recording medium.

The computer program 151, when loaded into the memory 120, may include one or more instructions that cause the processor 110 to perform methods/operations according to various embodiments of the present invention. That is, the processor 110 can perform the method/operation according to various embodiments of the present invention by executing the one or more instructions.

In one embodiment, the computer program 151 generates a plurality of sensor data by scanning a martial arts game taking place in a predetermined space and analyzes the plurality of sensor data generated using a previously learned artificial intelligence model. Accordingly, it may include one or more instructions for performing a martial arts artificial intelligence judgment method using video information, including the step of calculating scores for players participating in a martial arts game.

The steps of the method or algorithm described in connection with embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

The components of the present invention may be implemented as a program (or application) and stored in a medium in order to be executed in conjunction with a hardware computer. Components of the invention may be implemented as software programming or software elements, and similarly, embodiments may include various algorithms implemented as combinations of data structures, processes, routines or other programming constructs, such as C, C++, , may be implemented in a programming or scripting language such as Java, assembler, etc. Functional aspects may be implemented as algorithms running on one or more processors. Hereinafter, with reference to FIGS. 3 to 7, a method for determining martial arts artificial intelligence using image information performed by the server 100 will be described.

Figure 3 is a flowchart of a martial arts artificial intelligence determination method using image information according to another embodiment of the present invention.

Referring to FIG. 3, in step S110, the server 100 may collect a plurality of sensor data about a predetermined space where a martial arts game is held. For example, as shown in FIG. 4, a plurality of sensor modules 200 may be installed at different positions, angles, and heights within a predetermined space, and the server 100 may be installed with a plurality of sensor modules installed within a predetermined space ( 200), a plurality of sensor data generated by scanning a predetermined space through a plurality of sensor modules 200 can be collected.

In step S120, the server 100 may analyze a plurality of sensor data collected through step S110 to make a decision about a martial arts match taking place in a predetermined space. Here, judging a martial arts game may mean calculating scores for players participating in a martial arts game.

In various embodiments, the server 100 may analyze a plurality of sensor data using a previously learned artificial intelligence model to calculate scores to be awarded to athletes participating in a martial arts game. Hereinafter, it will be described with reference to FIGS. 5 to 7.

Figure 5 is a flowchart illustrating a method of calculating a score using a plurality of image data in various embodiments.

Referring to FIG. 5, in step S210, when a plurality of image data is generated through a plurality of camera modules installed in a predetermined space, the computing device 100 provides information about players participating in a martial arts game from each of the plurality of image data. Multiple two-dimensional skeleton information can be extracted.

Here, the two-dimensional skeleton information may include information about a plurality of points and a line between the two points.

More specifically, two-dimensional skeleton information can be expressed as 30 points and a line connecting two different points, as shown in FIG. 6.

First, in the head area, point 0 may represent the part corresponding to the nose (or philtrum), point 1 may represent the part corresponding to the left ear (or temple), and point 2 may represent the part corresponding to the left ear (or temple). It can represent the corresponding part corresponding to the eye, point 3 can represent the part corresponding to the right eye, point 4 can represent the part corresponding to the right ear (or temple), and point 5 can represent the part corresponding to the tip of the chin.

Next, in the shoulder area, point 6 can represent the part corresponding to the center of the neck or the ulnar bone, point 7 can represent the part corresponding to the left shoulder, and point 12 can represent the part corresponding to the right shoulder. It can indicate the part that is

Next, in the arm area, point 8 can represent the part corresponding to the left elbow, point 9 can represent the part corresponding to the left wrist, and point 10 can represent the part corresponding to the tip of the left thumb. point 11 can represent the part corresponding to the tip of the left middle finger.

Additionally, point 13 may represent the part corresponding to the right elbow, point 14 may represent the part corresponding to the right wrist, point 15 may represent the part corresponding to the tip of the right thumb, and point 16 The point may represent the part corresponding to the tip of the right middle finger.

Next, in the torso region, point 17 may represent a part corresponding to the center of the chest (or solar plexus), and point 18 may represent a part corresponding to the center of the waist (or dan tian).

Finally, in the leg area, point 19 can represent the part corresponding to the center of the left pelvis (or hip), point 20 can indicate the part corresponding to the left knee, and point 21 can represent the part corresponding to the left ankle. can represent the part corresponding to, point 22 can represent the part corresponding to the heel of the left foot, point 23 can represent the part corresponding to the tip of the left big toe, and point 24 can represent the part corresponding to the tip of the left little toe. It can indicate the part that is

Additionally, point 25 may represent the part corresponding to the center of the right pelvis (or hip), point 26 may indicate the part corresponding to the right knee, and point 27 may indicate the part corresponding to the right ankle. Point 28 can represent the part corresponding to the heel of the right foot, point 29 can represent the part corresponding to the tip of the right big toe, and point 30 can represent the part corresponding to the tip of the right little toe. .

Here, the point (numbered 0 to 30) may be expressed as an edge.

Here, a method of extracting skeleton information of an object included in image data from image data (e.g., image) is described in “Cheol-min Lee, and Dong-myeong Lee. “On extracting body joint coordinates for user action recognition in YOLO-Open Pose environment. Research. "Korea Communications Society Conference Proceedings (2020): 174-175." and "Yunmin Cho, Seunggyu Lee. (2019). Extraction of 2D human joints using machine learning and application of 3D movement using Kinect. Proceedings of the Korea Information Science Society, (), 1711-1713." Various technologies are known, and these known technologies can be selectively applied, so this specification does not describe a specific method of extracting two-dimensional skeleton information.

In step S220, the server 100 may generate 3D coordinate information (eg, FIG. 7) using a plurality of 2D skeleton information extracted through step S210. For example, the server 100 may generate one 3D coordinate information for a specific athlete by merging 2D skeleton information extracted from image data taken from different points of view of the specific athlete. . That is, the server 100 may perform 3D modeling on a specific object using a plurality of 2D skeleton information on the specific object.

In step S230, the server 100 may extract motion information of players participating in a martial arts game using the 3D coordinate information generated through step S220.

Here, the player's motion information extracted from the 3D coordinate information may include information about the player's current posture as well as information about the player's movements and actions.

For example, the server 100 generates 3D coordinate information using image data generated by filming a specific athlete at a specific time, and uses the 3D coordinate information to determine the number of images taken by a specific athlete at a specific time. The posture can be determined, and information about the posture of a specific player can be extracted as motion information of the specific player.

As another example, the server 100 generates a plurality of 3D coordinate information using image data generated by photographing a specific athlete over a predetermined period of time, and uses each of the plurality of 3D coordinate information to determine multiple 3D coordinate information for a specific athlete. Postural information can be extracted, and by analyzing multiple postural information according to time series, the movement and behavior information of a specific player can be determined over a certain period of time, and the movement and behavior information of a specific player can be converted into the movement information of a specific player. It can be extracted as.

In step S240, the server 100 may determine whether a blow has occurred between players participating in a martial arts game.

In various embodiments, the server 100 may determine whether a hit has occurred between players participating in a martial arts game based on motion information of the players participating in the martial arts game. For example, for a martial arts game in which a first player and a second player participate, the server 100 may determine whether the first player and/or the second player strikes based on the motion information of the first player and the motion information of the second player. If it is determined that the action has been performed, it may be determined that a strike occurred between the first player and the second player.

At this time, if the server 100 determines that the first player has performed an action of hitting the second player based on the motion information of the first player, the first player has hit the second player (or The blow is judged to have occurred in the direction from the first player to the second player, but the blow was applied by the first player to the second player based on the movement of the second player in response to the blow applied by the first player to the second player. The effectiveness of can be verified.

For example, the server 100 determines that the first player has performed a hitting action on the second player based on the motion information of the first player, and based on the motion information of the second player, the second player has performed a hitting motion. Only when it is determined that a receiving action has been performed, the blow inflicted by the first player on the second player is judged to be valid, and accordingly, it can be determined that the first player has inflicted a blow on the second player. However, it is not limited to this.

In various embodiments, the server 100 may determine whether a hit has occurred between players participating in a martial arts game based on whether there is contact and/or overlap in areas corresponding to the players participating in the martial arts game.

More specifically, first, the server 100 analyzes video data captured in the martial arts game in which the first player and the second player participated, and analyzes the area corresponding to the first player and the area corresponding to the second player. can be set. For example, the server 100 may generate 3D coordinate information corresponding to the first athlete using a plurality of image data, and define the area defined by the 3D coordinate information of the first athlete as the body area of the first athlete. It can be set to . Additionally, the server 100 may generate 3D coordinate information corresponding to the second athlete using a plurality of image data, and define the area defined by the 3D coordinate information of the second athlete as the body area of the second athlete. It can be set to . However, it is not limited to this.

Thereafter, the server 100 may determine that a hit has occurred between the first player and the second player when at least a portion of the area corresponding to the first player and the area corresponding to the second player contact or overlap. .

In various embodiments, when it is determined that a strike occurred between players participating in a martial arts game, the server 100 may verify whether a strike occurred based on voice information collected at the time the strike occurred. For example, each of the plurality of sensor modules 200 may include a plurality of camera modules for collecting a plurality of image data and a plurality of microphone modules for collecting a plurality of voice data, and the server 100 may collect the plurality of image data. When a blow occurs between players participating in a martial arts game, sound information at the time the blow occurs can be extracted based on a plurality of voice data, and sound information at the time the blow occurs can be extracted based on the extracted sound information. If the volume of the sound is greater than the standard value, it can be determined that a blow occurred between players participating in a martial arts game. Meanwhile, the server 100 may determine that a strike between players participating in a martial arts game has not occurred if the volume of the sound generated at the time the strike occurs is less than the standard value.

In step S250, if it is determined that a blow occurred between players participating in the martial arts game through step S240, the server 100 uses a plurality of video data to determine the blow that occurred between the players participating in the martial arts game. Related indicators can be extracted.

In various embodiments, in a martial arts game in which a first athlete and a second athlete participate, when the server 100 determines that the first athlete has struck the second athlete, the server 100 strikes based on the motion information of the first athlete. As an indicator, batting technique can be judged. Here, the striking technique may include, but is not limited to, striking using the hand (e.g., punching) and striking using the foot (e.g., straight kick, rotating kick, etc.).

In various embodiments, in a martial arts game in which a first athlete and a second athlete participate, when the server 100 determines that the first athlete has struck the second athlete, the server 100 strikes based on the motion information of the first athlete. As an indicator, the hitting area can be determined. Here, the hitting area may be the head, torso, and arms, but is not limited thereto.

In various embodiments, in a martial arts game in which a first player and a second player participate, the server 100, when it is determined that the first player has struck the second player, strikes based on the motion information of the first player. As an indicator, the amount of impact can be determined. For example, the server 100 calculates the striking speed and striking angle based on the motion information of the first athlete, and calculates the amount of impact for the blow applied by the first athlete based on the striking speed and striking angle and the weight class of the martial arts game. You can.

At this time, when it is determined that the first player hit the second player, the server 100 provides sound information (e.g., sound information extracted from a plurality of voice data) at the time the first player hit the second player. ), the size of the impulse can be corrected. For example, the server 100 may define and store in advance the sound volume value according to the amount of impact, and when it is determined that the first player has hit the second player, the server 100 can determine the value of the sound volume according to the amount of impact at the time the hit occurred from the pre-stored data. The amount of impact corresponding to the size of the generated sound can be extracted, and the size of the calculated amount of impact can be corrected based on the motion information of the first athlete based on the amount of impact extracted from pre-stored data (e.g., in advance Calculation of the average value of the impulse extracted from stored data and the impulse calculated based on the motion information of the first athlete, etc.)

In various embodiments, in a martial arts game in which a first athlete and a second athlete participate, when the server 100 determines that the first athlete has struck the second athlete, the server 100 determines the amount of impact of the first athlete as an indicator of the blow. The impact amount of the player and the second player can be calculated individually. For example, when it is determined that the first player has hit the second player with the instep, the server 100 may calculate the acceleration of the first player's kick by tracking the trajectory of the first player's instep, Based on this, the amount of impact of the first player can be estimated. In addition, when it is determined that the second player has received a blow to the head from the first player, the server 100 can track the trajectory of the head and determine whether the second player's head trajectory is an avoidance motion or is moved by the blow. Then, the amount of impact the second player would have suffered can be relatively evaluated and calculated.

In various embodiments, in a martial arts game in which a first athlete and a second athlete participate, when the server 100 determines that the first athlete has struck the second athlete, the server 100 strikes based on the motion information of the first athlete. As an indicator for , the number of rotations can be calculated.

Meanwhile, in step S240, if the server 100 determines that no blow has occurred between the players participating in the martial arts game, the server 100 operates to extract motion information of the players until a hit occurs among the players participating in the martial arts game. The operation of determining whether an overstroke has occurred can be performed repeatedly.

In step S260, the server 100 may calculate a score to be awarded to the player who made the hit using the index regarding the hit extracted through step S250.

In various embodiments, a score table for each indicator related to hitting (e.g., score data for each hitting technique, score data for each hitting part, score data for each impact amount, score data for each rotation, etc.) may be defined in advance, and the server 100 When the index related to the hit inflicted by the first player to the second player is extracted as it is determined that the first player hit the second player, the index is extracted based on a predefined score table for each index related to the hit. The score corresponding to the given indicator can be extracted.

In various embodiments, when a plurality of indicators are extracted regarding a hit inflicted by a first player to a second player, the server 100 calculates a unit score corresponding to each of the plurality of indicators based on a score table for each indicator related to the hit. It can be extracted, and the extracted unit scores can be added to finally calculate the score for the blow inflicted by the first player to the second player.

In various embodiments, when a plurality of indicators are extracted regarding a blow inflicted by the first player to the second player, the server 100 provides a score to be awarded to the first player based on a combination of indicators including the plurality of extracted indicators. can be calculated. For example, the server 100 may predefine score data for each combination of indicators including indicators of different attributes (e.g., type and number), and send the score corresponding to the combination of indicators extracted with respect to hitting to the first player. It can be extracted as a score to be given to .

In various embodiments, when it is determined that a blow has occurred between players participating in a martial arts game, the server 100 sends a user terminal 200 of a pre-registered assistant referee for the martial arts game to a strike among a plurality of sensor data. At least one sensor data containing information about (eg, image data including a hitting scene) and information about a score calculated for the hitting may be provided.

At this time, the server 100 may obtain feedback from the assistant referee in response to providing at least one sensor data and information about the score to the assistant referee's user terminal 200, and the assistant referee's user terminal 200 Verification of the score calculated for hitting can be performed based on the feedback obtained from.

That is, the server 100 performs a process of receiving confirmation from the referee in charge of the martial arts game whether the score derived through video data analysis is accurately calculated, thereby allowing the score to be calculated and reflected more accurately.

In various embodiments, when the score regarding the hit applied by the first player to the second player is calculated and confirmed, the server 100 may provide information regarding the hit applied by the first player to the second player (e.g., hitting time, hit) Information on the player who struck, the score of the hit, indicators related to the hit, etc.) can be recorded in chronological order.

In various embodiments, the server 100 analyzes a plurality of video data to primarily determine whether a hit has occurred between players participating in a martial arts game, and places a target at a point where it is predicted that a hit has occurred based on the results of the primary judgment. By secondarily determining whether a hit has occurred using only the corresponding image data, it is possible to more accurately determine whether a hit has occurred.

More specifically, first, the server 100 based on the motion information of the first player and the motion information of the second player derived by analyzing a plurality of video data with respect to the martial arts game in which the first player and the second player participated. Therefore, if it is primarily determined that the first player hit the second player, the hitting area of the hit that the first player inflicted on the second player can be determined as an indicator of the hit.

Thereafter, the server 100 selects at least one image data from a plurality of image data based on the striking part of the blow applied by the first player to the second player, and uses only the selected at least one image data to strike the first player. The motion information of and the motion information of the second player can be derived, and using this, it is possible to secondarily determine whether it is correct that the first player hit the second player.

In various embodiments, the server 100 randomly selects and combines N pieces of image data among a plurality of image data generated by filming a martial arts match in which the first athlete and the second athlete participate, thereby combining a plurality of image data. It can be generated, and each combination of multiple image data can be used individually to verify whether a hit has occurred between the first player and the second player.

More specifically, first, when eight pieces of image data are generated by filming a martial arts game using eight camera modules, the server 100 selects and combines three random pieces of image data among the eight image data. A combination of multiple image data can be created.

Thereafter, the server 100 may individually use each of the plurality of image data combinations to determine whether a hit has occurred between the first player and the second player, thereby deriving a plurality of result values.

Thereafter, the server 100 may collect a plurality of result values (e.g., soft voting or hard voting) to finally determine whether a hit has occurred between the first player and the second player.

In various embodiments, with respect to a martial arts game in which a first player and a second player participate, the server 100 determines that the first player has hit the second player and sends a message to the first player based on the index regarding the hit. Calculate the score to be awarded, but if an objection to the calculated score is received from an official related to the second player (e.g., the second player himself or the second player's manager, coach, etc.), the first player Information on hitting may be collected and provided to the user terminal 200 of the second player and related officials and/or assistant referees.

For example, when an objection to the score calculated for a blow inflicted by the first player to the second player is received, the server 100 displays a scene in which the first player strikes the second player in response to the objection. Original sensor data (e.g., image data), including 2-dimensional skeleton information (and/or 3-dimensional coordinate information) extracted from the original sensor data, and an indicator regarding the hit made by the first player to the second player (e.g., hit Collects information related to technique, hitting area, impact amount (impact amount of each first player and second player) and number of rotations), and loudness of sound corresponding to the hit (decibel, etc.), and collects information related to the user terminal of the officials and/or assistant referees ( 200), it is possible to verify the blow applied by the first player to the second player.

The martial arts artificial intelligence judgment method using the above-described video information was explained with reference to the flow chart shown in the drawing. For simple explanation, the martial arts artificial intelligence judgment method using image information is illustrated and explained as a series of blocks, but the present invention is not limited to the order of the blocks, and some blocks are in an order different from that shown and performed herein. It may be performed separately or simultaneously. Additionally, new blocks not described in this specification and drawings may be added, or some blocks may be deleted or changed.

Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: Server
200: sensor module
300: user terminal
400: external server
500: Network

Claims (10)

In a martial arts artificial intelligence judgment system using video information,
A plurality of sensor modules that generate sensor data by scanning a martial arts match taking place in a predetermined space; and
a server that calculates scores for players participating in the martial arts game by analyzing a plurality of sensor data generated from each of the plurality of sensor modules using a previously learned artificial intelligence model; Includes,
The server is,
Determine whether a blow occurred between players participating in the above martial arts game,
When it is determined that the first player participating in the martial arts game performed an action of striking the second player, the first player hits the second player based on the motion information of the second player extracted from the plurality of sensor data. Determine the movement of the second player in response to the blow applied to the second player,
If the determined movement of the second player is a movement of receiving a blow, the blow applied by the first player to the second player is judged to be valid,
When it is determined that a blow inflicted by the first player to the second player is effective, it is determined that a blow in the direction from the first player to the second player occurred,
A martial arts artificial intelligence judgment system using video information. According to paragraph 1,
The plurality of sensor modules are,
It includes a plurality of camera modules installed at different positions within a predetermined space,
The plurality of sensor data is,
Contains a plurality of image data generated by photographing the predetermined space from different viewpoints through the plurality of camera modules,
The server is,
Extracting a plurality of two-dimensional skeleton information about players participating in the martial arts game from each of the plurality of image data, and generating three-dimensional coordinate information using the extracted plurality of two-dimensional skeleton information,
By analyzing the 3D coordinate information generated using video data collected over a predetermined period of time, motion information of the players participating in the martial arts game is extracted, and based on the extracted motion information, the players participating in the martial arts game are Calculate the score to be awarded,
A martial arts artificial intelligence judgment system using video information. According to paragraph 1,
The server is,
If it is determined that a blow has occurred between players participating in the martial arts game, the plurality of sensor data is analyzed to extract an index regarding the hit, and the extracted index is used to determine the hit. calculating scores,
A martial arts artificial intelligence judgment system using video information. According to paragraph 3,
The server is,
When it is determined that the first player participating in the martial arts game struck the second player, the first player strikes the second player based on the motion information of the first player extracted from the plurality of sensor data. Determining the hitting technique as an indicator of the hit applied, and calculating the score to be awarded to the first player based on the determined hitting technique,
A martial arts artificial intelligence judgment system using video information. According to paragraph 3,
The server is,
When it is determined that the first player participating in the martial arts game struck the second player, the first player strikes the second player based on the motion information of the first player extracted from the plurality of sensor data. Determining the hitting area as an indicator of the hit applied, and calculating the score to be awarded to the first player based on the determined hitting area,
A martial arts artificial intelligence judgment system using video information. According to paragraph 3,
The server is,
When it is determined that the first player participating in the martial arts game has struck the second player, the hitting speed and hitting angle are calculated based on the motion information of the first player extracted from the plurality of sensor data, and the calculation Based on the striking speed and striking angle and the weight class of the martial arts game, the amount of impact is calculated as an indicator of the blow applied by the first athlete to the first athlete, and the amount of impact is to be given to the first athlete based on the calculated amount of impact. calculating scores,
A martial arts artificial intelligence judgment system using video information. According to paragraph 3,
The server is,
When it is determined that the first player participating in the martial arts game struck the second player, the first player strikes the second player based on the motion information of the first player extracted from the plurality of sensor data. Calculating the number of rotations as an indicator of the hit applied, and calculating the score to be awarded to the first player based on the calculated number of rotations,
A martial arts artificial intelligence judgment system using video information. delete According to paragraph 1,
The server is,
When it is determined that a blow has occurred between players participating in the martial arts game, at least one of the plurality of sensor data containing information about the occurred blow is sent to the terminal of an assistant referee registered in advance for the martial arts game. Provides information about the sensor data and the score calculated for the generated hit, and verifies the score calculated for the generated hit based on feedback obtained from the assistant referee's terminal in response to the provided information. To perform,
A martial arts artificial intelligence judgment system using video information. In a method performed through a martial arts artificial intelligence judgment system using image information including a plurality of sensor modules and servers installed at different locations within a predetermined space,
Generating a plurality of sensor data by scanning a martial arts game taking place in the predetermined space through the plurality of sensor modules; and
Comprising a step of calculating scores for players who participated in the martial arts game by analyzing the plurality of sensor data generated using a previously learned artificial intelligence model through the server,
The step of calculating the score is,
Including the step of determining whether a blow occurred between players participating in the martial arts game,
The step of determining whether the hit occurred is,
When it is determined that the first player participating in the martial arts game performed an action of striking the second player, the first player hits the second player based on the motion information of the second player extracted from the plurality of sensor data. determining the movement of the second player corresponding to a blow applied to the second player;
If the determined movement of the second player is a motion of receiving a blow, determining that the blow applied by the first player to the second player is effective; and
When it is determined that a blow inflicted by the first player to the second player is effective, determining that a blow in the direction from the first player to the second player has occurred,
Martial arts artificial intelligence judgment method using video information.