KR20210114815A - Deep Learning based score display system and operation method using mobile app - Google Patents

Abstract

An objective of the present invention is to provide a deep learning-based score display system using a mobile application, which accurately records a score even without a referee or athletic staff, and an operation method thereof. According to the present invention, the deep learning-based score display system comprises: a mobile terminal capable of collecting images and sounds of athletic matches between players, teams, or users; a mobile application transmitting the image and sound received from the mobile terminal to a deep learning server and displaying an analysis result to the players, teams, or users; and a server providing a model for extracting, analyzing, and reading mixed features of the image and sound received from the mobile application, and storing results and history of the athletic matches.

Description

Deep Learning based score display system and operation method using mobile app

An object of the present invention is to provide a score display system and operation method based on deep learning using a mobile app.

In more detail, when players, teams, or users play an athletic event, the video and sound generated from the athletic event are collected by a mobile terminal, and compared with the standard video and sound from the score and loss learned by deep learning,

It relates to a deep learning-based score display system and operation method for displaying the score determination result on a mobile app.

In general, the scoring of an athletic event is done by a referee or by agreement between an athletic event organizer, scorer, or a user who knows the rules. Scores are displayed using media such as number cards and electronic scoreboards, and the points earned between both teams, both players, or both users are displayed.

In addition, the score is displayed by the judgment of a referee or a person who plays a corresponding role in scoring and conceding by applying the rules of the competition or the rules provided by the party in charge of the athletic event.

The score display method as described above requires a person who always plays a role in determining the score and a score display medium.

In addition, there is a disadvantage that the judgment result for the score may be different depending on the standard interpretation and judgment of the individual serving as the judge.

In order to overcome the above shortcomings, a deep learning-based score display system that automatically determines the score by artificial intelligence using the video and sound generated in the situation of scoring and conceding between players, teams, or users during an athletic event; and You need a way to work.

Domestic Registered Patent No. 10-2039584: Machine learning-based scoring system and method using smart yoga mat

Recently, various deep learning applications are being attempted abroad. However, no non-patent literature could be found for a deep learning-based score display system and operation method using a mobile app only by mixing video and sound generated during an athletic event.

The present invention was devised to solve the above problems, by collecting images and sounds generated in the course of an athletic match between players, teams, or users, and classifying the scoring situation that results in scoring and loss by deep learning, and then determining it It is intended to provide results to players, teams or users via a mobile app.

In order to solve the above problems,

Deep learning-based score display system and operation method using a mobile app according to the present invention

a mobile terminal capable of collecting images and sounds of athletic matches between players, teams, or users;

a mobile app for transmitting the image and sound received from the mobile terminal to a server, and displaying a score result determined by the deep learning learning model;

a deep learning learning model for determining a score by extracting and analyzing the mixed features of the image and sound received from the mobile app, and a server for storing scores and athletic history of the players, teams or users;

characterized by including.

The deep learning learning model trains the image and sound for the standard score and loss in advance with a deep learning convolutional neural network (CNN), and when the input image and sound come in, the reference image and sound learned by the CNN learning model By comparing them, it is judged whether or not a score is scored based on the similarity.

The specific details of other embodiments are included in the specific contents and drawings for carrying out the invention.

According to the solution of the present invention, since the deep learning-based score display system using a mobile app is automatically determined only by video and sound, it is possible to accurately record the score even if there is no referee or an athletic event facilitator. It has the advantage of running smoothly.

In addition, even if there is no separate score recording medium, players, teams, or users can easily display them through the mobile app they have.

In addition, it is possible to check the athletic performance history of players, teams, or users, so that it is possible to determine whether one's own athletic performance is continuously improved.

1 is an overall schematic diagram of a deep learning-based score display system and operation method using a mobile app according to an embodiment of the present invention.
2 is a detailed configuration diagram of a mobile app according to an embodiment of the present invention.
3 is a detailed configuration diagram of a deep learning server according to an embodiment of the present invention.
4 is a detailed configuration diagram of a management server according to an embodiment of the present invention.
5 is a flowchart illustrating a functional flow of a CNN learning model according to an embodiment of the present invention.
6 is a diagram for explaining the structure of a CNN learning model according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily carry out the present invention.

However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein.

And, in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification. In the drawings, the same reference numbers are used for the same parts.

In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

When it is said that any one component "includes" another component in the detailed description or claims of the invention, it is not construed as being limited to only the component unless otherwise stated, and other components are not It should be understood that more may be included.

In addition, in describing the components of the present invention, the term “athletic competition” may be used.

These terms refer to various sporting events in which victory or defeat is determined through scores, and is not limited to official competitions such as table tennis, tennis, basketball, volleyball, baseball, and badminton. It should be understood that this may be applied to any type of athletic event as agreed upon between players, teams or users, such as friendly sports events and practice sports competitions.

In addition, in describing the components of the present invention, the term “user” may be used.

It should be understood that these terms can also be applied to players, referees, facilitators, scorers, third parties or stakeholders involved in the game as a person who uses a deep learning-based score display system using a mobile app.

In addition, in describing the components of the present invention, the implementation of the mobile app 200 is a native app (NATIVE APP), a mobile web (MOBILE WEB), a mobile web app (MOBILE WEB APP), a hybrid app (HYBRID APP) The general application implementation method of the mobile app can be applied. This does not limit the implementation of mobile apps in a specific way.

In addition, in describing the components of the present invention, the deep learning server 310 and the management server 320 may be physically implemented as one or more servers as a logical division.

In addition, in explaining the components of the present invention, the mobile terminal 100 is composed of one or more, and when it is composed of two or more, images and sounds from different angles for the same situation are collected and determined. It can be implemented in the form of a synthesis of the results.

Hereinafter, an example in which a deep learning-based score display system and an operating method using a mobile app according to the present invention are implemented will be described with reference to the accompanying drawings and specific embodiments.

The system may be changed in the same manner for various sports events such as table tennis, basketball, volleyball, tennis, and badminton, and will be described below as an embodiment in table tennis sports.

The scoring system for table tennis sports consists of a set for each game and a scoring point within each set. In a table tennis sport, the score of one game is composed of odd sets of 3, 5, 7, 9, etc., and is generally performed in 5 or 7 sets. In each set, if the first to score 11 points or a 10-10 tie results in a deuce, the first player to score 2 points in a row wins the set.

1 is an overall schematic diagram of a deep learning-based score display system using a mobile app according to an embodiment of the present invention.

As shown in the overall schematic diagram, the deep learning-based score display system using a mobile app includes a mobile terminal 100 , a mobile app 200 , and a server 300 .

The deep learning-based score display system using the mobile app 200 uses images and sounds generated in athletic competitions between players, teams, or users to determine the score by deep learning and displays the results to the players, teams, or users. It is a system that informs

The mobile terminal 100 may be a computer medium capable of operating an application program such as a mobile app.

In addition, the server 300 includes a deep learning server 310 that manages a deep learning learning model and a management server 320 that manages a score history for each player, team or user.

In addition, communication between the mobile app 200 and the server 300 may be connected and communicated through wired communication, wireless communication, or mobile communication.

Referring to FIG. 1 , in an athletic competition between players, teams, or users, images and sounds generated in the sports competition may be collected through the mobile terminal 100 and transmitted to the mobile app 200 .

In the mobile app 200, it is possible to determine a score by matching the image and sound that are scored or lost in the image and sound generated during the athletic event using the deep learning learning model learned in advance, respectively.

The deep learning learning model collects various images and sounds related to a standard score and a loss and trains it in the deep learning server 310 in advance. In addition, the learning model is divided into a deep learning learning model for images and a deep learning learning model for sound.

As shown in FIG. 5 , the determination of the score result of a score or a loss can be performed by dividing it into a case in which reading is possible only with an image, a case where reading is possible only with sound, and a case where reading is required using both image and sound.

Athletics video and sound, which are input for score determination in the learning model, are collected in real time and transmitted to the mobile app 200 and input, or pre-recorded video and sound of an athletic event are selected from the mobile app 200 so it can be entered.

The deep learning learning model managed by the server 300 extracts and analyzes features in a situation where a score is scored from an image and sound generated in a table tennis game and a situation where a score is scored, and scores for the input image and sound It detects the error between the standard image or the standard sound of a goal or loss and provides it to the player, team, or user as an interpreted result through the mobile app.

The deep learning learning model uses a separate CNN learning model 312 that can separately classify images and sounds, and can learn the reference images and sounds of scoring and missing points in an independent form.

In addition, information on which parts are scored and lost for each video and sound may be connected as separate information in advance, and this may be provided through the mobile app 200 .

In this way, the reference images and sounds judged by deep learning are images and sounds that can identify both scores and losses by various types of images and sounds occurring in athletic events, and when an input image is received, deep learning CNN learning The model 312 automatically maps and classifies the input image.

The calculation work for classification of the CNN learning model 312 may be performed in the mobile app 200 in the mobile terminal 100 on which the mobile app 200 is mounted, and only the result calculated by the server 300 It may be delivered to the mobile app 200 of the mobile terminal 100 .

In addition, the mobile app 200 provides a direct download path from the management server 320, or

It is provided through an application software download service (eg, App Store or Play Store, etc.), and players, teams, or users can download it to their respective mobile terminals 100 through wired communication, wireless communication, or mobile communication. .

Then, the player, the team, or the user decides whether to download the mobile app 200 from his/her mobile terminal 100 and receives the download.

As shown in FIG. 2 , the mobile app 200 configures user log-in functions 211 and 221 for managing the content and histories of their own athletic competition scores for each player, team, or user.

As shown in FIG. 4 , when a request for user logins 211 and 221 occurs, whether the requested user is a legitimate user is provided through the user authentication unit 321 in the management server 320 .

The video receivers 212 and 222 show images captured in real time by the mobile terminal 100 or recorded images selected by a player, team, or user. It plays the collected sounds or sounds from recorded videos selected by players, teams or users.

The score display units 214 and 224 of the mobile app 200 may display the score as a reference image and sound determined by deep learning.

In addition, the score information displayed to the player, team, or user is stored in the management server 320 through the score information storage (215, 225) function in order to manage individual histories.

Accordingly, the player, team, or user can view his/her user game history 322 stored in the database at any time through the mobile app 200 .

The mobile app providing unit 323 of the management server 320 may provide various statistical analyzes using the athletic score analysis and accumulated data history through the mobile app 200, and may be utilized by players, teams or users. It can also be provided as raw historical data.

For example, since it is stored in the database about the history of athletic matches between players, teams, or users, it is possible to provide information such as providing changes in athletics by using the data. In addition, it is to match the video and sound of the frequently occurring loss video or the main scoring point for each player, team, or user, and display it.

The similarity determination between the input image and the reference image to be scored or lost is identified as an error range. The error range can be set by a player, team, user, or manager by setting a standard arbitrarily. For example, if the reference error is set to 5%, the same image must be identified as having a similarity of 95%. Such an error can improve its accuracy through continuous learning and correction.

In addition, as shown in FIG. 3 , the deep learning server 310 includes a CNN learning model 312 used for deep learning and a management unit 311 for learning data. The learning data management unit 311 stores images and sounds in scores and losses, which are the criteria for determining images, as image data for each type.

In addition, the CNN learning model 312 stores information about weights, the number of nodes, the number of layers, the threshold, the objective function, and the activation function for the CNN model learned from the most recently collected reference image and sound.

The CNN learning models 312 manage the generated histories for each version through the learning model management unit 313 , and select the most suitable learning model for the player, team, user, or manager through the learning model providing unit 314 . provide you with

In addition, as shown in FIG. 5, the CNN learning model 312 is an input unit that converts data into a matrix-type numerical value for image and sound, respectively, a part that extracts the features of scoring and loss through the converted numerical value, classification It consists of a part to do, and a result part that provides classification for exercise postures. And when independent judgment is difficult in the video learning model or the sound learning model, mixed judgment is performed using the AND condition or OR condition by combining the results of both sides.

For example, an image when a ping-pong ball hits the body other than the racquet hand or when the ping-pong ball hits the net and fails to pass can be classified as a loss video. can be classified. In addition, if the ping-pong ball is thrown over the racquet side without a rubber attached, it is possible to determine whether a score has been scored by mixing the image and sound of hitting the ping-pong ball with the racquet.

Other than that, if your opponent fails to serve or return. When the ball, served or returned by the opponent, hits outside the net before it is bound to his court. When the opponent's ball crosses the court without bouncing. If your opponent blocks the ball. Your opponent hits the ball twice in a row. If your opponent makes the ping-pong table move or touches the net. You can learn the video and sound in advance in situations such as when the opponent touches the table tennis table with the hand that does not hold the racket.

6 is a schematic diagram illustrating a calculation process from a node of the deep learning CNN learning model 312 structure according to an embodiment of the present invention.

As shown in FIG. 6 , the CNN learning model 312 according to an embodiment of the present invention is a multi-layer neural network structure in which a convolution layer and a pooling layer are stacked several times.

In addition, the regions that classify images into appropriate categories by learning the finally extracted features well are all connected in a fully connected layer. This can be used appropriately for the classification of images and sounds.

Above, the technical idea of the present invention has been reviewed through specific implementations of the present invention.

In addition, even if not explicitly shown or described, a person of ordinary skill in the art to which the present invention pertains may make various modifications including the technical idea according to the present invention from the description of the present invention. is self-evident, which still falls within the scope of the present invention.

The above embodiments described with reference to the accompanying drawings have been described for the purpose of explaining the present invention, and the scope of the present invention is not limited to these embodiments.

100: user device
110: mobile terminal
120: personal computer
200 : mobile app
211 : user login
212: video receiver
213: sound receiver
214: score display unit
215: save score information
300 : server
310: Deep Learning Server
311: learning data management unit
312: CNN learning model
313: learning model management unit
314: learning model providing unit
320: management server
321: user authentication unit
322: user exercise history
323: mobile app provider

Claims (6)

Through the camera and microphone, the video and sound of the sports game between players, teams, or users with the mobile terminal;
a mobile app for transmitting the image and sound received from the mobile terminal to a server, and providing the determination result to a player, team, or user and a mobile terminal;
a server that provides a deep learning learning model for determining a score and a loss by extracting and analyzing the mixed features of the image and sound received from the mobile app, and storing the results and history of the athletic game;
Deep learning-based score display system and operation method using a mobile app, characterized in that it includes. According to claim 1,
The mobile app is
An application in the form of a mobile app, characterized in that it transmits the video and sound received from the mobile terminal to the server, and provides the result of scoring and conceding to a player, team, or user. According to claim 1,
the server
A system characterized in that it analyzes the mixed features of the image and the sound received from the mobile app with a deep learning learning model to provide a model for judging scores and losses, and provides the results and history of the athletic event. A first step of collecting sports video and sound through a mobile terminal;
a second step of transmitting the images and sounds collected from the mobile terminal to a server, and providing the results of scoring and conceding by deep learning to a mobile app of a player, team or user;
A third step of extracting and analyzing the mixed features of the image and sound received through the mobile app to provide a deep learning learning model that determines the score and the loss, and storing the results and history of the athletic game;
Deep learning-based score display operation method using a mobile app, characterized in that it further comprises. 5. The method of claim 4,
The deep learning-based score display operation method using the mobile app is,
After the first step,
Transmitting the image and sound received from the mobile terminal to the server,
providing a score and loss determination result by deep learning to a player, a team, or a user and a mobile terminal through a mobile app;
Method of operation, characterized in that it further comprises. 5. The method of claim 4,
The deep learning-based score display operation method using the mobile app is,
After the second step,
providing a deep learning learning model that extracts and analyzes mixed features of images and sounds delivered through the mobile app to determine a score and a loss, and storing the results and history of the athletic game;
Method of operation, characterized in that it further comprises.

Images