KR20210151232A - How a server that provides a sports video-based platform service operates - Google Patents

Abstract

A method of operating a server that provides a sports video-based platform service includes: tracking a ball using preprocessing that leaves dynamic pixels in a sports video for a ball game; determining an unidentified player associated with a score-related scene of a sports video by using the ball tracking result; identifying the non-identifying player by tracking the non-identifying player to an identifiable adjacent frame; and generating a time section of a sports video and identification information of an unidentified player in response to a score-related scene.

Description

How a server that provides a sports video-based platform service operates

The following embodiments relate to a method of operating a server that provides a sports video-based platform service.

In order to analyze sports videos, human resources by professional personnel are required at a considerable level. Several solutions for analyzing sports videos have been proposed, but these solutions still require dedicated filming equipment or a dedicated team. Therefore, there is a need for a technique for efficiently analyzing sports videos.

Embodiments provide a technique for remarkably reducing human resources required to analyze sports videos by using artificial intelligence technology. In addition, the embodiments provide a technique for searching sports videos in detail by linking the analysis results of sports videos with a corresponding scene and forming a database.

The method of operating a video analysis server according to one side comprises the steps of: receiving an analysis request signal including a link to a sports video of a ball game; filtering out static pixels from a plurality of frames included in the sports video and performing pre-processing to leave dynamic pixels; tracking the ball of the sports video based on the pre-processed video; detecting a score-related scene of the sports video from the pre-processed video; in response to detection of the scoring scene, determining an unidentified player associated with the scoring scene using the ball tracking result; identifying the unidentified player by tracking the unidentified player up to an adjacent frame in which the unidentified player is identifiable; and outputting a time section of the sports video and identification information of the unidentified player in response to the score-related scene.

Tracking the ball may include, corresponding to each of the frames, detecting the ball based on dynamic pixels of the corresponding frame.

The detecting of the score-related scene may include: corresponding to each of the frames, detecting a rim based on dynamic pixels of the corresponding frame; and determining frames adjacent to the frame in which the rim is detected as the score-related scenes.

The determining of the non-identified player associated with the score-related scene may include: detecting dynamic pixels related to the player who attempted to score in frames included in the scoring-related scene, using the ball tracking result; and determining a non-identified player associated with the scoring-related scene by instance-segmenting a frame in which dynamic pixels related to the player who attempted to score are detected.

The step of identifying the non-identified player may include extracting a feature from the determined non-identifying player; comparing the extracted features with features of previously registered players; determining whether the non-identified player can be identified as a result of the comparison; and tracking the unidentified player by instance segmenting adjacent frames according to a determination that the unidentified player cannot be identified.

The performing of the pre-processing may include: filtering out static pixels based on a change in pixel values between adjacent frames within a predetermined range when the sports video is a video shot from a fixed viewpoint; and filtering out static pixels based on statistical values of optical flow of pixels within a frame when the sports video is a video taken at a moving time point.

According to one aspect, a method of operating a server for providing a sports video-based platform service includes: transmitting a signal requesting analysis of the sports video to a video analysis module based on a link of the sports video; storing player-specific clusters received from the video analysis module in a database; providing information for extracting player-specific video clips from the sports video to a user terminal based on the database; receiving an input for identifying an unidentified player of at least one cluster from the user terminal provided with the video clips for each player from a streaming server; and updating identification information of at least one corresponding cluster in the database based on the input.

The method of operating a server providing the sports video-based platform service may include: providing statistics indicating the contribution of players to the user terminal; receiving an input for selecting detailed records included in the statistics from the user terminal; obtaining at least one sub-cluster associated with the selected detailed record based on the database; and providing information for extracting a video clip from the sports moving picture to the user terminal based on the at least one sub-cluster.

The method of operating a server for providing the sports video-based platform service includes: receiving a search query including a search target player and the search target scene from the user terminal; retrieving, from the database, a sub-cluster corresponding to the search query; and based on the found sub-cluster, providing information for extracting a video clip from the sports moving picture to the user terminal.

The method of operating a server providing the sports video-based platform service may include: determining a charging level based on the reliability of the clusters; and determining a compensation level based on a feedback input for modifying the clusters.

The method of operating a server for providing the sports video-based platform service includes: receiving a feedback input from the user that a player of at least one section included in the at least one cluster does not belong to the corresponding cluster; and excluding the corresponding section from the corresponding cluster based on the feedback input.

The method of operating a server for providing the sports video-based platform service may include: receiving, from the user, a feedback input indicating that a player of at least one section included in the at least one cluster belongs to another cluster; and excluding the corresponding section from the corresponding cluster and including the corresponding section in the other cluster based on the feedback input.

The method of operating a server for providing the sports video-based platform service includes: generating training data dependent on the updated database; and learning a specialized model for estimating at least one of detection information, identification information, and motion type information of players based on the training data.

1 is a view for explaining a system for providing a sports video-based platform service according to an embodiment.
2 is an operation flowchart illustrating a method of operating a service server that provides a sports video-based platform service according to an embodiment.
3 is a view for explaining a clustering operation according to an embodiment;
4 is a view for explaining a sports video-based platform service according to an embodiment.
5 is a view for explaining a sports video-based platform service according to an embodiment.
6 is a view for explaining distribution of feature vectors according to a motion type according to an embodiment;
7 is a view for explaining a clustering operation according to a motion type according to an embodiment;
8 is a view for explaining a general-purpose model and a specialized model according to an embodiment;
9 is a view for explaining video clips provided in association with statistics indicating a contribution level for each player in a basketball game according to an embodiment;
10 is a view for explaining a function of reflecting a user's feedback on a video clip according to an embodiment;
11 is a view for explaining a search function according to an embodiment;
12 is a view for explaining an operation of generating tracking clusters according to an embodiment;
13 is a view for explaining an operation of matching tracking clusters according to an embodiment;
14 is a view for explaining an operation of detecting a scoring event according to an embodiment;

Specific structural or functional descriptions disclosed in this specification are merely illustrative for the purpose of describing embodiments according to technical concepts, and the embodiments may be embodied in various other forms and are limited to the embodiments described herein. doesn't happen

Terms such as first or second may be used to describe various elements, but these terms should be understood only for the purpose of distinguishing one element from another element. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Expressions describing the relationship between components, for example, “between” and “between” or “neighboring to” and “directly adjacent to”, etc. should be interpreted similarly.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Embodiments may be serviced in various types of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, wearable devices, and the like.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

1 is a diagram illustrating a system for providing a sports video-based platform service according to an embodiment. Referring to FIG. 1 , a system according to an embodiment includes a service server, and according to the design of the system, at least one of a streaming server, a storage server, a social network service (SNS) server, and an instant messaging service (IMS) server is further added. may include The service server may include a front-end server that communicates with an application installed in the user terminal and a video analysis server that analyzes sports videos. Depending on the design of the system, the front-end server and the video analysis server may be implemented in the form of modules within the same server, or may be implemented as different independent servers to communicate with each other through a network.

The service server receives the sports video. A sports video is an image of a sports game and includes a plurality of frames. Sports events include real-time ball games such as basketball, soccer, volleyball, handball, hockey, ice hockey, and tennis; It may correspond to any one of the stocks. Hereinafter, a basketball game will be described as an example, but embodiments may be applied in substantially the same manner to other sports events other than the basketball game.

The sports video may have already been uploaded to a streaming server or storage server. In this case, the service server may receive information (eg, URL, etc.) for accessing the sports video through a streaming server or a storage server. Alternatively, the service server may receive a request to upload a sports video. The service server may upload the requested sports video to a streaming server or a storage server. The storage server or the storage server may be operated by the same entity as the service server, or may be operated by a different entity according to an embodiment.

The sports video includes a plurality of unidentified players. The unidentified player is a player whose identity is not identified, and may include, for example, a player for which identification information of the corresponding player is not set. A plurality of unidentified players included in the sports video may perform various movements for a sports game, and by analyzing the sports video, individual or team capabilities of players participating in the corresponding sports event may be evaluated. However, in order to analyze the sports video, one or more experts who are familiar with the rules of the corresponding sports competition must perform recordings for each player throughout the entire time of the corresponding sports competition.

The embodiments described below provide a technology for remarkably reducing human resources required to analyze sports videos by using artificial intelligence technology.

According to an embodiment, the video analysis server may receive an analysis request signal including a link of a sports video of a ball game from the front-end server. The video analysis server may filter out static pixels from a plurality of frames included in the sports video and perform preprocessing to leave dynamic pixels. For example, when the sports video is a video recorded with a fixed viewpoint (a video recorded with a fixed camera angle), static pixels are filtered out based on a change in pixel value between adjacent frames within a predetermined range. )can do. When a pixel value at the same position between adjacent frames changes by exceeding a predetermined threshold range, it may be classified as a dynamic pixel.

If the sports video is a video shot with a moving point (a video shot with a camera angle moving along the ball), static pixels may be filtered out based on the statistical value of the optical flow of pixels in the frame. . Due to the movement of the camera angle, pixels in a frame may have a component of a common optical flow. If the optical flow of the common component included in the pixels is removed, the optical flow component of the pixels corresponding to an actually moving object (a moving player, a moving ball, or a vibrating rim, etc.) can be obtained. After the optical flow of the common component is removed, a pixel having an optical flow having a size exceeding a predetermined threshold size may be classified as a dynamic pixel.

The video analysis server may track the ball of the sports video based on the pre-processed video. For example, the video analysis server may track the ball of the sports video by detecting the ball based on dynamic pixels of the corresponding frame in response to each frame of the sports video. Since the ball included in the sports video is recorded relatively small compared to other objects (player or goalpost, etc.) in the image, it is difficult to track the ball through the existing object tracking model. Embodiments may remarkably improve the performance of ball tracking by using an artificial neural network trained to track a ball using dynamic pixels obtained through preprocessing.

The video analysis server may detect a score-related scene of a sports video from the pre-processed video. For example, the video analysis server corresponds to each of the frames, detects a rim based on dynamic pixels of the corresponding frame, and determines frames adjacent to the frame in which the rim is detected as a score-related scene. It is possible to detect a score-related scene of a video.

The rim refers to a structure of a predetermined shape for determining whether a ball passes through and scores in a sports game in a ball game, and it may be understood that a net attached to the rim is also included in the rim below. The rim is normally classified as a static pixel because it does not move, but it can be classified as a dynamic pixel in a scene related to scoring, such as when a ball hits the rim and the rim actually moves, or when a moving ball passes through the rim. Accordingly, when a rim is detected in the pre-processed image, it may be determined that the scene is related to a score.

The video analysis server may determine a non-identified player related to the score-related scene by using the ball tracking result in response to detection of the scoring-related scene. For example, the video analysis server detects dynamic pixels related to the player who attempted to score from frames included in the scoring-related scene using the ball tracking result, and the frame in which the dynamic pixels related to the player who attempted to score are detected By performing instance segmentation, it is possible to determine an unidentified player associated with a scoring-related scene.

Since the video analysis server tracks the ball in the pre-processed image, when a rim is detected in the pre-processed image, it is possible to acquire the movement trajectory of the ball included in adjacent frames starting from the frame in which the rim is detected. The video analysis server may select dynamic pixels that come close to the ball within a predetermined reference distance from among the dynamic pixels of the player included in the adjacent frames while searching the movement trajectory of the ball. Selected dynamic pixels may include an unidentified player attempting to score.

When dynamic pixels approaching within a predetermined reference distance from the ball are selected, the video analysis server may instance segment the corresponding frame to obtain a mask of the non-identified player who attempted to score. In this case, an image of an original frame that has not been pre-processed may be input as an input to the neural network model performing instance segmentation.

The video analysis server may identify the non-identified player by tracking the non-identifying player up to an adjacent frame in which the non-identifying player can be identified. For example, the video analysis server may extract a feature (feature) from the determined non-identified player. The video analysis server may obtain pixel values corresponding to the unidentified player by using the mask of the unidentified player, and extract a feature based on the obtained pixel values.

The video analysis server may determine whether a non-identified player can be identified by comparing the extracted features with features of previously registered players. The video analysis server may receive registration information of players participating in the sports event requested for analysis from the service server. Registration information may include a list of players and photos of the players. The video analysis server may extract features from photos of previously registered players and compare them with features of non-identified players. The video analysis server may match the unidentified player with one of the pre-registered players according to the comparison result.

When it is determined that the non-identified player cannot be identified as a result of the comparison, the video analysis server may track the non-identified player by instance-segmenting an adjacent frame (a past or future adjacent frame). Due to various factors, such as the unidentified player being obscured by other players, video information sufficient to identify the unidentified player may not be included in the frame attempting to shoot. The video analysis server may track the non-identified player to an adjacent frame that contains sufficient video information to identify the non-identified player. After tracking the unidentified player in the adjacent frame, the video analysis server may determine whether to identify the unidentified player based on image information of the unidentified player tracked in the corresponding adjacent frame. The video analysis server may repeatedly perform a tracking operation and an identification operation in an adjacent frame until a non-identifying player can be identified.

The video analysis server may determine whether the goal was successful or failed in the goal-related scene. For example, the video analysis server may use a neural network model trained to determine whether scoring is successful by inputting the movement trajectory of the ball based on the position of the rim in a plurality of frames included in the scoring-related scene as an input.

The video analysis server may respond to the analysis request signal of the front-end server by outputting a time section of a sports video, identification information of an unidentified player, and/or whether or not the score was successful, in response to each of the scoring-related scenes. The front-end server may build a database based on the response signal.

According to an embodiment, the service server may distinguish the non-identified players included in the sports video from each other by using an unsupervised learning technique. The service server may distinguish that the first non-identified player and the second non-identified player are different from each other even in a situation in which each of the non-identified players cannot be identified. For example, the external features of the first unidentified player and the external features of the second unidentified player may be distinguished from each other in the sports video. Appearance characteristics are the external characteristics of the non-identified player, for example, physical characteristics of the non-identified player itself, such as physique, height, skin color, hair style, and face, as well as external characteristics of sportswear, jersey numbers, shoes, protectors, and accessories. may include Also, the motion characteristics of the first unidentified player and the motion characteristics of the second unidentified player may be distinguished from each other. A motion characteristic is a movement characteristic of an unidentified player, for example, jump shot motion, set shot motion, layup motion, dunk motion, dribble motion, pass motion, pick motion, rebound motion, blocking motion, defense motion, etc. in a basketball game. In various motions, the characteristic according to the posture or movement of the non-identified player may be included. The service server may distinguish non-identified players from each other by using appearance characteristics and/or motion characteristics.

The service server may generate a video clip for each non-identified player that is distinguished from each other. For example, the service server generates a first video clip selectively including only the image of the first unidentified player among the entire game image, and a second video clip selectively including only the image of the second unidentified player can create The service server may receive information identifying the non-identified player of the video clip from the user by providing the non-identified player-specific video clips to the user. The user may be one of the non-identified players in the corresponding sports video, and in some cases, the user may be a person who is not a non-identified player in the sports video but can identify the non-identified players in the corresponding sports video.

The user may input information identifying a non-identified player corresponding to at least one of the plurality of video clips. For example, the user inputs an input to confirm his/her own video clip among video clips, or identifies the non-identified player in a video clip of an unidentified player that he/she can identify among video clips. You can enter information.

The service server may analyze the sports video based on identification information input for each video clip. Since most sports events are conducted based on predetermined rules, motion types allowed according to the rules of the sports competition may be predetermined. Due to this, one motion type and another motion type can be distinguished from each other. For example, in a basketball game, a shooting motion and a dribbling motion may be distinguished from each other. Furthermore, among the shooting motions, a jump shoot motion and a dunk shoot motion may be distinguished from each other. Accordingly, the service server may extract feature scenes or frames from the sports video and classify motion types of the extracted scenes. As will be described in detail below, the service server may extract some regions rather than the entire scene or frame, and classify motion types of the extracted regions.

For this reason, the embodiments may provide a technique for analyzing various moments in a sports game only by receiving identification information for each video clip from the user. Furthermore, as image data identified as the same player is accumulated, a specialized model for automatically identifying the player may be learned. The specialized model not only automatically identifies the player, but can also be trained to more accurately classify the player's movements.

In addition, the service server may build a database by tagging identification information and motion type information at moments detected in a sports video. A user can search for a desired image in the database through various queries. For example, the user may input a query for i) designating a player and ii) searching for a desired scene. The service server may search from the database to which time point (frame) in which sports video footage corresponds to scenes corresponding to the query, and may generate a video clip based on the search result and provide it to the user. According to an embodiment, a desired scene by designating a plurality of players, for example, a scene in which player B scores with an assist from player A, or a scene in which player C is blocked by player D, may be searched.

As will be described in detail below, the service server may generate statistics indicating the contribution of each player in a sports game. For example, the service server may generate a box score expressing the result of a basketball game numerically. The service server may provide a service for linking scenes of the detailed record for each detailed record in the box score. For example, if a 'still' detail of a particular player is selected in a box score, a video clip containing scenes 'still' of that player may be provided. According to an embodiment, the front-end server may search for still-related clusters of a specific game in response to a selection input of 'still' in a box score of a specific game. The front-end server may provide, to the user terminal, information (eg, an image URL and at least one time interval) through which a related video clip can be streamed, based on the found clusters.

The service server may share the video clip provided to the user to a social network server or an instant messaging server. The service server may provide a service to a user through a web interface or may provide a service to a user through an app interface.

2 is an operation flowchart illustrating a method of operating a service server that provides a sports video-based platform service according to an embodiment. As described above with reference to FIG. 1 , the service server may be implemented as a single server of the front-end module and the video analysis module, or may be implemented as separate servers of the front-end server and the video analysis server. Hereinafter, for convenience of description, an embodiment implemented as a single server will be described.

Referring to FIG. 2 , the front-end module of the service server transmits a signal requesting analysis of the sports video to the video analysis module based on the link of the sports video ( 210 ). The front-end module of the service server stores player-specific clusters received from the video analysis module in the database ( 220 ). The player-specific clusters may include section information regarding scenes (eg, a score-related scene, a defense-related scene, an error or foul-related scene, etc.)

The front-end module of the service server provides information for extracting player-specific video clips from the sports video to the user terminal based on the database ( 230 ). The information provided to the user terminal may include a time section for a link of a video and a main scene for each player. The user terminal may selectively request streaming only for a time section corresponding to a main scene corresponding to the streaming server, based on the received information.

The front-end module of the service server receives an input for identifying an unidentified player of at least one cluster from the user terminal ( 240 ). The front-end module of the service server updates identification information of at least one corresponding cluster in the database based on a user input ( 250 ).

The service server receives input from the user identifying the unidentified player of the at least one cluster. Since the user is provided with a video clip for each non-identified player, he/she can input for identifying the non-identified player for each video clip. A video clip is extracted or generated in units of clusters, and the received input corresponds to information identifying an unidentified player corresponding to the corresponding cluster.

The service server sets identification information of at least one corresponding cluster based on the user's input. According to an embodiment, the cluster may include fields of {access information of sports video, identification information, indexes of areas}. In this case, the service server may set the cluster identification information based on the user's input. The identification information may correspond to account information of a sports video-based platform service. Alternatively, the identification information may correspond to account information of a social network service interworking with a sports video-based platform service or account information of an instant messaging service interworking with a sports video-based platform service. The identification information may include information (eg, last name, first name, nickname, affiliation team, jersey number, gender, age, height, weight, position, etc.) of a predetermined template for identifying the non-identified player. According to an embodiment, the identification information may be stored in the area. In this case, the cluster may include fields of {indices of regions}, and each region may include a field of {identification information}.

Alternatively, the service server may receive a feedback input from the user terminal indicating that the player of at least one section included in at least one cluster does not belong to the corresponding cluster, and may exclude the corresponding section from the corresponding cluster.

Alternatively, the service server may receive a feedback input from the user terminal stating that the player of at least one section included in at least one cluster belongs to another cluster, and exclude the section from the corresponding cluster and include it in another cluster.

Also, the front-end module of the service server may provide a sports video-based platform service ( 260 ). For example, the service server may provide a sports video-based platform service based on clusters (or regions) in which identification information is set. The service server automatically analyzes sports events, provides statistical information that indexes the contents of sports events, provides video clips linked with statistical information, provides detailed search functions for sports videos, or provides search results Various services, such as providing a video clip corresponding to In addition, the service server may also provide a function of sharing the video clip to a social network service or an instant messaging service.

The service server may provide statistics indicating the contribution of players to the user terminal, and receive an input for selecting detailed records included in the statistics from the user terminal. The service server may obtain at least one sub-cluster related to the selected detailed record based on the database, and provide the user terminal with information for extracting a video clip from the sports video based on the at least one sub-cluster. .

Alternatively, the service server may receive a search query including a search target player and a search target scene from the user terminal, and search for a sub-cluster corresponding to the search query from the database. The service server may provide information for extracting a video clip from a sports video to the user terminal based on the found sub-cluster.

Alternatively, the service server may determine a charging level based on the reliability of the clusters, or may determine a compensation level based on a feedback input for modifying the clusters.

In addition, the service server may learn a specialized model for estimating at least one of detection information, identification information, and motion type information of players by generating training data depending on the updated database.

The service server may learn a specialized model for the identified player based on clusters (or regions) in which identification information is set. Alternatively, the service server may learn a specialized model for a team to which a plurality of identified players belong. The specialized model may be trained as a detection module for detecting regions of players in a sports video, a classification module for classifying regions, an identification module for identifying regions, or a complex module for various combinations of the above-described functions.

The service server may provide a higher quality service regarding the player to which the identification information has been input by using the specialized model. As an example, as the specialized model is learned by accumulating data in which identification information is set, from a new sports video to identification information of a corresponding player may be automatically set. In addition, the game content of the corresponding player can be analyzed more accurately through the specialized model.

Although not shown in the drawings, according to an embodiment, the service server detects regions corresponding to the non-identified players from a sports video shot of the non-identified players. 'Detection' may be an operation of determining a partial region corresponding to a detection target in an image. A sports video includes a plurality of frames, and the service server may detect an area occupied by an individual unidentified player in each frame. The service server may detect regions corresponding to non-identified players by using a detector that detects players playing a corresponding sporting event in the sports video. Regions corresponding to unidentified players may have a predetermined shape, for example, a rectangular window shape. The service server may detect windows corresponding to non-identified players in the respective frame unit of the sports video. As an example, the service server may obtain a frame index indicating a frame, (x, y)-coordinates that are information indicating the position of the detected window, and (width, height) information indicating the size of the detected window. have. In this case, the individual region may be defined as {frame_index, x-coordinate, y-coordinate, width, height}. The service server may directly drive the detection module, or may request detection from another server that operates the detection module in conjunction with the service server.

The service server creates unidentified player-specific clusters by clustering the detected regions to distinguish unidentified players from each other ( 220 ). The service server can cluster regions based on unsupervised learning techniques. For example, the service server may cluster regions by using a cluster analysis technique, which is represented by a K-means clustering technique. The generated cluster may include information indicating areas belonging to the corresponding cluster (eg, indexes of areas, etc.).

The K parameter for clustering may be set by receiving the number of players participating in the game as input from the user. Alternatively, the number of players that can participate at the same time according to a sports event (for example, in case of basketball, 5 players per team for a total of 10 players) is set as the initial value of the K parameter, and iteratively while adjusting the value of the K parameter By performing clustering, it is also possible to estimate the number of players participating in a sports game in the corresponding sports video. Considering replacement members, the number of players participating in a sporting event may be greater than the number of players participating simultaneously. Although described in detail below, according to an embodiment, the service server may use a hierarchical clustering technique as a cluster analysis technique.

The service server may extract the characteristics of a region for clustering. According to an embodiment, the service server may extract an appearance feature from an individual area. Appearance characteristics can be defined in a variety of ways. For example, the appearance feature is a multi-dimensional vector, and may include information related to a physique, height, skin color, hair style, face, sportswear, uniform number, shoes, protector, and/or accessories in each dimension. Alternatively, the service server may extract the motion feature from the sequence of regions. A motion characteristic can be defined in a variety of ways. For example, the motion feature is a multidimensional vector and may include information extracted from the posture of the unidentified player or information extracted from the movement of the unidentified player.

The service server may cluster regions based on appearance characteristics or motion characteristics. For example, the service server may classify regions having similar external features into the same cluster, and classify regions having different external features into different clusters. Alternatively, the service server may classify regions having similar motion characteristics into the same cluster, and classify regions having different motion characteristics into different clusters. According to an embodiment, the service server may cluster regions based on a combination of an appearance feature and a motion feature. For example, referring to FIG. 3 , the service server may classify regions having similar combinations of external features and motion features into the same cluster, and classify regions with different combinations of external features and motion features into different clusters. . For convenience of explanation, only three clusters are shown in FIG. 3, but as described above, as many clusters as the number of unidentified players included in the sports video may be generated. In addition, although the appearance feature and the motion feature are respectively illustrated in a single dimension in FIG. 3 , as described above, the appearance feature or the motion feature may include multidimensional information.

According to an embodiment, the appearance feature may be extracted from a region of a single frame, and the motion feature may be extracted from a region sequence of a plurality of frames. In this case, a synchronization operation between the region and the region sequence may be required. As an example, the external features of the region detected in a specific frame may be statistically processed (average, etc.) together with the external features of regions detected in the previous frame and subsequent frames with respect to the corresponding region. The range of the previous frame and the subsequent frame may correspond to the frame range of the region sequence for extracting the motion feature.

The service server extracts unidentified player-specific video clips from the sports video based on the generated clusters (230). In response to each cluster, the service server may acquire frame indices of regions included in the cluster, and the location and size of regions within each frame. The service server may generate a video clip for a corresponding cluster by extracting frames of frame indices corresponding to individual clusters from the sports video.

According to an embodiment, the service server may provide a visual effect to the video clip. The service server may cut the frame extracted from the sports video based on the location and size of the region within the frame. Alternatively, the service server may provide a visual effect of emphasizing the corresponding region in the corresponding frame based on the location and size of the region in the frame. Alternatively, the service server may add cluster-related information or region-related information in the form of captions or the like.

The service server provides the extracted video clips to the user for each non-identified player. The service server may provide the video clips to the user through a web interface and/or an app interface. Depending on the embodiment, the service server may provide the video clips to the user through a social networking service and/or an instant messaging service.

4 is a diagram for explaining a sports video-based platform service according to an embodiment. 4 , a sports video may include a plurality of frames (..., k, k+1, ..., l, l+1, ??, m, m+1, ...). can The service server may detect the regions 401 - 416 of the plurality of unidentified players in the plurality of frames.

The service server may cluster the regions 401 to 416 by using an unsupervised learning technique. For example, the service server classifies the regions 401 , 403 , 413 , and 416 into the first cluster C1 , and the regions 405 , 408 , and 411 . ) and region 414 into a second cluster C2, region 402, region 404, region 406, and region 409 into a third cluster C3, and region 407 ), the region 410 , the region 412 , and the region 415 may be classified into a fourth cluster C4 .

The service server may extract video clips for each non-identified player from the sports video based on the first to fourth clusters. For example, the service server may generate a video clip for the first non-identified player by extracting frames (k, k+1, m, m+1) corresponding to the first cluster. Also, the service server may generate a video clip for the second unidentified player by extracting frames (l, l+1, m, m+1) corresponding to the second cluster. The service server may generate a video clip for the third unidentified player by extracting frames (k, k+1, l, l+1) corresponding to the third cluster. The service server may generate a video clip for the fourth unidentified player by extracting frames (l, l+1, m, m+1) corresponding to the fourth cluster.

The service server may receive identification information for each video clip by providing the video clips generated for each non-identified player to the user. For example, the service server receives identification information of the first player of team A corresponding to the first cluster, receives identification information of the second player of team A corresponding to the second cluster, and corresponding to the third cluster Accordingly, identification information as the first player of team B may be received, and identification information as the second player of team B may be received corresponding to the fourth cluster.

The service server may collect identification information received from a plurality of users. For example, the service server may receive an input identifying a video clip of a first cluster from a first user, and receive an input identifying a video clip of a second cluster from a second user. Alternatively, the service server may receive information identifying a video clip of the same cluster from a plurality of users, and adopt identification information of the highest reliability. For example, the service server may receive different identification information for video clips of the same cluster from a plurality of users. In this case, the service server may adopt the identification information input by the most users. Alternatively, the service server may adopt the identification information input by the user with the highest reliability. Alternatively, the service server may assign a score to the identification information input by the corresponding user based on the user's reliability and adopt the identification information having the highest score. The user's reliability may be determined based on the user's previous history of using a sports video-based platform service or the user's user authentication level.

5 is a diagram illustrating a sports video-based platform service according to an embodiment. The service server may perform clustering for each motion type. To this end, the service server may classify regions of non-identified players detected in a sports video according to a predetermined motion type, and may cluster motion type-specific regions. Alternatively, the service server may hierarchically cluster regions using a hierarchical clustering technique.

Motion characteristics for each type of motion appearing in a sports game may be included in common among unidentified players. For example, the dunk shooting motion of the first unidentified player and the dunk shooting motion of the second unidentified player may include a motion characteristic for a dunk shot in common. Also, the jump shoot motion of the first unidentified player and the jump shoot motion of the second unidentified player may include a motion characteristic for the jump shoot motion in common. Referring to FIG. 6 , the dunk shoot motion of the first unidentified player is represented by a motion vector 610 , the jump shoot motion of the first unidentified player is represented by a motion vector 620 , and the second unidentified player’s The dunk shoot motion may be expressed as a motion vector 630 , and the jump shot motion of the second unidentified player may be expressed as a motion vector 640 .

In the embodiment of FIG. 6 , the distance between motion vector 610 and motion vector 620 is greater than the distance between motion vector 610 and motion vector 630 , and between motion vector 640 and motion vector 630 . The distance of may be greater than the distance between the motion vector 640 and the motion vector 620 . In this case, clustering may not be performed for each non-identified player, but may be clustered for each motion type.

According to an embodiment, after classifying motion types of regions, the service server may perform clustering among regions corresponding to the same motion type. In this case, the regions may be classified into a first cluster 650 corresponding to a dunk shoot motion and a second cluster 660 corresponding to a jump shoot motion. According to an embodiment, region detection and motion type classification may be performed simultaneously. For example, a learned detection module may be used to classify a motion type of a corresponding region while detecting non-identified players from a sports video. Of course, according to embodiments, region detection and motion type classification may be performed by a separate module (or neural network). After the regions are classified by motion types, the service server may cluster motion type-specific regions to distinguish non-identified players from each other by motion types.

According to an embodiment, the service server may first perform primary clustering and then perform secondary clustering within each cluster using a hierarchical clustering technique. In this case, the first cluster 650 and the second cluster 660 are generated through the primary clustering, and the first unidentified player and the second unidentified player in the first cluster 650 through the secondary clustering is distinguished, and the first unidentified player and the second unidentified player may be distinguished in the second cluster 660 . K1 for primary clustering may correspond to the number of motion types allowed in the corresponding sporting event, and K2 for secondary clustering may correspond to the number of players participating in the corresponding sporting event.

Referring to FIG. 7 , an embodiment in which sub-clusters are generated by clustering regions according to a motion type, a motion characteristic, and an appearance characteristic is illustrated. Referring back to FIG. 6 , as will be described in detail below, a sub-cluster corresponding to the first unidentified player in the first cluster 650 and a sub-cluster corresponding to the first unidentified player in the second cluster 660 . may be matched with each other based on tracking information, appearance information, or a combination thereof. Of course, the sub-cluster corresponding to the second unidentified player in the first cluster 650 and the sub-cluster corresponding to the second unidentified player in the second cluster 660 are also based on tracking information, appearance information, or a combination thereof. so they can be matched with each other.

Referring to FIG. 5 , the service server may classify regions 501 to 516 by motion type. For example, area 501 and area 511 are classified as dribble motion, area 503, area 510, and area 514 are classified as shooting motion, and area 504 is classified as block motion. , region 506 may be classified as a pass motion, and region 513 and region 516 may be classified as a screen motion.

The service server may cluster regions for each motion type to distinguish non-identified players from each other for each motion type. For example, the service server may cluster the area 501 and the area 511 classified as dribbling motion into different sub-clusters DR1 and DR2. The service server may cluster the areas 503 , 510 , and 514 classified as shooting motion into different sub-clusters SH1 , SH2 , and SH3 . The service server may cluster the regions 513 and 516 classified as screen motion into the same sub-cluster SC1.

The service server may match sub-clusters of different motion types using the tracking information of regions. For example, the service server may classify DR1 and SH1 into the same cluster C1 by matching DR1 and SH1 based on tracking information that the region 501 and the region 503 are continuous regions. The service server may classify the same cluster C2 by matching DR2 and SH3 based on the tracking information that the regions 511 and 514 are continuous regions.

The service server may match sub-clusters of different motion types by using the appearance information of the regions. For example, the service server may classify BR1 and PA1 into the same cluster C3 according to determining that the external features of the area 514 and the external features of the region 506 are similar. The service server may classify SC1 into the cluster C1 according to the determination that the external features of the region 513 and the external features of the region 502 or 503 are similar.

In the embodiment of FIG. 5 , an example in which regions are classified by motion types and then clustered has been described. However, even when the hierarchical clustering technique is used as described above, the operation may be substantially the same. In addition, although an example of applying the appearance information after applying the tracking information for matching sub-clusters has been described, the order or method of applying the tracking information and the appearance information may be variously modified.

According to an embodiment, the service server may generate a video clip using sub-clusters according to a predetermined motion type. For example, the service server may generate a video clip using only sub-clusters of motion types corresponding to the attack motion. Furthermore, the service server may generate a video clip by using only the region sequence corresponding to a scene that has been scored successfully during the attack motion.

8 is a diagram illustrating a general-purpose model and a specialized model according to an embodiment. Referring to FIG. 8 , the general-purpose model may include a detector for universally detecting unidentified players from a sports video and a classifier for universally classifying the motion type of the detected region. A general-purpose model may include a detector or classifier that is independently trained on the data of non-identifying players.

The specialized model utilizes a database in which identification information is set according to a sports video-based platform service, and is specialized for a specific player, a specific group, or a specific team, a detector that detects a player, a classifier that classifies a motion type, and an identifier that identifies a player , or a mixing module in which a detection function, a classification function, and/or an identification function are variously combined. The general-purpose model and the specialized model may be an artificial neural network-based model. In this case, the database may contain training data dependent on data of players for which identification information is set.

The specialized model may be a model newly learned for a specific group, or may be a model additionally trained to suit a specific group based on a general-purpose model according to embodiments. For example, the general-purpose model may be additionally learned to be specialized to a specific group according to nationality, age group, gender, and the like to provide higher performance, thereby generating a specialized model.

The service server may automatically set identification information of clusters for each unidentified player from a newly received sports video using a specialized model. The sports video may be a live streaming video, and in this case, the service server may generate, in real time, statistics indicating the contribution of players automatically identified from the sports video based on the specialized model. Statistics showing the contribution of players include not only the scores of both teams during a basketball game, but also the number of successful goals scored by each player, the number of attempts to score, the number of goals scored by the team during playing time, the total number of goals scored, the number of offensive rebounds, the number of defensive rebounds, the number of assists, and the number of steals. It may include detailed records such as the number, the number of blocks, the number of blocks, the number of fouls, and the number of turnovers.

9 is a view for explaining video clips provided in association with statistics indicating a contribution level for each player in a basketball game according to an exemplary embodiment. Referring to FIG. 9 , the service server may generate a box score statistically indicating the contribution of each player to a game according to identification information of a region detected in a basketball video and a motion type of the region.

The service server may determine whether to additionally analyze the sports video according to the motion type of the region. For example, the service server may additionally analyze whether the shot was successful according to the determination that the motion type is a shooting motion. The service server may check whether the ball has passed the rim after the frame of the shooting motion. The service server may detect regions corresponding to the rim in frames after the shooting motion, and may determine whether the ball has passed in the detected regions.

In addition to providing the box score, the service server may provide a video clip comprising scenes of the selected detail in response to selection of the detail in the box score. For example, the FG in the box score may be expressed as 'number of successful scoring' - 'number of scoring attempts', such as 08-14. When the user selects '08' corresponding to the 'number of successful scoring', the service server may generate a video clip including scenes in which the corresponding player succeeds in scoring in the corresponding game and provide it to the user.

More specifically, the detailed records of the box score may store corresponding regions (or sub-clusters) in a data structure such as a linked list. The service server may obtain related areas along the linked list, and extract a video clip from the basketball moving picture based on information (frame index, window position, window size, etc.) stored in each area. As described above, the service server may provide a visual effect to the video clip.

10 is a diagram for describing a function of reflecting a user's feedback on a video clip according to an exemplary embodiment. Referring to FIG. 10 , the service server may receive a user's feedback on a video clip. For example, the service server may receive a feedback input 1010 from the user indicating that the unidentified player of at least one region included in the cluster of video clips does not belong to the cluster. The service server may exclude or remove the corresponding area from the corresponding cluster, or further exclude or remove the sub-cluster 1015 including the corresponding area from the corresponding cluster. In this case, according to an embodiment, the service server may temporarily store the excluded area or sub-cluster in an unclassified pool. The service server may inquire to which cluster the sub-clusters belong while providing the user with video clips of the sub-clusters temporarily stored in the unclassified pool.

Alternatively, the service server may receive a feedback input 1020 from the user indicating that the unidentified player of at least one region included in the cluster of video clips belongs to another cluster. The service server may exclude or remove the corresponding area or the sub-cluster 1025 including the corresponding area from the corresponding cluster. The service server can include excluded zones or sub-clusters into other clusters designated by the user.

Although not shown in the drawing, the service server may differentially determine the charging level according to the accuracy (or reliability) of the service. For example, the service server may discount the service cost according to detection accuracy, classification accuracy, or a combination thereof based on the service cost per game (eg, $1). Detection accuracy and/or classification accuracy may be referred to as cluster accuracy.

Also, the service server may determine the reward level according to the user's feedback input. For example, the service server is used for the next service according to the degree to which the accuracy (or reliability) of the cluster is improved according to the user's feedback input, or the performance of the specialized model is improved according to the user's feedback input. Possible points can be accumulated to the corresponding user.

11 is a diagram for describing a search function according to an exemplary embodiment. The service server may build a database for searching sports videos. The service server may receive a search query from the user terminal. The search query may include a search target player and a search target scene. The service server may search for a URL of a sports video corresponding to the search query and time section(s) within the sports video corresponding to the search query from the database.

The service server provides information for extracting the video clip to the streaming server, so that the video clip is directly transmitted from the streaming server to the user. The service server may provide a search result including the URL and time section(s) of the sports video to the user terminal. The user terminal may request an image of the corresponding time period of the sports video from the streaming server or the storage server based on the search result.

Although not shown in the figure, according to an embodiment, the service server may extract a video clip from a streaming server or a storage server based on {sports video access information} of a cluster including the found sub-cluster (or section sequence). have. The service server may provide the video clip to the user. The service server may cache pre-generated video clips. The service server may store whether the video clip is cached in the database. When it is determined that the service server needs to provide the cached video clip as a result of the query processing, the service server may omit the operation of extracting (or creating) the video clip and immediately provide the cached video clip to the user.

According to an embodiment, an appearance feature or a motion feature may be extracted differently according to a viewpoint of a camera that captures a sports video. Here, a viewpoint of the camera may have a value of 6 degrees of freedom (DOF) as a 3D position of the camera and a 3D orientation of the camera.

The service server may use an appearance feature or motion feature that is robust to changes in the camera's viewpoint. For example, the service server may encode an appearance feature so that the information included in the multidimensional appearance vector is independent without depending on the viewpoint of the camera. Appearance characteristics independent of the camera view include the player's own style characteristics, such as hair style, skin style (or skin type) and/or tattoo style; style characteristics of the object worn by the player, such as jersey styles, basketball shoe styles, and/or other accessory styles; and/or may include physical characteristics of the player, such as the player's height or physique, normalized based on the reference object.

The style characteristics of the player itself and the characteristics of the object worn by the player can be defined in a form independent of the camera's viewpoint. The reference object is an object including a standardized appearance in a corresponding sporting event, and may include, for example, lines drawn on a goalpost or a stadium. Physical characteristics such as height or physique of a player may be normalized based on the size of a reference object captured in a sports video. Since the size of the reference object is standardized in the corresponding sporting event, the physical feature normalized based on the reference object may be independent without depending on the viewpoint of the camera.

In addition, the service server may encode a motion feature so that information included in the multidimensional motion vector is independent without depending on the viewpoint of the camera. The camera viewpoint-independent motion characteristics may include normalized motion (eg, direction, size, speed, etc.) or pose with respect to a reference object. If the player's joints are recognizable, the motion characteristics may include normalized motions or poses for each major joint. Here, major joints used for motion characteristics may be defined for each motion type.

According to an embodiment, a sports video may be recorded while a part of the stadium is cut off. The service server may utilize the region sequence to estimate the movement of the player in the uncaptured and cut space. For example, if the ball passes through the rim or an event that the ball hits the rim is detected while no shooting motion is detected, the service server may estimate that the shooting motion occurred in the cut space without being photographed. Furthermore, the service server may find a player who has moved to a cut space without being photographed from adjacent previous frames of the corresponding frame, and may estimate that a shooting motion has occurred by the corresponding player.

According to an embodiment, the service server may analyze the sports video by additionally utilizing the motion of the referee, the whistle sound of the referee, or the buzzer sound of the scoring system. For example, the service server may distinguish whether a two-point shot or a three-point shot is made by using the motion of the referee. Alternatively, the service server may use a whistle sound or a buzzer sound to determine whether the game is stopped at a corresponding time point.

According to an embodiment, the service server may recognize a connection situation according to a motion type. For example, the service server may recognize the assist situation in connection with the pass motion before the shooting motion. Alternatively, the service server may recognize the block situation in connection with the block motion after the shoot motion. In the case of a block situation, it can also be recognized by considering the direction of the ball after the shooting motion.

12 is a diagram for explaining an operation of generating tracking clusters according to an embodiment. Referring to FIG. 12 , a user may upload a sports video to the streaming server 1210 . The user may provide a link of the streaming server as a service server. As described above, the sports video may be uploaded to a storage server other than the streaming server 1210 or may be directly uploaded to a service server.

The service server may receive a sports video by accessing the streaming server 1210 using the corresponding link. According to an embodiment, the service server may process the sports video while streaming it or perform the processing after downloading the sports video.

The pre-processing module 1220 of the service server may pre-process a sports video. For example, the service server may extract an image of the game time, excluding the break time, from the sports video. If the sports game consists of four quarters, the service server may extract the match images of the first to fourth quarters. In addition, the service server may extract the game progress image from the game images except for the non-match video. For example, even within a quarter, the game may be stopped due to a foul or operation time. The service server may extract the video of the game in progress, except for the non-game video where the game is interrupted. The game progress video may be subdivided according to the rules of the sports game. For example, in a basketball game, a free throw may be awarded to a player who is fouled, and in a soccer game, a free kick may be awarded. The service server may extract a free throw image or a free kick image according to the rules of the corresponding sporting event.

The service server may extract the video by extracting a frame section or a time section within the sports video. Pre-processing of sports videos may be operated in an automatic mode in which automatic processing is performed, a semi-automatic mode in which user feedback is received after being automatically processed, or a manual mode in which input from a user is received. For example, the service server may receive a start time and an end time of 1 to 4 quarters from the user. Alternatively, the service server may automatically extract start scene candidates and end scene candidates of the first to fourth quarters through video analysis, and receive the user's selection of the actual start scene and the actual end scene of each quarter. In addition, the service server may recognize the match suspension due to a foul by analyzing the referee's whistle sound or the referee's motion. Of course, the service server can recognize the game suspension situation through video analysis. For example, there may be a difference between the amount of motion of the players during a match and the amount of motion of the players during a game stop. The service server may recognize the game interruption situation by acquiring the amount of motion of the players through video analysis.

The service server may detect and track (1230) unidentified players in the pre-processed video. The service server may generate tracking clusters i0, i1, j0, and k0 by tracking regions corresponding to non-identified players from the sports video. The sports video may include a plurality of frames, and the tracking cluster may include regions of the same unidentified player included in successive frames of at least some of the plurality of frames.

The service server may assign a unique identifier to the tracking clusters i0, i1, j0, k0. For example, the service server may assign an identifier to the tracking cluster with a combination of a start frame and a unique number. If there is a tracking cluster starting in the i-th frame, an identifier of i0 may be assigned. If there is another tracking cluster starting in the i-th frame, an identifier of i1 may be assigned. Similarly, an identifier of j0 may be assigned to a tracking cluster starting from a j-th frame, and an identifier of k0 may be assigned to a tracking cluster starting from a k-th frame.

Also, the service server may add information indicating the end frame of the tracking cluster or the number of frames of the tracking cluster to the identifier. As will be described below, when matching between tracking clusters is performed, matching may be omitted if even some frames overlap. The service server can easily determine whether the tracking clusters overlap each other by using the end frame added to the identifier or the number of frames.

When processing a sports video in a streaming manner, the service server may add information indicating whether the tracking cluster is still valid in the current frame to the identifier.

In the process of tracking unidentified players in a sports video, tracking of unidentified players may be interrupted due to various factors. For example, depending on the viewpoint of the camera, non-identified players may overlap each other. Alternatively, an unidentified player may deviate from the camera's viewpoint. Alternatively, some or all of the screen may be blocked due to an obstacle while shooting a sports video. Or, for other technical reasons, the tracking module may miss tracking of an unidentified player. In this case, a first tracking cluster until the tracking is stopped may be generated, and a second tracking cluster newly tracked after the tracking is stopped may be generated. In FIG. 13 below, an embodiment of merging tracking clusters that are separated and generated corresponding to the same unidentified player through matching will be described.

13 is a diagram illustrating an operation of matching tracking clusters according to an embodiment. Referring to FIG. 13 , the service server may classify motion types for each tracking cluster ( 1310 ) and extract features ( 1320 ). The service server may extract the appearance characteristics of the non-identified player included in the tracking cluster. As described above, the appearance feature may include a multidimensional feature vector independent of the viewpoint of the camera.

The service server may classify the motion types of at least some sections included in the tracking cluster based on the predetermined motion types. For example, the service server may detect at least some continuous areas corresponding to any one of predetermined motion types among a plurality of areas included in the tracking cluster. For example, referring to the tracking cluster i0, the service server may detect a dribble section and a pass section. The service server may use a neural network model that sequentially receives consecutive scenes included in the tracking cluster and outputs at least one of predetermined motion types. The service server may extract the motion characteristics of the unidentified player included in the corresponding section in the tracking cluster based on the motion type. As described above, the motion feature may include a multidimensional feature vector independent of the viewpoint of the camera.

When a shot section is detected, the service server may separately tag the location on the court where the unidentified player attempted to shoot. As will be described below, the service server may detect a scoring event and additionally tag whether the unidentified player's shot attempt was successful.

According to an embodiment, the service server extracts areas corresponding to a state of possession of a ball from among a plurality of areas included in the tracking cluster, and extracts at least some of the extracted areas corresponding to any one of predetermined motion types. It is also possible to detect contiguous regions.

In the embodiment of FIG. 13 , the service server may extract the appearance feature af_i0 from the tracking cluster i0. The service server may extract the motion feature mf_i0_dribble from the dribble section included in the tracking cluster i0 and extract the motion feature mf_i0_pass from the pass section. In substantially the same way, the service server may extract the appearance feature af_i1 from the tracking cluster i1 and extract the motion feature mf_i1_shoot. Also, the service server may extract the appearance feature af_j0 and the motion feature mf_j0_shoot from the tracking cluster j0, and extract the appearance feature af_k0 and the motion feature mf_k0_block from the tracking cluster k0.

The service server may match 1330 the tracking clusters based on at least one of appearance characteristics and motion characteristics. The service server may not perform matching between tracking clusters overlapping even if the frame period is partially. This is because different tracking clusters with overlapping frame sections can be regarded as different unidentified players. Accordingly, the service server may omit matching between the tracking cluster i0 and the tracking cluster i1. After matching the external features of the tracking cluster i0 with the external features of the tracking cluster j0, the service server may determine that the matching has failed. After matching the external features of the tracking cluster i0 with the external features of the tracking cluster k0, the service server may determine that the matching has been successful. If matching is successful, the service server may merge the tracking cluster i0 and the tracking cluster k0. Here, merging may be understood as an operation of including the tracking cluster i0 and the tracking cluster k0 into a non-identified player-specific cluster for the same non-identified player.

Also, the service server may match motion characteristics of the same motion type among tracking clusters whose frame sections do not overlap each other. For example, the service server may match mf_i1_shoot extracted from the shoot section of the tracking cluster i1 with mf_j0_shoot extracted from the shoot section of the tracking cluster j0, and then determine that the matching has failed.

The service server may generate clusters for each non-identified player through the above-described matching operation. In the above embodiments, the 'cluster' refers to a cluster for each unidentified player, and may be understood as a concept distinct from the 'tracking cluster'. The service server may generate ( 1340 ) video clips for each non-identified player based on the clusters. As described above with reference to FIGS. 1 to 11 may be applied to operations after the clusters for each non-identified player are generated, a more detailed description will be omitted.

14 is a diagram for describing an operation of detecting a scoring event according to an exemplary embodiment. Referring to FIG. 14 , the service server may detect a scoring event in which the ball passes through the goalpost in the sports video. In the case of a basketball game, it is possible to detect whether the ball passes through the rim of the goalpost. The service server may detect the goal area or the rim area of the goal in the sports video, and determine whether a scoring event through which the ball passes in the detected area as the frame progresses is detected.

When a scoring event is detected, the service server may identify a tracking cluster associated with the corresponding event from among the tracking clusters based on the corresponding frame. For example, the service server may identify a tracking cluster including a section of a motion type associated with a corresponding event in the corresponding frame and previous frames from among the tracking clusters. When a scoring event is detected, the service server may identify a tracking cluster that includes the shot period. The service server may identify the tracking cluster containing the closest shot interval in previous frames of the scoring event. The service server may tag information indicating that the attempted shot was successful in the corresponding shot section.

According to an embodiment, when generating video clips for each non-identified player, the service server selectively acquires sub-clusters associated with the scoring event, and then, based on the sub-clusters associated with the scoring event, video from the sports video. You can extract clips.

The embodiments described above may be implemented as a processing device using a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Claims (15)

Receiving an analysis request signal including a link of a sports video of a ball game;
filtering out static pixels from a plurality of frames included in the sports video and performing pre-processing to leave dynamic pixels;
tracking the ball of the sports video based on the pre-processed video;
detecting a score-related scene of the sports video from the pre-processed video;
in response to detection of the scoring scene, determining an unidentified player associated with the scoring scene using the ball tracking result;
identifying the unidentified player by tracking the unidentified player up to an adjacent frame in which the unidentified player is identifiable; and
outputting a time section of the sports video and identification information of the non-identified player in response to the score-related scene
A method of operation of a video analysis server comprising a. According to claim 1,
The step of tracking the ball is
corresponding to each of the frames, detecting a ball based on dynamic pixels of the corresponding frame;
Including, the method of operation of the video analysis server. According to claim 1,
The step of detecting the score related scene is
corresponding to each of the frames, detecting a rim based on dynamic pixels of the corresponding frame; and
determining frames adjacent to the frame in which the rim is detected as the score-related scenes;
Including, the method of operation of the video analysis server. According to claim 1,
The step of determining a non-identified player associated with the score-related scene comprises:
detecting dynamic pixels related to a player who attempted to score in frames included in the scoring-related scene by using the ball tracking result; and
determining an unidentified player associated with the scoring-related scene by instance segmenting a frame in which dynamic pixels related to the player who attempted to score are detected;
Including, the method of operation of the video analysis server. According to claim 1,
The step of identifying the unidentified player is
extracting a feature from the determined non-identified player;
comparing the extracted features with features of previously registered players;
determining whether the non-identified player can be identified as a result of the comparison; and
tracking the unidentified player by instance segmenting adjacent frames in response to determining that the unidentified player cannot be identified;
Including, the method of operation of the video analysis server. According to claim 1,
The pre-processing step is
filtering out static pixels based on a change in pixel values between adjacent frames within a predetermined range when the sports video is a video taken from a fixed viewpoint; and
Filtering out static pixels based on statistical values of optical flow of pixels in a frame when the sports video is a video taken at a moving time
At least one of, the method of operation of the video analysis server. A computer-readable recording medium in which a program for executing the method of claim 1 is recorded. transmitting a signal requesting analysis of the sports video to a video analysis module based on the link of the sports video;
storing player-specific clusters received from the video analysis module in a database;
providing information for extracting player-specific video clips from the sports video to a user terminal based on the database;
receiving an input for identifying an unidentified player of at least one cluster from the user terminal provided with the video clips for each player from a streaming server; and
updating identification information of at least one corresponding cluster in the database based on the input;
A method of operating a server that provides a sports video-based platform service comprising a. 9. The method of claim 8,
providing statistics indicating the contribution of players to the user terminal;
receiving an input for selecting detailed records included in the statistics from the user terminal;
obtaining at least one sub-cluster associated with the selected detailed record based on the database; and
providing information for extracting a video clip from the sports video to the user terminal based on the at least one sub-cluster
A method of operating a server that provides a sports video-based platform service further comprising a. 9. The method of claim 8,
receiving a search query including a search target player and a search target scene from the user terminal;
retrieving, from the database, a sub-cluster corresponding to the search query; and
providing information for extracting a video clip from the sports video to the user terminal based on the found sub-cluster
A method of operating a server that provides a sports video-based platform service further comprising a. 9. The method of claim 8,
determining a charging level based on the reliability of the clusters; and
determining a reward level based on a feedback input that modifies the clusters;
A method of operating a server that provides a sports video-based platform service further comprising at least one of. 9. The method of claim 8,
receiving, from the user terminal, a feedback input indicating that a player of at least one section included in the at least one cluster does not belong to the corresponding cluster; and
excluding the corresponding section from the corresponding cluster based on the feedback input
A method of operating a server that provides a sports video-based platform service further comprising a. 9. The method of claim 8,
receiving, from the user terminal, a feedback input indicating that a player of at least one section included in the at least one cluster belongs to another cluster; and
excluding the corresponding section from the corresponding cluster and including it in the other cluster based on the feedback input
A method of operating a server that provides a sports video-based platform service further comprising a. 9. The method of claim 8,
generating training data dependent on the updated database; and
Learning a specialized model for estimating at least one of detection information, identification information, and motion type information of players based on the training data
A method of operating a server that provides a sports video-based platform service further comprising a. A computer-readable recording medium in which a program for executing the method of claim 8 is recorded.

Images