KR102423760B1 - System and method for video segmentation based on events - Google Patents

Abstract

The present invention relates to a video event unit segmentation system and method, and more particularly, to a system and method for segmenting a relay video in event units using text relay and image information.
The video event unit segmentation system according to the present invention includes an information collecting unit for receiving a relay image and text relay information and extracting image information, an event section identification unit for identifying an event section within the video, and an image generated based on the event section It is characterized in that it comprises a video service unit for servicing the segment.

Description

Video event unit segmentation system and method

The present invention relates to a video event unit segmentation system and a method therefor, and more particularly, to a system and method for segmenting a relayed video in an event unit using text relay and image information.

With the expansion of IPTV and OTT markets, a vast amount of content is being provided to viewers.

In such a vast amount of sports videos, viewers have a limited time to consume content, so the average recyclability of sports content is lower than that of other types of content such as dramas and documentaries.

In addition, there is a demand for viewers to select and watch only the desired team or player's play, but there is a problem that personalized service is impossible due to one-sided broadcast programming.

In addition, recently, services to edit and provide main game scenes of events that viewers want are provided mainly through large portals. There are many problems with players centered or concentrated on a specific game.

The present invention has been proposed to solve the above-described problem, and by automatically performing semantic indexing on a relay image, it is possible to support the viewer to select and watch a desired player or event type, event unit image segmentation An object of the present invention is to provide a system and a method therefor.

The video event unit segmentation system according to the present invention includes an information collecting unit for receiving a relay image and text relay information and extracting image information, an event section identification unit for identifying an event section within the video, and an image generated based on the event section It is characterized in that it comprises a video service unit for servicing the segment.

The video event unit segmentation method according to the present invention includes the steps of receiving a relay image and text relay information, extracting image information from the relay image, and using the image information to define an event section corresponding to an event included in text relay information. and identifying and generating an image segment.

A video event unit segmentation system and method according to an embodiment of the present invention include information about the shot class and replay section as well as camera panning information, whether a game is in progress, whether the game time is displayed in the video, and whether the shot class is currently There is an effect of improving the identification accuracy of the event video section by using a number of qualities, such as the time required to change from .

According to the present invention, semantic index information for an image is generated, and it is possible to select and provide content by player, team, or event type preferred by viewers in a media service environment. There is a possible effect of automatically generating a highlight image for the .

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram showing the configuration of a video event unit segmentation system according to an embodiment of the present invention.
2 is a diagram illustrating text relay information according to an embodiment of the present invention.
3 is a diagram illustrating a rule-based event section identification process according to an embodiment of the present invention.
4 is a diagram illustrating learning data of Haksim model-based event section identification according to an embodiment of the present invention.
5 is a flowchart illustrating a video event unit segmentation method according to an embodiment of the present invention.

The above and other objects, advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the following embodiments are intended for those of ordinary skill in the art to which the present invention pertains. It is only provided to easily inform the composition and effect, and the scope of the present invention is defined by the description of the claims.

Meanwhile, the terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural, unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” means that a referenced component, step, operation, and/or element is the presence of one or more other components, steps, operations and/or elements. or added.

Before describing a preferred embodiment of the present invention, a background in which the present invention is proposed to help those of ordinary skill in the art in the following will first look at the background.

The highlight content configuration according to the prior art suggests a configuration in which the main part is extracted by analyzing the game video. It is impossible to provide a personalized image service, and there is a limit in that a viewer can only view a highlight image created/provided by the service provider.

The present invention has been proposed to solve the above problems, and by automatically performing semantic indexing on sports relay images, when a viewer requests a desired player or event type, it is possible to extract and service an image segment for the desired player or event type. A unit segmentation system and method are proposed.

In general, when an event such as a shot occurs, the point corresponding to the event in the video is a scene where the ball enters the goal, but the event video that the viewer is interested in shows the process of the event and a certain scene after the event. It's supposed to be an embedded video.

However, if a scene with little relevance to the event is included in the event video, user satisfaction will decrease significantly. Therefore, it is important to identify the exact starting point and ending point for each event to provide the event video, and to achieve this technical task A detailed description of the embodiments of the present invention proposed in order to do so will be described below.

1 is a diagram showing the configuration of a video event unit segmentation system according to an embodiment of the present invention.

The video event unit segmentation system according to an embodiment of the present invention extracts the event occurrence time and contents from the text relay, finds the corresponding point in the relay image, identifies and extracts the event section, and automatically generates an event unit video clip and serve it to viewers.

The video event unit segmentation system according to an embodiment of the present invention includes an information collecting unit 310 that receives a relay image and text relay information and extracts image information, and uses the text relay information and the image information to define an event section in an image. It includes an event section identification unit 320 to identify and a video service unit 330 for servicing the video segment generated based on the event section.

The information collection unit 310 according to an embodiment of the present invention includes an image information extraction unit 310 and a text relay information pre-processing unit 320, and receives a relay image from the video streaming capture unit 100, and performs real-time crawling. Receives text relay information from the unit 200 .

The information collection unit 310 according to an embodiment of the present invention extracts at least one of time information, shot class information, and camera panning information from a relay image, and preferably uses a Convolutional Neural Network (CNN) technique to extract the relay image. Time information and shot class (court view, close-up) information are extracted from the , and camera panning information is extracted using optical flow estimation and flow segmentation algorithms.

The information collection unit 310 according to an embodiment of the present invention processes the extracted information again to configure additional qualities, that is, whether the game is in progress, whether the game time is displayed in the image, and when the shot class is changed. Image information is extracted by generating at least one of the required times as a feature.

The video service unit 330 according to an embodiment of the present invention stores the video segment generated based on the event section as a lower video clip file, stores event information for the video clip file as metadata, and builds a database. , an event unit service is provided to the user terminal 400 such as a TV or tablet according to a viewer's selection.

As shown in FIG. 2 , the text relay information received by the information collection unit 310 according to an embodiment of the present invention includes the event occurrence time and a player's action (in the case of basketball, scoring, rebounding, assisting, fouling). , replacement, etc.).

The information collection unit 310 according to an embodiment of the present invention extracts image information from the relay image, so that the event section identification unit 320, which will be described later, accurately identifies the start point and the end point of the event.

The most basic information for accurately finding the corresponding video section for one event is the game time.

That is, as can be seen in the example of the text relay information shown in FIG. 2 , since all events have an occurrence time, it is possible to find a point at which the corresponding occurrence time is exposed by using a number recognition technique in the image.

However, since viewers tend to want to see a scene in which the event occurred and a scene after the event, a scene in which the event occurred and a scene after the event should also be included in the section of the event, and scenes with little relevance to the event should be included. If it is included, the user's satisfaction will be lowered, so it is an important task to accurately identify the start and end points of the event.

Since there is an error of about 1 to 3 seconds between the event occurrence time in the text relay and the actual game occurrence time, the start point and the end point of the event are different for each event.

Therefore, according to an embodiment of the present invention, a general production rule of a sports relay is used to automatically find an event section.

For example, in basketball games, after most shot or goal events, the camera will often switch from a court view shot to a close-up shot to illuminate the player.

In addition, many events occur at both ends of the court where the goal is located, and there is little or no panning of the camera (a technique of shooting while moving the camera according to the speed or direction of the body).

In addition, if the game time stops or disappears, it is the case that the game is stopped, so it is desirable to filter the relevant part when selecting the event section.

According to an embodiment of the present invention, event section identification using rules and event section identification using machine learning algorithms (eg, CRFs) are combined, and the above-described broadcast production rules are naturally melted into the learning model.

3 is a diagram illustrating a rule-based event interval identification process according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating learning data of Haksum model-based event interval identification according to an embodiment of the present invention.

The event section identification unit 320 according to an embodiment of the present invention identifies an event section in the image based on a rule and a learning model, and identifies the event section using weights and average values for the event start point and end point. .

The learning model is built using the image information extracted from the relay image as a feature, which will be described later with a detailed description of FIG. 4 , and a rule-based event section identification process will be described with reference to FIG. 3 below.

The event section identification unit 320 according to an embodiment of the present invention finds a point corresponding to the game time on the text relay based on the text relay information, but considers errors in the text relay information.

As described above, the event occurrence time of the text relay is usually 1 to 3 seconds slower than the occurrence time of the actual game. For example, the event occurring at 11 minutes 40 seconds in the text relay is usually 11 minutes 42 seconds to 44 seconds in the actual video occurs in seconds.

The event section identification unit 320 according to an embodiment of the present invention recognizes the point of the corresponding game time in the relay image in consideration of the error regarding the game time of the text relay information.

The event section identification unit 320 according to an embodiment of the present invention defines a first event section by applying a preset time before and after the recognition point.

Referring to FIG. 3 , 15 seconds before and 5 seconds after the recognition point are defined as the primary event period, which mainly includes the event generation process and secondaryly includes the period after the event occurrence.

The event section identification unit 320 according to an embodiment of the present invention defines a second event section in which a partial region within the first event section is removed by using shot class information among the image information.

That is, the event section identification unit 320 defines the second event section by removing the close-up shot class in the section boundary portion using the shot class information extracted from each frame for the first event section. .

In most sports broadcasts, when a game is in progress, it is a far-shot, and when the game is temporarily suspended or an event occurs, a close-up of a specific player is sometimes shown.

The event section identification unit 320 according to an embodiment of the present invention refines the first event section in consideration of such a broadcast production pattern to define a second event section.

The event section identification unit 320 according to an embodiment of the present invention selects a final event section in which a partial region within the secondary event section is removed by using camera panning information among image information.

At this time, the event section identification unit 320 selects the final event section by removing the portion if the camera panning rapidly increases near the section boundary using camera panning information for the second event section.

For example, in a basketball game, most events occur near the goalposts on both sides, and camera panning rarely occurs at this time.

Because panning means that players have moved from left to right (or vice versa), removing these sections makes it possible to more accurately pinpoint the start and end points of the event.

As another embodiment, even when camera panning occurs, the event section identification unit 320 according to an embodiment of the present invention may rapidly increase the camera panning when the camera panning and event occurrence time are within a preset time. Even if it is a part, the final event section is selected without removing it.

For example, in the case of a successful fast attack in a basketball game, the camera panning increased rapidly, but the rapid increase in the camera panning and the event occurrence time are within a preset time (eg 1.5 seconds), so It is not an unnecessary scene, but a main scene related to the event.

Hereinafter, a learning model-based event section identification process will be described with reference to FIG. 4 .

According to an embodiment of the present invention, image information (time information, shot class information, camera panning information, etc.) in a relay image is used as a feature in order to build a learning model.

In addition, as additional qualities, whether the game is in progress, whether the game time is displayed in the video, and the time taken from the time the shot class is changed from the present to the time taken to build the learning data.

Learning data for one frame for learning CRFs, which is a machine learning algorithm according to an embodiment of the present invention, is shown in FIG. 4 .

Referring to FIG. 4 , time represents time information of the current frame, -1 means no game time, and ##:## means game time.

The panning represents the numerical value of the camera panning degree by comparing the current frame with the next frame.

shotcls indicates the shot class information of the current frame, far indicates a far shot, and close indicates a close-up shot.

The board indicates whether the game time is displayed in the current frame.

isStop compares the current frame with the frame after 1 second to indicate whether the game time is stopped.

close2far indicates the length of the video until the far shot frame appears if the current frame is a close-up, and far2close: indicates the video length until the close-up frame if the current frame is a far shot.

Items prefixed with '-#' indicate information of the previous #th frame, and items prefixed with '#' indicate information of the #th frame after.

The output permutation indicates whether the current frame corresponds to the event section, none means a normal video, and yes means the event section.

CRFs are undirected graph models trained to maximize conditional probabilities, with parameters Λ={λ,… } is defined as a conditional probability for a label random variable y when an input permutation x is given as shown in [Equation 1] below.

Here, Zx is a normalization constant that makes the sum of the probability values of the label column for the input data column equal to 1.

As described above, the event section identification unit 320 according to an embodiment of the present invention combines the rule-based and learning model-based event section identification, and the start and end points of the event section identified based on the rule-based and the learning model. The final event section is identified using weights and average values for , and the identification accuracy is increased.

5 is a flowchart illustrating a video event unit segmentation method according to an embodiment of the present invention.

The video event unit segmentation method according to an embodiment of the present invention includes the steps of receiving a relay image and text relay information (S100), extracting image information from the relay image (S200), and using the image information to obtain text relay information. and identifying an event section corresponding to the included event and generating an image segment (S300).

Step S100 is a step of receiving a relay image from the video streaming capture unit, and receiving text relay information through real-time crawling.

Step S200 extracts at least one of time information, shot class information, and camera panning information, and processes the extracted information to determine whether the game is in progress, whether the game time is displayed in the video, and the time it takes when the shot class is changed. Image information is extracted by generating at least one of time as a feature.

Step S300 is to identify the event section based on the learning model and rule-based constructed using the image information extracted in step S200 as a feature, and selects the final event section by combining the rule-based and learning model-based event section identification. .

In step S300, in consideration of the error in the text relay information, a point corresponding to the game time on the text relay among the relay images is found, and an event section is identified based on this.

Since there is an error of about 1 to 3 seconds between the event occurrence time on the actual text relay and the actual game time, step S300 recognizes the occurrence point of the event in consideration of the text relay error.

In step S300, the first event section is defined by applying a preset time before and after (eg 15 seconds before, 5 seconds after) based on the event occurrence recognition point, and close-up shot class within the first event section using shot class information Defines a secondary event section in which some areas including .

In addition, step S300 selects a final event section in which a portion in which the camera panning within the secondary event section rapidly increases than a preset value near the section boundary is removed by using the camera panning information.

Step S300 stores the video segment generated using the event identification result as a sub-video clip file, provides an event unit service according to the viewer's selection, and stores event information about the video clip file as metadata to build a database do.

So far, the embodiments of the present invention have been mainly looked at. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: video streaming capture unit 200: real-time crawling unit
310: information collection unit 311: image information extraction unit
312: text relay information pre-processing unit 320: event section identification unit
330: video service unit 400: viewer terminal

Claims (20)

an information collecting unit for receiving a relay image and text relay information and extracting the image information;
an event section identification unit for identifying an event section in an image by using the text relay information and the image information; and
And a video service unit for servicing the video segment generated based on the event section,
The event section identification unit identifies the final event section by removing the corresponding part according to the degree to which the camera panning increases near the section boundary by using the camera panning information for the event section, but when the camera panning occurs and the event If the occurrence time is within a preset time, selecting the final event section without removing the corresponding part
In-video event unit segmentation system.
According to claim 1,
The information collection unit receives the relay image from the video streaming capture unit, and receives the text relay information from the real-time crawling unit
In-video event unit segmentation system.
According to claim 1,
The information collecting unit extracts at least one of time information, shot class information, and the camera panning information from the relay image
In-video event unit segmentation system.
4. The method of claim 3,
The information collection unit processes the extracted information to generate at least one of the characteristics of whether the game is in progress, whether the game time is displayed in the image, and the time required when the shot class is changed, and extracting the image information
In-video event unit segmentation system.
According to claim 1,
The event section identification unit identifies the event section in the image based on a rule and a learning model, and identifies the event section using a weight and an average value for an event start point and an end point, and the learning model in the relay image Constructed using the extracted image information as a feature
In-video event unit segmentation system.
The method of claim 1,
In consideration of the error of the text relay information, the event section identification unit finds a point corresponding to the game time on the text relay among the relay images, and defines the event section based on the recognition point
In-video event unit segmentation system.
7. The method of claim 6,
The event section identification unit selects the final event section for the event section by using the image information
In-video event unit segmentation system.
8. The method of claim 7,
The event section identification unit defines a first event section by applying a preset time before and after the recognition point, and defines a second event section in which a partial region within the first event section is removed using shot class information, , selecting the final event section in which a partial area within the second event section is removed using the camera panning information
In-video event unit segmentation system.
9. The method of claim 8,
The event section identification unit defines the second event section by removing the part including the close-up shot class from the first event section
In-video event unit segmentation system.
9. The method of claim 8,
The event section identification unit selects the final event section by removing a portion in which the camera panning rapidly increases than a preset value near the section boundary in the second event section
In-video event unit segmentation system.
According to claim 1,
The video service unit stores the video segment generated based on the event section as a lower video clip file, stores event information for the video clip file as metadata to build a database, and provides event unit service according to a viewer's selection to provide
In-video event unit segmentation system.
In the video event unit segmentation method performed by the video event unit segmentation system,
(a) receiving a relay image and text relay information;
(b) extracting image information from the relay image; and
(c) identifying an event section corresponding to an event included in the text relay information using the image information, and generating an image segment;
In step (c), the final event section is identified by using camera panning information for the event section, and the corresponding part is removed according to the degree to which the camera panning increases in the vicinity of the section boundary, but when the camera panning occurs and the event If the occurrence time is within a preset time, selecting the final event section without removing the corresponding part
In-video event unit segmentation method.
13. The method of claim 12,
The step (a) is to receive the text relay information through real-time crawling.
In-video event unit segmentation method.
13. The method of claim 12,
Step (b) extracts at least one of time information, shot class information, and the camera panning information, and processes the extracted information to determine whether the game is in progress, whether the game time is displayed in the video, and the shot class is changed. Extracting the image information by generating at least one of the time required for the viewpoint as a quality
In-video event unit segmentation method.
13. The method of claim 12,
The step (c) is to identify the event section based on a learning model and rule built using the image information extracted from the relay image as a feature
In-video event unit segmentation method.
16. The method of claim 15,
The step (c) is to find a point corresponding to the game time on the text relay among the relay images in consideration of the error of the text relay information, and to identify the event section based on this
In-video event unit segmentation method.
16. The method of claim 15,
In step (c), a first event section is defined by applying a preset time before and after the event occurrence recognition point, and a second event section in which a partial region within the first event section is removed using shot class information defining and selecting the final event section in which a partial area within the second event section is removed using the camera panning information
In-video event unit segmentation method.
delete 13. The method of claim 12,
The step (c) stores the video segment generated using the event identification result as a lower video clip file, and provides an event unit service according to a viewer's selection.
In-video event unit segmentation method.
20. The method of claim 19,
The step (c) is to build a database by storing the event information for the video clip file as metadata.
In-video event unit segmentation method.

Images