KR102151668B1 - Apparatus of extracting highlight and method thereof - Google Patents

Abstract

Embodiments disclosed in the present specification relate to a highlight extraction method and a highlight extraction apparatus for extracting a highlight image from an original game image using machine learning. As a technical means for achieving the technical problem, according to an embodiment, a highlight extraction method performed by a highlight extraction apparatus, comprising: receiving an original game image, sampling an image included in the original game image, and sampled Disclosed is a highlight extraction method including grouping images in units of scenes and extracting highlight images based on the grouped scenes.

Description

Highlight extraction method and highlight extraction device {APPARATUS OF EXTRACTING HIGHLIGHT AND METHOD THEREOF}

Embodiments disclosed in the present specification relate to a highlight extraction method and a highlight extraction apparatus for extracting a highlight image from an original game image using machine learning (Machine Learnig).

With the development of technology, the production and distribution of video content is overflowing. Accordingly, there is a need for a technology that effectively processes image content and improves utilization.

For example, according to Korean Patent Registration No. 10-1833943, a'method and system for extracting and searching for main scenes of a moving image' is disclosed, and a method and system for extracting a representative image of a moving image is disclosed.

However, according to the above-described prior art, the representative image extracted as a result by extracting the image visually protruding from the moving picture is unfortunate in terms of utilization as an image unrelated to the user's interest or interest is extracted.

On the other hand, the above-described background technology is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and is not necessarily known to be publicly known before filing the present invention. .

Embodiments disclosed in the present specification are intended to disclose a highlight extraction method and a highlight extraction apparatus for extracting a highlight image from a game image using machine learning.

In addition, the embodiments aim to extract a highlight image suitable for the interest and interest of the viewer by analyzing the content visually expressed in the game image.

As a technical means for achieving the above-described technical problem, according to an embodiment, a highlight extraction method performed by a highlight extraction apparatus, comprising: receiving an original game image, sampling an image included in the original game image, Disclosed is a highlight extraction method including grouping the sampled images in a scene unit and extracting a highlight image based on the grouped scenes.

According to another embodiment, a computer-readable recording medium on which a program for performing a highlight extraction method is recorded is disclosed. In this case, the highlight extraction method includes: receiving an original game image, sampling an image included in the original game image, grouping the sampled images in units of scenes, and extracting a highlight image based on the grouped scenes. It may include.

According to another embodiment, a computer program is disclosed that is performed by a job management apparatus and stored in a medium to perform a method of extracting highlights. In this case, the highlight extraction method includes: receiving an original game image, sampling an image included in the original game image, grouping the sampled images in units of scenes, and extracting a highlight image based on the grouped scenes. It may include.

According to another embodiment, a communication unit for receiving an original game image, a storage unit for storing the original game image, and a control unit for controlling the communication unit and the storage unit to extract a highlight image, wherein the control unit, Disclosed is a highlight extraction apparatus for sampling an image included in the original game video, grouping the sampled images in units of scenes, and extracting a highlight image based on the grouped scenes.

Embodiments disclosed in the present specification may disclose a highlight extraction method and a highlight extraction apparatus for extracting a highlight image from a game image using machine learning.

In addition, embodiments may extract a highlight image suitable for the interest and interest of the viewer by analyzing the content visually expressed in the game image.

The effects that can be obtained in the disclosed embodiments are not limited to the above-mentioned effects, and other effects not mentioned are obvious to those of ordinary skill in the art to which the embodiments disclosed from the following description belong. Can be understood.

1 is a block diagram showing the configuration of a highlight extraction apparatus according to an embodiment.
2 to 3 are exemplary views for explaining an embodiment of a highlight extraction method performed by the highlight extraction apparatus.
4 to 7 are flowcharts illustrating a method of extracting highlights according to an exemplary embodiment.
8 is an exemplary diagram illustrating a method of extracting highlights according to exemplary embodiments.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly describe the features of the embodiments, detailed descriptions of matters widely known to those of ordinary skill in the art to which the following embodiments belong are omitted. In addition, parts not related to the description of the embodiments are omitted in the drawings, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a configuration is said to be "connected" with another configuration, this includes not only a case of being'directly connected' but also a case of being'connected with another configuration in between'. In addition, when a certain configuration "includes" a certain configuration, this means that other configurations may be further included rather than excluding other configurations, unless otherwise specified.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a highlight extraction apparatus 10 according to an embodiment.

The highlight extraction apparatus 10 according to an exemplary embodiment disclosed in the present specification may extract a highlight image from an original game image based on machine learning. In this case, the game may include a sports game such as baseball, and the original game video refers to a video including the game situation.

According to an embodiment, the highlight extraction device 10 may be implemented as an electronic terminal in which an application for extracting a highlight image is installed. In addition, the highlight extraction device 10 may be implemented as a server or a server-client system. According to an embodiment, the highlight extraction device 10 implemented as a server-client system may include a user terminal in which a client for interaction with a user is installed, and in this case, the user terminal may be implemented as an electronic terminal.

Referring to FIG. 1, the highlight extraction device 10 may include an input/output unit 11, a communication unit 12, a storage unit 13, and a control unit 14.

The input/output unit 11 according to an embodiment may include an input unit for receiving an input from a user, and an output unit for displaying information on a result of performing an operation, for example, a result of performing a highlight extraction method. .

For example, the input unit may include various types of input receiving devices such as a keyboard, a physical button, a touch screen, a camera, or a microphone for receiving a user's input. In addition, the output unit may include a display panel, a speaker, or a headset. However, the input/output unit 11 is not limited to the above-described example and may include a configuration supporting various input/output.

Meanwhile, the communication unit 12 may perform wired or wireless communication with other devices or networks. To this end, the communication unit 12 may include a communication module that supports at least one of various wired and wireless communication methods. For example, the communication module may be implemented in the form of a chipset.

The wireless communication supported by the communication unit 12 may be, for example, Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Bluetooth, Ultra-Wide Band (UWB), or Near Field Communication (NFC). In addition, wired communication supported by the communication unit 12 may be, for example, USB or High Definition Multimedia Interface (HDMI).

According to an embodiment, the communication unit 12 may receive an original game image to be subjected to highlight extraction, and transmit/receive data for performing a highlight extraction method such as various data sets for performing machine learning.

Meanwhile, the storage unit 13 may install and store various types of data for performing the highlight extraction method. According to an embodiment, the storage unit 13 may store and drive software such as an application and/or a device driver for performing the highlight extraction method.

For example, the storage unit 13 may store data for performing a highlight extraction method, such as an original game image and a data set for machine learning.

Meanwhile, the control unit 14 controls the overall operation of the highlight extraction apparatus 10 and may include a processor such as a CPU or the like. According to an embodiment, the control unit 14 controls other components such as the input/output unit 11, the communication unit 12, and the storage unit 13 included in the highlight extraction device 10 to perform the highlight extraction method, and Various data can be processed to perform the extraction method.

Next, FIG. 2 is an exemplary diagram for explaining an embodiment of a method for extracting a highlight performed by the highlight extracting apparatus 10.

Referring to FIG. 2, the controller 14 may sample an image 22 from an original game image 21. In this case, the sampled image 22 may refer to a still image included in the original game image 21 which is a moving picture.

According to an embodiment, the control unit 14 selects a specific frame, for example, an image 22 corresponding to an I frame among I, P, and B frames of a Group of Picture (GOP) included in the original game video 21. Can be sampled. In addition, the controller 14 may decode the original game image 21 into an image, and then sample the image 22 at predetermined playback time intervals.

Meanwhile, the controller 14 may group the sampled images in units of scenes.

According to an embodiment, the controller 14 may group the sampled images in units of scenes based on the similarity of adjacent images among the sampled images. Here, the "adjacent image" may mean an image that is adjacent when sampled images are arranged in the order of time in which the images are played.

For example, the controller 14 may calculate similarity of images by performing feature matching on adjacent images. For example, the controller 14 may group images showing a degree of similarity of a predetermined degree or more into one scene by comparing feature points of adjacent images.

In addition, the controller 14 may determine whether to include a scoreboard by selecting a selected image from among images included in the grouped scene.

Here,'selected image' is a term for specifying an image selected to determine whether the scoreboard is included, and according to an embodiment, the control unit 14 may select an arbitrary image from among the images included in the scene as the selected image. Yes, one or more can be selected. In addition, when there is more than one grouped scene, the controller 14 may select one or more selected images for each scene.

In addition, the'scoreboard' is for visually conveying the main matters of the game progress, and can be implemented in the form of a table including symbols such as letters, numbers, and pictograms. It may be included, or included in the game video by inserting and processing a scoreboard implemented in computer graphics.

According to an embodiment, the control unit 14 determines whether or not the selected image includes a scoreboard, but the training data set including the image with the scoreboard and the test data set including the image without the scoreboard are trained in the scoreboard detection model. Based on the selection image, it can be determined whether or not the scoreboard is included.

To this end, the controller 14 may build a scoreboard detection model. For example, the control unit 14 is a CNN deep learning model including a convolution layer, a ReLu layer, a Max pooling layer, and a fully connected layer ( CNN Deep Learning Model) can be built. In this case, the CNN deep learning model may be designed to include two convolutional layers, a relu layer, a max pooling layer, and one fully connected layer.

In addition, the control unit 14 inputs and outputs a training data set including at least one sample image including a scoreboard and a test data set including at least one sample image without a scoreboard for learning a scoreboard detection model. 11) Alternatively, it can be input through the communication unit 12. Here, the sample images included in the data set may be extracted from various sample images.

According to an embodiment, the control unit 14 includes a training data set including a sample image including a scoreboard and a test data set (<image#) including a sample image without a scoreboard, as represented in [Table 1]. , The presence of a scoreboard>) can be entered.

In addition, the controller 14 may perform training by extracting an area where the scoreboard is expected to be present from the sample image included in the data set, and inputting the extracted area image into the scoreboard detection model. To this end, the control unit 14 may be configured with an area where the scoreboard is expected to be located. For example, as the area where the scoreboard is expected, the coordinate value of the upper left corner of the sample image (e.g., from (50, 0) to (400, 150)) is input, and the area corresponding to the input coordinate value is extracted. Training can be performed.

In addition, the control unit 14 may determine whether the selected image includes a scoreboard by substituting the selected image into the learned scoreboard detection model as described above. In this case, the selection image that is substituted into the scoreboard detection model may mean a partial area of the selection image in which the scoreboard is expected to exist. That is, the control unit 14 may determine whether or not the selected image includes the scoreboard by extracting a partial area expected to have a scoreboard from the selection image and replacing it in the scoreboard detection model. For example, the control unit 14 receives the coordinates of the area where the scoreboard is expected to be present in the selected image, or extracts the same area as the coordinates of the area extracted from the image included in the data set used for learning from the selected image to score. Can be substituted into the plate analysis model.

In addition, if the scoreboard is included in the selection image, the controller 14 may analyze information on the scoreboard of the selection image.

According to an embodiment, the controller 14 may analyze the scoreboard information based on the scoreboard analysis model.

To this end, the controller 14 may build a scoreboard analysis model. For example, the control unit 14 is a CNN deep learning model including a convolution layer, a ReLu layer, a Max pooling layer, and a fully connected layer ( CNN Deep Learning Model) can be built. In this case, the CNN deep learning model may be constructed to include two convolutional layers, a relu layer, a max pooling layer, and one fully connected layer.

In addition, the control unit 14 is for learning a scoreboard analysis model, and can receive a data set including sample images including a scoreboard matching interest information through the input/output unit 11 or the communication unit 12. have.

At this time,'interest information' refers to an event displayed through the scoreboard as an important matter in the progress of the game.For example, in the case of a baseball game, information on the round, score situation, ball count, advance situation, out count, etc. I can. In other words, each scoreboard can visually express interest information intended to be represented by the scoreboard. Accordingly, the image included in the data set for training of the scoreboard analysis model is matched with interest information indicated by the inserted scoreboard as a scoreboard inserted to form a data set.

According to an embodiment, the controller 14 may receive a data set in which a sample image and interest information visually expressed in each sample image are matched, as represented in Table 2 above.

In addition, the control unit 14 may perform training by extracting a region where the scoreboard is expected to be present from the sample image included in the data set, and substituting the extracted region image into the scoreboard analysis model. At this time, the control unit 14 may receive a setting of an area where the scoreboard is expected to be located. For example, as the area where the scoreboard is expected, the coordinate value of the upper left corner of the image (e.g., from (50, 0) to (400, 150)) is input, and the area corresponding to the input coordinate value is extracted and trained. Can be done.

In addition, the control unit 14 may analyze information on the scoreboard by substituting the selected image into the learned scoreboard analysis model as described above. In this case, the selection image that is substituted into the scoreboard analysis model may mean a partial area of the selection image in which the scoreboard is expected to be present. For example, the control unit 14 receives the coordinates of the area where the scoreboard is expected to be present in the selected image, or extracts the same area as the coordinates of the area extracted from the image included in the data set used for learning from the selected image to score Can be substituted into the plate analysis model.

In this case, the information on the scoreboard analyzed by the control unit 14 may include interest information indicated by the scoreboard included in the selection image. For example, in the case of a baseball game, the information on the scoreboard analyzed by the control unit 14 includes information on the round, score situation, ball count, advance situation, out count, etc. in the scene including the selection image and the selection image. can do.

In addition, the control unit 14 analyzes the information on the scoreboard, but analyzes the source of the original game video, and analyzes the scoreboard information of the selected image based on the scoreboard analysis model learning a data set matching the analyzed source. can do.

Here, the “source of the original game video” refers to a producer who processed the original game video including the selected image, and may mean each broadcaster such as KBS, SBS, and MBC. The format of the scoreboard inserted into the game video may differ depending on the source of the game video. Therefore, the accuracy of analysis can be improved by preparing and learning different data sets for each source of the original game video.

To this end, the controller 14 may build a source analysis model. For example, the control unit 14 is a CNN deep learning model including a convolution layer, a ReLu layer, a Max pooling layer, and a fully connected layer ( CNN Deep Learning Model) can be built. In this case, the CNN deep learning model may be constructed to include two convolutional layers, a relu layer, a max pooling layer, and one fully connected layer.

In addition, the control unit 14 may receive and learn a training data set and a test data set for each of one or more sources.

According to an embodiment, the controller 14 may receive a sample image and a data set (<image#, source>) in which the source of each sample image matched as shown in [Table 3].

In addition, the control unit 14 may extract an area where the logo of the source is expected to be present from the data set, and input the extracted area image into the source analysis model to perform training. In this case, the control unit 14 may be configured to a region where the logo of the source is expected to be present. For example, as the area where the logo of the source is expected to be located, the coordinate value of the upper right corner of the image (e.g., from (50, 0) to (400, 150)) is input, and the area corresponding to the input coordinate value is extracted. You can do the training.

In addition, the control unit 14 may analyze the source by substituting the selected image into the learned source analysis model as described above. In this case, the selected image substituted in the source analysis model may mean a partial area of the selected image in which the logo of the source is expected to be present.

In addition, the control unit 14 may analyze the scoreboard information by substituting the selected image from which the source is analyzed to the scoreboard analysis model.

According to an embodiment, the control unit 14 may include a scoreboard analysis model according to each source. That is, the controller 14 may design one or more scoreboard analysis models, and learn data sets corresponding to different sources through each scoreboard analysis model. To this end, the controller 14 may receive a data set corresponding to each source.

In addition, the controller 14 may analyze scoreboard information of the selected image based on the scoreboard analysis model in which the data set corresponding to the source of the selected image is learned.

In addition, the controller 14 may extract the highlight image based on the analyzed scoreboard information.

For example, the controller 14 may extract the highlight image in units of sequence based on the analyzed information on the scoreboard.

In this case, the'sequence' refers to an image unit in which one or more scenes are sequentially arranged to form a flow in content. According to the embodiment disclosed in the present specification, the controller 14 may group one or more scenes to generate one sequence.

To this end, the controller 14 may analyze the type of each scene.

In this case, the'scene type' refers to the type of situation expressed in each scene. For example, in the case of a baseball game, the type of scene is a pitch view, a wide shot of receiving a ball (Catch Overview), a close shot of receiving a ball (Catch Close-up), and a runner. A scene in which the moving situation is widely photographed (Running Overview), a scene in which a runner is moving closely (Running Close-up), a scene in the audience seat (Audience View), a scene in which the user touches the base is photographed closely (Touch base Close-up).

According to an embodiment, the controller 14 may build a scene type analysis model as a CNN Deep Learning Model and learn the above-described data set. In this case, the CNN deep learning model may be designed to include two convolutional layers, a relu layer, a max pooling layer, and one fully connected layer. In addition, according to an embodiment, the control unit 14 constructs a feature sequence in the order of the maps of the convolution feature maps calculated from the CNN using the RCNN technique, and then short-shorts each feature sequence. You can learn by substituting for Long Short Term Memory Networks (LSTM).

In addition, the controller 14 is for learning a scene type analysis model, and may receive a data set including sample images corresponding to each scene type through the input/output unit 11 or the communication unit 12.

According to an embodiment, the controller 14 receives and learns a data set (<image#, scene type>) in which a sample image and a scene type corresponding to each sample image are matched as shown in [Table 4]. I can.

In addition, the controller 14 may determine the type of each scene by substituting each scene into the learned scene type analysis model as described above. In this case, each scene substituted in the scene type analysis model may mean an image constituting each scene. That is, the controller 14 may determine the type of each scene by selecting one or more images constituting each scene and substituting it into the scene type analysis model.

Also, the controller 14 may classify a sequence based on the type of scene. For example, from a specific type of scene to a specific type of scene can be divided into one sequence. To this end, the control unit 14 may be configured to set the type of scene to classify the sequence, and may receive different types of scene to classify the sequence according to a corresponding sport. For example, in the case of a baseball game, from a pitching scene to a scene immediately before the next pitching scene may be set as one sequence.

Meanwhile, the controller 14 may analyze the game situation of the sequence based on the analyzed scoreboard information. For example, the analyzed scoreboard information can be compared to each other to analyze the game situation. For example, it is possible to analyze the game situation according to the change of scoreboard information within the sequence.

In addition, the controller 14 may extract the highlight image according to the analyzed game situation. In this case, the controller 14 may extract the highlight image in sequence units. For example, when the main situation is set and the analyzed game situation corresponds to the set main situation, the controller 14 may generate a highlight image including a sequence including the game situation corresponding to the main situation. For example, when the score situation is set as the main situation, a sequence including the score situation may be extracted as a highlight image. In addition, in the case of a baseball game, the main situation may be set as an out situation in which the out count increases, a long hit situation that advances to second base or more.

According to an embodiment, the controller 14 may generate a highlight image including a sequence including a game situation corresponding to a major situation and a sequence before and after, and generate a plurality of sequences including a game situation corresponding to the major situation. It is also possible to extract and generate a highlight image.

Here, referring to FIG. 3, FIG. 3 is an exemplary diagram for explaining an embodiment of a highlight extraction method performed by the highlight extraction apparatus 10.

According to FIG. 3, the controller 14 may analyze the selected image and scoreboard information 31 of the scene corresponding to the selected image based on the selected image included in each scene. Also, the controller 14 may analyze the scene type 32 of each scene and divide the scene into sequence units 33 based on the analyzed scene type.

In addition, the control unit 14 may analyze the game situation by comparing the scoreboard information. For example, if the score in scene A is 0:0 and the score in scene D is 4:0, scoreboard information of two scenes is compared to analyze that a score of 4 points has occurred in sequence 1. In this case, when the analyzed game situation, that is, the score situation is set as the main situation, the controller 14 may extract the highlight image to include the sequence 1.

Also, the controller 14 may generate a title of the extracted highlight image. For example, the controller 14 may generate a title of an image based on scoreboard information corresponding to the extracted highlight image. For example, if a match situation in which team A scored 4 points in sequence 1 is analyzed, the control unit 14 may generate the title of the highlight image including sequence 1 as a'team A full home run scene'.

Meanwhile, the controller 14 may extract the highlight image based on the scene order. In other words, in the scene grouped from the image extracted from the original game video, the controller 14 can analyze the scene type of each consecutive scene and extract the highlight image based on the sequence of the scenes according to the analyzed type. have. For example, in the case of a baseball game video, four consecutive scenes may have a sequence of pitching -> batting -> infield -> spectators.

In addition, if the analyzed scene order corresponds to a preset order, the controller 14 may extract consecutive scenes corresponding to the preset order as highlight images.

According to an embodiment, the control unit 14 determines whether or not the selected image contains a scoreboard, and if the scoreboard is not included, the control unit 14 can improve the efficiency of the highlight image by extracting the highlight image based on the order of the scene types. have.

According to an embodiment, the controller 14 may learn a training data set including an image edited as a highlight and a test data set including an image other than the highlight based on the highlight prediction model. In this case, the highlight prediction model may learn a relationship between the highlight image and the scene order by analyzing the scene order of the image edited as the highlight of the training data set and analyzing the scene order of the image other than the highlight of the test data set.

According to an embodiment, the controller 14 may generate a training data set using the highlight image extracted based on scoreboard information, and generate a test data set using an image excluding the highlight image from the original game image, You can learn this.

In addition, the controller 14 analyzes whether the analyzed scene order is a highlight image based on the learned highlight prediction model, and selects a continuous scene having a scene order corresponding to a preset order as analyzed as a highlight image. Can be extracted with In this case, the controller 14 may calculate a result value by specifying the continuous scene analyzed as the highlight image as the start time and the end time of the continuous scene.

Hereinafter, an embodiment of a highlight extraction method performed by the highlight extraction device 10 will be described in more detail with reference to FIGS. 4 to 7.

4 to 7 are flowcharts illustrating an embodiment of a method for extracting highlights performed by the highlight extracting apparatus 10. The highlight extraction method illustrated in FIGS. 4 to 7 includes steps processed in a time series in the highlight extraction apparatus 10 illustrated in FIGS. 1 to 3. Therefore, even if omitted below, the contents described above with respect to the highlight extraction method illustrated in FIGS. 1 to 3 may also be applied to the highlight extraction method according to the exemplary embodiments illustrated in FIGS. 4 to 7.

First, referring to FIG. 4, the highlight extraction device 10 may receive an original game image (S41), sample an image included in the original game image (S42), and group the sampled images in units of scenes ( S43). According to an embodiment, the highlight extraction apparatus 10 may group the sampled images in units of scenes based on the similarity of adjacent images among the sampled images.

In addition, the highlight extraction apparatus 10 may extract a highlight image based on the grouped scene (S44).

In this case, referring to FIG. 5, the highlight extraction device 10 may determine whether or not the selected image includes a scoreboard by selecting a selected image among images included in the scene (S51 ). According to an embodiment, the highlight extraction device 10 includes a scoreboard of a selected image based on a scoreboard detection model obtained by learning a training data set including an image with a scoreboard and a test data set including an image without a scoreboard. You can judge whether or not.

In addition, if the scoreboard is included in the selected image, the highlight extraction device 10 may analyze scoreboard information of the selected image (S52). According to an embodiment, the highlight extraction apparatus 10 may analyze scoreboard information of a selected image based on a scoreboard analysis model obtained by learning a data set including images including a scoreboard matching interest information.

According to an embodiment, in order to improve the accuracy of the scoreboard information analysis, the highlight extraction device 10 analyzes the source of an original game video and learns a data set matching the analyzed source based on a scoreboard analysis model. The scoreboard information of the selected image can be analyzed. In this case, the data set matching the source may include images that match the source and include a scoreboard matching interest information.

In addition, the highlight extraction device 10 may extract the highlight image based on the analyzed scoreboard information (S53).

According to an embodiment, the highlight extraction device 10 extracts a highlight image based on scoreboard information (S53), and analyzes the game situation of the sequence based on the scoreboard information (S61), and based on the analyzed game situation Thus, the highlight image may be extracted (S61).

To this end, the highlight extraction apparatus 10 may group scenes in units of sequences. According to an embodiment, after the step (S43) of grouping the sampled images in units of scenes, the highlight extraction apparatus 10 analyzes the scene types of each scene, and groups the scenes in units of sequences based on the analyzed scene types. can do.

Meanwhile, in extracting the highlight image based on the grouped scene (S44), the highlight extraction device 10 analyzes the type of each scene (S71), and extracts the highlight image based on the order of the analyzed scene types. It can be done (S72). According to an embodiment, if the order of the analyzed scene types corresponds to a preset order, the highlight extracting apparatus 10 may extract a highlight image including one or more scenes corresponding to the preset order.

At this time, in order to improve the efficiency of the highlight image extraction, the highlight extraction device 10 analyzes the scene type (S71), selects a selected image from among images included in the scene, and if the scoreboard of the selected image is not included, You can analyze the scene type of each scene.

That is, when the scoreboard is included in the selected image, the highlight extraction device 10 analyzes the scoreboard information, extracts the highlight image based on the scoreboard information, and when the scoreboard is not included in the selected image, the sequence of scenes May be analyzed, and a highlight image may be extracted based on the scene order.

Meanwhile, FIG. 8 is a diagram illustrating an embodiment of a highlight extraction method performed by the highlight extraction apparatus 10.

Referring to FIG. 8, the highlight extraction apparatus 10 may receive an original game image (S81), sample an image included in the original game image (S82), and group the sampled images into scene units (S83). .

In addition, the highlight extraction device 10 may determine whether the selected image includes a scoreboard (S84), and when the selected image includes a scoreboard, analyze the scoreboard information of the selected image (S85). In addition, the highlight extraction device 10 analyzes the game situation of the sequence based on the scoreboard information, but analyzes the type of each scene (S86), grouping them in sequence units (S87), and the grouped sequence and scoreboard information. Based on the sequence, it is possible to analyze the game situation (S88). In this case, the highlight extraction device 10 may extract the highlight image based on the game situation of the sequence (S89).

Meanwhile, when the selected image does not include a scoreboard, the highlight extraction device 10 may analyze the scene type of each scene (S810) and analyze the scene order (S811). In this case, the highlight extraction apparatus 10 may extract the highlight image based on the scene order (S812).

As described above, the highlight extraction apparatus 10 may extract a highlight image from an original game image using machine learning, and may perform a highlight extraction method that effectively improves accuracy according to whether or not a scoreboard is included.

The term'~ unit' used in the above embodiments refers to software or hardware components such as field programmable gate array (FPGA) or ASIC, and the'~ unit' performs certain roles. However,'~ part' is not limited to software or hardware. The'~ unit' may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Therefore, as an example,'~ unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , Subroutines, segments of program patent code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays, and variables.

The components and functions provided in the'~ units' may be combined into a smaller number of elements and'~ units' or separated from the additional elements and'~ units'.

In addition, components and'~ units' may be implemented to play one or more CPUs in a device or a security multimedia card.

The highlight extraction method according to the embodiments described with reference to FIGS. 4 to 7 may be implemented in the form of a computer-readable medium storing instructions and data executable by a computer. In this case, the instructions and data may be stored in the form of a program code, and when executed by a processor, a predetermined program module may be generated to perform a predetermined operation. Further, the computer-readable medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, and removable and non-removable media. In addition, the computer-readable medium may be a computer recording medium, which is volatile and non-volatile implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It can include both volatile, removable and non-removable media. For example, the computer recording medium may be a magnetic storage medium such as HDD and SSD, an optical recording medium such as CD, DVD, and Blu-ray disk, or a memory included in a server accessible through a network.

In addition, the highlight extraction method according to the embodiments described with reference to FIGS. 4 to 7 may be implemented as a computer program (or a computer program product) including instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor, and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language. . Further, the computer program may be recorded on a tangible computer-readable recording medium (eg, memory, hard disk, magnetic/optical medium, solid-state drive (SSD), etc.).

Accordingly, the highlight extraction method according to the embodiments described with reference to FIGS. 4 to 7 may be implemented by executing the above-described computer program by the computing device. The computing device may include at least some of a processor, a memory, a storage device, a high speed interface connected to the memory and a high speed expansion port, and a low speed interface connected to the low speed bus and the storage device. Each of these components is connected to each other using various buses and can be mounted on a common motherboard or in other suitable manner.

Here, the processor can process commands within the computing device. Such commands include, for example, to display graphic information for providing a GUI (Graphic User Interface) on an external input or output device, such as a display connected to a high-speed interface. Examples are instructions stored in memory or storage devices. As another embodiment, multiple processors and/or multiple buses may be utilized with multiple memories and memory types as appropriate. In addition, the processor may be implemented as a chipset formed by chips including a plurality of independent analog and/or digital processors.

The memory also stores information within the computing device. As an example, the memory may be composed of volatile memory units or a set of them. As another example, the memory may be composed of a nonvolatile memory unit or a set of them. Also, the memory may be another type of computer-readable medium such as a magnetic or optical disk.

In addition, the storage device may provide a large-capacity storage space to the computing device. The storage device may be a computer-readable medium or a configuration including such a medium, for example, devices in a storage area network (SAN) or other configurations, a floppy disk device, a hard disk device, an optical disk device, Or it may be a tape device, a flash memory, or another semiconductor memory device or device array similar thereto.

The above-described embodiments are for illustration only, and those of ordinary skill in the art to which the above-described embodiments belong can easily transform into other specific forms without changing the technical idea or essential features of the above-described embodiments. You can understand. Therefore, it should be understood that the above-described embodiments are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

10: highlight extraction device
11: input/output unit 12: communication unit
13: storage unit 14: control unit

Claims (23)

In the highlight extraction method performed by the highlight extraction device,
Receiving an original game video;
Sampling an image included in the original game video;
Grouping the sampled images in units of scenes; And
Including the step of extracting the highlight image based on the grouped scene,
The step of extracting the highlight image,
The score of the selected image is based on a scoreboard detection model that selects a selection image from among the images included in the scene and learns a training data set including an image with a scoreboard and a test data set including an image without a scoreboard. Determining whether a plate is included;
If the scoreboard is included in the selection image, analyzing scoreboard information of the selection image; And
And extracting a highlight image based on the scoreboard information. The method of claim 1,
Grouping the sampled images by scene unit,
And grouping the sampled images in a scene unit based on a similarity of images adjacent to each other among the sampled images. delete The method of claim 1,
After the step of grouping the sampled image by scene unit,
Analyzing a scene type of each of the scenes; And
Grouping the scenes in sequence units based on the analyzed scene type,
Extracting the highlight image based on the scoreboard information,
Analyzing a game situation of the sequence based on the scoreboard information; And
And extracting a highlight image based on the game situation. delete The method of claim 1,
Analyzing the scoreboard information,
A method for extracting highlights, comprising analyzing scoreboard information of the selected image based on a scoreboard analysis model obtained by learning a data set including images including a scoreboard matching interest information. The method of claim 1,
Analyzing the scoreboard information,
Analyzing the source of the original game video; And
Analyzing scoreboard information of the selected image, based on the scoreboard analysis model obtained by learning the analyzed data set matching the source,
Data sets matching the above sources,
As images matching the source, including images including a scoreboard matching interest information. The method of claim 7,
Analyzing the source of the original game video,
A method for extracting highlights, characterized in that the source of the game image source is analyzed based on a source analysis model obtained by learning a training data set and a test data set corresponding to each of one or more sources. The method of claim 1,
The step of determining whether the selected image includes a scoreboard,
If the selected image does not include a scoreboard, analyzing a scene type of each of the scenes; And
And extracting a highlight image based on the analyzed scene order according to the type. The method of claim 9,
Analyzing the scene type,
Selecting a selected image from among the images included in the scene, and determining whether the selected image includes a scoreboard; And
If the scoreboard is not included in the selected image, comprising the step of analyzing the scene type of each of the scenes,
Extracting the highlight image based on the scene order,
When the scene order corresponds to a preset order, the successive scenes corresponding to the scene order are extracted as a highlight image. The method of claim 9,
Extracting the highlight image based on the scene order,
A training data set including an image edited as a highlight and a test data set including a non-highlight image are learned, but the relationship between the highlight image and the scene order is learned by analyzing the sequence of the training data set and the test data set. A highlight extraction method, characterized in that, based on one highlight prediction model, a highlight image is extracted. A computer-readable recording medium on which a program for performing the method according to claim 1 is recorded. A computer program performed by a highlight extraction device and stored on a recording medium to perform the method according to claim 1. Communication unit for receiving the original game video;
A storage unit for storing the original game image received; And
Comprising a control unit for controlling the communication unit and the storage unit to extract a highlight image,
The control unit,
Sampling the images included in the original game video, grouping the sampled images in units of scenes, and extracting a highlight image based on the grouped scenes, including a scoreboard by selecting a selected image from among the images included in the scene It is determined whether or not, based on a scoreboard detection model obtained by learning a training data set including an image with a scoreboard and a test data set including an image without a scoreboard, determining whether or not the selected image contains a scoreboard, and the If the scoreboard is included in the selected image, the scoreboard information of the selected image is analyzed, and a highlight image is extracted based on the scoreboard information. The method of claim 14,
The control unit,
And grouping the sampled images in units of scenes based on the similarity of images adjacent to each other among the sampled images. delete The method of claim 14,
The control unit,
Analyzing the scene type of each of the scenes, grouping the scenes in sequence units based on the analyzed scene type, analyzing the game situation of the sequence based on the scoreboard information, and highlighting images based on the game situation To extract, highlight extraction device. delete The method of claim 14,
The control unit,
A highlight extraction device that analyzes scoreboard information, but analyzes scoreboard information of the selected image based on a scoreboard analysis model obtained by learning a data set including images including a scoreboard matching interest information. The method of claim 14,
The control unit,
Analyzing the information on the scoreboard, analyzing the source of the original game image, and analyzing the scoreboard information of the selected image based on a scoreboard analysis model that learned a data set matching the analyzed source,
Data sets matching the above sources,
An image matching the source, including images including a scoreboard matching interest information. The method of claim 20,
The control unit,
Analyzing the source of the original game video, but analyzing the source of the original game video based on a source analysis model obtained by learning a training data set and a test data set corresponding to each of one or more sources. The method of claim 14,
The control unit,
If the selected image does not include a scoreboard, analyzing the scene type of each of the scenes, and extracting a highlight image based on the analyzed scene order according to the type, highlight extraction device. The method of claim 22,
The control unit,
A training data set including an image edited as a highlight and a test data set including a non-highlight image are learned, but the relationship between the highlight image and the scene order is learned by analyzing the sequence of the training data set and the test data set. A highlight extraction device, characterized in that for extracting a highlight image based on one highlight prediction model.

Images