JP2022040665A - Video processing device, video processing method, and model generation device - Google Patents

Abstract

To provide a video processing device, a video processing method, and a model generation device, capable of generating appropriate highlight scenes.SOLUTION: A video generation device (10) includes: an extraction part (highlight extraction part 17) to extract, from a sport video, a reference scene including characteristic movements and characteristic voices; and a generation part (highlight extraction part 17) to generate, as a highlight scene, a scene including from a first scene that returns only by a first predetermined time to a second scene that advances only by a second predetermined time, with the reference scene extracted by the extraction part as a reference.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video processing device, a video processing method, and a model generation device.

Conventionally, an invention for generating a highlight scene has been known (Patent Document 1). The invention described in Patent Document 1 generates a scene as a highlight scene when the volume of cheers is larger than a predetermined value.

Japanese Unexamined Patent Publication No. 2012-147296

However, there are scenes that do not show the players even when the cheers are loud, and such scenes are not always appropriate as highlight scenes.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a video processing device, a video processing method, and a model generating device capable of generating an appropriate highlight scene.

The image processing apparatus according to one aspect of the present invention has a first extraction unit that extracts a reference scene containing a characteristic motion or a characteristic sound from a sports image, and a reference scene extracted by the extraction unit as a reference. It is provided with a generation unit that generates a scene including a first scene that has returned by a predetermined time to a second scene that has advanced by a second predetermined time as a highlight scene.

According to the present invention, it is possible to generate an appropriate highlight scene.

FIG. 1 is a schematic configuration diagram of a video processing device 10 and a model generation device 50 according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a highlight scene extraction method. FIG. 3 is a diagram illustrating another example of the highlight scene extraction method. FIG. 4 is a flowchart illustrating an operation example of the video processing apparatus 10 according to the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

(Configuration example of video processing device)
The image processing device 10 is a device that extracts a highlight scene and generates the extracted highlight scene as a highlight image. When there are multiple highlight scenes, the highlight image is a combination of multiple highlight scenes. In the present embodiment, the highlight scene means a so-called highlight of sports, and is typically a sports scoring scene. However, some sports do not have a scoring scene (for example, sumo). In sumo wrestling where there is no scoring scene, the highlight scene corresponds to the scene where the wrestler's skill is decided. In the present embodiment, the image processing device 10 will be described as a device different from the model generation device 50, but some or all the functions of the model generation device 50 may be incorporated in the image processing device 10.

An example of the configuration of the video processing device 10 and the model generation device 50 will be described with reference to FIG. First, the video processing apparatus 10 will be described. As shown in FIG. 1, the image processing device 10 includes a storage device 11, a control unit 12, and an interface 13.

The storage device 11 is a device for recording or recording the content acquired by the video acquisition device 20, and is typically a recorder. As an example, such a recorder is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Note that cloud storage using cloud computing may be used as the storage device 11. Therefore, the image processing device 10 does not necessarily have to include the storage device 11.

The video acquisition device 20 is, for example, a device for acquiring a television program, and is typically a television. Further, the video acquisition device 20 may be a smartphone, a tablet terminal, a personal computer, or a game machine that acquires video content through an Internet line. Where TV programs or video content include sports broadcasts, dramas, movies, documentaries, animations, news, etc., the target of highlight scene extraction in this embodiment is sports broadcasts. Regarding the types of sports, if the sport is provided through a television or an internet line, it will be the target of highlight scene extraction. Typical sports offered via television or internet access include baseball, basketball, American football, ice hockey, soccer, tennis, volleyball, table tennis, rugby, cricket, fencing, golf and sumo. In this embodiment, sports broadcasts and sports contents are collectively referred to as sports images. Therefore, the sports video includes sports broadcasts acquired through television and sports contents acquired via the Internet line. However, in order to simplify the explanation, the target of highlight scene extraction is the TV program below. The function of the video acquisition device 20 may be incorporated in the video processing device 10. In this case, the video processing device 10 can function as a video camera that is easy to carry. A video camera that is easy to carry is sometimes called a handy camera. The above-mentioned sports video also includes sports video taken by a general user with a video camera.

As an example, the interface 13 is implemented as hardware such as a network adapter, communication software, or a combination thereof, and is configured to realize wired or wireless communication. Further, the interface 13 has a function as an input unit and an output unit for transmitting and receiving data.

The interface 13 is used for communication with the user terminal 30 and communication with the model generation device 50 via the network 40. The user terminal 30 is a device operated by the user, and is typically a remote controller for operating the recorder. However, the user terminal 30 is not limited to the remote controller. The user terminal 30 may be a smartphone, a tablet terminal, a personal computer, or a game machine. In this embodiment, the user terminal 30 will be described as a remote controller. The network 40 may be configured by either a wireless or wired system, or both, and the network 40 may include the Internet. In the present embodiment, the video processing device 10 and the model generation device 50 are connected to the network 40 by a wireless communication method.

The user operates the remote controller (user terminal 30) to select a sports broadcast program for which the highlight scene is to be extracted. That is, in the present embodiment, it is premised that the sports broadcast program for which the highlight scene is to be extracted is recorded in the storage device 11. When the user operates the remote controller to select a sports broadcast program for which the highlight scene is to be extracted, the signal transmitted from the remote controller is output to the control unit 12 via the interface 13. The control unit 12 extracts the highlight scene of the sports broadcast program selected by the user based on the received signal.

The control unit 12 is a general-purpose microcomputer including a CPU (Central Processing Unit), a memory, an input / output unit, and the like. The CPU, memory, input / output unit, and the like are electrically connected via a bus (not shown). A computer program for functioning as the video processing apparatus 10 is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits included in the video processing apparatus 10. Here, an example of realizing a plurality of information processing circuits included in the video processing apparatus 10 by software is shown, but of course, the information processing circuit is configured by preparing dedicated hardware for executing each of the following information processing. It is also possible to do. Further, a plurality of information processing circuits may be configured by individual hardware. The control unit 12 includes a video analysis unit 14 and a highlight extraction unit 17 as a plurality of information processing circuits. Further, the video analysis unit 14 is classified into a competition specific unit 15 and a feature detection unit 16.

When the competition specifying unit 15 receives the signal transmitted from the user terminal 30, the competition specifying unit 15 identifies the type of the sports broadcast program (competition) selected by the user. An example of the specific method will be described. When a sports broadcast program is recorded in the storage device 11, tag information indicating the type of the recorded sports broadcast program may be stored at the same time. In such a case, the competition specifying unit 15 can specify the type of the sports broadcast program by referring to the tag information. Alternatively, the competition specifying unit 15 may analyze the video of the sports broadcast program and specify the type of the sports broadcast program. For example, when a line is drawn on the court or the ground, it is possible to specify basketball if a 3-point line is detected, soccer if a center circle and a penalty box are detected. It can also be specified from the shape of the uniform, the shape of the ball used in the competition, the number of players, and so on. The competition specifying unit 15 outputs the type of the specified sports broadcast program to the feature detection unit 16.

The feature detection unit 16 analyzes a sports broadcast program selected by the user, and detects a characteristic facial expression, a characteristic motion, and a characteristic voice. Well-known image recognition technology is used to detect characteristic facial expressions and characteristic movements. As an example of the image recognition technology, there is template matching using a model (template). Template matching is a technique for detecting scenes with a high degree of similarity by performing matching while changing the position and size of the model. SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference) and the like are used as an index of similarity. Since these are well-known techniques, detailed description thereof will be omitted.

The characteristic facial expression is, for example, the facial expression of the player's joy. Characteristic movements include a player's guts pose, a player raising his hand to respond to the spectator, a high five between the players, a player's applause or a spectator's applause. That is, a characteristic movement is a movement showing joy. Well-known speech recognition techniques are used to detect characteristic speech. An example of a characteristic voice is a specific wording when a player scores, such as "I did it" or "come on". Alternatively, a voice having a volume equal to or higher than a predetermined value is also included in the characteristic voice.

A model of a characteristic facial expression and a characteristic motion is generated by the model generator 50. Although the details will be described later, the model generation device 50 uses deep learning as an example to generate a model of a characteristic facial expression and a characteristic motion.

The feature detection unit 16 analyzes the sports broadcast program by dividing it into predetermined time intervals. For example, when the sports broadcast program is soccer, the match time is 45 minutes in the first half of the game, so the feature detection unit 16 divides the video into 5 minute units and analyzes them by dividing them into 9 parts. The feature detection unit 16 outputs a characteristic facial expression, a characteristic motion, and a characteristic voice detected by using template matching and voice recognition technology to the highlight extraction unit 17. The time of 5 minutes is an example and can be changed as appropriate depending on the performance of the computer and the like.

The highlight extraction unit 17 (extraction unit, generation unit) is a scene including a characteristic facial expression, a characteristic motion, and a characteristic voice detected by the feature detection unit 16 (hereinafter, may be referred to as a reference scene). And the scenes before and after it are extracted as highlight scenes. The scenes before and after are defined as a scene including the first scene that has returned by the first predetermined time and the second scene that has advanced by the second predetermined time with respect to the reference scene. The highlight extraction unit 17 stores the extracted highlight scene as a highlight image in the storage device 11. When a plurality of highlight scenes are extracted, the highlight extraction unit 17 connects the plurality of highlight scenes to generate one highlight image.

When the highlight image is stored in the storage device 11, the image processing device 10 may inform the user that the generation of the highlight image is completed via the interface 13. This allows the user to enjoy the highlight video.

Next, the model generator 50 will be described. As shown in FIG. 1, the model generation device 50 includes a server 51, a control unit 52, a storage device 55, and an interface 56. The installation location of the model generation device 50 is not particularly limited, but for example, the model generation device 50 is installed in a management center owned by a business operator who manages the model generation device 50.

A large number of past sports broadcast programs and sports contents are stored in the server 51. In addition, sports broadcast programs and sports contents to be broadcast in the future are also stored in the server 51. It does not have to be a server as long as it is a structure that functions as a storage medium. Examples of the structure that functions as a storage medium include a cassette tape, a CD-ROM, a DVD-ROM, a BLU-RAY disc, and the like.

The control unit 52 is a general-purpose microcomputer like the control unit 12. The control unit 52 includes a video acquisition unit 53 and a model generation unit 54 as a plurality of information processing circuits.

The video acquisition unit 53 acquires sports broadcast programs and sports contents stored in the past from the server 51. The video acquisition unit 53 outputs the acquired sports broadcast program and sports content to the model generation unit 54.

The model generation unit 54 analyzes the sports broadcast program and sports content acquired from the video acquisition unit 53 to generate a model. As described above, the model referred to here is a model with a characteristic facial expression and a model with a characteristic motion. These models are used in video recognition technology. Well-known AI techniques are used to generate the model. Deep learning is used as an example of AI technology. In deep learning, feature quantities (characteristic facial expressions, characteristic movements) are extracted from a large amount of video data (data related to sports broadcast programs and sports contents) using a neural network. This makes it possible to extract the characteristics of movements unique to each sport, in addition to the features such as facial expressions of joy and guts poses that are common to many sports. The same applies to voice. In deep learning, a voice model is generated by extracting features such as sound intensity, frequency, volume, and specific words of input voice data. When the sports video is input to the neural network, the sports video is classified for each competition, and the classified video is input to the neural network. The images classified by the neural network are learned, and a model of a characteristic facial expression, a model of a characteristic motion, and a model of a characteristic voice are generated for each competition. The model thus generated is stored in the storage device 55. In the following, a model showing a characteristic facial expression generated by the model generation unit 54 may be referred to as a facial expression model, a model showing a characteristic motion may be referred to as an motion model, and a characteristic voice model may be referred to as a speech model.

The model generation device 50 transmits the model to the video processing device 10 via the network 40. The transmitted model is stored in the storage device 11. The model stored in the storage device 11 is called when extracting the highlight scene.

Next, an example of the highlight scene extraction method will be described with reference to FIGS. 2 to 3. In FIGS. 2 to 3, the competition is soccer.

The horizontal axis shown in FIG. 2 is the soccer match time. In FIG. 2, a score is generated 30 minutes after the start of the match, and the player who scored at 30 minutes and 5 seconds makes a guts pose, and the cheers of the spectators are increasing. It is assumed that the user wants to extract a highlight scene in a soccer broadcast program having such a scene. As described above, the user's wishes are operated by the remote controller.

First, the competition specifying unit 15 identifies the type of sports broadcast program (competition) selected by the user when receiving the signal transmitted from the remote controller. Here, it is assumed that soccer is specified. Next, the feature detection unit 16 refers to the storage device 11 to acquire a facial expression model, an motion model, and a voice model related to soccer. The feature detection unit 16 divides the soccer broadcast program into predetermined time intervals and starts analysis. Here, 25 minutes to 35 minutes are divided into one part. The feature detection unit 16 analyzes whether there is a scene that matches or is similar to the facial expression model, the motion model, and the voice model in the video for 25 to 35 minutes. As a result of the analysis, the feature detection unit 16 detects the facial expression of the player's joy, the player's guts pose, and the voice (cheers) having a volume equal to or higher than a predetermined value in the scene of 30 minutes and 5 seconds.

The highlight extraction unit 17 extracts a scene including a scene including a first scene that has returned by a first predetermined time and a second scene that has advanced by a second predetermined time, based on a scene of 30 minutes and 5 seconds. In the example shown in FIG. 2, the first predetermined time is 10 seconds, and the first scene is the scene at 29 minutes 55 seconds. The second predetermined time is 5 seconds, and the second scene is a scene at 30 minutes and 10 seconds. In the example shown in FIG. 2, the highlight scene is 15 seconds.

When a player scores a goal in soccer, the subsequent scenes include a scene in which the scored player is pleased, a scene in which the scored player is in a guts pose, and a scene in which a big cheer is raised. According to the present embodiment, the feature of the scene after the scoring scene is generated is detected, and the scene including the scenes before and after the scene is generated as the highlight scene. This makes it possible to generate a scene including the scoring scene as a highlight scene without extracting the scoring scene itself. Further, since the highlight scene is generated in consideration of the facial expression of the player and the movement of the player in addition to the voice, it is possible to generate an appropriate highlight scene as compared with the conventional technique.

The feature detection unit 16 extracts scenes in which the facial expression model, the motion model, and the voice model match or are similar without any time difference. However, some time difference may be allowed. The allowable time difference is not particularly limited, but may be set in the range of, for example, 1 to 2 seconds.

In the example shown in FIG. 2, the scenes before and after the reference scene are included, but the present invention is not limited to this. For example, as shown in FIG. 3, the highlight scene may be extracted from the reference scene to the first scene that has returned by the first predetermined time (10 seconds). Even in this case, it is possible to extract the scene including the scoring scene as a highlight scene without extracting the scoring scene itself.

Next, an operation example of the video processing apparatus 10 will be described with reference to the flowchart of FIG.

In step S101, when the control unit 12 receives the signal transmitted from the user terminal 30 (YES in step S101), the process proceeds to step S103. When the signal is not transmitted from the user terminal 30 (NO in step S101), the process waits.

In step S103, the competition specifying unit 15 acquires the sports broadcast program selected by the user from the storage device 11 and specifies the type of the sports broadcast program. Here, it is assumed that the type of sports broadcast program is specified as soccer. The process proceeds to step S105, and the feature detection unit 16 refers to the storage device 11 to acquire a facial expression model, an motion model, and a voice model related to soccer.

The process proceeds to step S107, and the feature detection unit 16 divides the soccer broadcast program into predetermined time intervals and starts analysis. The feature detection unit 16 analyzes whether there is a scene that matches or is similar to the facial expression model, the motion model, and the voice model by using a well-known video recognition technique and voice recognition technique in each of the separated parts.

The process proceeds to step S109, and when a scene that matches or is similar to the facial expression model, motion model, and voice model is detected, that scene is extracted as a reference scene. The highlight extraction unit 17 extracts a scene including the first scene that has returned by the first predetermined time to the second scene that has advanced by the second predetermined time, with the reference scene as a reference. This extracts the highlight scene. The process proceeds to step S111, and the highlight extraction unit 17 stores the extracted highlight scene as a highlight image in the storage device 11.

(Action effect)
As described above, according to the video processing apparatus 10 according to the present embodiment, the following effects can be obtained.

The image processing device 10 extracts a reference scene containing a characteristic motion and a characteristic sound from a sports image, and returns only a first predetermined time based on the extracted reference scene for a second predetermined time. A scene including the advanced second scene is generated as a highlight scene. According to the present embodiment, the feature of the scene after the scoring scene is generated is detected, and the scene including the scenes before and after the scene is generated as the highlight scene. This makes it possible to generate a scene including the scoring scene as a highlight scene without extracting the scoring scene itself. Further, since the highlight scene is generated in consideration of the facial expression of the player and the movement of the player in addition to the voice, it is possible to generate an appropriate highlight scene as compared with the conventional technique. In the above-described embodiment, both a characteristic motion and a characteristic voice are used, but the present invention is not necessarily limited to this. The image processing device 10 may extract a reference scene including either a characteristic motion or a characteristic sound from a sports image and generate a highlight scene.

Each of the functions described in the above embodiments may be implemented by one or more processing circuits. The processing circuit includes a programmed processing device such as a processing device including an electric circuit. Processing circuits also include devices such as application specific integrated circuits (ASICs) and circuit components arranged to perform the described functions.

As mentioned above, embodiments of the invention have been described, but the statements and drawings that form part of this disclosure should not be understood to limit the invention. This disclosure will reveal to those skilled in the art various alternative embodiments, examples and operational techniques.

As another example of the scene after the scoring scene occurs, in the case of basketball, the scene where the goal was broken by a dunk, in the case of tennis, the scene where the ball boy picked up the ball (in the case of double fault, the score was scored). In the case of volleyball, the scene where the players form a circle is applicable.

The highlight scene is not limited to the scoring scene. In soccer, offside scenes, scenes where the referee issues a yellow card or red card, and scenes where one player continues to dribble for a predetermined period of time can be highlight scenes. In such a scene where the score is not involved, the facial expression of joy of the player is unlikely to occur, so there are two models to be generated, the motion model and the voice model.

The sports targeted for highlight scene extraction may be limited. For example, only sports in which one's own team and the other's team compete for points may be targeted for highlight scene extraction. Sports that compete for points on their own and on the enemy include soccer, basketball, American football, ice hockey, tennis, badminton, volleyball, table tennis, rugby, and futsal. In these sports as well, as in soccer described above, according to the present embodiment, it is possible to generate a scene including a scoring scene as a highlight scene without extracting the scoring scene itself.

Of course, the highlight scene is not limited to the scoring scene even in the sports where the score is competed between the own team and the enemy team. In the case of ice hockey, the scene where the acrylic board in front of the audience seats breaks can be a highlight scene. For tennis, badminton, and volleyball, the challenge scene can be a highlight scene. Highlight scenes can be generated by extracting these scenes and the scenes before and after them. The challenge scene means a scene in which a so-called "challenge" is performed, in which a reconfirmation of the judgment is requested by a video image or the like when the judgment is doubtful.

It can also be a highlight scene just before the end of the match.

The model generation device 50 may generate a model of a characteristic facial expression, a characteristic motion, and a characteristic sound by using only the image when the game is interrupted in the sports broadcast program. For example, in the case of soccer, the video when the match is interrupted is the video from the score being scored until the referee rings the whistle at the start of the match, and the referee rings the whistle due to a foul or the like, and then the referee plays the match again. It is a video until the whistle of the start is sounded. That is, the time when the game of the competition is interrupted may be defined as the period from the signal of the interruption of the game to the time when the signal of the start of the game is given again. The brawl scene is a scene when the game is interrupted in all sports.

In addition to detecting the degree of matching with the facial expression model, motion model, and audio model during video analysis, the player who appears in the center of the angle of view determines whether or not the player appears in the center of the angle of view. You may consider whether the players around you are wearing the same uniform or on the ground.

In the above embodiment, the highlight scene of the sports broadcast program selected by the user is extracted, but the present invention is not limited to this. Regardless of the user's instruction, if the sports broadcast program is stored in the storage device 11, the control unit 12 may automatically extract the highlight scene of the stored sports broadcast program.

10 Video processing equipment 11, 55 Storage equipment 12, 52 Control unit 13, 56 Interface 14 Video analysis unit 15 Competition identification unit 16 Feature detection unit 17 Highlight extraction unit 20 Video acquisition device 30 User terminal 40 Network 50 Model generation device 51 Server 53 Video acquisition unit 54 Model generation unit

Claims (6)

An extractor that extracts a reference scene containing characteristic movements or characteristic sounds from sports video,
With the reference scene extracted by the extraction unit as a reference, a generation unit that generates a scene including a scene including a first scene that has returned by a first predetermined time and a second scene that has advanced by a second predetermined time as a highlight scene. A video processing device characterized by being equipped with. The characteristic motion or the characteristic sound is generated in advance as a model for each competition related to the sports image and stored in a storage device, and the generated model interrupts the game of the competition. It ’s a model of time,
The video processing device according to claim 1, wherein the extraction unit extracts the reference scene using the model stored in the storage device. The characteristic movement is a joyful movement when the game of the competition is interrupted, or a competition-specific movement, and the characteristic voice is a voice containing a specific wording or a volume equal to or higher than a predetermined volume. The video processing apparatus according to claim 2, wherein the audio is provided. The video processing device according to claim 3, wherein the time when the game of the competition is interrupted means a period from the signal of the interruption of the game to the time when the signal of the start of the game is given again. Extract reference scenes containing characteristic movements or characteristic sounds from sports images,
A video processing method characterized by generating a scene including a scene including a first scene that has returned by a first predetermined time and a second scene that has advanced by a second predetermined time as a highlight scene with the extracted reference scene as a reference. .. It is a model generator that generates a model for extracting highlight scenes from sports images.
The sports video is classified for each competition, and the classified video is input to a neural network to be learned, and a model generation unit for generating a characteristic motion model or a characteristic audio model for each competition is provided. Model generator.

Images