JP4577774B2 - Sports video classification device and log generation device - Google Patents

Description

  The present invention relates to a sports video classification device and a log generation device provided with a broadcast medium and the like, and in particular, can classify sports video into various shot types by analyzing image characteristics, and further identify important events. Sports video classification device that can detect in detail, sports video log that can perform real-time classification of sports video shot types and detailed event detection, and can generate information on the progress of the game situation as a log The present invention relates to a generation device.

  A system for providing various information such as a game progress of a sports game and a specific scene has been proposed. For example, Patent Document 1 discloses a content description data creation system capable of creating content description data related to sports matches and providing it to a portable information terminal such as a mobile phone or a PDA. In this content description data creation system, the names of the players making up each team are displayed, and event buttons are provided for selecting various events of sports matches for each player name, and when the event button is selected, Content description data including the event and player name corresponding to the selected event button and the time on the sport game when the event button is selected is automatically generated.

  Also, in Patent Document 2, video data of sports matches, commentary image data relating to the game situation, and commentary document data relating to the game situation are transmitted from the streaming distribution server, and these data are synchronously displayed on the client side system. A sports video distribution system is disclosed.

Further, the applicant classifies sports video scenes using motion information and color information of moving image data of sports video, and audio data attached to the moving image data, and efficiently searches and browses the sports video. Patent Document 3 previously proposed a sports video classification device that made it possible to provide
JP 2003-305150 A Japanese Patent Laid-Open No. 2003-199060 Japanese Patent Application No. 2004-62867 (prior application)

  In the content description data creation system disclosed in Patent Document 1, the names of players constituting each team are displayed, and various events of a sport game are selected by event buttons for each player name, and events and players are selected according to the selection. Since the content description data including the name and the time on the sport match is generated, the creation of the content description data requires the intervention of a manual process of selecting an event button.

  In addition, the sports video distribution system of Patent Document 2 is characterized by distributing commentary image data and commentary document data together with video data of a sports game, and displaying these data synchronously on the client side system. It does not disclose how to generate commentary image data or commentary document data.

  In addition, the sports video classification device disclosed in Patent Document 3 can perform close-up of players and classification of attack scenes using motion information and color information of moving image data, and audio data accompanying the moving image data. Therefore, it is possible to provide efficient search and browsing of sports videos, but by classifying scenes in more detail, more efficient search and browsing can be expected, and logs that contribute to it can be generated be able to.

  The object of the present invention is to solve the above-mentioned problems, classify sports scenes with shot switching provided in broadcast media in detail, and generate game progress information as a log in real time. An object of the present invention is to provide a sports video classification device and a log generation device.

In order to solve the above-described problems, the present invention provides a sports video classification apparatus that involves shot switching, a shot division point detecting means for detecting a shot division point of moving image data of sports video, Shots for classifying shots divided at the shot division points using at least in-screen feature quantity extraction means for extracting color information and edge information, and color information and edge information extracted by the in-screen feature quantity extraction means The shot classification means includes a close-up shot determination means for determining whether or not the shot includes a close-up of a player in the screen, and a long shot determination means for determining whether or not the shot is a long shot. The close-up shot determining means divides the lower half area in the screen in an arbitrary number of ways in the vertical direction, When the number of pixels of the color information corresponding to the color of the stadium in the area is below a certain level, it is determined that the shot includes a close-up of the player in the screen, and the long shot determination means includes the color information and the edge information. A determination means for separately determining a shot with a spectator seat in the screen, a shot with a spectator seat near a corner of a stadium, and a shot with no spectator seat using at least one of the screens, , The determination result of the close-up shot determination means is negative, the edge of the horizontal component at the top of the screen has a certain value or more, and the variance of the sum of the edges of the oblique component in the region obtained by dividing the entire screen in the vertical direction is If the number of pixels corresponding to the stadium at the bottom of the screen has a certain value, the long seat with a spectator seat in the screen It is determined that the shot is not a long shot with a spectator seat in the screen, the edge of the horizontal component at the top of the screen has a certain value, and the sum of the edges of the diagonal component at the top right of the screen and the screen The difference from the sum of the edges of the diagonal component in the upper left part has a certain value or more, the variance of the edge of the oblique component in the horizontal center of the screen has a certain value or less, and the entire screen is a playing field. If the corresponding number of pixels has a certain value or more, it is judged as a long shot with a spectator seat near the corner in the screen, and not a long shot with a spectator seat near the corner in the screen. , The number of pixels corresponding to the color of the stadium on the entire screen has a certain value or more, and there is an edge distribution and an average value of the diagonal component in the center in the horizontal direction of the screen. When it has a value below a certain value, it is characterized in that it is determined as a long shot in which no spectator seat exists in the screen.

Further, in the present invention, when the close-up shot determining means determines that the shot includes a player's close-up on the screen, the region where the player is present is determined, and the color information of the region is used to determine the player's uniform. A uniform color determining means for determining the color is further provided .

Further, in the present invention, when the close-up shot determination unit determines that the shot includes a player's close-up in the screen, the face region is used to determine a region where the player exists and to determine a face region included in the region. and determining means is characterized by comprising a face recognition identification means for identifying and face recognition in determined face regions in the face area determination means.

The present invention is further characterized by further comprising event detection means for detecting an event using the transition information of the shots classified by the shot classification means and the audio information attached to the sports video .

Further, the present invention is characterized in that the sports video is a soccer video, and the event detection means detects a goal, a shot, a free kick, and a corner kick event .

Further, the present invention provides a sports video log generation apparatus that involves shot switching, a moving image data capturing unit that captures the moving image data of the sports video, and a moving image data captured by the moving image data capturing unit. A detection unit that executes shot classification and event detection in real time using a sports video classification device that includes the event detection unit, and records a result of shot classification and event detection by the detection unit as game progress information. It is characterized by having a recording means .

The present invention is further characterized by further comprising sending means for sending the game progress information recorded by the recording means to the server .

Further, the present invention is characterized in that the recording means describes information on the progress of the game situation in metadata .

Further, the present invention is characterized by comprising means for linking the game progress information recorded by the recording means to an actual sports video .

  According to the present invention, it is possible to perform shot classification and event detection in detail by using color information, audio information, and edge information for a sports video accompanied by shot switching provided on a broadcast medium or the like. A fine index can be easily assigned to the sports video.

  In addition, even when sports video data is encoded by an encoding method such as MPEG, color information, edge information, and audio information can be easily extracted from the encoded video data. Shot classification and event detection can be realized with low processing cost.

  Furthermore, by using shot classification and event detection with real-time sports video as an input, it is possible to automatically generate a log as game progress information, and send the generated log to a server for character-based game progress It is possible to facilitate the search and management of sports videos by providing notifications or outputting them as metadata.

  The present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of a sports video classification device according to the present invention. In the following description, sports video data encoded by the MPEG encoding method is assumed. However, the present invention is not limited to this, and the same encoding method or non-encoded uncompressed video is not limited thereto. The same applies to data. In addition, as a type of sports video, the following description will be made mainly on soccer video, but the present invention can also be applied to other sports video.

  First, moving image data of sports video (hereinafter simply referred to as video data) is input to the shot division point detection unit 12 via the video data input unit 11. The shot division point detection unit 12 detects a shot division point of the input video data, and divides each shot by the detected division point. For the shot division, for example, a technique described in JP-A-10-224741 can be used.

  The video data divided for each shot by the shot division point detection unit 12 is input to the image feature extraction unit 13 and the audio feature extraction unit 14. The image feature extraction unit 13 extracts at least color information and edge information in each shot. The audio feature extraction unit 14 extracts a cheering sound level associated with a sports video as an audio feature.

  The image feature extraction unit 13 can obtain the color information and edge information in each shot by using the beginning or intermediate screen of each shot, the representative screen of each shot, and the like as an extraction target. It is also possible to obtain color information and edge information in each shot from an average value or an intermediate value of information extracted from the extraction target. As the edge information here, an edge histogram in an arbitrary number of directions existing in the screen can be used, and an edge histogram in the entire screen or an edge histogram in a small region obtained by dividing the screen into an arbitrary number is used. be able to.

  In addition, when the sports image to be input is encoded by a method using discrete cosine transform such as MPEG, a DC component constituted by a DC component of an encoding coefficient obtained by variable length decoding and simple motion compensation Edge information such as an edge histogram can be extracted for an image (hereinafter referred to as a DC image). As the edge histogram, an edge histogram descriptor defined in MPEG-7 or the like can be used. The MPEG-7 edge histogram descriptor divides the screen into 4 × 4 sub-regions, and includes five types of horizontal component, vertical component, 45-degree component, 135-degree component, and no edge directionality in each sub-region. Consists of histograms for edge direction. FIG. 2 shows edge components of the edge histogram descriptor.

  When the color information is extracted by the image feature extraction unit 13, a reduced image obtained by reducing the size of an input image can be selected as an extraction target. For example, in consideration of the trade-off between color information extraction accuracy and calculation amount, the original sports video screen can be reduced to 8 × 8.

  From this degenerated image, color information corresponding to the stadium can be determined by a method similar to the method described in Patent Document 3. For example, the most frequently appearing color (most frequent color) is determined from the total of the color histograms of a certain section, and the most frequent color is set as the color of the turf of the stadium. In a sport video, since a screen in which the turf color is dominant frequently appears in the stadium, the most frequent color can be determined as the turf color of the stadium.

  The shot type determination unit 15 includes a close-up shot determination unit 15-1 and a long shot determination unit 15-2. The close-up shot determining unit 15-1 determines whether or not a player's close-up is included in the screen, and the long shot determining unit 15-2 is a long shot in which the screen is taken of a player or a stadium from a long distance. It is determined whether or not.

  In addition, the shot type determination unit 15 determines the shot that the close-up shot determination unit 15-1 determines that the player's close-up is not included in the screen and the long shot determination unit 15-2 determines that the shot is not a long shot. The player and the stadium are judged as medium shots taken from a medium distance.

  The long shot determination unit 15-2 separately determines three types of long shots: a shot with a spectator seat in the screen, a shot with a spectator seat near the corner of the stadium, and a shot with no spectator seat. .

  FIG. 3 is a flowchart showing processing in the close-up shot determination unit 15-1. When a close-up shot is determined using motion vector information in the screen as described in Patent Document 3, the close-up shot may be erroneously determined as a medium shot. Therefore, in the present invention, the close-up shot determination unit 15-1 uses color information of the stadium existing in the screen as an image feature. For example, in a soccer video, the color of the stadium corresponds to the color of the turf.

  Specifically, first, the color information of the shot extracted by the image feature amount extraction unit 13 is input to the close-up shot determination unit 15-1 (S31). Next, it is determined whether or not the number of turf color pixels in the specified area is smaller than the threshold THCL (S32). It is determined that the close-up shot includes the close-up of the player (S33). Otherwise, it is determined not to be a close-up shot.

  For example, if the lower half area in the screen is divided into an arbitrary number of areas in the horizontal direction and the number of turf color pixels in those areas is smaller than the threshold THCL, the player close-up in the screen Is determined to be a close-up shot. The area may be divided into three methods such as the left side, the center, and the right side, and the divided areas may partially overlap.

  A player area processing unit is provided after the close-up shot determination unit 15-1, so that the uniform color (team) and face (name) of the close-up player are identified from the shot determined to be the close-up shot. be able to. Shots that are not determined to be close-up shots are sent to the long shot determination processing unit 15-2. Further, the close-up determination result (S33) is sent to the log generation unit 17 and used for log generation.

  FIG. 4 is a flowchart showing processing in the long shot determination unit 15-2. Whether or not the shot is a long shot is determined for a shot that has not been determined to be close-up by the close-up determination unit 15-1. In the determination of a long shot, three types of long shots are separately determined: a shot with a spectator seat in the screen, a shot with a spectator seat near the corner of the stadium, and a shot with no spectator seat.

Since the following characteristics can be seen in a shot with a spectator seat, whether or not the shot has a spectator seat can be determined using edge information and color information corresponding to the color of the stadium.
1. Due to the influence of many spectators and advertising fences around the stadium, the edge of the horizontal component mainly becomes strong at the top of the screen.
2. Since a large subject is not included on the screen, the same edge characteristics can be seen in all areas obtained by dividing the screen into horizontal components.
3. Since the lower part of the screen does not include spectator seats or large subjects, the color of the turf, which is the color of the stadium, becomes the occupied color.

  Specifically, the edge histogram in the small area belonging to the upper part of the screen (or the sum if it is composed of multiple areas) is calculated, and the horizontal component sum EH_H is greater than a certain threshold THL11, and the entire screen is From the threshold THL13 where the variance σEH_V of the sum of the edges of the diagonal component (45 degree component or 135 degree component) in the divided area is smaller than a threshold THL12 and the number of pixels GC_B corresponding to the color of the stadium at the bottom of the screen If it is also larger (S41), it can be determined that the long shot has a spectator seat (S42).

  Here, the division method and the number of divisions for extracting the edge can be arbitrarily set. For example, when the MPEG-7 edge histogram descriptor is used, the horizontal direction is set as “a small region belonging to the upper part of the screen”. Edges in each of the four blocks in the vertical column can be used as the uppermost four blocks, “regions obtained by dividing the entire screen in the vertical direction”. Further, when extracting color information from an 8 × 8 reduced image, THL13 can be set to 30 (pixels) or the like in 32 pixels in the lower half of the screen.

Next, when it is determined whether or not there is a spectator seat, it is determined whether or not there is a spectator seat near the corner of the stadium. The following characteristics can be seen in shots with spectator seats near the corners. Therefore, whether or not there is a spectator seat near the corners is determined using edge information and color information corresponding to the color of the stadium. be able to.
1. Due to the influence of many spectators and advertising fences around the stadium, the edge of the horizontal component mainly becomes strong at the top of the screen.
2. Due to the influence of the advertising fence, goal line, etc., the edge of the diagonal component at the upper right corner of the screen near the corner on the right side of the screen and the edge of the diagonal component at the upper left corner of the screen near the corner on the left side of the screen are strong.
3. Since the large object is not included in the portion below the center in the horizontal direction of the screen, the edge component does not change greatly.
4). Since the entire screen does not include spectator seats or large subjects, the color of the turf, which is the color of the stadium, becomes the occupied color.

  Specifically, an edge histogram (or the sum of multiple regions) of the small area belonging to the upper part of the screen is obtained, and the sum EH_H of the horizontal components is larger than a threshold THL21, and the upper left part of the screen and the upper right part of the screen. Find the edge histogram in the small area belonging to the part (the sum if each consists of a plurality of areas), the absolute value difference ΔEH_S of the diagonal component (45 degree component or 135 degree component) is greater than a threshold THL22, From the threshold THL24 where the variance σEH_H of the edge of the diagonal component (45 degree component or 135 degree component) in the horizontal center of the screen is smaller than the threshold THL23 and the number of pixels GC_A corresponding to the color of the stadium on the entire screen If it is larger (S43), it can be determined that there is a long shot with a spectator seat near the corner (S44).

  The division method and the number of divisions for extracting the edge can be arbitrarily set. For example, when the MPEG-7 edge histogram descriptor is used, the “small area belonging to the upper part of the screen” is set as the topmost portion in the horizontal direction. 4 blocks, 4 blocks each in the upper left 2 × 2 and 2 × 2 in the upper right as the “upper left part of the screen” and “upper right part of the screen”. Edges in a total of 8 blocks of 4 blocks can be used. Further, when extracting color information from an 8 × 8 reduced image, THL24 can be set to 16 (pixels) or the like in 64 pixels of the entire screen.

Further, if it is determined whether or not there is a spectator seat near the corner of the stadium, it is determined whether or not the shot has no spectator seat. Since the following characteristics are seen in the shot without the audience seat, whether or not the shot has the audience seat can be determined using edge information and color information corresponding to the color of the stadium.
1. Since only the stadium is being photographed, the turf color that is the color of the stadium is the occupied color for the entire screen.
2. Since a large subject is not included in the central portion in the horizontal direction of the screen, no significant variation is observed in the edge component.

  Specifically, the number of pixels GC_A corresponding to the color of the stadium on the entire screen is larger than a threshold THL31, and the edge distribution σEH_H of the diagonal component (45-degree component or 135-degree component) in the center in the horizontal direction of the screen When the average value avgEH_H is smaller than certain threshold values THL32 and THL33 (S45), it can be determined that the long shot has no spectator seat (S46).

  The division method and the number of divisions for extracting the edge can be arbitrarily set. For example, when the MPEG-7 edge histogram descriptor is used, the second horizontal row as the “horizontal center portion of the screen” is used. Edges in a total of 8 blocks of 4 blocks and 4 blocks in the 3rd row can be used. Also, when extracting color information from an 8 × 8 reduced image, THL31 can be set to 32 (pixels) or the like in 64 pixels of the entire screen.

  Finally, if the determination whether or not there is no spectator seat (S45) is negative, that is, the close-up shot determination unit 15-1 determines that the player's close-up is not included in the screen, and the long shot If the determination unit 15-2 determines that the shot is not a long shot, the shot is determined to be a medium shot (S47). The determination result (S42, S44, S46, S47) in the long shot determination process is sent to the log generation unit 17 and used for log generation.

  A player area processing unit is provided downstream of the close-up shot determination unit 15-1, so that the uniform color (team name) and face (name) of the close-up player are identified from the shot determined to be the close-up shot. FIG. 5 shows a processing flow in the case of doing so.

  The player area processing unit includes a uniform color determination unit 51 and a player identification unit 52. In the uniform color determination unit 51, first, an area (hereinafter, referred to as a player area) in which a player is considered to be present is identified using edge information for the shot determined to be close-up by the close-up shot determination unit 15-1. (S51). Next, the color information in the player area is evaluated (S52). As color information in the player area, a color histogram, an average color, or a distance from a uniform color given in advance can be used. If the information on the team to be played and the uniform color are given in advance, uniform color matching processing can be performed (S53) to identify the team of the players who are close-up (S54).

  The player identification unit 52 further extracts a player's face area (hereinafter referred to as a player face area) from the player area (S55), and performs face recognition processing on the extracted player face area, thereby obtaining the close-up shot. The player's face included is identified (S56). For the player face area extraction, skin color detection or template matching with a face image given separately can be applied. As face recognition processing, matching at the image level with a player's face image database prepared in advance, matching using face-specific features, or the like can be applied.

  As a result, it is possible to determine the name of the player who is closed up (S57). The determination results (S54, S57) of the teams and player names of the players who are close-up are sent to the log generation unit 17 and used for log generation.

  An attack direction determination unit can be provided downstream of the long shot determination unit 15-2, and the attack direction can be determined from the shot that has been determined to be a long shot by the long shot determination unit 15-2.

  For the determination of the attack direction, a method using a motion vector disclosed in Patent Document 3 can be used. That is, since it is considered that the camera spans in the attacking direction, the attacking direction can be determined using the motion vector. For example, it is defined by the dominant direction attribute (Dominant Direction) of the activity descriptor defined in MPEG-7 and (the number of motion vectors in the screen matching the most frequent direction) / (total number of motion vectors in the screen) The attack direction can be determined using the uniformity of the movement direction as an evaluation index. By determining the attack direction, it is possible to acquire, as log information, information on the progress of the game, such as the attacking team in a long shot and the ball dominance rate in soccer.

  Returning to FIG. 1, the event type detection unit 16 can be provided after the shot type determination unit 15 to detect various events in the sports video. The event type detection unit 16 detects what event type a shot classified into a close-up shot, a medium shot, and a long shot belongs to using shot transition information and further audio information.

  When the sports video is, for example, a soccer video, the event type detection unit 16 includes a goal event determination unit 16-1, a free kick event determination unit 16-2, a corner kick event determination unit 16-3, and the like. Detects shots, free kicks, corner kicks, etc.

  A method for detecting a goal or a shot, a free kick, and a corner kick is disclosed in Patent Document 3. In the method disclosed here, if the subband energy sum indicating the cheering characteristic is larger than the first threshold value, it is regarded as a shoot or goal shot, and the subband energy sum indicating the whistle characteristic is larger than the second threshold value. In this case, it is regarded as a free kick shot, and when the difference in motion activity between two adjacent shots is larger than the third threshold, it is regarded as a corner kick shot.

  This method cannot distinguish between a goal and a shot, but can distinguish between a goal and a shot by using the goal event determination unit 16-1 as follows.

FIG. 6 is a flowchart showing processing in the goal event determination unit 16-1. First, the cheering level APL by the audience is evaluated in the audio feature extracting unit 14 in the previous stage (S61), and a shot whose cheering level APL is larger than a certain threshold THG1 is sent to the goal event determining unit 16-1. After the input shot, the goal event determination unit 16-1 determines a series of shot groups as a goal event when any of the following conditions is satisfied (S64), and when both conditions are not satisfied, the shot event (S65).
1. A shot in which the cheer duration DA is greater than a certain threshold THG2 (for example, 8 seconds) or a cheer level APL is greater than the threshold THG1 continues for more than SG1 (for example, 3) shots (S62).
2. After a shot with a cheer level APL larger than a certain threshold THG1, a close-up or medium shot continues for SG2 (eg, 3) shots or more (S63).

  Further, in the method disclosed in Patent Document 3, there are many erroneous corner kick detections, and the side on which the corner kick is performed cannot be determined, but the corner kick event determination unit 16-3 is as follows. Thus, the side where the corner kick is performed can be determined.

  FIG. 7 is a flowchart showing processing in the corner kick event determination unit 16-3. First, the long shot determination unit 15-2 at the preceding stage sends a shot determined as a long shot with a spectator seat near the corner to the corner kick event determination unit 16-3 (S71). The corner kick event determination unit 16-3 stores data for NC (for example, 3) shots before a long shot in which a spectator seat near the corner exists, and when the shot type is close-up or medium shot (S72) ), A series of shot groups is determined as a corner kick event (S73).

  Further, the corner side determination unit 71 determines whether the side on which the corner kick is performed is right or left using the color information in the screen. The corner side determination unit 71 compares the number of grass colors GC_CL on the left side of the screen with the number of grass colors GC_CR on the right side of the screen (S74). If GC_CL> GC_CR, the corner kick is on the right side. It is determined that the corner kick is performed (S75). If GC_CL <GC_CR, it is determined that the corner kick is performed on the left side (S76).

  The determination results by the goal event determination unit 16-1, the free kick event determination unit 16-2, the corner kick event determination unit 16-3, and the like are sent to the log generation unit 17 and used for log generation.

  Next, sports video log generation will be described. FIG. 8 is a functional block lock diagram showing an embodiment of a sports video log generating device according to the present invention. In FIG. 8, the same or equivalent parts as in FIG.

As in FIG. 1, sports video data sent to a broadcasting medium or the like is input to the video data input unit 11, and the sports video is defined in advance by the shot type determination unit 15 and the event type determination unit 16 in the subsequent stage. Into different shot types and event types. The classified shot type and event type are recorded as match progress information (hereinafter referred to as log information) together with corresponding time information in the log generation unit 17. The recording format of the log information can be arbitrarily determined. For example, it can be in the following format. However, it can be omitted if there is no corresponding information.

<Elapsed time>, <Shot type>, <Event type>, <Team identifier>, <Player identifier>

  Here, the elapsed time indicates the elapsed time from the start of the game. The shot type and the event type are obtained from the shot type determination unit 15 and the event type determination unit 16, respectively. The team identifier and the player identifier are obtained from the uniform color determination unit 51 and the player identification unit 52, respectively.

  It is also possible to provide the log generation unit 81 with the log generation device and sequentially transmit the log information to, for example, a server on the Internet and distribute it via the Internet. On the client side, the received log information can be shaped into an appropriate format, and the game progress can be viewed using log information display software or the like.

  In addition, log information can be described as metadata. The description format of the metadata can be arbitrarily determined. For example, MPEG-7 defined by ISO / IEC or SportsML (Sports Markup Language) defined by IPTC (The International Press Telecommunications Council) can be used. However, it can be omitted when the corresponding information cannot be described as metadata.

  Furthermore, it is possible to provide means for linking the shot type and event type recorded as log information with the location of the corresponding section of the actual video data corresponding to each. Thereby, for example, in the log information display software, by clicking an entry of the event type that occurred at a certain time, the actual video data can be acquired from the server and reproduced. The location format can be arbitrarily determined. When location resolution is required, the location resolution server is inquired to obtain the actual location and section information of the video data. FIG. 9 is a flowchart showing how video data is browsed from log information.

1 is a block diagram showing an embodiment of a sports video classification device according to the present invention. It is explanatory drawing of the edge component of an edge histogram descriptor. It is a flowchart which shows the process in a close-up shot determination part. It is a flowchart which shows the process in a long shot determination part. It is a flowchart which shows the process in a player area | region process part. It is a flowchart which shows the process in a goal event determination part. It is a flowchart which shows the process in a corner kick event determination part. It is a functional part lock figure showing one embodiment of the log generation device of the sports picture concerning the present invention. It is a flowchart which shows the mode of the video data browsing from log information.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Video data input part, 12 ... Shot division | segmentation point detection part, 13 ... Image feature extraction part, 14 ... Audio feature extraction part, 15 ... Shot classification determination part, 15-1. ..Close-up shot determination unit, 15-2 ... long shot determination unit, 16 ... event type determination unit, 16-1 ... goal event determination unit, 16-2 ... free kick event determination unit 16-3 ... Corner kick event determination unit, 17 ... Log generation unit, 51 ... Uniform color determination unit, 52 ... Player identification unit, 71 ... Corner side determination unit, 81 ...・ Log sending part

Claims (20)

In a sports video classification device with shot switching,
Shot division point detecting means for detecting shot division points of moving image data of sports video;
In-screen feature amount extracting means for extracting at least color information and edge information in the screen of the moving image data;
Using the color information and edge information extracted by the in-screen feature amount extraction means, and comprising shot classification means for classifying the shots divided at the shot division points ;
The shot classification means includes a close-up shot determination means for determining whether or not the shot includes a close-up of the player in the screen, and a long shot determination means for determining whether or not the shot is a long shot,
The close-up shot determining means divides the lower half area in the screen in an arbitrary number in the vertical direction, and when the number of pixels of color information corresponding to the color of the stadium in each area is less than a certain value, It is determined that the shot contains the player's close-up on the screen,
The long shot determination means uses at least one of color information and edge information to separately divide a shot with a spectator seat on the screen, a shot with a spectator seat near the corner of the stadium, and a shot with no spectator seat. Including determination means for determining
The determination means is the determination result of the close-up shot determination means is negative, the edge of the horizontal component at the upper part of the screen has a certain value or more, and the edge of the oblique component in the region obtained by dividing the entire screen in the vertical direction If the variance of the sum has a certain value or less and the number of pixels corresponding to the stadium at the bottom of the screen has a certain value or more, it is judged as a long shot with a spectator seat in the screen and The horizontal component edge at the top of the screen has a certain value or more, and the sum of the diagonal component edge at the top right corner of the screen and the diagonal component edge at the top left corner of the screen The difference between the sum and the sum of the values has a certain value or more, the dispersion of the edge of the diagonal component in the horizontal center of the screen has a certain value or less, and the whole screen If the number of pixels corresponding to the stadium has a certain value or more, it is judged as a long shot with a spectator seat near the corner in the screen, and not a long shot with a spectator seat near the corner in the screen. And the number of pixels corresponding to the color of the stadium on the entire screen has a certain value or more, and the variance and average value of the edge of the diagonal component in the horizontal center of the screen each have a certain value or less. In this case, the sports video classification device determines that the long shot has no spectator seat in the screen .   The sports video classification apparatus according to claim 1, wherein the edge information is an edge histogram in an arbitrary number of directions existing in a screen, and is calculated for each small region into which the screen is divided.   The moving image data is data that has been compression-encoded using discrete cosine transform, and the in-screen feature amount extraction unit extracts edge information only from a DC component of the moving image data. The sports video classification device according to 1.   2. The sports video classification apparatus according to claim 1, wherein the in-screen feature amount extraction unit extracts color information from a reduced screen having an arbitrary size. The shot classification unit determines that the close-up shot determination unit determines that the shot does not include a close-up of the player in the screen, and determines that the shot determined by the long shot determination unit is not a long shot is a medium shot. The sports video classification device according to claim 1 , wherein: The sports video according to claim 1 , wherein the close-up shot determining means determines whether or not the shot includes a close-up of the player in the screen using color information corresponding to the color of the stadium. Sorter. When the close-up shot determining means determines that the shot contains a player's close-up on the screen, the uniform color is used to determine the area where the player exists and to determine the color of the player's uniform using the color information of the area. The sports video classification device according to claim 1 , further comprising a determination unit. The sports video classification device according to claim 7 , further comprising a team determination unit that determines a team to which the player belongs using a result obtained by the uniform color determination unit. When the close-up shot determining means determines that the shot includes a close-up of the player in the screen, the face area determining means for determining an area where the player exists and determining a face area included in the area, and the face The sports video classification device according to claim 1 , further comprising face recognition identifying means for recognizing and identifying a face in the face area determined by the area determining means. 2. The sports video classification device according to claim 1 , further comprising event detection means for detecting an event using shot transition information classified by the shot classification means and audio information accompanying the sports video. The sports video classification apparatus according to claim 1 , wherein the sports video is a soccer video, and a turf color is used as color information of the stadium. 11. The sports video classification apparatus according to claim 10 , wherein the sports video is a soccer video, and the event detection means detects a goal, a shoot, a free kick, and a corner kick event. The event detection means, when the cheer level by the audience is greater than a certain value and the duration of the cheer thereafter is greater than a certain time, or a shot where the cheer level is greater than a predetermined level continues for a certain shot or more, 13. The sports video classification device according to claim 12 , wherein a goal event is detected when a close-up or medium shot continues for a certain shot or more after a shot with a cheer level greater than a certain value. The event detection means determines a close-up shot or a medium shot in which the shot classification means continues more than a certain number before the long shot determined by the long shot determination means that there is a spectator seat near the corner of the stadium. 13. The sports video classification device according to claim 12 , wherein a corner kick event is detected. 15. The sports video classification device according to claim 14 , wherein the event detection unit further determines a side where a corner kick is performed using color information in a screen. The event detection means compares the number of turf color pixels on the left and right sides of the screen, and when the number of pixels on the left side of the screen is large, corner kick from the right side and the number of pixels on the right side of the screen are large. The sports video classification device according to claim 15 , wherein a corner kick from the left side is determined. In a sports video log generator with shot switching,
Moving image data capturing means for capturing the moving image data of the sports video;
Detection means for executing shot classification and event detection of the moving image data captured by the moving image data capturing means in real time using the sports video classification device according to claim 10 ,
A sports video log generation device comprising recording means for recording the result of shot classification and event detection by the detection means as information on the progress of a game situation. 18. The sports video log generating device according to claim 17 , further comprising sending means for sending game progress information recorded by the recording means to a server. 18. The sports video log generation device according to claim 17 , wherein the recording means describes game progress information in metadata. 18. The sports video log generating device according to claim 17 , further comprising means for linking information on the progress of the game recorded by the recording means to an actual sports video.

Images