JP2005115529A - Video classification display method, its system, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To classify a video by a motion, and to display it in a list. <P>SOLUTION: This is a classification display method of a video in a video classification display system 1 equipped with: a video analysis part 11 which analyzes an inputted video; a video classification part 12 which classifies the video based on an analysis result in the video analysis part 11; a video display section 13 which displays classification video information based on a classification result in the video classification part 12; and a video database 14 holding information about a video stored by the video classification part 12. The video analysis part 11 generates a difference pattern image from the input image and has a procedure to analyze the video on the basis of this difference pattern image. The video classification part 12 has a partial video section to be used as a reference, and has a procedure to classify the plurality of other partial video sections as the videos of the same class, if these partial video sections are similar by comparison of the difference pattern image based on the above analysis result. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a system and method for classifying and displaying video and a program for executing the system and method.

  Recent developments in video broadcasting environments, such as satellite broadcasting, CATV broadcasting, optical networks, and Internet broadcasting in high-speed mopile networks, are remarkable. As described above, an environment in which a user can access a large amount of video information is being enhanced. On the other hand, the provision of a technology for searching for a video that a user wants to view or accessing a video that the user wants to watch from among such a large amount of video information remains unchanged. It is indispensable to handle these large amounts of video information so that they can be performed efficiently or simply. In addition, as video information diversifies, personal interest in them also diversifies, providing a means to access video adapted to personal interests and preferences, rather than broadcasting uniform video so far The number is also increasing.

  In view of these problems, a technique for structuring a video to be viewed using image information has been invented. For example, “Moving image display apparatus and display method” described in Japanese Patent Laid-Open No. 10-2340040 divides digital moving image data to be viewed into a plurality of scenes according to changes in scenes, and the like. An apparatus and a method for classifying a predetermined video of a person appearing in a group together with the number of appearances.

However, in general, the same person often appears in a video or live video as a certain work, and classification in a scene in which the character is reflected is not very effective. For this reason, video classification focusing on the actions performed by the characters has recently been required. However, these are not automated, and the current situation is to manually classify what operations are being performed on each scene of the video and describe them manually.
JP-A-10-234004 (paragraph number 0016 and FIG. 1)

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a new video classification display method and system and program for classifying and displaying videos. An object of the present invention is to provide a video classification display method, its system, and a program that can be displayed.

  In order to solve the above-described problem, the video classification display method and system according to the present invention determine similarity based on a difference pattern image with respect to a reference video section having a predetermined number of frames for another video section having the same number of frames. If they are similar, the same classification number as that of the reference video section is given, and this is processed for all video sections to classify the frames.

That is, the video classification display system of the present invention is
Means for generating a difference pattern image from the input image and analyzing the video based on the difference pattern image;
If there is a reference partial video section, and based on the analysis result, if another partial video section is similar in the comparison of the difference pattern images, the partial video section is classified as a video of the same class. Means,
And a video database for storing information on the video stored by the video classifying means.

  As the means for analyzing the video, for the input image group, for each pixel, when the difference in pixel value between adjacent frames first exceeds a threshold value, the difference is determined using the frame number as the pixel value. What generates a pattern image may be provided.

As the means for classifying the video, the step of inputting video information and storing it in the video database, the step of inputting the number of processing frames as a minimum unit for classification, and searching for an unclassified frame from the video database A step of acquiring images corresponding to the number of processed frames from the minimum frame number for the unclassified frame group of the search result, and a step of generating a reference difference pattern image from the image group in the means for analyzing the video A step of setting an initial start frame number from the minimum frame number and the number of processing frames, a step of acquiring images for the number of processing frames for each frame from the start frame number, and means for analyzing the video Generating a difference pattern image from the image group; and the difference pattern image Calculating a similarity between the reference difference pattern image and,
A step of assigning a classification number to a frame constituting the difference pattern image when the similarity satisfies a threshold value, and determining whether or not a classification number is assigned in the same reference difference pattern image, and a corresponding frame of the reference difference pattern image And a step of performing a step of assigning a classification number to and a step of updating a video database for the assigned classification number.

  The means for displaying the classified video information includes a step of inputting selection of video information to be displayed, a step of searching the selected video information from a video database, and obtaining a classification number for the video information A step of selecting a representative image for each classification number, a step of displaying a list of the representative images in descending order of the classification number to which the representative image is assigned, and a classification number of a classification display video to be displayed And a step of sequentially displaying the frames having the classification numbers in the ascending order of the frame numbers may be provided.

The video classification display method of the present invention includes a video analysis unit that analyzes an input video, a video classification unit that classifies videos based on an analysis result in the video analysis unit, and a classification result in the video classification unit. In accordance with the video classification display system, the video classification display system comprising: a video display unit that displays the classified video information; and a video database that stores information about the video stored by the video classification unit
The video analysis unit generates a difference pattern image from the input image and analyzes the video based on the difference pattern image;
If the video classification unit has a reference partial video section, and based on the analysis result, if another plurality of partial video sections are similar in the comparison of the difference pattern images, the partial video section is classified into the same class. And classifying the video as a video.

  In this video classification display method, the video analysis unit sets the frame number when the pixel value difference between adjacent frames first exceeds a threshold value for each pixel in the input image group. You may have the procedure which produces | generates a difference pattern image as a pixel value.

The video classification unit inputs video information and stores it in the video database, a step of inputting the number of processing frames as a minimum unit for classification, and a search for unclassified frames from the video database. A step of acquiring a process frame number from the minimum frame number for the unclassified frame group of the search result, and the video analysis unit generates a reference difference pattern image from the image group Procedure and
A procedure in which the video classification unit performs a step of setting an initial start frame number from the minimum frame number and the number of processed frames, and a step of acquiring images for the number of processed frames for each frame from the start frame number; ,
The video analysis unit generates a difference pattern image from the image group,
A step of calculating a similarity between the difference pattern image and a reference difference pattern image, and a step of assigning a classification number to a frame constituting the difference pattern image when the similarity satisfies a threshold; Determining whether or not a classification number is assigned to the same reference difference pattern image, assigning a classification number to a corresponding frame of the reference difference pattern image, and updating a video database for the assigned classification number You may have a procedure.

  A step of inputting a selection of video information to be displayed, a step of searching the selected video information from a video database, and a step of acquiring a classification number for the video information; A step of selecting a representative image for each classification number, a step of displaying a list of the representative images in descending order of the classification number to which the representative image is assigned, and a step of inputting a classification number of a classification display video to be displayed. And a step of sequentially displaying the frames of the classification numbers in the ascending order of the frame numbers.

  The above video classification display method and system can be realized by a program for executing the procedure and means on a computer. Further, the program can be recorded on a computer-readable recording medium or provided through a network. Is possible. Examples of the recording medium include a flexible disk, MO, ROM, memory card, CD, DVD, and removable disk.

  According to the video classification display method and system and program of the present invention, the similarity is determined based on the difference pattern image for the other video sections of the same number of frames with respect to the reference video section of the predetermined number of frames. If so, the same classification number as that of the reference video section is given, and this is processed for all the video sections to classify the frames, so that the videos can be classified by operation and displayed as a list.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic configuration diagram showing a video classification display system of the present invention.

  The video classification display system 1 of this embodiment includes a video analysis unit 11, a video classification unit 12, a video display unit 13, and a video database 14.

  The video analysis unit 11 generates a reference difference pattern image and a difference pattern image from the input image (video content 10), and analyzes the video based on these images. The video classification unit 12 classifies videos based on the analysis result from the video analysis unit 11. The video display unit 13 displays the classified video information based on the classification result in the video classification unit 12. The video database 14 holds information about the video stored by the video classification unit 12.

  In the generation of the reference difference pattern image and the difference pattern image in the video analysis unit 11, when the difference in pixel values between adjacent frames first exceeds the threshold for each pixel with respect to the input image group. The image may be generated using the frame number as a pixel value. The calculation of the similarity between images is preferably used in normal image processing, for example, a normalized cross-correlation value represented by the following equation.

  When the reference image is represented by T (x, y) and the input image is represented by I (x, y), an arithmetic expression for calculating the correlation value R (x, y) can be represented by the following equation (1).

  The threshold value to be used may be determined in advance, or may be dynamically calculated using a statistically calculated value such as an average value, a frequency, or a deviation value. Since the method is not particularly different from the conventional method, detailed description thereof is omitted here.

  The above processing unit can be realized by, for example, a personal computer provided with an input device such as a keyboard, a mouse, and a joystick as an interface with a user, and an output device such as a CRT display. In this case, each of the processing units, that is, the video analysis unit 11, the video classification unit 12, and the video display unit 13 can be realized by a program stored and executed in the memory of such an information processing device.

  FIG. 2 is a flowchart showing a video classification procedure in the system. FIG. 3 is a flowchart showing a video display procedure in the system. Hereinafter, it demonstrates in detail according to this flow.

  The video classification unit 12 classifies videos based on the comparison of the difference pattern images generated by the video analysis unit 11. This process is executed according to the following procedure (steps 101 to 113) shown in FIG.

  (Step 101) In the video classification unit 11, video information (video content 10) is input and stored in the video database 10.

  (Step 102) The number of processing frames as a minimum unit for classification is input.

  (Step 103) An unclassified frame is searched from the video database 14.

  (Step 104) For the unclassified frame group as a search result, images corresponding to the number of processing frames are acquired from the minimum frame number.

  (Step 105) The video analysis unit 11 generates a reference difference pattern image from the acquired image group.

  (Step 106) An initial start frame number is set from the minimum frame number and the number of processed frames.

  (Step 107) Images for the number of processing frames are acquired for each frame from the start frame number.

  (Step 108) The video analysis unit 11 generates a difference pattern image from the acquired image group.

  (Step 109) The similarity between the difference pattern image and the reference difference pattern image is calculated.

  (Step 110) If the degree of similarity satisfies the threshold, a classification number is assigned to the frames constituting the difference pattern image.

  (Step 111) It is determined whether or not a classification number is assigned in the same reference difference pattern image.

  (Step 112) When the classification number is assigned, the classification number is assigned to all frames of the reference difference pattern image. On the other hand, if the classification number is not assigned, the classification number is assigned to the first frame.

  (Step 113) The video database 14 is updated for the assigned classification number.

  In step 104, if it is not possible to acquire images for the number of processed frames from the minimum frame number for the unclassified frame group of the search result (if there are no consecutive unclassified frames for the number of processed frames), the frames are processed. It is assumed that the subsequent successive frames are acquired.

  The video display unit 13 displays the classified video information based on the classification result in the video classification unit 12. This process is performed according to the procedure shown in FIG. 3 (steps 201 to 207).

  (Step 201) Selection of video information to be displayed is input.

  (Step 202) The selected video information is searched from the video database 14.

  (Step 203) The number of classifications is acquired for the searched video information.

  (Stenna 204) For each classification number, the number of processed frames / 2nd frame is selected as a representative image.

  (Step 205) A list of representative images is displayed in descending order of the assigned classification numbers.

  (Step 206) The classification number of the classification display video to be displayed is input.

  (Step 207) The frames of the input classification numbers are sequentially displayed in ascending order of frame numbers.

  At this time, selection of a display target in these is performed using an input device such as a keyboard, a mouse, or a joystick that serves as an interface with the user, and video information as a processing result thereof is output using an output device such as a CRT display. Done. These may be realized by the same computer, or may be computers connected physically by a network such as the Internet, or a mopile terminal such as a mobile phone and a computer.

  Next, with reference to FIG. 4 and FIG. 5, a specific procedure for classifying videos by operation in the video classification display system described so far and displaying them as a list will be described.

  Assume that the video classification unit 11 inputs certain specified video information and stores it in the video database 14. It is assumed that the number of processing frames, which is the minimum unit for classification, for example, 3 frames is specified for this video. On the other hand, as a result of searching the unclassified frame from the video database 14, here, it is composed of an image group of all 12 frames as indicated by 401 in FIG. Subsequently, in the video analysis unit 12, as indicated by 402, an image of three frames from the minimum frame number 1 is acquired, and a reference difference pattern image is generated. On the other hand, three frames of images are sequentially acquired from the start frame number 4 for each frame as indicated by 403 to 409, and each difference pattern is generated. The similarity between the reference difference pattern image generated from 402 and the difference pattern images generated from 403 to 409 is calculated. Here, it is assumed that the similarity with the difference pattern image generated from 406 satisfies the threshold, and classification number 1 is assigned to these three frames. Since this assignment is performed, the classification number 1 is also assigned to the three frames indicated by 402 constituting the reference difference pattern image. For these six frames, the assigned classification numbers are registered in the video database 14. This process is repeated until there are no unclassified frames.

  In this case, 6 frames indicated by white circles 501 in FIG. 5 are subsequently searched as unclassified frames. Subsequently, as indicated by 502, an image of 3 frames from the minimum frame number 4 is acquired, and a reference difference pattern image is generated. On the other hand, when an image of 3 frames is acquired from the start frame number 10, only one is shown as 503. From this, a difference pattern image is generated and the similarity is calculated. Here, assuming that the threshold is satisfied, classification number 2 is assigned to these three frames. Since this assignment is performed, classification number 2 is also assigned to the three frames indicated by 502 constituting the reference difference pattern image. For these six frames, the assigned classification numbers are registered in the video database 14. Since all unclassified frames have been processed, the video classification process ends.

  Here, a specific processing procedure for generating a difference pattern image in the video analysis unit 11 for the difference pattern image used in the video classification unit 12 will be described. Here, for the sake of explanation, it is assumed that processing is performed on the above-described three-frame image group and the image group has a 5 × 5 pixel size as indicated by 601 to 603 in FIG. 6. For each pixel, a frame number is assigned when the pixel value difference between adjacent frames first exceeds a threshold value. If the pixel value difference between adjacent frames (601 and 602, 602 and 603) is calculated for the images indicated by 601 to 603 in FIG. 6, the pixels indicated by 605 to 606 exceed the threshold value. . On the other hand, first, since the pixel indicated by 606 first exceeds the threshold value, frame number 1 is assigned to this pixel, as indicated by 609. Subsequently, frame number 2 is assigned to the pixel indicated by 605 that does not exceed the threshold value at 606, as indicated by 608. In this way, finally, an image as shown in 610 is generated as the difference pattern image.

  The videos classified in this way and stored in the video database 14 are viewed. As shown in FIG. 7, the video display unit 13 presents video information to be displayed and designates information to be viewed. Here, it is assumed that a video related to sports is displayed and the user selects baseball. The selected video information is searched from the video database 14, and a representative image is selected for each number into which the video is classified. Here, as shown in FIG. 8, the top three hitting actions, throwing actions, and defending actions with many classification numbers are selected, and their representative images are displayed in a list. From now on, for example, if the user selects a hitting operation, the frame groups classified into this classification number are sequentially displayed in ascending order of the frame number.

  Although the present invention has been specifically described above based on the embodiments, it is obvious to those skilled in the art that the embodiments of the present invention can be variously modified and modified within the scope of the technical idea of the present invention. Of course, such variations and modifications fall within the scope of the appended claims.

  In addition, the video classification display method described in the embodiment is configured by a program that causes a computer to execute the processing steps shown in FIGS. 1 to 3 and can be executed by the computer. A program for realizing, or a program for causing a computer to execute the processing steps on a recording medium readable by the computer, for example, a flexible disk, MO, ROM, memory card, CD, DVD, removable disk, etc. It can be recorded, stored and distributed. Further, these programs can be provided via a network such as the Internet or e-mail. Then, the present invention can be implemented by installing the program from the recording medium into a computer or by downloading the program from a network and installing the program into the computer. However, installation on a computer is a computer unit, and when there are multiple computers to be installed due to multiple devices and systems, it is natural that the program is installed for each necessary processing part. is there. In this case, the program may be recorded on a recording medium corresponding to a computer, or downloaded via a network.

The schematic block diagram which showed the video classification display system of this invention. The flowchart explaining the procedure of the video classification in the video classification display system of this invention. The flowchart explaining the procedure of the video display in the video classification display system of this invention. Explanatory drawing which showed an example of the process by the image | video classification display system of this invention. Explanatory drawing which showed an example of the process by the image | video classification display system of this invention. Explanatory drawing which showed an example of the process by the image | video classification display system of this invention. Explanatory drawing which showed an example of the process by the image | video classification display system of this invention. Explanatory drawing which showed an example of the process by the image | video classification display system of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Video classification display system 10 ... Video content 11 ... Video analysis part 12 ... Video classification part 13 ... Video display part 14 ... Video database

Claims (9)

In a video classification display system for classifying and displaying video,
Means for generating a difference pattern image from the input image and analyzing the video based on the difference pattern image;
If there is a reference partial video section, and based on the analysis result, if another partial video section is similar in the comparison of the difference pattern images, the partial video section is classified as a video of the same class. Means,
A video classification display system, comprising: means for displaying classified video information based on a classification result in the video classification unit; and a video database for storing information relating to videos stored by the means for classifying the video. The means for analyzing the video, for the input image group, for each pixel, when the difference in pixel value between adjacent frames first exceeds a threshold value, the difference number pattern is set with the frame number as the pixel value. The video classification display system according to claim 1, wherein an image is generated. The means for classifying the video is:
Inputting video information and storing it in the video database;
Inputting the number of processing frames as a minimum unit for classification;
Retrieving unclassified frames from the video database;
Acquiring images for the number of processing frames from the minimum frame number for the unclassified frame group of the search results;
Generating a reference difference pattern image from the image group in the means for analyzing the video;
Setting an initial start frame number from the minimum frame number and the number of processing frames;
Acquiring images for the number of processing frames for each frame from the start frame number;
Generating a difference pattern image from the image group in the means for analyzing the video;
Calculating a similarity between the difference pattern image and a reference difference pattern image;
A step of assigning a classification number to a frame constituting the difference pattern image when the similarity satisfies a threshold;
Determining whether or not to assign a classification number in the same reference difference pattern image and assigning a classification number to a corresponding frame of the reference difference pattern image; and
The video classification display system according to claim 1, wherein the video database is updated with respect to the assigned classification number. The means for displaying the classified video information includes:
Inputting a selection of video information to be displayed;
Retrieving the selected video information from the video database;
Obtaining a classification number for the video information;
Selecting a representative image for each classification number;
Displaying a list of the representative images in descending order of assigned classification numbers;
A step of inputting a classification number of a classification display video to be displayed;
The video classification display system according to any one of claims 1 to 3, wherein a step of sequentially displaying the frames having the classification numbers in the ascending order of the frame numbers is performed. Video analysis unit for analyzing input video, video classification unit for classifying video based on the analysis result in the video analysis unit, and video display unit for displaying classified video information based on the classification result in the video classification unit And a video classification display method in a video classification display system comprising a video database that holds information about the video stored by the video classification unit,
The video analysis unit generates a difference pattern image from the input image and analyzes the video based on the difference pattern image;
If the video classification unit has a reference partial video section, and based on the analysis result, if another plurality of partial video sections are similar in the comparison of the difference pattern images, the partial video section is classified into the same class. A video classification display method comprising: a procedure for classifying the video as a video. When the difference in pixel value between adjacent frames for each pixel first exceeds a threshold for the input image group with respect to the input image group, the video analysis unit uses the frame number as the pixel value to generate a difference pattern image. 6. The video classification display method according to claim 5, further comprising a generation procedure. A step of inputting video information and storing it in the video database; a step of inputting a processing frame number as a minimum unit for classification; and a step of searching for an unclassified frame from the video database; , A procedure for executing the step of acquiring images for the number of processing frames from the minimum frame number for the unclassified frame group of the search result, and a procedure for the video analysis unit to generate a reference difference pattern image from the image group, ,
A procedure in which the video classification unit performs a step of setting an initial start frame number from the minimum frame number and the number of processed frames, and a step of acquiring images for the number of processed frames for each frame from the start frame number; ,
The video analysis unit generates a difference pattern image from the image group,
A step of calculating a similarity between the difference pattern image and a reference difference pattern image, and a step of assigning a classification number to a frame constituting the difference pattern image when the similarity satisfies a threshold; Determining whether or not a classification number is assigned to the same reference difference pattern image, assigning a classification number to a corresponding frame of the reference difference pattern image, and updating a video database for the assigned classification number 7. The video classification display method according to claim 5, further comprising: a procedure. A step of inputting a selection of video information to be displayed, a step of searching the selected video information from the video database, a step of acquiring a classification number for the video information, A step of selecting a representative image for the classification number, a step of displaying a list in descending order of the classification number assigned with the representative image, a step of inputting the classification number of the classification display video to be displayed, The video classification display method according to any one of claims 5 to 7, further comprising a step of sequentially displaying frames of classification numbers in ascending order of frame numbers. A program for causing a computer to execute the video classification display method according to any one of claims 1 to 8, and the procedure and means in the system.

Images