JPH05244475A - Method of extracting representative pattern - Google Patents

Abstract

PURPOSE:To obtain the method of extracting a representative pattern which is able to automatically extract a proper representative pattern that takes a physical characteristic of a content such as shot or take into account. CONSTITUTION:Take/shot of video information is specified. The presence of camera operation is specified, when it is specified that no camera is operated, a physical characteristic of the take/shot is extracted. When no physical characteristic is in existence, a head pattern of the take/shot is used for a representative pattern and when any physical characteristic is in existence, the pattern having the characteristic value is used for the representative pattern. When the presence of camera operation is specified, the physical characteristic of the take/shot is extracted. When no physical characteristic is in existence, the final pattern of the take/shot is used for the representative pattern and when any physical characteristic is in existence, the pattern having the characteristic value is used for the representative pattern. Furthermore, when the presence of the camera operation is specified, the pattern having the characteristic value is used for the representative pattern. Moreover, when the presence of the camera operation is specified, the final pattern of the take/shot is extracted as one of the representative patterns regardless of the physical characteristic.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a representative screen extracting method for selecting and extracting a screen representative of the contents of a video from a video continuous for a certain time in units of frames.

[0002]

2. Description of the Related Art Conventionally, a head screen of shots, takes, etc. has been used as a representative screen of video. This shot,
As a method of extracting the top screen of a take or the like, there is a method of analyzing video information and detecting a cut point which is a transition point of a shot, a take, etc. [eg, Yanaka et al. KAISHO 64-68084]. A method of analyzing voice information and extracting a representative screen based on the characteristic points thereof has also been reported [Tonomura et al., "Scene extraction processing method", JP-A-3-80782]. In addition, a method of extracting a representative screen by using the priority of video / image defined by a person has been reported [Abe et al., "Automatic moving image digest automatic generation method", Japanese Patent Laid-Open No. 3-90968].

[0003]

However, the above-mentioned conventional technique is an extraction method in which the representative screen of the image is considered to be only the top screen of shots, takes, etc., and the physical characteristics of the contents of shots, takes, etc. are considered. Since it is not taken into consideration, the contents of the screen such as shots and takes are not always appropriate representative screens. Further, the method of extracting the representative screen in consideration of the priority has a problem in the automation of the processing because the enormous video / image priority is defined by the person.

An object of the present invention is to provide a representative screen extracting method capable of automatically extracting an appropriate representative screen.

[0005]

In order to achieve the above object, the representative screen extracting method of the present invention performs a statistical analysis on a video image and temporal characteristics of the camera operation at the time of shooting the video image and the video characteristics. The spatiotemporal feature is calculated, the change point of the feature amount is obtained from the calculation result, and the image of the frame that gives this change point is selected and extracted as the representative screen.

[0006]

Since the physical characteristics of the contents such as shots and takes are taken into consideration, an appropriate representative screen can be extracted. Further, since the processing of the present invention can be calculated, the representative screen can be automatically extracted.

[0007]

Embodiments of the present invention will now be described with reference to the drawings.

FIG. 1 is a flow chart showing an algorithm of a representative screen extracting method according to an embodiment of the present invention, FIG. 2 is a random frequency distribution diagram of motion vector differences used for take / shot detection, and FIG. 3 is a camera operation presence / absence detection. FIG. 4 is a diagram showing the frequency distribution of the motion vector differences used, FIG. 4 is a diagram showing the characteristics of the camera operation parameters used for extracting the representative screen, and FIG. 5 is a diagram showing the features of the movement direction time change used for extracting the representative screen. A representative screen extraction method that has been conventionally used is shown by a broken line in FIG.
This conventional representative screen extraction method is configured only by the procedure for defining take / shot (procedure A), and the representative screen is the top screen of the defined take / shot.

The present embodiment is characterized in that procedures B, C, and D are newly added to the representative screen extraction processing.
A representative screen is extracted in consideration of physical characteristics of shot contents. Also, when shooting a video, the camera is always operated. In the present embodiment, the camera operation is roughly classified into two types, that is, the still state and the still state, and the still state is regarded as no camera operation. A feature of this embodiment is that a take with camera operation /
In the case of a shot, the final screen where the camera operation ends may be extracted as one of the representative screens. Normally, the end of the camera operation is the end of the take / shot, and in the case of the camera operation, the final screen of the take / shot becomes one of the representative screens. This representative screen extraction is called the "panning, zooming, etc." of the final screen of a take / shot shot with camera operation in the field of broadcasting stations, movies, etc., and is handled carefully when shooting / editing. Due to being Hereinafter, each procedure A,
The references for B, C, and D are cited and examples thereof are shown, and the representative screen extracting method of the present embodiment will be described in detail.

A number of announcements have recently been made regarding the take / shot definition (procedure A) (Otsuji et al., "Movie browsing using luminance information", IEICE Technical Report, IE90-
103, 1991 etc.). These methods basically use the difference between frames of video information. This method uses the fact that video information (luminance, chromaticity, etc.) changes abruptly between frames of take / shot joints (cut points). Here, as an example of the process A, FIG. 2 shows a take / shot definition result using a random frequency distribution of inter-frame difference vectors of motion vectors. Ranges 23, 24, and 25 in FIG. 2 are takes / shots, and points 21 and 22 are cut points. The method used here is
This is a discontinuity in the smoothness of motion between frames of a take / shot joint, and a difference vector of motion vectors, which is technically calculated from block-by-block correlation between frames, is used. When this method is used, at the joint of take / shot, the difference vector calculated as a result shows randomness within the frame. FIG. 2 is a diagram showing the size of each frame of this randomness. Frames with large randomness are the cut points that are the joints between takes / shots.

Next, in the camera operation presence / absence regulation (procedure B), a regulation result is shown in FIG. 3 as an example thereof. Take / shot 31 and take / shot 33 in the figure indicate that there is no camera operation, and take / shot 32 in FIG. 3 indicates that there is a camera operation. This method is based on the smoothness of the movement by camera operation seen between frames.
As a method, the sum of the time difference vectors of motion vectors calculated between frames in a frame of zero size is used. The magnitude of the time difference vector of zero represents the smoothness of the movement, and if the total sum in the frame is large, it can be defined that the camera operation is being performed. 2 and 3
It is indicated that the video used in the embodiment is composed of four takes / shots (takes / shots 31, 32, 33 and take / shot of the final frame). If you can specify that there is a camera operation at this processing stage,
The final screen of take / shot is extracted as one of the representative screens.

Next, when it is specified that the camera is not operated,
In step C, the physical characteristics of the take / shot are extracted. If no feature is specified in this procedure, the representative screen is the start screen of the take / shot. Take / Shot 31
In the case of. FIG. 5 shows how the motion vector of the take / shot 31 changes in direction with time. The distributions in FIG. 5 are clustered according to the spread of each distribution, and each cluster is shown by a symbol (□, Δ, *, ◇). The take / shot 31 is characterized by the temporal change in the direction of the motion vector, and a cluster of distribution can be seen in the vector direction vs time space. FIG. 5 is a diagram showing the characteristics. The representative screen of the take / shot 31 is points 51 and 5 which are physical feature points (center of gravity of cluster distribution, etc.).
Two screens are shown at points 2 and 53, respectively.

Next, in procedure D, the physical characteristics of the takes / shots specified to have camera operation in procedure B are extracted in procedure D. Regarding the camera operation parameter estimation method which is one of the physical characteristics, "Akutsu et al.," Camera operation specification method for the purpose of moving image indexing ", No. 2
Annual Symposium on Functional Graphic Information Systems, p,
p. 107-112, 1991 ”and the like. These methods use motion vectors calculated from the video. Spatiotemporal regularity of motion vector generated by camera operation (temporal continuity of motion vector,
The camera operation is defined by divergence point / convergence point position, vector size, direction). As an example of the process D, the result of defining the camera operation using the method of Akutsu et al. Is shown in FIG.

FIG. 4 shows the camera operation parameters estimated for the take / shot 32. 41 in FIG.
Indicates a camera operation parameter space (without a time axis), pan (f) is a pan parameter, and tilt (f) is a pan parameter.
Indicates a tilt parameter and zoom (f) indicates a zoom parameter, respectively, and represents a screen speed [pixel / frame]. The representative screen of the take / shot 32 is the final screen, the pan parameter pan (f), and the tilt parameter til.
t (f), and a total of four screens shown by points 41, 42, and 43, which are physical feature points (extreme values or the like) of the zoom parameter zoom (f).

In addition, the estimated camera operation is taken / taken
It is also possible to subtract from the shot, return to procedure B without camera operation, and extract physical features in procedure C.

FIG. 6 is a block diagram of an apparatus for realizing the representative screen extracting method of this embodiment. This apparatus includes a video signal input section 61 for inputting a video signal, a take / shot defining section 62 for performing the procedure A, a camera operation defining section 63 for performing the procedure B, and a procedure C, D. It is composed of a physical feature extraction unit 64 for performing and a representative screen output unit 65 for selecting and extracting a representative screen.

Although the present invention has been specifically described based on the embodiments, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Needless to say. It goes without saying that the physical feature space of the image used for extracting the representative screen includes all the image information such as the movement, the luminance, the hue, and the sound in addition to the movement shown in the embodiment.

[0018]

As described above, according to the present invention, the statistical analysis of a video is performed to calculate the temporal characteristics of the camera operation and the spatio-temporal characteristics of the video when the video is photographed, and based on the calculation result, By obtaining the change point of the characteristic amount, selecting the image of the frame that gives this change point as the representative screen, and extracting the representative screen, an appropriate representative screen can be automatically extracted.

[Brief description of drawings]

FIG. 1 is a flowchart showing an algorithm of a representative screen extracting method according to an embodiment of the present invention.

FIG. 2 is a random frequency distribution diagram of motion vector differences used for take / shot detection.

FIG. 3 is a frequency distribution diagram of motion vector differences used for camera operation presence / absence detection.

FIG. 4 is a diagram showing characteristics of camera operation parameters used for extracting a representative screen.

[Fig. 5] Fig. 5 is a diagram showing characteristics of temporal changes in a moving direction used for extracting a representative screen.

FIG. 6 is a block diagram of an apparatus that implements the representative screen extraction method of FIGS.

[Explanation of Codes] A to D Procedures 21, 22 Cut points 23, 24, 25 Take / shot 31, 32, 33 Take / shot 41 Camera operation parameter space 42, 43, 44 points 51, 52, 53 points 61 Video signal Input part 62 Take / shot definition part 63 Camera operation definition part 64 Physical feature extraction part 65 Representative screen output part

Claims (1)

[Claims] 1. A representative screen extraction method for selecting and extracting an image representative of the contents of a video from a video continuous for a certain time with a frame as a unit, and performing statistical analysis on the video, The temporal characteristics of the camera operation at the time of shooting the video and the spatiotemporal characteristics of the video are calculated, the change point of the feature amount is obtained from the calculation result, and the image of the frame giving the change point is selected as the representative screen. , A representative screen extracting method for extracting.