JP2002140712A - Av signal processor, av signal processing method, program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect the boundary of scenes. SOLUTION: In a step S1, inputted video data is divided into a video segment or an audio segment or into both segments if possible. In a step S2, feature variable showing the feature of the segment is calculated. In a step S3, the similarity of the segment is measured by using feature variable. In a step S4, it is judged whether the segment is in the gap of the scenes or not. Namely, a video/audio processor considers the respective segments to be in the present by using a non-similarity measurement reference calculated in the previous step S3 and the feature variable extracted in the previous step S2. It is detected in which part of the past or the future the existing ratio of adjacent similar segments is higher compared to reference segments. The pattern of the change of the existing ratio is checked and it is judged whether it is the boundary of the scenes or not.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an AV signal processing apparatus and method, a program, and a recording medium, and more particularly to an AV signal suitable for use when selecting and reproducing a desired portion from a series of video signals. The present invention relates to a signal processing device and method, a program, and a recording medium.

[0002]

2. Description of the Related Art For example, a user may want to search for and reproduce a desired part such as a part of interest from a video application composed of a large amount of different video data such as a television program recorded on video data. .

As described above, as a general technique for extracting desired video contents, there is a storyboard which is a panel formed by arranging a series of videos depicting main scenes of an application. This storyboard is a video board that breaks down video data into so-called shots and displays a video image represented by each shot. Most of such video extraction technologies are, for example, "G. Ahanger a
nd TDC Little, Asurvey of technologies for pars
ing and indexing digital video, J. of Visual Commu
As described in “Nication and Image Representation 7: 28-4, 1996”, a shot is automatically detected and extracted from video data.

[0004]

By the way, for example, a typical 30-minute television program contains hundreds of shots. Therefore, in the above-described conventional video extraction technology, the user needs to examine a storyboard in which a huge number of extracted shots are arranged, and to understand such a storyboard, it is necessary to impose a heavy burden on the user. was there.

Further, the conventional video extraction technology has a problem that, for example, shots in a conversation scene in which two persons are alternately photographed according to a change in a speaker are often redundant. In this way, shots are too low in the hierarchy for extracting video structure and have a lot of wasted information,
Conventional video extraction technology that extracts such shots,
It was not convenient for the user.

[0006] Other video extraction techniques include, for example, "A. Merlino, D. Morey and M. Maybury, Broadcast n.
ews navigation using story segmentation, Proc. of
ACM Multimedia 97, 1997 ”and JP-A-10-136297.
Some use very specialized knowledge about a particular content genre, such as news or football games, as described in the publication. However, this conventional video extraction technique can obtain a good result with respect to a target genre, but is useless for other genres, and cannot be easily generalized as a result of being limited to a genre. There was a problem.

Further, as another video extraction technique, there is a technique for extracting a so-called story unit as described in, for example, US Pat. No. 5,708,767.
However, this conventional video extraction technique is not completely automated, and requires a user operation to determine which shot has the same content. In addition, this conventional video extraction technique has a problem that the calculation required for the processing is complicated and the application target is limited to only video information.

[0008] Further, as another video extraction technology,
For example, as described in Japanese Patent Application Laid-Open No. 9-21879, a scene is identified by combining shot detection and silent portion detection. However, this conventional video extraction technique is limited to a case where a silent portion corresponds to a shot boundary.

As another video extraction technique, for example, “H. Aoki, S. Shimotsuji and O. Hori, A shot classi”
fication method to select effective key-frames for
video browsing, IPSJ Human Interface SIG Notes,
7: 43-50, 1996 "and Japanese Patent Application Laid-Open No. 9-93588, some of which detect repeated similar shots in order to reduce the redundancy of the display on the storyboard. However, this conventional video extraction technique can be applied only to video information, and cannot be applied to audio information.

[0010] Further, these conventional techniques have a plurality of problems when they are mounted on home appliances such as set-top boxes and digital video recorders. that is,
This is mainly because post-processing is premised in the prior art. Specifically, there are the following three problems.

The first problem is that the number of segments depends on the length of the content. Even if the number of segments is constant, the number of shots contained therein is not constant. Therefore, the amount of memory required for scene detection cannot be fixed, so that the amount of memory required has to be set excessively. This was a major problem for home appliances with a small amount of memory.

The second problem is that home appliances require real-time processing in which a predetermined process must be completed within a predetermined time. However, since the number of segments cannot be fixed and post-processing must be performed, it has always been difficult to end the processing within a predetermined time. This means that it is difficult to perform further real-time processing when a low-performance CPU mounted on a household device must be used.

The third problem is that post-processing is required as described above, and the processing result of scene detection cannot be ended every time a segment is generated. This means that if the recording state stops for some reason during the recording, it is not possible to obtain an intermediate result. This means that it is not possible to perform sequential processing while recording, which is a major problem in home appliances.

Further, in the prior art, when a scene is determined, a method based on a repetition pattern of segments or grouping of other segments is used, so that the detection result of the scene is unique. Therefore, it is impossible to determine whether the detected boundary is likely to be an actual scene boundary or not, and the number of scenes detected cannot be controlled step by step.

Further, in listing videos, it is necessary to minimize the number of obtained scenes for easy viewing. Therefore, when the number of detected scenes is limited, a problem arises as to which scene should be displayed. Therefore, if the importance of each of the obtained scenes is known, the scenes can be viewed and listed according to the order of importance. However, the prior art does not provide a measure of how important the resulting scene is.

The present invention has been made in view of such circumstances, and has as its object to detect scene boundaries so that recorded video data can be reproduced from any scene.

[0017]

According to the present invention, there is provided an AV signal processing apparatus comprising: a feature extracting means for extracting a feature of a segment formed by a series of frames constituting an AV signal; Calculating means for calculating a metric for measuring the similarity of a feature with a segment; similarity measuring means for measuring the similarity between a reference segment and another segment using the metric; Measurement value calculation means for calculating a measurement value indicating that a reference segment may be a scene boundary using the similarity measured by the gender measurement means; and a temporal pattern of the measurement value calculated by the measurement value calculation means. And a boundary determining means for determining whether or not the reference segment is a boundary of the scene based on the analysis result.

The AV signal can include at least one of a video signal and an audio signal.

The AV signal processing apparatus according to the present invention may further include an intensity value calculating means for calculating an intensity value indicating a degree of change of the measured value corresponding to the reference segment.

The measured value calculating means finds a similar segment in a predetermined time domain with respect to a reference segment, analyzes the time distribution of the similar segment, and quantifies the ratio existing in the past and the future. Measurements can be calculated.

The boundary determining means may determine whether or not the reference segment is a boundary of the scene based on the sum of the absolute values of the measured values.

The AV signal processing apparatus of the present invention can further include an audio segment generating means for detecting a shot which is a basic unit of the video segment and generating an audio segment when the AV signal includes a video signal. .

In the AV signal processing apparatus according to the present invention, when the audio signal is included in the AV signal, the audio signal generating means for generating an audio segment by using at least one of the characteristic amount of the audio signal and the silent section. Can be included.

The feature amount of the video signal can include at least a color histogram.

The feature amount of the audio signal may include at least one of a volume and a spectrum.

The boundary determining means may determine whether or not the reference segment is a boundary of the scene by comparing a preset threshold value with the measured value.

According to the AV signal processing method of the present invention, a feature amount extracting step of extracting a feature amount of a segment formed by a series of frames constituting an AV signal, and a feature amount of a reference segment and another segment are extracted. A calculating step of calculating a metric for measuring similarity, a similarity measuring step of measuring the similarity between the reference segment and another segment using the metric, and a processing of the similarity measuring step. A measured value calculating step of calculating a measured value indicating that the reference segment may be a scene boundary using the measured similarity; and a temporal pattern of the measured values calculated in the processing of the measured value calculating step. Analyzing the change of the image, and determining whether the reference segment is a boundary of the scene based on the analysis result. That.

A program according to the present invention includes a feature value extracting step of extracting a feature value of a segment formed by a series of frames constituting an AV signal, and a similarity feature value between a reference segment and another segment. A calculation step of calculating a measurement standard for measurement, a similarity measurement step of measuring the similarity between a reference segment and another segment using the measurement standard, and a similarity measurement step. Using a similarity, a measurement value calculation step of calculating a measurement value indicating that the reference segment may be a scene boundary, and a change in a temporal pattern of the measurement value calculated in the processing of the measurement value calculation step. Analyze,
A boundary determining step of determining whether or not the reference segment is a boundary of the scene based on the analysis result.

A program of a recording medium according to the present invention includes a feature amount extracting step of extracting a feature amount of a segment formed by a series of frames constituting an AV signal, and a feature amount extracting step of a feature amount between a reference segment and another segment. A calculating step of calculating a metric for measuring similarity, a similarity measuring step of measuring the similarity between the reference segment and another segment using the metric, and a processing of the similarity measuring step. A measured value calculating step of calculating a measured value indicating that the reference segment may be a scene boundary using the measured similarity; and a temporal pattern of the measured values calculated in the processing of the measured value calculating step. And a boundary determination step of determining whether or not a reference segment is a scene boundary based on the analysis result. To.

In the AV signal processing apparatus and method and the program according to the present invention, the characteristic amount of a segment formed by a series of frames constituting an AV signal is extracted, and the characteristic amount of a reference segment and another segment is extracted. A metric for measuring the similarity of the segment is calculated, the similarity between the reference segment and other segments is measured using the metric, and the reference segment is determined using the measured similarity. A measurement is calculated that indicates a possible boundary of the scene. Further, a change in the temporal pattern of the calculated measurement value is analyzed, and it is determined whether or not the reference segment is a scene boundary based on the analysis result.

[0031]

DETAILED DESCRIPTION OF THE INVENTION An object of the present invention is to segment video data into scenes. The meaning of the separation means that a boundary between scenes is detected. A scene is composed of one or more segments. Each scene has its own unique features, so that when comparing segments at each boundary of adjacent scenes, those features show significant differences. In other words, a place where such a remarkable difference appears is the boundary of the scene, and by detecting the boundary, the scene can be separated in segment units.

In performing this processing, first, the target video data is divided into segments, as in the above-described prior art. The segments obtained by division form a time series, and it is necessary to determine whether or not each segment has a scene boundary between the next segment. Based on each segment, it is examined where similar segments are located in time in the neighboring segments.

When it is determined that there is a scene boundary, a change point where a peculiar change appears in a short time from a pattern that has been concentrated in the past to a pattern that is concentrated in the future is detected. A scene from the change point to the next change point is one scene. Sufficient information can be obtained simply by looking at local changes before and after scene boundaries to find where such pattern changes occur.

Further, by measuring the magnitude of the local change, scene segmentation can be controlled stepwise. This is because it has been empirically found that the visual change point coincides well with the semantic change point of the scene. The present invention is for detecting a scene boundary and separating a scene such as video data on the basis of the above. Also, video data can be displayed in an easily viewable manner based on the scene boundary information.

Next, the outline of the present invention will be specifically described.
First, each feature will be described separately for a case where there is a boundary between scenes and a case where it does not exist. FIG. 2 shows a specific example of certain video data. In the figure, the unit of video data is shown in segment units, and is constituted by three scenes 1 to 3. In the figure, it is assumed that the time axis is directed to the right. A region where no boundary exists is referred to as a non-boundary region, and a region where a boundary exists is referred to as a boundary region, which is shown in detail in FIG.

The non-boundary region shown in FIG. 4A is the portion of scene 2 in time, and is the time region of segments 3 to 11 where no boundary exists with other scenes. Also, in contrast to this, the boundary region in FIG. 4B shows the time region of segments 8 to 15 adjacent to the scene including the boundary region of scene 2 and scene 3.

First, the characteristics of the non-boundary area, which represents the case where no boundary exists, will be described. Since the non-boundary region is composed of only similar segments, similar segments exist almost equally when divided into past and future time zones with respect to the reference segment in the non-boundary region. Therefore, no peculiar change pattern appears in the distribution pattern of the similar segments.

The boundary area is different from the non-boundary area and represents a part of a time zone including a boundary point where two scenes are adjacent to each other. Here, the scene means a scene composed of segments having high similarity to each other. Therefore, the segments 8 to 11 constituting the scene 2 and the segments 1 constituting the different scene 3
The segments 2 to 15 are adjacent to each other, and the features of the segments of the scene are different from each other with the boundary therebetween.

In order to detect scene boundaries, each segment is first assumed to be a temporal reference (current). For each of them, it can be realized by examining the change of the temporal distribution pattern of the most similar segments (past or future when viewed from the reference).

As can be seen from the boundary region shown in FIG. 4B, as the segments 8 to 11 sequentially approach the boundary as a temporal reference, the most similar segment exists in the past with respect to the future. The ratio increases and reaches 100% near the boundary (end of the scene). Immediately after the boundary (the beginning of the next scene), the ratio of the future to the past becomes 100%, and the segment 1
As 2 to segment 15 become the temporal reference in order,
The ratio goes down.

Therefore, by changing the pattern of the time distribution ratio of the most similar segments, it is possible to specify a place which is likely to be a boundary of a scene. In addition, since this typical pattern has a very high probability of appearing in a local portion near the boundary of the scene, the boundary can be identified from a change in the pattern by examining only the vicinity of the boundary. In other words, it is not necessary to take an unnecessarily large time region for examining the distribution pattern of similar segments.

When these changes in the pattern are digitized, the degree of the change is linked to the degree of the visual change of the scene. It has been empirically and experimentally known that the degree of the visual change of the scene is linked to the degree of the semantic change of the scene. Therefore, if this digitized value is used as a boundary measurement value, it is possible to detect a scene corresponding to the magnitude of the semantic degree of the scene by the magnitude of this value.

Next, a video and audio processing apparatus according to an embodiment of the present invention will be described. Before that, video data to be processed by the video and audio processing apparatus will be described.

In the present invention, it is assumed that video data to be processed is modeled as shown in FIG. 1 and has a data structure hierarchized into three levels of frames, segments, and scenes. That is, the video data is
The lowest layer is composed of a series of frames.
Further, the video data is constituted by segments formed from a series of consecutive frames, as a layer immediately above the frame. Furthermore, the video data is composed of a scene formed in the highest layer by grouping the segments based on a meaningful association.

This video data generally contains both video and audio information. That is, in the video data, the frame includes a video frame which is a single still image and an audio frame representing audio information sampled at a sampling rate of several KHz to several tens KHz.

A video segment is composed of a series of video frames continuously photographed by a single camera, and is generally called a shot.

On the other hand, a voice segment can be defined in many ways, and the following can be considered as an example. Some audio segments are defined by silence periods in video data detected by generally well-known methods. The audio segment is “D. Kimber and L. Wilcox, Acoustic Segm
entation for Audio Browsers, Xerox Parc Technical
As described in “Report”, for example, some are formed from a series of speech frames classified into a small number of categories such as speech, music, noise, and silence.
In addition, the audio segment is "S. Pfeiffer, S. Fische
r and E. Wolfgang, Automatic Audio Content Analysi
s, Proceeding of ACM Multimedia 96, Nov. 1996, pp2
As described in 1-30 ", a large change in a certain feature between two consecutive audio frames may be detected as a transition between audio and determined based on this.

A scene is a higher level based on the meaning of the content of the video data. Scenes are subjective and depend on the content or genre of the video data. A scene is composed of a video segment or an audio segment whose characteristics are similar to each other.

Here, for each segment in the video data, a segment having similar characteristics existing in the vicinity of the segment is changed from a pattern concentrated in the past to a pattern concentrated in the future. And a change point showing a peculiar change is detected, and the next change point is regarded as one scene from the change point. Such patterns correspond to scene breaks because similar characteristics of the segments greatly change at the boundary of the scene because the characteristics of the segments included in each scene are different. This is very relevant to the high level meaningful structure in the video data, and the scenes represent such a high level meaningful unity in the video data.

Next, an example of the configuration of a video / audio processing apparatus according to an embodiment of the present invention will be described with reference to FIG. The video and audio processing apparatus measures the similarity between segments using the feature amounts of the segments in the video data described above, collects these segments into scenes, and automatically extracts a video structure. It can be applied to both audio segments.

As shown in FIG. 3, the video / audio processing apparatus
A video dividing unit 11 for dividing an input video data stream into video or audio or both segments, a video segment memory 12 for storing video data division information, and a video feature extraction for extracting a feature in each video segment. Unit 13, an audio feature amount extraction unit 14 for extracting the feature amount of each audio segment, a segment feature amount memory 15 for storing the feature amounts of the video segment and the audio segment, a scene detection unit 16 for combining the video segment and the audio segment into a scene, And a feature amount similarity measuring unit 17 for measuring the similarity between the two segments.

The video dividing unit 11 receives, for example, MPEG (Moving Picture Experts Group) 1, MPEG2,
Alternatively, a video data stream composed of video data and audio data in various digitized formats including a compressed video data format such as so-called DV (Digital Video) is divided into video, audio, or both segments. is there.

When the input video data is in a compressed format, the video dividing unit 11 can directly process the compressed video data without completely expanding the compressed video data. The video division unit 11 processes the input video data and divides the video data into a video segment and an audio segment.
Further, the video division unit 11 outputs division information, which is a result of dividing the input video data, to the video segment memory 12 at the subsequent stage. Further, the video division unit 11 divides the division information into the video characteristic amount extraction unit 13 and the audio characteristic amount extraction unit 1 in the subsequent stage according to the video segment and the audio segment.
4 is output.

The video segment memory 12 stores the division information of the video data supplied from the video division section 11. In addition, the video segment memory 12 outputs division information to the scene detection unit 16 in response to an inquiry from the scene detection unit 16 described later.

The video feature quantity extraction unit 13 includes the video division unit 1
The feature amount of each video segment obtained by dividing the video data by 1 is extracted. The video feature amount extraction unit 13 can directly process the compressed video data without completely expanding it. The video feature extraction unit 13 outputs the extracted feature of each video segment to the segment feature memory 15 at the subsequent stage.

The audio feature amount extraction unit 14 includes the video division unit 1
The feature amount of each audio segment obtained by dividing the video data by 1 is extracted. The audio feature amount extraction unit 14 can directly process the compressed audio data without completely expanding it. The audio feature amount extraction unit 14 outputs the extracted feature amount of each audio segment to the subsequent segment feature amount memory 15.

The segment feature memory 15 stores the features of the video segment and the audio segment supplied from the video feature extractor 13 and the audio feature extractor 14, respectively. The segment feature memory 15 responds to an inquiry from a feature similarity measuring unit 17 described later,
The stored feature values and segments are output to the feature value similarity measurement unit 17.

The scene detector 16 determines whether the video segment and the audio segment are scene boundaries using the division information held in the video segment memory 12 and the similarity between the segments. Scene detector 16
Detects the boundary of the scene by identifying the transition point where the distribution pattern of the segment with the most similar feature in the vicinity of each segment switches from the state concentrated in the past to the state concentrated in the future, and detects the boundary of the scene and Confirm the last part.
The scene detector 16 moves one segment in time series every time a segment is generated, and measures the distribution pattern of the nearest similar segments in the vicinity. The scene detecting unit 16 uses the feature similarity measuring unit 17 to specify the number of similar segments in the neighboring segments. That is, the number of the nearest neighbors of the feature amount in the feature space is obtained. Then, the boundary of the scene is specified from a change in the pattern of the difference in the number of similar segments closest to the segment existing in the past and those existing in the future with the segment as a boundary.

The feature quantity similarity measuring unit 17 measures the similarity between each segment and its neighboring segments. The feature similarity measurement unit 17 queries the segment feature memory 15 to search for a feature related to a certain segment.

The video data recording section 18 stores various data related to a video stream and video data.
The so-called additional information data is recorded. Here, the scene boundary information output from the scene detection unit 16 and the intensity value calculated for the scene are stored.

The video display unit 19 realizes a display method such as a thumbnail or a random access method based on the video data from the video data recording unit 18 based on various types of additional information data. This increases the degree of freedom in the user's viewing method, and displays video data with high convenience.

The control unit 20 controls the drive 21 to
Magnetic disk 22, optical disk 23, magneto-optical disk 2
Or the control program stored in the semiconductor memory 25 is read, and each unit of the video / audio processing device is controlled based on the read control program.

The video / audio processing apparatus detects a scene by performing a series of processes as schematically shown in FIG.

First, the video and audio processing apparatus performs video division in step S1, as shown in FIG. That is, the video and audio processing device divides the video data input to the video dividing unit 11 into either a video segment or an audio segment or, if possible, into both.

The video division method applied by the video / audio processing apparatus has no particular prerequisite. For example, the video and audio processing device is “G. Ahanger and TDC Little, A survey
oftechnologies for parsing and indexing digital v
ideo, J. of Visual Communication and Image Represe
ntation 7: 28-4, 1996 ". Video segmentation is performed in a manner well known in the art. Any video segmentation method shall be applicable.

Next, the video / audio processing apparatus performs step S2.
In, the feature amount is extracted. That is, the video / audio processing device includes the video feature amount extraction unit 13 and the audio feature amount extraction unit 14
, A feature quantity representing the feature of the segment is calculated. In the video / audio processing apparatus, for example, the time length of each segment, video feature amounts such as color histograms and texture features, audio feature amounts such as frequency analysis results, levels and pitches, activity measurement results, and the like are applicable feature amounts. Is calculated. Of course, the video and audio processing device is not limited to these as applicable feature amounts.

Subsequently, the video and audio processing apparatus performs step S
In 3, the similarity of the segments is measured using the feature values. That is, the video and audio processing apparatus performs the dissimilarity measurement by the feature amount similarity measurement unit 17 and measures the degree of similarity between the segment and the neighboring segments based on the measurement criterion. The video / audio processing device calculates the dissimilarity metric using the feature amount extracted in step S2.

Then, the video and audio processing apparatus proceeds to step S
At 4, it is determined whether the segment corresponds to a scene break. That is, using the dissimilarity metric calculated in the previous step S3 and the feature amount extracted in the previous step S2, the video and audio processing apparatus regards each segment as the current segment, It is determined whether the presence ratio is higher in the past or in the future with respect to the reference segment, the pattern of the change ratio of the presence ratio is examined, and it is determined whether or not the segment is a scene boundary. The video / audio processing device finally outputs whether or not each segment is a scene break.

Through such a series of processing, the video / audio processing apparatus can detect a scene from video data.

Therefore, by using the result, the user can summarize the contents of the video data and quickly access a point of interest in the video data.

Hereinafter, each step of the above processing will be described in more detail.

The video division in step S1 will be described. The video / audio processing apparatus divides the video data input to the video division unit 11 into either a video segment or an audio segment or, if possible, both, and automatically detects a segment boundary in the video data. As described above, there are many techniques for performing the above, and no special prerequisites are provided for this video division method in the video / audio processing apparatus.

On the other hand, in the video / audio processing apparatus, the accuracy of scene detection by the subsequent processing essentially depends on the accuracy of the underlying video division. It should be noted that the scene detection in the video / audio processing apparatus can tolerate an error at the time of video division to some extent. In particular, in the video / audio processing apparatus, video division is preferably performed when segment detection is excessively performed, rather than when segment detection is insufficient. In general, the video and audio processing apparatus can combine segments that are overdetected during scene detection as the same scene as long as the result of detection of similar segments is excessive.

The feature value extraction in step S2 will be described. The feature amount is an attribute of the segment that represents the feature of the segment and supplies data for measuring the similarity between different segments. In the video / audio processing apparatus, the video feature extraction unit 13 and the audio feature extraction unit 14 calculate the feature of each segment, and represent the feature of the segment.

Although the video / audio processing device does not depend on any specific details of the feature values, the feature values considered to be effective when used in the video / audio processing device include, for example, the following video feature values. There are such as quantity, audio feature quantity, and video / audio common feature quantity. A necessary condition of these feature amounts that can be applied to the video and audio processing device is that dissimilarity can be measured. In addition, the video and audio processing apparatus may simultaneously perform feature extraction and the above-described video division for efficiency. The feature values described below enable such processing.

As the above-mentioned feature amount, there is firstly one relating to a video. Hereinafter, this is referred to as an image feature amount. Since the video segment is composed of consecutive video frames, by extracting an appropriate video frame from the video segment, it is possible to characterize the depiction content of the video segment with the extracted video frame. That is, the similarity of video segments can be replaced with the similarity of appropriately extracted video frames. That is, the video feature is one of important features that can be used in the video and audio processing device. In this case, the video feature amount alone can represent only static information. However, the video and audio processing apparatus applies a method described later to obtain the dynamic feature of the video segment based on the video feature amount. Extract.

Although there are many known image feature amounts, it has been found that, for scene detection, the following color feature amounts (histogram) and image correlation give a good balance between calculation cost and accuracy. Therefore, the video and audio processing device uses the color feature amount and the video correlation as the video features.

In a video / audio processing apparatus, the color in a video is an important material when determining whether two videos are similar. Judging the similarity of images using a color histogram is described, for example, in “G. Ahanger and TDC.
Little, A survey of technologies for parsing and i
ndexing digital video, J. of Visual Communication
and Image Representation 7: 28-4, 1996 ”.

Here, the color histogram is, for example, LU
It is obtained by dividing a three-dimensional color space such as V or RGB into n regions and calculating the relative proportion of the appearance frequency of pixels in the video in each region. Then, an n-dimensional vector is provided from the obtained information. For compressed video data, a color histogram can be directly extracted from the compressed data, as described, for example, in US Pat. No. 5,708,767.

The video / audio processing apparatus obtains the original histogram vector in the YUV color space in the video (a generally used method such as MPEG1 / 2 or DV) constituting the segment.

In the video / audio processing apparatus, the original YUV color space of the video (MPEG1 / 2, DV, etc.) that constitutes a segment is sampled with 2 bits per color channel. 2 ^{2 3} = 64-dimensional histogram vector is obtained.

[0082] Such a histogram represents the overall color tone of the video, but does not include time information.
Therefore, the video / audio processing device calculates a video correlation as another video feature amount. In scene detection in the video and audio processing device, a structure in which a plurality of similar segments intersect with each other is a powerful indicator that the structure is a single scene structure.

For example, in a conversation scene, the position of the camera alternately moves between two speakers, but the camera usually returns to almost the same position when the same speaker is photographed again. In order to detect the structure in such a case, it has been found that the correlation based on the grayscale reduced video is a good indicator of the similarity of the segments. The image is thinned out and reduced to a grayscale image having a size of × N, and the image correlation is calculated using this. Here, a small value is sufficient for both M and N, for example, 8 × 8. That is, these reduced grayscale images are interpreted as MN-dimensional feature amount vectors.

Further, as a feature amount different from the above-mentioned video feature amount, there is a feature amount related to audio. Hereinafter, this feature will be referred to as a voice feature. The audio feature is a feature that can represent the content of an audio segment, and the video / audio processing device can use frequency analysis, pitch, level, and the like as the audio feature. These speech features are known from various documents.

First, the video and audio processing apparatus can determine the distribution of frequency information in a single audio frame by performing frequency analysis such as Fourier transform.
The video / audio processing apparatus, for example, expresses an FFT (Fast Four Fourier Display) to represent the distribution of frequency information over one audio segment.
ier Transform: Fast Fourier Transform component, frequency histogram, power spectrum, cepstrum (Cepstru)
m) and other feature values can be used.

The video / audio processing apparatus can also use pitches such as an average pitch and a maximum pitch, and audio levels such as an average loudness and a maximum loudness as effective audio feature amounts representing audio segments.

As another feature, there is a video / audio common feature. This is not a video feature amount nor an audio feature amount, but provides information useful for representing characteristics of a segment in a scene in a video / audio processing apparatus. The video / audio processing device uses the segment length and the activity as the video / audio common feature amount.

The video / audio processing apparatus can use the segment length as the video / audio common feature. This segment length is the length of time in the segment. Generally, a scene has rhythmic features that are unique to the scene. This rhythm feature appears as a change in segment length in the scene. For example, a short series of rapid segments represents a commercial. On the other hand, the segments in the conversation scene are longer than in the case of commercials, and the conversation scene is characterized in that mutually combined segments are similar to each other. The video and audio processing device
A segment length having such a feature can be used as a video / audio common feature amount.

The video / audio processing apparatus can use an activity as the video / audio common feature.
The activity is an index indicating how much the content of the segment feels dynamic or static. For example, if visually dynamic, the activity represents the degree to which the camera moves quickly along the object or the object being photographed changes rapidly.

This activity is calculated indirectly by measuring the average value of inter-frame dissimilarity of a feature quantity such as a color histogram. Here, if the dissimilarity metric for the feature value F measured between the frame i and the frame j is defined as dF (i, j), the video activity VF is defined as the following equation (1). You.

(Equation 1)

In the equation (1), b and f each represent 1
The frame numbers of the first and last frames in the segment. Specifically, the video / audio processing device uses, for example, the above-described histogram to generate the video activity VF.
Is calculated.

As described above, the feature values including the above-mentioned video feature values basically represent the static information of the segment. However, in order to accurately represent the feature of the segment, Needs to consider its dynamic information. Therefore, the video and audio processing apparatus represents the dynamic information by a feature amount sampling method as described below.

The video and audio processing apparatus extracts one or more static feature values from different points in one segment, for example, as shown in FIG. At this time, the video and audio processing apparatus determines the number of extracted feature amounts by balancing the maximization of the fidelity in the segment representation and the minimization of the data redundancy. For example, if one image in a segment can be specified as a key frame of the segment, the histogram calculated from the key frame is
This is the sampling feature to be extracted.

The video / audio processing apparatus determines which of the samples that can be extracted as features in the target segment is to be selected by using a sampling method described later.

Now, consider a case where a certain sample is always selected at a predetermined time, for example, the last time in a segment. In this case, for any two segments that change (fade) to a black frame, the samples have the same black frame, so that the same feature value may be obtained. That is, whatever the video content of these segments is, the selected two frames are determined to be extremely similar. Such a problem occurs because the sample is not a good representative value.

Therefore, the video / audio processing apparatus does not extract the characteristic amount at the fixed point as described above, but extracts a statistical representative value in the entire segment. Here, in the case of two general sampling methods of the feature amount,
That is, the first case in which the feature amount can be represented as a real n-dimensional vector and the second case in which only the dissimilarity metric can be used will be described. In the first case, the most well-known video feature amounts and audio feature amounts, such as a histogram and a power spectrum, are included.

In the first case, the number k of samples is determined in advance, and the video and audio processing apparatus uses “L. Kaufman
and PJ Rousseeuw, Finding Groups in Data: An Int
roduction to Cluster Analysis, John-Wiley and son
s, 1990 ”, using a well-known k-means-clustering method,
The feature amount of the entire segment is automatically divided into k different groups. And the video and audio processing device,
As a sample value, a centroid value of the group or a sample close to this centroid value is selected from each of the k groups. The complexity of this processing in a video and audio processing device only increases linearly with the number of samples.

[0098] On the other hand, in the second case, the video and audio processing apparatus is described in "L. Kaufman and PJ Rousseeuw, Finding.
Groups in Data: An Introduction to Cluster Analysi
s, John-Wiley and sons, 1990 ".
K-medoids algorithm method
Are used to form k groups. Then, the video and audio processing apparatus uses the medoid of the above-described group for each of the k groups as the sample value.

In the video / audio processing apparatus, the method of constructing the dissimilarity metric for the feature quantity representing the extracted dynamic feature is based on the dissimilarity metric of the static feature quantity on which the feature is based. , Which will be described later.

As described above, the video / audio processing apparatus can extract a plurality of static features and use the plurality of static features to represent a dynamic feature.

As described above, the video and audio processing apparatus can extract various feature amounts. Each of these features is generally insufficient by itself to represent the features of the segment. Thus, the video and audio processing device can select a set of feature amounts that complement each other by combining these various feature amounts. For example, the video and audio processing apparatus can obtain more information than the information of each feature amount by combining the above-described color histogram and video correlation.

Next, a description will be given of the similarity measurement of the segments using the feature amounts in step S3 of FIG. The video / audio processing apparatus uses a feature amount similarity measurement unit 1 using a dissimilarity metric that is a function for calculating a real value that measures how dissimilar the two feature amounts are.
7, the similarity of the segments is measured. In the dissimilarity metric, a small value indicates that the two feature amounts are similar, and a large value indicates dissimilarity. Here, a function for calculating the dissimilarity between the _two segments S ₁ and S ₂ related to the feature value F is defined as a dissimilarity metric dF (S ₁ , S ₂ ). This function needs to satisfy the relationship given by the following equation (2).

(Equation 2)

By the way, some of the dissimilarity metrics can be applied only to a specific feature, but “G.A.
hanger and TDC Little, A survey of technologies
forparsing and indexing digital video, J. of Visu
al Communication and Image Representation 7: 28-4,
1996 ”and“ L. Kaufman and PJ Rousseeuw, Finding G
roups in Data: An Introduction to Cluster Analysis,
In general, many dissimilarity metrics can be applied to measuring similarity for features represented as points in n-dimensional space, as described in John-Wiley and sons, 1990. It is.

The specific examples are Euclidean distance, inner product,
L1 distance or the like. Here, in particular, since the L1 distance effectively acts on various feature amounts including feature amounts such as histograms and video correlations, the video / audio processing apparatus uses the L1 distance.
Introduce the distance. Here, two n-dimensional vectors are A,
If B, the L1 distance dL1 (A, B) between A and B is given by the following equation (3).

(Equation 3) Here, the suffix i of A and B indicates the i-dimensional element of each of the n-dimensional vectors A and B.

As described above, the video and audio processing apparatus extracts static feature values at various points in the segment as feature values representing dynamic features. Then, the video and audio processing apparatus determines the similarity between the two extracted dynamic features by using the dissimilarity measurement between the underlying static features as its dissimilarity criterion. Use criteria. These dynamic feature dissimilarity metrics are often best determined using the dissimilarity values of the most similar static feature pair selected from each dynamic feature. is there. In this case, the two extracted dynamic features SF ₁ , S
Dissimilarity metric between F ₂ is defined by the following equation (4).

(Equation 4)

Here, the function dF (F
₁ , F ₂ ) indicates the dissimilarity metric for the static feature F on which it is based. In some cases, instead of taking the minimum value of the dissimilarity of the feature amount, the maximum value or the average value may be taken.

By the way, in the video / audio processing apparatus, a single feature amount is not enough to determine the similarity of segments, and it is necessary to combine information from a large number of feature amounts regarding the same segment. Often. As one of the methods, the video and audio processing apparatus calculates dissimilarity based on various feature amounts as a weighted combination of the respective feature amounts. That is, when there are k feature quantities F ₁ , F ₂ ,..., F _k , the video / audio processing apparatus sets the dissimilarity measurement criterion dF for the combined feature quantity as shown in the following equation (5). (S ₁ , S ₂ ) is used.

(Equation 5)

Here, {w _i } is a weighting coefficient that satisfies {iw _i = 1}.

As described above, the video and audio processing apparatus is configured as shown in FIG.
It is possible to calculate a dissimilarity metric using the feature amount extracted in step S2 of step (a) and measure the similarity between the segments.

Next, the division of the scene in step S4 of FIG. 5 will be described. The video / audio processing device detects a change in the distribution pattern of the closest similar segment to each segment using the dissimilarity metric and the extracted feature amount, and determines whether or not the segment is a scene break and outputs the result. I do.

The video / audio processing apparatus performs the following four processes when detecting a scene.

In the process (1), based on each segment, a certain number of the most similar segments in a certain time frame is detected.

In the processing of (1), after the processing of (2), the ratio of the number of similar segments existing in the past and future time zones with respect to the reference segment is calculated (actually, the number of similar segments existing in the future is calculated from the number of similar segments existing in the future). , Etc.), and the calculation result is used as a boundary measurement value.

In the processing of (1), the borderline measurement value obtained in the processing of (2) is examined with respect to time change with respect to each segment. Or a segment position indicating a continuous pattern is detected.

In the process (1), the absolute values of the measured boundary values are summed in the process (1), and this sum is referred to as a scene intensity value. If this scene intensity value exceeds a predetermined threshold, it is determined as a scene boundary.

These processes will be specifically described with reference to FIG. 6A, for example, as shown in FIG. 6A, a time frame of arbitrary k segments is set for each segment in the past, and a time frame of k segments is set for the future (for example, 5 frames in this case). N segments are detected in this time frame (here, four segments). Time progresses into the future as the number representing each segment increases. The darkly shaded segment 7 in the middle of the figure is a reference segment at a certain time, while the similar segment is a lighter shaded segment 4,
6, 9, and 10. Here, four similar segments are extracted, two in the past and two in the future.

In this process, the measured boundary value is
The calculation is performed by dividing (the number in the past) by (the number in the future) or subtracting (the number in the past) from (the number in the future). Here, the boundary value is calculated by the latter method. Here, representing each borderline measured and F _i. i
Is the position (number) of each segment. Now, when the calculation is performed by the latter method, the measured boundary value F _{6 in} FIG.

In the process (1), the calculation in the process (2) is performed along the time axis. FIG. 8B shows three segments 5, 8, and 9 in the past and one segment 11 in the future with respect to the segment 10 advanced by three segments from FIG.
Similar segments exist. The borderline measurements F ₁₀ = 1-3 = -2 in this case.

FIG. 11C shows a state in which the image has advanced one segment to reach just before the boundary of the scene, and similar segments 6, 7, 9, and 10 of segment 11 are all concentrated in the past. At this time, the measured boundary value is F ₁₁ = 0-4.
= -4.

Next, FIG. 11D shows a state advanced by one segment from FIG. 10C, immediately after entering a new scene beyond the boundary, and where the beginning of the scene is segment 12. is there. Similar segments are 13, 14, 15,
Sixteen. At this time, all the similar segments have changed to patterns that exist in the future. F12 = 4-0 = 4.

Finally, FIG. 11E shows the case of the segment 13 which is further advanced by one segment. Similarly,
A F ₁₃ = 3-1 = 2. In this method, when the ratio of similar segments is large in the past, a negative sign (minus sign) is used, and a positive sign (plus sign) indicates that the ratio is large in the future. The change of the measured boundary value F _i at this time shows a pattern such as 0... (−2) → (−4) → (+4) → (+2) (6).

The portion changing from (−4) to (+4) corresponds to the boundary of the scene. In the case where the similar segment is located in the middle of the scene as shown in FIG. 6A, the similar segments within the time frame exist almost equally in the past and the future with each segment interposed therebetween. However, as the position approaches the boundary of the scene, the ratio existing in the past becomes higher as shown in FIG. 2B, and the existing ratio in the past becomes 100% in FIG. 2C, as shown in FIG. Immediately after the boundary is exceeded, it indicates that there is a pattern in which the existence ratio changes to 100% in the future. By detecting such a pattern, almost 100
A change point that largely fluctuates from the% existence ratio to the future existence ratio of approximately 100% is associated with a scene break.

Even in a non-boundary region of a scene, a pattern having a high past ratio may temporarily change to a high future ratio (only for one segment). However, it is often not a scene boundary. This is because many of these temporary changes occur accidentally. After a number of similarity measures, such as non-boundary areas, that have a high prevalence in the past,
When a pattern in which a plurality of boundary measurement values having a large existence ratio continue in the future is detected, it is determined that the possibility of a scene boundary is high. If not, there is a high possibility that it is not a scene boundary, so it is not regarded as a scene boundary.

In the processing (1), after the processing (2), the measured values of the borderlines are summed up to calculate the “strength” of the scene boundary point. In order to measure the strength, the absolute value of the boundary measurement value is added. The degree of the change in the value corresponds to the degree of the visual change of the scene, and the degree of the visual change of the scene corresponds to the degree of the semantic change of the scene. Therefore, it is possible to detect a scene corresponding to the magnitude of the semantic degree of the scene by the magnitude of this value.

Here, the sum of the absolute values is defined as a scene intensity value Vi. In that definition, i represents the segment number. For example, four boundary measurement values (two segments in the past, one segment in the future and one segment in the future, and a total of four segments F _i-2 , F _i-1 , F _i , F of boundary measurement values of the segment) _{i + 1} ) using the sum of the absolute values.

The pattern of the change in the measured boundary value at the boundary of the scene is theoretically 100% similar segment in the past, such as F _i−1 → F _i −4 → + 4 as shown above. It is considered that a change that exists 100% in the future will occur from the case where.

As described above, at the boundary of a scene, a large change occurs between one segment. Then, like the pattern of Expression (6), the possibility that the pattern will change while the absolute value of the measured boundary value is large over four or more segments is not high except near the boundary of the scene. The characteristics of the change in this pattern, by determining only those of a certain size or greater scene intensity values V _i and the actual scene boundary, it is possible to perform scene detection to desired.

FIG. 7 is a graph showing a result of using video data of about 30 minutes obtained by recording an actual music program. The vertical axis represents the scene intensity value, and the horizontal axis represents each segment. The segment at the dark bar is the actual scene boundary (here, the top segment of the scene).
In the case of this result, if the scene intensity value is 12 or more as the boundary of the scene, the scene coincides with the actual scene with a probability of 6/7.

The process of generating the graph of FIG. 7 will be described with reference to FIG. What is described here is performed by the scene detection unit 16 shown in the video and audio processing apparatus. This processing performs the following processing every time a segment is generated.

In step S11, for each segment,
Within the ± k segment ranges centered on the segment, N nearest similar segments are detected by using the feature similarity measuring unit 17, and the number of those existing in the past and the number present in the future are determined. Ask.

In step S12, the number of similar segments existing in the future from the number of similar segments existing in the future among the N similar segments obtained in the processing in step S11 is set as the boundary characteristic measurement value F _i of each segment. Calculate and save the reduced number.

[0132] At step S13, 2n segments of borderline measurements F _in, ···, F i, to identify the high places from a change in the pattern of F _i + _n potential scene boundaries. n is the number of boundary measurements needed to see the pattern change of past ratio and future ratio from i segment.

Here, three conditions for a change pattern indicating a scene boundary are defined as follows. Borderline measure F _i to F _{i + n} be borderline measurements F _in to F _{i + n} is uniformly Borderline measurement is not a 0 value F _in to F _i-1 is less than or equal to 0 is 0 or more Being

Then, it is determined whether or not all of the above three conditions are satisfied. If it is determined that all three conditions are satisfied, it is determined that the possibility of a scene boundary is high,
Move to the next step S14. Otherwise, the process proceeds to step S16.

In step S14, step S1
3 is applied to the following equation to calculate the boundary value F
_in, ···, F _i, to calculate the scene intensity V _i from F _{i + n.} V _i = | F _in | +… + | F _i-1 | + | F _i | +… + | F
_{i + n} |

When a condition that exceeds the threshold value for the intensity value is set, and a scene intensity value that satisfies the condition appears, it is determined that the intensity of the visual change of the scene to be obtained is, and the video data being processed is determined. And outputs the position as one of the boundaries of the scene. When the condition regarding the intensity value is not required, the intensity value for each segment is used as the additional information data in the video data recording unit 18.
Output to and record.

The boundaries of the scene are detected by repeating the above processing. In the scene, a segment group included in the boundary is formed from the boundary.

As described above, the video / audio processing apparatus to which the present invention is applied is for extracting a scene structure. It has already been experimentally verified that a series of processes of the video and audio processing apparatus described above can extract a scene structure from video data of various contents such as a TV drama and a movie.

Note that the number of detected scene boundaries can be adjusted by arbitrarily changing the scene intensity value. Therefore, by adjusting the scene intensity value, it is possible to perform scene boundary detection that is better adapted to various contents.

Further, in the case of displaying a list of scenes of video data for a certain period of time, it is conceivable that the list display is made easier by limiting the number of scenes to be detected. In that case, there is a problem that which scenes should be included in the list display to make it easier to grasp the video data. For this purpose, the scenes to be used for the list display may be determined in accordance with the order of the obtained importance of each scene. In the present invention, it is possible to provide a scene intensity value, which is a measure of the importance of the obtained scene, and to change the scale (change the scene intensity threshold) to change the number of scenes. It is possible, and a convenient viewing expression can be performed according to the user's interest.

Further, when the number of scenes is changed, it is not necessary to perform the scene detection process again, and the stored intensity value time series can be easily processed only by changing the scene intensity threshold. is there.

As described above, the present invention has solved all the above-mentioned problems in the prior art.

First, in the video / audio processing apparatus, the user does not need to know the semantic structure of the video data in advance.

In the video / audio processing apparatus, the processing performed on each segment includes the following items. Extracting feature values Measuring dissimilarity between a pair of segments within a fixed number of time domains Extracting a fixed number of most similar segments using dissimilarity measurement results Calculate the borderline measurement Use the borderline measurement to determine the intensity value at the scene boundary point

Each of the processes has a small calculation load. Therefore, the present invention can be applied to home electronic devices such as set-top boxes, digital video recorders, and home servers.

As a result of detecting a scene, the video / audio processing apparatus can provide a new high-level access base for video browsing. Therefore, the video / audio processing device enables easy access to the video data based on the content by visualizing the content of the video data using a high-level video structure such as a scene instead of a segment. For example, by displaying a scene, the video and audio processing device allows the user to quickly know the gist of the program and quickly find a part of interest.

Further, the video / audio processing apparatus provides a basis for automatically creating an outline or summary of video data as a result of scene detection. In general, creating a coherent summary requires breaking down the video data into reconfigurable meaningful components, rather than combining random fragments from the video data. Scenes detected by the audiovisual processing device are the basis for creating such summaries.

The present invention is not limited to the above-described embodiments. For example, it is needless to say that the feature amounts used for measuring the similarity between segments may be other than those described above. Needless to say, other modifications can be made without departing from the spirit of the present invention.

Furthermore, according to the present invention, by arbitrarily changing the scene intensity value, a scene which is an important change point in the content structure can be obtained. This is because the intensity value can correspond to the degree of change in the content content. That is, when browsing a video, the number of detected scenes can be controlled by adjusting the scene intensity value threshold. In addition, the number of contents to be displayed can be increased or decreased according to the purpose.

That is, the so-called browsing granularity (g
ranularity) can be freely controlled according to the purpose. For example, when watching an hour of video, first set the intensity value high to show a short summary of scenes that are important to the content. Then, if interest increases and one wants to look in detail, lowering the intensity value can display a summary made up of more detailed scenes. Moreover, using the method of the present invention, unlike the prior art, there is no need to perform detection again each time the intensity value is adjusted, and it is sufficient to simply process the stored intensity value time series.

The following effects can be obtained when the present invention is implemented in household equipment such as a set-top box and a digital video recorder.

The first effect is that the number of segments to be examined can be fixed to a fixed number. The scene detection of the present invention can be realized by examining local changes of similar segments with respect to each segment, so that the amount of memory required for processing can be fixed, and a set-top box or digital Home equipment such as recorders can also be implemented.

The second effect is that the time required for processing each segment can be made constant. This is suitable for home devices such as set-top boxes and digital video recorders, which must complete a predetermined process within a predetermined time.

A third effect is that sequential processing for processing a new segment every time one processing is completed is possible. This means that when recording of a video signal or the like is completed in a household device such as a set-top box or a digital video recorder, it is possible to terminate the processing almost simultaneously with the end time. Also, even if the recording state is stopped for some reason, it is possible to keep the previous recording.

By the way, the above-described series of processing can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software can execute various functions by installing a computer built into dedicated hardware or installing various programs. It is installed from a recording medium into a possible general-purpose personal computer or the like.

As shown in FIG. 3, the recording medium is a magnetic disk 22 (including a floppy disk) on which the program is recorded and an optical disk 23 which are distributed separately from the computer to provide the program to the user.
(CD-ROM (Compact Disc-Read Only Memory), DVD (Digit
al Versatile Disc), magneto-optical disc 24 (M
D (including Mini Disc)), or a packaged medium including a semiconductor memory 25 and the like, and a ROM or a hard disk in which a program is recorded and provided to a user in a state where the program is incorporated in a computer in advance. It consists of.

[0157] In this specification, the step of describing a program recorded on a recording medium is not limited to processing performed in chronological order according to the described order. Alternatively, it also includes individually executed processing.

In this specification, the system is
It represents the entire device composed of a plurality of devices.

[0159]

As described above, according to the AV signal processing apparatus, method, and program of the present invention, a measurement criterion for measuring the similarity of the characteristic amount between a reference segment and another segment is calculated. Use the metric to measure the similarity between the reference segment and other segments, measure and use the similarity to calculate a measurement that indicates that the reference segment may be a scene boundary As a result, the boundary between scenes can be detected.

[Brief description of the drawings]

FIG. 1 is a diagram showing a hierarchical model of video data.

FIG. 2 is a diagram for explaining a boundary region and a non-boundary region of a scene.

FIG. 3 is a block diagram illustrating a configuration example of a video / audio processing device according to an embodiment of the present invention;

FIG. 4 is a diagram for explaining a boundary region of a scene.

FIG. 5 is a flowchart illustrating an operation of the video and audio processing device.

FIG. 6 is a diagram illustrating an example of a distribution pattern of similar segments.

FIG. 7 is a diagram showing a scene detection result.

FIG. 8 is a flowchart illustrating a process of a scene detection unit 16;

[Explanation of symbols]

Reference Signs List 11 video division unit, 12 video segment memory, 13 video feature extraction unit, 14 audio feature extraction unit, 15 segment feature memory, 16 scene detection unit, 17 feature similarity measurement unit, 18 video data recording unit, 19 Video display unit, 20 control unit,
21 Driver, 22 Magnetic Disk, 23 Optical Disk, 24 Magneto-Optical Disk, 25 Semiconductor Memory

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C053 FA14 GB09 HA29 LA06 LA11 5D015 FF06 5L096 AA02 CA04 FA23 FA37 HA01 JA03 JA11

Claims (13)

[Claims] 1. An AV signal processing apparatus for detecting and analyzing a pattern that reflects the semantic structure of the content of a supplied AV signal and detecting a scene that is a meaningful delimiter. Feature value extraction means for extracting a feature value of a segment formed by a frame; calculation means for calculating a metric for measuring the similarity of the feature value between a reference segment and another segment; and the measurement Using a criterion, a similarity measurement unit that measures the similarity between the reference segment and the other segment, and using the similarity measured by the similarity measurement unit, the reference segment is Measurement value calculation means for calculating a measurement value indicating a possibility of being a boundary of a scene; and analyzing a change in a temporal pattern of the measurement value calculated by the measurement value calculation means. , AV signal processing apparatus characterized by segments of said reference on the basis of the analysis result and a determining boundary determination means for determining whether or not a boundary of the scene. 2. The AV signal processing device according to claim 1, wherein the AV signal includes at least one of a video signal and an audio signal. 3. The AV signal processing apparatus according to claim 1, further comprising an intensity value calculating means for calculating an intensity value indicating a degree of change of said measured value corresponding to said reference segment. 4. The measurement value calculation means obtains a similar segment in a predetermined time domain with respect to the reference segment, analyzes a time distribution of the similar segment, and determines a ratio between the past and the future. 2. The AV signal processing device according to claim 1, wherein the measurement value is calculated by numerical conversion. 5. The apparatus according to claim 1, wherein the boundary determination unit determines whether the reference segment is a boundary of the scene based on a sum of absolute values of the measurement values. An AV signal processing apparatus as described in the above. 6. The apparatus according to claim 2, further comprising an audio segment generating means for detecting a shot as a basic unit of the video segment and generating the audio segment when the AV signal includes a video signal. A described in
V signal processing device. 7. When the audio signal is included in the AV signal, the audio signal further includes an audio segment generating unit that generates an audio segment using at least one of the feature amount and the silent section of the audio signal. The AV signal processing device according to claim 2, wherein 8. The AV signal processing apparatus according to claim 2, wherein the feature amount of the video signal includes at least a color histogram. 9. The AV signal processing device according to claim 2, wherein the feature amount of the audio signal includes at least one of a volume and a spectrum. 10. The apparatus according to claim 1, wherein the boundary determining unit determines whether the reference segment is a boundary of the scene by comparing the measured value with a preset threshold value. 2. The AV signal processing device according to 1. 11. An AV signal processing method of an AV signal processing apparatus for detecting and analyzing a pattern that reflects a semantic structure of the content of a supplied AV signal and detecting a scene that is a meaningful break, A feature value extraction step of extracting a feature value of a segment formed by a series of frames constituting the above, and a calculation for calculating a measurement reference for measuring the similarity of the feature value between a reference segment and another segment And a similarity measurement step of measuring the similarity between the reference segment and the other segment using the measurement criterion, using the similarity measured in the process of the similarity measurement step. A measurement value calculating step of calculating a measurement value indicating that the reference segment may be a boundary of the scene; Analyzing a change in the temporal pattern of the measurement value calculated in the processing of the map, and determining whether or not the reference segment is a boundary of the scene based on the analysis result. An AV signal processing method comprising: 12. A computer that detects and analyzes a pattern that reflects the semantic structure of the content of the supplied AV signal and detects a scene that is a meaningful delimiter, comprising a series of frames constituting the AV signal. A feature value extraction step of extracting a feature value of a segment to be performed; a calculation step of calculating a metric for measuring the similarity of the feature value between a reference segment and another segment; and using the metric. A similarity measurement step of measuring the similarity between the reference segment and the other segment; and using the similarity measured in the similarity measurement step, the reference segment is determined by the A measurement value calculation step of calculating a measurement value indicating the possibility of being a scene boundary; and the measurement value calculated in the processing of the measurement value calculation step. Analyzes the changes in the temporal pattern of values, analysis program segment serving as the reference, based on the result of executing the boundary determination step of determining whether a boundary of the scene. 13. An AV signal processing program for detecting and analyzing a pattern that reflects the semantic structure of the content of a supplied AV signal and detecting a scene that is a meaningful delimiter. A feature value extraction step of extracting a feature value of a segment formed by a series of frames constituting the frame, and a calculation step of calculating a metric for measuring the similarity of the feature value between a reference segment and another segment And, using the measurement criterion, a similarity measurement step of measuring the similarity between the reference segment and the other segment, using the similarity measured in the process of the similarity measurement step, A measurement value calculating step of calculating a measurement value indicating a possibility that the reference segment is a boundary of the scene; Analyzing a change in the temporal pattern of the measurement values calculated in the above, and a boundary determination step of determining whether the reference segment is a boundary of the scene based on the analysis result. Recording medium on which a computer-readable program to be executed is recorded.