JP2010033532A - Electronic apparatus, motion vector detection method and program thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent false detection of a motion vector as much as possible. <P>SOLUTION: A video characteristic detection part 4 of a recording/reproducing device 100 sequentially stores input thumbnail images in each 2/4/8/16-field interval, detects the motion vector of each block by block matching processing between a referred field and a reference field having each field interval, discriminates validity of the detected motion vector based on a discriminant function, normalizes each motion vector in accordance with the motion vector in the 2-field interval, combines the normalized valid motion vectors for each block to generate a composite motion vector. The composite motion vector is used for camera characteristic detection processing. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an electronic device capable of detecting a motion vector from video content, a motion vector detection method in the electronic device, and a program thereof.

  In recent years, the amount of video content stored in electronic devices such as HDD (Hard Disk Drive) / BD (Blu-ray Disc) recorders is increasing due to the increase in recording media capacity and content diversification. Yes. As a method for allowing users to view such a large amount of video content efficiently, a digest (highlight) scene is extracted from the video content and played back, or a chapter is added to the video content and played back for each chapter. Has been done. The digest scene extraction and chapter generation are performed by detecting, for example, panning, tilting, zooming, and the like, the motion characteristics of the camera that captured the video content (hereinafter referred to as camera characteristics), and using the characteristics based on the characteristics. This is done by extracting a section.

  This camera feature can be detected, for example, by detecting a motion vector in the video content. For example, in Patent Document 1, a block matching process is performed by calculating a sum of absolute differences between a reference block cut out from a current frame and a search block cut out from a reference frame one frame after the current frame. A technique for detecting a motion vector is disclosed.

JP 2003-299040 A (FIG. 3 etc.)

  However, when a motion vector is detected between frames (fields), for example, if there is a fast motion in the video content, the amount of motion exceeds the search range, and the detection process cannot be followed. In some cases, it could not be detected. The erroneous motion vector is used for the camera feature detection process, and thus the camera feature is erroneously detected.

  In view of the circumstances as described above, an object of the present invention is to provide an electronic device, a motion vector detection method, and a program thereof that can prevent erroneous detection of a motion vector as much as possible.

In order to solve the above-described problem, an electronic device according to an embodiment of the present invention includes an input unit, a detection unit, a storage unit, a determination unit, a normalization unit, and a generation unit.
The input means includes a first image data, a second image data having a first time length between the first image data, and a plurality of pieces of image data constituting the video data; Third image data having a second time length longer than the first time length is input to each first block with respect to the first image data.
The detection means includes a first motion vector between a predetermined reference block in the first image data and a first reference block in the second image data, the reference block, and the third block. The second motion vector between the second reference block in the image data is detected.
The storage means stores determination data for determining whether or not the detected first and second motion vectors are valid.
The discriminating unit discriminates for each block whether or not the detected first and second motion vectors are valid using the stored discrimination data.
The normalizing means normalizes the detected first and second motion vectors for each block with respect to a time length.
The generating means generates a combined motion vector by combining the first and second motion vectors for each of the blocks determined to be valid and normalized.
Here, the electronic device is an electrical appliance such as a recording / reproducing device such as an HDD / DVD / BD recorder, a television device, a PC (Personal Computer), a game device, a mobile phone, or a portable audio / visual device. The first time length is, for example, 2 field times (or 2 frame times, the same applies hereinafter), 4 field times, 8 field times, etc., and the second time length is, for example, 4 field times, 8 field times, 16 field times. However, the present invention is not limited to these. In addition, the above configuration does not limit the case where two types of motion vectors are detected and synthesized at two different time lengths, and three or more types of motion vectors are detected at three or more different time lengths. It is a concept that includes the case of synthesis. The input means may be one piece of hardware to which the first to third image data are inputted, or a plurality of different pieces of hardware to which the first to third image data are inputted respectively. It may be. The predetermined block is an area obtained by dividing each image data into a matrix of 8 × 8, for example.
With this configuration, the validity of the first and second motion vectors detected between the first image data and the second and third image data is determined, and each motion vector is normalized. Are combined to generate a combined motion vector. Therefore, since the electronic device can generate a combined motion vector using only valid motion vectors detected between fields (frames) having different time lengths, it is possible to improve the detection accuracy of motion vectors in video data. . That is, since fast motion and slow motion in video data are appropriately detected between different time length fields (frames) and the validity thereof is determined, erroneous detection can be minimized. This synthesized motion vector can be used for, for example, determination of camera characteristics in video data such as pan, tilt, and zoom, and object coding of video data.

The normalization unit may normalize the first motion vector in accordance with the second time length.
In this case, the first motion vector is normalized according to the ratio of the first time length and the second time length while the second motion vector is left as it is. Thereby, the dynamic range of the detected first motion vector can be expanded and the resolution can be improved. In addition, the second motion vector can be normalized without degrading the resolution of the second motion vector as compared with the case where the second motion vector is normalized in accordance with the first time length. Therefore, it is possible to improve the accuracy of the combined motion vector generated based on each motion vector.

The generating means determines the synthesized motion vector using the first motion vector when it is determined that both the first motion vector and the second motion vector of the corresponding block are valid. It may be generated.
Here, the first motion vector is used instead of the second motion vector because the longer the motion vector detection time length, the faster the motion in the video data due to the greater amount of motion. This is because there is a high possibility that the user cannot follow the movement and falls out of the search range. That is, the electronic device considers that the first motion vector is more reliable than the second motion vector in detecting fast motion, and that both the first and second motion vectors are valid. If it is determined, the first motion vector is adopted. Thereby, the detection accuracy of particularly fast motion can be enhanced.

The storage means stores a plurality of learning video data and a plurality of learning motion vector data respectively corresponding to the first and second time lengths to be detected from the plurality of learning video data. May be.
In this case, the electronic device detects a motion vector between the reference block and the reference block for each of the first and second time lengths from the stored plurality of learning video data, and performs the detection. Learning means for learning whether or not the detected motion vector is valid using the stored motion vector and the stored learning motion vector data, and generating the discrimination data. Good.
Accordingly, the discrimination data can be generated by learning whether or not the detected motion vector is valid using the learning video data.

The learning means includes a motion vector detected from the learning video data, a residual signal between the base block and the reference block, an average value and a variance value of pixel data of the base block, and the reference block The discriminant function as the discriminant data may be generated by performing discriminant analysis using the average value and the variance value of the pixel data.
In this configuration, in addition to the motion vector detected from the learning video data and the residual signal between the standard block and the reference block, each average value and each variance value of the standard block and the reference block are used as the basis. Generate a discriminant function. Accordingly, since the validity of the first and second motion vectors can be discriminated learning by the discriminant function generated based on the learning video data, the validity can be more accurately performed. it can.

  In this case, the discriminant function related to the first motion vector detected in the first time length and the discriminant function related to the second motion vector detected in the second time length may be generated separately. For example, the first motion vector and the second motion vector may have different motion speeds even if the motion amounts are the same. In this case, a difference occurs in the residual signal, each average value, and each variance value. However, if a common discriminant function is used for the first and second motion vectors, this difference in motion speed is used as a discrimination result. It may not be reflected. However, by generating the discriminant function separately as described above, the validity of the motion vector can be discriminated with higher accuracy reflecting the difference in the motion speed of the video.

When the detected first motion vector is determined to be valid, the detection means uses the first reference block when the first motion vector is detected as the reference block and uses the second reference block as the reference block. A third motion vector may be detected from the reference block, and the detected second motion vector may be detected by adding the detected third motion vector and the first motion vector.
As a result, the second reference block can be detected more quickly because the distance is shorter than when the third motion vector is detected between the base block and the second reference block in the first image data. And detection efficiency can be improved.

The electronic apparatus may further include a determination unit that determines camera characteristics in the video data generated by the camera operation based on the generated combined motion vector.
Accordingly, the camera feature in the video data can be determined with higher accuracy by using the synthesized motion vector with higher accuracy. Here, the camera features are features such as pan, tilt, and zoom.

The determination unit continues the determination of the camera feature for a first determination time when the combined motion vector is a combination of the first motion vector for each block, and the combined motion vector is When the second motion vector for each block is synthesized, the camera feature determination may be continued for a second determination time longer than the first determination time.
With this configuration, the detection accuracy of the camera feature can be improved by detecting the camera feature by varying the determination time according to the time length in which the motion vector that is the basis of the combined motion vector is detected. That is, since the first motion vector has a detection time shorter than that of the second motion vector, it is considered that the first motion vector is a motion vector detected from a relatively fast motion image. This is considered to be a motion vector detected from a video with slow target movement. Therefore, since the camera feature can be detected from the fast motion video in a short time, the determination is continued for the first judgment time, and the camera feature cannot be detected from the slow motion video in a short time, so the second judgment time. Continue judgment. This can reduce erroneous detection of camera features.

Further, the determining means may synthesize any one of the first and second motion vectors when the combined motion vector is a combination of the first and second motion vectors for each block. The determination of the camera characteristics may be continued for the first determination time or the second determination time depending on whether the determination is made.
Thereby, even when the first and second motion vectors for each block are mixed in the synthesized motion vector, the determination time can be set adaptively depending on which motion vector is larger.

The electronic apparatus may further include first learning video data generation means.
The first learning video data generation means moves the rectangular area within the predetermined learning image data by the first and second time lengths, and moves the image data in the rectangular area to the first and second image data, respectively. By extracting the second time length, the learning video data necessary for learning whether or not each movement detected as the first and second motion vectors is appropriate is generated.
Thus, the electronic device artificially creates learning video data for learning the validity of the motion vector as the movement in the vertical and horizontal directions by moving the rectangular area in the learning image data. be able to. Therefore, it is possible to efficiently collect learning video data with less variation in characteristics compared to a case where learning video data for learning movement as a motion vector is manually created. In other words, the electronic device can reduce the labor of the collection and can improve the accuracy of the learning process.

The electronic apparatus may further include second learning video data generation means.
The second learning video data generation means is configured to display an image in a first rectangular area having a first area within a predetermined learning image data with a second area different from the first area. Whether each of the enlargements or reductions detected as the first and second motion vectors is appropriate by extracting and changing the images in the second rectangular area having the first and second time lengths, respectively. The learning video data necessary for learning whether or not is generated.
Thereby, the electronic device gradually changes the image in the first rectangular area to the image in the second rectangular area, thereby learning the validity of the motion vector as the enlargement or reduction. Data can be created in a pseudo manner. Therefore, it is possible to efficiently collect learning video data with less variation in characteristics compared to the case of manually creating learning video data for learning expansion or reduction as a motion vector. In other words, the electronic device can reduce the labor of the collection and can improve the accuracy of the learning process.

The first or second learning video data generating means is a residual signal for each block between the first sample image data and the second sample image data that are consecutive in predetermined sample video data. And the second sample image data may be extracted as the learning image data when the sum is equal to or greater than a predetermined threshold.
The second sample image data when the sum of the residual signals is equal to or greater than a predetermined threshold is a so-called cut point in the sample video data. Thereby, the electronic device can extract various image data with different scenes as the learning image data from one sample video data, and can further reduce the labor of collecting appropriate learning image data. it can.

In this case, the first or second learning video data generation means may extract the learning image data using the video data from which the first and second motion vectors are detected as the sample video data. Good.
Thereby, the electronic device uses the video data from which the first and second motion vectors are actually detected as the sample video data, thereby detecting a motion vector closer to the first and second motion vectors. Learning video data can be generated. Therefore, the electronic device can more reliably determine the validity of the first and second motion vectors. For example, when an electronic device creates a digest video based on a motion vector detected from the video data, when the electronic device is instructed to create the digest video, the electronic device first uses the video data for learning. And the learning process may be executed.

A motion vector detection method according to another aspect of the present invention has a first time length between first image data and the first image data among a plurality of image data constituting video data. Inputting, for each predetermined block, third image data having a second time length longer than the first time length between the second image data and the first image data. .
A first motion vector is detected between a predetermined reference block in the input first image data and a first reference block in the input second image data.
A second motion vector is detected between the reference block and the second reference block in the input third image data.
Determination data for determining whether or not the detected first and second motion vectors are valid is stored.
Using the stored discrimination data, it is discriminated for each block whether or not the detected first and second motion vectors are valid.
The detected first and second motion vectors are normalized for each block with respect to time length.
The first and second motion vectors for each of the blocks determined to be valid and normalized are combined to generate a combined motion vector.
With this configuration, since a combined motion vector can be generated using only valid motion vectors detected between fields having different time lengths, it is possible to improve the detection accuracy of motion vectors in video data.

In the motion vector detection method, when the composite motion vector is a composite of the first motion vector for each block, the camera feature in the video data generated by the camera operation is the composite motion vector. Based on the above, the determination is continued for the first determination time.
Further, when the synthesized motion vector is a synthesized second motion vector for each block, the camera feature is longer than the first determination time based on the synthesized motion vector. The determination is continued for 2 determination times.
With this configuration, the detection accuracy of the camera feature can be improved by detecting the camera feature by varying the determination time according to the time length in which the motion vector that is the basis of the combined motion vector is detected.

A program according to still another aspect of the present invention causes an electronic device to execute an input step, a detection step, a storage step, a determination step, a normalization step, and a generation step.
The input step includes: first image data out of a plurality of image data constituting video data; second image data having a first time length between the first image data; Third image data having a second time length longer than the first time length is input to each predetermined block between the first image data and the first image data.
The detecting step includes a first motion vector between a predetermined reference block in the first image data and a first reference block in the second image data, the reference block, and the third block. The second motion vector between the second reference block in the image data is detected.
The storing step stores determination data for determining whether or not the detected first and second motion vectors are valid.
The discrimination step discriminates for each block whether or not the detected first and second motion vectors are valid using the stored discrimination data.
The normalizing step normalizes the detected first and second motion vectors for each block with respect to a time length.
The generating step generates a combined motion vector by combining the first or second motion vector for each block that has been determined to be valid and normalized.

The program may further cause the electronic device to execute a determination step.
In the determination step, when the combined motion vector is a combination of the first motion vector for each block, the camera feature in the video data generated by the camera operation is determined based on the combined motion vector. The determination is continued for the first determination time.
In the determination step, when the combined motion vector is a combination of the second motion vector for each block, the camera feature is determined based on the combined motion vector from the first determination time. The determination is continued for a long second determination time.

  As described above, according to the present invention, erroneous detection of motion vectors can be prevented as much as possible.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Hardware configuration of recording / reproducing apparatus]
FIG. 1 is a diagram showing a configuration of a recording / reproducing apparatus 100 according to an embodiment of the present invention.
As shown in FIG. 1, a recording / reproducing apparatus 100 includes a CPU (Central Processing Unit) 1, a RAM (Random Access Memory) 2, an operation input unit 3, a video feature detection unit 4, a digital tuner 5, an IEEE1394 interface 6, an Ethernet ( (Registered trademark) / wireless LAN (Local Area Network) interface 7, USB (Universal Serial Bus) interface 8, memory card interface 9, HDD 10, optical disk drive 11, buffer controller 13, selector 14, demultiplexer 15, AV (Audio / Video) ) Decoder 16, OSD (On Screen Display) 17, video D / A (Digital / Analog) converter 18 and audio D / A converter 19.

  The CPU 1 appropriately accesses the RAM 2 or the like as necessary, and controls the entire blocks of the recording / reproducing apparatus 100. The RAM 2 is a memory that is used as a work area of the CPU 1 and temporarily stores an OS (Operating System), a program, processing data, and the like.

  The operation input unit 3 includes a button, a switch, a key, a touch panel, a light receiving unit for an infrared signal transmitted from a remote controller (not shown), and the like. Output to.

  Under the control of the CPU 1, the digital tuner 5 receives a broadcast signal of a digital broadcast program through an antenna (not shown), and selects and demodulates a broadcast signal of a specific channel. This broadcast signal is output to the demultiplexer 15 via the selector 14 for reproduction, or recorded on the HDD 10 via the buffer controller 13 or recorded on the optical disk 12 inserted in the optical disk drive 11.

  The IEEE1394 interface 6 can be connected to an external device such as a digital video camera. For example, video content shot and recorded by a digital video camera is played back or recorded on the HDD 10 or the optical disc 12 in the same manner as the video content of a broadcast program received by the digital tuner 5.

  The Ethernet (registered trademark) / wireless LAN interface 7 inputs, for example, video content recorded on a PC or other recording / playback apparatus via Ethernet (registered trademark) or wireless LAN. This video content can also be played back and recorded on the HDD 10 or the optical disc 12.

  The USB interface 8 inputs video content from a device such as a digital camera or an external storage device such as a so-called USB memory via the USB. This video content can also be played back and recorded on the HDD 10 or the optical disc 12.

  The memory card interface 9 is connected to, for example, a memory card with a built-in flash memory, and inputs video content recorded on the memory card. This video content can also be played back and recorded on the HDD 10 or the optical disc 12.

  The HDD 10 records various video contents received as a broadcast signal or input from an external device, various programs, data, and the like on a built-in hard disk, and reads them from the hard disk and outputs them to the buffer controller 13 during reproduction. In addition, the HDD 10 is a program for executing motion vector and camera feature detection processing from the OS and video content described later, a program and data for executing motion vector learning processing and discriminant analysis, and various other programs and data. Etc. are also stored.

  Note that the recording / reproducing apparatus 100 may store the OS, various programs, and data in another recording medium such as a flash memory (not shown) instead of the HDD 10. These programs may be recorded on a recording medium such as an optical disk 12 or a memory card, and installed in the recording / reproducing apparatus 100 by the optical disk drive 11.

  The optical disk drive 11 records the video content and the like on the optical disk 12, reads them during reproduction, and outputs them to the buffer controller 13. The optical disk 12 is, for example, a DVD, a BD, a CD, or the like.

  For example, the buffer controller 13 controls the timing and amount of data written to the HDD 10 or the optical disk 12 of video content continuously supplied from the digital tuner 5 and other various interfaces, and writes the video content intermittently. . Further, the buffer controller 13 controls the read timing and data amount of the video content recorded on the HDD 10 and the optical disk 12 and continuously supplies the video content read intermittently to the demultiplexer 15.

  The selector 14 selects video content input from any of the digital tuner 5, various interfaces, the HDD 10, and the optical disk drive 11 based on a control signal from the CPU 1.

  The demultiplexer 15 separates the multiplexed video content input from the buffer controller 13 into a video signal and an audio signal, and outputs them to the AV decoder 16.

  The AV decoder 16 decodes a video signal and an audio signal encoded in a format such as MPEG (Moving Picture Expert Group) -2 or MPEG-4, for example, and converts the video signal to the OSD 17 and the audio signal to the D / A. Output to the converter 19.

  The OSD 17 generates graphics or the like to be displayed on a display (not shown), performs synthesis processing and switching processing with the video signal, and outputs the processed video signal to the video D / A converter 18. The video D / A converter 18 converts the video signal subjected to graphic processing by the OSD 17 into an NTSC (National Television Standards Committee) signal by D / A conversion, and outputs and displays it on a display (not shown).

  The audio D / A converter 19 D / A converts the audio signal input from the AV decoder 16 and outputs it to a speaker (not shown) for reproduction.

  The video feature detection unit 4 detects video features from the video signal before decoding by the AV decoder 16 or the video signal after decoding.

[Description of video features]
2 and 3 are diagrams illustrating the video feature.
FIG. 2 (A) shows an image captured by moving (panning) the camera leftward or rightward as the process proceeds to scenes S1 to S6. FIG. 2B shows an image captured by zooming in (zooming in) the camera as it proceeds to scenes S1 to S6. In the present embodiment, video features generated by camera work such as pan, tilt (not shown), and zoom are referred to as camera features.

  FIG. 3A shows a state where the scene is switched at a cut point fa between the scenes S3 and S4. FIG. 3B shows that one scene gradually fades out as it proceeds to scenes S1 to S3, and another scene gradually fades in as it proceeds to scenes S4 to S6 instead. Yes. In the present embodiment, video features generated by video editing work, such as video effects such as cut and fade, are referred to as video editing features.

  The video feature detection unit 4 detects such camera features and video editing features by a common signal processing system to be described later. The recording / reproducing apparatus 100 performs processing such as highlight scene (digest scene) generation and chapter generation using the video feature.

[Configuration of video feature detector]
FIG. 4 is a block diagram showing a specific configuration of the video feature detection unit 4. Each block described below may be realized as hardware or software.
As shown in the figure, the video feature detection unit 4 includes a block processing unit 21, an inter-field memory unit 22, an inter-field memory unit 23, an inter-field memory unit 24, an inter-field memory unit 25, and these memory units. Each matching processing unit 26, 28, 30 and 32, fade / cut processing units 27, 29, 31 and 33, a motion vector processing unit 34, a camera feature determination unit 36 and a fade / cut determination unit 35 are included.

  The block processing unit 21 inputs image data of video content in the order of field numbers. The image data input here is, for example, image data constituting a thumbnail video generated by reducing the image size from the original video content. The size of the thumbnail image data is, for example, 96x128, but is not limited thereto. Thereby, the load of the video feature detection process can be suppressed. Then, based on the image data, the block processing unit 21 performs processing such as setting a search region that is a target for motion vector detection and extraction of a block that is a reference for block matching processing (hereinafter referred to as a reference block). .

  The thumbnail image data input to the block processing unit 21 is, for example, baseband image data (specifically, a luminance signal Y, each field of the thumbnail video content decoded by the AV decoder 16). Color difference signals Cb and Cr). As video contents to be played back by the recording / playback apparatus 100, various types of data such as MPEG, digital recording DV (Digital Video), analog recording VHS (Video Home System), and 8 mm are mixed. Can be assumed. Therefore, the recording / reproducing apparatus 100 can perform processing of extracting video features from these video contents by using a common signal processing system as much as possible by performing processing in the baseband.

  The 2/4/8/16 inter-field memory units 22 to 25 have time lengths of 2 fields, 4 fields, 8 fields, and 16 fields respectively from the field from which the reference block is extracted (hereinafter referred to as a reference field). Each image data up to each field is stored. Of course, the time length between fields is not limited to these. The 2/4/8/16 inter-field memory units 22 to 25 may be provided as separate memory elements for each field, or may be provided as separate storage areas in a single memory element.

  The matching processing units 26, 28, 30 and 32 respectively search for the reference field input from the block processing unit 21 and each field (hereinafter referred to as a reference field) input from the inter-field memory units 22-25. Block matching processing is performed between regions. The result of the block matching process is output to the motion vector processing unit 34. Each matching processing unit 26, 28, 30, and 32 moves a block having the same shape as the standard block of the standard field (hereinafter referred to as a reference block) within the reference field, while similar to the standard block and the reference block. Search for the position with the maximum degree. Then, each of the matching processing units 26, 28, 30 and 32 performs motion vector amounts from the reference position to the searched position (that is, movement amounts in the x direction (horizontal direction) and the y direction (vertical direction) and (Movement direction) is output to the motion vector processing unit 34. The matching processing units 26, 28, 30 and 32 output Y, Cb and Cr residual signals between the base block and the reference block to the fade / cut processing units 27, 29, 31 and 33, respectively. To do. Details of these processes will also be described later.

  Fade / cut processing units 27, 29, 31 and 33 generate fade / cut evaluation values based on the residual signals after matching input from the matching processing units 26, 28, 30 and 32, respectively. Output to the fade / cut determination unit 35. Details of this processing will also be described later.

  The fade / cut processing units 27, 29, 31 and 33 are the reference blocks input from the block processing unit 21 and the references used for the block matching processing input from the inter-field memory units 22 to 25. The residual with the block may be calculated independently.

  The motion vector processing unit 34 determines whether or not the motion vector as a result of the block matching process input from each of the matching processing units 26, 28, 30 and 32 is valid (valid), and determines an effective motion vector. Based on this, a combined motion vector is generated. The combined motion vector is output to the camera feature determination unit 36. Details of this processing will also be described later.

  Based on the combined motion vector input from the motion vector processing unit 34, the camera feature determination unit 36 determines each camera feature in the video content by multiple regression analysis using an affine transformation model to be described later. Is output to CPU1. Details of this processing will also be described later.

  The fade / cut determination unit 35 determines each video editing feature of fade or cut in the video content based on the fade / cut evaluation values input from the respective fade / cut processing units 27, 29, 31 and 33. , Output to CPU1.

[Operation of recording and playback device]
Next, the operation of the recording / reproducing apparatus 100 configured as described above will be described. The operations described below are all executed under the control of the CPU 1 of the recording / reproducing apparatus 100 regardless of the operation subject. Each of these operations may be a hardware operation, or may be a software (program) operation cooperating with the hardware.

  The recording / reproducing apparatus 100 in the present embodiment detects two video features, a camera feature and a video editing feature. Of these, a motion vector is used to detect camera features. At that time, as described above, the motion vector processing unit 34 determines whether or not the motion vectors detected by the matching processing units 26, 28, 30 and 32 are valid. A discrimination function is used to determine the validity of the motion vector. The discriminant function generation process will be described below.

(Discriminant function generation process)
FIG. 5 is a flowchart showing a flow of discriminant function generation processing.
As shown in the figure, first, the CPU 1 of the recording / reproducing apparatus 100 inputs various learning contents stored in a storage medium such as the HDD 10 to the video feature detection unit 4 (step 51). Here, the learning content includes video content that can detect motion vectors well and video content that cannot detect motion vectors well.

  The video content that can detect a motion vector well is video content that easily detects a motion feature amount when a motion vector is detected from a subject. Specifically, the motion feature amount is, for example, feature data such as a residual between the standard block and the reference block, an average value and a variance value of the standard block, an average value and a variance value of the reference block (hereinafter referred to as a discrimination parameter). Called). Examples of such video content include video content in which a subject such as a building or a person with a sharp edge is photographed.

  Video content in which a motion vector cannot be detected well is video content in which it is difficult to detect the motion feature amount (discrimination parameter) when a motion vector is detected from a subject. As such video content, for example, video content without blurring of the subject, such as blurred, can be cited.

  The learning content includes video content with different motion speeds such as fast motion and slow motion for both video content that can detect the motion vector well and video content that cannot be detected well.

  The learning content can be obtained by, for example, the designer of the recording / reproducing apparatus 100 actually moving the video camera in a horizontal (pan) direction and a vertical (tilt) direction in a predetermined time, or zooming in or out. It is generated by shooting the subject.

  For example, when a learning content having a motion time of 2 fields (1/30 seconds) and a motion amount of 10 pixels is generated, the designer measures how many centimeters corresponds to how many pixels on the monitor of the video camera. Can be measured. Therefore, the designer can generate learning content having a desired amount of movement based on the measurement data. In this case, since the motion time is 2 fields (1/30 seconds) and the motion amount is difficult to measure at 10 pixels, the designer increases the time scale, for example, moves 300 pixels in 60 fields (1 second). You may measure as.

  Of the learning contents photographed by the designer in this manner, those with a subject sharpening are video contents that can detect the motion vector well, and those without a subject sharpening cannot detect the motion vector well. It is.

  Subsequently, the CPU 1 executes a learning process using the learning content generated as described above (step 52). Specifically, the CPU 1 first performs block matching processing on the learning content by the matching processing units 26, 28, 30 and 32 to detect a motion vector, and further detects the motion vector by the motion vector processing unit 34. Get the parameters.

  Subsequently, the designer compares the amount of motion of the motion vector detected from the learning content with the amount of motion that should be detected in advance when the learning content is generated, and compares the validity of the motion vector (valid / (Invalid) is determined, and the determination result is input to the recording / reproducing apparatus 100. That is, in the learning content, the motion vector detected from the subject with the presence of the subject is determined to be valid, and the motion vector detected from the subject without the subject is determined to be invalid.

  Subsequently, the CPU 1 uses the motion vector processing unit 34 to perform discriminant analysis using the input motion vector validity / invalidity judgment result and the discrimination parameter at the time of motion vector detection as learning data (step 53). . The discriminant analysis is, for example, linear discriminant analysis.

Then, the CPU 1 determines a discriminant function for judging the validity / invalidity of the motion vector based on the result of the discriminant analysis (step 54). In the case of linear discriminant analysis, this discriminant function is expressed as follows, for example.
y = a ₁ x ₁ + a ₂ x ₂ + a ₃ x ₃ + a ₄ x ₄ + a ₅ x ₅
In the discriminant function, the variables x _{1 to} x ₅ are the residuals between the standard block and the reference block, which are the discriminant parameters, the average value and variance value of the standard block, the average value and variance value of the reference block It corresponds to. That is, the discriminant analysis is a process for obtaining coefficients a _{1 to} a ₅ related to the variables x _{1 to} x ₅ of the discriminant function using the discriminant parameters.

  As this discriminant function, for example, a common function is generated for each of the matching processing units 26, 28, 30 and 32 of the 2/4/8/16 field. However, different (four) discriminant functions may be generated so as to correspond to each matching process in each matching processing unit 26, 28, 30 and 32, respectively.

  In this case, the learning content is also generated with different field times (motion times) so that each matching processing unit 26, 28, 30 and 32 can detect a motion vector. In this case, the discriminant parameter is also detected separately from each motion vector detected by each matching processing unit 26, 28, 30 and 32, and used for discriminant analysis for obtaining each discriminant function.

  By providing a discriminant function for each of the matching processing units 26, 28, 30 and 32, the CPU 1 enables / disables motion vectors corresponding to video contents having different motion speeds even with the same motion amount. Judgment can be made. Even with the same amount of motion, when the motion is fast, that is, when the detection field time is short, the image becomes blurred, and the discrimination parameters differ from those of the slow motion. However, by providing a discriminant function for each of the matching processing units 26, 28, 30 and 32 having different detection field times, the difference in the discriminant parameters can be reflected in the determination of the validity of the motion vector. The generated discriminant function is stored in, for example, the HDD 10 or the RAM 2 (step 55).

(Outline of video feature detection processing)
Next, the recording / reproducing apparatus 100 proceeds to a video feature detection process from video content stored in the HDD 10 or the like. This video feature is detected based on an effective motion vector detected from each video content using the determined discriminant function.

FIG. 6 is a flowchart showing a rough flow of video feature detection processing by the video feature detection unit 4.
As shown in the figure, the video feature detection unit 4 first sequentially inputs the respective images (hereinafter referred to as thumbnail images) constituting the thumbnail video of the video content stored in the HDD 10 or the like (step 61). Subsequently, the video feature detection unit 4 performs block matching processing between the input thumbnail images and detects a motion vector (step 62). Subsequently, the video feature detection unit 4 estimates camera features in the video content based on the detected motion vector, and estimates video editing features in the video content based on a fade / cut evaluation value described later. (Step 63). Then, the video feature detector 4 outputs each estimated video feature (step 64).
Hereinafter, the processing of each of these steps will be described in detail. First, details of the motion vector detection process will be described.

(Motion vector detection process)
FIG. 7 is a flowchart showing the flow of motion vector detection processing in the present embodiment. FIG. 8 is a functional block diagram of the motion vector detection process. FIG. 9 is a diagram conceptually showing a synthetic motion vector generation process.
7 and 8, when the thumbnail image data is input from the CPU 1 (step 71), the video feature detection unit 4 performs block processing on the thumbnail image data by the block processing unit 21 (step 71). 72). Specifically, first, the block processing unit 21 sets one of the input thumbnail image data as a reference field. Then, the block processing unit 21 sets, in the reference field, a reference image region that defines an image range that is a target of motion vector detection and a search region that defines a motion vector search range (reference range). Then, the block processing unit 21 divides the reference image area into, for example, 8 × 8 = 64 areas. Each divided block is extracted as a reference block in the subsequent block matching process.

  Subsequently, as shown in FIG. 7, the video feature detection unit 4 stores the thumbnail images sequentially input and subjected to the block processing in the 2/4/8/16 inter-field memory units 22 to 25 as reference fields, respectively. (Step 73). As described above, the 2/4/8/16 inter-field memory units 22 to 25 may be configured by a single memory element. In this case, as shown in FIG. 9, the copy of the thumbnail image buffered in the single memory element is delayed by 2/4/8/16 fields, respectively, and sent to each matching processing unit 26, 28, 30, 32. Supplied. Thereby, the same function as the case where the 2/4/8/16 inter-field memory units 22 to 25 exist as separate memory elements is realized.

  Subsequently, the video feature detection unit 4 uses the matching processing units 26, 28, 30, and 32 to compare the reference field and the reference fields input from the 2/4/8/16 inter-field memory units 22 to 25. Multistage block matching processing is performed between them (step 74). Specifically, the matching processing units 26, 28, 30, and 32 are configured so that the standard block extracted from the standard field is stored in the 2/4/8/16 inter-field memory units 22 to 25. Which position it has moved to is detected by pattern matching.

  In this case, the matching processing units 26, 28, 30, and 32 move the reference block set in the reference field one pixel at a time in the search field of the reference field, and each data component (Y, Cb) constituting the base block And Cr), the position where each vector distance is minimum is detected. A movement amount and a movement direction from the reference block to the detection position (reference block) are detected as motion vectors. The block matching process is executed for each block (for example, 64 blocks) constituting the reference field. In addition, the matching processing units 26, 28, 30, and 32 also calculate residuals after matching between the base block and the reference block together with the motion vector. The detected residual is output to each of the fade / cut processing units 27, 29, 31 and 33.

  Subsequently, the video feature detection unit 4 performs discriminant analysis on the motion vector detection data using the generated discriminant function (step 76). That is, as shown in FIG. 9, the video feature detection unit 4 has motion vectors 1a to 64a, 1b to 64b, 1c to 64c, 1d to 64d (detected for each block between 2/4/8/16 fields). The discriminant analysis is performed on each of the discriminant functions (not shown). That is, the residual between the standard block and the reference block, which is the discrimination parameter associated with the motion vector, the average value and variance value of the standard block, and the average value and variance value of the reference block are input as variables of the discrimination function. The In FIG. 9, for convenience of explanation, processing for motion vectors between 16 fields is not shown.

  Subsequently, the video feature detection unit 4 confirms whether the output of the discriminant function as a result of the discriminant analysis is positive or negative (step 77). When the output of the discriminant function is a positive value, the video feature detection unit 4 determines that the motion vector is valid (step 78), and when the output is a negative value, the video feature detection unit 4 determines that the motion vector is invalid (step 79). ). Of course, this positive / negative setting may be reversed.

  Subsequently, the video feature detection unit 4 normalizes the motion vectors detected in multiple stages (step 80). That is, the video feature detection unit 4 normalizes motion vectors detected between 16 fields, for example, by multiplying motion vectors detected between 2/4/8 fields by 8 times / 4 times / 2 times, respectively. To do. Thereby, the dynamic range of the motion vector detected between 2/4/8 fields can be expanded, the resolution can be improved, and the precision of the synthesized motion vector generated thereafter can be increased. Of course, the video feature detection unit 4 may normalize the motion vector detected between, for example, 4/8/16 fields by reducing it according to the motion vector detected between two fields. However, in this case, the resolution decreases as the motion vector detected between long fields.

  Subsequently, the video feature detection unit 4 determines that the motion vector processing unit 34 is valid by the discriminant analysis and generates a combined motion vector by combining the normalized motion vectors (step 81). For example, as shown in FIGS. 8 and 9, the motion vector processing unit 34 selects, for each block, the motion vector determined to be valid from the normalized data for each block of the motion vector detected between the fields. To synthesize. In the example of FIG. 9, among the 64 blocks, the motion vector between the two fields (1a to 4a, 9a to 12a, 17a to 20a, 25a to 28a) is located in the block located in the upper left quarter portion of the field. Is determined to be effective. For the blocks located in the upper right and lower left quarters, motion vectors between four fields (5b-8b, 13b-16b, 21b-24b, 29b-36b, 41b-44b, 49b-52b, 57b- 60b) is determined to be effective. For the block located in the lower right quarter portion, motion vectors (37c to 40c, 45c to 48c, 53c to 56c, 61c to 64c) between 8 fields are determined to be valid. Therefore, the motion vector processing unit 34 combines the motion vectors determined to be valid between the fields into one field to generate a combined motion vector.

  Here, when each motion vector between different fields is valid, the motion vector processing unit 34 uses a shorter motion vector between the fields to generate a combined motion vector. Accordingly, when all the motion vectors between 2/4/8/16 fields are valid, the motion vector processing unit 34 sets the motion vector between the two fields as a combined motion vector. As described above, the motion vector between shorter fields is preferentially used because the larger the field interval, the faster the motion content in the video content cannot follow the large motion amount. This is because there is a possibility of deviating from the search range. Therefore, the motion vector processing unit 34 can improve the accuracy of fast motion detection by giving priority to the motion vector between shorter fields.

  Then, the video feature detection unit 4 outputs the generated synthesized motion vector as motion data in the video content from the motion vector processing unit 34 to the camera feature determination unit 36 (step 82). This synthesized motion vector is used for the multiple regression analysis process in the subsequent video feature estimation process.

  8 and 9, the discriminant analysis process and the normalization process are illustrated to be executed separately for each field, but this is different from the discriminant analysis process and the normalization between the fields. It does not indicate that the processing is performed on separate hardware and software. The discriminant analysis process and the normalization process between the fields may be executed by a single hardware and software, or may be executed by separate hardware and software.

  By the way, the block matching process is performed in the order of shortness between fields, that is, the matching processing unit 26, the matching processing unit 28, the matching processing unit 30, and the matching processing unit 32 in this order. In the present embodiment, each of the matching processing units 26, 28, 30 and 32 can sequentially use the motion vector detected between the previous fields for the detection of the motion vector between the next fields. FIG. 10 is a diagram conceptually showing such block matching processing.

  As shown in the figure, the matching processing unit 26 between the two fields moves from the initial position (x0, y0) of the reference block of the reference field by block matching, and the movement position of the reference block of the reference field two fields ahead Search for (x2, y2). Here, the amount of motion between the four fields is considered to be larger than the amount of motion between the two fields. Therefore, the matching processing unit 28 between the four fields refers to the four fields ahead of the reference field with the movement position (x2, y2) between the two fields as the initial position, not the initial position (x0, y0) of the reference block. The movement position (x4, y4) in the block is searched.

  Then, the matching processing unit 28 between the four fields adds (x0, y0) to (x4) by adding the motion vector detected between the two fields and the motion vector detected with (x2, y2) as the initial position. , Y4) as a motion vector.

  This process is executed when a motion vector detected between two fields is determined to be valid by the discriminant analysis. That is, the matching processing unit 28 between the four fields determines the initial position of the block matching processing between the four fields in accordance with the motion vector discriminant analysis result detected between the two fields.

  Thus, by using the search position searched by the block matching process between the short fields as the initial position of the block matching process between the next fields, it is faster than searching as the initial position (x0, y0). Matching position can be detected.

  In FIG. 10, the processing for detecting motion vectors between two fields and between four fields has been described. Of course, the present invention can be similarly applied to detecting motion vectors between eight fields and sixteen fields. For example, when motion vectors between 2 fields and 4 fields are both valid and motion vectors between 8 fields are invalid, the matching processing unit 32 between 16 fields sets the search position between 4 fields as the initial position. Search for motion vectors.

(Video feature estimation processing)
Next, details of video feature estimation processing, that is, camera feature and video editing feature estimation processing will be described.

FIG. 11 is a flowchart showing a flow of video feature estimation processing by the recording / reproducing apparatus 100.
As shown in the figure, first, the camera feature determination unit 36 of the video feature detection unit 4 performs initial setting of a detection flag for each video feature (step 101). Here, the detection flag is a flag indicating that the pan, tilt, and zoom camera features and the fade and cut video editing features are detected from the video content. The detection flag of each video feature is represented by Dpan, Dtilt, Dzoom, Dfade, and Dcut, and each initial setting is performed by setting each flag value to 0.

  Subsequently, the camera feature determination unit 36 performs multiple regression analysis processing by, for example, the least square method based on the combined motion vector data output from the motion vector processing unit 34 (step 102), and calculates an affine coefficient. (Step 103). Here, an affine transformation model for calculating an affine coefficient by the multiple regression analysis process will be described.

  FIG. 12 is a diagram showing an affine transformation model. The affine transformation model is a model for describing translation, enlargement / reduction, and rotation of a three-dimensional object as coordinate transformation processing using a matrix. Since the camera features such as pan, tilt, and zoom are considered to be parallel movement and enlargement / reduction of an object in the reference field, the camera features can be described by using an affine transformation model.

Here, in the video content, when the field interval is not large, the following approximation processing can be performed on the assumption that the rotation angle θ is small for the rotation feature.
sinθ ≒ θ
cos θ ≒ 1

  Therefore, the affine transformation model can be modified as shown in FIG. And a camera characteristic can be detected by calculating | requiring each coefficient from this synthetic | combination motion vector using this affine transformation model. That is, predetermined camera thresholds Pth, Tth, and Zth are set for pan, tilt, and zoom, and each camera feature can be detected by comparing with each affine coefficient calculated from the combined motion vector. .

  FIG. 13 is a diagram showing processing for obtaining an affine coefficient by multiple regression analysis. As shown in the figure, the camera feature determination unit 36 uses the explanatory variables as x and y coordinates (xn, yn) of the initial position in the reference field, and sets the non-explaining variables (object variables) as motion vectors in the reference field. X and y coordinates (xm, ym) of the detected position (search position). Then, using the explanatory variable and the non-explanatory variable, the camera feature determination unit 36 performs a multiple regression analysis process on the synthesized motion vector to obtain pan, tilt, and zoom coefficients Px, Py, and Zx (step 103). ).

  Returning to FIG. 11, the camera feature determination unit 36 inputs the pan coefficient Px among the calculated affine coefficients (step 104). Then, the camera feature determination unit 36 determines whether or not the Px is greater than the threshold value Pth (step 105). If the Px is greater than Pth (Yes), the pan detection flag Dpan = 1 is set (step 1). 106) If Pth or less (No), the pan detection flag Dpan = 0 is set (step 107).

  Subsequently, the camera feature determination unit 36 inputs the tilt coefficient Py among the calculated affine coefficients (step 108). Then, the camera feature determination unit 36 determines whether or not the Py is greater than the threshold value Tth (step 109). If the Py is greater than Tth (Yes), the tilt detection flag Dtilt = 1 is set (step 1). 110), if Tth or less (No), the tilt detection flag Dtilt = 0 is set (step 111).

  Subsequently, the camera feature determination unit 36 inputs zoom coefficients Zx and Zy among the calculated affine coefficients (step 112). Then, the camera feature determination unit 36 determines whether or not the Zx or Zy is greater than the threshold value Zth (step 113). If at least one is greater than Zth (Yes), the zoom detection flag Dzoom = 1 (step 114). If both are equal to or smaller than Zth (No), the zoom detection flag Dzoom = 0 is set (step 115).

  The camera feature determination unit 36 may separately detect left pan and right pan, left tilt and right tilt, and zoom in and zoom out for each camera feature of pan, tilt, and zoom. This distinction can be easily made by referring to the sign of the affine coefficient. Here, when the camera is moved, the actual camera movement direction is opposite to the subject movement direction. The pan and tilt camera features detected by the camera feature determination unit 36 are detected not as camera motion but as subject motion. However, the camera feature determination unit 36 can detect the camera feature in accordance with the movement direction of the camera by reversing the sign of the affine coefficient.

  By calculating each affine coefficient by the above multiple regression analysis and comparing it with each threshold value, it is determined whether or not a camera feature exists in the video content. In this embodiment, the camera feature determination duration is varied depending on which field the motion vector of each block from which the synthesized motion vector is generated is detected. Hereinafter, the determination duration time variable process will be described.

  FIG. 14 is a flowchart showing a flow of variable processing of the determination duration time of the camera feature. This figure shows the variable processing as common processing for pan, tilt, zoom affine coefficients and camera features.

  As shown in the figure, the camera feature determination unit 36 first inputs each affine coefficient (step 141). Next, the camera feature determination unit 36 uses the motion vector finally output from the matching processing unit 26 between two fields as the motion vector of each block constituting the combined motion vector that is the basis for calculating the affine coefficient. It is determined whether or not many are included (step 142). If the camera feature determination unit 36 determines that the number of motion vectors between two fields is the largest (Yes), it sets the determination time length of each camera feature to t2 (step 143).

  If the camera feature determination unit 36 determines that the number of motion vectors between two fields is not the largest (No), does the combined motion vector include the largest number of motion vectors finally output from the matching processing unit 28 between the four fields? It is determined whether or not (step 144). When the camera feature determination unit 36 determines that the motion vectors between the four fields are the largest (Yes), the camera feature determination unit 36 sets the determination time length of each camera feature to t4 (step 145).

  If the camera feature determination unit 36 determines that the number of motion vectors between the four fields is not the largest (No), does the combined motion vector include the largest number of motion vectors finally output from the matching processing unit 30 between the eight fields? It is determined whether or not (step 146). When the camera feature determination unit 36 determines that the motion vectors between the eight fields are the largest (Yes), the camera feature determination unit 36 sets the determination time length of each camera feature to t8 (step 147).

  If the camera feature determination unit 36 determines that the motion vector between the eight fields is not the largest (No), the camera feature determination unit 36 sets the determination time length of each camera feature to t16 (step 148). Here, t2 <t4 <t6 <t16.

  Subsequently, the camera feature determination unit 36 performs a process for detecting the presence or absence of each camera feature based on the predetermined threshold values Pth, Tth, and Zth for each block constituting the combined motion vector. Continue for a long time.

  Then, the camera feature determination unit 36 determines whether or not each camera feature is detected within a predetermined time ratio within each determination time (step 149). That is, the camera feature determination unit 36 determines whether or not each affine coefficient exceeding the predetermined thresholds Pth, Tth, and Zth has been detected for a predetermined time ratio or more.

  The camera feature determination unit 36 determines that there is a camera feature if each camera feature is detected within a predetermined time ratio within each determination time (step 150), and otherwise indicates that there is no camera feature. Determination is made (step 151). The determination result is recorded as each detection flag.

  In this case, the non-explanatory variable (objective variable) in the multiple regression analysis process is also determined according to the field in which each motion vector constituting the combined motion vector is detected. That is, the x and y coordinates (xm, ym) of the motion vector detection position (search position) in the reference field between the fields where the motion vector is the largest are set as non-explanatory variables.

  As described above, changing the camera feature determination time length according to which field the detected most motion vector based on the combined motion vector is based on the fast motion in the video content. This is to cope with both slow movements. FIG. 15 is a diagram illustrating the movement amount of the motion vector detected between 2 fields and 16 fields when the motion is fast and when the motion is slow.

  As shown in the figure (A), the motion vector that is effectively detected in a short field interval is considered to indicate a fast motion, and as shown in the figure (B), it is detected effectively in a long field interval. It is considered that the obtained motion vector indicates a slow motion. As shown in FIG. 4A, when the motion is slow, the matching processing unit 32 can detect an effective motion vector between 16 fields. However, between the two fields, the matching processing unit 26 may not be able to detect an effective motion vector because the motion amount of each block of the combined motion vector is too small. Even if the matching processing unit 26 can detect an effective motion vector, the motion vector of each block of the combined motion vector may be too small to be detected by the camera feature determination unit 36 as a camera feature.

  On the other hand, as shown in FIG. 5B, when the motion is fast, the matching processing unit 26 can detect an effective motion vector between two fields. However, there is a possibility that the matching processing unit 32 cannot detect an effective motion vector between 16 fields because the amount of motion is too large to follow.

  Therefore, the camera feature determination unit 36 increases the determination duration of the camera feature when the distance between the fields where the most motion vectors are detected is large. Thereby, it is possible to reliably detect a slow movement. The camera feature determination unit 36 reduces the determination time length of the camera feature when the distance between the fields where the most motion vectors are detected is small. Accordingly, it is possible to reliably detect a fast movement, to prevent useless determination processing, and to prevent erroneous determination due to noise that may be mixed by performing long-time determination. FIG. 16 is a graph showing the variable processing of the camera feature determination continuation length corresponding to the field in which the most motion vectors are detected as described above. As shown in the figure, the camera feature determination duration time increases as the distance between 2 fields, 4 fields, 8 fields, 16 fields, and the fields increases.

(Fade / cut detection process)
Returning to FIG. 11, the video feature detection unit 4 performs a fade and cut detection process.
First, the processing of the fade / cut processing units 27, 29, 31 and 33 will be described.

  The fade / cut processing units 27, 29, 31 and 33 respectively input the residuals after the matching processing from the matching processing units 26, 28, 30 and 32, and based on these residuals, respectively. A fade / cut evaluation value is generated and output to the fade / cut determination unit 35 (step 116).

Here, the fade / cut evaluation value H can be obtained by the following equation, where En (n = 0 to 63) is the residual.
₆₃
H = ΣEn
^{n = 0}

  Therefore, each fade / cut processing unit 27, 29, 31 and 33 uses the residuals from the matching processing units 26, 28, 30 and 32 until n = 63, that is, all the reference fields. Are input until the residuals for the reference blocks are input, and the sum of these is calculated.

  17 and 18 are graphs showing the relationship between the calculation result of the fade / cut evaluation value and the field progress for each field interval. FIG. 17 shows a graph when cut points are included, and FIG. 18 shows a graph when fades are included.

  The fade / cut determination unit 35 determines fade and cut based on the fade / cut evaluation value shown in FIGS. 17 and 18 (step 117 in FIG. 11). That is, when the change in the fade / cut evaluation value with the progress of the field is steep (Yes in Step 118), the fade / cut determination unit 35 determines that the cut is made and sets the cut detection flag Dcut = 1 ( Step 120). Further, when the change in the fade / cut evaluation value with the progress of the field is moderate (Yes in Step 119), the fade / cut determination unit 35 determines that the fade is made and sets the fade detection flag Dfade = 1. (Step 121). If neither of them can be determined (No in step 119), the fade / cut determination unit 35 sets the cut detection flag Dcut = 0 and the fade detection flag Dfade = 0 (step 122).

  Specifically, the fade / cut determination unit 35 analyzes the change in the fade / cut evaluation value between two fields, and when a peak characteristic as shown in the graph a of FIG. Is determined as a cut point.

  Further, when the peak characteristic is not detected, the fade / cut determination unit 35 performs the fade evaluation value (graph a) between two fields and the four field intervals at a predetermined time t as shown in FIG. The difference Va between the fade evaluation value (graph b) and the fade evaluation value between four fields and the fade evaluation value (graph c) between eight fields and the fade evaluation value between eight fields and 16 fields A difference Vc from the fade evaluation value (graph d) is calculated.

  As shown in FIG. 18, in the case of fading, since the video gradually changes, the fade / cut evaluation value varies in the amount of change depending on the field interval, so that each value of Va, Vb, Vc is All appear prominently as positive and relatively close numbers. On the other hand, in the case of a cut, as shown in FIG. 17, there are large differences in the values of Va, Vb, and Vc, and there may be negative values. Therefore, the fade / cut determination unit 35 can determine whether or not it is a fade by analyzing the Va, Vb, and Vc.

  Then, the video feature detection unit 4 outputs each camera feature and video editing feature detected by the above processing (step 123).

  FIG. 19 is a table showing the determination results of the video features determined by the camera feature determination unit 36 and the fade / cut determination unit 35. The CPU 1 controls the video feature detection unit 4 so as to store data equivalent to this table in, for example, the RAM 2 or the HDD 10.

  The recording / reproducing apparatus 100 can create, for example, a digest video (highlight video) from video content or divide the video content into chapters using the detected video features.

[Summary]
As described above, according to the present embodiment, the recording / reproducing apparatus 100 determines the effectiveness of the motion vector detected between the image data having different time lengths, and determines the synthesized motion vector based on the effective motion vector. Therefore, the motion vector detection accuracy can be improved.

  In addition, since the recording / playback apparatus varies the duration of the camera feature determination according to the field in which each motion vector that is the basis of the combined motion vector is detected, it supports both fast and slow motion. Thus, camera features can be reliably detected.

[Modification]
The present invention is not limited to the embodiments described above, and various modifications can be made without departing from the scope of the present invention.

  In FIG. 4, the video feature detection unit 4 is configured by connecting the inter-field memory units 22 to 25 in series. However, the video feature detection is performed by connecting the inter-field memory units 22 to 25 in parallel. The unit 4 may be configured. FIG. 20 is a diagram showing the configuration of the video feature detection unit 4 in this case. Even if comprised in this way, the same process as the case where it connected in parallel may be performed, and the same effect can be acquired.

  In the above-described embodiment, the video feature detection unit 4 processes video content in units of fields, but may of course process in units of frames.

  In the above-described embodiment, the video feature detection unit 4 uses each thumbnail image constituting the thumbnail video of the original video content. However, the original video content image that has not been reduced may be used. .

  In the above-described embodiment, the video feature detection unit 4 uses a linear discriminant function as a discriminant function, but may use a non-linear discriminant function. For the non-linear discriminant analysis, for example, a method of performing discrimination by calculating the Mahalanobis distance, a support vector machine, a neural network, or other machine learning methods may be used.

  In the method of calculating the Mahalanobis distance, the discriminant function is not used, but learning content in which the validity / invalidity of the motion vector is known in advance is used as in the above-described embodiment. First, the variance-covariance matrix and average of each valid / invalid data group are calculated based on the discrimination parameters of motion vectors detected from several learning contents.

  Then, the Mahalanobis distance between the motion vector discrimination parameter detected from each block and each valid / invalid data group is calculated between each field of the video content to be subjected to the video feature detection processing. Then, it is determined which data group the Mahalanobis distance is closer to, and the detected motion vector discrimination parameter belongs to the group of the closer distance (the one with the smallest distance). By such processing, the validity / invalidity of the detected motion vector is determined.

Here, the data above learning, the variance-covariance matrix H _v of valid discrimination _parameter, averaged a _y, a variance-covariance matrix of the invalid determining parameter H _n, average and a _n, subject discriminating process Assuming that the detected motion vector discrimination parameter is v, the Mahalanobis distance D _y between the detected motion vector discrimination parameter and the effective data group is calculated by the following equation.
^{_{D y 2 = (v-a}} y) T H y -1 (v-a y)
Here, (v−a _y ) ^T indicates a transposed matrix, and H _y ⁻¹ indicates an inverse matrix. Further, since the discrimination parameter v is multivariate data, it is regarded as a (feature) vector.

Similarly, Mahalanobis distance D _n of the determining parameter of the detected motion vector, and invalid data group is calculated by the following equation.
^{_{D n 2 = (v-a}} n) T H n -1 (v-a n)

As described above, the validity / invalidity of the detected motion vector is determined as follows.
D _y ² <··· invalid cases of ... effective D _y ² ≧ D _n ² of D _n ²

(Learning content generation processing)
Next, another embodiment relating to the learning data generation process used in the discrimination process using the Mahalanobis distance will be described.

As described above, in the discrimination processing using the Mahalanobis distance, learning content (learning video data) whose motion vector is known in advance is used to generate learning data used for the discrimination processing. It is done. A motion vector is detected from the learning content, and the discrimination parameter is acquired when the motion vector is valid and invalid, and the discrimination parameter is used for the discrimination process. However, it is very troublesome to manually create the learning content by adjusting the amount of movement and the direction of movement for each of the different time lengths, for example, by camera photography. In addition, when the discrimination parameter is acquired from the manually created learning content, there is a possibility that the characteristics of the discrimination parameter vary and the accuracy of the discrimination process is lowered.
Therefore, in this embodiment, the learning / playback apparatus 100 automatically creates the learning content.

  FIG. 21 is a block diagram showing a configuration of the video feature detection unit 4 in the present embodiment. In the figure, the same parts as those in the above-described embodiment are denoted by the same reference numerals, and description thereof is omitted or simplified.

  As shown in the figure, the video feature detection unit 4 in the present embodiment newly includes a learning image extraction unit 211, a window processing unit 212, and a learning processing unit 213. Block processing units 21, 2/4/8/16 fields between memory units 22-25, matching processing units 26, 28, 30, 32, fade / cut processing units 27, 29, 31, 33, discrimination processing The unit 214, the normalization unit 215, the motion vector processing unit 34, the fade / cut determination unit 35, and the camera feature determination unit 36 have the same configuration and functions as those of the above-described embodiment.

  The learning image extraction unit 211 receives a video signal of sample video content or sample still image data for generating the learning content. The learning image extraction unit 211 detects a cut point (scene change) from the input sample video content, and extracts an image of the cut point as learning image data. The extracted learning image data is supplied to the window processing unit 212. Specifically, the learning image extraction unit 211 detects, for each block, residual signals in the histograms of Y, Cb, and Cr between the above-described fields of the input sample video content, and the residual of each block. The sum of the signals is calculated as a cut evaluation value. The learning image extraction unit 211 detects a cut point by analyzing the change in the cut evaluation value, and extracts the field of the cut point as learning image data. That is, when the cut evaluation value changes abruptly (for example, a predetermined threshold value or more), the subsequent field is extracted as learning image data among the fields before and after the change.

  In addition, when using still image data instead of video content as a sample, the learning image extraction unit 211 supplies the still image data to the window processing unit 212 as learning image data.

  The window processing unit 212 moves the window (rectangular frame) in the learning image data supplied from the learning image extraction unit 211 or changes the image in the window (window processing), so that the learning content ( Learning video data). The learning content includes the detection of each camera feature for each of the 2/4/8/16 fields and for each pan, tilt, and zoom camera feature detected as the motion vector. Generated to learn validity. The generated learning content is supplied to the learning processing unit 213. Details of this window processing will be described later.

  The learning processing unit 213 uses the learning content supplied from the window processing unit 212 to execute a learning process for valid / invalid motion vectors. Specifically, the learning processing unit 213 detects a motion vector between the different fields from the learning content, and obtains the determination result of the validity / invalidity of the motion vector and the determination parameter at the time of detecting the motion vector as learning data. To the discrimination processing unit 214.

  The discrimination processing unit 214 uses the learning data supplied from the learning processing unit 213 for the validity / invalidity of the motion vectors detected from the matching processing units 26, 28, 30, and 32, respectively, and discriminant analysis based on the Mahalanobis distance I do. The discrimination processing unit 214 outputs the result of the discrimination analysis (valid / invalid of motion vector) to the normalization unit 215.

  Here, the sample video content may be the same content as the video content that is input to the block processing unit 21 and that is the target of video feature detection. In this case, for example, when the generation of a digest video from a certain video content is instructed by a user operation, the video feature detection unit 4 inputs the video content to the block processing unit 21 and also uses the learning image extraction unit 211. To enter. Thus, the learning content generation process and the learning process are executed in parallel with the motion vector detection process by each matching processing unit for the same video content as the digest video generation target video content.

  When the sample video content is not the same as the video content whose video feature is to be detected, the learning content generation process and the learning process are performed periodically or when the video content is newly stored in the HDD 10 or the like. Prior to the motion vector detection process, it is executed in advance.

  The block processing unit 21 receives a video signal of video content that is a target of video feature detection. The video signals are stored in the memory units 22 to 25, and motion vectors are detected from the video signals by the matching processing units 26, 28, 30, and 32, and are output to the discrimination processing unit 214.

  Further, the residual signals as matching results in the respective matching processing units 26, 28, 30 and 32 are supplied to the respective fade / cut processing units 27, 29, 31 and 33. As described above, each fade / cut processing unit 27, 29, 31, 33 generates a fade / cut evaluation value using the residual signal and outputs the fade / cut evaluation value to the fade / cut determination unit 35. The fade / cut determination unit 35 analyzes the change in the fade / cut evaluation value, detects fade and cut from the video signal, and outputs the result. In the present embodiment, the fade / cut determination unit 35 can also supply the image data at the detected cut point to the window processing unit 212 as the learning image data. That is, as described above, in the case where the sample video content is the same content as the video content that is the detection target of the video feature, the image of the cut point detected from the video content that is the detection target of the video feature is This is supplied to the window processing unit 212.

  The functions of the normalization unit 215, the motion vector processing unit 34, and the camera feature determination unit 36 are the same as those in the above-described embodiment, and thus are omitted.

Next, learning content generation processing by the video feature detection unit 4 configured as described above will be described.
FIG. 22 is a flowchart showing a rough flow of the learning content generation processing in the present embodiment. In the figure, a case where the video feature detection unit 4 generates learning content using learning image data extracted from sample video content will be described.

  As shown in FIG. 22, the video feature detection unit 4 first inputs various video contents stored in the HDD 10 or the like as sample video content to the learning image extraction unit 211 (step 221). The learning image extraction unit 211 extracts cut points from the input sample video content by the above-described method (step 222). When a cut point, that is, a scene change is detected (Yes), the learning image extraction unit 211 extracts an image of the cut point (the image after the cut point) as learning image data, and the RAM 2 or the like. (Step 224). The video feature detection unit 4 repeats the processing from step 221 to step 224 for all the sample video contents stored in the HDD 10 or the like (step 225).

FIG. 25 is a diagram illustrating a state in which an image of a cut point is extracted as learning image data from sample video content.
As shown in the figure, images f0, f2, and f4 are extracted as cut points 1 to 3 from the sample video content. The images V1, V2, and V3 of the cut points are output as learning image data to the learning image extraction unit 211 and stored in the RAM 2 or the like.

  Here, when the sample video content is the same as the video content whose video feature is to be detected, each image of the cut point is input from the fade / cut determination unit 35 to the learning image extraction unit 211, It is stored in the RAM 2 or the like as learning image data. In this case, the fade / cut determination unit 35 extracts images of all cut points from the video content whose video features are to be detected, and outputs the images to the learning image extraction unit 211.

  Further, when still image data is used for generating learning content, as shown in FIG. 24, for example, all the still image data V1, V2, V3,. Is input to the learning image extraction unit 211 and stored in the RAM 2 or the like.

Subsequently, the window processing unit 212 sets a window processing flag w to be executed to 0 (step 226).
Here, the window processing flag w is set to 0-2. w = 0 is set when the window is moved in the horizontal direction, w = 1 is set when the window is moved in the vertical direction, and w = 2 is set when the window is enlarged and reduced. The initial value of the window processing flag is set to 0.

  Subsequently, the window processing unit 212 reads the learning image data stored in the RAM 2 or the like (step 227), and executes window processing on the learning image data (step 228). Hereinafter, this window processing will be described in detail.

FIG. 23 is a flowchart showing a detailed flow of the window processing.
As shown in the figure, the window processing unit 212 determines whether or not the window processing flag w is 0, that is, whether or not the window processing unit 212 is set to move the window in the horizontal direction (left-right direction). Is determined (step 241).

Subsequently, the window processing unit 212 first sets n = 1 in order to generate learning content for learning the validity of motion vectors detected between 2 ⁿ fields (step 242). Here, 1 ≦ n ≦ 4 (n is an integer). That is, the learning content is generated for each field in order to learn the validity of the motion vector between the fields of 2/4/8/16 fields detected by the matching processing units. . The initial value of n in the learning content generation process for each field is set to 1.

Subsequently, the window processing unit 212 sets a horizontal moving speed of the window for generating learning content corresponding to motion vector detection between 2 ⁿ fields. Here, the moving speed is set to a minimum speed at which each matching processing unit can finally detect between 2 ⁿ fields. First, a moving speed for generating learning content for motion vectors between two fields is set.

Subsequently, the window processing unit 212 moves the window in the horizontal direction at the set moving speed in the read learning image data, thereby performing the panning (left panning and right panning) between 2 ⁿ fields. ) Learning content for learning is generated (step 244).

FIG. 26 is a diagram illustrating how the learning content for bread is generated.
As shown in FIG. 6A, the window S is set to a predetermined area smaller than the size of the learning image data V (for example, 720 × 480 pixels) and is moved in the horizontal direction. For example, the window processing unit 212 gradually moves the window S shown in FIG. 5B to the right as shown in FIGS. 5C to 5E, and cuts out each image in the moved window S. Thus, it is possible to generate learning content that is actually taken by panning the camera to the right. In this learning content, since the subject (building) is gradually moving in the left direction, the movement in the left direction is detected as the motion vector. Similarly, the window processing unit 212 can generate learning content that is actually taken by panning the camera leftward by moving the window S leftward. From this learning content, the movement of the subject in the right direction is detected as a motion vector. The window processing unit 212 generates learning content imitating the right pan of the camera by moving the window S in the right direction with respect to certain learning image data V, and then moves the window S moved in the right direction to the left By returning to, a learning content imitating the left pan may be created. As described above, since the moving speed of the window S is set in advance, the window processing unit 212 can easily grasp the moving amount of the subject in these learning contents.

  Subsequently, the window processing unit 212 increments the value of n by 1 and sets it to n = n + 1 (step 245), and determines whether or not the incremented value of n exceeds 5 (step 246). ).

  If the value of n does not exceed 5 (No), the window processing unit 212 returns to step 243 and executes the learning content generation process for pan corresponding to the motion vector between the next fields. Thereby, the learning content for bread corresponding to each motion vector between 2/4/8/16 fields is generated. When the value of n exceeds 5 (Yes), the window processing unit 212 ends the learning content generation process for bread.

  When the window processing unit 212 determines in step 241 that the window processing flag w is not 0 (No), the window processing unit 212 determines whether or not the window processing flag w is 1 (step 247). That is, the window processing unit 212 determines whether or not the processing for moving the window in the vertical direction (up and down direction) is set (step 247).

  Subsequently, the window processing unit 212 moves the window S in the learning image data V in the vertical direction in the same manner as the learning content generation processing for bread learning, so that a vertical motion vector between the fields is obtained. Learning content for tilt learning corresponding to is generated (steps 248 to 252). That is, the window processing unit 212 generates learning content that is actually taken by tilting the camera in the vertical direction. In this case, for example, the window processing unit 212 moves the window S upward in the learning image data V, generates learning content imitating the upward tilt of the camera, and then moves in the upward direction. The learning content imitating the downward tilt of the camera may be generated by returning the window S thus moved downward.

When the window processing unit 212 determines in step 247 that the window processing flag w is not 1 (2) (No), the window processing unit 212 executes learning content generation processing for zoom learning (steps 253 to 253). 256). That is, the window processing unit 212 generates learning content for zoom learning corresponding to the motion vector for enlargement and reduction between fields by enlarging and reducing the window S in the learning image data V. The rough flow of learning content generation processing for zoom learning (setting of the value of n, speed setting, etc.) is the same as the generation processing of learning content for pan and tilt learning. However, the learning content generation process for zoom learning differs from the above-described learning content generation process for panning and tilt learning in that the window S is enlarged and reduced. Hereinafter, the details of the process of generating learning content for zoom learning will be described focusing on the enlargement and reduction processes.
FIG. 27 is a diagram conceptually illustrating a process for generating learning content for zoom learning.
As shown in the figure, when generating learning content for reduction (zoom-out) learning, the window processing unit 212 includes a window Sa and a window Sb having a larger area in the learning image data V. Define The sizes of the window Sa and the window Sb are set in a range not exceeding the size of the learning image data V. Then, the window processing unit 212 gradually changes the image data in the window Sa to the image data in the window Sb capturing a wider area image during the time ta (seconds). This is equivalent to gradually expanding the window Sa to the window Sb and normalizing the size of each image cut out by the window Sa being enlarged to the size of the window Sa.

  Further, when generating the learning content for enlargement (zoom-in) learning, the window processing unit 212 gradually converts the image data in the window Sb stored in the window Sa into the window Sa during the time tb (seconds). Return to the image data. This is equivalent to gradually reducing the window Sb to the window Sa and normalizing the size of each image cut out by the window Sb being reduced to the size of the window Sa.

FIG. 28 is a diagram illustrating a process of generating learning content for zoom-out (reduction) learning. In the figure, for simplification of explanation, processing for 10 × 10 pixel data is shown. Of course, processing for pixel data of other sizes is also executed in the same manner.
As shown in FIG. 5A, the window processing unit 212 sets a window surrounding 6 × 6 pixel data as the window Sa and a window surrounding 10 × 10 pixel image data as the window Sb. Then, the window processing unit 212 changes 8 pixels in the horizontal direction from the pixel data of 8 × 8 pixels to 6 pixels in accordance with the window Sa, and 6 pixels in the vertical direction according to the window Sa. To change. This process is executed after t1 (seconds) from the start of the process. As a result, as shown in FIG. 5B, the image in the window Sa is reduced to (6 × 6) / (8 × 8) = 0.56 times.

  Then, as shown in FIG. 5A, the window processing unit 212 finally changes 10 pixels in the horizontal direction from 10 × 10 pixel data to 6 pixels in accordance with the window Sa, and vertically 10 pixels in the direction are changed to 6 pixels in accordance with the window Sa. This process is executed ta (seconds) after the start of the process. As a result, as shown in FIG. 5B, the image in the window Sa is reduced to (6 × 6) / (10 × 10) = 0.36 times.

FIG. 29 is a diagram for explaining a calculation method when the number of pixel data is changed in the horizontal direction and the vertical direction.
As shown in FIGS. 6A and 6B, for example, when data of 8 pixels is changed to data of 6 pixels, the pixel data ek (k: 0 to 5) after the change is the pixel data dn before the change. It is generated by weighted addition of (n: 0 to 7). As shown in FIG. 5B, the weighting factor w _kn is varied in proportion to the distance between the pixel data dn (d0 to d7) before the change and the pixel data ek (e0 to e5) after the change. That is, the weight coefficient is set large when the position (distance) between the pixel data before and after the change is close, and the weight coefficient is set small when the position is far. For example, as shown in Fig (A), in the case where the pixel data e0 is generated, the weighting factor w ₀₁ is maximized for the pixel data d1 distance between the pixel data e0 is minimized, the distance between the pixel data e1 weighting factor w ₀₇ for the pixel data d7 to the maximum is minimized. The window processing unit 212 executes this processing in the horizontal direction and the vertical direction, respectively.

FIG. 30 is a diagram showing the generation process of learning content for zoom-in (enlargement) learning in the same manner as FIG.
As shown in the figure, the window processing unit 212 returns learning data for zoom-in (enlargement) learning by returning the data generated in the process for generating learning content for zoom-out (reduction) learning. Is generated. That is, as shown in FIG. 8A, the window processing unit 212 uses the pixel data (0.36 times) generated by the process of FIG. 28 and changed from 10 × 10 pixels to 6 × 6 pixels. To start the process. The window processing unit 212 converts the pixel data changed from 10 × 10 pixels to 6 × 6 pixels into pixel data obtained by changing pixel data of 8 × 8 pixels to pixel data of 6 × 6 pixels after t2 seconds. Change. As a result, as shown in FIG. 5B, the image in the window Sa is enlarged from the above 0.36 times to (6 × 6) / (8 × 8) = 0.56 times.

  Then, the window processing unit 212 finally returns the pixel data of 6 × 6 pixels in the window Sa to the original pixel data after tb seconds, as shown in FIG. As a result, as shown in FIG. 5B, the image in the window Sa is enlarged from 0.36 times to 1 time.

  The window processing unit 212 can separately generate the zoom-in learning content without using the data generated in the zoom-out learning content generation processing. However, it is more efficient to use the data generated by the zoom-out learning content generation process because it saves the trouble of generating new data.

FIG. 31 is a diagram showing how the image in the window Sa changes in the learning content generation process for zoom-out (reduction).
The window processing unit 212 defines windows Sa and Sb in the learning image data V as shown in FIGS. Then, the window processing unit 212 actually zooms out with the camera by gradually changing the image in the window Sa to the image in the window Sb, as shown in FIGS. It is possible to generate learning content such as a photograph taken. By the reverse process, learning content that is actually captured by zooming in with the camera is generated.

  When the learning content is generated by the above processing, the window processing unit 212 stores the learning content in the RAM 2 or the HDD 10 or the like. In this case, data relating to the moving amount and moving direction of the window S is also stored in association with the learning content.

  Returning to FIG. 22, the learning processing unit 213 of the video feature detection unit 4 executes the learning process using the learning content generated as described above (step 229). That is, the learning processing unit 213 reads the generated learning content from the RAM 2 or the HDD 10, detects a motion vector from the learning content, and determines whether the motion vector is valid / invalid. The learning processing unit 213 can also acquire from the RAM 2 or the HDD 10 the data related to the moving direction and moving amount of the window S at the time of learning content generation, and therefore can easily determine whether the detected motion vector is valid / invalid. That is, the validity / invalidity of the motion vector is determined according to whether or not the movement amount and movement direction of the actually detected motion vector match the movement amount and movement direction of the window S stored at the time of learning content generation. To be judged. This learning process is executed for each different field of the 2/4/8/16 field.

  Then, the learning processing unit 213 uses the motion vector valid / invalid data and the discrimination parameters (residual between the base block and the reference block, the average value and the variance value of the base block and the reference block) as the learning data. (Step 230).

  Subsequently, the window processing unit 212 increments the window processing flag w by 1 (step 232), and determines whether or not the value of the window processing flag w exceeds 2 (step 233). That is, the window processing unit 212 determines whether or not learning content for learning of each of the pan, tilt, and zoom is generated from one learning image data. If the value of the window processing flag w is 2 or less (No), the window processing unit returns to Step 227 and executes the learning content generation processing corresponding to the window processing flag w after the increment.

  When the value of the window processing flag w exceeds 2 (Yes), the determination processing unit 214 reads the learning data from the RAM 2 or the like (step 234). When the discrimination processing unit 214 completes the reading of the learning data (Yes in Step 235), the discrimination processing unit 214 receives input of the detected motion vector discrimination parameter from the matching processing units 26, 28, 30, and 32 (Step 236). ). Then, as described above, the discrimination processing unit 214 calculates the motion vector by calculating the Mahalanobis distance between the valid discrimination parameter group and the invalid discrimination parameter group among the input discrimination parameter and the learning data. The validity / invalidity is determined (step 237).

  As described above, according to the present embodiment, the video feature detection unit 4 can efficiently generate learning content with small variations in characteristics by executing window processing in the learning image data. Thereby, the video feature detection unit 4 can accurately execute the learning process using the learning content and the discrimination process using the result of the learning process.

  The video feature detection unit 4 also uses the video content of the video feature detection target as the learning video content, thereby generating a motion vector that will be detected from the video content in a pseudo manner. You can learn about validity. Therefore, the video feature detection unit 4 can learn about the validity of a motion vector similar to the motion vector actually detected by each matching processing unit. That is, in this case, the learning content generated by the window processing unit 212 is a video content different from that based on the video content of the video feature detection target. Therefore, since the video feature detection unit 4 can comprehensively learn motion vectors that may be detected from the video content for video feature detection, the validity of the actually detected motion vector is determined. It can be determined with higher accuracy.

  In the present embodiment, the recording / reproducing apparatus 100 executes the validity / invalidity of the motion vector by discriminant analysis based on the Mahalanobis distance. However, the recording / reproducing apparatus 100 may perform the discriminant analysis using another machine learning method such as SVM (Support Vector Machine) or a neural network. The learning content generation process can be similarly applied to the case where the discriminant function is used to determine whether a motion vector is valid / invalid.

  In the above-described embodiment, the recording / reproducing apparatus 100 executes the discrimination function generation processing and the learning data generation processing for performing the discrimination processing using the Mahalanobis distance. However, the discriminant function and the learning data may be generated by another device and input and stored in the recording / reproducing device 100 via various interfaces.

It is the figure which showed the structure of the recording / reproducing apparatus which concerns on one Embodiment of this invention. It is the figure shown about the camera characteristic in one Embodiment of this invention. It is the figure shown about the video editing characteristic in one Embodiment of this invention. It is the block diagram which showed the specific structure of the image | video feature detection part in one Embodiment of this invention. It is the flowchart which showed the flow of the production | generation process of the discriminant function in one Embodiment of this invention. It is the flowchart which showed the rough flow of the image | video feature detection process by the image | video feature detection part in one Embodiment of this invention. It is the flowchart which showed the flow of the motion vector detection process in one Embodiment of this invention. It is a functional block diagram of the motion vector detection process in one Embodiment of this invention. It is a conceptual diagram of the production | generation process of the synthetic | combination motion vector in one Embodiment of this invention. It is a conceptual diagram of the block matching process in one Embodiment of this invention. It is the flowchart which showed the flow of the image | video feature estimation process by the recording / reproducing apparatus which concerns on one Embodiment of this invention. It is the figure which showed the affine transformation model used in one Embodiment of this invention. It is the figure which showed the process which calculates | requires an affine coefficient by multiple regression analysis in one Embodiment of this invention. It is the flowchart which showed the flow of the variable process of the determination continuation time length of the camera characteristic in one Embodiment of this invention. In one Embodiment of this invention, it is a figure explaining the amount of movement of the motion vector detected between 2 fields and 16 fields about the case where a motion is fast and the case where a motion is slow. It is the graph shown about the variable process of the determination duration length of the camera characteristic according to between the fields from which the motion vector was detected in one Embodiment of this invention. It is the graph which showed the relationship between the calculation result of fade / cut evaluation value, and progress of a field for every said field interval in one Embodiment of this invention. It is the graph which showed the relationship between the calculation result of fade / cut evaluation value, and progress of a field for every said field interval in one Embodiment of this invention. It is the table | surface which showed the determination result of each image | video characteristic determined in one Embodiment of this invention. It is the block diagram which showed the specific structure of the image | video feature detection part in other embodiment of this invention. It is the block diagram which showed the structure of the image | video feature detection part in other embodiment of this invention. It is the flowchart which showed the rough flow of the content generation process for learning in other embodiment of this invention. It is a flowchart which shows the detailed flow of the window process in other embodiment of this invention. It is the figure which showed a mode that the still image was extracted as image data for learning in other embodiment of this invention. In other embodiment of this invention, it is the figure which showed a mode that the image of the cut point was extracted as learning image data from the sample video content. It is the figure which showed a mode that the content for learning for bread | pans is produced | generated in other embodiment of this invention. It is the figure which showed notionally the production | generation process of the learning content for zoom learning in other embodiment of this invention. It is the figure which showed the mode of the production | generation process of the content for learning for zoom out (reduction | reduction) learning in other embodiment of this invention. FIG. 11 is a diagram for explaining a calculation method when changing the number of pixel data in the horizontal direction and the vertical direction in another embodiment of the present invention. It is the figure which showed the mode of the production | generation process of the content for learning for zoom-in (enlargement) learning in other embodiment of this invention similarly to the said FIG. It is the figure which showed a mode that the image in window Sa changed in the learning content production | generation process for the zoom out (reduction | reduction) in other embodiment of this invention.

Explanation of symbols

1 ... CPU
2 ... RAM
3 ... Operation input unit 4 ... Video feature detection unit 10 ... HDD
DESCRIPTION OF SYMBOLS 11 ... Optical disk drive 16 ... AV decoder 21 ... Block processing part 22 ... Memory field between 2 fields 23 ... Memory part between 4 fields 24 ... Memory part between 8 fields 25 ... Memory part between 16 fields 26, 28, 30, 32 ... Matching Processing unit 27, 29, 31, 33 ... Fade / cut processing unit 34 ... Motion vector processing unit 35 ... Fade / cut determination unit 36 ... Camera feature determination unit 100 ... Recording / playback device 211 ... Learning image extraction unit 212 ... Window processing Unit 213 ... Learning processing unit 214 ... Discrimination processing unit V ... Image data for learning Sa ... Window Sb ... Window

Claims (17)

Of the plurality of image data constituting the video data, the first image data, the second image data having a first time length between the first image data, and the first image data Input means for inputting, for each predetermined block, third image data having a second time length longer than the first time length during
A first motion vector between a predetermined reference block in the first image data and a first reference block in the second image data; and a first motion vector in the reference image and the third image data. Detecting means for respectively detecting a second motion vector between the two reference blocks;
Storage means for storing determination data for determining whether or not each of the detected first and second motion vectors is valid;
Discriminating means for discriminating, for each block, whether or not the detected first and second motion vectors are valid using the stored discrimination data;
Normalization means for normalizing the detected first and second motion vectors for each block with respect to time length;
An electronic apparatus comprising: generating means for generating a combined motion vector by combining the first and second motion vectors for each of the blocks determined to be valid and normalized. The electronic device according to claim 1,
The electronic device that normalizes the first motion vector according to the second time length. The electronic device according to claim 2,
The generation unit determines the combined motion vector using the first motion vector when it is determined that both the first motion vector and the second motion vector of the corresponding block are valid. Generate electronic equipment. The electronic device according to claim 3,
The storage means stores a plurality of learning video data and a plurality of learning motion vector data respectively corresponding to the first and second time lengths to be detected from the plurality of learning video data. ,
The electronic device
A motion vector between the reference block and the reference block is detected from the stored plurality of learning video data with the first and second time lengths, and the detected motion vector and the storage An electronic device further comprising learning means for learning whether or not the detected motion vector is valid using the learned motion vector data and generating the discrimination data. The electronic device according to claim 4,
The learning means includes a motion vector detected from the learning video data, a residual signal between the base block and the reference block, an average value and a variance value of pixel data of the base block, and the reference block An electronic apparatus that generates a discriminant function as the discriminant data by performing discriminant analysis using an average value and a variance value of pixel data. The electronic device according to claim 3,
When it is determined that the detected first motion vector is valid, the detection means uses the first reference block when the first motion vector is detected as a reference block and uses the second reference block as a reference block. An electronic device that detects a second motion vector by detecting a third motion vector between the reference block and the detected third motion vector and the first motion vector. The electronic device according to claim 3,
An electronic apparatus further comprising: a determination unit that determines a camera feature in the video data generated by a camera operation based on the generated combined motion vector. The electronic device according to claim 7,
The determination unit continues the determination of the camera feature for a first determination time when the combined motion vector is a combination of the first motion vector for each block, and the combined motion vector is An electronic device that continues the determination of the camera characteristics for a second determination time longer than the first determination time when the second motion vector for each block is synthesized. The electronic device according to claim 8,
The determining means determines that if the combined motion vector is a combination of the first and second motion vectors for each block, any one of the first and second motion vectors is combined. The electronic device continues the determination of the camera characteristics according to whether the first or second determination time is satisfied. The electronic device according to claim 4,
In the predetermined learning image data, the rectangular area is moved by the first and second time lengths, and the image data in the rectangular area is extracted by the first and second time lengths, respectively. An electronic device further comprising learning video data generating means for generating the learning video data necessary for learning whether or not each movement detected as the first and second motion vectors is appropriate. The electronic device according to claim 4,
In the predetermined learning image data, the image in the first rectangular area having the first area is changed to the image in the second rectangular area having the second area different from the first area. The learning is necessary for learning whether or not each expansion or reduction detected as the first and second motion vectors is appropriate by changing and extracting the first and second time lengths, respectively. An electronic device further comprising learning video data generation means for generating video data. The electronic device according to claim 10 or 11,
The learning video data generation means calculates a sum of residual signals for each block between continuous first sample image data and second sample image data in predetermined sample video data, An electronic apparatus that extracts the second sample image data as the learning image data when the sum is equal to or greater than a predetermined threshold. The electronic device according to claim 12,
The learning video data generating means extracts the learning image data using the video data in which the first and second motion vectors are detected as the sample video data. Of the plurality of image data constituting the video data, the first image data, the second image data having a first time length between the first image data, and the first image data And input the third image data having a second time length longer than the first time length for each predetermined block,
A first motion vector between a predetermined reference block in the first image data and a first reference block in the second image data; and a first motion vector in the reference image and the third image data. Each detecting a second motion vector between two reference blocks;
Storing discrimination data for discriminating whether or not each of the detected first and second motion vectors is valid;
Using the stored discrimination data, each block determines whether or not the detected first and second motion vectors are valid,
Normalizing the detected first and second motion vectors for each block with respect to time length;
A motion vector detection method for generating a combined motion vector by combining the first or second motion vector for each of the blocks determined to be valid and normalized. 15. The motion vector detection method according to claim 14, further comprising:
When the composite motion vector is a composite of the first motion vector for each block, the camera feature in the video data generated by the camera operation is determined based on the composite motion vector for a first determination time. Continue to judge,
When the synthesized motion vector is a synthesized second motion vector for each block, the camera feature is determined based on the synthesized motion vector as a second determination longer than the first determination time. A motion vector detection method for continuous determination. Electronic equipment,
Of the plurality of image data constituting the video data, the first image data, the second image data having a first time length between the first image data, and the first image data Inputting, for each predetermined block, third image data having a second time length longer than the first time length during
A first motion vector between a predetermined reference block in the first image data and a first reference block in the second image data; and a first motion vector in the reference image and the third image data. Each detecting a second motion vector between two reference blocks;
Storing discrimination data for discriminating whether or not each of the detected first and second motion vectors is valid;
Using the stored discrimination data to discriminate for each block whether or not the detected first and second motion vectors are valid;
Normalizing the detected first and second motion vectors for each block with respect to time length; and
Generating a synthesized motion vector by synthesizing the first or second motion vector for each of the blocks determined to be valid and normalized. The program according to claim 16, further comprising:
When the composite motion vector is a composite of the first motion vector for each block, the camera feature in the video data generated by the camera operation is determined based on the composite motion vector for a first determination time. A step of continually determining;
When the synthesized motion vector is a synthesized second motion vector for each block, the camera feature is determined based on the synthesized motion vector as a second determination longer than the first determination time. A program for executing the step of determining continuously.

Images