JPH089314A - Animation image recording medium, still image extracting device, animation image recorder and still image automatic extracting method - Google Patents

Abstract

PURPOSE:To extract a representative image expressing sufficiently a content of the animation image as a still image automatically by providing a representative image extracting information evaluation section to the device so as to extract the image based on image extract information. CONSTITUTION:A camera operation information acquisition section 2, an image processing information acquisition section 3 and an image pickup state information acquisition section 4 respectively read out camera operation information, image processing information and image pickup information from a reproduced signal inputted by a reproduction signal input section 1. Then a representative image extracting information evaluation section 6 detects video recording start operation information included in the camera operation information to evaluate whether or not each image is proper to a representative image of a cut image as to each image in the same cut image till the succeeding video recording start operation information is detected. As a result, a representative image management section 7 fetches the image of one frame from the video signal acquisition section 5 as to the image evaluated to be proper as a representative image and stores the received image to a representative image storage section 8 and then the image stored in the storage section 8 is extracted and outputted through an output circuit 9.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a still image extracting device for automatically extracting a representative image from moving images captured by a video camera, a moving image recording medium used for the still image extracting device, and The present invention relates to a moving image recording device and a still image automatic extraction method.

[0002]

2. Description of the Related Art As a conventional technique, for example, in a VTR, information for managing a recorded moving image is recorded on a video tape together with the moving image. As an example, a VISS (VHS Index Search System) will be described. VISS was developed for high-speed cueing in a VHS type VTR. The video tape has a control track for recording a VISS signal for performing this high-speed cueing, in addition to a video track for recording normal image information. This VISS signal is used when recording image information on a video tape.
Automatically recorded on the control track. It is also possible to record the VISS signal for the scene that the user wants to see. Using the VISS signal recorded on the video tape in this way, fast-forward reproduction called intro search can be performed. The intro search is a process in which when a VISS signal is found during fast-forwarding, the playback state is maintained for a certain time and then fast-forwarding is repeated until the end of the tape.

[0003]

However, since the VISS signal automatically recorded in the conventional VTR is recorded when the image information is started to be recorded on the video tape, the VISS signal of the intro search is changed. When the image of the attached part is extracted, the extracted image does not sufficiently express the content of the moving image. In addition, if a VISS signal is to be added to a portion that sufficiently expresses the content of a moving image, the user has to do it manually, which takes a great deal of time and effort.

In view of the above points, the present invention is directed to a still image extracting apparatus for automatically extracting a representative image that sufficiently expresses the content of a moving image, a moving image recording medium used for the still image extracting apparatus, and a moving image recording apparatus and still image. An object is to provide an image automatic extraction method.

[0005]

In order to achieve the above object, a moving image recording medium of the present invention is a camera operation information storage in which camera operation information of a camera operated by a photographer at the time of shooting a moving image is stored. A region, an image processing information storage region that stores image processing information obtained by processing a captured image, and a shooting state that stores shooting state information during shooting obtained by processing a signal from a sensor This is a configuration including at least one of the information storage areas.

Further, the still image extracting apparatus of the present invention includes a camera operation information acquisition means for taking in camera operation information of a camera operated by a photographer when capturing a moving image, and an image obtained by processing the captured image. Image processing information acquisition means for capturing processing information, shooting state information acquisition means for capturing shooting state information during shooting obtained by processing a signal from a sensor, camera operation information from the camera operation information acquisition means, and Based on at least one of the image processing information from the image processing information acquisition unit and the shooting state information from the shooting state information acquisition unit, from the time when the photographer performs the shooting start operation to the shooting end operation Still image extraction means is provided for extracting at least one still image from the moving images captured in step 1.

Further, the moving image recording apparatus of the present invention records a moving image captured by the image pickup apparatus on a moving image recording medium, and also records still image information recording means for recording information of the still image extracted by the still image extracting means. Equipped with.

[0008]

With the above structure, the camera operation information such as the zoom of the photographer at the time of photographing, the reliability of the focus and the iris obtained by performing the image processing, the position of the subject, the presence of obstacles, etc. An evaluation is performed based on image processing information and shooting state information such as pan obtained from a sensor, and an image having a high evaluation value or satisfying a certain condition is extracted from the moving image. As a result, the extracted still image sufficiently expresses the content of the moving image.

Further, by recording the information of the extracted still image together with the moving image, it becomes possible to easily retrieve or output the representative image based on the still image information.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows an embodiment of a moving image recording medium of the present invention. Although FIG. 1 shows a video tape as an example of the moving image recording medium, it may be another recording medium such as a video disk or an IC memory.

As shown in FIG. 1, on the video tape, along with the video signal, the extraction information of the representative image is recorded in each area in frame units corresponding to the video signal. Here, the extraction information of the representative image is camera operation information, image processing information, and shooting state information. It goes without saying that the order of recording in each area of the tracks of the video tape is not limited to the order shown in the figure.

[0012] The camera operation is an operation for starting recording or zooming when shooting with a video camera, or an operation for producing a shooting effect such as fade or continuous shooting. It is information indicating a recording start point by setting a flag at the time when the zoom operation is performed, and zoom operation information is information indicating a zoom magnification. The recording start operation information, the zoom operation information, the fade operation information, and the continuous shooting operation information are all information that can be detected based on the button operation of the video camera.

The image processing information is information which is automatically or based on a video signal picked up by an image pickup device and extracted by a human being. For example, the frequency of the high frequency component of the video signal obtained for performing focus control or The size, or the interframe difference value obtained by calculating the difference between the luminance signal and the chrominance signal between frames, or the information such as the position and size of the subject area extracted from the video signal, the state of backlight or over-light, and the contrast Further, it is a γ correction value and color temperature.

The photographing state information is information obtained by detecting the photographing state of the camera by a sensor, for example, camera movement information such as panning detected by an angular velocity sensor, or the light amount of an object detected by a light amount sensor or the aperture of a lens detected by an aperture opening sensor. For example, the opening distance or the focus distance by detecting the lens position, the distance to the subject by a distance measuring sensor, or the like.

By providing the moving image recording medium with the representative image extraction information as described above, it is possible to extract the representative image in the moving image as described in the following embodiments.

Next, a first embodiment of the still image extracting apparatus of the present invention will be described. FIG. 2 shows a block diagram of this embodiment. In this embodiment, information for extracting a representative image together with a video signal is recorded in a moving image recording medium, the extracted information of the representative image is read out from the moving image recording medium and evaluated, and one cut is made based on the evaluation result. A typical image is extracted as a still image from (a group of moving images continuously shot between the recording start operation and the recording end operation in the camera).

In FIG. 2, 1 is a reproduction signal input unit, 2 is a camera operation information acquisition unit, 3 is an image processing information acquisition unit, 4 is a shooting state information acquisition unit, 5 is a video signal acquisition unit, and 6 is a representative image extraction. An information evaluation unit, 7 is a representative image management unit, 8 is a representative image storage unit, and 9 is an output device. The operation of each unit in the above configuration will be described in detail below.

First, the reproduction signal input section 1 reproduces and inputs the information recorded on the moving image recording medium. The camera operation information acquisition unit 2, the image processing information acquisition unit 3, and the shooting state information acquisition unit 4 read camera operation information, image processing information, and shooting state information from the reproduction signal input to the reproduction signal input unit 1, respectively. In this embodiment, each coded information is decoded. The representative image extraction information evaluation unit 6 detects the recording start operation information included in the camera operation information, and for each image in the same cut until the next recording start operation information is detected, the representative image extraction information evaluation unit 6 Based on the information other than the recording start operation information, the image processing information, and the shooting state information, it is evaluated whether each image is an appropriate image as a representative image of cuts. Regarding the image evaluated as suitable as the representative image, the representative image management unit 7 takes in one frame image from the video signal acquisition unit 5 and stores it in the representative image storage unit 8. The output device 9 extracts and outputs the representative image stored in the representative image storage unit 8, and is a display, a printer, or the like. The operation of the representative image extraction information evaluation unit 6 will be described in detail in an example of a still image automatic extraction method described later.

In the above embodiment, the information for extracting the representative image is recorded in advance in the moving image recording medium together with the video signal, and the extraction information of the representative image is read from the moving image recording medium to extract the representative image. The case was explained. However, even if some or all of the information for extracting the representative image does not exist in the moving image recording medium, the information for extracting the representative image is processed by processing the video signal recorded in the moving image recording medium. The representative image can be extracted based on the acquired information. This will be described in detail in the still image extracting apparatus of the second embodiment below.

In the second embodiment, all representative image extraction information is acquired from only the video signal. FIG. 3 shows an example of the configuration of an apparatus that acquires representative image extraction information from a video signal. In FIG. 3, 10 is an inter-frame difference value detection unit, 11 is a memory, 12 is a change amount detection unit, 13 is a cut change detection unit, 14 is a camera work detection unit, 15 is a motion vector detection unit, and 16 is camera work parameter estimation. Reference numeral 17 is a subject information detection unit, 18 is a moving region detection unit, 19 is a region feature amount extraction unit, 20 is a focus information detection unit, 21 is a high-pass filter, 22 is an average value calculation unit, and 47 is contrast information. It is a detection unit. The operation of each unit in the above configuration will be described in detail below.

First, the operations of the inter-frame difference value detecting section 10 and the cut change detecting section 13 will be described. The interframe difference value detection unit 10 includes a memory 11 for delaying the moving image signal by one frame, and a change amount detection unit 12 that obtains a difference between the moving image signals between consecutive frames.
A signal for obtaining a difference between consecutive frames of a moving image uses a luminance value, an rgb value, or the like, and the change amount detection unit 12 performs a difference calculation of image signals between consecutive frames in pixel units to obtain a difference value for each pixel. The sum is calculated and output as the interframe difference value. The cut change detection unit 13 performs threshold processing on the interframe difference value obtained by the interframe difference value detection unit 10. That is, a predetermined threshold value is compared with an inter-frame difference value, and if the inter-frame difference value is larger than the threshold value, it is considered that the image content is greatly changed between the two frames, and the cut change is performed at that portion. Judge that there was. In a video camera, a cut change occurs when a recording start operation is performed, and conversely, the recording start operation can be estimated by detecting the cut change from the image signal. Therefore, the cut change detection unit 13 outputs the recording start operation information when the inter-frame difference value exceeding the threshold value is detected.

The configuration of the inter-frame difference value detection unit 10 shown in FIG. 3 is an example, and another configuration as shown in FIG. 4 may be used. In FIG. 4, reference numeral 44 is a color histogram detection unit for obtaining a color histogram in one frame of the moving image signal, and 45.
Is a histogram memory that stores the obtained histogram,
Reference numeral 46 is a histogram difference detection unit that detects a difference in color histogram between consecutive frames. In the configuration shown in FIG. 4, the comparison is not performed for each pixel between frames but for the entire frame. However, the screen is divided into a plurality of blocks and the difference between the frames is obtained in block units. Good.

Next, the camera work detecting section 14 will be described. First, the operation of the motion vector detection unit 15 will be described. FIG. 5 is a diagram for explaining the position of the detected motion vector on the screen. Figure 5 (a) shows
FIG. 7 is a diagram in which M and N straight lines are arranged in a grid pattern in the vertical direction, and the MN intersections indicate the positions of motion vectors to be detected. Hereinafter, the MN intersections are referred to as grid points, and the i and jth grid points in the horizontal and vertical directions are referred to as grid points (i, j) (1≤i≤M, 1≤j≤N). .

In this embodiment, the motion vector at the grid point position is obtained by selecting a plurality of representative points around each grid point and performing representative point matching. FIG. 5 (b) is an enlarged view of the vicinity of the grid point (i, j) of FIG. 5 (a), where (2 · m + 1) · (2 · n + 1) grid points and their surroundings. The positional relationship of the representative points of is shown. Hereinafter, among the representative points of the grid point (i, j), the k-th and the l-th ones in the horizontal and vertical directions are represented by the representative point (i, j, k, l) (-m ≤ k ≤ m, -n ≦ l ≦ n). As can be seen from FIG. 5 (b), the representative points (i, j,
0,0) is equal to the grid point (i, j).

A method of obtaining a motion vector will be described below with reference to FIG. 6 showing a concrete block diagram of the motion vector detecting section 15. It is assumed that the input of the motion vector detecting unit 15 is a video signal and is set to be input once (r: a predetermined number) in r frames. Here, a certain time t
Image as the image of the 0th frame, and after that, at time (t +
The image of τ) will be called the image of the (30 · τ) th frame.

Now, it is assumed that the image of the Rth frame has been input. The input image is first passed through a bandpass filter in the BPF 23. Here, the value of the image after the BPF processing at the coordinate position (x, y) is set to I (x, y).

On the other hand, the representative point value storage unit 24 stores the value of the representative point of the image before the r frame, that is, after the BPF processing of the (R−r) th frame. That is, the value Y (i, j, k, l) of the representative point (i, j, k, l) is stored.

Y (i, j, k, l) = I (pos_x (i, k), pos_y (j, l)) 1 ≦ i ≦ M, 1 ≦ j ≦ N, −m ≦ k ≦ m,- n ≦ l ≦ n pos_x (i, k): x coordinate of the representative point (i, j, k, l) pos_y (i, k): y coordinate of the representative point (i, j, k, l) Matching unit 25 Inputs the image I (x, y) after the BPF processing from the BPF 23 and the value Y (i, j, k, l) of the representative point before r frames from the representative point value storage unit 24, and performs the representative point matching. The motion vector at each grid point is calculated. That is, for the grid point (i, j), as shown in the following (Equation 1), g and h where K is the minimum are within the range of (2G) (2H) (-G).
≦ g ≦ G, −H ≦ h ≦ H), the motion vector (g, h) is obtained.

[0029]

[Equation 1]

The contents of the representative point value storage section 24 are updated after the processing of the matching section 25 is completed. In particular,
Coordinates pos_x (i, j, k, l) and pos_y (i, j, k, l) of the representative points stored in the representative point position storage unit 26 1 ≦ i ≦ M, 1 ≦ j ≦ N, -m The value at the representative point of the image after the BPF processing of the Rth frame is recorded by using ≤k≤m and -n≤l≤n.

As described above, the motion vector can be obtained from the two images, the input image and the image r frames before.

Next, the camera work parameter estimation unit 16
In, a method of estimating a camerawork parameter from a motion vector will be described.

Camera work that can be estimated from a moving image includes horizontal and vertical changes of the camera (panning and tilting), changes in the angle of view of the camera (zooming), changes in the horizontal / vertical / forward and backward positions of the camera (tracking, Booming, dolling) etc. are considered. In this embodiment, for simplicity, panning, tilting, and zooming are performed.
A method for estimating the type of operation will be described.

First, consider how the points projected on the image pickup surface of the camera move by the above three types of camera work. FIG. 7 is a diagram showing the positional relationship between the imaging surface of the camera and the subject, and the three-dimensional coordinates in the space of the camera are (x, y, z).
, And the two-dimensional coordinates on the imaging surface are represented by (X, Y). Further, the position of the camera is the origin of the three-dimensional coordinates, and the optical axis of the camera is the z axis. The imaging plane is located at z = F (F: focal length), and the coordinates u1 = (x1, y) of an arbitrary point on the subject.
1, z1) is projected onto U1 = (X1, Y1) on the imaging surface. Here, the relationship between the coordinates of the subject and the coordinates on the imaging surface can be expressed by (Equation 2).

[0035]

[Equation 2]

First, using the coordinates of FIG. 7, consider the movement of the coordinates of the object on the image pickup surface by zooming. Figure 8 (a)
Shows zooming caused by a change in focal length. As shown in the figure, the focal length is from F to F '.
, The projection of the subject of u1 is U1 = (X1, Y
Move from 1) to U2 = (X2, Y2). However, from (Equation 2), U2 satisfies U2 = U1 · F ′ / F = f · U1.
However, f = F '/ F.

Similarly, using FIG. 8B, panning,
Consider the case of tilting. Panning and tilting are equivalent to rotating the camera about the y-axis and the x-axis, respectively. When the camera rotates about the x-axis by θx as shown in the figure, the coordinate u1 in the space of the subject moves to u3. However, u3 satisfies (Equation 3).

[0038]

[Equation 3]

Assuming that the rotation angle θx with respect to x is sufficiently small, the coordinate U3 = (X3, Y3) on the image plane after movement is expressed by the relationship of (Equation 2): X3 = X1, Y3 = Y1 + F · θx Relationship is guided. Generalizing this, in the case of a camera operation in which both the x-axis and the y-axis rotate, the relationship before and after the operation of arbitrary coordinates can be expressed as follows.

U3 = U1 + P However, P = (px, py) px, py: rotational components of x-axis and y-axis From the above, the camera operation is different from the general camera operation in which zooming, panning and tilting are combined. It can be seen that the coordinates U1 = (X1, Y1) and U ′ = (X ′, Y ′) before and after the operation satisfy U ′ = f · U + P. Hereinafter, f is called a zoom element and P is called a rotation vector. Therefore, it is understood that the operation amount of the camera can be estimated by obtaining the zoom element and the rotation vector.

A method of estimating the zoom element and the rotation vector from the motion vector obtained by the motion vector detecting section 15 will be described below. Here, with respect to the lattice point (i, j), the position (two-dimensional coordinate) is Ui, j, and the motion vector obtained by the motion vector detection unit 15 is vi, j. Now, when the camera operation of the zoom element f and the rotation vector P occurs, the grid point (i, j) should move to the position of U'i, j (f, P) = f · Ui, j + P. . Therefore, in order to estimate f and P of the actual camera operation, the error E (f, P) = Σ (U'i, j with the actually moved position U ^real i, j = Ui, j + vi, j It suffices to find f and P such that (f, P) -U ^real i, j) ² is minimized. Since the error E is a quadratic equation with respect to f and P, f, which minimizes the error E,
P is uniquely determined as shown in (Equation 4).

[0042]

[Equation 4]

However, <·, ·> indicates an inner product. Therefore, in the camerawork parameter estimation unit 16, the motion vector detection unit 15 detects the motion vector vi, j and the grid point position U.
By inputting i, j and calculating f, P using (Equation 4), each camera work parameter of zooming, panning, and tilting can be estimated.

Next, the operation of the subject information detecting section 17 will be described. The subject information detection unit 17 extracts subject information such as the position, size, and color of the subject while the subject is being tracked by the video camera.
That is, when the camerawork detection unit 14 detects panning and the moving region detection unit 18 can detect a moving region, the in-region characteristic amount extraction unit 19 extracts the characteristic amount in the region from the moving region. . The operation of the moving area detector 18 will be described in more detail.

The motion area detecting unit 18 detects the motion vectors vi, j of M / N lattice points in the screen detected by the motion vector detecting unit 15 and the motion of the camera by the panning detected by the camera work detecting unit 14. The vector Vp is input.
The moving area detection unit 18 extracts grid points that satisfy (Equation 5) and extracts an area different from the motion vector of the camera by panning based on the connection relationship of the extracted grid points.

[0046]

[Equation 5]

The in-region feature amount extraction unit 19 extracts the center of gravity position, area, and color from the moving region detected by the moving region detection unit 18 as the feature amount in the region.

Next, the operation of the focus information detector 20 will be described. The focus information detector 20 is for detecting the out-of-focus state of the image, and is based on the amount of high frequency components of the image. That is, the value of the high frequency component of the video signal becomes small when the image is blurred due to defocus of the lens or the like. Therefore, the high-pass filter 21 takes out the high-frequency component of the image, and the average value calculation unit 22 obtains the average value of the high-frequency component in the entire screen or in the designated area.

Next, the operation of the contrast information detecting section 47 will be described. To detect the contrast information, the brightness histogram detection unit 48 obtains a one-frame brightness histogram, the histogram bias amount detection unit 49 detects the bias of the histogram based on the obtained histogram, and this is used as the contrast information. Here, as an example of the histogram bias, a value having a higher frequency in the low luminance portion and the high luminance portion of the luminance histogram is set. Note that the histogram bias scale is not limited to this, and may be a deviation amount from a typical luminance histogram distribution.

By processing the video signal as described above, the information for extracting the representative image can be obtained. In the present embodiment, the γ correction value, color temperature, backlight or over-illumination state, subject light amount, etc. were not described, but these information can also be obtained by processing the video signal. The structure and method for acquiring the information for extracting the representative image and then extracting the representative image based on the acquired information are the same as those in the first embodiment of the still image extracting apparatus, and the description thereof will be omitted.

The apparatus for acquiring the representative image extraction information described above represents the representative image based on the video signal read from the moving image recording medium when the information for extracting the representative image does not exist in the moving image recording medium. Although the image extraction information is obtained, similarly, the representative image extraction information can be obtained based on the video signal taken in from the image pickup device during photographing by the video camera. Although the configuration in this case is similar to that of FIG. 3 and the description thereof is omitted, it is not necessary to acquire the representative image extraction information from the video signal when the representative image extraction information that can be detected by the sensor included in the video camera exists. Needless to say. Further, the representative image extraction information acquired during shooting with the video camera may be recorded in the moving image recording medium together with the video signal.

FIG. 9 shows the arrangement of an embodiment of the moving image recording apparatus of the present invention. In this embodiment, the video camera is provided with a still image extracting device, and a still image to be extracted as a representative image is determined during shooting by the video camera, a video signal is recorded on a moving image recording medium, and the still image is extracted. The information of is recorded. In FIG. 9, the moving image recording apparatus 27 of this embodiment includes a camera operation information acquisition unit 28, an image processing information acquisition unit 29, a shooting state information acquisition unit 30, a video signal acquisition unit 31, a representative image extraction information evaluation unit 32, and a still image. The image information recording unit 33 and the video signal recording unit 34 are included. The operation of each unit in the above configuration will be described in detail below.

The camera operation information acquisition section 28 is a section for acquiring information such as a recording start operation or a zoom operation when shooting with a video camera, or an operation for producing a shooting effect such as fade or continuous shooting. The recording start operation information is information indicating a recording start point by setting a flag when the recording start operation is performed, and the zoom operation information is information indicating a zoom magnification. The recording start operation information, the zoom operation information, the fade operation information, and the continuous shooting operation information are all detected based on the button operation of the video camera. The image processing information acquisition unit 29 is a part that acquires information obtained by processing a video signal captured by an image sensor, and is, for example, the frequency or magnitude of the high frequency component of the video signal obtained for performing focus control, or the brightness between frames. Difference value between frames obtained from the difference between the signals and color signals, or information such as the position and size of the subject area obtained from the video signal, backlit and overlit conditions, contrast, and γ correction value and color temperature To extract. The shooting state information acquisition unit 30 is a unit that acquires information obtained by detecting the shooting state of the camera with a sensor. For example, camera movement information such as panning detected by an angular velocity sensor, or a subject light amount or aperture opening sensor with a light amount sensor. Acquires the detected aperture of the lens, the focus distance by detecting the lens position, the distance to the subject by a distance measuring sensor, and the like.

The representative image extraction information evaluation unit 32 detects the recording start operation information included in the camera operation information, and the camera operation information for each image in the same cut until the next recording start operation information is detected. Based on the information other than the recording start operation information, the image processing information, and the shooting state information, it is evaluated whether each image is an appropriate image as a representative image of cuts. Regarding the image evaluated as suitable as the representative image, still image information is recorded on the still image information recording medium of the recording medium 35 via the still image information recording unit 33.
Regarding the operation of the representative image extraction information evaluation unit 32,
This will be described in detail in an embodiment of a still image automatic extraction method described later.

The still image information recorded by the still image information recording section 33 will be further described below. The still image information is the still image itself that has been evaluated as suitable as a representative image by the representative image extraction information evaluation unit 32, image processing such as reduction, or storage on a moving image recording medium corresponding to the still image. It is position information or a flag added to a moving image corresponding to a still image.

When the still image information is the still image itself or a reduced image, what is the recording position on the moving image recording medium for recording the moving image captured by the camera by the video signal acquisition unit 31 and the video signal recording unit 34? An image, which is still image information, is recorded at a different position or a recording medium different from a moving image recording medium that records a moving image. For example, when the moving image recording medium is a video tape, only the still image information images are collectively recorded at the beginning portion of the tape or the end portion of the tape, or only the still image information is stored in an IC memory provided separately from the tape. To record.

When the still image information is the storage position information on the moving image recording medium corresponding to the still image, the video signal acquisition unit 3
1 and still image information on a recording medium different from the recording position on the moving image recording medium on which the moving image recorded by the camera is recorded by the video signal recording unit 34 or on a recording medium different from the moving image recording medium on which the moving image is recorded. The storage position information on the moving image recording medium corresponding to the still image is recorded.

When the still image information is a flag added to the moving image corresponding to the still image, recording on the moving image recording medium for recording the moving image photographed by the camera by the image signal acquisition unit 31 and the image signal recording unit 34. The still image information is recorded at the same position as the position. That is, for example, the flag of the still image information is recorded at the beginning of the video signal recorded in 1-frame units.

As described above, the video signal captured by the video camera is recorded in the moving image recording medium, the representative image is extracted from the captured moving image, and the still image information of the representative image is recorded in the recording medium. . This makes it possible to read the recorded still image information and output the representative image to a display or a printer.

Next, an embodiment of the still image automatic extraction method according to the present invention will be described. The still image automatic extraction method is a processing method in the representative image extraction information evaluation unit 6 of FIG. 2 and the representative image extraction information evaluation unit 32 of FIG.

The still image automatic extracting method of the present invention automatically selects a typical image as a still image from the moving images continuously photographed between the recording start operation and the recording end operation. To be extracted. Here, the representative image is an image selected by evaluation based on the intention of the photographer, the state of the image taken, and the state of the subject.

The photographer's intention is reflected in camera work such as zooming and panning. That is, when zooming in, the subject of interest exists on the screen, and is considered to be an important image. Also, when panning, you are moving from one scene to another,
Not considered important. Furthermore, even if you are panning, if there is an object you are tracking,
Considered to be important. In this way, it is desirable to estimate the intention of the photographer from the camera work and extract the important part as the representative image.

The state of the photographed image is a blurred image state when focus control is not properly performed at the time of photographing, a state of excessive forward light or backlight when iris control is inadequate, and gamma correction. Is a state such as when the value is inappropriate, a state where the contrast is poor, or a state where the aperture or focus is being adjusted. These image states can be determined based on information of focus control and iris control at the time of shooting with a video camera, or a γ correction value. Further, even if there is no information on focus control, iris control, γ correction, and contrast, it can be obtained by processing the video signal. It is desirable to evaluate these image states and extract a representative image having a good image state.

The state of the subject means the position and size of the subject being photographed, the presence of obstacles such as when a person crosses the front of the camera during photographing, and the flash flashes during photographing. A case or the like, a state in which a subject is illuminated by a spotlight, or the like. Regarding the position and size of the subject, it is desirable that the position is large in the center of the camera and the area is large, and it is desirable that there is no obstacle. Further, it is desirable not to extract the image when the flash is emitted as the representative image. Further, the image illuminated by the spotlight is the image of interest, and it is desirable to extract it as a representative image.

Here, the method of detecting the position and area of the object has been described in the object information detecting section 17 of FIG. 3 in the third embodiment. Further, regarding the method of detecting the flash and the obstacle, the inter-frame difference value detection unit 10 of FIG.
It is possible to detect based on the inter-frame difference value obtained in. That is, in the flash, the inter-frame difference value changes abruptly, so that the flash can be detected based on a predetermined threshold. In the case of an obstacle, the interframe difference value changes when the obstacle enters the screen and when the obstacle exits from the screen. If the inter-frame difference value again exceeds the threshold value within the time, it can be detected that an obstacle exists in the screen. The irradiation of the spotlight can be detected based on the light amount of the subject.

Based on the knowledge for extracting the representative image as described above, a specific representative image extracting method will be described below. A configuration example of this embodiment is shown in FIG. In FIG. 10, 36 is a weighted addition section, 37 is a gate signal generation section, 38
Is a gate unit, and 39 is a maximum value detection unit. The zoom magnification and the subject information are input to the weighting addition unit 36, and the respective signals are weighted and added. Here, the subject information is information obtained when the subject is being tracked by the camera, and is based on the position and size of the subject at the time of tracking. The subject information has a larger value as the subject position is closer to the center of the camera and as the subject area is larger. The gate unit 38 turns on and off the switch based on the gate signal of the gate signal generation unit 37. The maximum value detecting unit 39 detects the maximum value of the values input from the gate unit 38.

The gate signal generator 37 generates a gate signal based on the high frequency component values of the pan signal and the video signal, the interframe difference value, and the like. A method of generating the gate signal is shown in FIG. FIG. 11A shows a pan signal, which is 0 during panning.
Thus, the signal is 1 when not panning. (B)
Is the high frequency component value of the video signal, and the smaller the value, the more blurred the image. (C) is a signal obtained by thresholding the signal of (b) and binarizing it. When the signal is below the threshold, it is set to zero. (D) is a difference value between frames. (E) thresholds the signal of (d), sets it to 0 when it is above the threshold, and exceeds the threshold again within a predetermined time after the signal of (d) exceeds the threshold. In the case where the value is 0, the processing for setting the interval between 0 to 0 is also performed.

That is, when the inter-frame difference value alone becomes large, it is determined that an image abnormality due to flash or the like has occurred, and it is set to 0 only during the period when the inter-frame difference value is equal to or more than the threshold value. However, as described above, when the obstacle passes in front of the camera, the inter-frame difference value takes a plurality of peaks, so the inter-frame difference value is the threshold value during the period when the obstacle is present in the screen. Even if it is less than or equal to 0, it is set to 0. A gate signal is generated by taking the logical product of the three signals (a), (c), and (e) that are binarized as described above.

As described above, by obtaining the image having the maximum evaluation value with the configuration shown in FIG. 10, the panning period, the image blurring period, and the period in which a flash or an obstacle exists is determined. An image in which the zoom magnification is high and the subject is largely reflected in the center of the screen can be extracted as a representative image from the removed images. The maximum value detection unit 3
The maximum value is detected in 9 even if the whole one cut
Alternatively, it may be a plurality of sections within one cut.

Note that in FIG. 10, one evaluation value is obtained from a plurality of inputs by the configuration of the weighting addition unit 36, the gate signal generation unit 37, and the gate unit 38, but the present invention is not limited to this configuration, and fuzzy inference is possible. It is possible to use a rule based on such rules, or a configuration obtained by a neural network. Further, although the processing of the information regarding the γ correction value, the light amount of the subject, the backlight or over-forwarding condition, the aperture opening, the contrast, and the focus distance is not shown in the configuration of the present embodiment, these signals are also used in the same manner. be able to. That is, the gate signal may be generated so as not to be extracted as the representative image when the γ correction value, the aperture opening value, or the focus distance value is fluctuating, or when the backlight or over-illumination state occurs. Alternatively, the evaluation value may be increased by detecting that the spotlight is emitted from the amount of light of the subject.

FIG. 12 shows another configuration example of the still image automatic extraction method. In this embodiment, the evaluation value is calculated from the input information such as the zoom magnification and the inter-frame difference value, and each evaluation value is weighted and added to obtain the maximum total evaluation value. In FIG. 12, reference numeral 50 is an evaluation value calculation unit that calculates an evaluation value for each input information. For example, when the input information is the zoom magnification, the higher the magnification, the higher the evaluation value is output. If the input information is the inter-frame difference value, the smaller the difference value is, the higher the evaluation value is output. If the input information is contrast, the higher the contrast, the higher the evaluation value is output.

As described above, by obtaining the image having the maximum evaluation value with the configuration shown in FIG. 12, the image is high in contrast and the image is not blurred, and there is a flash or an obstacle. Instead, an image having a high zoom magnification and a large subject in the center of the screen can be extracted as a representative image.

In the above embodiment, when the still image is extracted, the zoom magnification and the size of the subject, which is one of the subject information, are used as independent information. In the method, the still image is automatically extracted by estimating the evaluation value related to the relative size of the subject from the zoom magnification and the subject distance.

The size of the object imaged by the imaging device is proportional to the zoom magnification and inversely proportional to the object distance. Therefore, if the subject being photographed is the same, the relative size of the photographed subject can be estimated from the zoom magnification and the subject distance.

FIG. 13 shows the configuration of the still image automatic extraction method of this embodiment. In FIG. 13, 51 is an evaluation value calculation unit, and 5
Reference numeral 1a is an evaluation value calculation unit that inputs a zoom magnification and an object distance and outputs an evaluation value related to the relative size of the object. Hereinafter, the operation of the evaluation value calculation unit 51a will be described in more detail.

Now, let us say that the maximum value of the zoom magnification is Zmax, and the maximum distance within the target range as the object distance is Dmax. At this time, the minimum distance within the target range as the subject distance is Dmax / Zmax. Here, when the zoom magnification z and the subject distance d are input to the evaluation value calculation unit 51a,
The evaluation value s is obtained based on s = z · Dmax / d.

When d ≧ Dmax, d = Dmax, d ≦ Dm
When ax / Zmax, d = Dmax / Zmax. The evaluation value s thus obtained is a value proportional to the size of the imaged subject.

The method for extracting the representative image from the weighting addition unit 36 and the maximum value detection unit 39 based on the evaluation value obtained by the evaluation value calculation unit 51 as described above is as described in the previous embodiment. Therefore, the description is omitted. The input information used to extract the representative image in this embodiment is only the contrast and the inter-frame difference value in addition to the zoom magnification and the subject distance, but other input information such as a high-frequency component is used. Good.

Further, the object distance is detected by a method of directly detecting it using a distance measuring sensor such as an infrared sensor or an ultrasonic sensor, or by calculating the object distance based on the focus lens position at the time of focusing. There is a way.

In the configuration of the embodiment of the still image automatic extraction method described above, all the moving images photographed between the photographing start operation and the photographing end operation by the photographer are evaluated. Although the representative image is extracted by the photographer, the image may be evaluated after a lapse of a predetermined time after the photographer starts the photographing operation, and the image at the time when the predetermined condition is satisfied may be extracted as the representative image. An example of this configuration will be described below.

The structure of this embodiment is shown in FIG. In FIG. 14, 40 is a timer, 41 is a gate unit, and 42 is an evaluation unit. The timer 40 measures the elapsed time after the shooting start operation is performed, and generates a gate signal to open the gate of the gate unit 41 when a certain time has elapsed from the start of shooting. The evaluation unit 42 evaluates whether the high-frequency component value of the focus that has passed through the gate unit 41 and the inter-frame difference value satisfy the conditions. The condition in the evaluation unit 42 is that the high frequency component value of focus is equal to or higher than a predetermined threshold value and the inter-frame difference value is equal to or lower than the predetermined threshold value. When the condition is satisfied, the evaluation unit 42 stops the subsequent evaluation and extracts a still image at the time when the condition is satisfied as a representative image. In this embodiment, the signals used for evaluation are only the high-frequency component value of focus and the inter-frame difference value, but other signals such as pan and zoom may be used.

Further, the configuration of another embodiment of the still image automatic extraction method will be described. The structure of this embodiment is shown in FIG. 15, and the same parts as those in FIG.
In the present embodiment, evaluation is performed from the number of images obtained by multiplying the number of images captured between the image capturing start operation and the image capturing end operation by a fixed ratio, and the image at the time when a predetermined condition is satisfied is extracted as a representative image. It is a thing. In the configuration of FIG. 15, for images captured between the shooting start operation and the shooting end operation,
The image after the intermediate frame is evaluated. Therefore, the intermediate frame detection unit 43 detects an intermediate frame of frames captured from the shooting start operation to the shooting end operation, and when the intermediate frame is detected, the gate unit 4 is detected.
Generate a gate signal to open the gate of 1. The operations of the gate unit 41 and the evaluation unit 42 are the same as those in the configuration of FIG.

[0083]

As described above, the moving image recording medium of the present invention is provided with the image extraction information, so that the representative image in the moving images recorded in the moving image recording medium based on the image extraction information. Can be easily extracted.

Further, the still image extracting apparatus of the present invention can extract a still image that sufficiently expresses the content of the moving image as a representative image from the moving image based on the image extraction information. It is possible to grasp the content of the moving image in a short time.

Further, the moving image recording apparatus of the present invention records the video signal photographed by the camera in the moving image recording medium, extracts the representative image from the photographed moving image, and obtains the still image information of the representative image. By recording on a recording medium, it is possible to read the recorded still image information and output the representative image to a display or a printer at high speed.

[Brief description of drawings]

FIG. 1 is a diagram showing a moving image recording medium according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a still image extracting apparatus according to the first embodiment of the present invention.

FIG. 3 is a block diagram showing a configuration of a representative image information extracting device in a still image extracting device according to a second embodiment of the present invention.

FIG. 4 is a block diagram showing another configuration of an interframe difference value detection unit in the second embodiment.

5A is a diagram showing grid points for detecting a motion vector in a motion vector detection unit in the still image automatic extraction apparatus according to the second embodiment; FIG. 5B is a grid point (i, j) Enlarged view of the neighborhood

FIG. 6 is a block diagram showing the configuration of a motion vector detection unit in the second embodiment.

FIG. 7 is a diagram showing a positional relationship between an imaging surface of a camera and a subject.

FIG. 8A is a diagram showing a positional relationship between an image pickup surface of a camera and a subject in zooming. FIG. 8B is a diagram showing a positional relation between an image pickup surface of a camera and a subject in tilting.

FIG. 9 is a block diagram showing a configuration of a moving image recording apparatus according to an embodiment of the present invention.

FIG. 10 is a block diagram showing the configuration of a representative image extraction information evaluation unit of the still image automatic extraction method according to the first embodiment of the present invention.

11A to 11E are diagrams showing the operation of the gate signal generator in the still image automatic extraction method according to the first embodiment of the present invention.

FIG. 12 is a block diagram showing the configuration of a representative image extraction information evaluation unit of a still image automatic extraction method according to the second embodiment of the present invention.

FIG. 13 is a block diagram showing a configuration of a representative image extraction information evaluation unit of a still image automatic extraction method according to a third embodiment of the present invention.

FIG. 14 is a block diagram showing a configuration of a representative image extraction information evaluation section of a still image automatic extraction method according to a fourth example of the present invention.

FIG. 15 is a block diagram showing a configuration of a representative image extraction information evaluation section of a still image automatic extraction method according to a fifth example of the present invention.

[Explanation of symbols]

 1 reproduction signal input unit 2 camera operation information acquisition unit 3 image processing information acquisition unit 4 shooting state information acquisition unit 5 video signal acquisition unit 6 representative image extraction information evaluation unit 7 representative image management unit 8 representative image storage unit 9 output device 28 camera Operation information acquisition unit 29 Image processing information acquisition unit 30 Imaging state information acquisition unit 31 Video signal acquisition unit 32 Representative image extraction information evaluation unit 33 Still image information recording unit 36 Weighting addition unit 37 Gate signal generation unit 38 Gate unit 39 Maximum value detection Unit 40 timer 41 gate unit 42 evaluation unit 43 intermediate frame detection unit 50 evaluation value calculation unit 51 evaluation value calculation unit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Office reference number FI technical display location H04N 5/91

Claims (7)

[Claims] 1. A camera operation information storage area in which camera operation information of a camera operated by a photographer when shooting a moving image and image processing information obtained by processing the captured image are stored. A moving image recording medium comprising at least one of an image processing information storage area and a shooting state information storage area in which shooting state information during shooting obtained by processing a signal from a sensor is stored. 2. A camera operation information acquisition means for acquiring camera operation information of a camera operated by a photographer when shooting a moving image, and an image processing information acquisition for acquiring image processing information obtained by processing the captured image. Means, a shooting state information acquisition means for capturing shooting state information during shooting obtained by processing a signal from the sensor, camera operation information from the camera operation information acquisition means, and an image from the image processing information acquisition means. Based on at least one of the processing information and the shooting state information from the shooting state information acquisition means, among the moving images shot between the shooting start operation and the shooting end operation by the photographer. A still image extracting device comprising a still image extracting means for extracting at least one still image from the image. 3. A camera operation information acquisition means for acquiring camera operation information of a camera operated by a photographer when shooting a moving image, and an image processing information acquisition for acquiring image processing information obtained by processing the captured image. Means, a shooting state information acquisition means for capturing shooting state information during shooting obtained by processing a signal from the sensor, camera operation information from the camera operation information acquisition means, and an image from the image processing information acquisition means. Based on at least one of the processing information and the shooting state information from the shooting state information acquisition means, among the moving images shot between the shooting start operation and the shooting end operation by the photographer. A still image extracting means for extracting at least one still image from the still image, a moving image captured by the image pickup device is recorded in a moving image recording medium, and a still image extracted by the still image extracting means. Moving picture recording apparatus comprising: a still image information recording means for recording information of images. 4. A zoom or image pickup of an image pickup device for extracting at least one still image from a moving image shot by a photographer between a shooting start operation and a shooting end operation. At least one of camera operation information such as start operation, image processing information obtained by processing the captured image, and shooting state information during shooting obtained by processing the signal from the sensor is input and still image extraction An automatic still image extraction method characterized by obtaining an evaluation value for each image based on knowledge and extracting an image having a high evaluation value. 5. Camera operation information of an image pickup device for extracting at least one still image from a moving image shot between a shooting start operation and a shooting end operation by a photographer. And at least one of the image processing information obtained by processing the captured image and the shooting state information during shooting obtained by processing the signal from the sensor, and inputting the camera operation information or the image processing information or An automatic still image extracting method, characterized in that an image whose extraction condition information satisfies an extraction condition is extracted. 6. A certain period of time from a shooting start operation in order to extract at least one still image from the moving images taken between the shooting start operation and the shooting end operation by the photographer. For the image after the elapse, the camera operation information such as the zoom of the imaging device, the image processing information obtained by processing the captured image, and the shooting state information during shooting obtained by processing the signal from the sensor 6. The automatic still image extraction method according to claim 5, wherein at least one of the inputs is input, and an image in which the camera operation information, the image processing information, or the shooting state information satisfies an extraction condition is extracted. 7. A shooting start operation to a shooting end in order to extract at least one still image from the moving images taken by the photographer between the shooting start operation and the shooting end operation. An image obtained by processing camera operation information such as the zoom of the imaging device and the captured image for the image captured after the number of images obtained by multiplying the number of images captured by the operation by a certain ratio An image in which at least one of the shooting state information during shooting obtained by processing the signal from the processing information and the sensor is input, and the image in which the camera operation information, the image processing information, or the shooting state information satisfies the extraction condition is input. The still image automatic extraction method according to claim 5, wherein the still image automatic extraction method is performed.