JP2001024933A - Device and method for inputting image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To effectively correct hand shakes even on various photographing conditions corresponding to the hand shakes caused by the motion of a device and a hand shake vector detected when compressing the moving image. SOLUTION: A device 101 is constituted so that image information provided by picking up the image of an object and recorded. In this case, this device is provided with an image pickup means 103 for a acquiring the image information, a motion detecting means 104 for detecting the motion of the image input device for each frame of image information, a converting means 105 for converting the motion to an image blurring quantity, a correcting means 107 for correcting the image information on the basis of the image blurring quantity, a signal processing means 108 for processing the image information, an encoding means 109 for encoding the image information, a data recording means 111 for recording the image information and a hand shake vector detecting means 110 for detecting the hand shake vector of any arbitrary frame from the image information and on the basis of the hand shake vector, the correcting means 107 corrects the image blurring quantity of a frame following the arbitrary frame.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image input device such as a video camera and an image input method.

[0002]

2. Description of the Related Art Image input devices such as video cameras are widely used in ordinary households along with image recording devices such as video tape recorders (VTRs) and personal computers.

Many of such image input devices are equipped with an autofocus function for automatically adjusting the focus and a camera shake correction function for photographing and recording an image with little blur.

[0004] Among these functions, a technique for realizing a camera shake correction function includes a camera shake detection method and a camera shake correction method.

[0005] As a method of detecting a camera shake, a method of detecting a motion applied to an image input device itself such as a video camera by a camera shake sensor such as an acceleration sensor, a method of calculating an image blur amount from image signals of a plurality of frames, and the like are available. Can be classified.

As a method of correcting camera shake, there is a method of mechanically controlling the direction of a lens or the direction of a variable prism in accordance with the detected camera shake. Calculates the amount of image blur corresponding to the amount of movement on the image sensor from the camera shake, changes the read position of the image signal stored in the memory, and controls the timing of reading signals from the image sensor in accordance with the amount of image blur. Therefore, it can be classified into, for example, those that read out a part of the captured signal.

The camera shake correction function is realized by combining the camera shake detection method and the correction method classified as described above, for example, as shown in FIG. In the case of FIG. 7, the output from the motion sensor is sent to the optical mechanism control unit, and the camera direction is corrected by mechanically controlling the direction of the lens and the direction of the variable prism in accordance with the output.

On the other hand, in recent image input devices, in addition to the analog type which has been performing analog signal processing and recording as in the past, a photographed image signal is converted into digital data to perform digital signal processing and recording. Digital types that perform are becoming widely used. The reasons why this digital type is attracting attention are not only image quality such as resolution and color reproducibility and sound quality, but also information exchange and transmission speed using various digital interfaces, and the use of moving image compression technology. The recording efficiency is high, the image signal is easily processed and edited due to the affinity with a personal computer or the like, or the size of the apparatus is reduced in size and weight.

Some of the above-mentioned digital type image input devices use a motion vector in moving image compression. In this case, if camera shake occurs during shooting, an overall motion vector is generated due to the camera shake,
This causes a reduction in image quality. That is, there is a possibility that the camera shake is detected as a motion vector for all the blocks constituting the image. In the case of an inexpensive, small, and lightweight portable image input device, particularly an image input device that performs moving image compression at a low bit rate, the effect of image quality deterioration due to camera shake is great. Therefore, it is important to have a camera shake correction function for photographing and recording an image with less blur.

[0010]

However, there are some problems with the current camera shake correction function.

For example, depending on the specifications of the optical system of the image input device, the distance from the subject, and the number of pixels and the size of the image pickup device, the magnitude of the image blur amount with respect to the magnitude of the motion of the image input device is determined. The percentage may change,
In this case, there is a possibility that appropriate camera shake correction is not performed. FIG. 8 shows a case where three frames are extracted from the output of the image sensor unit. In the distant view shooting of FIG. 8A and the near view shooting of FIG. The impact on For this reason, if correction processing such as clipping of an image is performed only based on a relationship that has been grasped in advance, the effect of camera shake correction is reduced.

In view of the above, the present invention has a means for detecting a motion of the apparatus and a means for detecting a camera shake vector by using a motion vector detected in a process of compressing a moving image. The purpose is to perform effective camera shake correction under the circumstances.

[0013]

SUMMARY OF THE INVENTION The present invention provides an image input apparatus for compressing and recording image information obtained by imaging a subject, comprising: an image pickup means for acquiring image information; A motion detection unit that detects a motion of the image input device, a conversion unit that converts the motion detected by the motion detection unit into an image blur amount, and an image obtained by the imaging unit based on the image blur amount obtained by the conversion unit. Correction means for correcting the information, signal processing means for processing the image information corrected by the correction means, coding means for coding the image information processed by the signal processing means, and coding means for coding the image information processed by the signal processing means. A data recording unit for recording image information; and a shake vector detection unit for detecting a shake vector of an arbitrary frame from the image information encoded by the encoding unit. To provide an image input apparatus, characterized in that to correct the image blur amount in the next frame of any frame based on the detected hand movement vector detection means.

The correcting means of the image input device may correct the image blur amount of the next frame of the arbitrary frame based on the change amount of the integrated value of the camera shake vector of a plurality of frames up to the arbitrary frame. .

According to the present invention, there is provided an image input method for an image input apparatus for compressing and recording image information obtained by imaging a subject, comprising:
A motion detecting step for detecting a motion of the image input device for each frame of the image information; a converting step for converting the motion detected in the motion detecting step into an image blur amount; A correction step of correcting the image information obtained in the imaging step, a signal processing step of processing the image information corrected in the correction step,
An encoding step of encoding the image information processed in the signal processing step, a data recording step of recording the image information encoded in the encoding step, and an arbitrary frame from the image information encoded in the encoding step. A camera shake vector detecting step of detecting a camera shake vector, and a correcting step of correcting an image blur amount of an arbitrary frame next to an arbitrary frame based on the camera shake vector detected in the camera shake vector detecting step. Provide an image input method.

[0016]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to these embodiments.

First, an embodiment of the present invention will be described. The image input device of the present embodiment has a unit that detects a camera shake from the motion of the image input device, and a unit that detects a camera shake vector using a motion vector detected in the process of moving image compression. The camera shake correction is performed based on the information detected by the means, and the features of the respective camera shake correction methods are utilized.

The image input device according to the present embodiment is shown in the block diagram of FIG.

As shown in FIG. 1, the present image input device 101
Denotes an optical system 102, an image sensor 103 (image pickup means),
Motion sensor unit 104 (motion detection unit), data conversion unit 105 (conversion unit), conversion table unit 106, camera shake correction unit 107 (correction unit), signal processing unit 108 (signal processing unit), encoding Unit 109 (encoding unit) and shake vector detection unit 110 (shake vector detection unit)
, A data recording unit 111 (data recording unit), and a display unit 112.

First, an image of a subject is formed by an optical system 102 using a lens.
The subject formed by the optical system 102 is imaged to obtain image information. As a lens used for the optical system 102, a zoom lens capable of continuously changing the magnification of an optical image of a subject or a fixed focus lens may be used. The image sensor 103 includes an image sensor such as a CCD image sensor and an image sensor driving circuit.

The motion sensor unit 104 includes a camera shake sensor such as an acceleration sensor for detecting camera shake, a sensor output signal processing circuit such as a filter circuit for limiting the band of a sensor output signal, an amplifier circuit, and the like. A reference value holding circuit that holds a signal serving as a reference to be calculated, a sensor output integration circuit for obtaining a moving distance, an angle, and the like using the reference value, and an AD that converts the integration value to digital data
It is composed of a conversion circuit and the like. Then, the imaging element unit 10
For example, the image sensor 103 is installed on a substrate to which the image sensor unit 103 is fixed so as to detect the horizontal and vertical movements, and the movement applied to the image input device 101 is detected.

As the camera shake sensor of the motion sensor unit 104, other sensors such as a speed sensor and an angular speed sensor may be used.

The data conversion unit 105 includes the motion sensor unit 10
4 is for performing data conversion for converting the output from the device No. 4 into an image blur amount. The data conversion unit 105 refers to a plurality of conversion tables provided in the conversion table unit 106 and converts the camera shake sensor data into an image blur amount. The conversion table unit 106 stores the image blur amount for each output of the motion sensor unit 104 in advance as a conversion table, and may be configured by a semiconductor memory or the like.

Here, a method of converting the output of the motion sensor unit 104 into an image blur amount using the conversion table in the data conversion unit 105 will be described in detail with reference to FIG.

In FIG. 2A, the integrated value in the horizontal direction X of the output obtained by the motion sensor unit 104 is used as a reference value X _Ref.
Is quantized in five steps. Further, in FIG. 2B, the integrated value in the vertical direction Y is quantized in five stages around the reference value Y _Ref . Here, the motion sensor unit 10
Although the quantization of the integrated value of the output of No. 4 is linearly performed, non-linear quantization may be performed according to the relationship between the magnitude of the motion of the motion sensor unit 104 and the actual image blur amount.

FIG. 2C shows the amount of image blur (S) with respect to the quantized output (X _n , Y _n ) of the motion sensor unit 104.
x _n , Syn _n ) in advance.
3 is an example of a conversion table. For example, using this conversion table, the output of the motion sensor unit 104 is (X ₀ , Y ₀ )
In the case of, an image blurring amount (Sx ₀ , Sy ₀ ) is obtained. Note that the image blurring amount (Sx ₀ , Sy ₀ ) at this time is (0,
0), the output of the motion sensor unit 104 is (X ₀ ,
In the case of Y ₀ ), it can be expressed that there is no influence of camera shake.
Further, as in the example of FIG. 2D, the image blur amount (S) with respect to the absolute value of the output (X _n , Y _n ) of the motion sensor unit 104 is determined.
x _n, stored in advance as a conversion table Sy _n),
If the sign of the image blur amount is matched to the sign of the output of the motion sensor unit 104, the data to be stored can be reduced to almost half.

With such a configuration, the data conversion unit 105 receives the output from the motion sensor unit 104, refers to a predetermined conversion table of the conversion table unit 106, or prepares a plurality of prepared conversion tables. The image blur amount is output by performing switching or the like. In other words, a conversion table provides a correspondence between the output from the motion sensor unit 104 and the amount of image blur, and data for converting the output from the motion sensor unit 104 to the optimal amount of image blur using this conversion table. The data conversion unit 105 performs the conversion.

In the camera shake correction unit 107, the data conversion unit 1
Based on the image blur amount obtained in step 05, image blur correction is performed on the image information from the image sensor 103. For example, as shown in FIG. 3, all image information from the image sensor 103 is temporarily stored in a memory, and a part of the image information stored in the memory is cut out in accordance with the amount of image blur obtained by the data converter 105. In this way, camera shake correction is performed.

The signal processing unit 108 includes a camera shake correction unit 107
Is for converting the image information subjected to the camera shake correction into an image signal. In the signal processing unit 108, the image information subjected to the camera shake correction is converted into image signals such as a luminance signal and a color difference signal by performing digital signal processing after analog processing.

The encoding unit 109 compresses and expands an enormous amount of image signals processed by the signal processing unit 108. At the time of compression and expansion, H.264 is used for communication images such as videophones and video conferences. H.263, MPEG-1 for storage media images such as CD-ROM, MPEG-2 for broadcast images, and MPEG-4 for communication images on analog telephone lines and mobile communications May be used. In these compression / decompression methods, motion prediction is performed during compression.

The camera shake vector detecting section 110 detects a camera shake vector using a motion vector obtained from an image signal when the coding section 109 performs motion prediction. The motion vector detected from the image signal in the encoding unit 109 is obtained by dividing the image signal into blocks of 8 × 8 pixels or 16 × 16 pixels, determining a search range, and performing block matching. As a method of detecting a camera shake vector from the motion vector obtained in this way, a method using an average value, a median value, or the like of the entire motion vector can be considered.

The camera shake vector of an arbitrary frame detected by the camera shake vector detection unit 110 is used by the camera shake correction unit 107 to correct the image shake amount by adding it to the image shake amount of the next frame. The method of correcting the image blur amount will be described later.

The data recording section 111 records the image signal encoded by the encoding section 109. The data recording unit 111 may be fixed in the image input apparatus 101 or may be constituted by a detachable semiconductor memory, a tape medium, a disk medium, or the like.

The display section 112 is for displaying an image signal. The display unit 112 may be a display such as an LCD monitor, and displays an image signal from the signal processing unit 108 or an image signal obtained by expanding data recorded in the data recording unit 111 by the encoding unit 109.
Note that the display section 112 can be omitted.

The image input device 101 of this embodiment is configured as described above. Here, the image input device 101
A method for correcting the image blur amount of the next frame by the camera shake correction unit 107 based on the camera shake vector detected by the camera shake vector detection unit 110 will be described.

First, consider a case where a camera shake MV (t) occurs in the horizontal direction as shown in FIG. At this time, the image blur amount S (t) in the horizontal direction obtained by the data conversion unit 105 with respect to the output of the motion sensor unit 104 is represented by the camera shake MV.
In frames except when (t) is 0, actual camera shake M
An example when the amplitude is smaller than V (t) by C will be described. In FIG. 4, MV (t), S (t),
V (t) is the camera shake at the time of the frame number t,
The image blur amount and the camera shake vector are shown.

In this case, when the correction in the horizontal direction of FIG. 4A is performed, the actual camera shake MV (t) is corrected by the image blur amount S (t) based on the sensor output.
As shown in (b), the shake vector V (t) remains as a residual shake component. This shake vector V (t)
4C, it can be seen that the center coordinates of the image subjected to camera shake correction have moved, as shown in FIG. For example, if the positive direction in FIG. 4C is a rightward movement, as time elapses, that is, as the frame number t increases,
The center coordinate has moved to the right by a maximum of 5C and has returned to the original position.

Thus, in order to suppress the movement of the center coordinates of the image subjected to the camera shake correction, for example, the camera shake vector V (i) of the frame number i is replaced by the image shake amount S (i + 1) of the next frame number i + 1. Used to correct. This processing will be described with reference to the flowchart of FIG.

First, the camera shake vector V (i) is read (S101). Next, obtained by the data conversion unit 105,
The image blur amount S (i + 1) corresponding to the camera shake MV (i + 1) of the next frame is read (S102). Next, from the camera shake vector V (i) and the image blur amount S (i + 1),
Corrected image blur amount S (i + 1) '= S (i + 1) +
V (i) is obtained (S103).

Thereafter, the camera shake correction section 107 may perform camera shake correction based on the corrected image shake amount S (i + 1) '. As a result, the image blur amount S (t) ′ becomes a value closer to the actual camera shake MV (t) as shown in FIG. 4D, and is a residual camera shake component as shown in FIG. 4E. The shake vector V (t) 'which can be said also has a smaller value. Further, the movement of the center coordinates of the image subjected to camera shake correction is represented by ± C across the original position as shown in FIG.
Can be suppressed.

Next, consider a case where a camera shake MV (t) occurs in the horizontal direction as shown in FIG. At this time, the image blur amount S (t) in the horizontal direction obtained by the data conversion unit 105 with respect to the output of the motion sensor unit 104 is represented by the camera shake MV.
An example will be described in which the amplitude is α times the actual camera shake MV (t) in a frame other than when (t) is 0.

In FIG. 6A, for the same reason as in FIG. 4A, a camera shake vector V (t) remains as a remaining camera shake component as shown in FIG. 6B. Therefore, as shown in FIG. 6C, the center coordinates of the image subjected to camera shake correction move.

Therefore, even in the case as shown in FIG. 6A, by correcting the image blur amount using the flowchart of FIG. 5, as shown in FIG. The amount S (t) ′ is a value closer to the actual camera shake MV (t). In addition, as shown in FIG. 6E, the shake vector V (t) ′, which can be said to be a residual shake component, also has a smaller value. Further, the movement of the center coordinates of the image whose camera shake has been corrected as shown in FIG. 6F can be suppressed to a smaller movement than in the case where the image blur amount is not corrected.

When a motion associated with panning, tilting, or the like is detected, the image blur amount S in FIG.
The process of correcting (i + 1) may be stopped, and the user's intention may be prioritized. Also, when the image blur amount of the next frame is corrected by the camera shake correction unit 107 based on the camera shake vector V (t) detected by the camera shake vector detection unit 110, the camera shake vector detection unit 110 as in the present embodiment. Instead of the detected shake vector V (t) itself, for example, the shake vector V for several frames
The change amount of the integral value of (t) can be used.

[0045]

As described above, the apparatus has means for detecting a camera shake caused by the movement of the apparatus itself and means for detecting a camera shake vector using a motion vector detected in the process of moving image compression. Provided is an image input device capable of performing effective camera shake correction even in various shooting situations by making use of the advantages of the means and compensating for the defects.

[Brief description of the drawings]

FIG. 1 is a block diagram of a basic embodiment of the present invention.

FIGS. 2A, 2B, 2C, and 2D are diagrams for explaining a conversion table; FIG.

FIG. 3 is a view for explaining a method of camera shake correction.

FIG. 4 is a view for explaining correction of an image blur amount using a camera shake vector.

FIG. 5 is a flowchart of a process for correcting an image blur amount.

FIG. 6 is another diagram illustrating correction of an image blur amount using a camera shake vector.

FIG. 7 is a block diagram of a conventional apparatus having a camera shake correction function.

FIG. 8 is a view for explaining the influence of camera shake in distant view shooting and near view shooting.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 101 ... Image input device, 102 ... Optical system, 103 ... Image sensor unit, 104 ... Motion sensor unit, 105 ... Data conversion unit, 106 ... Conversion table unit, 107 ... Camera shake correction unit, 108 ... Signal processing unit, 109 ... Code Conversion unit 110: camera shake vector detection unit 111: data recording unit 112: display unit

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Masafumi Umeda 8th Shinsugita-cho, Isogo-ku, Yokohama-shi, Kanagawa Prefecture F-term in Toshiba Yokohama Office (reference) 5C022 AB55 AC03 AC41 AC51 AC51 AC69 AC79

Claims (3)

[Claims] 1. An image input apparatus for compressing and recording image information obtained by imaging a subject, comprising: an imaging unit for acquiring the image information; and a motion of the image input apparatus for each frame of the image information. Motion detecting means for detecting, converting means for converting the motion detected by the motion detecting means into an image blur amount, and the image acquired by the imaging means based on the image blur amount obtained by the converting means Correction means for correcting information, signal processing means for processing the image information corrected by the correction means, coding means for coding the image information processed by the signal processing means, and coding means Data recording means for recording the image information encoded in the, the camera shake vector detection means for detecting a camera shake vector of any frame from the image information encoded by the encoding means An image input device, wherein the correction means corrects an image blur amount in a frame next to the arbitrary frame based on the camera shake vector detected by the camera shake vector detection means. 2. The method according to claim 1, wherein the correcting unit corrects an image blur amount of a frame next to the arbitrary frame based on a change amount of an integral value of a camera shake vector of the plurality of frames up to the arbitrary frame. The image input device according to claim 1, wherein 3. An image input method in an image input device for compressing and recording image information obtained by imaging a subject, wherein: an image capturing step of acquiring the image information; and the image input for each frame of the image information. A motion detecting step of detecting a motion of the device, and a converting step of converting the motion detected in the motion detecting step into an image blur amount;
A correction step of correcting the image information obtained in the imaging step based on the image blur amount converted in the conversion step, a signal processing step of processing the image information corrected in the correction step, and the signal An encoding step of encoding the image information processed in the processing step; a data recording step of recording the image information encoded in the encoding step; and the image information encoded in the encoding step A camera shake vector detection step of detecting a camera shake vector of an arbitrary frame from
A correcting step of correcting an image blur amount of a frame next to the arbitrary frame based on the blur vector detected in the blur vector detecting step.