JP2000244797A - Device and method for processing image signal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a still image of high resolution from a plurality of image signals even when a dark object is photographed by a hand-held digital camera. SOLUTION: The image of an object passing through a lens group 1 not provided with an optical axis variable element is supplied to a CCD image pickup element 2 while a shutter 10 is pressed. At this moment, the image is picked up at a shutter speed of 1/1000 sec. The picked up image is supplied to an image processing circuit 3. In the image processing circuit 3, cylinder conversion, conversion of optical axis direction and the aligning of 1/m pixel accuracy are executed to the image signals, the resultant images are stored in an image memory 4 and a plurality of photographed image signals are successively added in real time while aligning reference image signals and present image signals. In a compression circuit 5, compression is executed to supplied synthetic image signals. Further, compressed image signals to which sub data are added are supplied to a recording medium 6.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image signal processing apparatus and method capable of generating one still image from a plurality of image signals.

[0002]

2. Description of the Related Art At present, when a still image is photographed using a camera-integrated digital VTR and a digital still camera having a still image mode (hereinafter, these are collectively referred to as a digital camera), a moving subject It freezes and shoots the moment, and records it.

[0003]

However, as in the case of shooting a moving object, only a momentary light is used when shooting a still object. Therefore, a still image captured by a digital camera is S / N
And the resolution was inadequate.

Further, when combining a plurality of image signals, an optical path difference between the digital camera and the subject occurs in each image. There is a problem that image deformation such as rotation, expansion and contraction, and trapezoidal distortion occurs due to the optical path difference.

[0005] Further, when a night view or the like is photographed by a digital camera on hand, an expensive and heavy variable optical axis element must be used in order to obtain a clear image. if,
Unless the optical axis variable element is used, there is a problem that a clear image cannot be obtained due to camera shake.

Furthermore, recently, the number of pixels of a CCD image pickup device has tended to increase several times, and the problem of pixel shake has become more pronounced. For example, suppose that a subject having a brightness that requires a shutter speed of 1/100 sec is photographed at a telephoto angle of 5 degrees in horizontal angle of view (TELE end). The exposure time at this time is
10 msec. The number of horizontal pixels of the image is 640 pixels (V
Assuming GA (Video Graphics Array), the number of pixels per degree is 640/5 = 128 pixels. Therefore, within 10 msec, only 0.1 degree movement is 0.1 ×
There is a problem that a shake of 128 = 12.8 pixels occurs.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image signal processing apparatus and method capable of obtaining a high-resolution still image from a plurality of image signals even when a hand-held image of a dark subject is taken.

[0008]

According to a first aspect of the present invention, there is provided an image pickup device for sequentially picking up images with a short exposure time which is not affected by camera shake while pressing a shutter;
Cylinder conversion means for converting a captured image from a plane to a cylinder, brightness correction of the cylinder-converted image, a reference image in the cylinder-converted image, and a positional shift within a predetermined range with respect to the reference image A displacement detection unit that detects a displacement with an accuracy of 1 / m pixel between the image having the displacement, and a displacement correction unit that corrects the image according to the detected displacement.
Adding means for adding the image corrected for positional deviation, averaging means for dividing the added image by the number of images added when the shutter is released, and averaging, and converting the averaged image from a cylinder to a plane An image signal processing apparatus comprising:

According to a fifth aspect of the present invention, there is provided an image pickup device for sequentially picking up images with a short exposure time which is not affected by camera shake while pressing a shutter, and an angle of view for picking up a plurality of images. In the case of a relatively small range, an optical axis converting means for converting the plurality of images in the optical axis direction, brightness correction of the optical axis converted image, a reference image in the optical axis converted image, A position shift detecting means for detecting a position shift with an accuracy of 1 / m pixel between an image having a position shift within a predetermined range with respect to the image, and a position for correcting the image according to the detected position shift It is characterized by comprising a shift correcting means, an adding means for adding an image whose position shift has been corrected, and an averaging means for dividing the added image by the added number and averaging when the shutter is released. An image signal processing device.

According to a thirteenth aspect of the present invention, while the shutter is pressed, the image pickup device is driven with a short exposure time which is not affected by camera shake, and the images are sequentially taken, and From the step of converting to a cylinder, the brightness correction of the cylinder-transformed image, in the cylinder-transformed image, between the reference image and an image having a position shift within a predetermined range with respect to the reference image, A step of detecting a position shift with an accuracy of 1 / m pixel, a step of correcting an image according to the detected position shift, a step of adding the image whose position shift has been corrected, and a step of releasing the shutter. And dividing the averaged image by the number of added images and averaging the divided images, and converting the averaged image from a cylinder to a plane.

According to a fourteenth aspect of the present invention, while the shutter is pressed, the image pickup device is driven with a short exposure time that is not affected by camera shake, and images are sequentially taken, and a plurality of images are taken. When the angle of view at the time is in a relatively small range, a step of performing conversion of the plurality of images in the optical axis direction, brightness correction of the optical axis converted image, a reference image in the optical axis converted image, A step of detecting a displacement with an accuracy of 1 / m pixel between an image having a displacement within a predetermined range with respect to the reference image, and a step of correcting the image according to the detected displacement. An image signal processing method comprising the steps of: adding an image whose positional deviation has been corrected; and, when the shutter is released, dividing the added image by the number of added images and averaging.

When a still object is photographed while the direction of the digital camera being held is moved up, down, left and right, a plurality of image signals are photographed at a shutter speed of 1/1000 sec. In order to align the photographed image signal P1 and the image signal P2 photographed in the next frame with an accuracy of 1 / m pixel, a cylinder transformation for transforming a plane into a cylinder is performed, and the magnification is magnified m times. The position detection circuit detects the position of the image signal P2 with respect to the image signal P1. Pixel shift interpolation, image deformation,
Cylindrical conversion and further, pixel-by-pixel positioning of n image signals are performed. The synthesized n image signals are divided by n for averaging, converted from a cylinder to a plane, and further subjected to a high-frequency emphasis filter. These image processes are performed in real time.

[0013]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows an overall configuration of a first embodiment to which the present invention is applied. While the shutter 10 is being pressed, the image of the subject incident via the lens group indicated by 1 is supplied to the CCD image sensor 2. The lens group 1 is subjected to zoom control and focus control by a system controller (system controller) 9.

In the CCD image pickup device 2, incident light from a subject is accumulated as electric charges. On / off of the electronic shutter of the CCD image pickup device 2 is controlled by the system controller 9. Thus, the electronic shutter of the CCD imaging device 2 is driven, and the supplied image of the subject is captured. The captured image of the subject is digitized by an A / D converter (not shown) and supplied to the image processing circuit 3 as a digital image signal (hereinafter, referred to as an image signal).
The image signal supplied to the image processing circuit 3 is temporarily stored in the image memory 4.

In the image processing circuit 3, as described later, a plurality of image signals stored in the image memory 4 are sequentially added in real time while performing positioning. The image processing circuit 3 is controlled by the system controller 9. The synthesized image signal synthesized by the image processing circuit 3 is supplied to the compression circuit 5. The composite image signal supplied to the compression circuit 5 is supplied to the image memory 4 once.

The composite image signal stored in the image memory 4 is subjected to compression processing by a compression circuit 5. As an example, the combined image signal stored as a still image is JP
EG (Joint Photographic Experts Group) is administered. Sub data supplied from the system controller 9 is added to the generated compressed image signal. The sub data is information when an image signal such as a date, a time, a focus state, a shutter speed, an aperture state, the total number, the number of sheets,...

The compressed image signal to which the sub data is added is
The recording medium 6 is supplied. The compressed image signal and the sub data supplied to the recording medium 6 are recorded under the control of the system controller 9. As an example of the recording medium 6, a magnetic tape,
A recording medium appropriately selected from a magnetic disk, a magneto-optical disk, a semiconductor memory, and the like is used.

System controller 9 according to designation from operation key system
, The compressed image signal is read from the recording medium 6. The read compressed image signal is temporarily stored in the image memory 4 via the decompression circuit 7, and is subjected to decompression processing by the decompression circuit 7. That is, the decompression circuit 7 performs JPEG decoding. Further, the sub data separated from the compressed image signal is supplied to the system controller 9.
From the supplied sub data, date, time, focus state, shutter speed, aperture state, total number of sheets,
・ Information such as is read. The expanded image signal is supplied from the expansion circuit 7 to the display circuit 8.

The above-described image memory 4 is used when performing image processing on a plurality of image signals, when performing compression on a composite image signal, and when expanding a compressed image signal. At this time, the area where the image processing is performed, the area where the compression is performed, and the area where the expansion is performed may be divided according to the address, or a flag for image processing, a compression A flag for expansion or a flag for expansion may be added. Further, a memory for image processing, a memory for compression, and a memory for decompression may be separately provided.

The above-described lens group 1 does not include an optical axis variable element that can change the direction of the optical axis. Further, in this embodiment, an angular velocity sensor for detecting camera shake is not used.

The effect of camera shake in a still image is remarkable when the subject is dark, and when the subject is bright, the shutter speed is high, so that there is no significant problem. For example, 1 / 100se
In a case where it is desired to reduce the camera shake to 1/10 when photographing a subject having a brightness that requires the shutter speed c, in this embodiment, the shutter speed is first increased by a factor of 10 to 1/1000 sec.
, The shutter button 29 is kept pressed for 2 seconds, and then released. Then, during this two seconds, 60 image signals are captured, and since each image signal has a high shutter speed,
Shake is reduced to 1/10. However, the brightness of the captured image signal will be somewhat darker and the S / N
Is also bad. Therefore, S / N is performed by performing brightness correction on each of the 60 image signals and performing averaging processing and high-frequency enhancement while performing alignment with an accuracy of 1/1 pixel to 1/8 pixel. , And a sharp image signal can be obtained.

As a first method of photographing an image signal, the image signal is photographed at a shutter speed of 1/1000 sec while the shutter button 29 is pressed as described above without setting the number of image signals to be photographed. There is a way to continue. Second
As a method, while the shutter button 29 is pressed,
The camera continuously shoots image signals at a shutter speed of 1/1000 sec, and performs brightness correction, cylindrical conversion,
After performing registration, averaging and high-frequency emphasis, S / N
There is a method of automatically terminating the photographing when the number of images has been obtained. As a third method, while the shutter button 29 is pressed, image signals are continuously captured at a shutter speed of 1/1000 sec. When an extreme image change is detected in the middle of sequentially captured image signals, There is a method in which only the image signal up to that immediately before is used and the shooting is terminated. At this time, if the S / N is not sufficient, a warning to that effect is issued. In this example, the luminance level of each image signal is used to determine the number of sheets at which the S / N can be obtained.

Here, the above-described image processing circuit 3 will be described with reference to FIG. An image signal supplied from the CCD image sensor 2 is input from an input terminal 21. The input image signal is supplied to the input image memory 22. The input image memory 22 stores a captured current image signal. The buffer memory 23 stores an image signal serving as a reference for one frame (hereinafter, referred to as a reference image signal). As an example, the input image memory 22 and the buffer memory 23 have an 8-bit VGA standard capacity.

The position detection circuit 24 examines the position of the current image signal with respect to the reference image signal of one frame, and also what kind of geometric deformation has occurred. The position detection circuit 24 is provided with the enlargement interpolation circuit 2
5, 27, a position detection circuit 26 for each block and a processing operation circuit 28.

In this example, the buffer memory 23 has
The reference image signal is stored. As this reference image signal,
First, the first image signal is selected. Then, the reference image signal and the current image signal are compared. When the position between the reference image signal and the current image signal is separated by about 30%, the current image signal separated by about 30% is stored as the reference image signal.
That is, the reference image signal is used as a reference image signal for position detection until the current image signal is separated from the reference image signal by about 30%. When the current image signal is separated from the reference image signal by about 30%, the current image signal is used. The image signal is used for position detection as a reference image signal. In this way, each time a current image signal 30% apart from the reference image signal is captured, the current image signal is stored as the reference image signal. Then, the position of the current image signal with respect to the reference image signal is detected.

The current image signal is supplied to the enlargement interpolation circuit 25, and the reference image signal is supplied to the enlargement interpolation circuit 27.
In the enlargement interpolation circuits 25 and 27, the supplied image signal is subjected to a cylinder conversion for converting a plane into a cylinder, and the image signal subjected to the cylinder conversion is enlarged to 1 to 8 times.
Further, since the processing is performed for each small block, the enlargement interpolation circuit 27 only needs to have a block size, and the enlargement interpolation circuit 25 requires a search range.
A size larger than 7 is required. This enlarged interpolation circuit 25
In (27) and (27), the supplied image signal is subjected to cylindrical conversion and then enlarged. However, the image signal may be enlarged and then subjected to cylindrical conversion. The enlarged current image signal and reference image signal are supplied to the position detection circuit 26 for each block.

The position detection circuit 26 for each block detects the position of each block with an accuracy of 1/1 to 1/8 of one pixel. At this time, if the image in a certain block is flat,
Position detection is not possible. Therefore, the variance Va of a block is calculated, and when the variance is small, the block is not used for position detection. The variance Va is calculated as follows: Va = Σ (yi ² ) / K − (− (yi / K)) ² where yi is the luminance value and K is the number of pixels in the block.
Also, when there is an angular velocity sensor 29 as shown in FIG.
When the changes in the directions of the vertical and horizontal optical axes detected by the angular velocity sensor 29 are supplied to the position detection circuit 26, more accurate position detection is performed.

In the processing operation circuit 28, the block number is set to i
The horizontal and vertical translation components are obtained for each block.
The obtained vertical translation component is y [i], and the horizontal translation component is x [i]. When the same value is obtained from x [i] and y [i] for many blocks, it is considered that the entire screen has been translated.
The position data detected by the processing operation circuit 28 has an integer component exceeding one pixel and a decimal component less than one pixel. The integer component of the detected position data is supplied to the output image memory 33, and the decimal component is supplied to the pixel shift interpolation circuit 30. The image deformation coefficient detected by the processing operation circuit 28 is supplied to the image deformation circuit 31.

In the pixel shift interpolation circuit 30, image signals sequentially photographed are supplied from the buffer memory 23 in accordance with the supplied decimal components, and the image signals are subjected to pixel shift interpolation. For example, 3.7
If it is determined that the pixel is shifted in the horizontal direction by the number of pixels, the pixel shift interpolation circuit 30 is supplied with the decimal component 0.7 from the position detection circuit 24. Therefore, the pixel shift interpolation circuit 30 generates the pixel C at a position shifted by 0.7 pixels from the pixel A to the pixel B by the weighted average. In this example, the pixel C is generated from A × (1−0.7) + B × 0.7 = C. In this way, the supplied image signal is shifted by 0.7 pixels, and an image signal shifted by 0.7 pixels is newly generated. The newly generated image signal is supplied from the pixel shift interpolation circuit 30 to the image transformation circuit 31.

The image deformation circuit 31 performs image deformation, for example, rotation, expansion and contraction, and trapezoidal distortion, on the image signal subjected to the pixel shift interpolation in accordance with the supplied image deformation coefficient. In the image transformation circuit 31, the supplied image signal is subjected to a cylinder conversion for converting a plane into a cylinder. The image signal thus subjected to the image transformation is supplied from the image transformation circuit 31 to the addition circuit 32.
If cylindrical conversion is not applied to the image signal, it is good if the angle of view of the finished synthesized image signal is as small as about 10 degrees, but if the angle of view is large, it becomes difficult to perform positioning, and at the same time, the added result is blurred. Occur.

In the addition circuit 32, the image signal from the output image memory 33 and the image signal from the image transformation circuit 31 are added. The added image signal is supplied to the output image memory 33.

In the output image memory 33, the position detection circuit 2
The image signal is written so as to correct the deviation of the integer component supplied from the step S4. For example, if the position detection circuit 24 determines that the pixel is horizontally shifted by 3.7 pixels,
The output image memory 33 is supplied with the integer component 3 from the position detection circuit 24. In the output image memory 33, 3 is shifted in the horizontal direction so as to correct the shift of 3 of the integer component.
An image signal is written so as to be a position shifted by a pixel. That is, the pixel is shifted by 0.7 pixels in the pixel shift interpolation circuit 30, and is shifted by 3 pixels in the output image memory 33.
As a result, the image signal of the next frame which is shifted by 3.7 pixels in the horizontal direction is aligned with the stored image signal. The stored image signal having undergone the alignment and the image signal of the next frame are added by the adding circuit 32 as described above. The vertical and horizontal sizes of the output image memory 33 are several times to several tens times the number of pixels of the CCD image sensor 2.

In this example, 60 image signals can be added in 2 seconds. When the number of added images is 64 or less, the output image memory 33 has a capacity conforming to the 14-bit VGA standard. . The combined image signal obtained by adding all the image signals is supplied from the output image memory 33 to the adding circuit 32 and the dividing circuit 34.

In the division circuit 34, the synthesized image signal obtained by adding the n images is divided by n, and the synthesized image signal is averaged. At this time, since the image is taken while moving the digital camera up, down, left, and right, the combined image signal added to the output image memory 33 exceeds the image size of the CCD image sensor 2. Since the image is taken while being appropriately moved by hand, the number of images to be added in the output image memory 33 differs. Therefore, when the whole is finally divided by the number of sheets, it is divided by a different number of sheets in pixel units. Therefore, since the value of n differs depending on the address, a memory for storing the value of n for each pixel is required. The averaged composite image signal is
From the division circuit 34 to the inverse conversion circuit 35 and the switch circuit 3
6 is supplied to one input terminal.

In the inverse conversion circuit 35, the supplied composite image signal is subjected to an inverse conversion for converting a cylinder into a plane. The composite image signal that has been inversely converted and converted into a plane is supplied from the inverse conversion circuit 35 to the other input terminal of the switch circuit 36. In the switch circuit 36, a composite image signal that has been subjected to inverse conversion for converting a cylinder into a plane or a composite image signal that has not been subjected to inverse conversion is selected.

In this embodiment, when it is known in advance that the angle of view of the finished synthesized image signal is very small, that is, 10 degrees or less, the cylinder formed by the above-described enlargement interpolation circuits 25 and 27 and the image deformation circuit 31 is used. Since there is no need to perform conversion, there is no need to perform inverse conversion in the inverse conversion circuit 35. Therefore, in such a case, the switch circuit 36 selects the synthesized image signal that has not been subjected to the inverse conversion and is supplied from the division circuit 34.

The angle of view of the finished composite image signal is about 6
When it is known in advance that the angle is smaller than 0 degrees, the cylinder transformation is performed in the enlargement interpolation circuits 25 and 27 and the image transformation circuit 31, and the inverse transformation for converting the cylinder into a plane is performed in the inverse transformation circuit 35. Instead, the enlargement interpolation circuits 25 and 27 and the image transformation circuit 31 may directly perform a later-described optical axis direction conversion (hereinafter referred to as an optical axis conversion) on the supplied image signal.
Therefore, in such a case, the switch circuit 36 selects the synthesized image signal that has not been subjected to the inverse conversion and is supplied from the division circuit 34.

Further, when the angle of view of the finished composite image signal is not less than about 120 degrees and the inverse conversion from the cylinder to the plane cannot be performed, the enlargement interpolation circuits 25 and 27 and the image transformation circuit 31 keep the cylinder-converted state. Are supplied to the high-frequency emphasis filter 37. However, the vertical axis, that is, latitude (−9)
Only the conversion from 0 to +90 degrees) to height is performed. This makes it possible to finish the image more easily. Therefore, in such a case, the switch circuit 36 selects the synthesized image signal that has not been subjected to the inverse conversion and is supplied from the division circuit 34.

The composite image signal selected by the switch circuit 36 is supplied to the high-frequency emphasis filter 37. In the high-frequency emphasis filter 37, the supplied synthesized image signal is finished to be a clearer synthesized image signal, and a still image with a good S / N is obtained. As an example, the high-frequency emphasis filter 37 is configured by an HBF (high boost filter). The sharply synthesized image signal is output to an output terminal 38.
Is supplied to the compression circuit 5 via the.

As described above, the image signals to be aligned are still images and have a correlation with each other. Further, noises of a plurality of image signals are random and have no correlation. Therefore, noise is canceled by adding and averaging a plurality of image signals, so that S / N is improved. When performing pixel-shift interpolation and further synthesizing a plurality of image signals, the information of a pixel at a position different from the pixel of the original image signal is stored, so that the resolution is improved.

In this embodiment, the current image signal is stored in the input image memory 22 and the reference image signal is stored in the buffer memory 23. However, the input image memory 22 and the buffer memory 23 are temporally adjacent to each other. Two matching image signals may be stored. In that case, position detection is performed using two temporally adjacent image signals. However, at this time, errors accumulate. Also, if it is known in advance that all the captured image signals do not exceed the range (100%) of the first captured image signal,
The position detection of all the image signals may be performed by using the first image signal as a reference image signal.

The image size of the input image memory 22 and the buffer memory 23 may be set to + α for one input image. For example, when α = 0.2, the number of horizontal pixels is 764 and the number of vertical pixels is 576. The number of bits of the image signals in the input image memory 22 and the buffer memory 23 may be the same as that of the input image. For example, 8
Bit x 3 colors is sufficient.

The image size of the output image memory 33 may be one image of the input image. Output image memory 3
The number of bits of the image signal of No. 3 depends on the number of image signals to be added, and increases by one bit every time the number of images is doubled. For example, if the number of image signals to be added is 16, 12 bits ×
It becomes 3 colors, and if it is 64 sheets, it will be 14 bits × 3 colors, so
If there are 16 bits × 3 color bits, 256 image signals can be added.

At this time, an image is shot while moving the digital camera up, down, left, and right toward a stationary subject by hand. As an example, when 12 image signals have been captured, as shown in FIG.
The second image signal, the second image signal, the third image signal,..., And the twelfth image signal are added and stored. FIG. 3A shows the size of the output image memory 33, and FIG. 3B shows the size output from the CCD image sensor 2. Numerals 1, 2, 3,..., And 12 indicate the order of sequentially captured image signals. From FIG. 3, it can be seen that the position of the image is slightly different since the image was taken while moving the digital camera by hand vertically and horizontally.

Here, a timing chart is shown in FIG. As shown in FIG. 4A, while the shutter 10 is being pressed, as shown in FIG. 4B, an image signal serving as a still image is continuously output from the CCD image sensor 2 every frame. FIG.
As shown in C, the output image signal P1 is stored in the input image memory 22. Then, as shown in FIG. 4D, the image signal P1 is stored in the buffer memory 2 in the next frame.
3 is stored.

The image signal P is stored in the input image memory 22.
2 is stored, and when the image signal P1 is stored in the buffer memory 23, the position detection circuit 24 detects the position of the image signal P1 with respect to the image signal P2. FIG. 4E
As shown in (2), the detection result of the position detection includes an integer component, a decimal component, and an image deformation coefficient. As described above, the integer component is supplied to the output image memory 33, and the decimal component is supplied to the pixel shift interpolation circuit 30. , The image transformation coefficient is supplied to the image transformation circuit 31.

As shown in FIG. 4F, based on the detection result, the pixel shift interpolation circuit 30 and the image transformation circuit 31
In, the image signal is processed. And FIG. 4G
As shown in (3), the processed image is stored in the output image memory 33. At this time, as described above, by controlling the writing of the image signal to the output image memory 33 based on the integer component of the detection result of the position detection, the alignment is performed and a plurality of image signals are synthesized.

When the shutter 10 is released, the taking in of a new image is stopped, the value in the output image memory 33 is divided by the number of added images, and an inverse conversion is performed to convert from a cylinder to a plane. The image signal subjected to the inverse transformation passes through the high-frequency emphasis filter 37 as shown in FIG.

FIG. 5 shows the overall configuration of the second embodiment to which the present invention is applied. The second embodiment is an example in which image processing is performed by software. The same blocks as those in the first embodiment described above are denoted by the same reference numerals, and description thereof will be omitted. The switch circuit 41 selects one of a composite image signal output from the image processing circuit 44 included in the system controller 9 and an image signal from the CCD image sensor 2. The composite image signal or image signal selected by the switch circuit 41 is supplied to the compression circuit 5 and the switch circuit 42.

In the switch circuit 42, the composite image signal or image signal reproduced by the decompression circuit 7 and the switch circuit 4
1 is selected from the composite image signal or the image signal supplied via the control unit 1. The selected composite image signal or image signal is output to an external monitor via the output terminal 43 and is also supplied to the display circuit 8.

A plurality of image signals output from the expansion circuit 7 are supplied to an image processing unit 44 and a data conversion circuit 45. In the image processing unit 44, image processing similar to that of the image processing circuit 3 described above is performed by software. The data conversion circuit 45 outputs the data to an external personal computer (personal computer) via an output terminal 46,
The image signal is converted so that it can be received by a personal computer.

As described above, all the n image signals are recorded on the recording medium 6 simultaneously with the photographing. When image processing is performed by the system controller 9, processing similar to that of the hardware shown in the block diagram of the image processing circuit in FIG. 2 described above is performed, and the processed image signal is recorded again in another area of the recording medium 6. Is done. When image processing is performed by an external personal computer, all image signals are transferred to the personal computer, and the same processing as in the case of hardware is performed, and the result is recorded on a hard disk of the personal computer.

Here, a cylinder conversion for converting an image signal from a plane to a cylinder will be described. For example, when rotating a digital camera fixed on a tripod or the like and photographing a mural painted on a large wall in multiple parts, if the subject in front of the digital camera is used as the basis for cylindrical conversion, subjects other than the front Differs in optical path difference from a digital camera. When an object having a different optical path difference is photographed by a digital camera, the photographed image undergoes deformation such as rotation, expansion and contraction, and / or trapezoidal distortion corresponding to the different optical path difference.

Therefore, in the above-described enlargement interpolation circuits 25 and 27, the supplied image signal is subjected to the cylindrical conversion so that it is on the same plane as the reference image of the cylindrical conversion. By performing the cylindrical transformation, deformation such as rotation, expansion and contraction, and / or trapezoidal distortion of the image caused by the optical path difference is corrected. That is, an image signal having the same optical path difference is generated. Further, even when a reference image obtained by photographing a subject in front of a digital camera is compared with the center of the image and the four corners of the image, an optical path difference occurs.

Therefore, when performing position detection between the reference image signal and the current image signal, the image signal is subjected to cylindrical transformation.
In this digital camera, as shown in FIG. 6B, L (m
h-v plane of the distance m) is adapted to take in the size of L _H × L _V. This light beam is projected onto a cylindrical surface having a radius of L (mm), and this cylindrical surface is simply expanded in a plane to form x-
Let it be the y plane (FIG. 6A). This is projected onto a mesh-like pixel (t, v) of an integer tmax × vmax (FIG. 6C).

When t and v are determined, x and y corresponding thereto are determined.
X = (t− (tmax−1) / 2) · (2 · θ _H / tmax) (1) y = (v− (vmax−1) / 2) · (2 · L _V / vmax) ( 2)

If the longitude P1 is determined as shown in FIG. 6B, P1 = π / 2−x (3)

The latitude Q1 is as follows: Q1 = tan ^-1 (y / L) (4)

Equation (1), Equation (2), Equation (3), Equation (4)
Are the directions (P1, Q) corresponding to an arbitrary pixel (t, v).
1 shows a method for obtaining (1). On the other hand, (P1, Q1)
The coordinates (h4, v4) at which the direction of intersects the hv plane are: h4 = L / tan (P1) v4 = Ltan (Q1) / sin (P1)

The relationship between the coordinates (h4, v4) and the pixel (i, j) of the CCD image sensor is as follows: i = (h4 · tmax / L _H + tmax−1) / 2 j = (v4 · vmax / L _V + Vmax-1) / 2.

These equations are obtained by expanding the coordinates on the CCD image sensor corresponding to the directions (P1, Q1) to real points (i,
j). The direction (P1, Q1) is obtained by giving an integer (t, v), and after the real number (i, j) is obtained, an interpolation operation is performed from some of the integers surrounding the point. It is assumed that the data is a coordinate point (t, v).
Also, as described above, cylindrical conversion is necessary even when the finished composite image signal is wide and cannot be inversely converted to a plane. However, if it is not so wide and it is known that the plane is always returned to the plane, there is no need to perform the conversion twice, such as a cylinder transformation for converting a plane to a cylinder and an inverse transformation for converting a cylinder to a plane. This can be done in one conversion. As an example, optical axis conversion will be described.

As shown in FIG. 7, the viewpoint of the digital camera is set at the origin O, and the coordinate system of x, y, z is determined as shown in the figure. The image plane G is located at the position of the focal length f [m]. The point Q on the subject plane H is projected on the point P on the image. The point C on the subject plane H is the center of the subject, and may be located anywhere on the z-axis.

Dx: pixel size in the x direction [m / pixel] dy: pixel size in the y direction [m / pixel] ξ ': position in the x direction [pixel] η': position in the y direction [Pixel].

Ex: pixel size in the x direction [m / pixel] ey: pixel size in the y direction [m / pixel] ξ ': position in the x direction [pixel] η': y direction Is the position [pixel].

The point P on the image can be expressed as follows.

[0066]

(Equation 1)

The point Q on the subject corresponding to the point P is

[0068]

(Equation 2)

Then, since the points P and Q are on a straight line,

[0070]

(Equation 3)

Solving for k = −f / zq ξ, η from the z coordinate gives:

[0072]

(Equation 4)

To find the point Q from the positions ξ ′ and η ′ on the converted image,

[0074]

(Equation 5)

R is a matrix representing the attitude of the digital camera.

[0076]

(Equation 6)

Here, θ, φ, and ρ are pan angles, tilt angles,
Roll angle. Therefore, when the inverse matrix is obtained,

[0078]

(Equation 7)

Is obtained.

FIGS. 8 and 9 show an example of an optical axis conversion program in C language based on the above theory.
This optical axis conversion program pans to the right by pa [deg], tilts up by qa [deg], and raises ra [de] counterclockwise.
g] Converts an image photographed by rolling onto the same plane as an image signal photographed facing forward. FIG.
Is an image signal of the G1 plane taken in front and pa to the right.
[Deg] is a schematic diagram for explaining an example of converting an image signal of the G2 plane photographed by panning into the same plane.

FIG. 11 shows an example in which the inside of a rectangular parallelepiped room having a lattice pattern wall surface is photographed from one place by using a digital camera in various directions. The horizontal angle of view at this time is 52 [degpp]. FIG. 11A is an image signal obtained by photographing the front with a digital camera. FIG. 11B
Moves the digital camera leftward from the front shown in FIG.
This is an image signal captured by panning 0 degrees. FIG. 11C is an image signal obtained by photographing the digital camera by tilting the digital camera upward by 20 degrees from the front shown in FIG. 11A. FIG. 11D corresponds to FIG.
3A is an image signal obtained by rolling the digital camera clockwise by 20 degrees from the front shown in FIG. FIG. 11E shows FIG.
This is an image signal obtained by panning the digital camera 20 degrees to the left and tilting the digital camera 20 degrees upward from the front shown in FIG. 1A. FIG. 11F shows the digital camera panned left 20 degrees, tilted up 20 degrees, and rotated clockwise by 2 degrees from the front shown in FIG. 11A.
This is an image signal captured by rolling by 0 degrees.

FIG. 12 shows an example of conversion into the same plane as the front image signal shown in FIG. 11A by the optical axis conversion program shown in FIGS. 8 and 9 described above. FIG. 12A corresponds to FIG.
Is an image signal obtained by optical axis conversion of the image signal shown in FIG. FIG.
B is an image signal obtained by optical axis conversion of the image signal shown in FIG. 11C. FIG. 12C is an image signal obtained by optical axis conversion of the image signal shown in FIG. 11D. FIG. 12D is an image signal obtained by optical axis conversion of the image signal shown in FIG. 11E. FIG. 12E is an image signal obtained by optical axis conversion of the image signal shown in FIG. 11F.
The image signal shown in FIG. 12 is not square because the size of the output image is limited to 120%.

As described above, by performing the cylindrical conversion or the optical axis conversion in the above-described enlarged interpolation circuits 25 and 27, the distortion due to the optical path difference can be corrected, so that the position can be correctly detected.

[0084]

According to the present invention, by sequentially photographing at a shutter speed of 1/1000 sec, pixel shake due to camera shake can be suppressed. By cylindrical conversion or conversion in the optical axis direction, image deformation such as rotation, expansion, contraction, or trapezoidal distortion caused by an optical path difference between the digital camera and the subject can be suppressed. By performing positioning with 1 / m pixel accuracy, adding and averaging a plurality of image signals, noise of the image signal is canceled, S / N is improved, and the pixel is different from the pixel of the original image signal. Since the information of the pixel at the position is stored, the resolution can be improved.

Therefore, without using an optical axis variable element such as an active prism or a shift lens, a wide angle of view, such as a ceiling or floor, can be obtained simply by moving a digital camera with a handheld camera while moving up, down, left, and right toward a stationary subject. Can be obtained with high resolution and without the influence of camera shake.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of a camera-integrated digital VTR to which the present invention is applied.

FIG. 2 is a block diagram illustrating an example of an image processing circuit to which the present invention is applied;

FIG. 3 is a schematic diagram illustrating an image signal captured by the digital camera of the present invention.

FIG. 4 is a timing chart for explaining the present invention.

FIG. 5 is a block diagram showing a second embodiment of a camera-integrated digital VTR to which the present invention is applied.

FIG. 6 is a schematic diagram used for explaining a cylinder conversion applied to the present invention.

FIG. 7 is a schematic diagram used for explaining optical axis conversion applied to the present invention.

FIG. 8 is an example of an optical axis conversion algorithm applied to the present invention.

FIG. 9 is an example of an optical axis conversion algorithm applied to the present invention.

FIG. 10 is a schematic diagram used for explaining optical axis conversion applied to the present invention.

FIG. 11 is a schematic diagram used for explaining optical axis conversion applied to the present invention.

FIG. 12 is a schematic diagram used for explaining optical axis conversion applied to the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Lens group, 2 ... CCD image sensor, 3 ...
Image processing circuit, 4 image memory, 5 compression circuit, 6 recording medium, 7 expansion circuit, 8 display circuit, 9 syscon, 10 shutter

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2H059 AA02 AA18 5C022 AA13 AB01 AB68 AC32 AC42 AC69 5C024 AA01 CA11 CA12 DA04 FA01 GA11 HA08 HA17 HA19 HA24 5C053 FA08 FA27 GB36 KA04 KA22 LA01

Claims (14)

[Claims] 1. An image pickup device for sequentially picking up images with a short exposure time that is not affected by camera shake while pressing a shutter, a cylinder converting means for converting the picked-up image from a plane to a cylinder, and a cylinder. The brightness correction of the converted image, and, among the cylindrically converted image, a reference image and an image having a positional shift within a predetermined range with respect to the reference image, with an accuracy of 1 / m pixel. Displacement detection means for detecting the displacement, displacement correction means for correcting the image in accordance with the detected displacement, addition means for adding the image corrected for the displacement, and releasing the shutter Image signal processing comprising: averaging means for dividing the averaged image by the number of images added and averaging the result; and inverse conversion means for converting the averaged image from a cylinder to a plane. apparatus . 2. The method according to claim 1, wherein when the angle of view of the added image is much smaller than a first predetermined value, the processing of the cylindrical conversion means and the processing of the inverse conversion means are not performed. An image signal processing device characterized by the above-mentioned. 3. The image signal processing device according to claim 1, wherein when the added angle of view of the image is larger than a second predetermined value, the processing of the inverse conversion means is not performed. 4. An image signal processing apparatus according to claim 3, wherein said cylindrical conversion means converts the vertical axis from latitude to height. 5. An imaging device for sequentially capturing images with a short exposure time that is not affected by camera shake while the shutter is pressed, and an image sensor for capturing a plurality of images in a relatively small angle of view. An optical axis converting means for converting the plurality of images in the optical axis direction; a lightness correction of the optical axis converted image; a reference image in the optical axis converted image; Displacement detection means for detecting a displacement with an accuracy of 1 / m pixel between an image having a displacement within a predetermined range, and displacement correction for correcting the image in accordance with the detected displacement Means, an adding means for adding the image whose positional deviation has been corrected, and an averaging means for dividing the added image by the added number and averaging when the shutter is released. Image signal processing device. 6. The image signal processing apparatus according to claim 1, further comprising a high-frequency emphasis unit that performs high-frequency emphasis on the image. 7. The image processing apparatus according to claim 1, further comprising: a compression unit configured to compress the image; a recording medium configured to record the compressed image; and an expansion unit configured to expand the image read from the recording medium. An image signal processing device comprising: 8. The image signal processing device according to claim 7, wherein the image picked up by the image pickup element and the image added by the adding means are recorded on the recording medium. 9. The image signal processing device according to claim 7, wherein only the image added in real time by the adding means is recorded on the recording medium. 10. The image signal processing device according to claim 1, wherein m is an integer of 1 or more. 11. The image signal processing device according to claim 1, further comprising a transfer unit configured to transfer the image captured by the image sensor to an external image processing device. 12. The image signal processing device according to claim 1, wherein when the image within the predetermined range is captured, the captured image is stored as a reference image. 13. A step of driving an image sensor with a short exposure time that is not affected by camera shake while pressing a shutter, and sequentially capturing images, and a step of converting the captured image from a plane to a cylinder. And 1 / m pixels between a reference image and an image having a displacement within a predetermined range with respect to the reference image in the cylinder-converted image. Detecting the positional deviation with the accuracy of: correcting the image in accordance with the detected positional deviation; adding the image with the corrected positional deviation; and releasing the shutter when the shutter is released. An image signal processing method, comprising: dividing the averaged image by the number of added images and averaging the image; and converting the averaged image from a cylinder to a plane. 14. While the shutter is being pressed, the imaging device is driven with a short exposure time that is not affected by camera shake, and images are sequentially captured, and the angle of view when capturing a plurality of images is relatively small. Performing a conversion in the optical axis direction of the plurality of images in the case of a small range; correcting the lightness of the images subjected to the optical axis conversion; A step of detecting a position shift with an accuracy of 1 / m pixel from an image having a position shift within a predetermined range, a step of correcting the image according to the detected position shift, An image signal processing method, comprising: a step of adding the image whose displacement has been corrected; and a step of dividing the added image by the number of added images and averaging when the shutter is released.