JP3240339B2 - Image capturing device and image processing device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a photographing apparatus and an image processing apparatus capable of obtaining a high quality image with a desired shutter chance.

[0002]

2. Description of the Related Art Conventional technologies are compared in terms of photo opportunity and image quality.

[0003] Although a photograph using a silver halide film has a very good image quality, the photograph is taken at a specific moment. Therefore, when the photograph is developed later, the photographer often misses the moment he wants to take. For example, there is a case where, when a photographer intends to take a picture of a moment when a child smiles and develops it, he / she did not smile because of the time lag.

Further, image processing is performed using silver halide photography,
For example, the expression of the subject's person is changed, but the movement of each part must be predicted by simulating the movement of the person's muscles, making the process extremely complicated and easy to complete. Because it is impossible to perform a natural simulation, the expression becomes unnatural.

[0005] When shooting using a video camera, 1/6
Since photographing is performed every 0 seconds, an image photographed at a desired shutter chance is often included in any one of them. However, since the resolution of a video camera image is generally inferior to that of a photograph of a silver halide film, the image quality is often unsatisfactory even if a shutter chance is desired.

Since a high-definition video camera shoots a high-resolution image every 1/60 second, a high-resolution image can be obtained at a desired photo opportunity.
High definition video cameras and high definition video recorders are very expensive and difficult for ordinary people to use.
Also, high resolution is significantly inferior to silver halide film photographs.

In order to use both a high-resolution camera using a silver halide film and a low-resolution video camera, a photographing apparatus in which the two are simply assembled together to form an integrated camera,
Although disclosed in JP-A-2-61626 and JP-A-1-107267,
Since the image processing that combines the two is not performed, an image with high resolution cannot be obtained at a desired shutter chance.

[0008]

An object of the present invention is to solve the above-mentioned problems of the prior art, and to make it possible to photograph a high-quality image at a desired photo opportunity with an inexpensive device. Also,
Simplify the processing when the image is reproduced later.

[0009]

An image photographing apparatus having a moving image photographing section for photographing a moving image at a low resolution and a still image photographing section for photographing a still image at a high resolution. For specific images in the video,
Recording means for recording a code or data for identifying the specific image from other images in the moving image together with the specific image is provided. Further, the relationship between corresponding pixels between the still image and the image captured at the same timing is recorded.

In order to accurately correspond the pixels of a still image and the pixels of a moving image, it is easy to use the same image sensor. For a still image, all pixels are read, and for a moving image, the number of read pixels is reduced by thinning out pixels or adding a plurality of pixels. This makes it possible to achieve both high image quality of a still image and low power consumption during a moving image.

At the time of reproduction, a low-resolution moving image of a specific period including a still image shot at a high resolution and an image shot at the same timing as the still image is input. In the image processing apparatus that performs the interpolation process, the image to be subjected to the interpolation process, the still image, the image captured at the same timing as the still image in the moving image, and the image to be subjected to the interpolation process Using the image in the moving image captured at the timing, the correspondence of the pixel between the image to be subjected to the interpolation process and the still image is obtained, and the data of the image to be subjected to the interpolation process and the data of the still image corresponding thereto are obtained. Is used to perform an interpolation operation.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to FIGS.
This will be described in detail.

First, a first embodiment of the image photographing apparatus is shown in FIG.
This will be described with reference to FIG.

The image photographing apparatus according to the first embodiment has a camera for exposing a still image on a silver halide film and a camera for exposing a moving image on a CCD as shown in FIGS. 1 is an image photographing device using a rotary shutter,
FIG. 2 is a diagram of an image photographing device using a focal plane shutter, and FIG. 3 is a diagram of an image photographing device in which a mirror is omitted from FIG.

In each of the figures, reference numeral 1 denotes a photographing lens, 2 denotes a silver halide film (hereinafter referred to as a film), and 3 denotes a CCD. 4
Is a half mirror for distributing light to the film 2 and the CCD 3, and the distribution ratio depends on the ratio of the sensitivity of the CCD to the sensitivity of the film. For example, CCD sensitivity is equivalent to ISO100 and film is mainly ISO
If using 100, distribute 1: 1. In this case, when a film of ISO 400 is used, the gain of the CCD is quadrupled. Assuming that ISO400 film is mainly used, the light going to the CCD and the light going to the film are 4:
Let it be 1. When the film sensitivity is ISO100, the CCD gain is reduced to 1/4, and when the film sensitivity is ISO1600, the CCD gain is increased to 4 times. 5 is a CCD3 with different exposure size
This is an intermediate lens for focusing on

As the exposure method, a method of controlling both the film camera unit and the CCD camera unit with a mechanical shutter, and a method of controlling the CCD camera unit with an electronic shutter can be considered. As a method of performing both of them by a mechanical shutter, a method of placing a rotary shutter 7 in front of the half mirror 4 as shown in FIG. 1 can be considered. In other words, since the CCD camera unit shoots a moving image, it must be repeatedly exposed,
A normal shutter cannot be used, and a rotary shutter is used. However, since the film 2 must not be repeatedly exposed, a shutter 8 different from the rotary shutter 7 is required before the film 2. However, the exposure time is controlled by the rotary shutter 7
Therefore, the operation of the shutter 8 does not need to be accurate. In this method, since the exposure control is performed before dividing the optical path, the exposure time of the film 2 and the exposure time of the CCD 3 are completely equal, and their timings are also completely matched.

FIGS. 2 and 3 are views of an image photographing apparatus for controlling the CCD camera section with an electronic shutter. FIG. 3 is a view in which the mirror 6 is omitted from FIG. 2, and 9 is a focal plane shutter. .

The recording medium for recording a still image need not be limited to a film, but may be mounted on a still video camera and recorded on a CCD.

Since the electronic shutter and the focal plane shutter 9 of the CCD camera unit operate independently, it is necessary to devise a method of matching the exposure amount and the timing. FIG. 4 is a perspective view of a focal plane shutter, and FIG. 5 is a CCD.
6 is a timing chart of an electronic shutter of the camera unit.
In FIG. 4, reference numeral 11 denotes a leading curtain, 12 denotes a trailing curtain, and 13 denotes a sensor for detecting the traveling of the focal plane shutter. When the traveling of the leading curtain 11 is detected, the electronic shutter controls the exposure of the CCD camera. It starts and ends the exposure of the CCD camera with the electronic shutter when the travel of the second curtain is detected. As a result, the exposure amount of the CCD and the exposure amount of the film match, and the exposure timing also substantially matches. However, in the case of the focal plane shutter, the exposure timing differs depending on the position of the screen, so that the exposure timing coincides with the position of the sensor in the film screen and the entire screen of the CCD camera.

When a still image is photographed by the above-described method, a moving image is also photographed at the same timing as the still image, and which of the moving images is photographed in such a manner. If the images are compared, it is possible to identify them, but it takes time and effort. Therefore, it is convenient to record a code or data for identification at the time of photographing. Various methods have been proposed for recording data when a still image is a film. For example, optically record by flashing the LED in sync with the film feed on the outside of the screen on the film, apply a magnetic layer outside the screen and record magnetically, apply a transparent magnetic layer on the entire screen Recording magnetically. Any of these methods may be used, and another method may be used. When the still image is a still video camera, a digital signal can be recorded by superimposing a digital signal on an image signal by the DPSK method. In the case of a digital still camera, there is an area for recording related data in a header or the like of an image signal. The data to be recorded is a symbol / number indicating which image in the moving image it is. For this purpose, it is necessary to record the symbols and numbers for each screen in advance on the moving image side. When an analog VTR such as a VHS is used, recording may be performed on a control track. In the case of digital recording using an MO or a semiconductor memory, it may be recorded in the header of the image data or recorded as another file.

Since there is mechanical variation and fluctuation in film feeding, it is not always possible to always expose the same position on the film. Since the CCD camera has an image sensor fixed to the optical system, the positional relationship of exposure is always constant. So CC
In order to know the exact positional relationship between the image of the D camera and the image of the film, it is necessary to record a mark on the film. For example, as shown in FIG. 6, LEDs are provided at the four corners of the screen so that they do not enter the screen in normal printing, and the LEDs emit light and are exposed simultaneously at the time of photographing. Since the LED and the CCD are fixed in the camera, their optical positional relationship is constant. Even if there is a variation in manufacturing, you can know the exact positional relationship by measuring one by one,
It is kept. Therefore, if the data is recorded in the form of a ROM in a camera, and is recorded together with the image in the CCD camera unit, the exact positional relationship between the image in the CCD camera unit and the image on the film can be known later.

It is also conceivable to use an autodate number as one of the markers. Use an LED or a combination of LCD and lamp to write the auto date numbers. Since this is fixed to the camera, the CC corresponding to the position of the number written by this is
Since the position in the D-camera screen can be determined accurately, it can be used for one of the four corner markers described above.

FIG. 7 is a diagram showing the correspondence of positions.
FIG. 7A is a diagram of a film, and FIG. 7B is a diagram of CCD pixels. A rectangle is created at the four corners of the film that has been photographed, and each side is divided according to the number of pixels of the CCD camera. If the dividing points are connected in a grid, each grid point corresponds to each pixel of the CCD. For example, b between a and c on the film corresponds to B between pixels A and C.

Since the screen size is different between the film and the CCD, the optical system after dividing the optical path is different. Since the distortions of the optical systems are different, it is not strictly required to correspond the positions only by the marks at the four corners. For example, a point which divides each side of the quadrilateral into four is added, and a grid as shown in FIG. 8 is formed to obtain a point corresponding to each pixel of the CCD. In this case, the pixels at the left and right ends are outside the range of the CCD screen, so they need to be interpolated.

On the other hand, when accuracy is not required very much, or when the image can be supplemented by image processing at the time of image reproduction / synthesis, the number of mark points can be set to 2 or 3 instead of 4 points. 9A and 9B are diagrams of three points, FIG. 9A is a diagram of a film, and FIG. 9B is a diagram of a pixel of a CCD. In the case of three points, if each side a to b and a to c is divided and a line is drawn from each division point in parallel to the other side to form a grid, D corresponding to an arbitrary d can be obtained. In the case of two points, the side is divided, a line is drawn perpendicularly to the side from the division point, and the line is divided at a rate according to the arrangement of the CCD pixels. In the case of a square array, the line may be divided at the same distance as the distance between the first divided points.

If the two-point or three-point method is applied, it is also possible to determine the correspondence of pixels only with the auto date number. If it is known which pixel of the CCD corresponds to each point of a, b, and c of the date as shown in FIG.
You can create dimensional coordinates to determine which pixels on the film correspond to other pixels on the CCD. In this case, an error may occur at a pixel far from the auto date number. It can be corrected at the time of reproducing and synthesizing the image. This will be described in a later embodiment.

Next, a second embodiment of the image taking apparatus will be described with reference to FIG. The second embodiment is an image capturing apparatus using the same image capturing apparatus for the still image capturing section and the moving image capturing section. FIG. 11A is a diagram of the structure of a still image CCD and the transfer of pixels, and shows the operation of distributing a signal to two horizontal transfer CCDs every other pixel and transferring them. FIG. 11B is a diagram showing the movement of a horizontal transfer CCD of a still image, and shows how the signals of each pixel are transferred independently. FIG. 11 (c)
Is a diagram of the configuration of a moving image CCD and the transfer of pixels, and shows an operation in which all pixels are sent to one horizontal transfer CCD. Figure 1
FIG. 1 (d) is a diagram showing the motion of the horizontal transfer CCD of a moving image, and shows a state in which signals of two adjacent pixels are mixed and then transferred in such an operation. FIG. 11E is a diagram of a vertical transfer clock and a horizontal transfer clock of a moving image. Since there are two vertical transfer clocks before horizontal transfer, before the first vertically transferred signal is sent in the horizontal direction. The signal of the next scanning line is transferred vertically. Therefore, two pixels in the vertical direction are mixed on the horizontal transfer CCD. Since two adjacent pixels are mixed on the horizontal transfer CCD, signals of four pixels are eventually mixed.

In a CCD having the configuration shown in FIG. 11, signals are mixed and read every four pixels in a moving image shooting state, and all pixels are read out independently in a still image shooting state to increase the number of pixels in a still image. In addition, the pixel reading rate in the moving image shooting state can be reduced to save power consumption. If signals for two lines are sent without moving the horizontal transfer CCD, two pixels are mixed on the horizontal transfer CCD. Furthermore, if the sum is added at the output of the horizontal transfer CCD, 4
A mix of pixels. Although the number of electrodes to be transferred is the same, but the time is equivalent to two lines, the readout rate is 1 /
It becomes 2. If only one of the horizontal transfer CCDs is moved and all the pixels are transferred there, four pixels are mixed and read. The reading rate is 1/2,
Since there is one moving horizontal transfer CCD, the power consumption is close to 1/4.

Since there is only one image sensor, the optical system is simple, and there is no need to worry about errors (such as distortion) due to differences in the optical systems between the two image sensors. Further, since each pixel of the moving image is obtained by adding four of each pixel of the still image, the correspondence between the pixels between the still image and the moving image is clear, and it is easy to interpolate the pixels of the moving image later. .

As an image taken at the same timing as a still image, an image to be included in a moving image may be created by adding still image data by four pixels. The addition is performed by software using a CPU or by hardware using an adder.

It is possible to use even a CCD whose reading rate cannot be increased. Even if two horizontal transfer CCDs are used at the rate at which four pixels are added and read, it takes twice as long as one screen of a moving image to read all the pixels independently. In the meantime, the next image pickup and readout cannot be performed, so that the image read out as a still image is added by 4 pixels each, so that two screens in a moving image are obtained. The movement will stop for a moment, but it is only 1/60 second, and it is only a moment, so there is almost no problem when watching the video.

When four pixels are added, the signal amount increases, so that the sensitivity of the imaging unit increases. Since the sensitivity differs between still image shooting and moving image shooting, it is necessary to change the exposure amount or the amplifier gain in each case in order to shoot a normal image in both cases. Since it is difficult to move a mechanical object such as an aperture at a high speed to change the exposure amount, the exposure time is changed by a CCD electronic shutter. Since the gain of the amplifier is electrically changed, a configuration in which the gain can be changed at a high speed is possible. However, when the gain is changed, the SN ratio of an image greatly changes. When handling image data later, it is easier to handle each image data under the same conditions as much as possible. Therefore, it is not desirable that the exposure time and the SN ratio change.
For this purpose, a method called thinning instead of addition is effective. As the configuration of the CCD, one having a drain for discharging electric charges beside each horizontal transfer CCD is used. Then, a signal on a certain line is transferred as shown in FIG. 11A, and all the signals are swept out to the drain. Horizontal transfer CC for the next line
The signal is transferred to D, but this time only one horizontal transfer CCD signal is swept out, and the other CCD signal is transferred and read out by the normal horizontal transfer CCD operation. This is repeated alternately. In this way, only one pixel out of the four pixels is read out, so that the sensitivity is not different from the case where all the pixels are read out independently. Therefore, a still image and a moving image can be captured under the same conditions.

A reproducing apparatus according to a third embodiment is shown in FIGS.
Explanation will be made based on 14.

The reproducing apparatus comprises a still image data input unit, a moving image data input unit, a processing unit, and an output unit. When a still image and a moving image are recorded on the same medium in different formats, the still image data input unit and the moving image data input unit can be used in common. If the still image is in the form of a film or photographic paper, the data can be read by a scanner,
The data read by the scanner as in D is recorded on some medium and supplied. Therefore, in this case, the still image data input section serves as an interface with a scanner or a CD-ROM drive.

In the still image data and the moving image data, data on which image in the moving image data is exposed at the same timing as which still image, and how the pixels of those images correspond, are recorded. I have. The process after the correspondence between the images and the correspondence between the pixels are known will be described below.
First, the corresponding correction of pixels will be described. Even if a mark for pixel correspondence is provided, the correspondence of pixels may be shifted at points far from the mark due to the characteristics of the lens and the like. To realize a high-quality image, it is necessary to correct it. FIG. 12 is a diagram of the correction, FIG. 12 (a) is a diagram of a film screen, and FIG. 12 (b) is a diagram of a CCD screen. For example, an image signal is viewed in the horizontal direction, a portion (edge) where the signal level changes extremely is detected, and if the signal level is different, the pixel is shifted in the horizontal direction so as to match so that the correction is performed. Even the corresponding point of a was A in correspondence of the original pixels, in this case A has no edges, it corrects the point of the edge on the L ₂ to a corresponding point of a. Similarly, when an edge is detected by looking at the image signal in the vertical direction, the pixel is shifted in the vertical direction at that portion to perform correction. If processing is performed using edges, the processing becomes susceptible to noise, so care must be taken to prevent such processing. For example:

With reference to a plurality of lines, only when there is a similar edge at a similar position in not only that line but also the lines before and after that line, it is regarded as an edge (l ₁ , l ₂ , l ₃₎
All have edges, and all of L ₁ , L ₂ and L ₃ also have edges. ), An edge is regarded as an edge only when there is an edge within a certain range of both corresponding images (there is an edge close to A which is the same as the edge of a).

Further, as shown in FIG. 13, the position of the edge of a moving image having a low number of pixels is the center of the pixel between the line whose edge is shifted by one pixel and the center line between the line whose edge is shifted by one pixel. Regard it.

When the correspondence of pixels is obtained in a portion where an edge exists in this way, this is used as a mark for new pixel correspondence, and correction of the intermediate pixels is performed by connecting them with a straight line. Note that these are irrelevant in the case of a method of reading a mix only from the same CCD at the time of moving images.

When the pixels can be associated with each other as described above, the next step is to associate pixels between images at different timings in a moving image. In other words, if an image at a certain timing in a moving image is a desired image, but there is no still image taken at the same timing, pixels are traced using pixel correspondence between different images in the moving image. When an image captured at the same timing as a still image arrives, a signal to be used for interpolation is obtained from the correspondence between the image and the pixel of the still image, and the originally necessary image is interpolated using the signal.

FIG. 14 is a diagram for obtaining a desired image.
14 (a), 14 (b) and 14 (c) are moving images, respectively, and the photographing is performed from (c) to (a). Fig. 14
FIG. 14D is a still image in which the photographing in the state shown in FIG. 14C is desired, but the shutter timing is shifted to become the same image as FIG. 14A. If an image at the timing of (c) is desired, there is no high-quality still image taken at the same timing as in (c), so a high-quality image cannot be obtained as it is. However, a still image (d) photographed at the same timing is used in (a) at a timing shifted by several images from (c), and this is used. That is,
(C) and (b) timing is 1/60 (or 1/50)
Even if the subject is moving, the movement is slight, and the movement can be followed. For example, if an edge in an image is detected and a similar connection is found at a similar position in the two images and there is an edge having a similar signal level, the same subject has moved. It is a method of considering it as something. The similarity is an ambiguous expression, but the criterion is to optimize according to the signal level and noise level of each camera. Alternatively, a method of inputting and specifying the corresponding pixel by the user may be used. Even if not all the pixels are specified, if a few points are specified, the nearby pixels can be inferred from them, so that the operation is not so troublesome.

In this way, if the correspondence between the pixels between (c) and (b) and between (b) and (a) is known, one of the images in (c) is obtained.
Since the correspondence between the pixels 1 to 5 and the pixels 1 to 5 in the image of (a) is known, and the correspondence between the pixels (a) and (d) is originally known, 6 to 9 exist between the pixels 1 to 5. It can be seen that interpolation should be performed with the pixel of. Therefore, pixels 1 to 5 in the image (c) are interpolated by pixels 6 to 9 in (d). Thereby, the image of (c) can be interpolated to achieve high image quality.

[0042]

As described above in detail, according to the present invention, it is possible to obtain a high-quality image with a desired shutter chance at a low cost by a simple process.

[Brief description of the drawings]

FIG. 1 is a diagram of an image processing apparatus using a rotary shutter.

FIG. 2 is a diagram of an image processing apparatus using a focal plane shutter.

FIG. 3 is a view in which a mirror is omitted from FIG. 2;

FIG. 4 is a perspective view of a focal plane shutter.

FIG. 5 is a timing chart of an electronic shutter of the CCD camera unit.

FIG. 6 is a diagram in which marks are recorded on a film.

FIG. 7 is a diagram showing correspondence between pixels at four corners.

FIG. 8 is a diagram in which each side is divided into four to correspond to pixels.

FIG. 9 is a diagram illustrating correspondence between pixels at three points.

FIG. 10 is a diagram illustrating correspondence between pixels using numbers in an auto date.

FIG. 11 is a diagram illustrating transfer of CCD pixels.

FIG. 12 is a diagram for correcting a shifted pixel.

FIG. 13 is a diagram for determining the position of an edge of a moving image.

FIG. 14 is a diagram for obtaining a desired high-quality image.

[Explanation of symbols]

 1 shooting lens 2 film 3 CCD 9 focal plane shutter 13 sensor

Claims (4)

(57) [Claims] 1. An image capturing apparatus having a moving image capturing section for capturing a moving image at a low resolution and a still image capturing section for capturing a still image at a high resolution, wherein a specific one of the moving images captured at the same timing as the still image is specified. An image capturing apparatus, comprising: a recording unit that records a code or data for identifying the specific image from other images in the moving image together with the specific image for the image. 2. A method for recording data relating to a relationship between a plurality of pixels in the specific image and a corresponding position in the still image on a medium for recording the moving image or the still image together with the recording of the image. The image photographing apparatus according to claim 1, wherein: 3. The still image pickup unit and the moving image pickup unit use the same image pickup unit, output signals of all pixels from the still image pickup unit, and output a signal or a signal obtained by thinning out pixels from the moving image pickup unit. 2. The image photographing apparatus according to claim 1, wherein a signal obtained by adding a plurality of pixels is output. 4. A low-resolution moving image of a specific period including a still image shot at a high resolution and an image shot at the same timing as the still image is input, and an image at a certain timing in the moving image is input. In an image processing apparatus that performs an interpolation process, an image to be subjected to the interpolation process, the still image, an image captured at the same timing as the still image in the moving image, and a timing between the image to be subjected to the interpolation process Using the image in the moving image captured in the above, the correspondence of the pixel between the image to be subjected to the interpolation process and the still image is obtained, and the data of the image to be subjected to the interpolation process and the data of the still image corresponding thereto are used. An image processing apparatus for performing an interpolation operation by using an image processing apparatus.