JP2003234968A - Image processor and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that the conventional imager uses a high-cost positioning unit for precisely positioning a plurality of imaging devices to obtain high resolution but takes much time gives influence on the cost. <P>SOLUTION: An image processor and the imaging device are as follows: An object image is divided into a plurality of images and photographed so that divided images have mutually overlapped portions. Using overlapped regions provided in their respective image data, the position deviation of one image from another is detected to do an interpolating operation, and the interpolated image data are combined with previously stored image data read from an image memory means to reproduce a high-resolution image having neither discontinuity of brightness nor geometrical discontinuity. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus using a plurality of solid-state image pickup elements to obtain high resolution. Furthermore, the present invention relates to an image processing device that obtains a high-definition image using a plurality of images.

[0002]

2. Description of the Related Art Generally, an image pickup device using a solid-state image pickup device such as a CCD is widely used in electronic still cameras, video cameras and the like, and it is desired to improve their resolution. The first technology to achieve such high performance is
There are two methods, namely, a method of increasing the resolution of the solid-state image sensor, and a second method of increasing the resolution equivalently by using a plurality of solid-state image sensors and synthesizing each captured image.

The first method is to increase the number of pixels per unit area of the element chip to improve the resolution of the image pickup element itself. That is, the number of pixels is increased in the conventional element chip area by reducing the area of each pixel and increasing the density or integration.

The second method is, for example, Japanese Patent Laid-Open No. 60-24.
There is one proposed in Japanese Patent No. 8079. In this conventional example, as shown in FIG. 45, one optical image is distributed by a prism in four directions, and four image pickup devices for respectively receiving different portions of the distributed optical image are provided. The outputs of the image sensors are combined.

At this time, each image pickup element is positioned so as to pick up an image at a predetermined position as shown by the shaded area in FIG. Further, Japanese Patent Application Laid-Open No. 63-191483 also proposes an image pickup apparatus using four similar image pickup elements. As shown in FIG. 47, the respective image pickup devices are arranged close to each other on the same surface with a certain distance therebetween, and the visual field image (broken line) is intermittently moved to capture an image of the entire visual field. For this movement, the focusing optical system is moved vertically and horizontally. Unlike the one disclosed in Japanese Patent Laid-Open No. 60-248079, no optical path distributing means such as a prism is required.

[0006]

However, the prior art has the following drawbacks. In the first method, the number of pixels per unit area of the element chip is increased to improve the resolution of the image pickup device itself, which is limited in terms of manufacturing technology and sensitivity. Further, although it is possible to increase the number of pixels by increasing the size of the element chip, in the conventional manufacturing technology, the ratio of defective pixels generated in the element chip is also high, the yield is remarkably reduced, and the imaging area is large. It was difficult to obtain a solid-state image sensor.

Japanese Unexamined Patent Application Publication No. 60-
In Japanese Patent No. 248079, when video signals output from four image pickup devices are combined to form a video signal that represents the same image as an input optical image, as described in that embodiment, 4
The positioning accuracy of the two image pickup devices is very strict.

That is, unless one image pickup device and the other image pickup device are arranged in such a positional relationship that they are separated from each other by one pixel, the images will be overlapped or dropped. Further, when they are attached so as to rotate with respect to each other, each area becomes completely discontinuous, resulting in a composite screen that is very difficult to see. At the time of manufacturing, it takes a lot of time and labor to position the image pickup element, which lowers the productivity and affects the cost.

On the other hand, a conventional example, Japanese Patent Laid-Open No. 63-191483
Similarly in the publication, positioning of each element requires extremely high accuracy. Further, in this case, since the condensing optical system is also driven, its driving accuracy is required. Moreover, it is difficult to reduce the size of the product by providing such a driving device,
This leads to higher costs.

As described above, in the conventional example, the positioning accuracy of a plurality of image pickup elements used for increasing the resolution directly affects the image quality. Therefore, an expensive positioning device is introduced for this positioning, and it takes time to obtain the accuracy. Had to do. This increases the cost as it is, and also causes the electronic still camera not to spread. Further, providing a member for moving the light flux is disadvantageous in reducing the size and weight.

Therefore, an object of the present invention is to provide an image pickup device which can form pixels by a certain degree of precise positioning and which has no movable part and has a high resolution at a low cost, and also to form a plurality of images. It is possible to realize an image processing device that can obtain a high-definition image by using the image processing device.

[0012]

In order to achieve the above object, the present invention provides an image processing apparatus for creating one composite image in which each end of a plurality of images is overlapped with each other as overlapping regions. Image storage means for storing the image as image data, displacement amount detection means for detecting a positional displacement amount between the images based on data of an overlap area in the image data read from the image storage means, An image processing apparatus is provided, which includes a combining unit that combines the plurality of images based on the detected amount of positional deviation so as not to cause luminance discontinuity and geometric discontinuity.

Further, the image pickup device is equipped with the image processing device, and further corresponds to an optical path dividing means for dividing an object image into a plurality of images and the number of images divided by the optical path dividing means, and the images overlap each other. And an image pickup unit including an image pickup device that photoelectrically converts each image, and the image storage unit stores the image signal from the image pickup unit. Further, the synthesizing means calculates a rotation amount R and a shift amount S between the plurality of images as conversion coefficients from the image data of the overlap area read from the image storage means, and based on the conversion coefficients,
Interpolation means for interpolating the image data read from the image storage means to generate interpolated image data, and image synthesis for synthesizing the interpolated image data from the interpolation means and the image data read from the image storage means And at least means.

The image pickup processing apparatus having the above configuration detects the amount of positional deviation between images by using the overlap area provided in the image data and performs interpolation calculation. The image data read from the stored image storage means is combined to reproduce a high-resolution image. Further, the image pickup device photoelectrically converts each image by the image pickup element to obtain an image signal so that the images obtained by dividing the subject image into a plurality of pieces by the dividing means have areas overlapping each other.

[0015]

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. First, FIG. 1 shows a schematic configuration of an image pickup apparatus of the present invention, and the outline will be described. In the image pickup apparatus of the present invention, the image pickup lens 6 is formed by the optical path splitting unit 1.
The subject image formed in step 1 is divided into a plurality of images.

As shown in the conceptual diagram of FIG. 1 (b), the image pickup unit 2 composed of a solid-state image pickup device such as CMD8, 9 having a one-to-one correspondence with the number of divided images overlaps each other in part of the area. So as to overlap (overlap), they are arranged in the optical path dividing section 1 with a certain degree of positioning accuracy. Each image signal photoelectrically converted by the image pickup unit 2 is temporarily stored in the image storage unit 3 such as a frame memory.

Next, the displacement amount detection unit 4 positions the pixel position of the image sensor and a predetermined pixel position (pixel position of the screen to be displayed) based on the image signal of the overlap area read from the image storage unit 3. Detect relationships. For example, the pixel value at a predetermined position is calculated by using the correlation value for the pixel value of the image sensor and the "rotation amount R" and "deviation amount S" as conversion coefficients. Based on this conversion coefficient, the pixel position of the image sensor is converted and the predetermined pixel position is specified.

Further, the interpolating section 5 corrects the values of the other predetermined pixels by sequentially using a predetermined pixel value near the predetermined pixel for the pixel values of the plurality of solid-state image pickup devices by interpolation calculation based on the conversion coefficient. As a result, an interpolated image signal is obtained. The interpolated image signal and the image signal from the image storage unit are combined by the image combining unit 7, and the display unit 31 realizes high-resolution image display as a continuous image signal.

2A shows an image pickup area a (M × N pixels) of the image pickup element 8 and an image pickup area b (M) of the image pickup element 9.
3 is a schematic representation of an area (object imaging area) c in which a subject is displayed as an image. The positions of the pixels in the subject imaging area c are indicated by white circles.
Therefore, the object imaging areas a and b are completely covered by the imaging areas of the two imaging elements, and in this case, the pixel position of the imaging area a and the pixel position of the object imaging area c match.

The subject imaging area c is (u + v) ×
It has w pixels and an overlap area (overlap portion) d exists as an overlap portion. That is, the positional relationship (deviation amount and rotation amount) of the areas imaged by the imaging elements 8 and 9 is derived from the image signal of the overlap area d. Here, in order to derive the positional relationship of the subject imaging area c, the positions (d11, d21) indicated by white circles are drawn.
~ D (u + v) w) pixel values are required. For the pixels dij (i = 1 to u, j = 1 to w) in the area c, the pixel values detected by the image sensor 8 are used as they are.

Then, the remaining pixels (dij; i = u + 1)
Up to u + v, j = 1 to w), the values of the white circle pixels are obtained by interpolation from the values of the four pixels indicated by the black circles of the image sensor 9, and the interpolation coefficient has the above-mentioned positional relationship (deviation amount and rotation). Amount). For example, as shown in the area surrounded by the broken line in FIG. 2A (FIG. 2B), the pixel value of the white circle pixel A is the black circle pixels B, C, and It is obtained by interpolation calculation from the pixel values of D and E.

As described above, since the pixel value at the desired position is calculated using the interpolation calculation, the mounting accuracy of the image pickup devices 8 and 9 is such that an appropriate overlap is provided so as to cover the desired image pickup area. It is sufficient, and there is no problem even if it shifts by several pixels. Since it is not necessary to obtain an accuracy of 1 pixel or less as in the conventional example, manufacturing becomes very easy and cost reduction can be realized. Further, an optical member or the like for moving the visual field image is unnecessary.

Next, FIG. 3 shows a specific structure of an image pickup apparatus as a first embodiment according to the present invention, which will be described.

This image pickup device is provided with a half prism 1a in which two right-angle prisms are joined so as to divide the light flux of the subject image formed by the photographing optical system 6 into two. On the upper surface and the back surface of the half prism 1a,
For example, as described above, the two-dimensional solid-state image pickup devices 8 and 9 such as CMD (Charge Modulation Device) are arranged so as to have an overlapping portion in the image pickup area.
It is controlled by the D driver 32.

The image signals output from the CMDs 8 and 9 are amplified by preamplifiers 10 and 11, low-pass filters (LPF) 12 and 13 remove noise components, and A / D converters 14 and 15 digital signals. Is input to the subtracters 16 and 17. The CMDs 8 and 9 previously stored from the FPN storage memories 18 and 19
Fixed pattern noise (Fixed Pattern Noise) is read out, the FPN is subtracted from the image signal by the subtracters 16 and 17, and a signal processing circuit (Signal Procesor; SP) 2
With 0 and 21, γ correction and edge enhancement are performed.

Next, the image signal is transferred to the frame memory 2
2 and 23. These frame memories 22,
The displacement amount detection circuit 24 detects the displacement amount (conversion coefficient R, S) of the overlap area of the CMDs 8 and 9 in the image signal read from 23. Here, the conversion coefficient R represents a rotation matrix due to rotation, and the conversion coefficient S represents a shift vector due to parallel movement.

Based on the conversion coefficients R and S, the interpolation circuit 2
5 interpolates the pixel value read from the frame memory 23, and further, parallel-serial (Parallel-Ser)
ial; PS) conversion circuit 29 performs conversion. The output signal and the output signal read from the frame memory 22 are written in the frame memory 30 as continuous image signals. A display unit 31 that displays the signals read from the frame memory 30 is provided. The CMD driver 32, the FPN storage memories 18 and 19, the frame memories 22 and 23, the interpolation circuit 25, and the PS circuit 29 are controlled by the system controller 33.

Next, a specific circuit configuration of the displacement amount detecting circuit 24 and the interpolating circuit 25 described above is shown in FIG. 4 and will be described. The displacement amount detection circuit 24 includes correlators 24a and 24b and a coefficient calculator 24c. Frame memory 2
The image signals read out from the channels
a and 24b, and after performing the correlation calculation respectively, the coefficient calculator 24c causes the displacement amounts (conversion coefficients R, R) of the overlapping areas of the CMDs 8, 9 (not shown).
S) is detected.

The conversion coefficients R and S output from the displacement amount detecting circuit 24 are stored in memories 26 and 27, respectively. The conversion coefficients R and S read from the memories 26 and 27 are input to the coordinate conversion circuit 35. Also,
Coordinate value X of the position specified by the system controller 33
1 is the coordinate conversion circuit 3 via the coordinate selection circuit 34.
5, the conversion coefficients R and S and the coordinate value X1 are converted into the corresponding coordinate value X2 on the CMD 9 by a predetermined conversion equation (equation (10) described later). This coordinate value X2
Is output to the data reading circuit 36 and the interpolation coefficient calculator 37.

Then, the data reading circuit 36 uses the pixel value vb of the coordinate value obtained based on the coordinate value x2,
vc, vd, ve are read out, and the interpolation coefficient calculator 3
Using the interpolation coefficients a, b, c, and d output from 7, the linear interpolation calculation is performed by the multiplier 39 and the adder 40 of the linear interpolation calculation circuit 38, and the interpolation value va is output. Further, in obtaining the conversion coefficients R and S described above, it is important to set reference points for rotation and translation.

In this embodiment, as shown in FIG.
The center of the overlap area of D8 is used as a reference point for rotation and translation. The overlap area shown corresponds to the area where the CMD8 and CMD9 are exactly overlapped when they are accurately positioned. The positions C1 and C2 indicate the centers of overlap of the CMDs 8 and 9, respectively. That is, the CM
The positional deviation between D8 and CMD9 can be represented by the amount of deviation between positions C1 and C2 (corresponding to conversion coefficient S) and the amount of rotation around position C1 (corresponding to conversion coefficient R).

The conversion coefficients S and R can be obtained from the shift vectors v1 and v2 at the positions P1 and P2 with the position C1 as the center of symmetry in the overlap area, for example. That is, the shift vector v1
, V2 can be expressed by the following equation with reference to FIG. 6 by the shift vector r due to rotation and the shift vector s due to parallel movement.

[0033]

[Equation 1] Therefore, the shift vector s due to the parallel movement is

[Equation 2] The displacement vector r due to rotation is

[Equation 3] It is calculated by the formula. The rotation matrix R is

[Equation 4] And θ is given by (2),

[Equation 5] Here, “l” is a known amount, and “r” is obtained from the equation (4), so that θ is obtained and the rotation matrix R is also obtained.

In this way, the rotation matrix R and the displacement vector S due to the parallel movement are obtained from the displacement vectors v1 and v2 at the positions P1 and P2, and the conversion coefficients R and
S is stored in the memories 26 and 27, respectively.

Various methods have been conventionally proposed for the correlation calculation. Here, as shown in FIG. 7, the reference areas r1 and r2 of the CMD8 are replaced by the search area S of the CMD9.
This is done by detecting the position within 1, S2 where the sum of the absolute values is the minimum. Position x1 thus obtained
, Y1 gives the vector v1 x2, y2 gives the vector v2. This x1, y1, x2, y2
Is given to the coefficient detector 24c (3), (4),
The rotation matrix R and the displacement vector S are obtained by sequentially using the equations (6) and (5). Note that the displacement vector S = (Sx, Sy) here.

Next, referring to FIGS. 3 and 4, the interpolation circuit 2
The operation of No. 5 will be described. In the interpolation circuit 25, the pixel value at the position given by the system controller 33 is read out by the frame memory 23 as 4 pixels.
It is calculated by linear interpolation using one pixel value.

First, the interpolation of pixel values by the linear interpolation calculation will be described with reference to FIG. In the figure, the value va of the pixel A is replaced by the pixel values vb, vc of the pixels B, C, D, E.
, Vd, ve. If the intersections of a line extending vertically through A and BC and DE are F and G, pixel values vf and vg at F and G are BF = DG = m,
FC = GE = n,

[Equation 6] Expressed as If FA = p and AG = q,

[Equation 7] As a result, the pixel value va is obtained. Here, if the inter-pixel distance is “1”, then m + n = p + q = 1.
From equations (7) a, (7) b, and (8),

[Equation 8] As, the value of the pixel value va can be obtained. That is, the pixel value va is immediately obtained from the values of m and p and the four neighboring pixel values.

Next, a method of obtaining m and p will be described.
In this embodiment, the center C1 of the overlap area of the CMD8 is the origin of the coordinates, and the pixel position is represented as x1. Further, with the center C2 of the overlap area of the CMD9 as the origin of the coordinates, as shown in FIG. 9, in order to obtain m and p representing the pixel position by the vector x2,
First, the coordinate value represented by the vector x1 is set to the vector X2.
Must be expressed by (coordinate conversion). Where the vector x
1 = (i1, j1), vector x2 = (i2
, J2), vector x1 and vector x2
Have different coordinate axes. Well, vector x1
, X2 is

[Equation 9] Can be expressed as R ⁻¹ represents the rotation of “−θ”. If these are expressed as components,

[Equation 10] Becomes That is, the coordinates represented by (i1, j1) on CMD8 are

[Equation 11] It is represented by. Since this (i2, j2) is a real number and corresponds to giving the coordinates of the pixel A in FIG. 8, m, p
Is

[Equation 12] Given in. Here, (int) means integerization.
Similarly, the coordinates of the pixels B, C, D and E are

[Equation 13] Given as.

In the image pickup apparatus of the first embodiment having the above-described structure, the conversion coefficients R and S are calculated at the time of manufacturing the image pickup apparatus and once stored in the memories 26 and 27 of the interpolation circuit 25, It is not necessary to detect again by the displacement amount detection circuit 24, and the stored conversion coefficients R and S may be read each time. Therefore, a general photographer needs the displacement amount detection circuit 24.
It is not used, but is usually removed at the factory and used for shooting.

In general photographing, when a subject to be photographed with high resolution is placed in the viewfinder and photographing is performed, the image signal is stored in the frame memories 22 and 23.
The operation of reading (u + v) × w pixels d11 to d (u + v) w shown in FIG. 2 from the frame memories 22 and 23 and outputting them to the frame memory 30 will be described. Here, the image pickup area of the image pickup device 4 of (u + v) × w pixels shown in FIG. 2 corresponds to the image pickup area of the CMD 8 shown in FIG. 3, and similarly, the image pickup area of the image pickup device 5 corresponds to the image pickup area of CMD9. And

That is, from the pixel d11 to the pixel dij (i
= 1 to u, j = 1 to w) are the same as the pixels on the CMD8, and the corresponding pixels are read. Remaining di
For j (i = u + 1 to u + v, j = 1 to w), the pixels of the CMD9 are interpolated and used. That is, the desired pixel dij (i = u + 1 to u + v, j = 1 to w) is supplied from the system controller 33 to the coordinate selection circuit 34 in the interpolation circuit 25. In the coordinate selection circuit 34, the CMD8
The coordinate value is output to the coordinate conversion circuit 35 as a coordinate value x1 centered on the overlap area of.

In the coordinate conversion circuit 35, the coordinate value x
1 and the coordinate value x2 on the CMD9 from the conversion coefficients R and S stored in the memories 26 and 27, respectively ((1
It is calculated based on the equation (0). In the data reading circuit 36, the coordinates of four neighboring pixels are calculated from the coordinate value x2 by using the equation (12), and the corresponding pixel values vb, vc, vd,
ve is read from the frame memory 23.

Further, in the interpolation coefficient calculator 37, the coordinate value x
From equation (2), m and p are calculated by the equation (11), and the interpolation coefficient a,
Calculate b, c, d. Then, the linear interpolation calculation circuit 38
Then, vb, vc, vd, ve and a, b,
The calculation based on the equation (9) is performed from c and d, the pixel value va of dij is calculated, and is written to a predetermined address of the frame memory 30 via the PS circuit 29, and the reproduced image is displayed on the display unit 31. Further, in the case where the output is always performed while performing the interpolation calculation, the frame memory 30 may be omitted.

As described above, in the first embodiment, by using the interpolation calculation, high resolution image pickup can be performed even when the plurality of image pickup elements are not accurately aligned. Further, since it is not necessary to perform highly accurate alignment, it can be manufactured at a low cost, and particularly, since no mechanical moving part is required, it is useful for downsizing and weight saving of the imaging device.

Further, in the first embodiment, the displacement amount detection circuit 24 is attached at the time of manufacture, the conversion coefficient is stored in the memory, and then the displacement amount detection circuit 24 itself is removed. However, a connector or the like is provided in the camera body. , It may be removable.
Further, as the interpolation calculation, a linear interpolation calculation using four neighboring pixels is used, but the invention is not limited to this, and a higher-level interpolation calculation such as spline interpolation or Sinc interpolation may be used.

Next, FIG. 10 shows the structure of an image pickup apparatus as a second embodiment according to the present invention, which will be described. In the above-described first embodiment, (9-b), (1
0), (11) and (12) are used for the calculation,
The amount of calculation becomes considerably large. Therefore, the second embodiment is performed every imaging (9-b), (10), (1).
It is configured so that the calculations of the expressions 1) and (12) are omitted.

That is, instead of the displacement amount detection circuit 24 and the interpolation circuit 25 in the first embodiment, the interpolation coefficient writing circuit 28 and the interpolation circuit 25a are used in the second embodiment. Here, the constituent members of the second embodiment shown in FIG. 10 that are the same as the constituent members of FIG. 3 are given the same reference numerals, and descriptions thereof will be omitted.

The interpolation coefficient writing circuit 28 is configured to have the functions of the displacement amount detection circuit 24, the coordinate conversion circuit 35, the interpolation coefficient calculator 37, and the data address detector 41 described above. In 35 (1
The calculation of the equation (0) is performed by the interpolation coefficient calculator 37 by (1
1) and (9-b) are calculated by the data address detector 4
In 1, the calculation of equation (12) is performed.

The coordinate value of the pixel used for interpolation detected by the data address detector 41 is stored in the data address memory 42, and the interpolation coefficients a, b, c, d calculated by the interpolation coefficient calculator 37 are: Coefficient a memory 43 and coefficient b, respectively
It is stored in the memory 44, the coefficient c memory 45, and the coefficient d memory 46. In the interpolation circuit 25a, the coordinate selection circuit 3
4, a data read circuit 36b, and a linear interpolation calculation circuit 38.

As described above, in the second embodiment, the coordinate conversion calculation (Equation (10)), the interpolation coefficient calculation calculation (Equation (11), (9) b), and the coordinate calculation (Eq. (12)
Expression) is performed at the time of manufacturing, and the data of the calculation results are stored in the memories 42 to 46. Therefore, in the actual photographing, only the calculation of the equation (9-a) performed by the linear interpolation calculation circuit 38 is performed.

As described above, in the second embodiment, by providing the data address memory and the coefficient memory in the interpolation circuit, the amount of calculation is drastically reduced, and the image signal processing by continuous image pickup is sufficient. Can be implemented. Also in the second embodiment, as in the first embodiment, the interpolation coefficient writing circuit 28 may be attached only at the time of manufacturing to calculate and record data, and then may be detached or detached.

Next, FIGS. 11 and 14 show the structure of an image pickup apparatus as a third embodiment according to the present invention, which will be described. Here, in the constituent members of the third embodiment shown in FIG. 14, the same members as the constituent members of FIG. 3 are designated by the same reference numerals, and the description thereof will be omitted.

In the first and second embodiments described above, the incoming light image is divided by the half prism 1. However, if the half prism is used, the light amount is half wasted. There is.

Therefore, in the third embodiment, a coating is applied to a part of the prism as an optical path splitting portion, as shown in FIG.
A semi-transmissive prism whose transmittance is changed as shown in (a) and (b) is used. FIG. 11A shows an image of a subject on the optical axis in the optical path splitting unit (the image is on the optical axis).
The light flux in the case is shown, and the shaded portion represents the light flux reflected by the reflection area of the prism. Further, FIG. 11B shows a light flux which forms an image at the end of the CMD 9 when a subject other than the optical axis is photographed.

Since the light quantity is proportional to the area of the exit pupil of the lens in this way, when a uniform light flux is imaged in the imaging area as shown in FIG. 11A, the light quantity distributions of the CMDs 8 and 9 are As shown in FIG. That is, the ratio of the amount of light received by the CMDs 8 and 9 differs depending on the imaging position, and the shape is symmetrical with respect to the optical axis position. In addition,
At the optical axis position, the amount of light becomes equal to that when a half prism is used.

However, as shown in FIG. 11B, the subject is not always on the optical axis in normal photographing,
The light amount distribution is different in the imaging area, especially in the overlap area. A good displacement amount cannot be detected from this different light amount distribution, and is detected with an error. Further, since there is an influence such as uneven brightness on the image, it is necessary to correct the light amount distribution.

Therefore, in the third embodiment, the light amount correction circuits 47 and 48 as shown in FIG. 14 are provided. The light amount correction circuits 47 and 48 amplify the image signal according to the imaging area, as shown in the reciprocal light amount distribution of FIG. 12 shown in FIG. 13, and correct the light amount so that the light amount distribution becomes uniform. Further, the prism 1b is a semi-transmissive prism having different transmittances shown in FIG. Further, the light amount correction circuit 4
Reference numerals 7 and 48 may be lookup tables.

As described above, in the third embodiment, the prisms having different transmittances are used as the optical path splitting means to reduce the loss of the light quantity, which is suitable for photographing a dark subject.

Next, FIG. 15 shows the structure of an image pickup apparatus as a fourth embodiment according to the present invention, and FIG. 16 shows a separator lens instead of the prism of the optical path splitting unit which is a characteristic part of this embodiment. The configuration when used will be shown and described. Here, in the constituent members of the fourth embodiment shown in FIG. 15, the same members as the constituent members of FIG. 3 are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 16 shows a light beam which is imaged on each of the image pickup devices CMD8 and CMD9 by using the separator lens 1c. The figure (a) shows the case where the subject (object point) is on the optical axis, and the figures (b) and (c) show the case where the object point is not on the optical axis, respectively. CM
The light flux that forms an image at the lower end of D8 is shown. in this way,
It can be seen that the images are formed so that they overlap each other. In addition, the separator lens 1c and the CMD8,
A light-shielding plate 50 is provided between the light-shielding plate 9 and the light-shielding plate 9 so that the divided light fluxes do not mix.

Further, the light quantity distribution shown in FIG.
Similar to the light amount distribution of, the light amount distribution changes when the object point is not located on the optical axis. That is, the light quantity correction circuits 47 and 48 are required as in the previous embodiment. Further, in FIG. 15, the separator lens 1c and the light shielding plate 50 shown in FIG. 16 are used. From the above, in the fourth embodiment, since the separator lens is used in place of the prism for dividing the optical path, the optical path dividing portion can be easily miniaturized.

Next, FIG. 17 shows the arrangement of an image pickup apparatus according to the fifth embodiment of the present invention, which will be described. Here, in the constituent members of the fifth embodiment shown in FIG. 17, the same members as those of FIG. 3 are designated by the same reference numerals, and the description thereof will be omitted. In the above-described first embodiment, as shown in FIG. 2, pixels dij (i = (u + 1) to (u + v),
j = 1 to w) are interpolated to calculate their pixel values, and the other pixels use the signals themselves received by the CMD8. Therefore, when the pixel value is obtained using the interpolation calculation, there is some image deterioration, and the right half and the left half of the screen may have different resolutions. Therefore, the fifth
In this embodiment, the entire screen has a uniform resolution.

In this embodiment, as shown in FIG. 18, the subject imaging area is tilted equally with respect to the areas imaged by the CMD8 and CMD9. That is, CMD8 and CMD9
However, if they differ by an angle θ, the imaging area is CM
D8 and CMD9 are set at different positions by θ / 2. Then, the pixel (dij; i =
CMD8 for 1 to (u + v) / 2, j = 1 to w)
Pixels on the right side of the broken line (dij; i = (u
+ V) / 2 + 1 to u + v, j = 1 to w)
A pixel value is calculated by interpolation using MD9 pixels.

That is, when the image pickup apparatus is held horizontally, the rotation mechanism section 49 of the image pickup element is provided for correction when the subject image pickup area is tilted with respect to the horizontal and vertical directions. The rotation amount of this rotation mechanism portion is determined according to the outputs R and S of the displacement amount detection circuit 24. Further, in the present embodiment, another interpolation circuit 25 is provided to perform the interpolation processing on the output of the image signal of the CMD8.

As described above, in the fifth embodiment, an image having the same resolution can be obtained on the entire screen by tilting the image pickup area evenly from the image pickup areas of the two image pickup elements. The rotation mechanism unit 4 used in this embodiment
9 may be used in each of the above-described first to fourth embodiments.

Next, FIG. 19 shows the arrangement of an image pickup apparatus as a sixth embodiment of the present invention, which will be described. here,
The same members as those of FIG. 3 in the members of the sixth embodiment shown in FIG. 19A are designated by the same reference numerals and the description thereof will be omitted.

In the sixth embodiment, as the image pickup device, as shown in FIG. 19B, the number of pixels is 1000 ×.
It is configured by using four 400 CMDs 51, 52, 53, 54. The number of pixels of each image sensor is a general-purpose NT
Since it is almost the same as the image pickup device for SC, it can be manufactured with a much higher yield than the image pickup device for HDTV (1920 × 1035). Then, as shown in FIG. 19B, the CMDs are arranged side by side with the CMD51 as a reference, and the CMD51,
52, 53, 54 are overlap areas a, b,
It is arranged on the half prism 1d so as to have c.

Then, the displacement amount detecting circuit 24 calculates each CND from the image signals of the three overlapping areas a, b and c.
The displacement amount (deviation amount S, rotation amount R) is detected and the conversion coefficient is output to each interpolation circuit. Although the half prism 1d is used as the optical path splitting unit 1 in this embodiment, prisms having different transmittances may be used as in the third embodiment.
Further, an interpolation coefficient memory may be provided inside the interpolation circuit 25 as in the second embodiment.

Next, FIGS. 20A to 20D show the structure of the optical path splitting unit of the image pickup apparatus as the seventh embodiment of the present invention, and will be described. In FIG. 20, only the characteristic parts of the seventh embodiment are shown, and the rest of the configuration is equivalent to that of the sixth embodiment. In the sixth embodiment described above, the half prism 1d is used and four image pickup devices are used, but in the seventh embodiment, another optical path splitting unit will be described. For the area imaged by this optical path splitting unit, see FIG.
3 shows.

FIG. 20A shows two wedge prisms 6.
By combining 0,61 and beam splitter 63,
In this example, the optical path of the imaging lens is branched into four, and an image is formed on the CMDs 55 to 58. Further, FIG. 21A shows a state of the optical path splitting portion of FIG. 20A viewed from the side, and FIG. 21B shows a state viewed from directly above. The wedge-shaped prism bifurcates the optical path into right and left parts, and the subsequent prism bifurcates the optical path into upper and lower parts. As a result, the optical path is divided into four parts. The prism is made by joining right-angle prisms, and the upper half of the joining surface is coated with a total reflection mirror.

Next, in FIG. 20B, the wedge prisms 60 and 61 described above are replaced with a pair of decentered lenses 64 and 65. In the wedge prisms 60 and 61,
The only function was to bend the optical path, but the decentering lens 64,
The imaging lens 6 has an image forming function in addition to the function of bending the optical path, and the imaging lens 6 can be of a type that emits an afocal light beam (see FIG. 22). Moreover, when such decentering lenses 64 and 65 are used, the positions where the imaging lens 6 and the optical path splitting unit 1 are arranged, the distances, etc. do not have to be strict, and the device assembly becomes easy. In the seventh embodiment, achromatic doublets are used for the decentering lenses 64 and 65, but the lens configuration is not limited to this.

Further, FIG. 20C shows an optical path splitting section 1 for splitting an optical path composed of four wedge prisms 66, 67, 68 and 69. In this example, the wedge prisms 66, 67, 68, 69 are arranged so that the center is most depressed, and the optical path of the imaging lens 6 is formed by these prisms.
It is divided into four parts: upper left, lower left, upper right and lower right. The wedge-shaped prism uses an achromatic prism formed by joining glass having different branches. Further, if an exit side telecentric optical system is used for the image pickup lens 6, image distortion does not occur even if the optical path is bent by the wedge prism, which is suitable for image synthesis.

Next, FIG. 20D shows an example in which the wedge prism described above is replaced by four decentered lenses 70, 71, 72 and 73. Similar to the optical path splitting unit in FIG. 20B, also in this example, the imaging lens 6 is preferably of a type that emits an afocal light beam. When these various prisms and lenses are used for the optical path splitting section, it is necessary to provide the above-mentioned light amount correction circuit.

As described above, in the seventh embodiment, an embodiment using four solid-state image pickup devices has been described. However, the solid-state image pickup device is not limited to CMD, and CCD, AMI or the like may be used. Is. For example, a CCD image sensor for NTSC (pixel number 768 × 4)
80), high-resolution imaging of about 1400 × 800 pixels can be performed. Further, if an image pickup device for PAL (the number of pixels is 820 × 640) is used, higher resolution image pickup is possible.

Next, FIG. 24 shows and describes a configuration as an eighth embodiment according to the present invention. Here, the configuration of FIG. 24 shows only the characteristic part of the eighth embodiment, and the other configuration is the same as that of the first embodiment shown in FIG.

In the above-described seventh embodiment, four image pickup devices are used in the optical path splitting unit, but the number is not limited, and as the eighth embodiment, for example, as shown in FIG. It is also possible to use a single lens (lens array) 74 and image pickup element array 75. In this case, one lens is made to correspond to each of the image pickup devices, and image pickup is performed so that each image pickup device has an overlap region. In the lens array 74, the area other than the lens is shielded from light.

Such a lens array 74 can be produced at low cost by, for example, press working.
Further, in the present invention, although CMD is adopted as the image pickup device,
Other image pickup devices such as D and MOS may be adopted.

Next, the alignment of the image pickup device in the above-described fourth to eighth embodiments will be described. In the fourth to eighth embodiments, a plurality of image pickup devices are adjacent to each other, and in a normal state of being sealed in a package, the CMD cannot be arranged at the position shown in FIG. .

FIG. 25 shows an embodiment in which a bare chip is mounted as it is. FIG. 25 shows an example in which the CMDs 81 and 82 are mounted on the ceramic substrate 80 as bare chips, and FIG. 26 is a sectional view taken along line AA ′. The CMD 81 and the CMD 82 are fitted into a groove formed in the ceramic substrate 80 and fixed with an adhesive material 83. The upper part of the groove is chamfered for mounting the adhesive material 83, and the lower part has a positioning recess. The electrodes of the CMDs 81 and 82 are bonded to the electrodes on the ceramic substrate 80 by wiring 84, and the electrodes are connected to the external members of the image pickup device via terminals 85. The terminal 85 may have a surface pattern as shown in FIG.
A metal terminal 86 may be provided as shown in FIG.

The reason why the grooves are provided in the ceramic substrate 80 in FIG. 26 is that the CMDs 81 and 82 are simply fitted into the grooves to perform positioning to some extent, and the effective pixel area shown in FIG. 18 can be widened. Further, the CMD8
The reason why the adhesive material 83 is flown on the side surfaces of the 1, 82 is to adjust the position in the optical axis direction on the back surface of the bare chip. This is because the position in the optical axis direction is displaced.

As shown by the broken line in FIG. 26, a hole 87 may be opened from the back surface side of the ceramic substrate 80 and fixed with an adhesive material 88. With this method, there is no possibility that the adhesive will cover the light receiving portion, and the work of fixing can be performed easily. Further, when the ceramic substrate 80 is attached to a prism or a crystal filter, if the spacer portions 90 are provided at both ends of the ceramic substrate 80 as shown in FIG. 28 and fixed, the wiring 91 at the time of wire bonding is not loaded. . This is the height of wire bonding h,
Assuming that the height of the spacer is H, the relationship of H> h may be set.

As another example of the bare chip mounting example of FIG. 26, as shown in FIG. 29, a bare chip having a space portion indicated by diagonal lines is provided in a groove on a rail, a plurality of bare chips are applied, and a horizontal direction is set. Position and fix.

Further, FIG. 30 shows a case where the mount example shown in FIG. 29 is applied to the image pickup apparatus of the sixth embodiment.
30A is a side view, FIG. 30B is a view from the direction B, and FIG. 30C is a view from the direction A in FIG. 30A. As shown in FIG. 30C, a spacer portion 90 is provided to protect the wiring (bonding wire) 91. FIG. 31 is another example of the mounting example shown in FIG. 29, in which a ceramic substrate 80 is fixed by a right-angled member 92 that is placed afterward, and a prism 93 is further fixed to the ceramic substrate 80. At that time, the prism 93 and the ceramic substrate 8
The distance from 0 is the height "H" of the spacer in FIG.

Next, FIG. 32 shows a specific structure of an image pickup apparatus as a ninth embodiment of the present invention, which will be described. In this embodiment, a synthetic circuit is further added to the configuration shown in FIG. 14, and the same members as those shown in FIG. 14 are designated by the same reference numerals, and only the characteristic portions will be described. To do.

One of the synthesis circuits 121 is shown in FIG.
3, the pixel values f and g of the target image to be combined are input to the pixel value conversion unit 122 and the pixel selection unit 123, respectively, and the pixel selection unit 123 is a vector indicating the pixel position corresponding to the output image. Pixels near the joint are selected based on the (coordinate value) X1, and the pixel value conversion unit 122
The corresponding pixel value is converted by.

This conversion is performed so that discontinuity does not occur in the image to be combined. For example, as shown in FIG.
The pixel value is converted according to the position in the overlapping area. Also,
This synthesizing circuit can be configured as shown in FIG. 35. The input images f and g are multiplied by the weighting factors a and b set by the coefficient setting unit 124 by the multiplication units 125a and 125b, respectively, and the addition unit is added. The sum is taken at 126 and the result is used as the output image.

As shown in FIG. 36, the coefficient setting by the coefficient setting section 124 is performed except for the overlapping area.
If it is within the overlapping area of "1" or "0", it is changed linearly. X1 is a coordinate in the image composition direction. P2-P1 is the length of the overlapping region.

The synthesizing circuit shown in FIG. 35 outputs the value of the input image f or g as it is in the case other than the overlapping area,
Within the overlapping area, the value of the coefficient a for the image f is linearly converted from "1" to "0", and the value of the coefficient b corresponding to the image g is "0" to "1". By adding two coefficient effects to the output image,
The brightness in the vicinity of the joint can be smoothly converted, and the discontinuity of the brightness, which is mainly caused by the variation in the sensitivity between the image sensors, is eliminated. Further, it is effective even when a geometrical discontinuity occurs at the joint due to the correlation detection and the interpolation processing.

As in the case of such a synthesizing circuit, the luminance difference and the geometrical difference can be eliminated to some extent in the vicinity of the joint. The brightness may be used together with the bias gain correction in the SP circuits 20 and 21.

Next, as a tenth embodiment, an example in which a structure emphasizing circuit is used in order to prevent image deterioration that occurs when linear interpolation or the like is used as the interpolation operation of the interpolation circuit will be shown below.

FIG. 37 shows a specific construction of an image pickup apparatus as a tenth embodiment of the present invention. The present embodiment is a circuit in which a structure emphasizing circuit 127 is added to the structure shown in FIG. 32 to restore an image deteriorated by interpolation, and the same members as those in FIG. The characteristic part will be described.

The structure emphasizing circuit 127 is, for example,
Compute the Laplacian using a local operator such as a digital filter. That is, as a first method, the Laplacian is calculated from the original image. Output image = the input image - ▽ ² input image × omega However, omega: constant, (see FIG. ^{39 (d)), ▽ 2} : the Laplacian operator. Here, as the weight coefficient, for example, the operators shown in FIGS. 39 (a) to 39 (c) are used. Alternatively, a selective image enhancement method is used.

[0093] The output image = the input image -h (x, y) * ▽ 2 input image where, h (x, y) is, for example, the line detection operator with respect to the input image is used.

As the second method, a high-frequency emphasis filter in the frequency domain is used. Inverse Fourier transform is performed by performing a Fourier transform on the input image and passing through a filter that emphasizes the values of high frequency components.

Next, in FIG. 38, each pixel of the reference image f is shifted (for example, 1/2 pixel, or 1 / pixel) in order to provide a uniform enhancement effect to the output image.
The structure may be emphasized after performing interpolation for about 3 pixels) and passing through the synthesis circuit 121.

Next, FIGS. 40 and 41 show the structure of an image pickup apparatus as an eleventh embodiment. These embodiments utilize the reference image to detect the mutation charge. First,
An image of the reference pattern shown in FIG. 42 is created, and the intersections of the straight lines [FIGS. 42 (a) and (c)] or the points [FIG. 42 (b)]
Accurately measure the position of.

Then, as in the image pickup apparatus shown in FIG.
A single reference image is photographed, and the reference pattern detection displacement amount calculation circuit 130.1 is obtained from the obtained left and right images.
The reference patterns are detected by 31, and the displacement amounts (shift value and rotation angle) corresponding to the left and right images are calculated using the previously measured position information.
These displacement amounts are stored in the displacement amount memories 132 and 133. Thereafter, as shown in FIG. 41, the displacement amount memory 13
Processing is performed in the same manner as in the tenth embodiment by using the respective displacement amount information stored in 1,132.

Regarding the detection of the reference pattern, feature extraction methods such as thinning (FIGS. 42 (a) and 42 (c)), line neighborhood tracking, and barycenter detection (FIG. 42 (b)) can be used. In addition to the types of reference patterns shown in FIG. 42,
There are many types, and the basic is that the respective displacement amounts (shift value and rotation angle) are obtained for the left and right images.

In the eleventh embodiment, the displacement amount can be detected by using the reference image even when the region where the images overlap is small and the displacement amount cannot be detected by the correlation calculation.

Next, FIG. 44 shows the arrangement of an image pickup apparatus as a twelfth embodiment of the invention. This image pickup device is characterized by additionally providing a reference pattern filter 135 when capturing an image. The reference pattern filter 13
As shown in FIG. 43, the patterns 5 are provided one by one near the bottom line and above the overlapping region, and are read together with the image. With respect to the read left and right images, the reference pattern detection displacement amount calculation circuit 136 detects the respective reference patterns from the regions near both end lines of the overlapping region, and obtains the relative displacement amounts of the left and right images. Subsequent processing is the tenth
Is the same as the embodiment of. This reference pattern filter is effective, for example, when a silver salt film is used as a subject.

The detection of the reference pattern is performed by thinning (FIG. 43A), tracking the neighborhood of the line, and detecting the center of gravity (FIG. 42).
A feature extraction method such as (b)) can be used. As described above, the feature of this embodiment is that the correlation points of the left and right images can be detected earlier. It is also applied to an image in which a correlation point is difficult to detect. Also,
By allowing the reference pattern filter to be taken in and out, the system can be easily calibrated.

Next, FIG. 49 shows a concrete construction of an image pickup apparatus as a thirteenth embodiment of the present invention. This embodiment is applied to composition of three or more images in the same direction by further adding a rotation amount determination 120, a circuit selection circuit 121, and a rotation interpolation circuit 123 to the configuration shown in FIG.

The rotation amount R detected by the displacement amount detection circuit 24 is sent to the rotation amount determination circuit 120, and is selected after the combining process. This selection is determined by a preset threshold value. For example, if the detected rotation amount R is larger than the threshold value as shown in FIG.
1 turns on the terminal A, the combined image passes through the rotation interpolation circuit 123, and is rotated by -R and output as shown in FIG. 48 (b) to perform a combining process with the next image. However, if the detected rotation amount R is smaller than the threshold value, the selection circuit 121
Turns on the terminal B and outputs it to the frame memory 30 as it is.

As described above, in the thirteenth embodiment, as shown in FIG. 48A, when the number of images to be combined increases, the reference left side image and the right side image to be combined will be displayed. The relative rotation angle between
Correlation may cause false detection.

In order to prevent this mismatching, the rotation amount R of the correlation is checked with respect to the image synthesized this time, and it is compared with a set threshold value (for example, 5 degrees). If it is larger, the combined image is rotated by -R and a combining process is performed with the next image (FIG. 48 (b)). Therefore, in this embodiment,
If this is applied to the case where three or more images are combined by adding the rotation amount determination circuit 120, the selection circuit 121, and the rotation interpolation circuit 123, selective correction is performed by the rotation amount, and correlation detection failure or error occurs. Matching is prevented.

FIG. 51 shows a concrete structure of an image pickup apparatus as a fourteenth embodiment according to the present invention. In the present embodiment, a boundary line end point detection circuit is further added to the configuration shown in FIG. 37, and the present embodiment is applied to composition of three or more images in the same direction. In this image pickup apparatus, the reference left-side image data shown in FIG. 50 is sent to a boundary line end point detection circuit 125, which detects the end points A and B of the right side boundary line of the image and sets a small coordinate value in the horizontal direction. The endpoints that it has are sent to the synthesis circuit 7. In the synthesizing circuit 7, the lateral coordinate values of the sent end points are shown in FIG.
Let it be the right end point of the joint processing shown in FIG.

Therefore, in this embodiment, as shown in FIG. 50 (a), the number of images to be combined is large, and the boundary line on the right side of the combined image may be slanted. When the next image is combined with the next image, an inconvenient part is prevented from occurring in the joint processing shown in FIGS. This inconvenient part is because the center of the overlapping area is used as it is as the center of the joint processing.

Therefore, as shown in FIG. 51, the boundary line end point detection circuit is used to detect the end points of the right side boundary line of the left image, and the end point A having a small coordinate value in the horizontal direction is set in the synthesis circuit 7. send. As shown in FIG. 50B, this point is the right end point of the joint processing area, and the center line of the joint processing is the center from the left end point of the overlapping area to this point. Therefore, it is possible to eliminate the occurrence of an inconvenient part in the joint processing.

As described above, according to the present embodiment, the boundary line end point detection circuit 125 is used to detect both end points of the right boundary line of the reference left image, and the end points having a small coordinate value in the horizontal direction are connected. By setting the right end point of the processing area, the inconvenient part of the joint processing is eliminated.

Next, FIG. 52 shows the structure of an image pickup apparatus as a fifteenth embodiment of the present invention. In this embodiment,
16 4000 × 500 CMDs are used to provide an overlapping area of about 60 pixels, and the synthesizing process allows 40
It is an example of an apparatus capable of obtaining an image having a resolution of a silver salt film of 00 × 6000.

The first composition processing section shown in FIG. 53 is configured as shown in FIG. The second synthesis processing unit is configured as shown in FIG. 54, one on the input side is the image signal from the CMD image sensor similar to the first synthesis processing unit, and the other is the image data (the image data previously synthesized). (Stored in memory) directly,
Input to the displacement amount detection circuit 24. Further, similarly to the fourteenth embodiment, a boundary line end point detection circuit 125 is added in order to eliminate the inconvenient part of the joint processing.

Then, as shown in FIG. 53, the image signals obtained from the 16 image pickup devices are sequentially subjected to a synthesizing process in 15 steps to obtain a resolution image similar to a silver salt film of 4000 × 6000. To be From the above, according to the present embodiment, an ultra-high resolution image can be obtained by a multi-step combining process using a plurality of image pickup devices.

Next, FIG. 55 shows the structure of an image pickup apparatus as a sixteenth embodiment of the present invention. In this imaging device, the first combination processing unit is equivalent to that of the above-described fifteenth embodiment, and the third combination processing unit is configured as shown in FIG. This third synthesis processing unit directly outputs from the memory placed in each of the two inputs, the result of the previous synthesis processing.
Input to the displacement amount detection circuit 24. Further, in order to eliminate the inconvenient part of the joint processing, the boundary line end point detection circuit 125
To provide.

This embodiment is characterized in that a large number of synthesizing processes are carried out in parallel to shorten the overall processing time. According to the sixteenth embodiment, high-speed processing is performed in four steps as compared with the sequential fifteen-step combining processing of the fifteenth embodiment. In addition, 16 CMDs of 4000 × 500 pixels are used in this embodiment and the 15th embodiment,
Not limited to this, it is possible to freely select the size and number of the image pickup elements as necessary.

Next, as a seventeenth embodiment of the present invention, an example applied to a projector will be described. FIG. 57 shows an external view of this projector, and FIG. 59 shows a configuration diagram of the projector. Further, FIG. 58 shows a configuration diagram in the vicinity of the half prism.

The feature of this embodiment is that a plurality of images displayed on a plurality of LCDs are projected by the projector 126 and are combined on the screen 127, so that even if there is an error in the mounting of this LCD, a function of correcting this is provided. Have.

As shown in FIG. 59, the half prism 12
8, LCDs 129 and 130.131 are arranged so that the projected images on the screen have overlapping regions. A crystal filter 132 is arranged on the light emission side. The crystal filter 132 functions as a low-pass filter for preventing pixels of the LCDs 129 to 131 from clearly forming an image on the screen and becoming difficult to see.

The S, R memory 133 shown in FIG. 58 is an LCD 129 and an LCD 130, which are required by the device described later.
This is a memory that stores the amount of positional deviation (parallel movement amount and rotation amount) between the LCD 130 and the LCD 131. The image signal to be displayed on the screen 127 is first stored in the frame memory 30, and the divided image signals to be displayed on the LCDs 129, 130 and 131 are interpolated by the interpolation circuits 134 and 134, respectively.
135, 136.

The interpolation circuits 134, 135 and 136 are
Based on the values from the S and R memories 133, interpolation calculation for parallel movement and rotation is performed so that the divided images are connected on the screen 127 without displacement.

Further, with respect to the overlapping portion of the images on the screen 127, the weighting factors from the weighting factor calculator 140 are multiplied by the adders 137 to 139 to adjust the brightness of the output image. This weighting factor is changed as in FIG. Then, the image signal multiplied by the weighting coefficient is stored in the memories 141 to 143, converted into analog by the D / A converters 144 to 146, and then LC.
The screen 1 is projected from D129, 130, 131.
An image is displayed on 27. A light source 147.148 is provided adjacent to such an LCD, and an image displayed on the LCD is projected on the screen 127 by the light of the light source. As described above, in the present embodiment, since a plurality of LCDs can be used, a high definition projector can be realized.

Further, since the LCD mounting error is detected and connected without displacement on the screen, the LCD mounting does not require high accuracy. Further, the overlapping area of the images on the screen is multiplied by the weighting coefficient, so that the joint is not noticeable. Further, since the crystal filter is used, the pixels of the LCD are not displayed on the screen, and the image has a high quality.

Next, referring to FIG. 60, the LCD 1 described above is used.
The detection of the positional deviation of 29, 130, 131 will be described. As shown in FIG. 60, the lens 6 and the crystal filter 132
A mirror 149 for detecting displacement is inserted between the two. That is, the image displayed on each LCD is configured so that the same image as that formed on the screen is formed on the CCD image pickup device 150 by the action of the mirror 149 and the lens 156.

At the time of detecting the positional deviation, a highly correlated subject is input as a reference video signal. In addition, the interpolation circuits 134 to 136 perform processing assuming that there is no displacement (S = R = 0), and multiply the weighting coefficient by "1". Then, at the time of detection, first, only the image corresponding to the LCD 129 is displayed and captured by the CCD image sensor 150,
After being A / D converted by the / D converter 151, it is stored in the memory 153 by the function of the switch 152. Next LCD
Only the image of 130 is displayed, the switch 152 is on the side b, and the image is stored in the memory 154. Then, in the S and R detector 155, the image signals of the memories 153 and 154 are displayed on the LCD.
A positional deviation between 129 and the LCD 130 is detected and stored in the S / R memory 133.

Similarly, next, only the image on the LCD 131 is displayed and stored in the memory 153.
The positional deviation from 1 is detected and stored in the S / R memory 133. From the above, it is possible to detect the amount of positional deviation when the LCDs 129, 130, 131 are attached. The reference video signal used here may use the pattern shown in FIG.

Further, in this embodiment, the position shift is detected by using the mirror. However, as shown in FIG.
It is also possible to project the image of 1) on the screen 127 once and re-image the reflected light on the CCD by the half mirror 156. Alternatively, a dedicated camera for detecting the displacement may be used.

The present invention also relates to a CRT as shown in FIG.
It can also be applied to monitors. In this example, a plurality of electron guns 186-
190, each of which displays a partial image on the fluorescent screen 193, and each of the interpolation circuits 161 to 165 translates and rotates the image so as to correct the positional deviation of these electron guns 186 to 190. I do. Here, the spatial filter 194 is a low-pass filter composed of a crystal filter or the like.

By using a plurality of electron guns in this way, the distance between the fluorescent screen and the electron gun can be shortened as compared with the case where only one electron gun is used. These electron guns may be, for example, lasers, or those using a light emitting diode array with a lens and a micromachine mirror.

The image distortion caused by the electromagnetic deflection may be corrected by an interpolation circuit. The cutoff frequency of the spatial filter may be determined according to the spacing between the scanning lines due to this distortion. The JQ and the spatial filter may be provided immediately before each electron gun, or an electrical LPF may be applied after D / A conversion.

Next, FIG. 63 shows a structure of an application example to a film taking-in device using a line sensor as an eighteenth embodiment of the present invention. In this film taking-in device, a film 301 for picking up an image and a loading mechanism 3 for taking up and rewinding the film 301
02 and its drive circuit 307 are provided, and the film 3
A light source 303 for irradiation is provided near 01.

Then, the light source 3 is sandwiched by the film 301.
A lens 304 for forming a film image on the image capturing unit 305 is provided at a position facing 03. The image signal detected by the image pickup unit 305 is the preamplifier 10
a, 10b, 10c are amplified, and A / D converter 14
a, 14b, 14c are converted into digital signals, and the signal processing circuits (SP) 20a, 20b, 20c perform γ correction and edge enhancement, and the frame memories 22a, 22
b, 22c.

The image signals read from the frame memory 22 are subjected to image composition by the image composition circuit 308 for each of R, G and B separately. This image composition circuit 308
Is composed of a synthesis processing unit as shown in FIG. Then, the combined image is displayed on the display device 3 such as a CRT.
09, the storage device 310, and the printer 311 respectively.

The image pickup section 305 has a structure shown in FIG. Three run sensor 306a, 30
6b and 306c are arranged. Then, each line sensor is provided with an RGB optical filter as shown in FIG. The present embodiment is characterized in that the film 301 is loaded and the obtained images are combined.

That is, the line sensors 306a, 306
b, the image captured by 306c is A in FIG.
The three images A, B, and C are arranged in B and C, and one frame of the film is picked up by combining these three images. The imaging unit 3
If there is an error in attaching the line sensor in 05, a position shift occurs between the images A, B, and C, but in the overlapping region, an image is generated using the above-described position rubbing correction.

According to the present embodiment, since the line sensor is used instead of the area sensor, high resolution photographing can be realized at a very low cost. Moreover, a color signal can be easily obtained by using the color filter. The line sensors may be arranged in a staggered arrangement as shown in FIG. 63, for example. Further, in the present embodiment, the film is driven to perform scanning for imaging, but the imaging unit itself may be driven.

Further, instead of using the color filter as shown in FIG. 64 (c), a line sensor dedicated to RGB may be used or a rotary color filter may be used. Next, FIG. 65 shows C as a nineteenth embodiment of the present invention.
A configuration of an image pickup apparatus that does not require a frame memory for interpolation calculation using MD will be shown and described. This image pickup device includes a CMD and has a random access function and a nondestructive read function.

This random access function is a function of reading out the signal value of a pixel at an arbitrary position, and the non-destructive reading function is a function of reading out the signal value of the pixel as many times as possible without losing the signal charge until the signal of the pixel is reset. is there. In other words, since non-destructive reading can be performed, the CMD itself can be treated like a memory for a short time.

By using the random access function and the nondestructive read function, interpolation calculation can be performed without using the frame memory. That is, the CMD is used as a substitute for the frame memory, and the pixel values required for interpolation are read out by random access. A configuration example of such an image pickup device is shown in FIG. Here the CMD driver 32
a and 32b generate a read address from the decoder address generator 312 to the x decoder 313 and the y decoder 314 according to the pixel position signal to be read as shown in FIG. y decoder 31
At 4, in accordance with this address, a signal pulse for reading the signal at the designated pixel position is sent to the CMD. The CMD drivers 32a and 32b are independently controlled.

The analog interpolating section 315 composed of an analog circuit includes a coefficient generating circuit 316, a multiplier 317,
It is composed of an adder 318, a sample hold 319, and a switch 320 for switching between the sample hold side and the ground side.

Next, the interpolation calculation, which is a feature of this embodiment, will be described. In this embodiment, the same interpolation calculation as that of the first embodiment is performed, that is, the signals corresponding to the pixels dij (i = 1 to u, j = 1 to w) as shown in FIG.
The data is read from D8, A / D converted, signal-processed, and then written in a predetermined position of the frame memory 30. On the other hand, pixels dij (i = u + 1 to u + v, j = 1 to w)
With respect to the above, the four pixels in the vicinity of the pixel dij are non-destructively read from the CMD9 by using the random access function, and the analog interpolating unit 315 uses the equations (9-a) and (9 The calculation corresponding to −b) is performed, and the calculated signal is A / D converted. Then, the A / D converted signal is subjected to signal processing, structure enhancement is performed, and the signal is written in a predetermined position of the frame memory 30. Since non-destructive reading is performed in this manner, the same pixel can be read many times.

Further, every time the signal value of one pixel is calculated by analog interpolation, the switch 320 is switched to the b side, and the value of the sample hold circuit (SH) 319 is reset to "0". As described above, a plurality of images can be combined without using the frame memories 22 and 23 required in the first embodiment, and the image pickup apparatus can be manufactured at low cost.

In this embodiment, the displacement amount of the CMDs 8 and 9 can be measured in the same manner as in the first embodiment. In addition, the coefficient generator 3 used in the interpolation calculation
The 16 coefficients can be implemented by a small-scale circuit, for example, if a representative coefficient set is prepared and the corresponding set is selected from them. Furthermore P
Instead of the S circuit 29, a synthesizing circuit as shown in FIG. 35 can be used.

As described above, the image pickup apparatus of the present invention detects the positional relationship between the image pickup elements by using the correlation operation, and uses the interpolation operation based on this to obtain the signal value of the image. In addition, it is possible to easily and inexpensively manufacture without being affected by the positioning accuracy of the image sensor. Further, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications and applications can be made without departing from the scope of the invention.

[0143]

As described in detail above, according to the present invention, pixels can be formed with a certain degree of precise positioning, and
It is possible to provide an imaging device having no moving parts and having high resolution at low cost, and it is possible to realize an image processing device capable of obtaining a high-definition image using a plurality of images.

[Brief description of drawings]

FIG. 1 is a diagram showing a schematic configuration for explaining an outline of an image pickup apparatus of the present invention.

FIG. 2 is a diagram schematically showing an image pickup area of an image pickup element and an area for displaying a subject as an image.

FIG. 3 is a diagram showing a specific configuration of the image pickup apparatus according to the first embodiment of the present invention.

FIG. 4 is a diagram showing a specific circuit configuration of a displacement amount detection circuit and an interpolation circuit shown in FIG.

5 is a diagram showing an overlap area of the CMD shown in FIG. 3;

FIG. 6 is a diagram showing a shift vector, a shift vector due to rotation, and a shift vector due to parallel movement.

FIG. 7 shows a reference area of the CMD8 shown in FIG.
It is a figure which shows the position where the sum of absolute values becomes the minimum in the search area of MD9.

FIG. 8 is a diagram showing a positional relationship between a predetermined pixel A and an imaged pixel in the pixel value interpolation by the linear interpolation calculation.

FIG. 9 is a diagram showing the positional relationship of each pixel of CMD8 and CMD9 by a vector.

FIG. 10 is a diagram showing a configuration of an image pickup apparatus as a second embodiment according to the present invention.

FIG. 11 is a diagram showing a configuration of an optical path splitting unit of an image pickup apparatus as a third embodiment according to the present invention.

FIG. 12 is a CMD according to a third embodiment.
It is a figure which shows the light amount distribution of 8 and 9.

FIG. 13 is a diagram showing a light amount distribution of the inverse function of FIG.

FIG. 14 is a diagram showing a configuration of an image pickup apparatus as a third embodiment according to the present invention.

FIG. 15 is a diagram showing a configuration of an imaging device as a fourth embodiment according to the present invention.

FIG. 16 is a diagram showing a configuration and a light flux when a separator lens is used in the optical path splitting unit of the fourth embodiment.

FIG. 17 is a diagram showing a configuration of an imaging device as a fifth embodiment according to the present invention.

FIG. 18 is a diagram schematically showing an image pickup area of an image pickup element according to the fifth embodiment and an area for displaying a subject as an image.

FIG. 19 is a diagram showing a configuration of an image pickup apparatus as a sixth embodiment according to the present invention.

20A to 20D are diagrams showing a configuration example of an optical path splitting unit of an image pickup apparatus as a seventh embodiment according to the present invention.

FIG. 21 is a diagram showing a state where the optical path splitting portion shown in FIG. 20 (a) is viewed from right side and right side.

22 is a diagram showing a state where the optical path splitting portion shown in FIG. 20 (b) is viewed from right side and right above.

FIG. 23 is a diagram schematically showing an image pickup area of an image pickup device imaged by an optical path splitting unit and an area for displaying a subject as an image in the seventh embodiment.

FIG. 24 is a diagram showing a configuration as an eighth exemplary embodiment of the present invention.

FIG. 25 is a diagram showing an embodiment of mounting a bare chip itself in a solid-state imaging device.

FIG. 26 is a view of A of the solid-state imaging device shown in FIG.
It is sectional drawing which shows a -A 'cross section.

FIG. 27 is a diagram showing the structure of a ceramic package provided with metal terminals.

FIG. 28 is a diagram showing a configuration of a package in which spacer portions are provided on both ends of a ceramic substrate.

FIG. 29 is a diagram showing an example of mounting a bare chip.

FIG. 30 is a diagram showing the configuration when the mount example shown in FIG. 29 is applied to the imaging device of the sixth embodiment.

FIG. 31 is a diagram showing an example of mounting a bare chip.

FIG. 32 is a diagram showing a configuration of an imaging device as a ninth embodiment according to the present invention.

FIG. 33 is a diagram showing a specific configuration of the first combining circuit shown in FIG. 32.

34 is a diagram showing a state of linear interpolation by the first combining circuit shown in FIG. 33.

FIG. 35 is a diagram showing a specific configuration of the second synthesis circuit shown in FIG. 32.

FIG. 36 is a diagram showing a state of linear interpolation by the second synthesizing circuit shown in FIG. 35.

FIG. 37 is a diagram showing a configuration of an image pickup apparatus as a tenth embodiment according to the present invention.

FIG. 38 is a diagram showing a configuration of an image pickup apparatus according to a modified example of the tenth embodiment.

FIG. 39 is an example of a weighting factor operator.

FIG. 40 is a diagram showing a configuration for creating a memory used in an imaging device as an eleventh embodiment according to the present invention.

FIG. 41 is a diagram showing the structure of an imaging device as an eleventh embodiment according to the present invention.

FIG. 42 is an image example of a reference pattern.

FIG. 43 is a diagram showing an example of a pattern provided in the vicinity of the lower end line on the overlapping region.

FIG. 44 is a diagram showing a configuration of an imaging device as a twelfth embodiment of the present invention.

FIG. 45 is a diagram showing a configuration of a conventional optical system and an image sensor.

FIG. 46 is a diagram showing a configuration of an image pickup element positioned so as to image a predetermined position in a conventional image pickup apparatus.

FIG. 47 is a diagram showing a configuration of an image pickup system that moves an image pickup element in a conventional image pickup apparatus to pick up an image of the entire field of view.

[Fig. 48] Fig. 48 is a diagram showing a rotated state of three or more image combining reference images.

FIG. 49 is a diagram showing a specific configuration of an imaging device as a thirteenth embodiment of the present invention.

[Fig. 50] Fig. 50 is a diagram showing a removal state of an inconvenient portion of an overlapping region of image combination of three or more images.

FIG. 51 is a diagram showing a specific configuration of an imaging device as a fourteenth embodiment according to the present invention.

FIG. 52 is a diagram showing a configuration of an imaging device as a fifteenth embodiment of the present invention.

FIG. 53 is a diagram showing a connection configuration of the synthesis processing unit shown in FIG. 52.

FIG. 54 is a diagram showing a specific configuration of the second synthesis processing unit shown in FIG. 53.

FIG. 55 is a diagram showing a configuration of an image pickup apparatus as a sixteenth embodiment according to the present invention.

FIG. 56 is a diagram showing a specific configuration of the synthesis processing unit shown in FIG. 55.

FIG. 57 is a diagram showing an example applied to a projector as a seventeenth embodiment of the present invention.

FIG. 58 is a diagram showing a configuration of a CMD.

FIG. 59 is a diagram showing a configuration near a half prism.

FIG. 60 is a diagram showing a configuration for detecting a positional deviation of an image from the LCD.

FIG. 61 is a diagram showing a configuration for detecting a positional deviation of an image from an LCD using a half mirror.

FIG. 62 is a diagram showing a configuration example in which the image pickup apparatus of the present invention is applied to a CRT monitor.

FIG. 63 is a diagram showing a configuration example applied to a film taking-in device using a line sensor as an eighteenth embodiment of the present invention.

64 is a diagram showing the arrangement of line sensors in the image capturing section of the film capturing device shown in FIG. 63 and the state of the captured image.

FIG. 65 is a diagram showing a specific configuration of an imaging device as a nineteenth embodiment according to the present invention.

[Explanation of symbols]

1 ... Optical path splitting unit, 1a ... Half prism, 2 ... Imaging unit,
3 ... Image storage unit, 4 ... Displacement amount detection unit, 5 ... Interpolation unit, 6 ...
Photographing optical system (imaging lens), 7 ... Image composition unit (composition circuit), 8, 9, 51, 52, 53, 54 ... CMD, 1
0, 11 ... Preamplifier, 12, 13 ... Low pass filter (LPF), 14, 15 ... A / D converter, 16, 17 ...
Subtractor, 18, 19 ... FPN storage memory, 20, 21 ...
Signal processing circuit (SP), 22, 23
... Frame memory, 24 ... Displacement amount detection circuit, 24a, 2
4b ... correlator, 24c ... coefficient calculator, 25 ... interpolation circuit,
26, 27 ... Memory, 28 ... Interpolation coefficient writing circuit, 2
9 ... Parallel-Serial (PS) conversion circuit, 30 ... Frame memory, 31 ... Display unit, 32 ...
CMD driver, 33 ... System controller, 34 ...
Coordinate selection circuit, 35 ... Coordinate conversion circuit, 36, 36a ... Data read circuit, 37 ... Interpolation coefficient calculator, 38 ... Linear interpolation calculation circuit, 39 ... Multiplier, 40 ... Adder, 41 ... Data address detector, 42 ... data address memory, 43 ...
Coefficient a memory, 44 ... Coefficient b memory, 45 ... Coefficient c memory, 46 ... Coefficient d memory, 47, 48 ... Light amount correction circuit,
49 ... Rotation mechanism part, 50 ... Shading plate, 61, 62, 66,
67, 68, 69 ... Wedge prism, 63 ... Beam splitter, 64, 65 ... Decentering lens, 70, 71, 72, 7
3 ... decentering lens, 74 ... lens (lens array), 75
... Imaging element array, 80 ... Ceramic substrate, 81, 82
... CMD, 83, 88 ... Adhesive material, 84, 91 ... Wiring (bonding wire), 85 ... Terminal, 86 ... Metal terminal, 8
7 ... Hole, 90 ... Spacer part, 92 ... Right angle member, 93 ... Prism.

─────────────────────────────────────────────────── ─── Continued Front Page (51) Int.Cl. ⁷ Identification Code FI Theme Coat (Reference) H04N 5/225 H04N 5/225 F 5/232 5/232 Z // H04N 101: 00 101: 00 (72 Inventor Tatsuo Nagasaki 2-43-2 Hatagaya, Shibuya-ku, Tokyo Olympus Optical Co., Ltd. (72) Inventor Ryoichi Sawaki 2-43-2 Hatagaya, Shibuya-ku, Tokyo Olympus Optical Co., Ltd. (72 ) Inventor Ken Mori 2-43-2 Hatagaya, Shibuya-ku, Tokyo Olympus Optical Co., Ltd. (72) Inventor Yoichi 2-43 Hatagaya, Shibuya-ku, Tokyo F-Term, Olympus Optical Co., Ltd. (Reference) 5B057 AA20 BA02 CA01 CA08 CA12 CA16 CB01 CB08 CB12 CB16 CC01 CE08 DA07 DB02 DB06 DB09 DC34 5C022 AA13 AC00 AC42 AC51 AC69 5C024 CX38 CX39 CY33 EX47 HX14 HX57 5C076 AA19 BA06

Claims (3)

[Claims] 1. An image processing apparatus for creating a single composite image in which respective end portions of a plurality of images are overlapped with each other as overlapping regions, and image storage means for storing the plurality of images as image data, Displacement amount detection means for detecting a positional deviation amount between the images based on the data of the overlap area in the image data read from the storage means, the plurality of images based on the detected positional deviation amount,
An image processing apparatus comprising: a synthesizing unit for synthesizing a luminance discontinuity and a geometric discontinuity. 2. The image processing apparatus according to claim 1, an optical path splitting unit for splitting a subject image into a plurality of images, and the images corresponding to the number of images split by the optical path splitting unit, and the images overlap each other. An image pickup unit which is arranged so as to have a region and which is configured by an image pickup device which photoelectrically converts each image, and the image storage unit stores the image signal from the image pickup unit. 3. The synthesizing unit calculates a rotation amount R and a shift amount S between the plurality of images as conversion coefficients from the image data of the overlap area read from the image storage unit, and the conversion is performed. Interpolation means for interpolating the image data read from the image storage means to generate interpolation image data based on a coefficient; and interpolation image data from the interpolation means and image data read from the image storage means. The image pickup apparatus according to claim 2, further comprising at least an image synthesizing unit for synthesizing.