JP2006135823A - Image processor, imaging apparatus and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for reducing image quality deterioration in the case of generating a single image on the basis of a plurality of images. <P>SOLUTION: This image processor generates a single image on the basis of pixel information of a plurality of input images. The image processor has a compositing means 115 capable of obtaining at least a part of pixels in the single image by compositing pixel information of at least two input images among the plurality of input images. The compositing means changes composition ratios of pixel information on the basis of information about a parallax between the input images. In addition, the image processor changes methods for generating the single image on the basis of the information about the parallax between the input images. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

The present invention relates to an apparatus for generating a single image based on a plurality of images acquired using a compound eye optical system or the like, and an imaging apparatus including the apparatus.

  2. Description of the Related Art Conventionally, there has been proposed a compound eye imaging apparatus that obtains a single high-definition image by synthesizing two images obtained by photographing a common (identical) subject using two sets of imaging systems. (For example, see Patent Documents 1 and 2). In this compound-eye imaging device, a left imaging system and a right imaging system are prepared, and the left imaging system and the right imaging system capture a subject with a sampling point shifted by ½ pitch in the spatial phase. By synthesizing the obtained image and the image obtained by the right imaging system, it is possible to obtain one output image having a higher resolution than when a subject is photographed by one imaging system.

  This principle can also be applied to color images. In this case, each imaging optical system is provided with an image sensor having a color filter such as a Bayer arrangement or a stripe arrangement, and the color sensitivity for each imaging optical system. There is a method using image sensors having different characteristics.

  Here, when the optical axes of the photographing optical systems are shifted to have parallax, the effect of obtaining a high-definition image by combining a plurality of images is reduced. The influence of this parallax varies depending on the distance to the subject.

  This will be described with reference to FIG. In FIG. 8, reference numerals 601 and 602 denote individual photographing optical systems constituting a compound eye optical system, and each photographing optical system forms an image of the same subject on image sensors 603 and 604.

  Each image sensor has a mosaic or striped color filter and picks up a color image independently. The pixel signals from the image sensors 603 and 604 are processed by the camera signal processing circuits 605 and 606 to generate two images, and these images are combined by the image combining processing circuit 607 to obtain a high-resolution single image (hereinafter, referred to as “image”). , Referred to as a composite image).

In this configuration, the two photographing optical systems 601 and 602 are arranged in a state where their optical axes are shifted from each other and inclined by an angle θ with respect to a specific subject position. Here, the angle θ is referred to as a convergence angle.
Japanese Patent No. 3478796 (paragraphs 0016-0019, FIGS. 1-4, etc.) JP-A-5-265081 (paragraphs 0002 to 0005, FIG. 11 etc.)

  When the convergence angle θ is set so that the optical axes of the two photographing optical systems 601 and 602 intersect at the subject position b in FIG. 8, the optimum convergence angle is different at the subject positions a and c. Deviation due to parallax occurs in the subject position on the image captured by the system.

  This will be described with reference to FIG. FIG. 9 shows a composite image obtained by combining a plurality of images. The circle in the figure is the subject.

  When the position of the subject is b in FIG. 8, since the convergence angle is appropriately set, a high-definition composite image without image displacement is output. When the subject position is “a” or “c” in FIG. 8, since the convergence angle is not appropriate, the images output from the image sensors 603 and 604 are images shifted from each other to the left and right. Deviation appears.

  Note that FIG. 8 illustrates a case where a plurality of photographing optical systems arranged in a one-dimensional direction only in the horizontal direction is used. However, a plurality of photographing optical systems are arranged in a two-dimensional direction in the horizontal direction and the vertical direction. In such a case, image shift also appears in the two-dimensional direction of the horizontal direction and the vertical direction.

  Here, the relationship between the convergence angle and the shift of the subject imaging position will be described with reference to FIG. When the photographing optical systems 601 and 602 are set to an infinitely focused state, these optical axes are parallel, and the respective optical axes pass through the centers of the image sensors 603 and 604. When providing a spatial offset between the image sensors 603 and 604, the sensor position with respect to the optical axis is shifted. In this state, the photographing optical system 601 has a convergence angle θ with respect to the position of the subject distance d, and an image picked up by the image sensor 603 corresponding to the photographing optical system 601 is set to an infinite focus state. In this case, the image is offset by the shift amount e with respect to the optical axis position when the infinite focus state is set.

At this time, the relationship between the subject distance d and the shift amount e is obtained. A triangle formed by the photographing optical systems 601 and 602 and the object distance position is similar to a triangle formed by the center of the photographing optical system 601 and the image sensor 603 and the shift position of the image sensor 603. If the distance between sensors is s and the focal length is f,
s / d = e / f
It becomes. Therefore,
e = (s · f) / d (1)
It becomes.

  When this is shown in the figure, a graph as shown by a solid line in FIG. 11 is obtained. The horizontal axis represents the subject distance d, and the vertical axis represents the shift amount e. As can be seen from this graph, when the subject distance increases in the short distance region, the shift amount e increases rapidly.

  In FIG. 11, the dotted line indicates the characteristic when the optical characteristic is set by matching the convergence angles of the two photographing optical systems so that the optical axes intersect at the position of the subject distance “1”. In this case, when the subject distance is longer than “1”, the shift amount becomes a negative value, and it is understood that the shift amount is shifted to the opposite side as compared with the case where the subject distance is closer than “1”.

  Thus, at a short distance, the amount of deviation due to parallax increases rapidly. This deteriorates the combined image as an image shift when a plurality of images are combined. For example, if the convergence angle at which the subject distance is 2 to 5 m and the deviation amount is 0 is set, the amount of image deviation becomes large at the time of shooting at a shorter distance than the set subject distance, particularly at the time of macro shooting.

  An object of the present invention is to provide an image processing apparatus, an imaging apparatus, and an image processing program capable of reducing deterioration in image quality when a single image is generated based on a plurality of images. Yes.

  The present invention as one aspect is an image processing apparatus (image processing program) that generates a single image based on pixel information of a plurality of input images, and at least some of the pixels in the single image are Combining means (synthesizing step) is obtained by combining pixel information of at least two of the plurality of input images. Then, the synthesizing unit (synthesizing step) changes the synthesis ratio of the pixel information based on the information regarding the parallax between the input images.

  Further, an image processing apparatus (image processing program) according to another aspect of the present invention includes image generation means (image generation step) that generates a single image based on pixel information of a plurality of input images. The generation means (image generation step) changes the generation method of the single image based on information on the parallax between the input images.

  According to the present invention, by generating a single image by combining the pixel information of the input image at a combination ratio according to the parallax, or by changing the generation method of the single image according to the parallax, A high-quality single image with little image quality degradation due to the influence of parallax can be obtained.

  FIG. 1 shows a configuration of an imaging apparatus including an image processing apparatus (image processing unit) that is Embodiment 1 of the present invention. In FIG. 1, reference numerals 101, 102, 103, and 104 denote photographing optical systems having parallax, and these four photographing optical systems 101 to 104 constitute a compound eye optical system. The optical axes of the four photographing optical systems 101 to 104 intersect (converge) with each other at a position of a set subject distance described later.

  In this embodiment, a case where a compound eye optical system is constituted by four photographing optical systems will be described, but the number of photographing optical systems constituting the compound eye optical system in the present invention is not limited to this.

  Reference numerals 105, 106, 107, and 108 denote image sensors (image sensors) that photoelectrically convert subject images formed by the respective photographing optical systems.

  Reference numerals 109, 110, 111, and 112 denote camera signal processing circuits that perform signal processing on pixel signals from the respective image sensors and output YUV signals or RGB signals (hereinafter referred to as imaging signals). An image (input image) is formed by this imaging signal.

  Reference numeral 113 denotes an image sensor driving / synchronizing signal generation circuit that drives the four image sensors 105 to 108 and the four camera signal processing circuits 109 to 112 in synchronization.

  Reference numeral 120 denotes an image processing unit. In the image processing unit 120, reference numeral 114 denotes a parallax detection circuit that detects parallax between input images by the four imaging elements 105 to 108. However, in this embodiment, since the distance between the image sensors is a fixed value, each input image is detected only with any two video signals (video signals from the camera signal processing circuits 109 and 110 in the figure). Detect parallax between. In addition, when the distance between each image pick-up element differs, the parallax between each input image is detectable based on the video signal from the four camera signal processing circuits 109-112.

  Reference numeral 115 denotes a synthesis processing circuit, which is a synthesis processing circuit that generates a single high-resolution synthesized image based on pixel values (pixel information) of four input images.

  Reference numeral 130 denotes a CPU as a controller that controls the entire image pickup apparatus, such as the image pickup element driving / synchronizing signal generation circuit 113 and the image processing unit 120.

  Here, the arrangement of the photographing optical system and the image sensor will be described with reference to FIG. In FIG. 2, reference numeral 201 denotes an aperture that restricts the amount of light incident on the four photographing optical systems 101 to 104 and improves the optical characteristics of the photographing optical systems 101 to 104.

  Reference numerals 101 to 104 denote the four photographing optical systems shown in FIG. 1, and reference numerals 105 to 108 denote the four imaging elements. The four imaging optical systems 101 to 104 and the four imaging elements 105 to 108 are arranged on the same plane, and each of the four imaging systems including the imaging optical system and the imaging element corresponding thereto captures the same subject. Take an image. The diaphragm 201, the compound eye optical system including the four photographing optical systems 101 to 104, and the four imaging elements 105 to 108 constitute an imaging system.

  FIG. 3 shows the arrangement of the imaging elements 105 to 108 as viewed from the light receiving surface side. These four image sensors 105 are configured to be driven independently from each other on the same semiconductor substrate (not shown). However, one imaging device may be divided into four imaging areas, and each parallax image may be generated from the pixel signal output from each imaging area, or as a component independent of each other without having a common substrate Four image sensors may be used.

  In FIG. 3, the image sensor 106 is disposed adjacent to the image sensor 105 in the horizontal direction, and the image sensors 107 and 108 are disposed adjacent to the image sensor 105 and 106 in the vertical direction.

  The imaging optical systems corresponding to the imaging elements are arranged so that the optical centers of the imaging elements 105 to 108 are shifted by 0.5 pixels in the horizontal direction and 0.5 pixels in the vertical direction. . As a result, when the imaging region of each imaging device is developed on the same optical plane, the pixels of the four imaging devices 105 to 108 are so-called nested in the horizontal and vertical directions, and as a result, the number of pixels of one imaging device is reduced. Thus, a single image having a resolution of 4 times in total is generated twice as much in the horizontal direction and twice in the vertical direction.

  This nested pixel array is shown in FIG. In FIG. 4, P <b> 105 is a pixel of the image sensor 105, and P <b> 106 is a pixel of the image sensor 106. P107 is a pixel of the image sensor 107, and P108 is a pixel of the image sensor 108. Coordinates such as (0, 0) and (1, 0) indicate the position of the pixel in each image sensor. In practice, the optical centers of the respective image sensors are arranged not only for the above-described 0.5 pixels but also for the amount of parallax correction at an arbitrarily set subject distance so as to be offset from each other.

  The image pickup device driving / synchronizing signal generation circuit 113 drives the four image pickup devices 105 to 108 and the four camera signal processing circuits 109 to 112 in synchronization, drives the image pickup devices 105 and 106 and camera signals in a certain horizontal period. The imaging signals are output from the processing circuits 109 and 110 to the synthesis processing circuit 115. In the next horizontal period, the imaging elements 107 and 108 are driven, and the imaging signals are output from the camera signal processing circuits 111 and 112 to the synthesis processing circuit 115.

  The parallax amount detection circuit 114 performs a known pattern matching process between input images, and detects the parallax amount between the input images. When there is a subject at a set subject distance, which is a subject distance at which the subject imaging position shifts between input images due to parallax, that is, the subject distance at which the image displacement is zero, the subject imaged by each imaging device Images are taken at the same position on the surface.

  However, when the subject is closer or farther than the set subject distance, the subject imaging position on the imaging surface of each imaging element is shifted. This shift amount has a magnitude corresponding to the parallax, and the image shift amount, that is, the parallax amount appearing on the image can be detected by the pattern matching. The amount of image shift or the amount of parallax corresponds to “information about parallax” in the claims, but is not limited thereto, and any information may be used as long as the information changes due to the parallax or information associated with the amount of parallax.

  For example, by providing a distance measuring unit that measures the distance from the imaging device to the subject, the distance (baseline length) between the imaging optical systems is known, so that the amount of parallax can be calculated uniquely. Furthermore, the photographer may estimate the parallax amount based on the shooting mode related to the distance to the subject. Shooting modes include a close-up (macro) shooting mode for shooting subjects that are closer than the set subject distance, a standard shooting mode for shooting subjects near the set subject distance, and a farther distance than the set subject distance. A long-distance shooting mode for shooting a subject at a distance can be considered.

  As described above, the image processing signal generated based on the pixel signals from the image sensor 105 and the image sensor 106 is input to the synthesis processing circuit 115 from the four camera signal processing circuits 109 to 112 in a certain horizontal period. In the next horizontal period, an image pickup signal generated based on pixel signals from the image pickup element 107 and the image pickup element 108 is input. In the following, an image signal generated based on a pixel signal from each image sensor is referred to as an image signal corresponding to the image sensor. The imaging signal represents a pixel value (pixel information) for each pixel of each image.

  When the parallax detected by the parallax detection circuit 114 is 0, that is, when the subject is at the set subject distance, the synthesis processing circuit 115 outputs the imaging signal input from the camera signal processing circuit to the imaging element 105 in a certain horizontal period. And the image pickup signal corresponding to the image pickup element 106 are alternately output with the same value (pixel value) as it is for each pixel, and in the next horizontal period, the image pickup signal corresponding to the image pickup element 107 and the image pickup element 108 are output. Are alternately output as they are for each pixel. As a result, a high-resolution composite image having the pixel arrangement shown in FIG. 4 in which the imaging signals (that is, pixels) from the four imaging elements 105 to 108 are nested is generated.

  On the other hand, when the subject distance is different from the set subject distance, that is, when an image shift due to parallax occurs, the synthesis processing circuit 115 outputs an imaging signal corresponding to a specific imaging element among the four imaging elements 105 to 108. As a reference imaging signal, a synthesized imaging signal generated by synthesizing the imaging signal corresponding to the reference imaging signal and another imaging element and the reference imaging signal are output for each pixel. In this embodiment, an imaging signal corresponding to the imaging element 105 is set as a reference imaging signal.

  In the composite image shown in FIG. 4, when the pixel interval is equal in the horizontal direction and the vertical direction, and this distance is 1, the pixel P106 (0, 0) is separated by 1 in the horizontal direction with reference to the pixel P105 (0, 0). , Not vertically separated (ie, zero apart). The pixel P107 (0, 0) is 0 in the horizontal direction and 1 in the vertical direction. Further, the pixel P108 is 1 in the horizontal direction and 1 in the vertical direction, and is about 1.4 away from the pixel center distance. That is, with reference to the pixel P105, the pixel P106 and the pixel P107 are separated by the same distance although there is a difference between the horizontal direction and the vertical direction, and the pixel P108 is located farthest.

  When the subject distance is different from the set subject distance, an image shift of the composite image occurs due to the influence of parallax. As described above, on the basis of an image signal corresponding to the image sensor 105, that is, an image captured by the image sensor 105, an image captured by the image sensor 106 affects the horizontal direction and is captured by the image sensor 107. The image that is captured affects the vertical direction, and the image captured by the image sensor 108 affects the horizontal and vertical directions.

  Therefore, in this embodiment, when the subject distance is the set subject distance in which the influence of the parallax is zero, that is, when the parallax is not detected by the parallax detection circuit 114, the image is captured by each image sensor by the synthesis processing circuit 115. A composite image is generated by nesting images as they are. In other words, each pixel of the composite image is a pixel having a pixel value of each input image.

  On the other hand, when parallax is detected by the parallax detection circuit 114, each pixel value of an image (first image, hereinafter referred to as a reference image) captured by the image sensor 105 is used as a pixel of the reference image in the composite image. And the pixel value of the reference image and the pixel value of the image photographed by the image sensor 106, the image sensor 107, or the image sensor 108 at a ratio according to the detected amount of parallax. The pixel value obtained in this way is used as the pixel value of the pixel that interpolates between the pixels corresponding to the reference image. Hereinafter, the generation of one pixel of a combined image by combining the pixel values of a plurality of input images in this way is referred to as a pixel value combining process.

  In this pixel value composition processing, the pixel value of a pixel (hereinafter referred to as a horizontal adjacent pixel) that is adjacent in the horizontal direction to a pixel corresponding to the reference image captured by the image sensor 105 in the composite image is the pixel value of the reference image. 3 and the pixel value of the image (second image) taken by the image sensor 106 adjacent to the image sensor 105 in FIG. Further, the pixel value of the pixel adjacent to the pixel corresponding to the reference image in the synthesized image in the vertical direction (hereinafter referred to as the vertical adjacent pixel) is the image value adjacent to the pixel value of the reference image in the vertical direction. It is obtained by synthesizing the pixel value of the photographed image (second image) by the element 107. Furthermore, the pixel value of a pixel (hereinafter referred to as a diagonally adjacent pixel) that is one distance apart in the horizontal direction and the vertical direction from the pixel corresponding to the reference image in the composite image is the pixel value of the reference image and the image sensor 105. It is obtained by synthesizing the pixel values of the image (second image) taken by the image sensor 108 separated in the horizontal and vertical directions.

  More specifically, for the synthesis of pixel values of horizontal adjacent pixels, the pixel values of two pixels of the reference image that are adjacent to the horizontal adjacent pixels on both sides in the horizontal direction are used, and the pixel values of the vertical adjacent pixels are also used. In the synthesis, the pixel values of two pixels of the reference image that are adjacent to the vertical adjacent pixels on both sides in the vertical direction are used. Further, pixel values of four pixels of the reference image that are separated by one in the horizontal direction and the vertical direction with respect to the diagonally adjacent pixels are used for the synthesis of the pixel values of the diagonally adjacent pixels.

  In the pixel value synthesis process of the present embodiment, the synthesis ratio is continuously changed according to the amount of parallax. This relationship is expressed as follows:

    Here, J is a synthesis coefficient (also referred to as an interpolation coefficient) for the pixel value of the captured image by the image sensor 106, and K is a synthesis coefficient for the pixel value of the captured image by the image sensor 107. L is a synthesis coefficient for the pixel value of the captured image by the image sensor 108. These synthesis coefficients continuously change from 1 to 0 according to the amount of parallax detected by the parallax detection circuit 114. Specifically, the larger the parallax amount is, the smaller the synthesis coefficient is set.

  When the amount of parallax detected by the parallax detection circuit 114 is 0, that is, when the subject distance is equal to the set subject distance, all of the synthesis coefficients J, K, and L are set to 1, and the calculation result by the above formula is the horizontal to the reference image. Output as pixel values of adjacent pixels, vertical adjacent pixels, and diagonally adjacent pixels.

  When the detected amount of parallax is not 0, that is, when the subject distance is different from the set subject distance, the synthesis coefficients J, K, and L are made closer to 0 as the amount of parallax increases. Accordingly, the amount of the pixel value of the captured image by the imaging elements 106, 107, and 108 having a large subject imaging position deviation with respect to the reference image is reduced, and the influence of the image having a large subject imaging position deviation in the synthesized image is reduced.

  FIG. 5 shows a graph in which the horizontal axis is the subject distance and the vertical axis is the subject imaging position deviation amount, that is, the parallax amount. Further, below the graph, the horizontal axis is the same subject distance, and the vertical axis is the synthesis coefficients J, K, and L.

  Here, the set subject distance is 1, and the parallax amount is 0 at the set subject distance. At this time, the synthesis processing circuit 115 outputs a high-resolution synthesized image using the pixel values of the captured images obtained by all the image sensors as they are. That is, the synthesis coefficients J, K, and L are set to 1 as shown in the figure.

  When the subject distance is shorter than 1, for example, close-up shooting, the amount of parallax increases abruptly. Therefore, the synthesis coefficients J, K, and L are changed from 1 to 0 according to the increased amount of parallax. As described above, since the pixel corresponding to the pixel of the image sensor 108 is farthest from the pixel corresponding to the reference image captured by the image sensor 105 in the composite image, the image is captured by the image sensor 108 having the greatest parallax effect. The decrease rate of the synthesis coefficient L with respect to the pixel value of the image with respect to the increase in the amount of parallax is larger than the decrease rate of the synthesis coefficients J and K with respect to the pixel value of the captured image by the imaging elements 106 and 107.

  As described above, the composition processing circuit 115 uses the reference values of the pixel values of the captured images by the image sensors 106, 107, and 108 to generate image shifts due to the influence of the parallax in accordance with the parallax amount that rapidly increases as the subject distance becomes shorter. The effect of parallax is reduced by reducing the composition ratio (ratio) with respect to the pixel value of the image.

  In addition, when the subject distance is longer than 1, the amount of parallax increases more slowly as the subject distance increases compared to close-up photography. In accordance with this, the synthesis coefficients J, K, and L are also decreased. At this time as well, as in close-up shooting, the reduction rate of the synthesis coefficient L with respect to the pixel value of the captured image by the image sensor 108 that has a large influence on the image shift of the composite image is set to It is made larger than the reduction rate of J and K which are synthesis coefficients.

  FIG. 6 shows a flowchart when the processing in this embodiment is performed by an image processing program which is a computer program. This image processing program is executed by the CPU 130 as the controller shown in FIG.

  When the processing is started, first, in step (abbreviated as S in the figure) 101, imaging by the imaging elements 105 to 108 and the camera signal processing circuits 109 to 112 is performed according to an instruction from the CPU 130, and the camera signal processing circuits 109 to 112 are performed. Each generate an input image.

  Next, in step 102, the parallax detection circuit 114 detects the amount of parallax. As a detection method, as described above, pattern matching between input images, distance measurement up to a subject, detection by photographing mode, and the like are used.

  In step 103, in accordance with an instruction from the CPU 130, the composition processing circuit 115 generates and outputs one composite image from a plurality of input images using the detected amount of parallax and the equations (2) to (5). .

  As described above, according to the present embodiment, when there is no parallax in an imaging apparatus that outputs a single composite image using pixel values of an input image captured using a plurality of optical systems having parallax. Can obtain a high-resolution composite image by using the pixel values of a plurality of input images as they are. On the other hand, when there is a parallax, the composition ratio between the pixel value of the reference input image and the pixel value of the other input image is continuously reduced according to the amount of the parallax as the parallax amount increases. By changing to, it is possible to obtain a high-quality and natural composite image with little image shift due to parallax. In this way, by changing the method for generating a composite image according to the presence or absence of the amount of parallax, image quality degradation due to the influence of parallax can be minimized.

  In this embodiment, horizontal, vertical, and vertical using a primary interpolation method that performs interpolation calculation with reference to pixels on a straight line passing through two points represented by the calculation formulas (2) to (5). The case where the pixel value of the diagonally adjacent pixel is calculated has been described. In addition, a nearest neighbor method using the pixel value of the adjacent pixel as it is (nearest neighbor), or a curve represented by a cubic polynomial with reference to surrounding pixels A bicubic (third-order interpolation) method that interpolates using equations may be used.

  In the above embodiment, the case where the method of generating the composite image is changed depending on whether there is no parallax or not is described. However, the parallax amount is a predetermined value (for example, the influence of the parallax in the composite image can be substantially ignored). The composite image generation method may be changed depending on whether it is smaller or larger than the amount of parallax. That is, when the parallax is smaller than the predetermined value, the pixel values of the plurality of input images are used as they are, and when the parallax is larger than the predetermined value, the pixel value of the reference input image according to the amount of the parallax and other values The composition ratio with the pixel value of the input image is decreased with respect to the increase in the amount of parallax.

  In addition, the single composite image described in the present embodiment may be a still image or a single frame image of a moving image. That is, the imaging apparatus of the present embodiment can be used as both a still image camera and a video camera.

  Further, in the first embodiment, the case where the image processing unit corresponding to the image processing apparatus is integrated in the imaging apparatus has been described. However, the image processing apparatus of the present invention is different from the imaging apparatus such as a personal computer. The apparatus may be used. In this case, a plurality of images taken by an image pickup apparatus having a compound eye optical system and an image pickup device are imaged via a cable, a recording medium such as a semiconductor memory, an optical disk, or a magnetic disk, or a network such as the Internet or a LAN. The image is taken into a processing apparatus, and a composite image is generated in the image processing apparatus by the method described above.

  FIG. 7 shows an example of the imaging apparatus described in the first embodiment and an electronic apparatus equipped with the imaging apparatus.

  FIG. 7A shows a card type camera as an imaging device. Reference numeral 701 denotes a camera body, and reference numeral 702 denotes an image pickup system having the compound eye optical system shown in FIG. Reference numeral 703 denotes a shooting button for starting a shooting operation, reference numeral 704 denotes a finder window, and reference numeral 705 denotes a flash light emitting unit.

  FIG. 7B illustrates a personal computer that is an electronic device on which an imaging device is mounted. Reference numeral 710 denotes a computer main body, and reference numeral 711 denotes a display unit that can be opened and closed with respect to the main body 710. An imaging device 712 is attached to the upper portion of the display unit 711 so as to be rotatable up and down. Reference numeral 713 denotes the imaging system shown in FIG.

  In addition to the computer shown in FIG. 7B, the image pickup apparatus according to the first embodiment can be mounted on various electronic devices such as a mobile phone, a PDA (Personal Digital Assistance), and the like.

1 is a block diagram illustrating a configuration of an imaging apparatus including an image processing unit that is Embodiment 1 of the present invention. FIG. 3 is a schematic diagram illustrating a configuration of an imaging system in the imaging apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an arrangement of image pickup elements that form the image pickup system according to the first embodiment. FIG. 3 is a diagram illustrating a pixel array of a composite image generated by the image processing unit according to the first embodiment. The graph which shows the relationship between parallax amount and a synthetic | combination coefficient. 6 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment. Schematic which shows the imaging device which is Example 2 of this invention, and an electronic device carrying this. The block diagram which shows the structure of the conventional compound eye imaging device. The schematic diagram which shows the relationship between a to-be-photographed object position and the image shift | offset | difference on a synthesized image. The schematic diagram which shows the relationship between the convergence angle and the shift | offset | difference of a to-be-photographed object position. The graph which shows the relationship between a to-be-photographed object distance and the deviation | shift amount of a to-be-photographed object position.

Explanation of symbols

101-104 Imaging optical system (compound eye optical system)
105-108 Image sensor 201 Aperture 701 Card type camera 710 Personal computer

Claims (10)

An image processing device that generates a single image based on pixel information of a plurality of input images,
Combining means for obtaining at least some of the pixels in the single image by combining pixel information of at least two input images of the plurality of input images;
The synthesizing unit changes the synthesis ratio of the pixel information based on information on parallax between the input images. The single image includes a pixel corresponding to the pixel information of the first input image and a pixel obtained by combining the pixel information of the first input image and the second input image,
The image processing apparatus according to claim 1, wherein the combining unit changes a combining ratio between pixel information of the first and second input images.   3. The image processing apparatus according to claim 2, wherein the synthesizing unit reduces the synthesis ratio of the pixel information of the second input image with respect to the pixel information of the first input image as the parallax increases. Image generating means for generating a single image based on pixel information of a plurality of input images;
The image processing device changes the method for generating the single image based on information on parallax between the input images.   If the parallax does not exist or is smaller than a predetermined value, the image generation means sets each pixel of the single image as a pixel corresponding to pixel information of each input image, and if the parallax is present or larger than the predetermined value The image processing apparatus according to claim 4, wherein at least some of the pixels in the single image are obtained by combining pixel information of at least two input images of the plurality of input images.   The image processing apparatus according to claim 1, further comprising a parallax acquisition unit configured to acquire information regarding the parallax. A compound eye optical system;
An image sensor that photoelectrically converts a plurality of subject images formed by the compound eye optical system;
7. The image generating apparatus according to claim 1, wherein the image generating apparatus generates a single image based on pixel information of a plurality of input images generated based on an output from the imaging element. An imaging device.   An electronic apparatus comprising the imaging device according to claim 7. An image processing program operating on a computer that generates a single image based on pixel information of a plurality of images,
Inputting the plurality of images;
A synthesis step of obtaining at least some of the pixels in the single image by synthesizing pixel information of at least two images of the plurality of input images;
Obtaining information on parallax between the input images,
An image processing program characterized in that, in the combining step, a combining ratio of the pixel information is changed based on information on the parallax. Inputting multiple images;
An image generation step of generating a single image based on pixel information of the plurality of input images;
Obtaining information on parallax between the input images,
An image processing program operating on a computer, wherein, in the image generation step, a method for generating the single image is changed based on information on the parallax.