JP2009290344A - Image processing device and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress the sensation of degradation in the resolution and generation of jaggies which are produced, when summing reading-out or decimation reading-out of pixel signals is performed. <P>SOLUTION: An image processing device performs summing reading-out of the pixel signals of a light-receiving pixel group which are two-dimensionally arranging on imaging elements of a single plane system, and acquires an original image. Here, a plurality of original images are acquired, (Fig.35(a) to (d)) whose pixel positions with pixel signals are different each other, by using a plurality of adding patterns whose combination of receiving pixels to be targets of the addition are different. Pixel signals of the same color contained in the pixel signal group of the original image are mixed with one another for respective images, and an interpolation image with a pixel signal obtained by the mixing is generated. Afterwards, a sheet of constituted output image is produced, by compositing a plurality of interpolation images produced from a plurality of original images. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs image processing on an acquired original image, and an imaging apparatus such as a digital video camera.

  When a moving image is captured by an image sensor (image sensor) that can capture a still image composed of multiple pixels, it is necessary to reduce the frame rate in accordance with the readout speed of the pixel signal from the image sensor. In order to realize a high frame rate, it is necessary to reduce image data by performing addition readout or thinning readout of pixel signals.

  However, although the pixel interval on the image sensor is uniform in the horizontal and vertical directions, when addition reading or thinning readout is performed, the interval at which pixel signals exist becomes unequal. If an image having pixel signals that are unevenly arranged is displayed as it is, jaggies and false colors are generated. As a technique for avoiding this problem, a technique has been proposed in which interpolation processing is performed so that pixel intervals in which pixel signals exist are equal (see, for example, Patent Documents 1 and 2).

  This method will be described with reference to FIG. In FIG. 54, a block 901 indicates a color filter array (Bayer array) arranged in front of the light receiving pixels of the image sensor that employs the single plate method. A block 902 indicates the presence positions of the R, G, and B signals obtained from the image sensor by performing addition reading. In addition reading, pixel signals in pixels near the target position are added, and the added signal is read out from the image sensor as a pixel signal at the target position. For example, by adding the pixel signals of the actual light receiving pixels adjacent to the upper left position, upper right position, lower left position, and lower right position of the target position where the G signal is to be generated, Generated. In FIG. 54, the target position for the G signal is indicated by a black circle and the state of signal addition is indicated by an arrow connected to the circle, but the same addition reading is also performed for the B and R signals. .

  As indicated by block 902, the pixel spacing of the image obtained by performing additive readout is uneven. By performing an interpolation process for correcting this unequal pixel interval to an equal pixel interval, an image having a pixel signal arrangement as shown in blocks 903 and 904 is obtained. That is, an image in which R, G, and B signals are arranged like a Bayer array is obtained. A so-called demosaicing process (color synchronization process) is performed on the image (RAW data) indicated by the block 904, whereby the output image indicated by the block 905 is obtained. The output image is a two-dimensional image in which pixels are arranged at equal intervals in the horizontal and vertical directions, and R, G, and B signals are assigned to each pixel in the output image.

JP 2003-092764 A JP 2003-299112 A

  In the output image obtained by using the conventional method as shown in FIG. 54, the pixel intervals in which the R, G, and B signals exist are uniform, so that the occurrence of jaggy and false colors is suppressed. However, an interpolation process for equalizing the pixel intervals is executed in order to eliminate the non-uniform pixel intervals resulting from the addition reading. Execution of such an interpolation process inevitably causes degradation in resolution (substantial degradation in resolution). The same problem occurs when thinning out pixel signals.

  Therefore, an object of the present invention is to provide an image processing apparatus and an imaging apparatus that contribute to the suppression of resolution degradation that may occur when pixel signals are added or read out.

  An image processing apparatus according to the present invention includes a reading unit that performs addition reading or thinning readout of pixel signals of a light receiving pixel group that is two-dimensionally arranged on a single-plate image pickup device, and is a light receiving pixel that is a target of addition or thinning. And a plurality of readout patterns having different combinations of the original image acquisition means for acquiring a plurality of original images having pixel positions different from each other, and each original image is included in the pixel signal group of the original image. And interpolating means for generating interpolated images having pixel signals obtained by mixing the pixel signals of the same color.

  The image processing apparatus is provided with original image acquisition means for acquiring a plurality of original images having different pixel positions having pixel signals, and an interpolated image is obtained by mixing pixel signals of the same color for each original image. Thereby, it is possible to omit the interpolation processing corresponding to FIG. 54 for equalizing the pixel position intervals, and as a result, it is possible to suppress degradation in resolution caused by the interpolation processing. Incidentally, for problems that may arise due to not performing this interpolation processing, for example, by synthesizing a plurality of interpolation images generated from the plurality of original images, or by combining the plurality of interpolation images. This can be dealt with by outputting as a moving image.

  Specifically, for example, the pixel signal group of each interpolation image includes a plurality of pixel signals including the first color, and in each interpolation image, an interval between pixel positions where the pixel signal of the first color exists is It is uneven.

  The interpolation processing corresponding to FIG. 54 for equalizing the pixel position intervals is not performed, and the pixel position intervals are unequal in each interpolated image. As a result, it is possible to suppress degradation of resolution due to the interpolation processing corresponding to FIG.

  Specifically, for example, a focused original image included in the plurality of original images is referred to as a focused original image, an interpolation image generated from the focused original image is referred to as a focused interpolation image, and the plurality When the pixel signal of the target color included in the color is called the target color pixel signal, the interpolation processing unit sets the interpolation pixel position at a position different from the pixel position where the target color pixel signal of the target original image exists. A plurality of pixel positions on the target original image where the target color pixel signal exists when generating an image having the target color pixel signal at the interpolation pixel position as the target interpolation image and generating the target interpolation image. Paying attention to generate a pixel signal for the interpolated pixel position by mixing a plurality of target color pixel signals at the plurality of pixel positions, and the interpolated pixel position is set to the center of gravity of the plurality of pixel positions It is.

  For example, the original image of interest and the interpolation image of interest are the original image 251 shown in FIGS. 12A to 12C and the color interpolation image 261 shown in FIGS. 13A to 13C, respectively. When the target color is green and the interpolation pixel position is the interpolation pixel position 301 shown in FIG. 12A, the pixel position is different from the pixel position where the target color pixel signal (G signal) of the target original image (251) exists. The interpolation pixel position (301) is set as the position, and an image having the pixel signal (G signal) of the target color at the interpolation pixel position (301) is generated as the target interpolation image (261). At this time, a plurality of pixel positions on the target original image (251) where the target color pixel signal (G signal) exists is noted, and the interpolation is performed by mixing a plurality of target color pixel signals at the plurality of pixel positions. A pixel signal for the pixel position (301) is generated. Then, the interpolation pixel position (301) is set to the barycentric position of the plurality of pixel positions.

  More specifically, for example, the interpolation processing unit generates a pixel signal for the interpolation pixel position by mixing the plurality of target color pixel signals at an equal ratio.

  Due to such mixing, the interval between the pixel positions where the pixel signals of the first color are present becomes uneven.

  For example, the image processing apparatus further includes output image generation means for generating one output image based on a plurality of interpolation images generated from the plurality of original images by the interpolation processing means, and the output image includes pixels arranged in an equal arrangement. Each of the positions has pixel signals for a plurality of colors, and the output image generation means is derived from a difference in a corresponding readout pattern between the plurality of interpolation images, and a difference in pixel positions between the plurality of interpolation images Based on the above, the output image is generated.

  Since the pixel interval of the output image is uniform, the occurrence of jaggy and false color is suppressed in the output image.

  In addition, for example, it further includes a motion detection unit that detects a motion of an object between the plurality of interpolation images, and the output image generation unit is configured to output the output based on not only the difference in the pixel position but also the magnitude of the motion. An image may be generated.

  More specifically, for example, the plurality of interpolated images include at least first and second interpolated images, and the output image generating unit is configured to detect the first interpolated image and the first interpolated image detected by the motion detecting unit. Weight coefficient setting means for setting a weight coefficient based on the magnitude of the motion of the object between the two interpolated images, and mixing the pixel signals of the first and second interpolated images according to the weight coefficient To generate the output image.

  Thereby, it is also possible to suppress the blurring of the outline portion and the generation of the double image in the output image.

  Further, for example, by repeatedly executing the operation of acquiring the plurality of original images using the plurality of readout patterns, an output image sequence arranged in a time series is generated by the output image generation unit, and the image processing apparatus Further includes image compression means for generating a compressed moving image including an intra-frame encoded image and an inter-frame prediction encoded image by performing an image compression process on the output image sequence, and the image compression means includes: An output image to be a target of the intra-frame encoded image may be selected from the output image sequence based on a weighting factor set corresponding to each output image forming the output image sequence. .

  This contributes to improving the overall image quality of the compressed moving image.

  Further, for example, a plurality of interpolation images generated from the plurality of original images by the interpolation processing means may be output as moving images.

  Further, for example, there are a plurality of readout patterns having different combinations of light receiving pixels to be added or thinned out, and the image processing apparatus further includes a motion detection unit that detects the motion of the object between the plurality of interpolation images. A set of read patterns used for acquiring the original image may be variably set based on the detected direction of motion.

  Thereby, a set of readout patterns suitable for the movement of the object in the image is dynamically variably set.

  An image pickup apparatus according to the present invention includes a single-plate image pickup device and any one of the image processing apparatuses described above.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the image processing apparatus and imaging device which contribute to suppression of the feeling of resolution which may arise when the addition reading or thinning-out reading of a pixel signal is performed.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to eighth embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital video camera, for example. The imaging device 1 can capture a moving image and a still image, and can also capture a still image simultaneously during moving image capturing.

[Description of basic configuration]
The imaging apparatus 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a video signal processing unit 13, a microphone 14, an audio signal processing unit 15, a compression processing unit 16, and a DRAM (Dynamic Random Access Memory). An internal memory 17 such as an SD (Secure Digital) card or a magnetic disk, a decompression processing unit 19, a VRAM (Video Random Access Memory) 20, an audio output circuit 21, and a TG (timing generator). 22, a CPU (Central Processing Unit) 23, a bus 24, a bus 25, an operation unit 26, a display unit 27, and a speaker 28. The operation unit 26 includes a recording button 26a, a shutter button 26b, an operation key 26c, and the like. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25.

  The TG 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and provides the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each unit in the imaging apparatus 1. The operation unit 26 receives an operation by a user. The operation content given to the operation unit 26 is transmitted to the CPU 23. Each unit in the imaging apparatus 1 temporarily records various data (digital signals) in the internal memory 17 during signal processing as necessary.

  The imaging unit 11 includes an imaging system (image sensor) 33, an optical system (not shown), a diaphragm, and a driver. Incident light from the subject enters the image sensor 33 via the optical system and the stop. Each lens constituting the optical system forms an optical image of the subject on the image sensor 33. The TG 22 generates a drive pulse for driving the image sensor 33 in synchronization with the timing control signal, and applies the drive pulse to the image sensor 33.

  The image sensor 33 is a solid-state image sensor composed of a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 33 photoelectrically converts an optical image incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels (not shown in FIG. 1) that are two-dimensionally arranged in a matrix, and in each photographing, each light receiving pixel has a charge amount signal corresponding to the exposure time. Stores charge. The electrical signal from each light receiving pixel having a magnitude proportional to the amount of the stored signal charge is sequentially output to the subsequent AFE 12 in accordance with the drive pulse from the TG 22.

  The AFE 12 amplifies the analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The degree of amplification of signal amplification in the AFE 12 is controlled by the CPU 23. The video signal processing unit 13 performs various types of image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The video signal is generally composed of a luminance signal Y representing the luminance of the image and color difference signals U and V representing the color of the image.

  The microphone 14 converts the peripheral sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 using a predetermined compression method. The compressed video signal is recorded in the external memory 18 at the time of capturing and recording a moving image or a still image. The compression processing unit 16 compresses the audio signal from the audio signal processing unit 15 using a predetermined compression method. At the time of moving image shooting and recording, the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 are compressed while being correlated with each other in time by the compression processing unit 16, and after compression, Recorded in the external memory 18.

  The recording button 26a is a push button switch for instructing start / end of moving image shooting and recording, and the shutter button 26b is a push button switch for instructing shooting and recording of a still image.

  The operation mode of the imaging apparatus 1 includes a shooting mode capable of shooting a moving image and a still image, and a playback mode for reproducing and displaying the moving image and the still image stored in the external memory 18 on the display unit 27. . Transition between the modes is performed according to the operation on the operation key 26c.

  In the shooting mode, shooting is sequentially performed at a predetermined frame period, and a shot image sequence is acquired from the image sensor 33. An image sequence typified by a captured image sequence refers to a collection of images arranged in time series. Data representing an image is called image data. Image data can also be considered as a kind of video signal. One image is represented by image data for one frame period. The video signal processing unit 13 performs various types of image processing on the image represented by the output signal of the AFE 12, and the image represented by the output signal itself of the AFE 12 before this image processing is referred to as an original image. . Accordingly, one original image is represented by the output signal of the AFE 12 for one frame period.

  When the user presses the recording button 26a in the shooting mode, under the control of the CPU 23, the video signal obtained after the pressing and the corresponding audio signal are sequentially recorded in the external memory 18 via the compression processing unit 16. . When the user presses the recording button 26a again after starting the moving image shooting, the recording of the video signal and the audio signal to the external memory 18 is completed, and the shooting of one moving image is completed. In the shooting mode, when the user presses the shutter button 26b, a still image is shot and recorded.

  In the playback mode, when the user performs a predetermined operation on the operation key 26c, a compressed video signal representing a moving image or a still image recorded in the external memory 18 is expanded by the expansion processing unit 19 and written to the VRAM 20. . In the shooting mode, the video signal is normally generated by the video signal processing 13 regardless of the operation contents of the recording button 26a and the shutter button 26b, and the video signal is written in the VRAM 20.

  The display unit 27 is a display device such as a liquid crystal display, and displays an image corresponding to the video signal written in the VRAM 20. In addition, when a moving image is reproduced in the reproduction mode, a compressed audio signal corresponding to the moving image recorded in the external memory 18 is also sent to the expansion processing unit 19. The decompression processing unit 19 decompresses the received audio signal and sends it to the audio output circuit 21. The audio output circuit 21 converts a given digital audio signal into an audio signal in a format that can be output by the speaker 28 (for example, an analog audio signal) and outputs the audio signal to the speaker 28. The speaker 28 outputs the sound signal from the sound output circuit 21 to the outside as sound (sound).

[Light receiving pixel array of image sensor]
FIG. 2 shows a light receiving pixel array in the effective area of the image sensor 33. The effective area of the image sensor 33 has a rectangular shape, and one vertex of the rectangle is regarded as the origin of the image sensor 33. It is assumed that the origin is located at the upper left corner of the effective area of the image sensor 33. The number of light receiving pixels corresponding to the product of the number of effective pixels in the vertical direction of the image sensor 33 and the number of effective pixels in the horizontal direction (for example, the square of several hundred to several thousand) is two-dimensionally arranged. The effective area is formed. Each light receiving pixel in the effective area of the image sensor 33 is represented by P _S [x, y]. Here, x and y are integers. It is assumed that the value of the corresponding variable x increases as the light receiving pixel located on the right side as viewed from the origin of the image sensor 33, and the value of the corresponding variable y increases as the light receiving pixel located below. In the image sensor 33, the vertical direction corresponds to the vertical direction, and the horizontal direction corresponds to the horizontal direction.

In FIG. 2, only a 10 × 10 light receiving pixel region is shown for convenience, and this light receiving pixel region is referred to by reference numeral 200. In the following description, attention is particularly paid to the light receiving pixels in the light receiving pixel region 200. In the light receiving pixel region 200, a total of 100 light receiving pixels P _S [x, y] satisfying the inequalities “1 ≦ x ≦ 10” and “1 ≦ y ≦ 10” are shown. Of the light receiving pixel group belonging to the light receiving pixel region 200, the arrangement position of the light receiving pixel P _S [1,1] is closest to the origin of the image sensor 33, and the arrangement position of the light receiving pixel P _S [10,10] is most imaged. It is far from the origin of the element 33.

The imaging device 1 employs a so-called single plate method that uses only one image sensor. FIG. 3 shows an arrangement of color filters arranged in front of each light receiving pixel of the image sensor 33. The arrangement shown in FIG. 3 is generally called a Bayer arrangement. Color filters include a red filter that transmits only the red component of light, a green filter that transmits only the green component of light, and a blue filter that transmits only the blue component of light. Red filter is placed in front of the light receiving pixels _{P S [2n A -1,2n B]} , the blue filter is disposed in front of the light receiving pixels _{P S [2n A, 2n B} -1], green filter, It is arranged in front of the light receiving pixel P _S [2n _A -1,2 n _B -1] or P _S [2n _A , 2n _B ]. Here, n _A and n _B are integers. In FIG. 3 and FIG. 5 to be described later, a part corresponding to the red filter is represented by R, a part corresponding to the green filter is represented by G, and a part corresponding to the blue filter is represented by B.

  The light receiving pixels in which the red filter, the green filter, and the blue filter are arranged in front are also referred to as a red light receiving pixel, a green light receiving pixel, and a blue light receiving pixel, respectively. Each light receiving pixel converts light incident on itself through a color filter into an electrical signal by photoelectric conversion. This electric signal represents a pixel signal of the light receiving pixel, and hereinafter, it may be referred to as a “light receiving pixel signal”. Each of the red light receiving pixel, the green light receiving pixel, and the blue light receiving pixel reacts only to the red component, the green component, and the blue component of the incident light of the optical system.

  As a method of reading out the light receiving pixel signal from the image sensor 33, there are an all-pixel reading method, an addition reading method, and a thinning-out reading method. When the light receiving pixel signal is read from the image sensor 33 by the all pixel reading method, the light receiving pixel signals from all the light receiving pixels located in the effective area of the image sensor 33 are individually given to the video signal processing unit 13 via the AFE 12. It is done. The addition reading method and the thinning reading method will be described later. In the following description, for simplification of description, signal amplification and digitization in the AFE 12 are ignored.

[Pixel array of original image]
FIG. 4A shows the pixel array of the original image. FIG. 4A shows only a partial image region of the original image corresponding to the light receiving pixel region 200 of FIG. It can be considered that an arbitrary image including the original image is formed from a pixel group two-dimensionally arranged on the image coordinate plane XY which is a two-dimensional orthogonal coordinate system (see FIG. 4B).

The symbol P [x, y] represents a pixel on the original image corresponding to the light receiving pixel P _S [x, y]. As viewed from the origin of the original image corresponding to the origin of the image sensor 33, the value of the variable x in the corresponding symbol P [x, y] increases as the pixel is located on the right side, and the pixel located on the lower side represents It is assumed that the value of the variable y in the corresponding symbol P [x, y] increases. In the original image, the vertical direction corresponds to the vertical direction, and the horizontal direction corresponds to the horizontal direction.

In the following description, the position of the image sensor 33 is represented by the symbol [x, y], and the position on the arbitrary image including the original image (position on the image coordinate plane XY) is also represented by the symbol [x, y]. Represented by The position [x, y] on the image sensor 33 matches the position on the image sensor 33 of the light receiving pixel P _S [x, y], and the position [x, y] on the image (image coordinate plane XY) is the original. It matches the position of the pixel P [x, y] of the image. However, since each light receiving pixel of the image sensor 33 and each pixel on the original image have a certain size that is not zero in the horizontal and vertical directions, strictly speaking, the position [x, y] in the image sensor 33 Matches the center position of the light receiving pixel P _S [x, y], and the position [x, y] in the image (image coordinate plane XY) matches the center position of the pixel P [x, y] of the original image. . In the following description, the symbol [x, y] may be used as a symbol representing the pixel position in order to clarify that the position of the pixel (or the light receiving pixel) is indicated.

  Note that the horizontal width of one pixel on the original image is represented by Wp (see FIG. 4A). The vertical width of one pixel on the original image is also Wp. Therefore, on the image (image coordinate plane XY), the distance between the position [x, y] and the position [x + 1, y] and the distance between the position [x, y] and the position [x, y + 1] are both Wp. is there.

When the all-pixel readout method is used, the light receiving pixel signal of the light receiving pixel P _S [x, y] output from the AFE 12 is the pixel signal of the pixel P [x, y] of the original image. FIG. 5 shows an image diagram of pixel signals in the original image 220 obtained by using the all-pixel readout method. In FIG. 5 and FIGS. 6A to 6E described later, only the portions corresponding to the pixel positions [1, 1] to [4, 4] are shown for simplification of illustration. Further, in FIG. 5 and FIGS. 6A to 6E described later, the color component (R, G, or B) represented by the pixel signal is shown corresponding to the pixel position.

In the original image 220, pixel signals of the pixel position [2n _A -1, 2n _B] is a light receiving pixel signals of the red light receiving pixels P _S output [2n _A -1, 2n _B] from AFE 12, the pixel position [ 2n _a, the pixel signals of 2n _B -1] is, blue light receiving pixels P _S [2n _a outputted from the AFE 12, a light receiving pixel signals of 2n _B -1], the pixel position [2n _a -1,2n _B - 1] or P [2n _A , 2n _B ] pixel signals are output from the AFE 12 as light receiving pixels P _S [2n _A −1, 2n _B −1] or P _S [2n _A , 2n _B ]. Signal (n _A and n _B are integers). As described above, when the all-pixel readout method is used, the pixel interval on the image is equal to the light receiving pixel interval on the image sensor 33.

  In the original image 220, a pixel signal of only one color component among the red component, the green component, and the blue component exists for one pixel position. The video signal processing unit 13 uses interpolation processing to perform processing for assigning pixel signals of three color components to each pixel forming the image. This process is called a color interpolation process. The color interpolation process is generally called a demosaicing process and sometimes called a color synchronization process.

  Hereinafter, in an arbitrary image including an original image, pixel signals representing data of a red component, a green component, and a blue component are referred to as an R signal, a G signal, and a B signal, respectively. In addition, any of the R signal, the G signal, and the B signal may be referred to as a color signal, and they may be collectively referred to as a color signal.

  6A to 6C are conceptual diagrams of color interpolation processing performed on the original image 220, and FIGS. 6D to 6E illustrate color interpolation processing performed on the original image 220. FIG. It is an image figure of the color interpolation image 230 obtained by this. 6A to 6C are conceptual diagrams of color interpolation processing for the G, B, and R signals, respectively. In FIGS. 6D to 6E, each pixel position of the color interpolation image 230 has G , B and R signals are shown. In FIGS. 6A to 6C, G, B, and R surrounded by circles are G, B, and R obtained by interpolation processing using peripheral pixels (pixels located at the roots of arrows), respectively. Represents a signal. In order to prevent complication, the G, B, and R signals in the color interpolation image 230 are shown separately, but one color interpolation image 230 is generated from the original image 220.

  As is well known, in the color interpolation process for the original image 220, the pixel signal of the target color in the target pixel is generated by mixing the pixel signals of the target color in the peripheral pixels of the target pixel. For example, as shown in FIG. 6A, an average signal of pixel signals at pixel positions [3, 1], [2, 2], [4, 2] and [3, 3] in the original image 220 is color interpolation. The G signal of the pixel position [3, 2] in the image 230 is generated and the pixel signals of the pixel positions [2, 2], [1, 3], [3, 3] and [2, 4] in the original image 220 are The average signal is generated as the G signal at the pixel position [2, 3] in the color interpolation image 230. The pixel signals at pixel positions [2, 2] and [3, 3] in the original image 220 are directly used as G signals at pixel positions [2, 2] and [3, 3] in the color interpolation image 230, respectively. . Similarly, B and R signals of each pixel in the color interpolation image 230 are also generated according to a known signal interpolation method.

  The imaging apparatus 1 performs a characteristic operation when using the addition reading method or the thinning reading method. Hereinafter, first to eighth embodiments will be described as embodiments for explaining a method for realizing this characteristic operation. As long as there is no contradiction, the matters described in one embodiment also apply to other embodiments.

<< First Example >>
First, the first embodiment will be described. In the first embodiment, as a method of reading out pixel signals from the image sensor 33, an addition reading method of reading out while adding a plurality of light receiving pixel signals is used. At this time, addition reading is performed while sequentially changing the addition pattern to be used between the plurality of addition patterns, and one output composite image is generated by combining a plurality of color interpolation images having different addition patterns. The addition pattern means a combination pattern of light receiving pixels to be added. The plurality of addition patterns used include two or more addition patterns among first, second, third, and fourth addition patterns that are different from each other.

FIGS. 7A, 7B, 8A, and 8B show how signals are added when the first, second, third, and fourth addition patterns are used, respectively. The first, second, third, and fourth addition patterns corresponding to FIGS. 7A, 7B, 8A, and 8B are added to the addition patterns P _A1 , P _A2 , and P _{A3, respectively.} And P _A4 . FIGS. 9A, 9B, 9C, and 9D are original images obtained when addition reading is performed using the first, second, third, and fourth addition patterns, respectively. The state of the pixel signal is shown. As described above, attention is paid to the light receiving pixel region 200 including the light receiving pixels P _S [1,1] to P _S [10, 10] (see FIG. 2).

  Black circles shown in FIGS. 7A, 7B, 8A, and 8B are assumed when the first, second, third, and fourth addition patterns are used, respectively. The arrangement positions of the virtual light receiving pixels are shown. In FIGS. 7A, 7B, 8A, and 8B, the arrows shown around the black circles generate pixel signals of virtual light receiving pixels corresponding to the circles. For this reason, the pixel signals of the peripheral light receiving pixels of the virtual light receiving pixels are added.

When using the addition pattern P _A1 as the first addition pattern,
Pixel position of the image sensor _{33 [2 + 4n A, 2} + 4n B] and virtual green light receiving pixels are arranged in a _{[3 + 4n A, 3 +} 4n B], virtual blue pixel positions of the image sensor _{33 [3 + 4n A, 2} + 4n B] It is assumed that a light receiving pixel is disposed and a virtual red light receiving pixel is disposed at a pixel position [2 + 4n _A , 3 + 4n _B ] of the image sensor 33.
When using the addition pattern P _A2 as the second addition pattern,
Virtual green light receiving pixels are arranged at pixel positions [4 + 4n _A , 4 + 4n _B ] and [5 + 4n _A , 5 + 4n _B ] of the image sensor 33, and virtual blue at the pixel positions [5 + 4n _A , 4 + 4n _B ] of the image sensor 33. It is assumed that a light receiving pixel is arranged and a virtual red light receiving pixel is arranged at a pixel position [4 + 4n _A , 5 + 4n _B ] of the image sensor 33.
When using the addition pattern P _A3 as the third addition pattern,
Virtual green light receiving pixels are arranged at pixel positions [4 + 4n _A , 2 + 4n _B ] and [5 + 4n _A , 3 + 4n _B ] of the image sensor 33, and virtual blue at the pixel positions [5 + 4n _A , 2 + 4n _B ] of the image sensor 33. It is assumed that a light receiving pixel is arranged and a virtual red light receiving pixel is arranged at the pixel position [4 + 4n _A , 3 + 4n _B ] of the image sensor 33.
When using the addition pattern P _A4 as the fourth addition pattern,
Pixel position of the image sensor _{33 [2 + 4n A, 4} + 4n B] and virtual green light receiving pixels are arranged in a _{[3 + 4n A, 5 +} 4n B], virtual blue pixel positions of the image sensor _{33 [3 + 4n A, 4} + 4n B] It is assumed that a light receiving pixel is arranged and a virtual red light receiving pixel is arranged at a pixel position [2 + 4n _A , 5 + 4n _B ] of the image sensor 33.
Note that n _A and n _B are integers as described above.

The pixel signal of one virtual light receiving pixel is an addition signal of the pixel signals of the actual light receiving pixels adjacent to the upper left, upper right, lower left, and lower right of the virtual light receiving pixel. . For example, when the addition pattern P _A1 is used, the pixel signals of the virtual green light receiving pixels arranged at the pixel position [2, 2] are the actual green light receiving pixels P _S [1, 1], [3, 1 ], [1,3] and [3,3] are added signals. In this manner, by adding the pixel signals of the four light receiving pixels in which the color filters of the same color are arranged, a pixel signal of one virtual light receiving pixel located at the center of the four light receiving pixels is formed. This is the same even when any addition pattern (including addition patterns P _{B1 to} P _B4 , P _{C1 to} P _C4, and P _{D1 to} P _D4 described later) is used.

  Then, the original image is acquired so that the pixel signal of the virtual light receiving pixel arranged at the position [x, y] is handled as the pixel signal at the position [x, y] on the image.

Therefore, as shown in FIG. 9A, the original image obtained by the addition reading using the first addition pattern (P _A1 ) is at pixel positions [2 + 4n _A , 2 + 4n _B ] and [3 + 4n _A , 3 + 4n _B ]. The pixel having only the G signal, the pixel having only the B signal, disposed at the pixel position [3 + 4n _A , 2 + 4n _B ], and the R signal disposed at the pixel position [2 + 4n _A , 3 + 4n _B ]. And an image including a pixel having only a pixel.
Similarly, as shown in FIG. 9B, the original image obtained by addition reading using the second addition pattern (P _A2 ) has pixel positions [4 + 4n _A , 4 + 4n _B ] and [5 + 4n _A , 5 + 4n _B ]. The pixel having only the G signal, the pixel having only the B signal, and the pixel having only the B signal, and the pixel position [4 + 4n _A , 5 + 4n _B ], arranged at the pixel position [5 + 4n _A , 4 + 4n _B ]. An image including pixels having only signals.
Similarly, the original image obtained by addition reading using the third addition pattern (P _A3 ) has pixel positions [4 + 4n _A , 2 + 4n _B ] and [5 + 4n _A , 3 + 4n _B ] as shown in FIG. The pixel having only the G signal, the pixel having only the B signal, and the pixel having only the B signal and the pixel position [4 + 4n _A , 3 + 4n _B ], which are arranged at the pixel position [5 + 4n _A , 2 + 4n _B ]. An image including pixels having only signals.
Similarly, the original image obtained by addition reading using the fourth addition pattern (P _A4 ) has pixel positions [2 + 4n _A , 4 + 4n _B ] and [3 + 4n _A , 5 + 4n _B ] as shown in FIG. The pixel having only the G signal, the pixel having only the B signal, and the pixel having only the B signal and the pixel position [2 + 4n _A , 5 + 4n _B ], which are arranged at the pixel position [3 + 4n _A , 4 + 4n _B ]. An image including pixels having only signals.

Hereinafter, original images obtained by addition reading using the first, second, third, and fourth addition patterns are referred to as original images of the first, second, third, and fourth addition patterns, respectively. In a certain original image, a pixel having an R, G, or B signal is also referred to as a real pixel, and a pixel in which none of the R, G, and B signals exists is also referred to as a blank pixel. Therefore, for example, in the original image of the first addition pattern, pixels arranged at positions [2 + 4n _A , 2 + 4n _B ], [3 + 4n _A , 3 + 4n _B ], [3 + 4n _A , 2 + 4n _B ] or [2 + 4n _A , 3 + 4n _B ]. Only pixels are actual pixels, and other pixels (for example, pixels arranged at the position [1, 1]) are blank pixels.

  FIG. 10 is a partial block diagram of the imaging apparatus 1 in FIG. 1 including an internal block diagram of the video signal processing unit 13a used as the video signal processing unit 13 in FIG. The video signal processing unit 13a includes portions that are referred to by reference numerals 51 to 56.

  The color interpolation processing unit 51 converts the RAW data into R, G, and B signals by performing color interpolation processing on the RAW data from the AFE 12. This conversion is executed for each frame, and R, G, and B signals obtained by this conversion are temporarily stored in the frame memory 52. The RAW data is image data representing an original image indicated by the output signal of the AFE 12.

  By color interpolation processing in the color interpolation processing unit 51, one color interpolation image is generated from one original image. Each time the frame period elapses, the first, second,..., (N−1) th, and nth original images are sequentially acquired from the image sensor 33 via the AFE 12, and the color interpolation processing unit 51 , First, second,..., (N−1) th, nth original image, first, second,..., (N−1) th, nth color interpolated image, respectively. Is generated. Here, n is an integer of 2 or more.

  The motion detection unit 53 is based on the R, G, and B signals of the current frame currently output from the color interpolation processing unit 51 and the R, G, and B signals of the previous frame stored in the frame memory 52. Thus, an optical flow between adjacent frames is obtained. That is, the optical flow between the two color interpolation images is obtained based on the image data of the (n−1) th and nth color interpolation images. The motion detection unit 53 detects the magnitude and direction of motion between the two-color interpolated images from the optical flow. The detection result of the motion detection unit 53 is stored in the memory 54.

  The image composition unit 55 receives the output signal of the color interpolation processing unit 51 and the signal stored in the frame memory 52, and generates one output composite image based on a plurality of color interpolation images represented by the received signal. To do. At the time of this generation, the detection result of the motion detection unit 53 stored in the memory 54 is also referred to. The signal processing unit 56 converts the R, G, and B signals of the output combined image output from the image combining unit 55 into a video signal composed of the luminance signal Y and the color difference signals U and V. The video signals (Y, U, and V) obtained by this conversion are sent to the compression processing unit 16 and are compressed and encoded according to a predetermined image compression method.

  In the configuration shown in FIG. 10, the color interpolation processing unit 51, the frame memory 52, the motion amount detection unit 53, the image synthesis unit 55, and the signal processing unit 56 are arranged in this order from the AFE 12 toward the compression processing unit 16. However, this order can be changed. Hereinafter, after describing the basic method of color interpolation processing, the functions of the color interpolation processing unit 51, the motion amount detection unit 53, and the image composition unit 55 will be described in detail.

[Basic method of color interpolation]
Focusing on the G signal, a basic method of color interpolation processing will be described with reference to FIGS. 11 (a) and 11 (b). When generating the G signal of the color interpolation image from the G signal of the original image, an interpolation pixel position is determined on the color interpolation image, and a plurality of the original image having a G signal are present in the vicinity of the interpolation pixel position. Real pixels are noticed. Then, the G signal at the interpolated pixel position is generated by mixing the G signals at the plurality of actual pixels of interest. A plurality of actual pixels that are noticed for generating the G signal at the interpolation pixel position are referred to as a reference actual pixel group for convenience.

When the number of real pixels forming the reference real pixel group is 2 and the reference real pixel group is composed of the first and second pixels, the G signal value at the interpolation pixel position is calculated according to the equation (A1). Here, as shown in FIG. 11A, d ₁ and d ₂ are the distance between the pixel position of the first pixel and the interpolation pixel position, and the distance between the pixel position of the second pixel and the interpolation pixel position, respectively. . The distance here is a distance on the image (a distance on the image coordinate plane XY). V _GT obtained by substituting the G signal values of the first and second pixels in the original image into V _G1 and V _G2 of equation (A1) respectively represents the G signal value at the interpolation pixel position. That is, the G signal value at the interpolation pixel position is calculated by linearly interpolating the G signal value of the reference actual pixel group according to the distances d ₁ and d ₂ . The G signal value indicates the value of the G signal (the same applies to the R signal value and the B signal value).

Even when the number of real pixels forming the reference real pixel group is four and the reference real pixel group is composed of the first to fourth pixels, the number of real pixels forming the reference real pixel group is two. The G signal value at the interpolation pixel position is calculated by the same linear interpolation. That is, by mixing the G signal values V _{G1 to} V _G4 of the first to fourth pixels at a ratio according to the distances d _{1 to} d ₄ between the pixel positions of the first to fourth pixels and the interpolation pixel position, A G signal value V _{GT at the} interpolation pixel position is calculated (see FIG. 11B).

  When the color interpolation processing unit 51 generates the G signal at the interpolation pixel position by mixing the G signals of the reference actual pixel group, the G signals of the plurality of actual pixels are mixed at an equal ratio (mixed at the same ratio). ). In other words, the interpolated pixel position is set to the position where the signal is to be interpolated by mixing the G signals of the reference real pixel group at an equal ratio. In order to satisfy this requirement, the interpolation pixel position is set to the barycentric position of the pixel positions of the actual pixels forming the reference actual pixel group. More specifically, the barycentric position of the figure formed by connecting the pixel positions of the respective real pixels forming the reference real pixel group is set as the interpolation pixel position.

Therefore, when the reference real pixel group is composed of the first and second pixels, the barycentric position of the figure (line segment) connecting the pixel position of the first pixel and the pixel position of the second pixel, that is, between both pixel positions. An interpolation pixel position is set at the center position. Then, since d ₁ = d ₂ inevitably, the expression (A1) is transformed into the following expression (A2). That is, the average value of the G signal values of the first and second pixels is calculated as the G signal value at the interpolation pixel position.

In addition, when the reference real pixel group includes the first to fourth pixels, the interpolation pixel position is set at the center of gravity of the quadrangle formed by connecting the pixel positions of the first to fourth pixels, and the G of the interpolation pixel position is set. The signal value V _GT is an average value of the G signal values V _{G1 to} V _G4 of the first to fourth pixels.

  Although the basic method of color interpolation processing has been described focusing on the G signal, the color interpolation processing is performed on the B signal and R signal according to the same method. That is, regardless of whether the target color is green, blue, or red, color interpolation processing is performed on the color signal of the target color according to the basic method described above. When considering the color interpolation processing for the B or R signal, it is sufficient to replace the above-mentioned “G” with “B” or “R”.

[Color interpolation processing unit]
The color interpolation processing unit 51 generates a color interpolation image by performing color interpolation processing on the original image obtained from the AFE 12. In the first embodiment and the second to sixth embodiments described later, the original image given from the AFE 12 to the color interpolation processing unit 51 is the original image of the first, second, third, or fourth addition pattern. Therefore, the pixel interval (interval of adjacent real pixels) in the original image that is the target of the color interpolation process is unequal as shown in FIGS. For such an original image, the color interpolation processing unit 51 performs color interpolation processing according to the basic method described above.

  With reference to FIGS. 12A to 12C and FIGS. 13A to 13C, color interpolation processing for generating the color interpolation image 261 from the original image 251 of the first addition pattern will be described. FIGS. 12A to 12C are diagrams showing how the G, B, and R signals of the real pixels of the original image 251 are mixed in order to generate the G, B, and R signals at the interpolation pixel positions, respectively. It is. FIGS. 13A to 13C are diagrams showing the G, B, and R signals on the color interpolation image 261, respectively. The black circles shown in FIGS. 12A to 12C indicate the interpolation pixel positions where the G, B, and R signals should be generated in the color interpolation image 261, respectively, and around each black circle. The arrows shown indicate how a plurality of color signals are mixed in order to generate a color signal at the interpolation pixel position. In order to prevent complication of illustration, the G, B, and R signals in the color interpolation image 261 are shown separately, but one color interpolation image 261 is generated from the original image 251.

  First, a color interpolation process for generating a G signal in the color interpolation image 261 from a G signal in the original image 251 will be described with reference to FIGS. Note the block 241 that contains the position [x, y] that satisfies the inequalities “2 ≦ x ≦ 7” and “2 ≦ y ≦ 7”. Then, consider the G signal at the interpolation pixel position of the color interpolation image 261 generated from the G signal of the real pixel belonging to the block 241. Note that the G signal (or B signal or R signal) generated for the interpolation pixel position is also referred to as an interpolation G signal (or interpolation B signal or interpolation R signal).

  Interpolated G signals for the two interpolated pixel positions 301 and 302 set in the color interpolated image 261 are generated from the G signals of actual pixels on the original image 251 belonging to the block 241. The interpolated pixel position 301 is a barycentric position [3.5, pixel positions of real pixels P [2,6], P [3,7], P [3,3] and P [6,6] having G signals. 5.5]. The position [3.5, 5.5] corresponds to the center position of the position [3, 6] and the position [4, 5]. The interpolation pixel position 302 is a barycentric position [5.5] of the pixel positions of the real pixels P [6,2], P [7,3], P [3,3] and P [6,6] having the G signal. 3.5]. The position [5.5, 3.5] corresponds to the center position of the position [6, 3] and the position [5, 4].

  In FIG. 13A, the interpolated G signals generated at the interpolated pixel positions 301 and 302 are indicated by reference numerals 311 and 312 respectively. The values of the G signal 311 generated at the interpolation pixel position 301 are the pixels of the actual pixels P [2,6], P [3,7], P [3,3] and P [6,6] in the original image 251. The average value of the values (that is, the G signal value) is used. That is, the G signal 311 is generated by mixing the pixel signals of the reference real pixel group for the G signal 311 at an equal ratio. Similarly, the values of the G signal 312 generated at the interpolation pixel position 302 are the real pixels P [6,2], P [7,3], P [3,3] and P [6,6] in the original image 251. ] Of pixel values (that is, G signal values). That is, the G signal 312 is generated by mixing the pixel signals of the reference real pixel group for the G signal 312 at an equal ratio. The pixel value refers to the value of the pixel signal.

  Further, the G signal of the actual pixel P [x, y] in the original image 251 is directly used as the G signal at the position [x, y] of the color interpolation image 261. That is, for example, the G signals of the actual pixels P [3, 3] and P [6, 6] in the original image 251 (that is, the G signals of the positions [3, 3] and [6, 6] in the original image 251) are These are the G signals 313 and 314 at the positions [3, 3] and [6, 6] of the color interpolation image 261, respectively. The same applies to other positions (for example, position [2, 2]).

When attention is paid to the block 241, two interpolation pixel positions 301 and 302 are set, and interpolation G signals 311 and 312 are generated for them. The block of interest is shifted by 4 pixels in the horizontal and vertical directions starting from the block 241 and the same interpolation G signal generation process is sequentially performed. As a result, a G signal on the color interpolation image 261 as shown in FIG. G1 _2,3 , G1 _3,2 , G1 _2,2 and G1 _3,3 in FIG. 20A respectively correspond to the G signals 311, 312, 313 and 314 in FIG. A detailed description of FIG. 20A will be described later. First, the color interpolation processing for the B and R signals and the color interpolation processing when the second to fourth addition patterns are used will be described.

  A color interpolation process for generating a B signal in the color interpolation image 261 from a B signal in the original image 251 will be described with reference to FIGS. 12B and 13B. Focusing on the block 241, consider the B signal at the interpolation pixel position of the color interpolation image 261 generated from the B signal of the real pixel belonging to the block 241.

  Interpolated B signals for the three interpolated pixel positions 321 to 323 set in the color interpolated image 261 are generated from the B signals of actual pixels belonging to the block 241. The interpolated pixel position 321 matches the barycentric position [3,4] of the pixel positions of the real pixels P [3,2] and P [3,6] having the B signal. The interpolated pixel position 322 matches the barycentric position [5, 6] of the pixel positions of the actual pixels P [3, 6] and P [7, 6] having the B signal. The interpolated pixel position 323 is the barycentric position [5, 4] of the pixel positions of the real pixels P [3, 2], P [7, 2], P [3, 6] and P [7, 6] having the B signal. Matches.

  In FIG. 13B, the interpolated B signals generated at the interpolated pixel positions 321 to 323 are indicated by reference numerals 331 to 333, respectively. The value of the B signal 331 generated at the interpolation pixel position 321 is an average value of the pixel values (that is, the B signal value) of the actual pixels P [3, 2] and P [3, 6] in the original image 251. That is, the B signal 331 is generated by mixing the pixel signals of the reference real pixel group for the B signal 331 at an equal ratio. The same applies to the B signals 332 and 333. That is, the value of the B signal 332 generated at the interpolation pixel position 322 is an average value of the pixel values (that is, the B signal value) of the actual pixels P [3, 6] and P [7, 6] in the original image 251. The values of the B signal 333 generated at the interpolation pixel position 323 are the real pixels P [3, 2], P [7, 2], P [3, 6] and P [7, 6] in the original image 251. The average value of the pixel values (that is, the B signal value) is used.

  Further, the B signal of the actual pixel P [x, y] in the original image 251 is directly used as the B signal at the position [x, y] of the color interpolation image 261. That is, for example, the B signal of the actual pixel P [3, 6] in the original image 251 (that is, the B signal at the position [3, 6] in the original image 251) is the B signal at the position [3, 6] of the color interpolation image 261. The signal 334 is used. The same applies to other positions (for example, position [3, 2]).

  When attention is paid to the block 241, three interpolation pixel positions 321 to 323 are set, and interpolation B signals 331 to 333 are generated for them. The block of interest is shifted by 4 pixels in the horizontal and vertical directions starting from the block 241, and the same interpolation B signal generation processing is sequentially performed. Thereby, a B signal on the color interpolation image 261 as shown in FIG. 20B is generated.

  With reference to FIGS. 12C and 13C, a color interpolation process for generating an R signal in the color interpolation image 261 from an R signal in the original image 251 will be described. Focusing on the block 241, consider the R signal at the interpolation pixel position of the color interpolation image 261 generated from the R signal of the real pixel belonging to the block 241.

  Interpolation B signals for the three interpolation pixel positions 341 to 343 set in the color interpolation image 261 are generated from the R signals of the real pixels belonging to the block 241. The interpolation pixel position 341 coincides with the barycentric position [4, 3] of the pixel positions of the real pixels P [2, 3] and P [6, 3] having the R signal. The interpolated pixel position 342 matches the barycentric position [6, 5] of the pixel positions of the real pixels P [6, 3] and P [6, 7] having the R signal. The interpolated pixel position 343 is the barycentric position [4, 5] of the pixel positions of the real pixels P [2,3], P [2,7], P [6,3] and P [6,7] having the R signal. Matches.

  In FIG. 13C, interpolation R signals generated at the interpolation pixel positions 341 to 343 are indicated by reference numerals 351 to 353, respectively. The value of the R signal 351 generated at the interpolation pixel position 341 is an average value of the pixel values (that is, R signal values) of the actual pixels P [2,3] and P [6,3] in the original image 251. That is, the R signal 351 is generated by mixing the pixel signals of the reference real pixel group for the R signal 351 at an equal ratio. The same applies to the R signals 352 and 353. That is, the value of the R signal 352 generated at the interpolation pixel position 342 is an average value of the pixel values (that is, the R signal value) of the actual pixels P [6, 3] and P [6, 7] in the original image 251. The values of the R signal 353 generated at the interpolation pixel position 343 are the real pixels P [2,3], P [2,7], P [6,3] and P [6,7] in the original image 251. The average value of pixel values (that is, R signal values) is used.

  In addition, the R signal of the actual pixel P [x, y] in the original image 251 is directly used as the R signal at the position [x, y] of the color interpolation image 261. That is, for example, the R signal of the actual pixel P [6, 3] in the original image 251 (that is, the R signal at the position [6, 3] in the original image 251) is the R signal at the position [6, 3] of the color interpolation image 261. The signal 354 is used. The same applies to other positions (for example, position [2, 3]).

  When attention is paid to the block 241, three interpolation pixel positions 341 to 343 are set, and interpolation R signals 351 to 353 are generated for them. The block of interest is shifted by 4 pixels in the horizontal and vertical directions starting from the block 241, and the same interpolation R signal generation processing is sequentially performed. Thereby, an R signal on the color interpolation image 261 as shown in FIG. 20C is generated.

  A color interpolation process for the original images of the second, third, and fourth addition patterns will be described. The original images of the second, third, and fourth addition patterns are referred to by reference numerals 252 253, and 254, respectively, and the color interpolation images generated from the original images 252, 253, and 254 are referred to by reference numerals 262, 263, and 264, respectively. To do.

  14 (a) to 14 (c), the G, B, and R signals of the actual pixels of the original image 252 are mixed to generate the G, B, and R signals of the interpolation pixel position in the color interpolation image 262, respectively. FIG. FIGS. 15A to 15C are diagrams showing the G, B, and R signals on the color interpolation image 262, respectively. In FIGS. 16A to 16C, the G, B, and R signals of the actual pixels of the original image 253 are mixed in order to generate the G, B, and R signals of the interpolation pixel positions in the color interpolation image 263, respectively. FIG. FIGS. 17A to 17C are diagrams showing the G, B, and R signals on the color interpolation image 263, respectively. 18 (a) to 18 (c), the G, B, and R signals of the actual pixels of the original image 254 are mixed to generate the G, B, and R signals of the interpolation pixel position in the color interpolation image 264, respectively. FIG. FIGS. 19A to 19C are diagrams showing the G, B, and R signals on the color interpolation image 264, respectively.

  The black circles shown in FIGS. 14A to 14C indicate the interpolation pixel positions where the G, B, or R signals are to be generated in the color interpolation image 262, respectively, and FIGS. The black circles shown in FIG. 18 indicate the interpolation pixel positions where the G, B, or R signals should be generated in the color interpolation image 263, and the black circles shown in FIGS. Respectively indicate interpolation pixel positions at which G, B, or R signals in the color interpolation image 264 are to be generated. An arrow shown around each black circle indicates a state in which a plurality of color signals are mixed to generate a color signal at the interpolation pixel position. In order to prevent complication of illustration, the G, B, and R signals in the color interpolation image 262 are separately shown, but one color interpolation image 262 is generated from the original image 252. The same applies to the color interpolation images 263 and 264.

  The color interpolation processing method for the original images of the second to fourth addition patterns is the same as that for the first addition pattern. However, with the actual pixel existing position in the original image of the first addition pattern as a reference, the actual pixel existing position in the second addition pattern original image is 2 · Wp in the right direction and 2 · Wp in the downward direction. The actual pixel existing position in the original image of the third addition pattern is shifted by 2 · Wp in the right direction, and the actual pixel existing position in the original image of the fourth addition pattern is downward. It is shifted by 2 · Wp (see also FIG. 4A). Accordingly, with the G, B, and R signal existing positions on the color interpolation image 261 as a reference, the G, B, and R signal existing positions on the color interpolation image 262 are 2 · Wp in the right direction and 2 · in the downward direction. The position of G, B and R signals on the color interpolation image 263 is shifted by 2 · Wp to the right, and the position of the G, B and R signals on the color interpolation image 264 is shifted by Wp. Is shifted downward by 2 · Wp. Therefore, the interpolation pixel position for the color interpolation images 262 to 264 is also shifted with reference to that of the color interpolation image 261 by an amount corresponding to these deviations.

  For example, regarding the original image 252 corresponding to FIG. 14A and the like, attention is paid to a block 242 including a position [x, y] that satisfies the inequalities “4 ≦ x ≦ 9” and “4 ≦ y ≦ 9”. Interpolation G signals for two interpolation pixel positions set in the color interpolation image 261 are generated from the G signals of the real pixels belonging to the block 242. Of the two interpolation pixel positions, one of the interpolation pixel positions is an actual pixel P [4,8], P [5,9], P [5,5] and P [8] having a G signal in the original image 252. , 8] is the barycentric position [5.5, 7.5] of the pixel position, and the other interpolation pixel position is the actual pixel P [8, 4], P [9, 5 having the G signal in the original image 252. ], P [5,5] and P [8,8] are the center-of-gravity positions [7.5, 5.5].

  The interpolated G signal at the interpolated pixel position set at the position [5.5, 7.5] is the actual pixel P [4,8], P [5,9], P [5,5] in the original image 252. ] And P [8,8] are average values, and the interpolation G signal at the interpolation pixel position set at the position [7.5,5.5] is the actual pixel P [8,8 in the original image 252. 4], P [9,5], P [5,5], and P [8,8]. Further, the G signal of the actual pixel P [x, y] in the original image 252 is directly used as the G signal at the position [x, y] of the color interpolation image 262.

  If the block of interest is shifted by 4 pixels in the horizontal and vertical directions starting from the block 242 and the same interpolation G signal generation process is performed in sequence, a color interpolation image as shown in FIG. A G signal on 262 is generated. By generating the interpolation B and R signals in the same manner, the B and R signals on the color interpolation image 262 as shown in FIGS. 21B and 21C are generated.

  FIGS. 20A to 20C are diagrams showing the positions of the G, B, and R signals in the color interpolation image 261, respectively, and FIGS. 21A to 21C are the color interpolation images 262, respectively. It is a figure which shows the existing position of G, B, and R signal of. The color interpolation images 263 and 264 are also generated from the original images 253 and 254 by a method similar to the method of generating the color interpolation image 261 (or 262) from the original image 251 (or 252). Drawings such as FIG. 20A corresponding to FIG. 20 are omitted.

20A to 20C, the G, B, and R signals on the color interpolation image 261 are indicated by circles, and the symbols indicated in the circles are G, B and R signals are represented. 21A to 21C, the G, B, and R signals on the color interpolation image 262 are indicated by circles, and the symbols shown in the circles are G, B and R signals are represented. G1 _{i, j} , B1 _{i, j} and R1 _{i, j} are used as symbols representing the G, B and R signals in the color interpolation image 261, respectively, and as symbols representing the G, B and R signals in the color interpolation image 262, respectively. , G2 _{i, j} , B2 _{i, j} and R2 _{i, j} are used, respectively. i and j are integers. G1 _{i, j} and G2 _{i, j} may be used as symbols representing the value of the G signal (the same applies to B1 _{i, j} , B2 _{i, j} , R1 _{i, j} and R2 _{i, j)} . ).

_{I and j in} the color signals G1 _{i, j} , B1 _{i, j} and R1 _{i, j} of the target pixel of the color interpolation image 261 indicate the horizontal pixel number and the vertical pixel number of the target pixel of the color interpolation image 261, respectively. (The same applies to the color signals G2 _{i, j} , B2 _{i, j} and R2 _{i, j} ).

The arrangement of the color signals G1 _{i, j} , B1 _{i, j} and R1 _{i, j in} the color interpolation image 261 will be described.
As shown in FIG. 20A, a G signal that coincides with the pixel signal at the position [2, 2] of the original image 251 exists at the position [2, 2] in the color interpolation image 261. This position [2 , 2] is taken as the G signal reference position, and the G signal at the G signal reference position is G1 _1,1 .
When the G signal on the color interpolation image 261 is scanned in the right direction from the G signal reference position (position [2, 2]), the G signals G1 _1,1 , G1 _2,1 , G1 _3,1 , G1 _{4 , 1} , ... exist in this order. However, it is assumed that the scanning line has a width Wp during the scanning in the right direction. Therefore, the position should be present G signal G1 _2,1 [3.5,1.5] is rests on the scanning line.
When the G signal on the color-interpolated image 261 is scanned downward from the G signal reference position (position [2, 2]), the G signals G1 _1,1 , G1 _1,2 , G1 _1,3 , G1 _{1 , 4} , ... exist in this order. However, it is assumed that the scanning line has a width Wp during the downward scanning. Accordingly, the position [1.5, 3.5] where the G signal G1 _1,2 should be located is on this scan line.
Similarly, when the G signal on the color interpolation image 261 is scanned from the starting point at an arbitrary position where the G signal G1 _{i, j} on the color interpolation image 261 exists, the G signal G1 _{i , j} , G1 _{i + 1, j} , G1 _{i + 2, j} , G1 _{i + 3, j} ,... exist in this order, and G on the color interpolation image 261 from the starting point downward. When the signal is scanned, the G signals G1 _{i, j} , G1 _{i, j + 1} , G1 _{i, j + 2} , G1 _{i, j + 3} ,... Exist in this order. However, it is assumed that the scanning line has a width Wp during the scanning in the right direction and the downward direction.

As shown in FIG. 20B, a B signal generated from B signals of a plurality of real pixels on the original image 251 exists at the position [1, 2] in the color interpolation image 261. , 2] is taken as the B signal reference position, and the B signal at the B signal reference position is B1 _1,1 .
When the B signal on the color interpolation image 261 is scanned in the right direction from the B signal reference position (position [1, 2]), the B signal B1 _1,1 , B1 _2,1 , B1 _3,1 , B1 ₄ is scanned. _{, 1} ,... Exist in this order, and when the B signal on the color interpolation image 261 is scanned downward from the B signal reference position, the B signal B1 _1,1 , B1 _1,2 , B1 _1,3 , B1 _1,4 , ... exist in this order. Similarly, when the B signal on the color interpolated image 261 is scanned from the starting point at an arbitrary position where the B signal B1 _{i, j} on the color interpolated image 261 exists, the B signal B1 _i is scanned. _{, j} , B1 _{i + 1, j} , B1 _{i + 2, j} , B1 _{i + 3, j} ,... exist in this order, and B on the color interpolation image 261 from the starting point downward. When the signal is scanned, B signals B1 _{i, j} , B1 _{i, j + 1} , B1 _{i, j + 2} , B1 _{i, j + 3} ,... Exist in this order.

As shown in FIG. 20C, an R signal generated from R signals of a plurality of real pixels on the original image 252 exists at the position [2, 1] in the color interpolation image 261. This position [2 , 1] is regarded as the R signal reference position, and the R signal at the R signal reference position is R1 _1,1 .
When the R signal on the color interpolation image 261 is scanned in the right direction from the R signal reference position (position [2, 1]), the R signal R1 _1,1 , R1 _2,1 , R1 _3,1 , R1 ₄ is scanned. _{, 1} ,... Exist in this order, and when the R signal on the color interpolation image 261 is scanned downward from the R signal reference position, the R signals R1 _1,1 , R1 _1,2 , R1 _1,3 , R1 _1,4 ... Exist in this order. Similarly, when an R signal R1 _{i, j} on the color interpolation image 261 starts from an arbitrary position and the R signal on the color interpolation image 261 is scanned from the starting point to the right, the R signal R1 _{i , j} , R1 _{i + 1, j} , R1 _{i + 2, j} , R1 _{i + 3, j} ,... exist in this order, and R on the color interpolation image 261 from the starting point downward. When the signal is scanned, R signals R1 _{i, j} , R1 _{i, j + 1} , R1 _{i, j + 2} , R1 _{i, j + 3} ,... Exist in this order.

The arrangement state of the color signals G1 _{i, j} , B1 _{i, j} and R1 _{i, j in} the color interpolation image 261 has been described, but the color signals G2 _{i, j} , B2 _{i, j} and R2 _{i, j} in the color interpolation image 262 have been described. The same applies to the arrangement state of. In the above description, the original image 251 and the color-interpolated image 261 are replaced with the original image 252 and the color-interpolated image 262, respectively, and “G1”, “B1”, and “R1” are replaced with “G2,” “B2,” and “ When read as R2 ″, the arrangement state of the signals G2 _{i, j} , B2 _{i, j} and R2 _{i, j} is determined. However, as shown in FIGS. 21A to 21C, the G, B, and R signal reference positions in the color interpolation image 262 are positions [4, 4], [3,4], and [4, 3], respectively. Therefore, the G signal at the position [4, 4], the B signal at the position [3, 4] and the R signal at the position [4, 3] are G2 _1,1 , B2 _1,1 and R2 _{1, respectively. 1}

The position where the color signal exists in the color interpolation image 261 is defined more clearly.
As shown in FIG. 20A, the color interpolation image 261 has positions [2 + 4n _A , 2 + 4n _B ], [3 + 4n _A , 3 + 4n _B ], [3.5 + 4n _A , 1.5 + 4n _B ] and [1.5 + 4n _A ]. , 3.5 + 4n _B ], there is a G signal (n _A and n _B are integers).
In the color interpolation image 261,
The G signal at the position [2 + 4n _A , 2 + 4n _B ] is represented by G1 _{i, j} when (i, j) = ((2 + 4n _A ) / 2, (2 + 4n _B ) / 2),
The G signal at the position [3 + 4n _A , 3 + 4n _B ] is represented by G1 _{i, j} when (i, j) = ((4 + 4n _A ) / 2, (4 + 4n _B ) / 2),
The G signal at the position [3.5 + 4n _A , 1.5 + 4n _B ] is represented by G1 _{i, j} when (i, j) = ((4 + 4n _A ) / 2, (2 + 4n _B ) / 2),
The G signal at the position [1.5 + 4n _A , 3.5 + 4n _B ] is represented by G1 _{i, j} when (i, j) = ((2 + 4n _A ) / 2, (4 + 4n _B ) / 2). .
As shown in FIGS. 20B and 20C, the color interpolation image 261 has a B signal at the position [2n _A -1,2n _B ] while the position [2n _A , 2n _B -1]. There are R signals (n _A and n _B are integers). In the color interpolation image 261, the B signal at the position [2n _A -1,2n _B ] is represented by B1 _{i, j} when (i, j) = (n _A , n _B ), and the position [2n The R signal in _A , 2n _B −1] is represented by R1 _{i, j} when (i, j) = (n _A , n _B ).

The position where the color signal exists in the color interpolation image 262 is defined more clearly.
As shown in FIG. 21A, the color interpolation image 262 has positions [4 + 4n _A , 4 + 4n _B ], [5 + 4n _A , 5 + 4n _B ], [5.5 + 4n _A , 3.5 + 4n _B ] and [3.5 + 4n _A ]. , 5.5 + 4n _B ], there is a G signal (n _A and n _B are integers).
And in the color interpolation image 262,
The G signal at the position [4 + 4n _A , 4 + 4n _B ] is represented by G2 _{i, j} when (i, j) = ((2 + 4n _A ) / 2, (2 + 4n _B ) / 2),
The G signal at the position [5 + 4n _A , 5 + 4n _B ] is represented by G2 _{i, j} when (i, j) = ((4 + 4n _A ) / 2, (4 + 4n _B ) / 2),
The G signal at the position [5.5 + 4n _A , 3.5 + 4n _B ] is represented by G2 _{i, j} when (i, j) = ((4 + 4n _A ) / 2, (2 + 4n _B ) / 2),
The G signal at the position [3.5 + 4n _A , 5.5 + 4n _B ] is represented by G2 _{i, j} when (i, j) = ((2 + 4n _A ) / 2, (4 + 4n _B ) / 2). .
Further, as shown in FIGS. 21B and 21C, in the color interpolation image 262, the B signal is present at the position [2n _A -1,2n _B ] while the position [2n _A , 2n _B -1]. There are R signals (n _A and n _B are integers). In the color interpolation image 262, the B signal at the position [2n _A +1, 2n _B +2] is represented by B2 _{i, j} when (i, j) = (n _A , n _B ), and the position [2n The R signal at _A +2, 2n _B +1] is represented by R2 _{i, j} when (i, j) = (n _A , n _B ).

[Motion detector]
The function of the motion detection unit 53 in FIG. 10 will be described. As described above, the motion detection unit 53 obtains the optical flow between the two color interpolation images based on the image data of the (n−1) th and nth color interpolation images. In the first embodiment, since the addition pattern to be used is sequentially changed between a plurality of addition patterns for each frame, the addition patterns corresponding to the (n−1) th and nth color interpolation images are different from each other. For example, one of the (n−1) th and nth color interpolation images is a color interpolation image generated from the original image of the first addition pattern, and the other is the original of the second addition pattern. It is a color interpolation image generated from an image.

  As an example, a method of deriving an optical flow between the color interpolation image 261 shown in FIG. 20A and the color interpolation image 262 shown in FIG. As shown in FIG. 22, the motion detection unit 53 first generates a luminance image 261Y from the R, G, and B signals of the color interpolation image 261, and generates a luminance image 262Y from the R, G, and B signals of the color interpolation image 262. To do. The luminance image is a grayscale image including only luminance signals. Each of the luminance images 261Y and 262Y is formed by arranging pixels having luminance signals at equal intervals in the horizontal and vertical directions. Note that “Y” in FIG. 22 represents a luminance signal.

The luminance signal of the pixel of interest on the luminance image 261Y is derived from the G, R, and B signals on the color interpolation image 261 located at or near the pixel of interest. For example, when the luminance signal at the position [4, 4] of the luminance image 261Y is generated, the position is determined from the G signals G1 _2,2 , G1 _3,3 , G1 _3,2 and G1 _2,3 of the color interpolation image 261. the G signal of [4,4] is calculated by linear interpolation, the B signal of the position [4,4] of B signals B1 _{2, 2} and B1 _3,2 color interpolated image 261 is calculated by linear interpolation, color interpolation The R signal at the position [4, 4] is calculated from the R signals R1, _{2, 2} and R1, _{2, 3 of the} image 261 by linear interpolation (see FIGS. 20A to 20C). Then, the luminance signal at the position [4, 4] in the luminance image 261Y is calculated from the G, B, and R signals at the position [4, 4] calculated based on the color interpolation image 261. The calculated luminance signal is handled as the luminance signal of the pixel existing at position [4, 4] on the luminance image 261Y.

When the luminance signal at the position [4, 4] of the luminance image 262Y is generated, the B signal at the position [4, 4] from the B signal B2 _1,1 and B2 _2,1 of the color interpolation image 262 is linearly interpolated. The R signal at the position [4, 4] is calculated by linear interpolation from the R signals R2 _1,1 and R2 _{1,2 of} the color interpolation image 262 (see FIGS. 21B and 21C). As G signal of the position [4,4], which is directly available G signal G2 _{1, 1} of the color-interpolated image 262 (see FIG. 21 (a)). Then, it calculates the G signal G2 _{1, 1,} was calculated on the basis of the color-interpolated image 262, and a B and R signals of position [4,4], the luminance signal of the position [4,4] of the luminance image 262Y . The calculated luminance signal is handled as the luminance signal of the pixel existing at position [4, 4] on the luminance image 262Y.

  The pixel existing at the position [4, 4] on the luminance image 261Y and the pixel existing at the position [4, 4] on the luminance image 262Y are mutually corresponding pixels. Although the calculation method of the luminance signal at the position [4, 4] has been described, the luminance signal is calculated according to the same method for other positions. Thereby, the luminance signal at an arbitrary pixel position [x, y] on the luminance image 261Y and the luminance signal at an arbitrary pixel position [x, y] on the luminance image 262Y are calculated.

  After generating the luminance images 261Y and 262Y, the motion detection unit 53 compares the luminance signal of the luminance image 261Y and the luminance signal of the luminance image 262Y to obtain an optical flow between the luminance images 261Y-262Y. As a method for deriving the optical flow, a block matching method, a representative point matching method, a gradient method, or the like can be used. The obtained optical flow is expressed by a motion vector representing the motion of the subject (object) on the image between the luminance images 261Y-262Y. The motion vector is a two-dimensional quantity indicating the direction and magnitude of the motion. The motion detection unit 53 treats the optical flow obtained for the luminance images 261Y-262Y as an optical flow between the color interpolation images 261-262, and stores it in the memory 54 as a motion detection result.

  As many motion detection results as possible between adjacent frames are stored in the memory 54. For example, a motion detection result between the (n-3) th and (n-2) th color interpolation images and a motion detection result between the (n-2) th and (n-1) th color interpolation images. And the motion detection result between the (n-1) -th and n-th color-interpolated images are stored in the memory 54, and read out from the memory 54 and synthesized, the (n-3) -th An optical flow (motion vector) between any two color interpolation images in the nth color interpolation image can be obtained.

  The “optical flow (or motion vector) between the luminance images 261Y-262Y” means “optical flow (or motion vector) between the luminance image 261Y and the luminance image 262Y”. A similar description method is also used when describing optical flows, motion vectors or motions, or matters related thereto for a plurality of images other than the luminance images 261Y and 262Y. Therefore, for example, “optical flow between the color interpolation images 261 to 262” refers to “optical flow between the color interpolation image 261 and the color interpolation image 262”.

[Image composition part]
The function of the image composition unit 55 in FIG. 10 will be described. The image synthesis unit 55 stores the color signal of the color interpolation image output from the color interpolation processing unit 51, the color signal of one or more other color interpolation images stored in the frame memory 52, and the memory 54. An output composite image is generated based on the motion detection result.

  The output combined image refers to a plurality of color interpolated images having different corresponding addition patterns, regards one of the plurality of referred color interpolated images as a combined reference image, and then selects the plurality of color interpolated images. Generated by synthesis. At this time, if the addition pattern corresponding to the color interpolation image used as the composite reference image changes with time, even if the subject is stationary in the real space, the subject seems to move in the output composite image sequence. It looks like. In order to avoid this, when generating a series of output composite image sequences, image data read from the frame memory 52 is controlled so that the addition pattern corresponding to the color interpolation image used as the composite reference image is always the same. Note that a color interpolation image that is not used as a synthesis reference image is referred to as a non-synthesis reference image.

  Now, the original images of the first and second addition patterns are alternately photographed, the composite reference image is a color interpolation image generated from the original image of the first addition pattern, and the non-composite reference image is the second image. It is assumed that the color interpolation image is generated from the original image of the addition pattern. Therefore, the (n-3) th, (n-2), (n-1), and nth original images are the original images of the first, second, first, and second addition patterns, respectively. If so, the color-interpolated image based on the (n-3) th and (n-1) th original images becomes the synthesis reference image, and the color based on the (n-2) th and nth original images. The interpolated image becomes a non-synthesized reference image. In the first embodiment, it is assumed that there is no movement of the subject on the image between two color-interpolated images obtained adjacent in time.

  Under this assumption, color interpolation shown in FIG. 20A and the like with reference to FIGS. 23A to 23C, FIGS. 24A and 24B, and FIGS. 25A to 25C. A process of generating one output composite image 270 from the image 261 and the color interpolation image 262 shown in FIG. 23A to 23C are diagrams showing the G, B, and R signals on the color interpolation images 261 and 262 for generating the G, B, and R signals on the output composite image 270. 25 (a) to 25 (c) are diagrams illustrating the positions where the G, B, and R signals of the output composite image 270 exist. FIGS. 24A and 24B are other diagrams showing the B and R signals on the color interpolation images 261 and 262 for generating the B and R signals on the output composite image 270. FIG.

As shown in FIGS. 25A to 25C, the output composite image 270 is a two-dimensional image in which pixels (pixel positions) are arranged at equal intervals in the horizontal and vertical directions, and each pixel of the output composite image 270. There are G, B, and R signals at each pixel position where. That is, unlike the original image and the color interpolation image, one G, B, and R signal is assigned to one pixel position where one pixel is arranged in the output composite image 270. As shown in FIGS. 25A to 25C, a pixel on the output composite image 270 is positioned at a position [2i−0.5, 2j−0.5] on the image coordinate plane XY (see FIG. 4B). Are arranged (i and j are integers). Then, the G signal, the B signal, and the R signal of the output composite image 270 at the position [2i-0.5, 2j-0.5] are converted into Go _{i, j} , Bo _{i, j} and Ro _{i, j} , respectively. Represent. Note that Go _{i, j} may be used as a symbol representing the value of the G signal (the same applies to Bo _{i, j} and Ro _{i, j} ).

_{I and j in} the color signals Go _{i, j} , Bo _{i, j} and Ro _{i, j} of the target pixel of the output composite image indicate the horizontal pixel number and the vertical pixel number of the target pixel of the output composite image.

  As described with reference to FIG. 20A, FIG. 21A, etc., the corresponding addition pattern is different between the color interpolation image 261 and the color interpolation image 262. The existence position is different, the existence position of the B signal is different, and the existence position of the R signal is different. Based on these differences, the image composition unit 55 mixes the G, B, and R signals of the color interpolation image 261 and the G, B, and R signals of the color interpolation image 262 to mix the G, B, and R signals of the output composite image 270. An R signal is generated.

Specifically, according to the following formulas (B1) to (B3), output synthesis is performed by weighted addition of the G, B, and R signal values of the color interpolation image 261 and the G, B, and R signal values of the color interpolation image 262. The G, B and R signal values Go _{i, j} , Bo _{i, j} and Ro _{i, j} of the image 270 are calculated.

Instead of equations (B2) and (B3), using equations (B4) and (B5) corresponding to FIGS. 24 (a) and (b), B and R signal values Bo _{i, j} and Ro _{i, j} May be calculated.

As a specific example, in FIGS. 23A to 23C and FIGS. 24A and 24B, the color signals Go _3,3 , Bo _3,3 and Ro existing at the position [5.5, 5.5] are shown. _{3 and 3} are generated. In FIGS. 23A to 23C and FIGS. 24A and 24B, asterisks are shown at positions where the color signals Go _3,3 , Bo _3,3 and Ro _3,3 should be present.
G signal Go _3,3, as shown in FIG. 23 (a), and a G signal G2 _{2, 2} at the position [5,5] and the G signal G1 _3,3 at the position [6,6] Produced by mixing.
B signal Bo _3,3, as shown in FIG. 23 (b), and a B signal B2 _{3, 1} at the position [7,4] and the B signal B1 _3,3 at the position [5,6] As shown in FIG. 24A, the B signal B1 _4,2 existing at the position [7,4] and the B signal B2 _2,2 existing at the position [5,6] are generated. And are mixed together.
As shown in FIG. 23 (c), the R signal Ro _3,3 is obtained by combining the R signal R1 _3,3 existing at the position [6,5] and the R signal R2 _1,3 existing at the position [4,7]. The R signal R1 _2,4 generated at the position [4,7] and the R signal R2 _2,2 present at the position [6,5] as shown in FIG. And are mixed together.

Mixing ratio when calculating Go _3,3 by mixing G1 _3,3 and G2 _2,2
Mixing ratio when calculating Bo _3,3 by mixing B1 _3,3 and B2 _3,1
Mixing ratio when calculating Bo _3,3 by mixing B1 _4,2 and B2 _2,2
Mixing ratio when calculating Ro _3,3 by mixing R1 _3,3 and R2 _1,3 , and
As for the mixing ratio when calculating Ro _3,3 by mixing R1 _2,4 and R2 _2,2 , V _GT is calculated by mixing V _G1 and V _G2 described with reference to the above formula (A1). This is the same as the mixing ratio at that time (see also FIG. 11A).

For example, when the signal Bo _3,3 existing at the position [5.5, 5.5] is generated by mixing B1 _3,3 and B2 _3,1, the position [5,5 at which the signal B1 _3,3 exists is generated. 6] and position [a distance d ₁ between 5.5, 5.5], the distance between the position where the signal B2 _{3, 1} is present [7,4] and position [5.5, 5.5] d ₂ Since d ₁ : d ₂ = 1: 3, B1 _3,3 and B2 _3,1 are mixed at a ratio of 3: 1 as shown in the formula (B2). That is, the signal values at the positions [5.5, 5.5] are obtained by linear interpolation based on the signal values at the positions [5, 6] and [7, 4].

Similar to the calculation method of the color signals Go _3,3 , Bo _3,3 and Ro _3,3 , the color signals Go _{i, j} , Bo _{i, j} and Ro _{i, j at} other positions are also obtained. G, B, and R signals at each pixel position of the output composite image 270 as shown in 25 (a) to (c) are obtained.

  The effect based on the above-described output composite image generation method will be considered. If the original image is obtained by the all-pixel readout method, as described with reference to FIG. 6A and the like, when obtaining the pixel signal of the target pixel by interpolation, The pixel signals of the peripheral pixels may be mixed at an equal ratio (at the same ratio), and the interpolated pixel signal is generated at the position where the pixel signal should originally exist by the mixing. Here, the “position where the pixel signal should originally exist” refers to a position [i, j] where i and j are integers.

  However, in the first embodiment, the original image given from the AFE 12 to the color interpolation processing unit 51 is an original image based on the first, second, third, or fourth addition pattern. In this case, if the same ratio mixing is performed as in the case of using the all-pixel readout method, the pixel signal is located at a position different from the position where the pixel signal should originally exist (for example, the interpolation pixel position 301 or 302 in FIG. 12A). Interpolation pixel signals are generated, and the intervals of pixels in which G signals are present on the color interpolation image generated by mixing are not uniform (see FIG. 20A). In addition, on one color interpolation image, the existence positions of the color signals are different among the G, B, and R signals (see FIGS. 20A to 20C).

  In the conventional method corresponding to FIG. 54, in order to avoid such non-uniformity, the interpolation process (see blocks 902 and 903 in FIG. 54) is once performed so that the pixel intervals are equalized, and then the demosaicing process is performed. Running. When the interpolation process for equalizing the pixel intervals is executed, the sense of resolution inevitably deteriorates (substantial resolution deteriorates). On the other hand, in the first embodiment according to the present invention, this non-uniformity is positively used, and an output composite image is generated using a plurality of color-interpolated images where the positions of the color signals are non-uniform. To do. In the color interpolation image, although the position of the color signal is not uniform, the pixel interval in the output composite image is uniform, so that jaggies and false colors are suppressed as in the conventional output image (see block 905 in FIG. 54). Is done. In addition, in the first embodiment according to the present invention, since the interpolation process for equalizing the pixel interval (see blocks 902 and 903 in FIG. 54) is not performed, deterioration in resolution is suppressed. That is, the resolution can be improved as compared with the conventional method corresponding to FIG.

  The method for generating an output composite image by combining two color-interpolated images based on the original images of the first and second addition patterns has been described above. The color for generating one output composite image The number of interpolated images may be 3 or more (this applies to other embodiments described later). For example, one output composite image may be generated from four color interpolation images based on the original images of the first to fourth addition patterns. However, the corresponding addition pattern is different between a plurality of color interpolation images for generating one output composite image (this applies to other embodiments described later).

<< Second Example >>
Next, a second embodiment will be described. In the first embodiment, it is assumed that there is no movement of the subject on the image between two color-interpolated images obtained adjacent in time. In the second embodiment, the movement is taken into consideration. The configuration and operation of the image composition unit 55 will be described. FIG. 26 is a partial block diagram of the imaging apparatus 1 of FIG. 1 according to the second embodiment. FIG. 26 is an internal block diagram of the video signal processing unit 13a used as the video signal processing unit 13 of FIG. And an internal block diagram of the image composition unit 55 is shown.

  The image composition unit 55 in FIG. 26 includes a weight coefficient calculation unit 61 and a composition processing unit 62. Since the configuration and operation in the video signal processing unit 13a excluding the weighting factor calculation unit 61 and the synthesis processing unit 62 are the same as those described in the first embodiment, hereinafter, the weighting factor calculation unit 61 and the synthesis processing unit 62 will be described. Will be described. The matters described in the first embodiment are applied to the second embodiment as long as there is no contradiction.

  As in the assumption in the first embodiment, the original images of the first and second addition patterns are alternately photographed, and the composite reference image is a color interpolation image generated from the original image of the first addition pattern. Further, it is assumed that the non-synthesis reference image is a color interpolation image generated from the original image of the second addition pattern. However, in the second embodiment, the position of the subject on the image can move between two color-interpolated images obtained adjacent in time. Under this assumption, a process of generating one output composite image 270 from the color interpolation image 261 shown in FIG. 20A and the color interpolation image 262 shown in FIG.

The weighting factor calculation unit 61 reads the motion vector obtained for the color interpolation images 261 to 262 from the memory 54, and calculates the weighting factor w based on the magnitude | M | of the motion vector. At this time, the weighting factor w is calculated so that the weighting factor w decreases as the magnitude | M | increases. However, the upper limit value and the lower limit value of the weight coefficient w (and w _{i, j} described later) are 0.5 and 0, respectively.

  FIG. 27 is a diagram illustrating a relationship example between the weighting coefficient w and the magnitude | M |. When this relationship example is employed, the weighting coefficient w is calculated according to the equation “w = −K · | M | +0.5”. However, w = 0 within the range of | M |> 0.5 / K. K is a slope in a relational expression between | M | and w having a predetermined positive value.

The optical flow obtained between the color interpolation images 261 to 262 by the motion detection unit 53 is formed by a bundle of motion vectors at various positions on the image coordinate plane XY. For example, the entire image area of each of the color interpolation images 261 and 262 is divided into a plurality of partial image areas, and one motion vector is obtained for one partial image area. Now, as shown in FIG. 28 (a), the whole image area of the image 260 is a color-interpolated image 261 or 262 is divided into nine partial image regions AR ₁ to Ar _9, some image areas AR ₁ to Ar ₉ Assume that one motion vector is obtained for each of the above. Of course, the number of partial image areas can be other than nine. As shown in FIG. 28 (b), were determined for partial image regions AR ₁ to Ar _9, the motion vector between the color interpolated images 261-262, respectively represented by the numeral M ₁ ~M _9. The magnitude of the motion vectors M ₁ ~M _9, respectively, | M ₁ | ~ | represented by | M _9.

The weighting factor calculation unit 61 calculates weighting factors w at various positions on the image coordinate plane XY based on the magnitudes | M ₁ | to | M ₉ | of the motion vectors M _{1 to} M ₉ . The weighting coefficient w when the horizontal pixel number and the vertical pixel number are i and j, respectively, is represented by w _{i, j} . The weight coefficient w _{i, j} is a weight coefficient for a pixel (pixel position) having the color signals Go _{i, j} , Bo _{i, j} and Ro _{i, j} , and is based on a motion vector for a partial image region to which the pixel belongs. Calculated. Therefore, for example, if the pixel position [1.5, 1.5] where the G signal Go _1,1 exists belongs to a part of the image area AR ₁ , the expression “w _1,1 is based on the size | M ₁ | _{. 1} = −K · | M ₁ | +0.5 ”, the weighting factor w _1,1 is calculated (where w _1,1 = 0 within the range of | M ₁ |> 0.5 / K), G If the pixel position [1.5, 1.5] where the signal Go _1,1 exists belongs to the partial image area AR ₂ , the expression “w _1,1 = −K based on the size | M ₂ |. The weighting factor w _1,1 is calculated according to | M ₂ | +0.5 ”(where w _1,1 = 0 within the range of | M ₂ |> 0.5 / K).

The composition processing unit 62 outputs the G, B, and R signals of the color interpolation image for the current frame currently output from the color interpolation processing unit 51 and the color interpolation image for the previous frame stored in the frame memory 52. The G, B, and R signals are mixed at a ratio corresponding to the weighting factor w _{i, j} calculated by the weighting factor calculation unit 61, thereby generating an output composite image 270 for the current frame.

As shown in FIG. 29 (a), the color interpolation image for the current frame is the color interpolation image 261 corresponding to FIG. 20 (a) and the color interpolation image for the previous frame corresponds to FIG. 21 (a) and the like. In the case of the color interpolation image 262 to be processed, the composition processing unit 62 performs the G, B, and R signal values of the color interpolation image 261 and the G, B, and R signals of the color interpolation image 262 according to the following equations (C1) to (C3). The G, B and R signal values Go _{i, j} , Bo _{i, j} and Ro _{i, j} of the output composite image 270 are calculated by weighted addition of the values. B and R signal values Bo _{i, j} and Ro _{i, j} may be calculated using equations (C4) and (C5) instead of equations (C2) and (C3).

On the other hand, as shown in FIG. 29B, the color interpolation image for the current frame is the color interpolation image 262 corresponding to FIG. 21A and the color interpolation image for the previous frame is FIG. In the case of the color interpolation image 261 corresponding to the color interpolation image 261, the composition processing unit 62 performs G, B, and R signal values of the color interpolation image 261 and G, B, and By weighting and adding the R signal values, the G and B and R signal values Go _{i, j} , Bo _{i, j} and Ro _{i, j} of the output composite image 270 are calculated. B and R signal values Bo _{i, j} and Ro _{i, j} may be calculated using equations (D4) and (D5) instead of equations (D2) and (D3).

  When an output composite image is generated by combining the color interpolation image for the current frame and the color interpolation image for the previous frame, if the movement of the subject between the two color interpolation images is relatively large, the contour portion in the output composite image The image may be blurred or a double image may appear. Therefore, as described above, if the magnitude of the motion vector between the two-color interpolated images is relatively large, the contribution ratio of the previous frame to the output composite image is reduced. As a result, blurring of the outline portion and double image generation in the output composite image are suppressed.

In the above example, the weighting factors w _{i, j} at various positions on the image coordinate plane XY are set. However, the number of weighting factors set when two color interpolation images are combined. , And one weight coefficient may be commonly used for the entire image area. For example, by averaging the motion vectors M _{1 to} M ₉ , an average motion vector M _AVE representing the average motion of the subject between the color interpolation images 261 to 262 is obtained, and the magnitude of the average motion vector M _AVE | Using M _AVE |, one weighting factor w is calculated according to the equation “w = −K · | M _AVE | +0.5” (provided that w = −K · in the range of | M _AVE |> 0.5 / K). 0). Then, according to each formula obtained by substituting the weighting factor w calculated using | M _AVE | into the weighting factors w _{i, j} of the above formulas (C1) to (C5) and formulas (D1) to (D5), The signal values Go _{i, j} , Bo _{i, j} and Ro _{i, j} may be obtained.

<< Third Example >>
Next, a third embodiment will be described. In the third embodiment, image contrast is taken into consideration in addition to subject movement between different color-interpolated images. FIG. 30 is a partial block diagram of the image pickup apparatus 1 of FIG. 1 according to the third embodiment. FIG. 30 shows an internal block diagram of the video signal processing unit 13b used as the video signal processing unit 13 of FIG.

  The video signal processing unit 13b includes parts referred to by reference numerals 51 to 54, 55b and 56, and among these parts, reference parts 51 to 54 and 56 are the same as those shown in FIG. is there. 30 includes a contrast calculation unit 70, a weight coefficient calculation unit 71, and a synthesis processing unit 72. The configuration and operation in the video signal processing unit 13b excluding the image synthesis unit 55b are the same as those in the video signal processing unit 13a described in the first or second embodiment. The configuration and operation will be described. The matters described in the first and second embodiments also apply to the third embodiment as long as there is no contradiction.

  As in the assumption in the first or second embodiment, the color interpolation image in which the original images of the first and second addition patterns are alternately photographed and the synthesized reference image is generated from the original image of the first addition pattern. Further, it is assumed that the non-synthesis reference image is a color interpolation image generated from the original image of the second addition pattern. However, in the third embodiment, as in the second embodiment, the position of the subject on the image can move between two color-interpolated images obtained adjacent in time. Under this assumption, a process of generating one output composite image 270 from the color interpolation image 261 shown in FIG. 20A and the color interpolation image 262 shown in FIG.

The contrast calculation unit 70 outputs the G, B, and R signals of the color interpolation image for the current frame currently output from the color interpolation processing unit 51 and the color interpolation image for the previous frame stored in the frame memory 52. G, B, and R signals are received as input signals, and the contrast amounts in various image regions of the current frame or the previous frame are calculated based on the input signals. Now, as shown in FIG. 28A, the entire image area of the image 260 which is the color interpolation image 261 or 262 is divided into _nine partial image areas AR _{1 to} AR ₉ , and the partial image areas AR _{1 to} AR _9. It is assumed that the contrast amount in each of the above is calculated. Of course, the number of partial image areas may be other than nine. The contrast amounts obtained for the partial image areas AR _{1 to} AR ₉ are represented by C _{1 to} C _9, respectively.

The contrast amount C _m used for the synthesis of the color interpolation image 261 shown in FIG. 20A and the color interpolation image 262 shown in FIG. 21A and the like is calculated as follows (m is Integer satisfying 1 ≦ m ≦ 9).
For example, paying attention to the luminance image 261Y or 262Y (see FIG. 22) generated from the color signal of the color interpolation image 261 or 262, the difference between the minimum luminance value and the maximum luminance value in the partial image area AR _m of the luminance image 261Y. Alternatively, the difference between the minimum luminance value and the maximum luminance value in the partial image area AR _m of the luminance image 262Y is obtained, and the obtained difference is handled as the contrast amount C _m . Alternatively, the average value of these differences is calculated as the contrast amount C _m .
Further, for example, the contrast amount C _m may be obtained by extracting a predetermined high frequency component in the partial image area AR _m of the luminance image 261Y or 262Y using a high pass filter. More specifically, for example, a high-pass filter is formed by a Laplacian filter having a predetermined filter size, and spatial filtering is performed so that the Laplacian filter is applied to each pixel in the partial image area AR _m of the luminance image 261Y or 262Y. Then, output values corresponding to the filter characteristics of the Laplacian filter are sequentially obtained from the high pass filter. The absolute value of the output value of this high pass filter (the magnitude of the high frequency component extracted by the high pass filter) may be integrated to obtain the integrated value as the contrast amount C _m . An average value of the integrated value calculated for the partial image area AR _m of the luminance image 261Y and the integrated value calculated for the partial image area AR _m of the luminance image 262Y may be handled as the contrast amount C _m. Is possible.
The contrast amount C _m obtained as described above takes a larger value as the contrast of the image in the corresponding image region is larger, and takes a smaller value as it is smaller.

The contrast calculation unit 70 calculates a reference motion value M _{O related} to the calculation of the weighting coefficient for each partial image region based on the contrast amounts C _{1 to} C ₉ . In particular, the reference motion value M _O calculated for the partial image area AR _m is represented by M _Om . In reference motion value M _Om, as shown in FIG. 31 (a), along with if the contrast amount C _m is zero is set to the minimum motion value M _OMIN, the contrast amount C _m is greater than or equal to a predetermined contrast threshold C _TH If there is, the maximum motion value M _OMAX is set. Within the range of 0 <C _m <C _TH , the reference motion value M _Om increases from the minimum motion value M _OMIN toward the maximum motion value M _OMAX as the contrast amount C _m increases from zero toward the contrast threshold value C _TH. To increase. More specifically, for example, within the range of 0 <C _m <C _TH , the reference motion value M _Om is calculated according to the formula “M _Om = Θ · C _m + M _OMIN ”. Here, C _TH > 0, 0 <M _OMIN <M _OMAX , and Θ = (M _OMAX −M _OMIN ) / C _TH . In the graph of FIG. 31A, the reference motion value M _O is a vertical axis as a representative of the reference motion value.

The weighting factor calculation unit 71 generates an image based on the reference motion values M _{O1 to} M _O9 calculated by the contrast calculation unit 70 and the magnitudes | M ₁ | to | M ₉ | of the motion vectors M _{1 to} M _9. Weight coefficients w _{i, j} at various positions on the coordinate plane XY are calculated. The significance of the magnitudes | M ₁ | to | M ₉ | of the motion vectors M _{1 to} M ₉ is as described in the second embodiment. The weight coefficient w _{i, j} is a weight coefficient for a pixel (pixel position) having the color signals Go _{i, j} , Bo _{i, j} and Ro _{i, j,} and a reference motion value for a partial image region to which the pixel belongs. And the motion vector. As described in the second embodiment _, the upper limit value and the lower limit value of the weight coefficient w _{i, j} are 0.5 and 0, respectively. The weighting coefficient w _{i, j} is set based on the reference motion value and the magnitude of the motion vector within the range of the upper and lower limit values. FIG. _31B shows a relationship example between the weighting coefficient, the reference motion value M _Om, and the motion vector magnitude | M _m |.

Specifically, for example, if the pixel position [1.5, 1.5] where the G signal Go _1,1 exists belongs to the partial image area AR ₁ , the reference motion amount M _O1 based on the contrast amount C ₁ is used. And the magnitude | M ₁ | of the motion vector M _1, the weight coefficient w _1,1 is calculated according to the expression “w _1,1 = −K · (| M ₁ | −M _O1 ) +0.5”. . However, w _1,1 = 0.5 within the range of | M ₁ | <M _O1 and w _1,1 = 0 within the range of (| M ₁ | −M _O1 )> 0.5 / K. It is said. For example, if the pixel position [1.5, 1.5] where the G signal Go _1,1 exists belongs to a part of the image area AR ₂ , the reference motion amount M _O2 and the motion vector based on the contrast amount C ₂ are used. the size of the M ₂ | based on the formula _{| M 2 "w 1,1 = -K} · (| M 2 | -M O2) +0.5" weighting factor w _{1, 1} according to is calculated. However, w _1,1 = 0.5 within the range of | M ₂ | <M _O2 and w _1,1 = 0 within the range of (| M ₂ | −M _O2 )> 0.5 / K. It is said.

The composition processing unit 72 outputs the G, B, and R signals of the color interpolation image for the current frame currently output from the color interpolation processing unit 51 and the color interpolation image for the previous frame stored in the frame memory 52. The G, B, and R signals are mixed at a ratio according to the weighting factor w _{i, j} set by the weighting factor calculation unit 71, thereby generating an output composite image 270 for the current frame. The calculation method of the G, B, and R signal values of the output combined image 270 by the combining processing unit 72 is the same as that by the combining processing unit 62 described in the second embodiment.

  An image region having a relatively large contrast amount is an image region containing a lot of edge components, and jaggies are conspicuous, so that the effect of reducing jaggy by image composition is great. On the other hand, an image region having a relatively small contrast amount is considered to be a flat image region, and jaggies are hardly noticeable (that is, there is little significance in performing image composition). Therefore, when generating an output composite image by combining the color-interpolated image for the current frame and the color-interpolated image for the previous frame, a relatively large weighting factor is set for an image region with a relatively large contrast amount. Thus, the contribution ratio of the previous frame to the output composite image is increased, and for an image region having a relatively small contrast amount, the weight coefficient is set to be relatively small to reduce the contribution ratio of the previous frame to the output composite image. As a result, an appropriate jaggy reduction effect can be obtained only for an image portion that requires jaggy reduction.

<< 4th Example >>
Next, a fourth embodiment will be described. In the fourth embodiment, a specific image compression method that can be adopted by the compression processing unit 16 (see FIG. 1 and the like) will be described. In the fourth embodiment, it is assumed that the compression processing unit 16 employs an MPEG (Moving Picture Experts Group) compression method, which is a typical compression method for a video signal, and compresses the video signal. In MPEG, an MPEG moving image, which is a compressed moving image, is generated using a difference between frames. FIG. 32 schematically shows the structure of this MPEG moving image. An MPEG moving picture is composed of three types of pictures, that is, an I picture, a P picture, and a B picture.

  An I picture is an intra-coded picture (Intra-Coded Picture), which is an image obtained by coding a video signal of one frame in the frame picture. It is possible to decode a video signal of one frame with an I picture alone.

  The P picture is an inter-frame predictive coded image (Predictive-Coded Picture), and is an image predicted from a temporally preceding I picture or P picture. A P picture is formed by data obtained by compressing and encoding a difference between an original image that is a target of a P picture and a temporally preceding I picture or P picture as viewed from the P picture. The B picture is a bi-directionally predictive-coded picture (Bidirectionally Predictive-Coded Picture), and is an image that is bi-directionally predicted from the later and previous I pictures or P pictures. The difference between the original picture that is the target of the B picture and the I picture or P picture that is temporally later as seen from the B picture, and the difference between the I picture or P picture that is temporally seen from the B picture A B picture is formed by the data obtained by compression encoding the.

  MPEG moving images are configured in units of GOP (Group Of Pictures). A GOP is a unit in which compression and expansion are performed, and one GOP is composed of pictures from a certain I picture to the next I picture. An MPEG video is composed of one or more GOPs. The number of pictures from one I picture to the next I picture may be fixed, but may be varied within a certain range.

  When an image compression method using inter-frame differences, represented by MPEG, is used, I picture provides difference data to both B and P pictures, so the picture quality of I picture is large compared to the overall picture quality of MPEG moving pictures. Influence. In consideration of this, the video signal processing unit 13 or the compression processing unit 16 selects an image number for which the weighting coefficient set in the image composition unit is relatively large and as a result it is determined that jaggies are effectively reduced. When the image is compressed, the output composite image corresponding to the recorded image number is preferentially used as an I picture target. Thereby, the overall image quality of the MPEG moving image obtained by the compression can be improved.

  A more specific example will be described with reference to FIG. As the video signal processing unit 13 according to the fourth embodiment, the video signal processing unit 13a or 13b shown in FIG. 26 or FIG. 30 is used. Now, by the color interpolation processing unit 51, the nth, (n + 1) th, (n + 2) th, nth, (n + 1) th, (n + 2), (n + 3), (n + 4),... ), (N + 3) th, (n + 4)... Color interpolation images 350, 351, 352, 353, 354... Are generated, and the color interpolation images 350 and 351 are generated in the image composition unit 55 or 55b. Output composite image 361, color interpolation images 351 and 352 generate output composite image 362, color interpolation images 352 and 353 generate output composite image 363, color interpolation images 353 and 354 generate output composite image 364,. think of. For example, the n-th, (n + 1) -th, (n + 2) -th, (n + 3) -th, and (n + 4) -th original images have the first, second, first, second, and first addition patterns, respectively. This is the original image. The output composite images 361 to 364 form an output composite image sequence arranged in time series in the order of the output composite images 361, 362, 363, and 364.

The method for generating one output composite image from the two color interpolation images of interest is the same as the method described in the second or third embodiment, and is calculated for the two color interpolation images of interest. One output composite image is generated by color signal mixing according to the weighting factors w _{i, j} . The weighting coefficient w _{i, j} used when generating the one output composite image can take various values depending on the horizontal pixel number i and the vertical pixel number j. The average value of the coefficients w _{i, j} is calculated as the total weight coefficient. The total weight coefficient is calculated by, for example, the weight coefficient calculation unit 61 or 71 (see FIG. 26 or FIG. 30). The total weight coefficients calculated for the output composite images 361 to 364 are represented by w _{T1 to} w _T4, respectively. Although it has been described in the second embodiment that the number of weighting factors set for the two color interpolation images of interest can be one, the two color interpolation images of interest have been described. When the number of weighting factors set for one is one, the one weighting factor may function as an overall weighting factor.

Reference numerals 361 to 364 indicating the output composite images 361 to 364 represent image numbers of the corresponding output composite images. The image numbers 361 to 364 of the output composite image and the total weight coefficients w _{T1 to} w _T4 are associated with each other and recorded in the video signal processing unit 13a or 13b so that the compression processing unit 16 can refer to them (FIG. 26). Or see FIG.

An output composite image corresponding to a relatively large total weight coefficient is estimated to be an image in which the degree of color signal mixing is relatively large and jaggy is relatively greatly reduced. Therefore, the compression processing unit 16 preferentially uses an output composite image corresponding to a relatively large total weight coefficient as an I picture target. Therefore, when one output composite image is selected from the output composite images 361 to 364 as the target of the I picture, the output composite image corresponding to the maximum value among the total weight coefficients w _{T1 to} w _T4 is selected as the I picture. Select as target. For example, if the total weight coefficient w _T2 is the maximum among the total weight coefficients w _{T1 to} w _T4 , the output composite image 362 is selected as the target of the I picture, and the output composite image 362 and the output composite images 361, 363 and P and B pictures are generated based on 364. The same applies when an I picture target is selected from a plurality of output composite images obtained after the output composite image 364.

  The compression processing unit 16 generates an I picture by encoding the output composite image selected as the target of the I picture according to the MPEG compression method, and also generates the I composite of the output composite image selected as the target of the I picture and the I picture. P and B pictures are generated based on the output composite image not selected as a target.

<< 5th Example >>
Next, a fifth embodiment will be described. In the first to fourth embodiments, the addition patterns P _{A1 to} P _A4 corresponding to FIGS. 7A, 7B, 8A, and 8B are used as the first to first patterns for acquiring the original image. Although it is assumed to be used as the fourth addition pattern, an addition pattern different from the addition patterns P _{A1 to} P _A4 can be used as the addition pattern for acquiring the original image. Available addition patterns include addition patterns P _{B1 to} P _B4 , addition patterns P _{C1 to} P _C4, and addition patterns P _{D1 to} P _D4 .

When the addition patterns P _{B1 to} P _B4 are used, the addition patterns P _{B1 to} P _B4 function as the first, second, third, and fourth addition patterns in the first to fourth embodiments, respectively.
When the addition patterns P _{C1 to} P _C4 are used, the addition patterns P _{C1 to} P _C4 function as the first, second, third, and fourth addition patterns in the first to fourth embodiments, respectively.
When the addition patterns P _{D1 to} P _D4 are used, the addition patterns P _{D1 to} P _D4 function as the first, second, third, and fourth addition patterns in the first to fourth embodiments, respectively.
Then, as described in the first to fourth embodiments, two or more addition patterns are selected from the first to fourth addition patterns, and addition reading is performed between the selected two or more addition patterns. The original image sequence is acquired while sequentially changing the addition pattern to be used. For example, when the addition patterns P _{B1 to} P _B4 are caused to function as the first, second, third and fourth addition patterns, the addition reading using the addition pattern P _B1 and the addition reading using the addition pattern P _B2 are alternately performed. , The original images of the addition patterns P _B1 , P _B2 , P _B1 , P _B2,.

34 (a) to (d) show how signals are added when the addition patterns P _{B1 to} P _B4 are used, respectively, and FIGS. 35 (a) to 35 (d) respectively show the addition patterns P _B1 to P _B1 to P _B1 . The state of the pixel signal of the original image when addition reading is performed using P _B4 is shown.
FIGS. 36A to 36D show how signals are added when the addition patterns P _{C1 to} P _C4 are used, respectively, and FIGS. 37A to 37D show the addition patterns P _C1 to P _C1 to P, respectively. in the case of performing the addition reading using a P _C4, showing the state of the pixel signals of the original image.
FIGS. 38A to 38D show the state of signal addition when the addition patterns P _{D1 to} P _D4 are used, respectively, and FIGS. 39A to 39D show the addition patterns P _D1 to P _D1 to P, respectively. in the case of performing the addition reading using a P _D4, showing the state of the pixel signals of the original image.

In FIGS. 34A to 34D, the black circles indicate the arrangement of virtual light receiving pixels that are assumed when the addition patterns P _{B1 to} P _B4 are used as the first to fourth addition patterns, respectively. Indicates the position. However, in FIGS. 34A to 34D, only the arrangement positions of the virtual light receiving pixels corresponding to the R signal among the assumed virtual light receiving pixels are clearly shown.
In FIGS. 36A to 36D, black circles indicate the arrangement of virtual light receiving pixels that are assumed when the addition patterns P _{C1 to} P _C4 are used as the first to fourth addition patterns, respectively. Indicates the position. However, in FIGS. 36A to 36D, only the arrangement positions of the virtual light receiving pixels corresponding to the B signal among the assumed virtual light receiving pixels are clearly shown.
In FIGS. 38A to 38D, black circles indicate the arrangement of virtual light receiving pixels that are assumed when the addition patterns P _{D1 to} P _D4 are used as the first to fourth addition patterns, respectively. Indicates the position. However, in FIGS. 38A to 38D, only a part of the arrangement positions of the virtual light receiving pixels corresponding to the G signal among the assumed virtual light receiving pixels is clearly shown.

  In FIGS. 34 (a) to (d), FIGS. 36 (a) to (d) and FIGS. 38 (a) to (d), an arrow shown around a black circle represents a virtual corresponding to the circle. In order to generate a pixel signal of a typical light receiving pixel, the pixel signals of the peripheral light receiving pixels of the virtual light receiving pixel are added.

When performing addition reading using an arbitrary addition pattern,
Virtual green light receiving pixels are arranged at pixel positions [p _G1 + 4n _A , p _G2 + 4n _B ] and [p _G3 + 4n _A , p _G4 + 4n _B ] of the image sensor 33, and pixel positions [p _B1 + 4n of the image sensor 33 are set. _{It is} assumed that a virtual blue light receiving pixel is disposed at _A , p _B2 + 4n _B ], and a virtual red light receiving pixel is disposed at the pixel position [p _R1 + 4n _A , p _R2 + 4n _B ] of the image sensor 33. (N _A and n _B are integers). However,
When the addition patterns P _B1 , P _B2 , P _B3, and P _B4 are used, (p _G1 , p _G2 , p _G3 , p _G4 , p _B1 , p _B2 , p _R1 , p _R2 ) are respectively
(4,2,3,3,3,2,4,3), (6,4,5,5,5,4,6,5),
(6, 2, 5, 3, 5, 2, 6, 3) and (4, 4, 3, 5, 3, 4, 4, 5),
When the addition patterns P _C1 , P _C2 , P _C3 and P _C4 are used, (p _G1 , p _G2 , p _G3 , p _G4 , p _B1 , p _B2 , p _R1 , p _R2 ) are respectively
(3, 3, 2, 4, 3, 4, 2, 3), (5, 5, 4, 6, 5, 6, 4, 5),
(5, 3, 4, 4, 5, 4, 4, 3) and (3, 5, 2, 6, 3, 6, 2, 5),
When the addition patterns P _D1 , P _D2 , P _D3, and P _D4 are used, (p _G1 , p _G2 , p _G3 , p _G4 , p _B1 , p _B2 , p _R1 , p _R2 ) are respectively
(3, 3, 4, 4, 3, 4, 4, 3), (5, 5, 6, 6, 5, 6, 6, 5),
(5, 3, 6, 4, 5, 4, 6, 3) and (3, 5, 4, 6, 3, 6, 4, 5).
( _PG1 , _pG2 , _pG3 , _pG4 , _pB1 , _pB2 , _pR1 , _pR2 ) are ( _pG1 ', _pG2 ', _pG3 ', _pG4 ', _pB1 ', p _B2 ', p _R1', 'is to _{be), p G1 = p G1'} p R2, p G2 = p G2 ', p G3 = p G3', p G4 = p G4 ', p B1 = p B1' , P _B2 = p _B2 ′, p _R1 = p _R1 ′ and p _R2 = p _R2 ′.

  As described in the first embodiment, the pixel signal of one virtual light receiving pixel is the actual light reception adjacent to the upper left, upper right, lower left and lower right of the virtual light receiving pixel. This is an addition signal of pixel signals of pixels. Then, the original image is acquired so that the pixel signal of the virtual light receiving pixel arranged at the position [x, y] is handled as the pixel signal at the position [x, y] on the image.

Therefore, the original images obtained by addition reading using an arbitrary addition pattern are as shown in FIGS. 35 (a) to (d), FIGS. 37 (a) to (d) and FIGS. 39 (a) to (d). , Pixels having only a G signal and pixel positions [p _B1 + 4n _A , p _B2 + 4n, which are arranged at pixel positions [p _G1 + 4n _A , p _G2 + 4n _B ] and [p _G3 + 4n _A , p _G4 + 4n _B ]. _B ] is an image including a pixel having only the B signal and a pixel having only the R signal, which is disposed at the pixel position [p _R1 + 4n _A , p _R2 + 4n _B ].

Incidentally, an addition pattern group consisting of addition patterns P _{A1 to} P _A4 , an addition pattern group consisting of addition patterns P _{B1 to} P _B4 , an addition pattern group consisting of addition patterns P _{C1 to} P _C4, and an addition pattern P _{D1 to} P _D4. The addition pattern group is represented by P _A , P _B , P _C and P _D , respectively.

<< Sixth Example >>
Next, a sixth embodiment will be described. In the sixth embodiment, the addition pattern group used for acquiring the original image is switched and used based on the detection result of the motion detection unit.

  First, the significance of this switching will be described. When a color interpolation image is generated from an original image, signal interpolation is performed depending on the pixel position. For example, when generating the G signal 311 on the color-interpolated image 261 shown in FIG. 13A, as described with reference to FIG. Signal interpolation using the G signal is performed. On the other hand, the G signal 313 in FIG. 13A is the G signal itself of one real pixel on the original image 251. That is, signal interpolation is not performed when the G signal 313 is generated. When signal interpolation is performed, the resolution (substantial resolution) is inevitably lowered.

Therefore, when comparing an image composed of color signals obtained by performing signal interpolation such as the G signal 311 and an image composed of color signals obtained without performing signal interpolation such as the G signal 313, the latter is preferred. However, it can be said that the resolution (substantial resolution) is higher than the former. Therefore, in the color interpolation image 261 shown in FIGS. 13A and 20A, the resolution in the direction from the upper left to the lower right is compared with that in other directions (particularly the direction from the lower left to the upper right). Is expensive. This is because the G signals G1 _1,1 , G1 _2,2 , G1 _3,3 and G1 _4,4 arranged in the direction from the upper left to the lower right can be obtained without performing signal interpolation (see FIG. 20A). . Conversely, for example, in a color interpolation image (see FIG. 34A, FIG. 35A, etc.) obtained using the addition pattern group P _B , the resolution in the direction from the lower left to the upper right is in other directions ( Especially in the direction from the upper left to the lower right).

  On the other hand, when the subject in the image has a motion in the image sequence, the contour portion that intersects the direction of the motion perpendicularly tends to be blurred. Considering these circumstances, in the sixth embodiment, a plurality of addition pattern groups to be used for the current frame are determined based on the motion detection result obtained for the past frame so that the blur is eliminated as much as possible. Dynamically select from the addition pattern group.

  Regarding the color interpolation image or motion vector, the direction from the upper left to the lower right indicates the direction from the position [1, 1] to the position [10, 10] on the image coordinate plane XY, and from the lower left to the upper right. The direction refers to a direction from the position [1, 10] on the image coordinate plane XY toward the position [10, 1]. A straight line along the direction from the upper left to the lower right or a line along the same direction as that direction is called a right-down straight line, a straight line along the direction from the lower left to the upper right or a straight line along the same direction as that direction. Is called a straight line going up to the right (see FIG. 40).

With reference to FIG. 41, the switching method according to the sixth embodiment will be specifically described on the assumption that the addition pattern group used between the addition pattern group P _A and the addition pattern group P _B is switched. In the sixth embodiment, the video signal processing unit 13a of FIG. 10 or FIG. 26 or the video signal processing unit 13b of FIG. 30 is used as the video signal processing unit 13 of FIG.

In the period of acquiring an original image by using the sum pattern group P _A, by performing the addition reading using an addition read with addition pattern P _A1 the addition pattern P _A2 alternately, sequentially adding pattern P _A1, Original images of P _A2 , P _A1 , P _A2 ,... _Are acquired, and one output composite image is generated from two color interpolation images based on two original images that are temporally adjacent. Similarly, during the period in which the original image is acquired using the addition pattern group P _B , the addition pattern using the addition pattern P _B1 and the addition reading using the addition pattern P _B2 are alternately executed, thereby sequentially adding the addition patterns. Original images of P _B1 , P _B2 , P _B1 , P _B2 ,... _Are acquired, and one output composite image is generated from two color interpolated images based on two temporally adjacent original images. The

  Now, as shown in FIG. 41, the color interpolation processing unit 51 performs the n-th, (n + 1) -th, (n + 2) -th, (n + 3) -th, (n + 4)... Original images 400, 401, 402, 403 and 404... To nth, (n + 1) th, (n + 2), (n + 3), (n + 4)... Color interpolation images 410, 411, 412, 413 and 414. Suppose that is generated.

The motion detection unit 53 obtains a motion vector between adjacent frames as described in the first embodiment. A motion vector between the color interpolation images 410-411, a motion vector between the color interpolation images 411-412, and a motion vector between the color interpolation images 412-413 are represented by M ₀₁ , M _12, and M _23, respectively. The motion vector M ₀₁ is assumed to be an average motion vector as described in the second embodiment, which represents the average motion of the subject between the color-interpolated images 410-411 (for motion vectors M ₁₂ and M ₂₃₎ . The same).

Early addition pattern groups used to acquire the original image is added pattern group P _A, the addition pattern group used at the time of acquisition of the original image 400 to 403 is assumed to be an addition pattern group P _A. At this time, the video signal processing unit 13 or the pattern switching control unit (not shown) included in the CPU 23 selects the addition pattern groups used when acquiring the original image 404 based on the selection motion vector as the addition pattern groups P _A and P _B. Choose from. The selection motion vector is formed from one or a plurality of motion vectors obtained before acquisition of the original image 404. The motion vector for selection includes, for example, a motion vector M _{23 and} may further include a motion vector M ₁₂ or motion vectors M ₁₂ and M ₀₁ . A motion vector obtained in the past than the motion vector M ₀₁ may be further included in the selection motion vector. Usually, the motion vector for selection is formed from a plurality of motion vectors.

When the selection motion vector is composed of a plurality of motion vectors (for example, M ₂₃ and M ₁₂ ), the pattern switching control unit pays attention to the plurality of motion vectors, and the directions of the plurality of motion vectors are all parallel to the right-up straight line. addition pattern case, the addition pattern group used at the time of acquisition of the original image 404 is switched to the addition pattern group P _B from the addition pattern group P _a, otherwise, that without the switching, used at the time of acquisition of the original image 404 the group that remains addition pattern groups P _a.

Although different from the above assumption, the addition pattern group used when acquiring the original images 400 to 403 is the addition pattern group P _B and the selection motion vector is composed of a plurality of motion vectors (for example, M ₂₃ and M ₁₂ ). In this case, the pattern switching control unit pays attention to the plurality of motion vectors, and when the directions of the plurality of motion vectors are all parallel to the right-downward straight line, the addition pattern group used when acquiring the original image 404 is selected as the addition pattern group P. Switching from _B to the addition pattern group P _A , otherwise, the switching is not performed and the addition pattern group used when acquiring the original image 404 remains the addition pattern group P _B.

If the selected motion vector consists only of the vector M ₂₃ motion, pattern switching control unit is focused on the motion vector M _23, if the direction of the motion vector M ₂₃ is upward sloping straight line and parallel, at the time of acquisition of the original image 404 used to select the sum pattern group P _B as an addition pattern group, if parallel to the linear direction of motion vector M ₂₃ is lowered right, by selecting the sum pattern group P _a as an addition pattern group used at the time of acquisition of the original image 404 Good.

  As described above, by variably setting the addition pattern group to be used, the optimum addition pattern group can be used according to the movement of the subject in the image, and the image quality of the output composite image sequence can be optimized.

A process for prohibiting frequent changes in the addition pattern group used for acquiring the original image may be added. For example, as shown in FIG. 41, the addition pattern group P sum pattern group from the addition pattern group P _A to be used for obtaining the sum pattern group adds pattern group P _A in a by and the original image 404 using the acquired original image 403 _When changed to _B , the addition pattern group P _B may be used without fail when acquiring a specified number of original images acquired after the original image 404.

Further, although the above-described example of switching between the sum pattern group to be used for obtaining the original image and addition pattern group P _A and the addition pattern group P _B, the addition pattern group to be used for obtaining the original image, adding pattern group P _A and between the addition pattern group P _C, or may be switched between a sum pattern group P _B and the addition pattern group P _D.

<< Seventh Embodiment >>
In the first to sixth embodiments, the pixel signal of the original image is acquired by addition reading, but it is also possible to acquire the pixel signal of the original image by thinning-out reading. An embodiment in which pixel signals of the original image are acquired by performing thinning readout will be described as a seventh embodiment. Even when the pixel signal of the original image is acquired by thinning-out reading, the matters described in the first to sixth embodiments are applicable as long as there is no contradiction.

  As is well known, in thinning readout, the light receiving pixel signals of the image sensor 33 are thinned out and read out. In the seventh embodiment, thinning-out reading is performed while sequentially changing a thinning pattern used for acquiring an original image between a plurality of thinning patterns, and a plurality of color-interpolated images having different thinning patterns are synthesized to produce one output composition. Generate an image. The thinning pattern means a combination pattern of light receiving pixels to be thinned. The plurality of thinning patterns used include two or more thinning patterns among first, second, third and fourth thinning patterns which are different from each other. The positions of thinning differ between the first to fourth thinning patterns.

As a thinning pattern group consisting of the first to fourth thinning patterns, a thinning pattern group Q _A consisting of thinning patterns Q _{A1 to} Q _{A4, a} thinning pattern group Q _B consisting of thinning patterns Q _{B1 to} Q _B4 , and a thinning pattern Q _C1 to A thinning pattern group Q _C composed of Q _C4 or a thinning pattern group Q _D composed of thinning patterns Q _{D1 to} Q _D4 can be used.

42A to 42D show thinning patterns Q _{A1 to} Q _A4 , respectively. FIGS. 43A to 43D show thinning-out readings using thinning patterns Q _{A1 to} Q _A4 , respectively. This shows the state of the pixel signal of the original image when
44A to 44D show thinning patterns Q _{B1 to} Q _B4 , respectively, and FIGS. 45A to 45D show thinning readout using the thinning patterns Q _{B1 to} Q _B4 , respectively. This shows the state of the pixel signal of the original image when
46A to 46D show thinning patterns Q _{C1 to} Q _C4 , respectively. FIGS. 47A to 47D show thinning-out readings using thinning patterns Q _{C1 to} Q _C4 , respectively. This shows the state of the pixel signal of the original image when
48A to 48D show thinning patterns Q _{D1 to} Q _D4 , respectively. FIGS. 49A to 49D show thinning-out readings using thinning patterns Q _{D1 to} Q _D4 , respectively. This shows the state of the pixel signal of the original image when

  42 (a) to (d), FIGS. 44 (a) to (d), FIGS. 46 (a) to (d), and FIGS. 48 (a) to 48 (d), the pixel signals of the light receiving pixels within the round frame. Are read out as pixel signals of actual pixels of the original image, and the pixel signals of the light receiving pixels located between the round frames adjacent in the horizontal or vertical direction are thinned out.

When performing thinning readout using an arbitrary thinning pattern,
Pixel position of the image sensor _{33 [p G1 + 4n A,} p G2 + 4n B] and _{[p G3 + 4n A, p} G4 + 4n B] pixel position of the pixel signal is the original image of the green light receiving pixels arranged in a [p _G1 + 4n _A , P _G2 + 4n _B ] and [p _G3 + 4n _A , p _G4 + 4n _B ] are read as G signals and arranged at the pixel position [p _B1 + 4n _A , p _B2 + 4n _B ] of the image sensor 33. Is read out as a B signal at the pixel position [p _B1 + 4n _A , p _B2 + 4n _B ] of the original image, and is arranged at the pixel position [p _R1 + 4n _A , p _R2 + 4n _B ] of the image sensor 33. The pixel signal of the light receiving pixel is read out as an R signal at the pixel position [p _R1 + 4n _A , p _R2 + 4n _B ] of the original image (n _A and n _B are integers). However,
When the thinning patterns Q _A1 , Q _A2 , Q _A3 and Q _A4 are used, (p _G1 , p _G2 , p _G3 , p _G4 , p _B1 , p _B2 , p _R1 , p _R2 ) are respectively
(1,1,2,2,2,1,1,2), (3,3,4,4,4,3,3,4),
(3, 1, 4, 2, 4, 1, 3, 2) and (1, 3, 2, 4, 2, 3, 1, 4),
When the thinning patterns Q _B1 , Q _B2 , Q _B3, and Q _B4 are used, (p _G1 , p _G2 , p _G3 , p _G4 , p _B1 , p _B2 , p _R1 , p _R2 ) are respectively
(3, 1, 2, 2, 2, 1, 3, 2), (5, 3, 4, 4, 4, 3, 5, 4),
(5, 1, 4, 2, 4, 1, 5, 2) and (3, 3, 2, 4, 2, 3, 3, 4),
When the thinning patterns Q _C1 , Q _C2 , Q _C3 and Q _C4 are used, (p _G1 , p _G2 , p _G3 , p _G4 , p _B1 , p _B2 , p _R1 , p _R2 ) are respectively
(2,2,1,3,2,3,1,2), (4,4,3,5,4,5,3,4),
(4, 2, 3, 3, 4, 3, 3, 2) and (2, 4, 1, 5, 2, 5, 1, 4),
When the thinning patterns Q _D1 , Q _D2 , Q _D3, and Q _D4 are used, (p _G1 , p _G2 , p _G3 , p _G4 , p _B1 , p _B2 , p _R1 , p _R2 ) are respectively
(2,2,3,3,2,3,3,2), (4,4,5,5,4,5,5,4),
(4, 2, 5, 3, 4, 3, 5, 2) and (2, 4, 3, 5, 2, 5, 3, 4).

  The pixel on the original image corresponding to the pixel position from which the G, B, or R signal is read is an actual pixel in which the G, B, or R signal exists, but all of the G, B, and R signals are read. The pixel on the original image corresponding to the pixel position that has not been output is a blank pixel in which none of the G, B, and R signals exist.

As an example, a process for generating an output composite image using a thinning pattern group Q _A composed of thinning patterns Q _{A1 to} Q _A4 will be described. As the video signal processing unit 13 according to the seventh embodiment, the video signal processing unit 13a of FIG. 10 or FIG. 26 or the video signal processing unit 13b of FIG. 30 can be used.

As can be seen from FIGS. 43 (a) to 43 (d) and FIGS. 9 (a) to 9 (d), the original image obtained by the thinning readout using the thinning patterns Q _{A1 to} Q _A4 and the addition patterns P _{A1 to} P _When comparing with the original image acquired by addition reading using _A4 , the relationship of the existence positions of the G, B, and R signals is the same between the two original images. However, with the latter original image as a reference, the positions of the G, B, and R signals in the former original image are shifted by Wp in the right direction and by Wp in the downward direction (see FIG. 4A). Therefore, when applying the above-described items on the premise of performing addition reading to an imaging apparatus that performs thinning-out reading, the above-described items may be corrected by an amount corresponding to this shift.

  Except for this deviation, both original images (the original image corresponding to FIGS. 43A to 43D and the original image corresponding to FIGS. 9A to 9D) can be treated as equivalent. Therefore, the matters described in the first to fourth embodiments can be applied to the seventh embodiment as they are. Basically, the addition pattern and addition reading in the first to fourth embodiments may be replaced with a thinning pattern and thinning reading.

That is, for example, assuming that the thinning patterns Q _A1 and Q _A2 are respectively used as the first and second thinning patterns, the first and second thinning patterns are used alternately. Acquire original images alternately. Then, by executing the color interpolation processing described in the first embodiment for each original image based on the thinning pattern, the color interpolation processing unit 51 generates a color interpolation image, while in the first embodiment. By executing the motion detection process described above, the motion detection unit 53 detects a motion vector between adjacent frames. Then, based on the detected motion vector, according to the method described in any of the first to third embodiments, the image composition unit 55 or 55b generates one output composite image from a plurality of color interpolation images. Generate. It is also possible to apply the image compression technique described in the fourth embodiment to the output composite image sequence based on the original image sequence obtained by the thinning readout.

Furthermore, even when thinning readout is used, the technique described in the sixth embodiment functions effectively. When the technique described in the sixth embodiment is applied to the seventh embodiment using thinning readout, the term “addition pattern” and “addition pattern group” appearing in the description of the sixth embodiment is referred to as the term “thinning pattern and thinning pattern”. Along with the replacement, the code corresponding to the addition pattern or the addition pattern group may be replaced with the code corresponding to the thinning pattern or the reduction pattern group. Specifically, the addition pattern groups P _A , P _B , P _C and P _D in the sixth embodiment are replaced with thinning pattern groups Q _A , Q _B , Q _C and Q _D , respectively, and addition in the sixth embodiment is performed. The patterns P _A1 , P _A2 , P _B1 and P _B2 may be read as thinning patterns Q _A1 , Q _A2 , Q _B1 and Q _B2 , respectively.

  Note that the light receiving pixel signal of the image sensor 33 may be read using a reading method (hereinafter referred to as an addition / decimation method) that combines the above-described addition reading method and the thinning reading method. A read pattern when the addition / decimation method is used is called an addition / decimation pattern. As an example, an addition / decimation pattern corresponding to FIGS. 50 and 51 can be adopted. This addition / decimation pattern functions as a first addition / decimation pattern. FIG. 50 shows a state of signal addition and a state of signal thinning when the first addition / decimation pattern is used, and FIG. 51 shows a case where a light receiving pixel signal is read according to the first addition / decimation pattern. The state of the pixel signal of the original image is shown.

When using this first addition / decimation pattern:
Pixel position of the image sensor _{33 [2 + 6n A, 2} + 6n B] and virtual green light receiving pixels are arranged in a _{[3 + 6n A, 3 +} 6n B], virtual blue pixel positions of the image sensor _{33 [3 + 6n A, 2} + 6n B] It is assumed that light receiving pixels are arranged and virtual red light receiving pixels are arranged at pixel positions [2 + 6n _A , 3 + 6n _B ] of the image sensor 33 (n _A and n _B are integers).

As described in the first embodiment, the pixel signal of one virtual light receiving pixel is the actual light reception adjacent to the upper left, upper right, lower left and lower right of the virtual light receiving pixel. This is an addition signal of pixel signals of pixels. Then, the original image is acquired so that the pixel signal of the virtual light receiving pixel arranged at the position [x, y] is handled as the pixel signal at the position [x, y] on the image. Therefore, an original image obtained by reading using the first addition / decimation pattern is a G signal arranged at pixel positions [2 + 6n _A , 2 + 6n _B ] and [3 + 6n _A , 3 + 6n _B ] as shown in FIG. _A pixel having only the B signal, a pixel having only the B signal arranged at the pixel position [3 + 6n _A , 2 + 6n _B ], and a pixel having only the R signal arranged at the pixel position [2 + 6n _A , 3 + 6n _B ], It becomes an image with

Thus, since the pixel signal of the original image is formed by adding a plurality of light receiving pixel signals, the addition / decimation method is a kind of addition reading method. At the same time, the light receiving pixel signals at the positions [5, n _B ], [6, n _B ], [n _A , 5] and [n _A , 6] do not contribute to the generation of the pixel signal of the original image. That is, when the original image is generated, the light receiving pixel signals at the positions [5, n _B ], [6, n _B ], [n _A , 5] and [n _A , 6] are thinned out. Therefore, it can be said that the addition / decimation method is a kind of decimation readout method.

  As described above, the addition pattern means a combination pattern of light receiving pixels to be added, and the thinning pattern means a combination pattern of light receiving pixels to be thinned. On the other hand, the addition / decimation pattern means a combination pattern of light receiving pixels to be added and thinned. Even when the addition / decimation method is used, a plurality of different addition / decimation patterns are set, and the addition / decimation pattern used to acquire the original image is sequentially changed between the plurality of addition / decimation patterns. One output composite image may be generated by performing readout and synthesizing a plurality of color interpolation images having different corresponding addition / decimation patterns.

<< Eighth Example >>
In each of the above-described embodiments, a plurality of color-interpolated images are synthesized, and an output synthesized image obtained by the synthesis is given to the signal processing unit 56. However, without performing this synthesis, R, The G and B signals can be given to the signal processing unit 56 as R, G, and B signals of one output image. An embodiment in which this synthesis is not performed will be described as an eighth embodiment. The matters described in the above embodiments can be applied to the eighth embodiment as long as there is no contradiction. However, since the synthesis process is not performed, the technique related to the synthesis is not applied to the eighth embodiment.

  The video signal processing unit 13c shown in FIG. 52 can be used as the video signal processing unit 13 according to the eighth embodiment. The video signal processing unit 13 c includes a color interpolation processing unit 51, a signal processing unit 56, and an image conversion unit 57. The functions of the color interpolation processing unit 51 and the signal processing unit 56 are the same as those described above.

  The color interpolation processing unit 51 performs the above-described color interpolation processing on the original image represented by the output signal of the AFE 12 to generate a color interpolation image. The R, G, and B signals of the color interpolation image generated by the color interpolation processing unit 51 are given to the image conversion unit 57. The image conversion unit 57 generates R, G, and B signals of the output image from the R, G, and B signals of the given color interpolation image. The signal processing unit 56 converts the R, G, and B signals of the output image generated by the image conversion unit 57 into a video signal composed of the luminance signal Y and the color difference signals U and V. The video signals (Y, U, and V) obtained by this conversion are sent to the compression processing unit 16 and are compressed and encoded according to a predetermined image compression method. By supplying the video signal of the output image sequence from the image conversion unit 57 to the display unit 27 in FIG. 1 or a display device (not shown), the output image sequence can be displayed as a moving image.

In each of the above-described embodiments, the G signal, the B signal, and the R signal of the output composite image 270 at the position [2i−0.5, 2j−0.5] are respectively represented as Go _{i, j} , Bo _{i, j} and Although represented by Ro _{i, j} (see FIGS. 25A to 25C), in the eighth embodiment, the output of the image conversion unit 57 at the position [2i−0.5, 2j−0.5]. The G signal, B signal, and R signal of the image are represented by Go _{i, j} , Bo _{i, j} and Ro _{i, j,} respectively.

With reference to FIG. 53, the operation of the video signal processing unit 13c will be described with a specific example. Now, using the addition patterns P _A1 and P _A2 as the first and second addition patterns (see FIGS. 7A and 7B), the original images of the first and second addition patterns are alternately photographed. Consider the case. The nth, (n + 1) th, (n + 2) th, and (n + 3) th original images are acquired sequentially. The nth, (n + 1) th, (n + 2) and (n + 3) th original images are the original images of the first, second, first and second addition patterns, respectively, and the nth and nth Assume that the color interpolation images generated from the (n + 1) th original image are color interpolation images 261 and 262, respectively (see FIG. 20A, FIG. 21A, etc.). Also, output images of the image conversion unit 57 generated from the color interpolation images 261 and 262 are output images 501 and 502, respectively.

  In the first embodiment, the position where the G signal is present, the position where the B signal is present, and the position where the R signal is present are different between the color interpolation image 261 and the color interpolation image 262. In consideration thereof, the G, B, and R signals of the color interpolation image 261 and the G, B, and R signals of the color interpolation image 262 are mixed (see equation (B1) and the like).

Based on these differences, the image conversion unit 57 according to the eighth embodiment outputs an output image according to Go _{i, j} = G1 _{i, j} , Bo _{i, j} = B1 _{i, j} and Ro _{i, j} = R1 _{i, j.} 501 G, B, and R signal values are obtained, and Go _{i, j} = G2 _{i−1, j−1} , Bo _{i, j} = B2 _{i−1, j−1} and Ro _{i, j} = R2 _i−1. The G, B, and R signal values of the output image 502 are obtained according to _j-1 . If the color-interpolated image based on the (n + 2) -th original image is also the color-interpolated image 261, the G, B, and R signal values of the output image based on the (n + 2) -th original image are also Bo _{i, j} = B1 _{i, j} and Ro _{i, j} = R1 _{i, j} . Similarly, if the color interpolation image based on the (n + 3) th original image is also the color interpolation image 262, the G, B, and R signal values of the output image based on the (n + 3) th original image are Go _{i, j} = G2 _{i-1, j-1} , Bo _{i, j} = B2 _{i-1, j-1} and Ro _{i, j} = R2 _{i-1, j-1} .

Seen from the positions [2i-0.5, 2j-0.5] of the signals Go _{i, j} , Bo _{i, j} and Ro _{i, j in} the output image, the signals G1 _{i, j} , B1 _i in the color interpolation image 261 _{, j} and R1 _{i, j} are slightly shifted, and the positions of the signals G2 _{i−1, j−1} , B2 _{i−1, j−1} and R2 _{i−1, j−1 in} the color interpolation image 262 are also different. Some deviation.

  Due to these deviations, image quality degradation such as jaggy is observed if each output image is viewed as a still image. However, output images are sequentially generated from the image conversion unit 57 in a frame cycle, and when the output image sequence is viewed as a moving image, the user hardly feels such image quality deterioration depending on the frame cycle. . The reason for this is the same as the reason why almost no video flicker can be detected in interlaced moving images, and the afterimage effect of the user's eyes is used.

On the other hand, the sampling point of the color signal at the same position [2i−0.5, 2j−0.5] is different between the output image 501 and the output image 502. For example, the sampling point (position [6, 6]) of the G signal G1 _3,3 of the color interpolation image 261 used as the G signal Go _{3,3 at} the position [5.5, 5.5] of the output image 501 And the sampling point (position [5, 5]) of the G signal G12, ₂ of the color interpolation image 262, which is used as the G signal Go _{3,3 at} the position [5.5, 5.5] of the output image 502 Is different. When such an output image sequence including the output images 501 and 502 is displayed as a moving image, the afterimage effect of the eyes works, and the user recognizes image information at both sampling points at once. That is, it is possible to compensate for image quality deterioration due to addition reading of the light-receiving pixel signals (image quality deterioration due to thinning readout when thinning readout is performed). In addition, since the interpolation process for equalizing the pixel intervals (see blocks 902 and 903 in FIG. 54) is not performed, deterioration in resolution is suppressed. That is, the resolution can be improved as compared with the conventional method corresponding to FIG.

  The process of generating the output image sequence including the output images 501 and 502 is useful when the frame period is relatively high (for example, when the frame period is 1/60 second). When the frame period is relatively low (for example, when the frame period is 1/30 second), the afterimage effect of the eyes is weakened. Therefore, a plurality of color-interpolated images as described in the first to seventh embodiments are used. It is better to generate an output composite image based on it.

  The block for realizing the function of the video signal processing unit 13c shown in FIG. 52 and the block for realizing the function of the video signal processing unit 13a or 13b shown in FIG. 10, FIG. 26 or FIG. It may be mounted on the unit 13 and used properly depending on the frame period. That is, when the frame period is larger than a predetermined reference period (for example, 1/30 second), the former block is operated to output the output image sequence from the image conversion unit 57, and the frame period is equal to or less than the reference period. In this case, the latter block may be operated to output the output composite image sequence from the image compositing unit 55 or 55b.

Further, using the sum pattern P _A1 and P _A2 of the first and second summing pattern has been described above an example of obtaining the original image of the addition pattern P _A1 and P _A2 are alternately first to fourth addition of with addition pattern P _A1 to P _A4 as pattern, it may be sequentially repeated to obtain the original image of the addition pattern P _A1 to P _A4. In this case, the original image is obtained by sequentially using the addition patterns P _A1 , P _A2 , P _A3 , P _A4 , P _A1 , P _A2 , and the like, and the image conversion unit 57 adds the addition patterns P _A1 , P _A2 , Output images based on the original images P _A3 , P _A4 , P _A1 , P _A2 ... _Are sequentially output. Also, as an addition pattern group consisting of the first to fourth summing patterns, instead of adding pattern group P _A consisting addition pattern P _A1 to P _A4, the addition pattern group P _B consisting addition pattern P _B1 to P _B4, An addition pattern group P _C composed of the addition patterns P _{C1 to} P _C4 or an addition pattern group P _D composed of the addition patterns P _{D1 to} P _D4 may be used (see FIG. 34A, etc.).

Further, the frame memory 52, the motion detection unit 53, and the memory 54 shown in FIG. 10 are added to the video signal processing unit 13c. The addition pattern group used for acquiring the original image may be switched and used. For example, as described in the sixth embodiment (see FIG. 41), the imaging device 1 is formed so that the addition pattern group used between the addition pattern group P _A and the addition pattern group P _B can be switched. . When it is from the original image 400 to 404 and the color interpolated images 410-414 is obtained, based on the selection for the motion vectors comprising motion vectors M _23, according to the method described in the sixth embodiment, at the time of acquisition of the original image 404 it may be selected from among the sum pattern group P _a and P _B addition pattern group is used.

Further, the items described in the eighth embodiment can be applied to thinning-out reading so that the first to sixth embodiments can be modified as in the seventh embodiment. In this case, the terms “addition pattern” and “addition pattern group” appearing in the above description in the eighth embodiment may be replaced with the terms “decimation pattern” and “decimation pattern”. Along with the replacement, the code corresponding to the addition pattern or the addition pattern group may be replaced with the code corresponding to the thinning pattern or the thinning pattern group (specifically, P _A , P _B , P _C and P _D are replaced with Each of them is read as Q _A , Q _B , Q _C and Q _D , and P _{A1 to} P _A4 , P _{B1 to} P _B4 , P _{C1 to} P _C4 and P _{D1 to} P _D4 are changed to Q _{A1 to} Q _A4 , Q respectively. _B1 to Q _B4, may be read as the Q _C1 to Q _C4 and Q _D1 ~Q _D4).

However, as described in the seventh embodiment, the original image obtained by the thinning readout using the thinning patterns Q _{A1 to} Q _A4 and the addition reading using the addition patterns P _{A1 to} P _A4 are obtained. When the original image is compared, the relationship between the positions of the G, B, and R signals is the same between the two original images, but the G, B, and R signals in the former original image are based on the latter original image. Is shifted by Wp in the right direction and by Wp in the downward direction (see FIG. 4A, FIG. 9A, FIG. 43A, etc.). A similar deviation also exists between the addition pattern group P _B and the thinning pattern group Q _B. Therefore, when thinning-out reading is performed, the matter described above in the eighth embodiment may be corrected by an amount corresponding to this shift.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 3 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
The addition pattern described above can be variously modified. In the above addition readout method, one pixel signal on the original image is formed by adding four light receiving pixel signals. However, a plurality of light receiving pixel signals other than four (for example, nine or sixteen light receiving pixels). (One light receiving pixel signal) may be added to form one pixel signal on the original image.

  Similarly, the thinning pattern described above can be variously modified. In the thinning readout method described above, the light receiving pixel signals are thinned out by two pixels in the horizontal and vertical directions, but the number of light receiving pixel signals to be thinned may be other than two. For example, the light receiving pixel signal may be thinned out by four pixels in the horizontal and vertical directions.

[Note 2]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, all or part of the processing executed in the video signal processing unit (13, 13a, 13b, or 13c) can be realized using software. Of course, it is also possible to form the video signal processing unit only by hardware. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

[Note 3]
For example, it can be considered as follows. The CPU 23 in FIG. 1 controls what addition pattern or thinning pattern is used when acquiring the original image, and under this control, a signal to be a pixel signal of the original image is read from the image sensor 33. . Therefore, it can be considered that the original image acquisition means for acquiring the original image is mainly realized by the CPU 23 and the video signal processing unit 13, and the original image acquisition means includes a reading means for performing addition reading or thinning reading. You can also think that Note that, as described above, the addition / decimation method that combines the addition readout method and the thinning-out readout method is a kind of the addition readout method or the thinning-out readout method. In addition, the readout of the light-receiving pixel signal by the addition readout method can be considered as a kind of addition readout or thinning readout.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is a figure which shows the light receiving pixel arrangement | sequence in the effective area | region of the image pick-up element of FIG. It is a figure which shows the color filter arrangement | sequence with respect to the image pick-up element of FIG. FIG. 2A is a diagram showing a pixel arrangement of an original image photographed by the imaging apparatus of FIG. 1 and FIG. 2B is a diagram showing an image coordinate plane for an arbitrary image including the original image. It is an image figure of the pixel signal in the original picture obtained using all the pixel readout methods. Conceptual diagrams (a) to (c) of color interpolation processing performed on the original image of FIG. 5, and an image diagram (d) of a color interpolation image obtained by performing color interpolation processing on the original image of FIG. To (e). It is a figure which concerns on 1st Example of this invention and shows the mode of the signal addition in the case of using the 1st and 2nd addition pattern. It is a figure which concerns on 1st Example of this invention and shows the mode of the signal addition in the case of using the 3rd and 4th addition pattern. FIG. 6 is a diagram illustrating a state of a pixel signal of an original image obtained when addition reading is performed using the first to fourth addition patterns according to the first embodiment of the present invention. FIG. 2 is a partial block diagram of the imaging apparatus according to the first embodiment of the present invention, including an internal block diagram of the video signal processing unit of FIG. 1. FIG. 6 is a diagram for describing a method of calculating a pixel signal at an interpolation pixel position from a plurality of pixel signals according to the first embodiment of the present invention. It is a figure which shows a mode that G, B, and R signal in the original image acquired using the 1st addition pattern are mixed according to 1st Example of this invention. FIG. 13 is a diagram illustrating G, B, and R signals on a color interpolation image generated from an original image corresponding to FIGS. 12A to 12C according to the first embodiment of the present invention. It is a figure which shows a mode that G, B, and R signal in the original image acquired using the 2nd addition pattern are mixed according to 1st Example of this invention. It is a figure which concerns on 1st Example of this invention and shows G, B, and R signal on the color interpolation image produced | generated from the original image corresponding to Fig.14 (a)-(c). It is a figure which shows a mode that G, B, and R signal in the original image acquired using the 3rd addition pattern are mixed according to 1st Example of this invention. It is a figure which concerns on 1st Example of this invention and shows G, B, and R signal on the color interpolation image produced | generated from the original image corresponding to Fig.16 (a)-(c). It is a figure which concerns on 1st Example of this invention and shows a mode that G, B, and R signal in the original image acquired using the 4th addition pattern are mixed. It is a figure which concerns on 1st Example of this invention and shows G, B, and R signal on the color interpolation image produced | generated from the original image corresponding to Fig.18 (a)-(c). FIG. 13 is a diagram illustrating G, B, and R signals on a color interpolation image generated from an original image corresponding to FIGS. 12A to 12C according to the first embodiment of the present invention. It is a figure which concerns on 1st Example of this invention and shows G, B, and R signal on the color interpolation image produced | generated from the original image corresponding to Fig.14 (a)-(c). FIG. 6 is a diagram illustrating a color interpolation image and a corresponding luminance image according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating G, B, and R signals on a color interpolation image for generating G, B, and R signals on an output composite image according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating B and R signals on a color interpolation image for generating B and R signals on an output composite image according to the first embodiment of the present invention. It is a figure which concerns on 1st Example of this invention and shows the presence position of G, B, and R signal of an output synthetic | combination image. FIG. 5 is a partial block diagram of an imaging apparatus according to a second embodiment of the present invention, including an internal block diagram of a video signal processing unit in FIG. 1. It is a figure which shows the example of a relationship between the magnitude | size of a motion vector and the weighting coefficient utilized for the color signal mixing of several color interpolation image concerning 2nd Example of this invention. FIG. 5A is a diagram illustrating a state in which an entire image region of one image is divided into nine partial image regions, and FIG. 5B is a diagram illustrating a motion vector group between two images. It is a figure which shows the relationship between two color interpolation images and one output synthetic | combination image. FIG. 7 is a partial block diagram of an imaging apparatus according to a third embodiment of the present invention, including an internal block diagram of a video signal processing unit in FIG. 1. FIG. 11A is a diagram illustrating an example of a relationship between a contrast amount of an image and a reference weight value used for mixing color signals of a plurality of color-interpolated images according to the third embodiment of the present invention; It is a figure (b) which shows the example of a relationship with the weighting coefficient utilized for the color signal mixing of the color interpolation image of. It is a figure which shows the structure of the MPEG moving image which concerns on 4th Example of this invention. It is a figure which concerns on 4th Example of this invention and shows the relationship between a color interpolation image sequence, an output synthetic | combination image sequence, and a total weight coefficient sequence. It is a figure which shows the mode of the signal addition in the case of using the addition pattern group (P _B1 -P _B4 ) according to the fifth embodiment of the present invention. FIG. 35 is a diagram illustrating a state of a pixel signal of an original image obtained when addition reading is performed using the addition pattern group of FIGS. 34A to 34D according to the fifth embodiment of the present invention. It is a figure which shows the mode of the signal addition in the case of using the addition pattern group (P _{C1 to} P _C4 ) according to the fifth embodiment of the present invention. FIG. 37 is a diagram illustrating a state of a pixel signal of an original image obtained when addition reading is performed using the addition pattern group of FIGS. 36A to 36D according to the fifth embodiment of the present invention. It relates to the fifth embodiment of the present invention, showing the state of signal addition in the case of using the sum pattern group (P _D1 ~P _D4). FIG. 39 is a diagram illustrating a state of a pixel signal of an original image obtained when addition reading is performed using the addition pattern group of FIGS. 38A to 38D according to the fifth example of the present invention. It is a figure which shows the downward slope straight line and the upward slope straight line which are defined in 6th Example of this invention. It is a figure which concerns on 6th Example of this invention and shows the relationship between an original image sequence, a color interpolation image sequence, a motion vector sequence, and the addition pattern group applied to each original image. It relates to a seventh embodiment of the present invention, showing a thinning pattern group (Q _A1 ~Q _A4). FIG. 45 is a diagram illustrating a state of a pixel signal of an original image obtained when thinning readout is performed using the thinning pattern group in FIGS. 42A to 42D according to the seventh embodiment of the present invention. It relates to a seventh embodiment of the present invention, showing a thinning pattern group (Q _B1 ~Q _B4). FIG. 45 is a diagram illustrating a state of a pixel signal of an original image obtained when thinning readout is performed using the thinning pattern group in FIGS. 44A to 44D according to the seventh embodiment of the present invention. It relates to a seventh embodiment of the present invention, showing a thinning pattern group (Q _C1 ~Q _C4). FIG. 49 is a diagram illustrating a state of a pixel signal of an original image obtained when thinning readout is performed using the thinning pattern group in FIGS. 46A to 46D according to the seventh embodiment of the present invention. It relates to a seventh embodiment of the present invention, showing a thinning pattern group (Q _D1 ~Q _D4). FIG. 49 is a diagram illustrating a state of a pixel signal of an original image obtained when thinning readout is performed using the thinning pattern group in FIGS. 48A to 48D according to the seventh embodiment of the present invention. It is a figure which shows the mode of signal addition and the state of signal decimation when an addition / thinning pattern is used according to the seventh embodiment of the present invention. FIG. 50 is a diagram illustrating a state of a pixel signal of an original image when a light receiving pixel signal is read according to the addition / thinning pattern of FIG. 50 according to the seventh embodiment of the present invention. FIG. 10 is a partial block diagram of an imaging apparatus according to an eighth embodiment of the present invention, including an internal block diagram of a video signal processing unit in FIG. 1. FIG. 20 is a diagram illustrating a relationship among an original image sequence, a color interpolation image sequence, and an output image sequence according to the eighth embodiment of the present invention. FIG. 10 is a diagram for describing processing for generating an output image from an original image obtained by performing addition reading of light receiving pixel signals of an image sensor according to a conventional technique.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 12 AFE
13, 13a, 13b, 13c Video signal processing unit 16 Compression processing unit 33 Image sensor 51 Color interpolation processing unit 52 Frame memory 53 Motion detection unit 54 Memory 55, 55b Image composition unit 56 Signal processing unit 57 Image conversion unit 61, 71 Weight Coefficient calculation unit 62, 72 Composition processing unit 70 Contrast calculation unit

Claims (11)

A reading means for performing addition reading or thinning readout of pixel signals of a light receiving pixel group arranged two-dimensionally on a single-plate image sensor, and having a plurality of readout patterns with different combinations of light receiving pixels to be added or thinned out By using the original image acquisition means for acquiring a plurality of original images having different pixel positions having pixel signals,
Interpolation processing means for mixing pixel signals of the same color included in the pixel signal group of the original image for each original image, and generating an interpolated image having pixel signals obtained by the mixing, An image processing apparatus. The pixel signal group of each interpolation image is composed of pixel signals of a plurality of colors including the first color,
2. The image processing apparatus according to claim 1, wherein in each interpolated image, intervals between pixel positions where the pixel signals of the first color are present are unequal. A focused original image included in the plurality of original images is referred to as a focused original image, an interpolation image generated from the focused original image is referred to as a focused interpolation image, and a focused color included in the plurality of colors When the pixel signal is called the target color pixel signal,
The interpolation processing unit sets an interpolation pixel position at a position different from a pixel position where the target color pixel signal of the target original image exists, and an image having a pixel signal of the target color at the interpolation pixel position is set as the target interpolation image. Produces as
When generating the target interpolation image, paying attention to a plurality of pixel positions on the target original image where a target color pixel signal exists, and mixing a plurality of target color pixel signals at the plurality of pixel positions Generate a pixel signal for the interpolated pixel position,
The image processing apparatus according to claim 2, wherein the interpolation pixel position is set to a gravity center position of the plurality of pixel positions. The image processing apparatus according to claim 3, wherein the interpolation processing unit generates a pixel signal for the interpolation pixel position by mixing the plurality of target color pixel signals at an equal ratio. Further comprising output image generation means for generating one output image based on a plurality of interpolation images generated from the plurality of original images by the interpolation processing means;
The output image has pixel signals for a plurality of colors at each of the equally arranged pixel positions,
The output image generation means generates the output image based on a difference in pixel position between the plurality of interpolation images, which is derived from a difference in a corresponding read pattern between the plurality of interpolation images. The image processing apparatus according to claim 1. A motion detecting means for detecting a motion of an object between the plurality of interpolation images;
The image processing apparatus according to claim 5, wherein the output image generation unit generates the output image based on not only the difference in the pixel position but also the magnitude of the movement. The plurality of interpolated images include at least first and second interpolated images;
The output image generation means sets a weight coefficient that sets a weight coefficient based on the magnitude of motion of the object between the first interpolation image and the second interpolation image detected by the motion detection means. 7. The image processing apparatus according to claim 6, further comprising: means for generating the output image by mixing pixel signals of the first and second interpolation images according to the weighting factor. By repeatedly executing the operation of acquiring the plurality of original images using the plurality of read patterns, an output image sequence arranged in a time series is generated by the output image generation unit,
The image processing apparatus
Image compression means for generating a compressed moving image including an intra-frame encoded image and an inter-frame predictive encoded image by performing an image compression process on the output image sequence;
The image compression means selects an output image to be a target of the intra-frame encoded image from the output image sequence based on a weighting factor set corresponding to each output image forming the output image sequence The image processing apparatus according to claim 7, wherein: The image processing apparatus according to claim 1, wherein a plurality of interpolation images generated from the plurality of original images by the interpolation processing unit are output as moving images. There are multiple sets of multiple readout patterns with different combinations of light receiving pixels to be added or thinned,
The image processing apparatus
A motion detecting means for detecting a motion of an object between the plurality of interpolation images;
The image processing according to claim 1, 2, 3, 4, 5, or 9, wherein a set of readout patterns used for acquiring the original image is variably set based on a detected direction of motion. apparatus. A single-plate image sensor;
An image processing apparatus comprising: the image processing apparatus according to claim 1.

Images