JP2010251882A - Image capturing apparatus and image reproducing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a recording data size of images without causing deterioration in image quality of reproduced images. <P>SOLUTION: By acquiring input images including M pixels by photographing and performing resampling by handling the input image as an image (300) to be resampled, a first reduced image (311) and a second reduced image (312) including M/4 pixels each are generated. The resampling is executed so that sampling positions deviate between the first and second reduced images. In photographing, image data of the first and second reduced images are recorded on an external memory instead of the input image. In reproduction, resolution of the reduced images is increased based on information on position deviation between the image data of the first and second reduced images read from the external memory and both the images, and an output image including resolution equivalent to that of the input image is generated and displayed. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital still camera and an image reproduction apparatus that reproduces an image.

  An imaging apparatus such as a digital still camera is usually equipped with a recording medium such as a semiconductor memory or a magnetic disk, and image data of a captured image is recorded on the recording medium. In recent years, the data size of a photographed image has increased with the increase in definition of a CCD or the like. In order to increase the number of images that can be recorded on a recording medium as much as possible, reduction of the recording data size is naturally desired.

  Also, in the process of recording image data on a recording medium, various image processing (such as noise reduction processing) is often performed on the image data while using an internal memory (such as SDRAM) mounted on the imaging device. The smaller the size of the image data to be subjected to image processing, the lower the power consumption required for executing the image processing. In image processing (for example, three-dimensional digital noise reduction processing) that performs noise reduction using the correlation between L processing target images, the image data of (L-1) processing target images is stored in an internal memory. In this state, the image processing of the remaining image to be processed is input to the image processing unit to realize the image processing (L is an integer of 2 or more). In this type of image processing, if the data size per image to be processed is reduced, the number of images that can be stored in the internal memory having the specified capacity increases, so that the value of L can be increased. In the image processing for noise reduction described above, increasing the value of L generally increases the noise reduction effect. For this reason, from the viewpoint of increasing the effect of image processing, it is desired to reduce the size of image data to be subjected to image processing.

  On the other hand, high resolution processing (super-resolution processing) for generating one high resolution image from a plurality of low resolution images has been proposed. In an imaging device formed so as to be able to perform super-resolution processing, a plurality of captured images are acquired as a plurality of low-resolution images by moving image shooting or continuous shooting of still images, and super-resolution processing is performed on the plurality of captured images. Thus, a high-resolution image exceeding the resolution of the captured image is generated (for example, see Patent Document 1 below).

JP 2006-41603 A

  As described above, it is desired to reduce the size of image data to be recorded and to reduce the size of image data to be subjected to image processing.

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging apparatus capable of reducing the size of image data to be recorded or the size of image data to be subjected to image processing. It is another object of the present invention to provide an image reproducing apparatus capable of generating a high-definition image from recorded data whose data size has been reduced.

  An imaging apparatus according to the present invention includes an imaging unit that acquires an input image including M pixels by imaging, and a reduced image generation unit that generates P reduced images each having N pixels from the input image (M, N And P is an integer of 2 or more, and M> N × P), and a recording control unit that records the P reduced images on a recording medium.

  Thereby, instead of recording data for M pixels, data for (N × P) pixels is recorded. Here, M> (N × P). Therefore, the size of the image data to be recorded is reduced.

  Specifically, for example, the P reduced images recorded on the recording medium include a high resolution unit that generates a high resolution image having a resolution higher than the resolution of the low resolution image from a plurality of low resolution images. It is preferable that the imaging apparatus further includes an output image generation unit that generates an output image as the high resolution image by inputting the image as the plurality of low resolution images to the high resolution unit.

  As a result, it is also possible to generate an output image having the same resolution as the input image from P reduced images. That is, it is possible to reduce the recording data size with almost no deterioration in the quality of the reproduced image.

  In addition, for example, the imaging apparatus further includes an image processing unit that performs predetermined image processing on the P reduced images generated by the reduced image generation unit, and the recording control unit is configured to perform post-image processing. P reduced images are recorded on the recording medium.

  The image reproduction apparatus according to the present invention is an image reproduction apparatus that generates an output image from P reduced images recorded on the recording medium, and that has a plurality of low resolution images that have a resolution lower than the resolution of the low resolution image. The output image as a high-resolution image by including a high-resolution part that generates a high-resolution image having a high resolution, and inputting the P reduced images as the plurality of low-resolution images to the high-resolution part. An output image generation unit for generating the image is provided.

  Thereby, it is also possible to generate an output image having the same resolution as the input image from P reduced images.

  Another imaging apparatus according to the present invention includes an imaging unit that acquires an input image including M pixels by imaging, and a reduced image generation unit that generates P reduced images each having N pixels from the input image. (M, N, and P are integers of 2 or more, and M> N × P), an image processing unit that performs predetermined image processing on the P reduced images, and a plurality of low-resolution images Including a high resolution unit for generating a high resolution image having a resolution higher than that of the low resolution image, and inputting the P reduced images after the image processing to the high resolution unit as the plurality of low resolution images Thus, an output image generation unit that generates an output image as the high-resolution image and a recording control unit that records the output image on a recording medium are provided.

  As a result, the size of the image data to be subjected to image processing is reduced, and as a result, a reduction in power consumption required for executing the image processing is expected. Further, the memory capacity necessary for executing the image processing is reduced by reducing the size of the image data to be subjected to the image processing. Alternatively, the number of images that can be handled by a memory of a predetermined capacity increases, and the effect is expected to increase depending on the contents of image processing.

  In the imaging apparatus, for example, the reduced image generation unit generates each reduced image by resampling the input image, and the resampled position on the input image is the P reduced images. The resolution increasing unit generates the output image as the high resolution image by using a shift of the resampled position between the P reduced images.

  According to the present invention, it is possible to provide an imaging apparatus capable of reducing the size of image data to be recorded or the size of image data to be subjected to image processing. In addition, it is possible to provide an image reproducing apparatus capable of generating a high-definition image from recorded data whose data size has been reduced.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is a conceptual diagram of the reconstruction type super-resolution process based on the MAP method using an iterative operation. 3 is a flowchart showing a flow of super-resolution processing corresponding to FIG. 2. FIG. 5A is a diagram showing a pixel array of a two-dimensional image, and FIG. It is a figure which shows the color filter arrangement | sequence of the image pick-up element of FIG. FIG. 6 is a block diagram (a) of a part responsible for processing during image capturing and recording and a block diagram (b) of a part responsible for processing during image reproduction according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating an original image and the number of pixels of a color interpolation image based on the original image according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating a relationship between a resampling target image and two reduced images according to the first embodiment of the present invention. It is a figure which shows the pixel arrangement | sequence of the resampling object image which concerns on 1st Example of this invention. In resampling method of the first embodiment of the present invention (A _1), showing the positional relationship between the pixels on the pixel and the first and second reduced images on resampling the target image drawing (a), the first FIG. 4B is a diagram showing a pixel array of a reduced image, and FIG. 3C is a diagram showing a pixel array of a second reduced image. It is a figure for demonstrating the basic policy of the resampling method. In resampling method of the first embodiment of the present invention (A _2), a diagram showing the positional relationship between the pixels on the pixel and the first and second reduced images on resampling the target image. In resampling method of the first embodiment of the present invention (A _3), a diagram showing the positional relationship between the pixels on the pixel and the first and second reduced images on resampling the target image. In resampling method of the first embodiment of the present invention (A _4), a diagram showing the positional relationship between the pixels on the pixel and the first and second reduced images on resampling the target image. In resampling method of the first embodiment of the present invention (A _5), a diagram showing the positional relationship between the pixels on the pixel and the first and second reduced images on resampling the target image. In resampling method of the first embodiment of the present invention (A _6), a diagram showing the positional relationship between the pixels on the pixel and the first and second reduced images on resampling the target image. It is a block diagram of the site | part responsible for the process at the time of image imaging | photography and recording in 2nd Example of this invention. In resampling method of the second embodiment of the present invention (B _1), it is a diagram showing a state of pixel signals of the first and second reduced image shown on resampling the target image. FIG. 10 is a diagram illustrating a total of (L × P) reduced images based on L original images and L original images according to the second embodiment of the present invention. It is a modification block diagram of the site | part responsible for the process at the time of image picking-up and recording in 2nd Example of this invention.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to third embodiments will be described later. First, matters common to each embodiment or items referred to in each embodiment will be described.

[Description of basic configuration]
FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 has each part referred by the codes | symbols 11-28. The imaging device 1 is a digital video camera, and can capture a moving image and a still image, and also can simultaneously capture a still image during moving image capturing. However, you may make it comprise the imaging device 1 as a digital still camera which can image | photograph only a still image. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25. The display unit 27 and / or the speaker 28 may be provided in an external device (not shown) of the imaging device 1.

  The imaging unit 11 includes an imaging system (image sensor) 33, an optical system (not shown), a diaphragm, and a driver. The image sensor 33 is formed by arranging a plurality of light receiving pixels in the horizontal and vertical directions. The image sensor 33 is a solid-state image sensor composed of a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. Each light receiving pixel of the image sensor 33 photoelectrically converts an optical image of an object incident through an optical system and a diaphragm, and outputs an electric signal obtained by the photoelectric conversion to an AFE 12 (Analog Front End). Each lens constituting the optical system forms an optical image of the subject on the image sensor 33.

  The AFE 12 amplifies the analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The amplification degree of signal amplification in the AFE 12 is controlled by a CPU (Central Processing Unit) 23. The video signal processing unit 13 performs necessary image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The microphone 14 converts the ambient sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 using a predetermined compression method. The internal memory 17 is composed of a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores various data. The external memory 18 as a recording medium is a non-volatile memory such as a semiconductor memory or a magnetic disk, and records a video signal and an audio signal compressed by the compression processing unit 16.

  The decompression processing unit 19 decompresses the compressed video signal and audio signal read from the external memory 18. The video signal expanded by the expansion processing unit 19 or the video signal from the video signal processing unit 13 is sent to the display unit 27 such as a liquid crystal display via the display processing unit 20 and displayed as an image. Further, the audio signal that has been expanded by the expansion processing unit 19 is sent to the speaker 28 via the audio output circuit 21 and output as sound.

  The TG (timing generator) 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and gives the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each part in the imaging apparatus 1. The operation unit 26 includes a recording button 26a for instructing start / end of moving image shooting and recording, a shutter button 26b for instructing shooting and recording of a still image, an operation key 26c, and the like. Accept the operation. The operation content for the operation unit 26 is transmitted to the CPU 23.

  The operation mode of the imaging apparatus 1 includes a shooting mode in which an image (still image or moving image) can be shot and recorded, and an image (still image or moving image) recorded in the external memory 18 is reproduced and displayed on the display unit 27. Playback mode to be included. Transition between the modes is performed according to the operation on the operation key 26c.

  In the shooting mode, a subject is periodically shot at a predetermined frame period, and shot images of the subject are sequentially acquired. A digital video signal representing an image is also called image data. One image is represented by image data for one frame period. One image represented by image data for one frame period is also called a frame image.

  The video signal processing unit 13 is configured to be able to perform super-resolution processing in cooperation with the CPU 23. A part responsible for the super-resolution processing is called a super-resolution processing unit (corresponding to a super-resolution processing unit 61 in FIG. 6B described later). Through the super-resolution processing, one high-resolution image is generated from a plurality of low-resolution images. The video signal of the high resolution image can be recorded in the external memory 18 via the compression processing unit 16. The resolution of the high resolution image is higher than that of the low resolution image, and the number of pixels in the horizontal and vertical directions of the high resolution image is larger than that of the low resolution image. For example, when a still image shooting instruction is given, a plurality of frame images as a plurality of low resolution images are acquired, and a high resolution image can be generated by performing super-resolution processing on these. Alternatively, for example, the super-resolution processing is performed on a plurality of frame images as a plurality of low resolution images obtained at the time of moving image shooting.

[Basic concept of super-resolution processing]
The basic concept of super-resolution processing will be briefly described. There is a method called a reconstruction type as a type of super-resolution processing. On the other hand, there is a method for realizing super-resolution processing by repetitive calculation (repetitive calculation algorithm), and this repetitive calculation can also be applied to reconstruction-type super-resolution processing. In the present embodiment, a main example is a reconstruction type super-resolution process based on a MAP (Maximum A Posterior) method using a repetitive calculation, which is a kind of super-resolution process that can be realized by a repetitive calculation.

  FIG. 2 shows a conceptual diagram of a reconstruction type super-resolution process based on the MAP method using repetitive operations. In the super-resolution processing, one high-resolution image is estimated from a plurality of low-resolution images input to the super-resolution processing unit, and the estimated plurality of low-resolution images are degraded by degrading the estimated high-resolution image. Is estimated. The low resolution image input to the super-resolution processing unit is particularly referred to as “real low resolution image”, and the estimated low resolution image is particularly referred to as “estimated low resolution image”. Thereafter, the high resolution image and the low resolution image are repeatedly estimated so that the error between the actual low resolution image and the estimated low resolution image is minimized, and the finally acquired high resolution image is output.

  FIG. 3 is a flowchart showing the flow of super-resolution processing corresponding to FIG. First, in step S11, an initial high resolution image is generated from an actual low resolution image. In subsequent step S12, the original actual low resolution image for constructing the current high resolution image is estimated. The estimated image is referred to as an estimated low resolution image as described above. In the subsequent step S13, an update amount for the current high resolution image is derived based on the difference (difference image) between the actual low resolution image and the estimated low resolution image. This update amount is derived so that the error between the actual low-resolution image and the estimated low-resolution image is minimized by repeatedly executing the processes in steps S12 to S14. In the subsequent step S14, the current high resolution image is updated using the updated amount, and a new high resolution image is generated. Thereafter, the process returns to step S12, and the newly generated high resolution image is regarded as the current high resolution image, and the processes of steps S12 to S14 are repeatedly executed. Basically, as the number of repetitions of each process of steps S12 to S14 increases, the resolution (substantial resolution) of the obtained high-resolution image is improved, and an ideal high-resolution image is obtained.

  The super-resolution processing based on the above-described operation flow is performed in the super-resolution processing unit in the imaging apparatus 1. In addition to super-resolution processing based on the MAP method, it is also possible to use super-resolution processing based on the ML (Maximum-Likelihood) method, the POCS (Projection Onto Convex Set) method, or the IBP (Iterative Back Projection) method. is there.

  In the reconfiguration-type super-resolution processing represented by the MAP method, after an initial high-resolution image is generated, calculation processing (super-resolution calculation) including calculation of the update amount and update of the high-resolution image based on the update amount. Process) is repeatedly executed, but it is not essential to repeat this calculation process. That is, an initial high resolution image may be generated from a plurality of low resolution images, and the initial high resolution image may be handled as a high resolution image to be finally obtained without being updated.

[Pixel array]

  The output signal of the AFE 12 is called RAW data, and a two-dimensional image represented by RAW data for one frame period is called an original image.

  FIG. 4A shows a pixel array of an arbitrary two-dimensional image 200 including the original image. It can be considered that the two-dimensional image 200 is formed from a pixel group two-dimensionally arranged on the image coordinate plane XY which is a two-dimensional orthogonal coordinate system (see FIG. 4B).

  The symbol P [x, y] represents a pixel on the two-dimensional image 200. Of all the pixels forming the two-dimensional image 200, the pixel closest to the origin of the two-dimensional image 200 is the pixel P [1,1]. In the two-dimensional image 200, pixels adjacent to the right, left, top, and bottom of the pixel P [x, y] are pixels P [x + 1, y], P [x-1, y], and P [x, y + 1] and P [x, y-1]. In the two-dimensional image 200, the vertical direction corresponds to the vertical direction, and the horizontal direction corresponds to the horizontal direction.

  A position on the two-dimensional image 200 (position on the image coordinate plane XY) is represented by a symbol [x, y]. If x and y are integers, the position [x, y] in the two-dimensional image 200 matches the center position of the pixel P [x, y] in the two-dimensional image 200. In the following description, the symbol [x, y] may be used as a symbol representing the pixel position in order to clarify that the pixel position is indicated.

  The imaging device 1 employs a so-called single plate method that uses only one image sensor. FIG. 5 shows an arrangement of color filters arranged in front of each light receiving pixel of the image sensor 33. The array shown in FIG. 5 is generally called a Bayer array. Color filters include a red filter that transmits only the red component of light, a green filter that transmits only the green component of light, and a blue filter that transmits only the blue component of light. In FIG. 5, a part corresponding to the red filter is represented by R, a part corresponding to the green filter is represented by G, and a part corresponding to the blue filter is represented by B.

The pixel signal of the pixel P [x, y] of the original image is an output signal of the AFE 12 for the light receiving pixel corresponding to the pixel P [x, y]. More specifically, in the original image, the pixel signal of the pixel P [2n _A -1,2n _B ] is an output signal of the AFE 12 for the light receiving pixel on which the red filter is arranged, and represents the intensity of the red component. The pixel signal of the pixel P [2n _A , 2n _B −1] is composed only of the B signal representing the intensity of the blue component, which is the output signal of the AFE 12 for the light receiving pixel in which the blue filter is disposed, The pixel signal of the pixel P [2n _A -1,2n _B -1] or P [2n _A , 2n _B ] is an output signal of the AFE 12 for the light receiving pixel in which the green filter is arranged, and represents the intensity of the green component. It consists only of G signals. n _A and n _B are natural numbers. Note that any of the R signal, the G signal, and the B signal may be referred to as a color signal, and they may be collectively referred to as a color signal.

  As described above, the imaging apparatus 1 has a function of generating a high-resolution image exceeding the resolution of the frame image by performing super-resolution processing using the frame image that is the captured image itself as a low-resolution image. When the user does not use this function, the presence of the super-resolution processing unit is wasted. Considering this, the imaging apparatus 1 effectively uses the super-resolution processing unit even when this function is not used.

  Hereinafter, first to third embodiments will be described as embodiments relating to effective use of the super-resolution processing unit. Matters described in one embodiment also apply to other embodiments as long as no contradiction arises.

<< First Example >>
First, the first embodiment will be described. FIG. 6A is a block diagram of a part that is included in the imaging apparatus 1 and that is responsible for processing during image capturing and recording. FIG. 6B is a block diagram of a portion that is included in the imaging apparatus 1 and that performs processing during image reproduction. Each part referred to by reference numerals 51 to 53 shown in FIG. 6A is provided in the video signal processing unit 13a, and each part referred to by reference numerals 61 to 64 shown in FIG. 6B is a display processing part. 20a. The video signal processing unit 13 a performs various processes using the internal memory 17. The video signal processing unit 13a and the display processing unit 20a can be used as the video signal processing unit 13 and the display processing unit 20 in FIG. 1, respectively.

  The raw data from the AFE 12 is sent to the demosaicing processing unit 51. As shown in FIG. 7, the number of pixels in the horizontal and vertical directions of the original image represented by the RAW data is represented by Mx and My, respectively (Mx and My are integers of 2 or more, for example, several hundred to several 1000). In the original image, R signals are arranged in a mosaic according to a Bayer array (the same applies to G signals and B signals). The demosaicing processing unit 51 generates image data in RGB format by executing a known demosaicing process on the RAW data. In the two-dimensional image generated by the demosaicing process, all R, G, and B signals are assigned to each pixel. A two-dimensional image generated by performing demosaicing processing on the original image is called a color interpolation image.

  The number of pixels of the color interpolation image is the same as the number of pixels of the original image. That is, the number of pixels in the horizontal direction and the vertical direction of the color interpolation image (and resampled image 300 described later; see FIG. 8) is Mx and My, respectively, as shown in FIG. The R, G, and B signals of the pixel P [x, y] of the color-interpolated image are the pixel signal in the pixel P [x, y] of the original image and the pixel signal in the peripheral pixels of the pixel P [x, y] of the original image. Is generated using a well-known interpolation process.

  The image processing unit 52 performs necessary image processing (such as noise reduction processing and edge enhancement processing) on the color interpolation image. As shown in FIG. 8, the resampling processing unit 53 handles the color-interpolated image after the image processing as the resampling target image 300 and generates P reduced images by resampling the resampling target image 300. P is an integer of 2 or more. FIG. 8 illustrates a case where P is 2. The number of pixels in the horizontal direction and the vertical direction of each reduced image generated by the resampling processing unit 53 is represented by Nx and Ny, respectively. Nx and Ny are integers of 2 or more (for example, several hundred to several thousand), but the inequality “(Mx × My)> P × (Nx × Ny)” is always satisfied. The reduction ratio of the reduced image with respect to the resampling target image is simply referred to as a reduction ratio. When the reduction ratio is 1 / Q, Nx = Mx / Q and Ny = My / Q.

  Image data of P reduced images is compressed by a predetermined compression method (for example, JPEG (Joint Photographic Experts Group) method) in the compression processing unit 16, and the compressed image data is recorded in the external memory 18.

  Before explaining the contents of resampling in detail, the operation of each part shown in FIG. 6B will be described. In the reproduction mode, the expansion processing unit 19 generates image data before compression by expanding the compressed image data of P reduced images read from the external memory 18. Image data of P reduced images that have not been compressed is input to the super-resolution processing unit 61. The super-resolution processing unit 61 regards the P reduced images as P low-resolution images, and generates a single high-resolution image by performing super-resolution processing on the P low-resolution images. . The number of pixels in the horizontal and vertical directions of the generated high-resolution image is the same as that of the original image.

  The video signal of the high resolution image generated by the super-resolution processing unit 61 is sent to the display video processing unit 62. The display video processing unit 62 converts the resolution of the image as necessary so that the image represented by the given video signal can be displayed on the display unit 27, and the image after the resolution conversion. Are written in a VRAM 63 (Video Random Access Memory). The VRAM 63 is a memory of a video display portion for the display unit 27. The display driver 64 displays an image represented by the video signal written in the VRAM 63 on the display screen of the display unit 27.

A resampling method for generating P reduced images will be described. As an example, resampling methods A _{1 to} A ₆ will be described. As long as there is no contradiction, the matters described in a certain resampling method can be applied to other resampling methods.

-Resampling method A _1-
Illustrating the resampling method A _1. In the resampling method A ₁ , P is 2 and the reduction ratio is 1/2. Two reduced images in the resampling method A ₁ are referred to by reference numerals 311 and 312.

  FIG. 9 shows a pixel array of the resampling target image 300. FIG. 9 shows (8 × 8) pixels on the resampling target image 300. As described above, if x and y are integers, the position [x, y] on the resampling target image 300 represents the center position of the pixel P [x, y] of the resampling target image 300.

  FIG. 10A shows the positional relationship between the pixels on the resampling target image 300 and the pixels on the first and second reduced images 311 and 312. The first reduced image 311 is formed from each pixel surrounded by a thick solid line in FIG. 10A, and the second reduced image 312 is formed from each pixel surrounded by a thick broken line in FIG. Is done. FIGS. 10B and 10C show pixel arrays of the first and second reduced images 311 and 312, respectively. From (8 × 8) pixels on the resampling target image 300, (4 × 4) pixels on the reduced image 311 and (4 × 4) pixels on the reduced image 312 are generated.

The pixel signal (R, G and B signals) at the position [2n _A -1, 2n _B -1] on the resampling target image 300 is the pixel signal (R, N _B ) at the position [n _A , n _B ] on the reduced image 311. G and B signals), and the pixel signals (R, G and B signals) at the positions [2n _A , 2n _B ] on the resampling target image 300 are the positions [n _A , n on the reduced image 312. _B ] is resampled so as to be a pixel signal (R, G, and B signals) (as described above, n _A and n _B are natural numbers).

The pixel signals at positions [2n _A -1,2n _B -1] and positions [2n _A , 2n _B ] on the resampling target image 300 are the pixels P [2n _A -1,2n _B ] on the resampling target image 300. −1] and [2n _A , 2n _B ]. Therefore, the resampling in the resampling method A ₁ can be said to be a simple thinning process (the same applies to the resampling method A ₆ corresponding to FIG. 16 described later).

Here, the basic policy of resampling will be described with reference to FIG. This policy is applicable to the resampling process comprising a resampling method A _1. A curve 305 in FIG. 11 represents the waveform of the analog R signal on the target horizontal line (y is an odd horizontal line) of the resampling target image 300. Two black circles in FIG. 11 represent digital R signals of x = 1 and x = 3 on the horizontal line of interest, and two white triangles in FIG. 11 represent digital of x = 2 and x = 4 on the horizontal line of interest. Represents the R signal. The image data of the resampling target image 300 includes these four digital R signals, but the image data of the first reduced image 311 includes only the digital R signals of x = 1 and x = 3. Yes. In order to accurately reproduce the analog R signal 305 using the first and second reduced images, the R signal information at x = 2 and x = 4 on the resampling target image 300 is used as the second reduced image. It is better to include them (the same applies to the G and B signals). The same applies to the case where attention is paid to the horizontal line, but attention is paid to the vertical line.

In accordance with this basic policy, in the resampling method A ₁ , in addition to the first reduced image 311, the second reduced image 312 as described above including signal information at x = 2n _A is generated.

Sampling positions (resampling positions) when P reduced images are generated by resampling differ among P reduced images. When generating the P reduced images, the resampling processing unit 53 also creates misregistration information indicating the difference between the sampling positions (resampling positions). The positional deviation information is recorded in the external memory 18 while being associated with the compressed image data of P reduced images. Incidentally, the creation and recording position shift information is performed also in different other resampling methods resampling method A _1. Further, the method for associating the positional deviation information with the compressed image data of P reduced images is arbitrary. For example, an image file having a main body area and a header area is created in the external memory 18, and compressed image data of P reduced images is stored in the main body area of the image file, while misregistration information is stored in the image file. Should be stored in the header area. Since the main body area and the header area in the same image file are recording areas associated with each other, the compressed image data and the positional deviation information of P reduced images are associated with each other by such storage.

  The misregistration information is vector information including horizontal and vertical components representing a difference in sampling position (resampling position) between the first reduced image and the pth reduced image with reference to the first reduced image. It is. Here, p is an integer of 2 or more and P or less. When viewed from the sampling position of the first reduced image 311 (for example, the position [1,1] on the reduced image 311), the corresponding sampling position of the second reduced image 312 (for example, the position [1,1 on the reduced image 312) 1]) is shifted to the right by one pixel on the resampling target image 300 and shifted downward by one pixel on the resampling target image 300.

“As seen from the sampling position of the first reduced image, the corresponding sampling position of the p-th reduced image is shifted to the right by Δx pixels on the resampling target image, and Δy pixels on the resampling target image. The content of “shifted downward” is represented by DIS _p [Δx, Δy]. Then, in the resampling method A ₁ , DIS ₂ [1, 1] is generated as the positional deviation information. In this example, Δx and Δy are expressed in units of adjacent pixel intervals of the resampling target image, but Δx and Δy may be expressed in units of adjacent pixel intervals of the reduced image.

-Resampling method A _2-
Illustrating the resampling method A _2. In resampling method A _2, a P 2 is the reduction ratio is 1/3. Two reduced images in the resampling method A ₂ are referred to by reference numerals 321 and 322.

  FIG. 12 shows the positional relationship between the pixels on the resampling target image 300 and the pixels on the first and second reduced images 321 and 322. The first reduced image 321 is formed from each pixel surrounded by a thick solid line in FIG. 12, and the second reduced image 322 is formed from each pixel surrounded by a thick broken line in FIG.

Position in resampling the target image 300 on the [3n _A -2,3n _B -2] pixel signals (R, G and B signals) located at the on the reduced image 321 [n _A, n _B] pixel signal (R in, G and B signals) and the pixel signals (R, G, and B signals) at positions [3n _A −0.5, 3n _B −0.5] on the resampling target image 300 are reduced images 322. Resampling is performed so that the pixel signal (R, G, and B signals) at the upper position [n _A , n _B ] is obtained.

When the values x and / or y representing the position [x, y] on the resampling target image 300 are values including decimal numbers, such as the position [3n _A −0.5, 3n _B −0.5], The pixel signal at the position [x, y] on the sampling target image 300 is interpolated (for example, linearly) using the pixel signal of the pixel on the resampling target image 300 arranged around the position [x, y]. Generated by interpolation or bilinear interpolation). For example, the pixel signal at the position [2.5, 2.5] on the resampling target image 300 to be the pixel signal at the position [1, 1] on the second reduced image 322 is Generated by interpolation from pixel signals of the pixels P [2,2], P [2,3], P [3,2] and P [3,3] (specifically, an average signal thereof) ). Of course, the interpolation is performed individually for each of the R, G, and B signals.

In the resampling method A ₂ , DIS ₂ [1.5, 1.5] is generated as misregistration information.

-Resampling method A _3-
Illustrating the resampling method A _3. In resampling method A _3, P is a 3 the reduced rate is 1/2. Three reduced images in the resampling method A ₃ are referred to by reference numerals 331 to 333.

  FIG. 13 shows the positional relationship between the pixels on the resampling target image 300 and the pixels on the first to third reduced images 331 to 333. The first reduced image 331 is formed from each pixel surrounded by the thick solid line in FIG. 13, and the second reduced image 332 is formed from each pixel surrounded by the thick broken line in FIG. The image 333 is formed from each pixel (corresponding to a hatched portion) surrounded by a dashed line in FIG.

Pixel signals (R, G) at positions [2n _A -1, 2n _B -1], [2n _A , 2n _B ] and [2n _A -0.5, 2n _B -0.5] on the resampling target image 300. And B signals) are resampled so that they become pixel signals (R, G, and B signals) at positions [n _A , n _B ] on the reduced images 331, 332, and 333, respectively, and DIS ₂ [1 , 1] and DIS ₃ [0.5, 0.5] are generated as misregistration information.

- re-sampling method A ₄ -
Illustrating the resampling method A _4. In the resampling method A ₄ , P is 3 and the reduction ratio is 1/2. The three reduced images in resampling method A ₄ referred to by numerals 341 to 343.

  FIG. 14 shows the positional relationship between the pixels on the resampling target image 300 and the pixels on the first to third reduced images 341 to 343. The first reduced image 341 is formed from each pixel surrounded by the thick solid line in FIG. 14, and the second reduced image 342 is formed from each pixel surrounded by the thick broken line in FIG. The image 343 is formed from each pixel (corresponding to a shaded area) surrounded by a dashed line in FIG.

Pixel signals (R) at positions [2n _A −1, 2n _B −1], [2n _A , 2n _B −1] and [2n _A −0.5, 2n _B −0.5] on the resampling target image 300 , G and B signals) are resampled so that they become pixel signals (R, G and B signals) at positions [n _A , n _B ] on the reduced images 341, 342 and 343, respectively, and DIS ₂ [1, 0] and DIS ₃ [0.5, 0.5] are generated as misregistration information.

-Resampling method A _5-
Illustrating the resampling method A _5. In the resampling method A ₅ , P is 3 and the reduction ratio is 1/3. The three reduced images in resampling method A ₅ referred to by numerals 351 to 353.

  FIG. 15 shows the positional relationship between the pixels on the resampling target image 300 and the pixels on the first to third reduced images 351 to 353. The first reduced image 351 is formed from each pixel surrounded by the thick solid line in FIG. 15, and the second reduced image 352 is formed from each pixel surrounded by the thick broken line in FIG. The image 353 is formed from each pixel (corresponding to a hatched portion) surrounded by a dashed line in FIG.

Positions [3n _A -2, 3n _B -2], [3n _A -0.5, 3n _B -0.5] and [3n _A -1.5, 3n _B -1.5] on the resampling target image 300 ] Pixel signals (R, G, and B signals) are resampled to be pixel signals (R, G, and B signals) at positions [n _A , n _B ] on the reduced images 351, 352, and 353, respectively. And DIS ₂ [1.5, 1.5] and DIS ₃ [0.5, 0.5] are generated as misregistration information.

-Resampling method A _6-
Illustrating the resampling method A _6. In the resampling method A ₆ , P is 3 and the reduction ratio is 1/3. Three reduced images in the resampling method A ₆ are referred to by reference numerals 361 to 363.

  FIG. 16 shows the positional relationship between the pixels on the resampling target image 300 and the pixels on the first to third reduced images 361 to 363. The first reduced image 361 is formed from each pixel surrounded by the thick solid line in FIG. 16, and the second reduced image 362 is formed from each pixel surrounded by the thick broken line in FIG. The image 363 is formed from each pixel (corresponding to a hatched portion) surrounded by a dashed line in FIG.

Position in resampling the target image 300 on the _{[3n A -2,3n B -2],} [3n A, 3n B] and the pixel signals of the _{[3n A -1,3n B -1] (} R, G and B signals) Are resampled to be pixel signals (R, G, and B signals) at positions [n _A , n _B ] on the reduced images 361, 362, and 363, respectively, and DIS ₂ [2, 2] and DIS ₃ [1,1] is generated as misregistration information.

In the resampling methods A _{1 to} A ₆ described above, the resampling method when P is 2 or 3 and the reduction ratio is 1/2 or 1/3 is exemplified. It is also possible to set it above, and it is also possible to make the reduction ratio other than 1/2 or 1/3 (the same applies to the second embodiment). In each of the resampling methods described above, it is possible to think upside down or upside down, or to swap the up and down direction and left and right direction (second embodiment). The same applies to the above). For example, in the resampling method A ₆ described above, the positional deviation information is DIS ₂ [2, 2] and DIS ₃ [1, 1], but the positional deviation information is DIS ₂ [2, 1] and DIS _3. Resampling may be performed so that [1,2].

[Operation of super-resolution processor]
In the reproduction mode, image data and positional deviation information of P reduced images are input from the external memory 18 to the super-resolution processing unit 61 via the expansion processing unit 19. This misregistration information indicates misregistration between P reduced images with a so-called sub-pixel resolution having a resolution higher than the pixel interval of the reduced images. The super-resolution processing unit 61 generates one high-resolution image based on the positional deviation information and P reduced images as P low-resolution images. Arbitrary super-resolution processing including known super-resolution processing is used as super-resolution processing for generating one high-resolution image based on positional deviation information having sub-pixel resolution and P low-resolution images. Is possible.

  In the first embodiment, the following usage example of the imaging apparatus 1 is assumed. When the user desires to reduce the recorded size of the captured image, the user performs an operation indicating that effect on the operation unit 26 and then issues a still image capturing instruction by pressing the shutter button 26b. Then, the video signal processing unit 13a generates P reduced images based on the original image acquired after the shutter button 26b is pressed, and uses the P reduced image data instead of the original image data. The compressed data is recorded in the external memory 18. Thereafter, when the user gives a reproduction instruction, one high resolution image having the same resolution (number of pixels) as that of the original image is obtained from the P reduced images using the expansion processing unit 19 and the display processing unit 20a. Generate and display. Such a use example is also applied to a second embodiment described later.

  By making such an operation executable, it is possible to reduce the recording size of the image with almost no deterioration in the quality of the reproduced image, according to the user's desire. In addition, since the number of pixels handled in the subsequent stage of the resampling processing unit is reduced as compared with the normal shooting operation in which resampling is not performed as described above, it is possible to reduce power consumption in the shooting operation.

<< Second Example >>
Next, a second embodiment will be described. FIG. 17 is a block diagram of a part that is included in the imaging apparatus 1 and performs processing during image capturing and recording. The block diagram of the part responsible for processing at the time of image reproduction according to the second embodiment is the same as that of the first embodiment (FIG. 6B). Each part referred by the codes | symbols 71-73 shown by FIG. 17 is provided in the video signal process part 13b. The video signal processing unit 13 b performs various processes using the internal memory 17. The video signal processing unit 13b can be used as the video signal processing unit 13 in FIG.

  RAW data from the AFE 12 is sent to the resampling processing unit 71. The resampling processing unit 71 treats the original image represented by the RAW data as a resampling target image, and generates P reduced images from the resampling target image (P is an integer of 2 or more). The number of pixels in the horizontal and vertical directions of the resampling target image and each reduced image is the same as that in the first embodiment (see FIG. 8). Also in the second embodiment, the inequality “(Mx × My)> P is always used. X (Nx x Ny) "is satisfied. The resampling processing unit 71 also creates misalignment information as described in the first embodiment at the time of resampling.

  In the reduced image generated by the resampling processing unit 71, the R, G, and B signals exist in a mosaic pattern as in the original image. The demosaicing processing unit 72 performs a well-known demosaicing process on each reduced image generated by the resampling processing unit 71, so that each pixel of each reduced image is subjected to R, G, and B signal. Assign all. The image processing unit 73 performs predetermined image processing on the reduced image after the demosaicing process. The image data of the P reduced images after the demosaicing process and the image process is compressed by a predetermined compression method (for example, JPEG method) in the compression processing unit 16, and the compressed image data is externally combined with the positional deviation information. Recorded in the memory 18.

  In the reproduction mode, the decompression processing unit 19 in FIG. 6B receives the compressed image data of P reduced images (P reduced images after the demosaicing process and the image processing) read from the external memory 18. Image data before compression is generated by decompression. Image data of P reduced images that are not compressed is input to the super-resolution processing unit 61 together with the positional deviation information. That is, the image data output from the image processing unit 73 is input to the super-resolution processing unit 61 via the compression processing unit 16, the external memory 18, and the expansion processing unit 19. The operation of the display processing unit 20a after inputting P reduced images is the same as that described in the first embodiment.

A resampling method for generating P reduced images employed in the resampling processing unit 71 will be described. As an example, illustrating the resampling method B ₁ .about.B ₆ corresponding to the resampling method A ₁ to A ₆ above. The positional deviation information generated in the resampling methods B _{1 to} B ₆ is the same as that generated in the resampling methods A _{1 to} A ₆ , respectively. As long as there is no contradiction, the matters described in a certain resampling method can be applied to other resampling methods. Also, the resample target image in the resampling processing unit 71 is referred to by reference numeral 400.

-Resampling method B _1-
Illustrating the resampling method B _1. In the resampling method B ₁ , P is 2 and the reduction ratio is 1/2. Two reduced images in the resampling method B ₁ are referred to by reference numerals 411 and 412.

  The pixel array of the resampling target image 400 is the same as that of the resampling target image 300 shown in FIG. 9, and the positions of the pixels on the resampling target image 400 and the pixels on the first and second reduced images 411 and 412. The relationship is the same as that shown in FIG.

Then, similarly to the resampling method A _1, the position in the re-sampled image 400 on [2n _A -1,2n _B -1] and [2n _A, 2n _B] pixel signals of, respectively, the reduced image 411 and the upper 412 Re-sampling is performed so that the pixel signal at position [n _A , n _B ] is obtained (as described above, n _A and n _B are natural numbers).

However, in the resampling method B ₁ , resampling is performed so that the color signal array on the reduced image becomes a Bayer array (the same applies to resampling methods B _{2 to} B ₆ described later). That is, in the resampling method according to the first embodiment, all of the R, G, and B signals are assigned to each pixel on the reduced image, whereas with the resampling method according to the second embodiment, on the reduced image. Only one of the R, G, and B signals is assigned to one pixel according to the Bayer array. Therefore, on each reduced image obtained by the resampling method according to the second embodiment, the pixel signal of the pixel P [2n _A -1,2n _B ] has only the R signal, and the pixel P [2n _A , 2n pixel signals _B -1] has only the B signal, the pixel signal of the pixel P [2n _a -1,2n _B -1] and P [2n _a, 2n _B] has only G signal.

More specifically, the resampling method B ₁ may be processed as follows. FIG. 18 shows the state of the pixel signals of the first and second reduced images 411 and 412 shown on the resampling target image 400. Position information such as [1, 1] in FIG. 18 represents position information on the resampling target image 400. As apparent from the above description, for example, the pixel position [5, 5] on the resampling target image 400 matches the pixel position [3, 3] on the first reduced image 411, and resampling is performed. The pixel position [6, 6] on the target image 400 matches the pixel position [3, 3] on the second reduced image 412.

The G signal 415 that is the pixel signal at the position [3, 3] on the reduced image 411 is the G signal itself that is the pixel signal at the position [5, 5] on the resampling target image 400. Alternatively, the G signal 415 is generated from G signals at pixel positions around the position [5, 5] and the position [5, 5] on the resampling target image 400 (for example, the positions [5, 5], [4 , 4], [4, 6], [6, 4] and [6, 6] are generated by mixing the G signals).
The G signal 416 that is the pixel signal at the position [4, 4] on the reduced image 411 is the G signal itself that is the pixel signal at the position [7, 7] on the resampling target image 400. Alternatively, the G signal 416 is generated from the G signals at the pixel positions around the position [7, 7] and the position [7, 7] on the resampling target image 400 (for example, the positions [7, 7], [6 , 6], [6, 8], [8, 6] and [8, 8] G signals are mixed).
The R signal 417 that is the pixel signal at the position [3, 4] on the reduced image 411 mixes the R signal that is the pixel signal at the positions [5, 6] and [5, 8] on the resampling target image 400. Is generated.
The B signal 418 that is the pixel signal at the position [4, 3] on the reduced image 411 is mixed with the B signal that is the pixel signal at the positions [6, 5] and [8, 5] on the resampling target image 400. Is generated.

  Although the calculation method of the pixel signal in the 2 × 2 pixel region has been described, the pixel signals at other pixel positions on the reduced image 411 are also calculated in the same manner. The pixel signal of each pixel on the reduced image 412 is calculated in the same manner. The above-described method for generating a color signal on a reduced image by mixing G signals, R signals, or B signals is also applied to other resampling methods described later.

-Resampling method B _2-
Illustrating the resampling method B _2. In resampling method B _2, a P 2 is the reduction ratio is 1/3. Two reduced images in the resampling method B ₂ are referred to by reference numerals 421 and 422. The positional relationship between the pixels on the resampling target image 400 and the pixels on the first and second reduced images 421 and 422 is the same as that in the resampling method A ₂ (see FIG. 12). Similar to the resampling method A ₂ , the pixel signals at positions [3n _A −2, 3n _B −2] and [3n _A −0.5, 3n _B −0.5] on the resampling target image 400 are Re-sampling is performed so that the pixel signals at positions [n _A , n _B ] on the reduced images 421 and 422 are obtained. However, this resampling is performed so that the color signal arrangement on each reduced image is a Bayer arrangement.

  As described in the first embodiment, when the value x and / or y representing the position [x, y] on the resampling target image is a value including a decimal, the position [x , Y] is generated by interpolation (for example, linear interpolation or bilinear interpolation) using pixel signals of pixels on the resampling target image arranged around the position [x, y]. Of course, the color of the pixel signal used for the interpolation is the same as the color of the pixel signal to be generated by the interpolation.

-Resampling method B _3-
Illustrating the resampling method B _3. In resampling method B _3, P is a 3 the reduced rate is 1/2. Three reduced images in the resampling method B ₃ are referred to by reference numerals 431 to 433. The positional relationship between the pixels on the resampling target image 400 and the pixels on the first to third reduced images 431 to 433 is the same as that in the resampling method A ₃ (see FIG. 13). Similarly to the resampling method A ₃ , the positions [2n _A −1, 2n _B −1], [2n _A , 2n _B ], and [2n _A −0.5, 2n _B −0] on the resampling target image 400. .5] is re-sampled so as to be pixel signals at positions [n _A , n _B ] on the reduced images 431, 432, and 433, respectively. However, this resampling is performed so that the color signal arrangement on each reduced image is a Bayer arrangement.

- resampling method B ₄ -
Describing a resampling method B _4. In resampling method B _4, P is a 3 the reduced rate is 1/2. The three reduced images in resampling method B ₄ referred to by numerals 441 to 443. The positional relationship between the pixels on the resampling target image 400 and the pixels on the first to third reduced images 441 to 443 is the same as that in the resampling method A ₄ (see FIG. 14). Then, similarly to the resampling method A _4, the position in the re-sampled image 400 on _{[2n A -1,2n B -1],} [2n A, 2n B -1] and [2n _A -0.5,2n _B Resampling is performed so that the pixel signal of −0.5] becomes the pixel signal of the position [n _A , n _B ] on the reduced images 441, 442, and 443, respectively. However, this resampling is performed so that the color signal arrangement on each reduced image is a Bayer arrangement.

-Resampling method B _5-
Illustrating the resampling method B _5. In resampling method B _5, the reduction ratio A P 3 is 1/3. Three reduced images in the resampling method B ₅ are referred to by reference numerals 451 to 453. The positional relationship between the pixels on the resampling target image 400 and the pixels on the first to third reduced images 451 to 453 is the same as that in the resampling method A ₅ (see FIG. 15). Then, similarly to the resampling method A _5, the position in the re-sampled image 400 on _{[3n A -2,3n B -2],} [3n A -0.5,3n B -0.5] and [3n _A - Resampling is performed so that the pixel signals of 1.5, 3n _B −1.5] become pixel signals at positions [n _A , n _B ] on the reduced images 451, 452, and 453, respectively. However, this resampling is performed so that the color signal arrangement on each reduced image is a Bayer arrangement.

-Resampling method B _6-
Illustrating the resampling method B _6. In the resampling method B ₆ , P is 3 and the reduction ratio is 1/3. The three reduced images in resampling method B ₆ referred to by numerals 461-463. The positional relationship between the pixels on the resampling target image 400 and the pixels on the first to third reduced images 461 to 463 is the same as that in the resampling method A ₆ (see FIG. 16). Similarly to the resampling method A ₆ , the positions [3n _A −2, 3n _B −2], [3n _A , 3n _B ], and [3n _A −1, 3n _B −1] on the resampling target image 400 are displayed. Resampling is performed so that the pixel signals become pixel signals at positions [n _A , n _B ] on the reduced images 461, 462, and 463, respectively. However, this resampling is performed so that the color signal arrangement on each reduced image is a Bayer arrangement.

[About image processing unit]
Image processing executed by the image processing unit 73 in FIG. 17 is, for example, edge enhancement processing and noise reduction processing. In particular, three-dimensional digital noise reduction processing (hereinafter referred to as 3DDNR), which is a kind of noise reduction processing, may be executed as image processing in the image processing unit 73.

  As is well known, 3DDNR removes noise contained in a plurality of processing target images based on the correlation between the plurality of processing target images. The processing target image in 3DDNR of the image processing unit 73 is a reduced image after demosaicing processing sent from the demosaicing processing unit 72. Hereinafter, for simplification of explanation, the presence of the demosaicing process may be ignored.

In 3DDNR, the noise of each reduced image can be reduced based on P reduced images after capturing P reduced images based on one original image as P processing target images. For example, when the resampling method B ₁ is used, the noise of the reduced images 411 and 412 can be reduced by 3DDNR based only on the two reduced images 411 and 412. However, there is a positional deviation between the two reduced images 411 and 412 based on one original image. Therefore, if 3DDNR is executed based only on the two reduced images 411 and 412, the correlation between the reduced images 411 and 412 is affected by the position shift, and as a result, the noise reduction effect by 3DDNR deteriorates. This also applies to the case of using the resampling method B ₂ .about.B _6.

  In order to prevent this noise reduction effect from deteriorating, 3DDNR may be performed on a plurality of reduced images using the correlation of a plurality of reduced images based on a plurality of original images. Specifically, the processing may be performed as follows.

  When the shutter button 26b is pressed in the shooting mode, as shown in FIG. 19, L original images including the first to Lth original images are acquired by continuous shooting (L is an integer of 2 or more). The resampling processing unit 71 performs the resampling described above for each of the L original images. Then, P reduced images based on the first original image, P reduced images based on the second original image,... P reduced images based on the (L−1) th original image, and P reduced images based on the Lth original image are obtained. That is, a total of (L × P) reduced images are obtained. The (L × P) reduced images include P sets of L reduced images that are not misaligned with each other. The image processing unit 73 executes 3DDNR after capturing L reduced images that have no positional deviation among the obtained (L × P) reduced images as L processing target images. .

For example, when the resampling method B ₁ is used, L first reduced images 411 and L second reduced images 412 are obtained from L original images. The L first reduced images 411 form a set of L reduced images that are not misaligned with each other, and the L second reduced images 412 are L reduced images that are not misaligned with each other. Form a set. The image processing unit 73 can reduce the noise of each reduced image 411 by executing 3DDNR based on the L first reduced images 411, and 3DDNR based on the L second reduced images 412. The noise of each reduced image 412 can be reduced by executing. Pixel data of P reduced images based on one original image among the first to L-th original images (more precisely, the P reduced image after the demosaicing process and 3DDNR are applied) Compressed image data) is recorded in the external memory 18.

  Also in the second embodiment, it is possible to reduce the recording size of an image with almost no deterioration in the image quality of the reproduced image according to the user's request. In addition, since the number of pixels handled in the subsequent stage of the resampling processing unit is reduced as compared with the normal shooting operation in which resampling is not performed as described above, it is possible to reduce power consumption in the shooting operation. In particular, since resampling is performed on the upstream side of the first embodiment, a higher power consumption reduction effect is expected.

  The 3DDNR based on the L processing target images is input to the image processing unit 73 with the Lth processing target image stored in the internal memory 17 in the first to (L-1) th processing target images. It is carried out by doing. For this reason, by making the processing target image a reduced image, the required memory capacity per processing target image is reduced, and as a result, the value of L can be increased compared to the case where the original image or the like is used as the processing target image. Is possible. By increasing the value of L, that is, by increasing the number of images to be processed, the noise reduction effect by 3DDNR can be enhanced.

  In the above example, the image processing unit 73 is provided in the subsequent stage of the demosaicing processing unit 72, but the image processing unit 73 may be provided in the previous stage of the demosaicing processing unit 72.

  Further, as shown in FIG. 20, image data of P reduced images obtained through the resampling processing unit 71, demosaicing processing unit 72, and image processing unit 73 (more precisely, demosaicing processing, 3DDNR, etc.) The image data of P reduced images after the pixel processing is directly input to the super-resolution processing unit 61 together with the positional deviation information, and thereby the high resolution obtained from the super-resolution processing unit 61 The image data of the image may be recorded in the external memory 18 through compression processing by the compression processing unit 16. According to the configuration of FIG. 20, although the effect of reducing the image recording size cannot be obtained, the power consumption reduction effect and the noise reduction effect by 3DDNR can be expected as in the configurations of FIGS. 17 and 6B.

<< Third Example >>
The super-resolution processing shown in the first or second embodiment based on the recorded data in the external memory 18 can also be realized by an electronic device (for example, an image reproduction device; not shown) different from the imaging device 1. It is. In this case, the expansion processing unit 19, the video signal processing unit 20a, and the display unit 27 shown in FIG. 6B may be provided in the electronic device, and the recording data in the external memory 18 may be given to the electronic device. The recording data in the external memory 18 includes the image data and the positional deviation information related to the reduced image shown in the first or second embodiment.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 and 2 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
The imaging apparatus 1 of FIG. 1 or the electronic apparatus according to the third embodiment can be realized by hardware or a combination of hardware and software. In particular, all or part of the processing executed in the video signal processing unit 13 and the display processing unit 20 can be realized using software. Of course, it is also possible to form them only by hardware. When the imaging apparatus 1 or the electronic apparatus according to the third embodiment is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

[Note 2]
For example, it can be considered as follows. A reduced image generating unit that generates P reduced images from one original image can be considered as a part including the resampling unit 53 or 71 (see FIG. 6A, FIG. 17, and FIG. 20). . The operation of recording the image data and the positional deviation information in the external memory 18 is controlled by a recording control unit (not shown). It can be considered that the function of the recording control unit is realized by the CPU 23, the compression processing unit 16, and / or a memory driver (not shown) provided in the preceding stage of the external memory 18. The super-resolution processing unit 61 functions as a high-resolution unit that generates one high-resolution image from a plurality of low-resolution images. The output image generation unit that generates an output image as a high resolution image by inputting P reduced images as P low resolution images into the high resolution image unit includes a super-resolution processing unit 61. It can be considered that the output image generation unit further includes a display video processing unit 62 and the like.

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13, 13a, 13b Video signal processing part 18 External memory 20, 20a Display processing part 33 Imaging element 51, 72 Demosaicing processing part 52, 73 Image processing part 53, 71 Resampling processing part 61 Super solution Image processing unit

Claims (6)

An imaging unit for acquiring an input image composed of M pixels by imaging;
A reduced image generation unit that generates P reduced images each having N pixels from the input image (M, N, and P are integers of 2 or more, and M> N × P);
An image pickup apparatus comprising: a recording control unit that records the P reduced images on a recording medium. A resolution increasing unit that generates a high-resolution image having a resolution higher than the resolution of the low-resolution image from a plurality of low-resolution images, and the P reduced images recorded on the recording medium are converted into the plurality of low-resolution images The imaging apparatus according to claim 1, further comprising an output image generation unit configured to generate an output image as the high resolution image by inputting the image to the high resolution unit as an image. An image processing unit that performs predetermined image processing on the P reduced images generated by the reduced image generation unit;
The imaging apparatus according to claim 1, wherein the recording control unit records P reduced images after the image processing on the recording medium. An image reproducing apparatus for generating an output image from P reduced images recorded on the recording medium according to claim 1,
A high-resolution unit that generates a high-resolution image having a higher resolution than the resolution of the low-resolution image from a plurality of low-resolution images, and the high-resolution unit using the P reduced images as the plurality of low-resolution images An image reproduction apparatus comprising: an output image generation unit configured to generate the output image as the high-resolution image by inputting to the image. An imaging unit for acquiring an input image composed of M pixels by imaging;
A reduced image generation unit that generates P reduced images each having N pixels from the input image (M, N, and P are integers of 2 or more, and M> N × P);
An image processing unit that performs predetermined image processing on the P reduced images;
A resolution increasing unit that generates a high-resolution image having a resolution higher than the resolution of the low-resolution image from a plurality of low-resolution images, and the P reduced images after the image processing are used as the plurality of low-resolution images An output image generation unit that generates an output image as the high resolution image by inputting to the high resolution unit;
An imaging apparatus comprising: a recording control unit that records the output image on a recording medium. The reduced image generation unit generates each reduced image by resampling the input image,
The resampled position on the input image differs between the P reduced images,
The said high resolution part produces | generates the said output image as the said high resolution image using the shift | offset | difference of the position which is resampled among the said P reduced images. The imaging device according to claim 3.

Images