JP2009237650A - Image processor and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the jaggy of an outline section in a high resolution image which is acquired by ultra-resolution processing. <P>SOLUTION: An ultra-resolution section generates one high resolution image from a plurality of low resolution images including position deviation. An arrangement position determination part arranges a plurality of low resolution images by shifting them on a common coordinate face based on position deviation value (lx) between a plurality of low resolution images, and determines the arrangement position (x) of the high resolution image on the common coordinate face. In this case, distances (d<SB>11</SB>, d<SB>21</SB>, d<SB>12</SB>, d<SB>22</SB>or d<SB>22</SB>') between pixels in the high resolution image and pixels in each low resolution image corresponding to those pixels are referenced, and the arrangement positions of the high resolution images are determined based on the variation of the distances. For example, the arrangement positions are determined so that the variation is minimized. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an imaging apparatus that perform image processing. In particular, the present invention relates to a super-resolution processing technique for generating a high-resolution image from a plurality of low-resolution images.

  Super-resolution processing has been proposed as an image processing technique for increasing the resolution of a low-resolution image. A device that performs super-resolution processing refers to a plurality of low-resolution images with positional deviation, and based on the amount of positional deviation between the plurality of low-resolution images and the image data of the plurality of low-resolution images, By performing the conversion, one high-resolution image is generated.

  The part for detecting the amount of misalignment sets one of a plurality of low-resolution images as a standard image and sets the remaining low-resolution image as a reference image. Based on the image data of the standard image and the reference image, The amount of misalignment between the reference image and the reference image is calculated so as to have subpixel resolution.

  A plurality of low resolution images are considered to be arranged on a common coordinate plane, and the arrangement positional relationship between the plurality of low resolution images is determined by the amount of positional deviation. When a plurality of low resolution images are composed of a first low resolution image as a reference image and a second low resolution image as a reference image, the pixels of the first and second low resolution images are arranged on the common coordinate plane as shown in FIG. Arranged as shown in (a). However, for simplification of description, consider a case where each image is a one-dimensional image. In FIG. 18A and FIGS. 18B and 18C described later, each triangle represents a pixel that forms a first low-resolution image, and each square represents a pixel that forms a second low-resolution image. ing. When the position where a certain pixel in the first low resolution image is arranged is considered as the reference position, the distance between the pixel and the pixel in the second low resolution image corresponding to the image is calculated. This matches the positional deviation amount lx.

  An apparatus that performs super-resolution processing arranges each pixel of a higher resolution image on the common coordinate plane. Then, the pixel value of the pixel of the high resolution image is obtained from the pixel value of the pixel of the first and second low resolution images located in the vicinity of the pixel, thereby generating a high resolution image.

  For example, the arrangement position of the high resolution image on the common coordinate plane is determined based on the arrangement position of the reference image. That is, in a certain conventional method (hereinafter referred to as conventional method 1), as shown in FIG. 18B, the pixels of the high-resolution image are arranged based on the same position as the reference position of the reference image (Patent Document). 1). Alternatively, in another conventional method (hereinafter referred to as the conventional method 2), as shown in FIG. 18C, the pixels of the high-resolution image are arranged with reference to the position moved by 1x / 2 from the reference position of the reference image. (See Patent Document 2).

18B and 18C, black circles P _A [0] to P _A [2] and white circles P _B [0] to P _B [2] represent pixels that form a high-resolution image. 18B and 18C illustrate a case where the enlargement ratio of the high resolution image is 2 with respect to the low resolution image, and the positional deviation amount lx is 1/8 of the adjacent pixel interval of the low resolution image. It corresponds to. In relation to the position of the pixel of the low-resolution image, the pixel group including the pixels P _A [0] to P _A [2] is considered as the first type pixel group, and the pixels P _B [0] to P _B [ 2] is considered a second type of pixel group.

JP 2005-109822 A JP 2006-245679 A

When the conventional method 1 is used, a distance between a pixel represented by a black circle (for example, P _A [0]) and a pixel of a low-resolution image located closest to the pixel is small, but a pixel represented by a white circle ( For example, the distance between P _B [0]) and the pixel of the low-resolution image located closest to the pixel increases. The same applies when the conventional method 2 is used.

  Therefore, when the conventional method 1 or 2 is used in this example, the high frequency component included in the pixel group belonging to the first type and the high frequency component included in the pixel group belonging to the second type Variation occurs between them. This variation causes jaggy (a jagged stepped outline) in the outline of the high resolution image. Adverse effects on image quality, as represented by jaggy, should be eliminated as much as possible.

  Accordingly, an object of the present invention is to provide an image processing apparatus and an imaging apparatus that can generate a high-resolution high-resolution image from a plurality of low-resolution images.

  An image processing apparatus according to the present invention is an image processing apparatus that generates a single high-resolution image from a plurality of low-resolution images that include misalignment with each other. Determining the arrangement positions of the plurality of low-resolution images on the common coordinate plane based on the amount of positional deviation so that the plurality of low-resolution images are shifted on the common coordinate plane; Arrangement position determining means for determining an arrangement position of the high resolution image on a plane, wherein the arrangement position determining means is a distance between a pixel in the high resolution image and a pixel in each low resolution image corresponding to the pixel. , And the arrangement position of the high-resolution image on the common coordinate plane is determined based on the variation in the distance.

  This is expected to reduce jaggies in the contour portion of the high resolution image.

  FIG. 19 shows an example of the arrangement position of each image determined in consideration of distance variation when two low-resolution images are used.

  Specifically, for example, the distance includes a distance between the position of the pixel of each low-resolution image and the position of the pixel of interest that is arranged in the nearest vicinity of the pixel of interest of the high-resolution image.

  More specifically, for example, the pixels of the high-resolution image are classified into m types (m is an integer of 2 or more), and the first to m-th target pixels belonging to the first to m-th types are set, The arrangement position determining means refers to the distance to each of the first to m-th target pixels, and determines the arrangement position of the high-resolution image on the common coordinate plane based on variations in all the reference distances. decide.

  More specifically, for example, the plurality of low resolution images include first and second low resolution images, and the distance to the jth pixel of interest includes first and second distances (j is 1 or more m The first and second distances with respect to the j-th pixel of interest are respectively arranged in the nearest to the j-th pixel of interest and the j-th pixel of interest. 2 is a distance from a pixel in the low-resolution image, and the arrangement position determining means is not only a variation in the distance but also a magnitude of a difference between the first and second distances for each pixel of interest. Based on this, an arrangement position of the high-resolution image on the common coordinate plane is determined.

  Further, for example, the arrangement position determining means determines an arrangement position of the high resolution image on the common coordinate plane so that the variation in the distance is minimized.

  Instead of this, for example, the arrangement position determining means obtains the magnitude of the difference for each pixel of interest, derives the sum of the obtained magnitudes of the differences, and includes the variation in the distance and the sum The arrangement position of the high resolution image on the common coordinate plane is determined so that the value of the evaluation function is minimized.

  Further, for example, the plurality of low-resolution images are three or more low-resolution images, including first to third low-resolution images, and the positional deviation amount derived by the positional deviation amount deriving unit is: A displacement amount between the first and second low-resolution images and a displacement amount between the first and third low-resolution images, and the arrangement position determining means includes the pixels in the high-resolution image and the pixels thereof. A plurality of distances derived for the first to third low-resolution images, comprising distance deriving means for deriving a distance from a pixel in the corresponding i-th low resolution image (i is 1, 2 or 3); The arrangement position of the high resolution image on the common coordinate plane is determined based on the evaluation value according to the variation of the difference, and the contribution degree of each distance to the evaluation value is changed according to the reliability of each displacement amount. Let

  Thereby, it is possible to relatively reduce the contribution of the distance with low detection accuracy that contributes to the determination of the arrangement position of the high-resolution image. As a result, optimization of determination of the arrangement position is further promoted.

  In addition, for example, the image processing apparatus has a sharpness of the high-resolution image according to a variation in the distance when the high-resolution image is arranged at the arrangement position of the high-resolution image determined by the arrangement position determining unit. Sharpness adjusting means is further provided.

  This also reduces jaggy.

  An imaging apparatus according to the present invention includes an imaging unit that acquires a plurality of captured images arranged in time series by sequential imaging, and a plurality of low-resolution images that include positional deviations between the plurality of captured images acquired by the imaging unit. And the image processing apparatus that generates one high-resolution image from the plurality of low-resolution images.

  According to the present invention, it is possible to provide an image processing apparatus and an imaging apparatus that can generate a high-quality high-resolution image from a plurality of low-resolution images.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to fourth embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital video camera, for example. The imaging device 1 can shoot moving images and still images, and can also shoot still images simultaneously during moving image shooting. Note that the moving image shooting function may be omitted, and the imaging apparatus 1 may be a digital still camera capable of shooting only a still image.

[Description of basic configuration]
The imaging device 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a video signal processing unit 13, a microphone 14, an audio signal processing unit 15, a compression processing unit 16, and an SDRAM (Synchronous Dynamic Random Access Memory). ), An external memory 18 such as an SD (Secure Digital) card or a magnetic disk, a decompression processing unit 19, a VRAM (Video Random Access Memory) 20, an audio output circuit 21, and a TG (timing generator). ) 22, a CPU (Central Processing Unit) 23, a bus 24, a bus 25, an operation unit 26, a display unit 27, and a speaker 28. The operation unit 26 includes a recording button 26a, a shutter button 26b, an operation key 26c, and the like. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25.

  The TG 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and provides the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each unit in the imaging apparatus 1. The operation unit 26 receives an operation by a user. The operation content given to the operation unit 26 is transmitted to the CPU 23. Each unit in the imaging apparatus 1 temporarily records various data (digital signals) in the internal memory 17 during signal processing as necessary.

  The imaging unit 11 includes an optical system, an aperture, and a driver (not shown in FIG. 1) in addition to the imaging element (image sensor) 33. Incident light from the subject enters the image sensor 33 via the optical system and the stop. Each lens constituting the optical system forms an optical image of the subject on the image sensor 33. The TG 22 generates a drive pulse for driving the image sensor 33 in synchronization with the timing control signal, and applies the drive pulse to the image sensor 33.

  The image sensor 33 is composed of, for example, a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 33 photoelectrically converts an optical image incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels arranged two-dimensionally in a matrix, and in each photographing, each light receiving pixel stores a signal charge having a charge amount corresponding to the exposure time. The electrical signal from each light receiving pixel having a magnitude proportional to the amount of the stored signal charge is sequentially output to the subsequent AFE 12 in accordance with the drive pulse from the TG 22.

  The AFE 12 amplifies the analog signal output from the image sensor 33, converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The degree of amplification of signal amplification in the AFE 12 is controlled by the CPU 23. The video signal processing unit 13 performs various types of image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The video signal is composed of a luminance signal Y representing the luminance of the image and color difference signals U and V representing the color of the image.

  The microphone 14 converts the peripheral sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 using a predetermined compression method. The compressed video signal is recorded in the external memory 18 at the time of shooting and recording a moving image or a still image. The compression processing unit 16 compresses the audio signal from the audio signal processing unit 15 using a predetermined compression method. At the time of moving image shooting and recording, the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 are compressed while being correlated with each other in time by the compression processing unit 16, and those after compression are externally transmitted. Recorded in the memory 18.

  The recording button 26a is a push button switch for instructing the start / end of moving image shooting and recording, and the shutter button 26b is a push button switch for instructing shooting and recording of a still image.

  The operation modes of the imaging apparatus 1 include a shooting mode in which moving images and still images can be shot, and a playback mode in which moving images and still images stored in the external memory 18 are played back and displayed on the display unit 27. Transition between the modes is performed according to the operation on the operation key 26c. In the shooting mode, shooting is sequentially performed at a predetermined frame period, and an image sequence arranged in time series is acquired from the image sensor 33. Each image forming this image sequence is called a “frame image”.

  When the user presses the recording button 26a in the shooting mode, under the control of the CPU 23, the video signal of each frame image and the corresponding audio signal obtained after the pressing are sequentially sent via the compression processing unit 16 to the external memory 18. To be recorded. When the user presses the recording button 26a again after starting the moving image shooting, the recording of the video signal and the audio signal in the external memory 18 is finished, and the shooting of one moving image is completed. In the shooting mode, when the user presses the shutter button 26b, a still image is shot and recorded.

  When the user performs a predetermined operation on the operation key 26c in the reproduction mode, a compressed video signal representing a moving image or a still image recorded in the external memory 18 is expanded by the expansion processing unit 19 and then written to the VRAM 20. It is. In the shooting mode, normally, video signals are generated sequentially by the video signal processing 13 regardless of the operation contents of the recording button 26a and the shutter button 26b, and the video signals are sequentially written in the VRAM 20.

  The display unit 27 is a display device such as a liquid crystal display, and displays an image corresponding to the video signal written in the VRAM 20. In addition, when a moving image is reproduced in the reproduction mode, a compressed audio signal corresponding to the moving image recorded in the external memory 18 is also sent to the expansion processing unit 19. The decompression processing unit 19 decompresses the received audio signal and sends it to the audio output circuit 21. The audio output circuit 21 converts a given digital audio signal into an audio signal in a format that can be output by the speaker 28 (for example, an analog audio signal) and outputs the audio signal to the speaker 28. The speaker 28 outputs the sound signal from the sound output circuit 21 to the outside as sound (sound).

  The video signal processing unit 13 is configured to be able to perform super-resolution processing in cooperation with the CPU 23. Through the super-resolution processing, one high-resolution image is generated from a plurality of low-resolution images. The video signal of the high resolution image can be recorded in the external memory 18 via the compression processing unit 16. The resolution of the high resolution image is higher than that of the low resolution image, and the number of pixels in the horizontal and vertical directions of the high resolution image is larger than that of the low resolution image. For example, when an instruction to capture a still image is given, a plurality of frame images as a plurality of low resolution images are acquired, and a high resolution image is generated by performing super-resolution processing on them. Alternatively, for example, the super-resolution processing is performed on a plurality of frame images as a plurality of low resolution images obtained at the time of moving image shooting.

  The basic concept of super-resolution processing will be briefly described. As an example, a super-resolution process using a reconstruction method will be described. FIG. 2 is a conceptual diagram of super-resolution processing using a MAP (Maximum A Posterior) method, which is a type of reconstruction method. In this super-resolution processing, one high-resolution image is estimated from a plurality of low-resolution images obtained by actual shooting, and the original plurality of low-resolution images are estimated by degrading the estimated high-resolution image. To do. A low resolution image obtained by actual photographing is particularly called an “observation low resolution image”, and an estimated low resolution image is particularly called an “estimated low resolution image”. Thereafter, the high resolution image and the low resolution image are repeatedly estimated so that the error between the observed low resolution image and the estimated low resolution image is minimized, and the finally acquired high resolution image is output.

  FIG. 3 is a flowchart showing the flow of super-resolution processing. First, in step S11, an initial high resolution image is generated from the observed low resolution image. In the subsequent step S12, the original observed low resolution image for constructing the current high resolution image is estimated. The estimated image is referred to as an estimated low resolution image as described above. In the subsequent step S13, an update amount for the current high resolution image is derived based on the difference image between the observed low resolution image and the estimated low resolution image. This update amount is derived so that an error between the observed low-resolution image and the estimated low-resolution image is minimized by repeatedly executing the processes in steps S12 to S14. In the subsequent step S14, the current high resolution image is updated using the updated amount, and a new high resolution image is generated. Thereafter, the process returns to step S12, and the newly generated high resolution image is regarded as the current high resolution image, and the processes of steps S12 to S14 are repeatedly executed. Basically, as the number of repetitions of each process of steps S12 to S14 increases, the resolution of the obtained high-resolution image is substantially improved, and an ideal high-resolution image is obtained.

  Super-resolution processing based on the above-described operation flow is performed in the imaging apparatus 1. The super-resolution processing performed in the imaging apparatus 1 may be any type of super-resolution processing, but in the present embodiment, a description will be given of a case where a reconfiguration type super-resolution processing is adopted. To do. The reconfiguration method includes an ML (Maximum-Likelihood) method, a MAP (Maximum A Posterior) method, a POCS (Projection Onto Convex Set) method, an IBP (Iterative Back Projection) method, and the like. In this embodiment, the MAP method is used. Take the configuration using as an example.

  As examples relating to the super-resolution processing, first to fourth examples will be described below. The matters described in one embodiment (particularly the first embodiment) can be applied to other embodiments as long as there is no contradiction.

<< First Example >>
First, the first embodiment will be described. FIG. 4 is an internal block diagram of the super-resolution unit 50 responsible for the super-resolution processing. 1 are provided in the video signal processing unit 13 of FIG. 1, and are each referred to as a low-resolution image frame memory 61 and a high-resolution image frame memory 62 (hereinafter simply referred to as frame memories 61 and 62). ) Is provided in the internal memory 17 of FIG. The function of the reference image setting unit 51 is realized by the CPU 23. However, the function of the reference image setting unit 51 may be realized by the video signal processing unit 13. The frame memory 61 is a memory for storing image data of a frame image as an observed low resolution image, and the frame memory 62 is a memory for storing image data of a high resolution image.

K _NUM frames images photographed continuously by the imaging section 11 is taken as the super-resolution unit 50 k _NUM pieces of observed low resolution images (k _NUM is an integer of two or more). Image data of k _NUM observed low-resolution images is stored in the frame memory 61, and each part in the super-resolution unit 50 reads the image data from the frame memory 61 as necessary.

The reference image setting unit 51 sets one of the k _NUM observation low-resolution images as a reference image, and sets the remaining (k _NUM −1) observation low-resolution images as reference images. The reference image can be read as a reference frame or a reference frame image, and the reference image can be read as a reference frame or a reference frame image. For example, out of k _NUM frame images, a frame image captured first, a frame image captured last, or a frame image captured at an intermediate time point is set as the reference image. Alternatively, based on the image data of each observed low resolution image, the size of the blur (hereinafter referred to as blur amount) included in each observed low resolution image is estimated, and the observed low resolution image with the least amount of blur is set as the reference image. You may make it do. For example, the amount (intensity) of a predetermined high frequency component in the observed low resolution image is calculated, and the amount is estimated as the blur amount of the observed low resolution image.

The misregistration amount detection unit 52 calculates the misregistration amount between two observation low-resolution images using a representative point matching method, a block matching method, a gradient method, or the like. The amount of displacement calculated here has a so-called sub-pixel resolution having a resolution higher than the pixel interval of the observed low-resolution image. That is, the amount of positional deviation is calculated with a distance shorter than the interval between two adjacent pixels in the observed low resolution image as the minimum unit. The positional deviation amount is a two-dimensional amount including a horizontal component and a vertical component, and is also called a motion amount or a motion vector. The interval between two adjacent pixels in the observed low resolution image is represented by pp _L.

  The misregistration detection unit 52 calculates the misregistration amount between the standard image and each reference image using the standard image as a standard. A known calculation method can be used as a method for calculating a positional deviation amount having sub-pixel resolution. For example, the method described in Japanese Patent Application Laid-Open No. 11-345315 and the method described in “Okutomi,“ Digital Image Processing ”, Second Edition, CG-ARTS Association, issued on March 1, 2007” (p. 31). 205).

k _NUM low resolution images that are k _NUM observed low resolution images or k _NUM estimated low resolution images, and one high resolution image including an initial high resolution image (such as H ₀ and H ₁ described below) ) Are formed from a plurality of pixels arranged in a two-dimensional grid, and in the super-resolution unit 50, these images are arranged on a two-dimensional common coordinate plane (hereinafter referred to as a common coordinate plane). Can be considered. _{Assume that} k _NUM low-resolution images include a first low-resolution image as a base image and a second low-resolution image as a reference image.

  FIG. 5 shows a grid and pixels of each image arranged on the common coordinate plane. In FIG. 5, a triangle denoted by reference numeral 201 represents a pixel of the first low-resolution image and represents a position where the pixel is arranged, and a square denoted by reference numeral 202 represents a pixel of the second low-resolution image. And a circle with a reference numeral 203 represents a pixel of a high-resolution image and a position where the pixel is arranged. In FIG. 5, reference numeral 201 is given only to a part of the triangle group representing the pixel group forming the first low-resolution image. The same applies to reference numerals 202 and 203.

Although there may be rotational misalignment between the first and second low-resolution images, there is only translational misalignment between them in order to prevent complication of illustration and simplify the description. Think about the case. It is assumed that the pixel interval in the horizontal direction and the vertical direction is the same in one low-resolution image, and the pixel interval is common to all the low-resolution images. Further, it is assumed that the pixel interval in the horizontal direction and the vertical direction is the same in one high-resolution image. In the example of FIG. 5, the pixel interval in the high resolution image is ½ that of the low resolution image. Further, consider a case where the horizontal component and the vertical component of the positional deviation amount between the first and second low resolution images are both less than the pixel interval pp _L of the low resolution image.

  In addition, it is assumed that the horizontal direction and the vertical direction of the common coordinate plane coincide with the X direction and the Y direction, respectively. In addition, the horizontal direction and the vertical direction of the common coordinate plane match the horizontal direction and the vertical direction of the low resolution image, respectively, and also match the horizontal direction and the vertical direction of the high resolution image.

  The triangles 201 are arranged in a grid at equal intervals in the horizontal and vertical directions of the common coordinate plane. A grid in which the triangles 201 are arranged (a grid indicated by a broken line in FIG. 5), that is, a grid in which the pixel group forming the first low-resolution image is arranged on the common coordinate plane is referred to as a first low-resolution grid. The rectangles 202 are arranged in a lattice pattern at equal intervals in the horizontal direction and the vertical direction of the common coordinate plane. A grid in which the rectangles 202 are arranged (a grid indicated by a one-dot chain line in FIG. 5), that is, a grid in which the pixel group forming the second low-resolution image is arranged on the common coordinate plane is referred to as a second low-resolution grid. . The circles 203 are arranged in a lattice pattern at equal intervals in the horizontal and vertical directions of the common coordinate plane. A grid in which the circles 203 are arranged (a grid indicated by a solid line in FIG. 5), that is, a grid in which a group of pixels forming a high resolution image is arranged on a common coordinate plane is called a high resolution grid.

  The symbol O represents the origin on the common coordinate plane. The point where the pixel closest to the origin O in the pixel group forming the first low-resolution image is called the reference point of the first low-resolution image, and the position of the reference point on the common coordinate plane is This is called the reference position of the first low-resolution image. In the example of FIG. 5, the reference point coincides with the origin O. The point where the pixel closest to the origin O in the pixel group forming the second low-resolution image is called the reference point of the second low-resolution image, and the position of the reference point on the common coordinate plane is This is called the reference position of the second low-resolution image. The point where the pixel closest to the origin O in the pixel group forming the high resolution image is called the reference point of the high resolution image, and the position of the reference point on the common coordinate plane is the reference of the high resolution image. Called position.

  Position displacement (position displacement on the common coordinate plane) between the reference position of the first low-resolution image and the reference position of the second low-resolution image, as viewed from the reference point of the first low-resolution image, is detected as displacement. This is the amount of positional deviation between both images calculated by the unit 52.

  The arrangement position determination unit 53 sets the first and second low-resolution grids on the common coordinate plane based on the positional deviation amount. In other words, the second low-resolution image is shifted and arranged on the common coordinate plane with reference to the arrangement position of the first low-resolution image based on the positional deviation amount. When the number of low-resolution images is three or more and the third low-resolution image exists, the position between the first low-resolution image that is the base image and the third low-resolution image that is the reference image Based on the amount of deviation, the third low-resolution image is shifted on the common coordinate plane with the arrangement position of the first low-resolution image as a reference (the same applies to the fourth low-resolution image).

  Furthermore, the arrangement position determination unit 53 determines the arrangement position of the high resolution grid on the common coordinate plane (that is, the arrangement position of the high resolution image) based on the arrangement position of each low resolution image on the common coordinate plane. Specifically, values of x and y, which will be described later, representing the arrangement position of the high resolution image are determined. This determination method will be described in detail later.

According to the arrangement positions of k _NUM low resolution images and one high resolution image determined by the arrangement position determination unit 53, an initial high resolution image generation unit 54 (hereinafter abbreviated as generation unit 54), a selection unit 55, and Super-resolution processing by the super-resolution processing unit 56 is executed.

The generation unit 54 uses the pixel interpolation to generate the initial high resolution image H ₀ using the observed low resolution image as the reference image. This generation process corresponds to the process in step S11 of FIG. The initial high resolution image H ₀ corresponds to an initial image of a high resolution image to be finally generated. Hereinafter, the initial high resolution image H ₀ may be simply referred to as a high resolution image H ₀ or an image H ₀ . For example, referring to the observed low-resolution image as the standard image, linear interpolation or bicubic interpolation is used to calculate the high-resolution grid from the pixel values of the pixels of the standard image located at the grid points of the first low-resolution grid. The pixel value of the pixel of the image H ₀ located at the grid point is calculated. The pixel value of a certain pixel is a digital value including information representing the luminance and color of the pixel. Note that the pixel values of the pixels of the image H ₀ may be calculated using the pixel values of both the standard image and the reference image. The generated image data of the image H ₀ is stored in the frame memory 62.

The selection unit 55 selects one of the image generated by the generation unit 54 (that is, the image H ₀ ) and the image stored in the frame memory 62, and performs super-resolution processing on the image data of the selected image. Part 56 is given. Immediately after the image H ₀ is generated, the image H ₀ is selected by the selection unit 55, and after a high resolution image different from the image H ₀ is generated by the super-resolution processing unit 56, it is stored in the frame memory 62. The selected image is selected by the selection unit 55.

Further, the image data of k _NUM observation low-resolution images is input from the frame memory 61 to the super-resolution processing unit 56. Based on the MAP method, the super-resolution processing unit 56 uses the image H ₀ , k _NUM observed low-resolution images, and the positional shift amount calculated by the positional shift detection unit 52, and generates an estimated low-resolution image. Thus, the update amount for the image H ₀ is obtained. This process corresponds to the first process of steps S12 and S13 (see FIG. 3). Then, the image H ₀ is updated with the update amount to generate a high-resolution image H ₁ (hereinafter sometimes abbreviated as image H ₁ ). This corresponds to the first step S14 (see FIG. 3). Note that after the image H ₁ is generated, only the lines indicated by bold lines in FIG. 4 function significantly.

The generated image data of the image H ₁ is overwritten and stored in the frame memory 62 and is input again to the super-resolution processing unit 56 via the selection unit 55. Super-resolution processing unit 56, by updating the image H ₁ from the image H ₀ in the same manner as that generated the image H ₁ high-resolution image H ₂ (hereinafter, sometimes abbreviated as image H ₂ ) Is generated. The image data of the image H ₂ is overwritten and stored in the frame memory 62. The process of generating the image H ₂ from the image H ₁ corresponds to the second process of steps S12 to S14.

As described above, the super-resolution processing unit 56 repeatedly executes the arithmetic processing including the processes of steps S12 to S14 for updating the high-resolution image and generating a new high-resolution image. Hereinafter, this calculation processing is referred to as super-resolution calculation processing. When n is a natural number, a high resolution image H _n is generated from the high resolution image H _{n −1} by the n-th super-resolution calculation process. In the following description, n is an integer of 0 or more.

FIG. 6 shows an internal block diagram of the super-resolution processing unit 56 when the number of observed low-resolution images is 3 (that is, when k _NUM = 3). Assume that three observation low-resolution images are composed of first, second, and third observation low-resolution images L ₁ , L _2, and L ₃ . Now, represents a high-resolution image H _n which is a matrix expressed by h ^n, representative of the matrix representation observed low resolution images L _k in l _k. That is, for example, the matrix h ⁿ is a line in which pixel values of pixels forming the high resolution image H _n are written. k takes a value of 1, 2 or 3.

Then, matrix h ^{n + 1} with respect to the high-resolution image H _{n + 1} is derived according to the following formula (A-1), high-resolution image H _{n + 1} are produced by this derivation. The second term on the right side of Expression (A-1) represents the update amount for the high-resolution image H _n that should be calculated by the super-resolution processing unit 56. By updating the high resolution image based on the equation (A-1), the evaluation function I in the MAP method represented by the following equation (A-2) is minimized. In Expression (A-2), h represents a high-resolution image expressed in a matrix when super-resolution calculation processing is performed a certain number of times.

The product of the matrix W _k and the matrix h ⁿ represents an estimated low resolution image as an estimated image of the observed low resolution image L _k . W _k is a matrix for generating an estimated image of the observed low resolution image L _k from the high resolution image H _{n, the} amount of misregistration calculated by the misregistration detection unit 52, and from the high resolution image to the low resolution image. It is an image conversion matrix including a point spread function representing image blur caused by the reduction in resolution and downsampling from a high resolution image to a low resolution image. (L _k −W _k h ⁿ ) represents a difference image between the observed low resolution image L _k and the estimated low resolution image corresponding to the estimated image. Note that the matrix with the subscript T represents the transposed matrix of the original matrix. Thus, for example, W _k ^T represents a transposed matrix of the matrix W _k .

  C is a matrix for regularization, and α is a regularization parameter. The matrix C is set based on prior knowledge that “a high-resolution image has few high-frequency components”, for example, and is formed by a Laplacian filter expressed in a matrix. Β is a parameter for controlling the feedback amount.

  When the number of iterations of the super-resolution calculation process in the super-resolution processing unit 56 reaches a specified number, the latest high-resolution image obtained by the specified number of super-resolution calculation processes is the high resolution that should be finally obtained. The image is output from the super-resolution processing unit 56 as an image. If it is determined that the update amount for the latest high-resolution image is sufficiently small and the update amount has converged regardless of the number of times of super-resolution calculation processing, the latest high-resolution image is finally obtained. The super-resolution processing unit 56 may output the high-resolution image to be obtained.

[Method for determining the location of the high-resolution grid]
A method for determining the arrangement position of the high resolution grid (that is, the arrangement position of the high resolution image) by the arrangement position determination unit 53 will be described in detail. In the following description, the pixel interval, distance, position, and the like refer to the pixel interval, distance, position, and the like on the common coordinate plane unless otherwise specified. Further, the interval pp _L between adjacent pixels in the low resolution image is considered as a unit (see FIG. 5), and the distance represented by pp _L on the common coordinate plane is 1.0.

First, consider a case where k _NUM = 2 and the first and second low-resolution images and high-resolution images are one-dimensional images in which a plurality of pixels are arranged only in the X direction in FIG. In this case, of course, the positional deviation amount is also a one-dimensional amount having only the X direction component. As described above, it is assumed that the first low-resolution image is a standard image and the second low-resolution image is a reference image. Then, the decimal point part of the positional deviation amount between the standard image and the reference image with reference to the standard image is represented by lx. For simplification of description, consider a case where the integer part of the positional deviation amount is zero. It is assumed that lx has a positive value when the X axis along the X direction is defined and the reference image is shifted in the positive direction of the X axis with reference to the base image.

FIG. 7 shows the pixel arrangement of the first and second low-resolution images. However, it is assumed that the inequality “0 ≦ lx ≦ 0.5” holds. Triangles L ₁ [0], L ₁ [1], L ₁ [2]... Represent consecutive pixels of the first low-resolution image, and squares L ₂ [0], L ₂ [1], L ₂ [2]... Represent consecutive pixels of the second low-resolution image. The pixel L ₁ [0] is located at the reference point of the first low-resolution image, and with reference to this reference point, the pixels L ₁ [0], L ₁ [1], L ₁ [2],... Are arranged in this order. The pixel L ₂ [0] is located at the reference point of the second low-resolution image, and with reference to this reference point, the pixels L ₂ [0], L ₂ [1], L ₂ [2],... Are arranged in this order. FIG. 8A shows the pixel arrangement of the first and second low-resolution images when lx is 1/4 of the pixel interval pp _L , that is, when lx = 0.25.

  When the actually calculated lx satisfies the inequality “0.5 <lx <1.0”, the arrangement position may be reversed horizontally. That is, for example, when lx = 0.75, the pixel arrangement of the first and second low-resolution images is actually represented as shown in FIG. 8B, but the positive and negative directions of the X axis are reversed. If the pixel arrangement as shown in FIG. 8C is assumed, it can be considered in the same manner as when the inequality “0 ≦ lx ≦ 0.5” is satisfied.

  Now, the reference position of the high resolution image (that is, the position of the reference point of the high resolution image) is represented by x. Further, it is assumed that the enlargement ratio of the high resolution image with respect to the low resolution image is 2. Then, the distance between adjacent pixels of the high resolution image is 0.5, and the arrangement position determination unit 53 determines the reference position x so as to satisfy the inequality “0 ≦ x <0.5”.

FIG. 9A shows a diagram in which the pixels of the high resolution image are superimposed on the pixels of the first and second low resolution images shown in FIG. In FIG. 9A, black circles H _A [0] to H _A [2] and white circles H _B [0] to H _B [2] are both pixels of a high resolution image. Each pixel forming the high resolution image is classified into a plurality of types in relation to the position of the pixel of the low resolution image. When considered in one dimension and the enlargement ratio is 2, the high-resolution image is formed from a pixel group belonging to the first type and a pixel group belonging to the second type. The former pixel group includes pixels H _A [0] to H _A [2], and the latter pixel group includes pixels H _B [0] to H _B [2]. When u is an integer greater than or equal to 0, the pixel H _A [u] is a pixel arranged between the pixel L ₁ [u] and the pixel L ₂ [u], and the pixel H _B [u] is the pixel L ₂ [u] is a pixel arranged between the pixel L ₁ [u + 1]. The pixel H _A [0] is located at a reference point of the high-resolution image, and the pixels H _A [0], H _B [0], H _A [1] are moved toward the positive direction of the X axis with reference to this reference point. ], H _B [1], H _A [2], H _B [2]... Are arranged in this order.

The arrangement position determination unit 53 calculates the distance between the pixel of the high resolution image and the pixels of the first and second low resolution images located in the vicinity of the pixel for each type. The neighborhood here includes at least the nearest neighborhood. In this example, only the pixels of the first and second low resolution images located in the nearest vicinity of the pixels of the high resolution image are considered. Therefore, the arrangement position determination unit 53 follows the following formulas (B-1) to (B-4), and the four distances d ₁₁ (x), d ₂₁ (x), d ₁₂ (x), and d ₂₂ (x) Is calculated. d ₁₁ (x), d ₂₁ (x) and d ₁₂ (x) are distances d ₁₁ , d ₂₁ and d ₁₂ shown in FIG. 9B, respectively, and d ₂₂ (x) is shown in FIG. D ₂₂ or d ₂₂ ′ shown in FIG.

The arrangement position determination unit 53 calculates the sum of absolute differences D (x) between different distances expressed by the following formula (B-5) and the variation in distance expressed by the following formula (B-6). Based on the index σ (x) representing the degree, the optimum arrangement position of the high-resolution image is determined. An index representing the degree of variation in distance is hereinafter referred to as variation amount. Here, the values of s _j (x) and ave (x) in Expression (B-6) are expressed by Expressions (B-7) and (B-8).

The sum D (x) becomes 0 when the reference position x of the high resolution image is lx / 2. When the sum D (x) is large, a feedback amount from the first low-resolution image (in other words, a pixel group that includes one type of pixel in the high-resolution image (for example, a pixel group including the pixel H _A [0]) (in other words, In this case, the contribution rate is increased while the feedback amount from the second low-resolution image is decreased. Conversely, for a pixel group belonging to the other type in the high-resolution image (for example, a pixel group including the pixel H _B [0]), the feedback amount from the first low-resolution image is reduced while the second amount is reduced. The amount of feedback from the low-resolution image increases. In this case, when there is a difference in the high-frequency component contained between the first and second low-resolution images, the high-frequency component varies between the two pixel groups, and the contour portion of the high-resolution image Can cause jaggy (a jagged stepped contour). Therefore, it is desirable to make the sum D (x) as small as possible. In the super-resolution calculation process, the high-resolution image is updated based on the update amount as described above, and the feedback amount from each low-resolution image is reflected in the update amount.

For example, if the reference position x of the high resolution image is 0.1 in the situation shown in FIG. 8A, the first type of pixel group including the pixels H _A [0] to H _A [2]. On the other hand, the feedback amount from the first low-resolution image increases. The distance between the pixel H _A [0] and the pixel L ₁ [0] is 0.1, while the distance between the pixel H _A [0] and the pixel L ₂ [0] is 0.15, This is because the former distance (0.1) corresponding to the first low-resolution image is shorter than the latter distance. Conversely, the feedback amount from the second low-resolution image is large for the second type of pixel group including the pixels H _B [0] to H _B [2]. The distance between the pixel H _B [0] and the pixel L ₁ [1] is 0.4 (= | 0.6-1 |), while the pixel H _B [0] and the pixel L ₂ [0] The distance between the two is 0.35 (= 0.6−0.25), and the latter distance (0.35) corresponding to the second low-resolution image is shorter than the former distance. . In such a case, when the amount of the high frequency component included in the first low-resolution image is larger than that of the second low-resolution image, the first high-feedback amount from the first low-resolution image is large. While the high frequency component in the type of pixel group becomes large, the high frequency component in the second type of pixel group having a large feedback amount from the second low resolution image becomes small. This variation in the high frequency components causes jaggy in the contour portion of the generated high resolution image.

  Also, when the distance variation amount σ (x) is large, the variation in feedback amount between different pixels of the high resolution image becomes large, and a certain pixel group in the high resolution image contains a lot of high frequency components. There may be a phenomenon that the high frequency component contained in the pixel group is small. As described above, since the variation of the high frequency component causes jaggy in the contour portion of the high resolution image, it is desirable to reduce the distance variation amount σ (x) as much as possible.

  In FIG. 10, a broken line 301 represents the positional deviation amount lx dependency of the reference position x when “σ (x) = 0”, and a broken line 302 represents the reference when “D (x) = 0”. This represents the dependency of the position x on the amount of displacement lx. In the graph of FIG. 10, the horizontal axis represents the positional deviation amount lx, and the vertical axis represents the reference position x. From FIG. 10, it can be seen that σ (x) and D (x) are both zero only when the positional deviation amount lx is 0 or 0.5. That is, in other cases, it is desirable to determine the reference position x in consideration of the balance between σ (x) and D (x).

  In FIG. 11, a broken line 311 represents the positional deviation amount lx dependence of the sum D (x) when “σ (x) = 0”, and a broken line 312 represents when “D (x) = 0”. Represents the positional deviation amount lx dependence of the distance variation amount σ (x). In the graph of FIG. 11, the horizontal axis represents the positional deviation amount lx, and the vertical axis represents D (x) or σ (x). From FIG. 11, when the positional deviation amount lx is up to about 0.2, the value of σ (x) when “D (x) = 0” is relatively large, and the positional deviation amount lx becomes larger than 0.2. In this case, it can be seen that the value of D (x) when “σ (x) = 0” is relatively large.

  Therefore, in consideration of the balance between σ (x) and D (x), for example, the reference position x of the high-resolution image may be determined according to the following formula (B-9). Here, TH is a preset threshold value that satisfies the inequality “0 <TH <0.5”. In this case, when lx is 0 or more and less than TH, the reference position x is determined so that σ (x) is minimized (in this example, σ (x) = 0), and lx is TH. When the value is 0.5 or less, the reference position x is determined so that D (x) is minimized (in this example, D (x) = 0).

  A broken line 321 in FIG. 12 shows the dependency of the reference position x to be determined on the positional deviation amount lx when TH = 0.25.

When the above-described concept is extended two-dimensionally, that is, as shown in FIG. 5, when the first and second low-resolution images and high-resolution images are two-dimensional images, the arrangement position determining unit 53 determines the amount of positional deviation. The reference position (x, y) of the high-resolution image with respect to (lx, ly) is determined according to the equation (C-5) based on the following equations (C-1) to (C-4). However, it is assumed that k _NUM = 2, and as described above, the first low-resolution image is the standard image and the second low-resolution image is the reference image.

  The matters described above for the X direction apply to each value for the Y direction as well. That is, lx and ly are an X-direction component and a Y-direction component in the decimal point part of the positional deviation amount between the base image and the reference image, respectively. Assume that the inequalities “0 ≦ lx ≦ 0.5” and “0 ≦ ly ≦ 0.5” hold. x and y are an X direction component and a Y direction component of the reference position of the high resolution image (that is, the position of the reference point of the high resolution image), respectively. The enlargement ratio of the high resolution image with respect to the low resolution image is 2 in the horizontal direction and the vertical direction. Then, the distance between adjacent pixels of the high resolution image is 0.5, and the arrangement position determination unit 53 satisfies the inequalities “0 ≦ x <0.5” and “0 ≦ y <0.5”. Determine (x, y).

  As described above, when the magnification ratio is 2 when considered in one dimension, the pixels that form a high-resolution image are classified into two types of pixels. Classification occurs. That is, the same pixel classification is performed for each of the X direction and the Y direction, and as a result, each pixel forming the high resolution image is classified into one of the first to fourth types. Of the pixels belonging to the j-th type in the high-resolution image, the pixel closest to the origin O is referred to as the j-th pixel of interest for convenience (where j is 1, 2, 3, or 4). The first to fourth target pixels are the four pixels in the high-resolution image located in the upper left corner of FIG.

d _1j (x, y) is a distance between the j-th pixel of interest and a pixel in the first low-resolution image that is arranged in the vicinity of the j-th pixel of interest. d _2j (x, y) is the distance between the j-th pixel of interest and the pixel in the second low-resolution image that is arranged in the vicinity of the j-th pixel of interest.

  σ (x, y) represents the variation amount of the total distance obtained for the first to fourth target pixels, and D (x, y) is the first to fourth target pixels. It represents the sum of the absolute values of the distance differences that are obtained. As understood from the equation (C-5), the arrangement position determining unit 53 determines that σ (x, y) or D (x, y) is minimized according to the values of lx and ly. And the value of y is determined.

  FIG. 13A shows a high resolution image obtained from two observed low resolution images when the method for determining the arrangement position of the high resolution image shown in this embodiment is used. Each observation low-resolution image is assumed to have a plurality of linear edges extending in the vertical direction, and a positional shift amount (lx, ly) between one observation low-resolution image and the other observation low-resolution image. ) Is (0.5, 0). In order to contrast with this, FIGS. 13B and 13C show high-resolution images obtained by a method different from the method shown in the present embodiment. The high-resolution image in FIG. 13B is obtained when the reference position of the high-resolution image is set to be the same as the reference position of one of the observed low-resolution images, which corresponds to the conventional method 1 (FIG. 18). (See (b)). The high-resolution image in FIG. 13C is obtained when the reference position of the high-resolution image is set to an intermediate position between the reference positions of the two observed low-resolution images, which corresponds to the above-described conventional method 2 ( (See FIG. 18C).

  As can be seen from FIGS. 13A to 13C, if the method for determining the arrangement position of the high resolution image shown in the present embodiment is used, the variation of the high frequency components included in the high resolution image is reduced. Jaggy in the contour portion (step edge portion) is reduced.

Further, more general calculation formulas for D (x, y) and σ (x, y) are shown in formulas (D-1) to (D-4). The k _NUM low resolution images are formed from the first to k _NUM low resolution images.

Here, m is an enlargement ratio (horizontal and vertical enlargement ratio) of the high resolution image with respect to the low resolution image. Let m be an integer. If and magnification considered in two dimensions is m, each pixel forming a high-resolution image are classified as either type of first to m ^2. Of the pixels belonging to the jth type in the high-resolution image, the pixel closest to the origin O is referred to as the jth pixel of interest for convenience (where j is an integer between 1 and m ² ). Then, d _ij (x, y) is a distance between the j-th target pixel and a pixel in the i-th low-resolution image that is arranged in the nearest vicinity of the j-th target pixel (where, i is an integer from 1 to k _NUM ). d _kj (x, y) is a distance between the j-th pixel of interest and a pixel in the k-th low-resolution image arranged closest to the j-th pixel of interest (where k is (I + 1) or more and k _NUM or less integer).

Formula (D-2) in accordance sigma (x, y) becomes a represent the variation amount of the total distance to be determined for the first to m ² of the pixel of interest, according to the formula (D-1) D (x , y) becomes represent the sum of the first to be determined for the m ² of the pixel of interest, the distance difference absolute value. The arrangement position determination unit 53 minimizes D (x, y) or σ (x, y) calculated based on the expressions (D-1) to (D-4) according to the values of lx and ly. As described above, the reference position (x, y) of the high-resolution image with respect to the positional deviation amount (lx, ly) is determined.

<< Second Example >>
Next, a second embodiment will be described. The second embodiment corresponds to an embodiment obtained by modifying a part of the first embodiment. The following description will be made by paying attention to differences from the first embodiment. For matters not specifically mentioned, the description of the first embodiment is applied.

First, for simplification of description, k _NUM = 2 and the first and second low-resolution images and high-resolution images are one-dimensional images in which a plurality of pixels are arranged only in the X direction in FIG. Consider a case. As described above, it is assumed that the first low-resolution image is a standard image, the second low-resolution image is a reference image, and the enlargement ratio of the high-resolution image with respect to the low-resolution image is 2. In addition, the matters described in the first embodiment, which should be taken into consideration when considering in one dimension, are also applied to the second embodiment.

  In the first embodiment, the reference position x of the high-resolution image is determined so that one of D (x) and σ (x) is minimized according to the positional deviation amount lx. In the second embodiment, , “E (x) = D (x) + gσ (x)” is introduced, and the reference position x is determined so that the evaluation function E (x) is minimized. . Here, g is a predetermined coefficient.

  When D (x) represented by the above formula (B-5) is expanded, the following formula (E-1) is obtained, and when σ (x) represented by the above formula (B-6) is expanded, the following formula ( E-2) is obtained. When the evaluation function E (x) is expanded using the equations (E-1) and (E-2), the evaluation function E (x) is expressed by the following equation (E-3).

  In FIG. 14, when g = 2, the positional deviation amount lx dependency of the reference position x for minimizing the evaluation function E (x) and the positional deviation amount lx dependence of the minimum value of the evaluation function E (x) are shown. Showing gender. A polygonal line 351 represents the former dependency, and a polygonal line 352 represents the latter dependency. In the graph of FIG. 14, the horizontal axis represents the positional deviation amount lx. The vertical axis for the broken line 351 represents the reference position x, and the vertical axis for the broken line 352 represents the minimum value of the evaluation function E (x).

  As shown in FIG. 14, the formula for calculating the reference position x for minimizing the evaluation function E (x) changes when the positional deviation amount lx is 0.25. Specifically, the reference position x in this numerical example is calculated according to the following formula (E-4). Formula (E-4) is a mathematical expression of the polygonal line 351. The equation (E-4) is the same as the equation (B-9) for calculating the reference position x described in the first embodiment. That is, when g = 2, the reference position x that minimizes the evaluation function E (x) and the reference position x determined by the method described in the first embodiment are the same as a result (g If ≠ 2, they can be different).

In the case of extending the above idea to two dimensions, the arrangement position determining unit 53, as in the first embodiment, determines D (x, y) and σ (x, x, y) according to the above formulas (C-1) to (C-4). y) is calculated. However, it is assumed that k _NUM = 2 and the enlargement ratio of the high resolution image with respect to the low resolution image is 2. In addition, the matters described in the first embodiment, which should be taken into consideration when considering in two dimensions, are also applied to the second embodiment. Then, the misregistration amount (lx, ly) is such that the evaluation function E (x, y) represented by “E (x, y) = D (x, y) + gσ (x, y)” is minimized. ) To determine the reference position (x, y) of the high resolution image. As in the case of one dimension, if g = 2, the reference position (x, y) is determined according to the above equation (C-5).

  Also in the second embodiment, as in the first embodiment, the high-resolution image is taken into consideration in consideration of the balance between the sum D (x, y) of the distance difference absolute value and the distance variation amount σ (x, y). Since the arrangement position is determined, variation in high frequency components included in the high resolution image is reduced, and jaggy in the contour portion (step edge portion) is reduced.

  Note that the method of defining the distance variation amount can be modified. When considered in one dimension, instead of σ (x), for example, σ ′ (x) represented by the following formula (E-5) may be regarded as the amount of variation in distance. In this case, an evaluation function E ′ (x) represented by “E ′ (x) = D (x) + g′σ ′ (x)” is introduced, and the evaluation function E ′ (x) is minimized. In this way, the reference position x may be determined. The ave (x) in the formula (E-5) is represented by the formula (E-6) (note that the formula (E-6) is a modified version of the formula (B-8)). Expressions (E-7a) and (E-7b) are expressions expressing the evaluation function E ′ (x) using x and lx. When lx <x, the value of the evaluation function E ′ (x) is calculated according to the equation (E-7a). When lx ≧ x, the value of the evaluation function E ′ (x) is calculated according to the equation (E-7b). Is done. g 'is a predetermined coefficient.

  FIG. 15 shows the dependence of the reference position x on the positional deviation amount lx for minimizing the evaluation function E ′ (x) and the positional deviation of the minimum value of the evaluation function E ′ (x) when g ′ = 1. The quantity lx dependence is shown. A polygonal line 361 represents the former dependency, and a polygonal line 362 represents the latter dependency. In the graph of FIG. 15, the horizontal axis represents the positional deviation amount lx. The vertical axis for the broken line 361 represents the reference position x, and the vertical axis for the broken line 362 represents the minimum value of the evaluation function E ′ (x).

  When considering two-dimensional expansion, σ (x) and E (x), which are one-dimensional functions, are expanded in the same manner as when two-dimensional functions, σ (x, y) and E (x, y), are expanded. Thus, the functions σ ′ (x, y) and E ′ (x, y) obtained by extending the functions σ ′ (x) and E ′ (x) to two dimensions are obtained, and E ′ (x, y) is minimized. The reference position (x, y) may be determined so that

<< Third Example >>
Next, a third embodiment will be described. In the first and second embodiments, the sum (D (x) or D (x, y)) of the distance difference absolute values is obtained by simple addition of a plurality of distances. In the third embodiment, the distance difference is calculated. The sum of absolute values is obtained by weighted addition of multiple distances. The value of the weighting coefficient in the weighted addition is determined based on the reliability of the positional deviation amount calculated by the positional deviation amount detection unit 52 in FIG.

  In the third embodiment, the misregistration amount detection unit 52 calculates the misregistration amount and evaluates the reliability of the misregistration amount. Various methods can be used to evaluate the reliability of the misregistration amount. The amount of misalignment between the reference image and the reference image is obtained by comparing the image in the detection block set in the reference image with the image in the detection block set in the reference image. Based on the flatness of the image in the detection block in the reference image, the reliability of the positional deviation amount is obtained. The flatness of the focused image means the flatness of the image. The flatter the image, the higher the flatness and the lower the reliability. For example, the flatness of the image is estimated from the amount of the high frequency component contained in the image in the detection block in the reference image (the flatness is estimated to be larger as the amount of the high frequency component is smaller).

  Alternatively, the reliability of the positional deviation amount may be evaluated using any known method. For example, the reliability of the misregistration amount may be evaluated by using a method described in Japanese Patent Application Laid-Open No. 2003-078807 or a method described in Japanese Patent Application Laid-Open No. 2002-016836. The reliability of the motion vector in those publications corresponds to the reliability of the positional deviation amount in this embodiment.

_Assume that k _NUM = 3, and the first observed low resolution image is a reference image and the second and third observed low resolution images are reference images. Then, the reliability of the evaluated misregistration amount is considered as a numerical value. The positional deviation amount of confidence between the first and second observation low-resolution image represented by R _2, represents an amount of positional deviation reliability between the first and third observation low-resolution images at R _3. For convenience, the reliability R ₁ is also introduced. The reliability R _i satisfies the inequality “0 ≦ R _i ≦ 1”, and R ₁ is always 1 (where i is 1, 2 or 3).

  For simplification of description, attention is paid only to one dimension in the X direction. Further, it is assumed that the enlargement ratio of the high resolution image with respect to the low resolution image is 2. Further, the decimal point part of the positional deviation amount between the first and second observed low resolution images and the decimal point part of the positional deviation amount between the first and third observed low resolution images are 0 on the common coordinate plane. It is assumed that it is 0.5 or less. In addition, the matters described in the first embodiment, which should be taken into consideration when considering in one dimension, are also applied to the third embodiment.

  FIG. 16 shows the pixel arrangement of the first to third low-resolution images and high-resolution images. In FIG. 16, the rhombus with a blank inside represents a pixel forming the third low-resolution image.

In the third embodiment, the arrangement position determination unit 53 is expressed by the following formula (F-2), which is expressed by the following formula (F-1), and the sum D _A (x) of the difference absolute values between different distances. The distance variation amount σ _A (x) is obtained. Here, the values of s _j (x) and ave (x) in Formula (F-2) are represented by Formulas (F-3) and (F-4). The distance d ₃₁ (x) is the distance between the pixel H _A [0] and the pixel in the third low-resolution image arranged closest to the pixel H _A [0], and the distance d ₃₂ (x ) Is the distance between the pixel H _B [0] and the pixel in the third low-resolution image arranged closest to the pixel H _B [0].

Then, the arrangement position determination unit 53 uses D _A (x) and σ _A (x) as D (x) and σ (x) in the first example, and as described in the first example, D D The reference position x of the high-resolution image is obtained so that _A (x) or σ _A (x) is minimized. Alternatively, the arrangement position determination unit 53 uses D _A (x) and σ _A (x) as D (x) and σ (x) in the second embodiment, and “E (x) = D _A (x) + gσ The value of the evaluation function E (x) is calculated according to “ _A (x)”, and the reference position x is determined so that the evaluation function E (x) is minimized.

For example, when R ₁ = 1.0, R ₂ = 1.0 and R ₃ = 0.5, the accuracy of the distance d _3j (x) is lower than that of the distance d _2j (x). Considering this, by using the equations (F-1) to (F-4), D _A (x) and σ _A (of the distance d _3j (x) obtained for the third low-resolution image are obtained. x) Make the degree of influence smaller than that of other distances. Thereby, the contribution degree of the distance with low detection accuracy that contributes to the determination of the arrangement position of the high-resolution image is relatively reduced, and the optimization of the determination is further promoted.

  Although the processing for the one-dimensional image has been described in detail, it is needless to say that the processing can be extended to the processing for the two-dimensional image as described in the first or second embodiment.

<< 4th Example >>
Next, a fourth embodiment will be described. The fourth embodiment is implemented in combination with any one of the first to third embodiments. In the fourth embodiment, the sharpness of the high-resolution image generated by the super-resolution processing unit 56 in FIG. 4 is determined according to the amount of variation in the distance after the arrangement position of the high-resolution image is determined according to the above-described method. Adjust. Hereinafter, (x, y) representing the arrangement position of the high-resolution image determined by the arrangement position determining unit 53 according to the method described in any of the first to third embodiments is expressed as (x ₀ , y ₀ ). It describes.

The super-resolution processing unit 56 or the arrangement position determination unit 53 calculates the amount of variation in distance when (x, y) = (x ₀ , y ₀ ), that is, σ (x ₀ , y ₀ ). Then, the super-resolution processing unit 56 adjusts the sharpness of the high-resolution image generated by itself according to the distance variation amount σ (x ₀ , y ₀ ). Specifically, the sharpness is relatively low when σ (x ₀ , y ₀ ) is relatively large, and the sharpness is relatively high when σ (x ₀ , y ₀ ) is relatively small. In an image with high sharpness, the pixel value change near the edge is steep, the amount of high frequency components contained in the image is relatively large, and in an image with low sharpness, the pixel value change near the edge is moderate. The amount of high frequency components contained in the image is relatively small.

When the MAP method is adopted and the high-resolution image is updated according to the equation (A-1), the sharpness is adjusted by adjusting the value of the regularization parameter α. When the matrix C in the formula (A-1) is formed by a Laplacian filter or the like, if the value of the regularization parameter α is increased, the sharpness of the high-resolution image generated by the super-resolution processing unit 56 is decreased. . Therefore, when σ (x ₀ , y ₀ ) is relatively large, the value of the regularization parameter α is relatively large. This blurs the high-resolution image, but reduces the jaggies of the high-resolution image resulting from the large σ (x ₀ , y ₀ ). On the other hand, when σ (x ₀ , y ₀ ) is relatively small, the value of the regularization parameter α is relatively small to increase the sharpness, thereby substantially improving the resolution.

For example, as shown in FIG. 17, when σ (x ₀ , y ₀ ) ≦ 0.2, the value of α is determined according to α = 0.5 × σ (x ₀ , y ₀ ), and σ (x ₀ , Y ₀ )> 0.2, α is fixed at 0.1.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 4 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the above-described embodiment, the super-resolution processing using the MAP method, which is a kind of reconstruction method, is exemplified as the super-resolution processing. However, what kind of super-resolution processing can be used in the present invention? Super-resolution processing may be used. In the above-described embodiment, after the initial high-resolution image is generated, the super-resolution calculation process including the calculation of the update amount and the update of the high-resolution image based on the update amount is repeatedly performed. The repetition of is not essential, and the high-resolution image H ₁ obtained by performing the super-resolution calculation process only once can be handled as a high-resolution image to be finally obtained.

[Note 2]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, part or all of the arithmetic processing executed in the super-resolution unit 50 in FIG. 4 can be realized using software. Of course, the super-resolution unit 50 can be formed only by hardware. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

[Note 3]
The function of the super-resolution unit 50 can be realized by an external device (for example, a personal computer; not shown) different from the imaging device 1. In this case, a super-resolution unit equivalent to the super-resolution unit 50 is provided in the external device, and after a plurality of low-resolution images are acquired by the imaging device 1, the image data of the plurality of low-resolution images is obtained. What is necessary is just to supply to the said external apparatus by a radio | wireless or a wire or a recording medium.

[Note 4]
For example, it can be considered as follows. The super-resolution unit 50 in FIG. 4 can also be called an image processing apparatus. It can be considered that the arrangement position determination unit 53 includes a distance deriving unit for deriving the above-described distance (d ₁₂ (x, y), etc.). The sharpness adjusting function described in the fourth embodiment is realized by the sharpness adjusting means included in the super-resolution processing unit 56.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is a conceptual diagram of the super-resolution process using a MAP system. It is a flowchart showing the flow of the super-resolution process which concerns on embodiment of this invention. It is an internal block diagram of the super-resolution part which bears the super-resolution process based on 1st Example of this invention. It is a figure which shows the grid | lattice and pixel of each image which are related to 1st Example of this invention and are arrange | positioned on the common coordinate plane. FIG. 5 is an internal block diagram of the super-resolution processing unit in FIG. 4 when the number of observed low-resolution images is three according to the first embodiment of the present invention. It is a figure which concerns on 1st Example of this invention and shows the pixel arrangement | positioning of the 1st and 2nd low resolution image. It is a figure which shows the specific example of the pixel arrangement | positioning of the 1st and 2nd low resolution image concerning 1st Example of this invention. It is a figure which shows pixel arrangement | positioning of the 1st and 2nd low resolution image and a high resolution image concerning 1st Example of this invention. According to the first embodiment of the present invention, when the distance variation (σ (x)) is zero, the positional deviation amount (lx) dependence of the reference position (x) of the high resolution image and the absolute difference value of the distance 6 is a graph showing the positional deviation amount (lx) dependence of the reference position (x) of the high resolution image when the sum (D (x)) is zero. According to the first embodiment of the present invention, when the distance variation (σ (x)) is zero, the sum of absolute differences of distances (D (x)) depends on the amount of positional deviation (lx), and It is a graph which shows the positional deviation amount (lx) dependence of the variation amount (σ (x)) of the distance when the sum is zero. 6 is a graph showing the positional deviation amount (lx) dependence of a reference position (x) of a high resolution image, which should be determined under certain conditions according to the first embodiment of the present invention. It is a figure showing the high resolution image (a) obtained using the method which concerns on 1st Example of this invention, and the high resolution image (b) and (c) obtained using another method. According to the second embodiment of the present invention, the positional deviation amount (lx) dependence of the reference position (x) of the high-resolution image for minimizing the evaluation function and the positional deviation amount (lx) of the minimum value of the evaluation function It is a graph which shows dependency. According to the second embodiment of the present invention, the positional deviation amount (lx) dependence of the reference position (x) of the high-resolution image for minimizing another evaluation function and the positional deviation of the minimum value of the other evaluation function It is a graph which shows quantity (lx) dependence. It is a figure which concerns on 3rd Example of this invention and shows the pixel arrangement | positioning of the 1st-3rd low resolution image and high resolution image. It is a figure for demonstrating the determination method of the value of the regularization parameter in MAP system concerning 4th Example of this invention. FIG. 4A is a diagram illustrating a pixel arrangement of first and second low-resolution images, and FIGS. 2B and 2C are diagrams illustrating pixel arrangements of first and second low-resolution images and high-resolution images. ). It is a figure which shows the example of arrangement | positioning position of several low resolution image and high resolution image concerning this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13 Video signal processing part 50 Super-resolution part 51 Reference image setting part 52 Position shift detection part 53 Arrangement position determination part 54 Initial high-resolution image generation part 55 Selection part 56 Super-resolution processing part

Claims (9)

In an image processing apparatus that generates one high-resolution image from a plurality of low-resolution images that include misalignment with each other,
A positional deviation amount deriving means for deriving a positional deviation amount between the plurality of low resolution images;
An arrangement position of the plurality of low resolution images on the common coordinate plane is determined on the basis of the positional deviation amount so that the plurality of low resolution images are shifted on a common coordinate plane, and the height on the common coordinate plane is determined. An arrangement position determining means for determining an arrangement position of the resolution image,
The arrangement position determining means refers to a distance between a pixel in the high-resolution image and a pixel in each low-resolution image corresponding to the pixel, and based on the variation in the distance, the high-resolution image on the common coordinate plane An image processing apparatus for determining an arrangement position of the image processing apparatus. The said distance contains the distance of the position of the pixel of each low-resolution image arrange | positioned in the nearest vicinity of the pixel of interest of the said high resolution image, and the position of the said pixel of interest. Image processing device. The pixels of the high-resolution image are classified into m types (m is an integer of 2 or more), and the first to m-th target pixels belonging to the first to m-th types are set,
The arrangement position determining means includes
With reference to the distance to each of the first to mth pixels of interest,
The image processing apparatus according to claim 2, wherein an arrangement position of the high-resolution image on the common coordinate plane is determined based on variations in all distances to be referred to. The plurality of low resolution images includes first and second low resolution images;
The distance to the jth pixel of interest includes first and second distances (j is an integer of 1 to m),
The first and second distances with respect to the j-th pixel of interest are first and second low-resolution images arranged in the nearest vicinity of the j-th pixel of interest and the j-th pixel of interest, respectively. Is the distance to the pixel at
The arrangement position determining means is configured to determine the high-resolution image on the common coordinate plane based not only on the variation in the distance but also on the size of the difference between the first and second distances for each pixel of interest. The image processing apparatus according to claim 3, wherein an arrangement position is determined. The image processing apparatus according to claim 1, wherein the arrangement position determination unit determines an arrangement position of the high-resolution image on the common coordinate plane so that variations in the distance are minimized. The arrangement position determining means obtains the magnitude of the difference for each pixel of interest, derives the sum of the obtained magnitudes of the differences, and minimizes the value of the evaluation function including the distance variation and the sum The image processing apparatus according to claim 4, wherein an arrangement position of the high resolution image on the common coordinate plane is determined so as to be realized. The plurality of low-resolution images are three or more low-resolution images, including first to third low-resolution images,
The misregistration amount derived by the misregistration amount deriving means includes a misregistration amount between the first and second low resolution images and a misregistration amount between the first and third low resolution images,
The arrangement position determining means includes
A distance deriving unit for deriving a distance between a pixel in the high-resolution image and a pixel in the i-th low-resolution image corresponding to the pixel (i is 1, 2 or 3);
Determining an arrangement position of the high-resolution image on the common coordinate plane based on an evaluation value according to a plurality of distance variations derived for the first to third low-resolution images;
The image processing apparatus according to claim 1, wherein the degree of contribution of each distance to the evaluation value is changed according to the reliability of each displacement amount. Sharpness adjusting means for adjusting the sharpness of the high resolution image according to the variation in the distance when the high resolution image is arranged at the arrangement position of the high resolution image determined by the arrangement position determining means. The image processing apparatus according to claim 1, further comprising an image processing apparatus. Imaging means for acquiring a plurality of photographed images arranged in time series by sequential photographing;
The plurality of captured images acquired by the imaging unit are regarded as a plurality of low resolution images including positional deviations, and one high resolution image is generated from the plurality of low resolution images. An image processing apparatus comprising: the image processing apparatus according to any one of the above.