JP2013021509A - Image processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image of higher resolution and definition than those of an image subjected to electronic zooming.SOLUTION: A lens 101 and an imaging device 102 capture the image data of a subject. A camera signal processing unit 105 generates the zoom image data by magnifying a part of an image represented by the image data output from the imaging device 102 according to the zoom magnification information being input. An imaging model database 108 stores the deterioration information of an image due to the optical system of the lens 101, and the deterioration information of an image due to the magnification, as the imaging model information. A file generation unit 112 generates a data file storing the information indicating on/off of magnification, the zoom magnification information, the imaging model information, the image data, and the zoom image data.

Description

  The present invention relates to an image processing apparatus and method using super-resolution processing.

  A technique for improving the resolution of an image or video by restoring signals lost when the image or video is acquired and increasing the pixels of the image or video itself, so-called super-resolution technology, is becoming widespread. As a super-resolution technique, a method based on MAP (maximum a posterior) estimation is famous (for example, Non-Patent Document 1).

  The MAP method is a method for estimating a high-resolution image that minimizes an evaluation function obtained by adding probability information of a high-resolution image to a square error. In other words, this is a super-resolution processing method that estimates a high-resolution image as an optimization problem that maximizes the posterior probability by using some foresight information for the high-resolution image. Specifically, using an imaging model that models the degradation process of the imaging device, the inverse problem of estimating a high resolution image from a low resolution image is solved by optimization processing, and a super-resolution image is obtained by calculation.

  On the other hand, there is an electronic zoom technique as an electronic image enlargement method that can be executed by the imaging apparatus during image acquisition. Electronic zoom is a technique for cutting out a portion of an image acquired by the imaging device and enlarging the portion when the zoom magnification of the imaging device at the time of image acquisition exceeds the limit of optical zoom (for example, Patent Document 1). .

  The technique disclosed in Non-Patent Document 1 is an inverse problem of estimating a high-resolution image from a low-resolution image using an imaging model, and a good super-resolution image cannot be obtained without an appropriate imaging model. The imaging model is a deterioration process that takes into account the optical characteristics of the imaging apparatus, and is information unique to the imaging apparatus. In other words, in order to obtain a good super-resolution image, it is necessary to hold an imaging model unique to the imaging apparatus together with the image data when recording the image data.

  In order to obtain a resolution higher than that of an electronic zoomed image, it is conceivable to perform super-resolution processing on the electronic zoomed image to electronically increase the resolution of the image. However, even if super-resolution processing considering the imaging model of the imaging device is performed on the image enlarged by electronic zoom, image degradation has occurred in the process of electronic zoom processing, and it is expected that sufficient resolution improvement will be expected Can not.

JP 2009-038627

Sung C. P., Min K. P. "Super-Resolution Image Reconstruction: A Technical Overview" IEEE Signal Proc. Magazine, Vol. 26, No. 3, pp. 21-36, 2003

  It is an object of the present invention to provide an image with higher resolution and higher definition than an image that has been subjected to electronic zoom processing.

  The present invention has the following configuration as one means for achieving the above object.

  An image processing apparatus according to the present invention includes an image capturing unit that captures image data of a subject, and zoom image data obtained by enlarging a part of an image represented by the image data output by the image capturing unit in accordance with input zoom magnification information. Generation means for generating; storage means for storing image degradation information by the optical system of the imaging means; and image degradation information by the enlargement process as imaging model information; information indicating on / off of the enlargement process; the zoom And creating means for creating a data file storing the magnification information, the imaging model information, the image data, and the zoom image data.

  According to the present invention, it is possible to provide a higher resolution and higher definition image than an image that has been subjected to electronic zoom processing.

1 is a block diagram illustrating a configuration example of an imaging apparatus according to Embodiment 1. FIG. The block diagram explaining the structural example of a camera signal processing part. The block diagram explaining the structural example of a file preparation part. The figure explaining the structural example of the data file which stores a moving image. FIG. 9 is a block diagram illustrating a structure example of a display device. The conceptual diagram explaining reproduction | regeneration of the electronic zoom image using a super-resolution process. The flowchart explaining the image processing of a camera signal processing part. The flowchart explaining the file creation process of a file creation part. 7 is a flowchart for explaining moving image reproduction processing of the display device. The block diagram explaining the structural example of a super-resolution image production | generation part. 6A and 6B illustrate a deterioration model of an imaging device. The figure explaining the size of the matrices B and D. The figure explaining the relationship between a two-dimensional image and the matrix X. FIG. The figure explaining the structural example of the matrix B. FIG. The figure explaining the structural example of the matrix D. FIG. The flowchart explaining the process example of a super-resolution image generation part.

  Hereinafter, an electronic zoom image generation process by super-resolution processing using an imaging model, which is image processing according to an embodiment of the present invention, will be described in detail with reference to the drawings.

[Device configuration]
A configuration example of the imaging apparatus 113 according to the first embodiment will be described with reference to the block diagram of FIG.

  Light from the subject forms an image on the imaging device 102 via the lens 101. An analog signal output from the imaging device 102 is converted into a digital signal (imaging data) by an analog-digital converter (A / D) 103 and stored in the buffer memory 104.

  The camera signal processing unit 105 performs development processing including demosaicing, white balance adjustment, gamma correction, and the like on the imaging data read from the buffer memory 104 to generate, for example, 8-bit image data for each color of RGB. The camera signal processing unit 105 realizes an electronic zoom function, which will be described in detail later.

  Although details will be described later, the file creation unit 112 inputs image data from the camera signal processing unit 105 and creates a data file. Then, the created data file is output to a display device 106 such as an LCD monitor and a storage device 107 such as a memory card or a hard disk drive (HDD). The imaging model DB 108 stores imaging model information indicating lens information of the lens 101 and image degradation information related to the device information of the imaging device 102.

  The zoom control unit 109 controls the zoom function of the imaging device 113. The zoom control unit 109 inputs zoom magnification information, which is a user instruction, via a user interface such as a zoom button (not shown). If the zoom magnification information is within the range of the optical zoom function, control information including the zoom magnification information is supplied to the lens 101 via the terminal 110 of the switch. When the zoom magnification information reaches the limit of the optical zoom function, the zoom control unit 109 supplies the electronic zoom magnification information to the camera signal processing unit 105 via the terminal 111 of the switch.

  The camera signal processing unit 105 determines whether the electronic zoom is on or off based on the electronic zoom magnification information input from the zoom control unit 109. When the electronic zoom is on, the electronic zoom function is activated to output electronic zoom image data that has been electronically zoomed according to the electronic zoom magnification information.

  In FIG. 1, the imaging device 113 and the display device 106 are connected via, for example, a video interface and a video cable, and the imaging device 113 and the storage device 107 are connected via, for example, a serial bus such as USB or a card interface. An example was given. However, at least one of the display device 106 and the storage device 107 may be configured integrally with the imaging device 113.

  In addition, the file creation unit 112 can read a data file from the storage device 107, output the read data file to the display device 106, and reproduce a moving image. Of course, when the display device 106 has an interface for the storage device 107, the display device 106 can reproduce the moving image recorded in the data file read directly from the storage device 107.

Camera Signal Processing Unit A configuration example of the camera signal processing unit 105 will be described with reference to the block diagram of FIG.

  The input terminal 201 inputs electronic zoom magnification information from the zoom control unit 109. The electronic zoom determination unit 203 determines whether the electronic zoom is on or off based on the electronic zoom magnification information, and outputs electronic zoom on / off information. When the electronic zoom on / off information input from the electronic zoom determination unit 203 indicates that the electronic zoom is on, the electronic zoom magnification acquisition unit 205 stores the electronic zoom magnification information in the electronic zoom magnification memory 212.

  The input terminal 202 inputs imaging data from the buffer memory 104. The image acquisition unit 204 generates image data by performing development processing on the imaging data, and stores the generated image data in the image memory 208.

  The electronic zoom image generation unit 206 generates electronic zoom image data corresponding to the electronic zoom magnification information stored in the electronic zoom magnification memory 212 from a part of the image data stored in the image memory 208. The generated electronic zoom image data is stored in the electronic zoom image memory 207.

  The electronic zoom on / off information is output from the output terminal 213. The electronic zoom magnification information stored in the electronic zoom magnification memory 212 is output from the output terminal 211. The electronic zoom image data stored in the electronic zoom image memory 207 is output from the output terminal 209. The image data stored in the image memory 208 is output from the output terminal 210. When the electronic zoom is off, the electronic zoom magnification memory 212 and the electronic zoom image memory 207 are empty.

File creation unit A configuration example of the file creation unit 112 will be described with reference to the block diagram of FIG.

  The input terminal 311 inputs electronic zoom on / off information from the camera signal processing unit 105 (output terminal 213). The electronic zoom on / off information is supplied to the data file generation unit 309.

  The input terminal 301 inputs electronic zoom magnification information from the camera signal processing unit 105 (output terminal 211). When significant electronic zoom magnification information is input, the electronic zoom magnification acquisition unit 305 supplies the electronic zoom magnification information to the data file generation unit 309.

  The input terminal 302 inputs electronic zoom image data from the camera signal processing unit 105 (output terminal 209). When significant electronic zoom image data is input, the electronic zoom image acquisition unit 306 supplies the electronic zoom image data to the data file generation unit 309.

  The input terminal 303 inputs image data from the camera signal processing unit 105 (output terminal 210). The image acquisition unit 307 supplies the input image data to the data file generation unit 309.

  The input terminal 304 inputs imaging model information from the imaging model database 108. The imaging model acquisition unit 308 supplies the input imaging model information to the data file generation unit 309.

  The data file generation unit 309 generates a data file in which electronic zoom on / off information, electronic zoom magnification information, electronic zoom image data, image data, and imaging model information are collected, and outputs the generated data file from the output terminal 310. Although the configuration of the data file will be described later, when the electronic zoom is off, null is stored in the storage area of the electronic zoom magnification information and the image data of the data file.

Data file structure An example of the data file structure for storing moving images will be described with reference to FIG.

  The data file format is, for example, a new area in which electronic zoom on / off information, electronic zoom magnification information, image data, imaging model information, and the like are provided in a file format for storing existing moving image data. Data files are roughly divided into a tag information unit 501, a moving image information unit 502, and an image information unit 503.

  The tag information unit 501 stores attribute information. For example, the attribute information 1 stores the imaging lens characteristic value of the imaging model information, the attribute information 2 stores the imaging device characteristic value of the imaging model information, the attribute information 3 stores the electronic zoom magnification information, and the attribute information 4 stores Stores electronic zoom on / off information. The electronic zoom magnification information is given as a numerical value, and the electronic zoom on / off information is a flag indicating “1” for on and “0” for off, and stores information for the number of frames of moving image data.

  The moving image information unit 502 is a storage area in which moving image data used for normal reproduction, that is, moving image data is stored regardless of optical zoom or electronic zoom. The image information unit 503 is an area for storing image data necessary for performing super-resolution processing. That is, image data of a frame (frame) for which the electronic zoom is on is stored in the image information unit 503 together with information indicating the frame.

  If the data file is supplied to a normal moving image playback device, the moving image can be played back using the moving image data stored in the moving image information unit 502.

Display Device A configuration example of the display device 106 will be described with reference to the block diagram of FIG.

  The input terminal 401 inputs a data file from the file creation unit 112 (output terminal 310 thereof). The file acquisition unit 402 separates tag information, moving image data, and image data from the data file. The tag information acquisition unit 403 separates the electronic zoom on / off information, the electronic zoom magnification information, and the imaging model information from the separated tag information.

  The super-resolution image generation unit 409 generates super-resolution image data from the imaging model information, the electronic zoom magnification information, and the image data for the frame whose electronic zoom on / off information is “1”.

  The moving image reconstruction unit 410 outputs the moving image data input from the file acquisition unit 402 to the display unit 411 when the electronic zoom on / off information is “0”. When the electronic zoom on / off information is “1”, super-resolution image data input from the super-resolution image generation unit 409 is output to the display unit 411. That is, the moving image reconstruction unit 410 performs a replacement operation for replacing an electronic zoom image of a frame with electronic zoom on with a super-resolution image.

  The display unit 411 displays the moving image data input from the moving image reconstruction unit 410 on, for example, an LCD monitor.

  The reproduction of an electronic zoom image using super-resolution processing will be described with reference to the conceptual diagram of FIG. In the upper part of FIG. 6, reference numerals 601 to 606 correspond to the frames recorded by the imaging device 113, and the zoom magnification is shown below each frame. An electronic zoom image is recorded in the frames 604-606 having the electronic zoom magnification of × 2 (double) in the frames 601-606. Reference numeral 607-609 corresponds to the original image of the electronic zoom image, and the central portion (hatched portion) of the image 607-609 is an image region used for generating the electronic zoom image.

  The lower part of FIG. 6 shows the reproduction of a moving image, and the recorded images are reproduced on the frames 601 to 603 where the electronic zoom is off. On the other hand, in the frames 604-606 in which the electronic zoom is off, the electronic zoom images 610-612 generated from the images 607-609 by the super-resolution processing are reproduced.

[Image processing]
Camera Signal Processing Unit Image processing of the camera signal processing unit 105 will be described with reference to the flowchart of FIG. Note that the camera signal processing unit 105 starts the processing illustrated in FIG. 7 with the start of moving image shooting, and executes it, for example, in units of frames.

  The camera signal processing unit 105 receives, for example, one frame of image data from the buffer memory 104 (S701), and performs development processing on the image data to generate image data (S702). The image data is stored in the image memory 208 (S703), and the image data stored in the image memory 208 is output (S704). Then, the electronic zoom magnification information is input to determine whether the electronic zoom is on or off (S705), and the electronic zoom on / off information and the electronic zoom magnification are output (S706).

  Subsequently, if the electronic zoom is off, the camera signal processing unit 105 proceeds with the process to step S709. If the electronic zoom is on, electronic zoom image data is generated from the image data stored in the image memory 208 according to the electronic zoom magnification information (S707), and the generated electronic zoom image data is output (S708). Note that linear interpolation, bicubic interpolation, or the like is used as an interpolation enlargement method in the electronic zoom.

  Subsequently, the camera signal processing unit 105 determines whether or not the moving image shooting is completed (S709), and if not completed, the process returns to step S701.

File Creation Unit The file creation process of the file creation unit 112 will be described with reference to the flowchart of FIG. The file creation unit 112 starts the process shown in FIG. 8 with the start of moving image shooting.

  The file creation unit 112 inputs the electronic zoom on / off information (S801), stores the electronic zoom on / off information in the tag information unit 501 of the data file (S802), and determines whether the electronic zoom is on or off (S803). When the electronic zoom is off, image data is input (S804), and the image data is stored in the moving image information section 502 of the data file (S805).

  If the electronic zoom is on, the file creation unit 112 inputs electronic zoom magnification information, electronic zoom image data, and image data (S806). The electronic zoom magnification information is stored in the tag information section 501 of the data file (S807), the electronic zoom image data is stored in the moving image information section 502 of the data file (S808), and the image data is stored in the image information section 503 of the data file. (S809).

  Subsequently, the file creation unit 112 determines whether or not the moving image shooting has been completed (S810). If the movie creation has not been completed, the process returns to step S801, and, for example, the processes of steps S801 to S809 are repeated for each frame.

  When the moving image shooting is completed, the file creation unit 112 acquires the imaging model information from the imaging model DB 108 (S811), and stores the imaging model information in the tag information unit 501 of the data file (S812). Then, the data file is closed and the data file is output (S813).

Display Device The moving image playback process of the display device 106 will be described with reference to the flowchart of FIG. For example, when a data file is input from the file creation unit 112, the display device 106 starts the processing illustrated in FIG.

  The display device 106 inputs a data file (S901), separates tag information from the data file (S902), and separates imaging model information, electronic zoom magnification information, and electronic zoom on / off information from the tag information (S903). Then, the electronic zoom on / off information is supplied to the super-resolution image generation unit 409 and the moving image reconstruction unit 410 (S904), and the imaging model information and the electronic zoom magnification information are supplied to the super-resolution image generation unit 409 (S905).

  Next, the display device 106 separates the moving image data from the data file for each frame and supplies it to the moving image reconstruction unit 410 (S906), and determines whether the electronic zoom is on or off based on the electronic zoom on / off information for each frame (S907). ). If the electronic zoom is on, the image data of the frame is separated from the data file and supplied to the super-resolution image generation unit 409 (S908).

  When the electronic zoom on / off information for each frame indicates that the electronic zoom is on, the super-resolution image generation unit 409 performs electronic zoom image data (hereinafter, referred to as electronic zoom image data by super-resolution processing from input image data based on the electronic zoom magnification information). Super-resolution electronic zoom image data) is generated. In addition, the moving image reconstruction unit 410 replaces moving image data of frames for which electronic zoom is on with super-resolution electronic zoom image data.

  Next, the display device 106 determines whether or not the moving image playback process of the data file has ended (S909). If not completed, the display device 106 returns the process to step S906 and continues the moving image playback process. Note that the end of the moving image playback process may be determined based on whether the electronic zoom on / off information for each frame has reached the end or the moving image data has reached the end.

[Super-resolution image generator]
A configuration example of the super-resolution image generation unit 409 will be described with reference to the block diagram of FIG. Hereinafter, a high resolution image is referred to as an “HR image”, and a low resolution image is referred to as an “LR image”.

  The super-resolution image generation unit 409 inputs image data from the input terminal 1001 and electronic zoom magnification information from the input terminal 1012. For simplicity of explanation, the input image data is assumed to be 8-bit grayscale image data.

  The image cutout unit 1009 outputs LR image data obtained by cropping the image data of the area corresponding to the electronic zoom magnification information at the center of the image represented by the input image data.

  The initial HR image creation unit 1002 creates initial HR image data that is an initial value of the HR image from the LR image data, for example, by linear interpolation processing, and outputs the initial HR image data to the HR image creation unit 1005. The HR image creation unit 1005 updates the HR image data by adding a correction amount calculated by a MAP estimation method described later to the HR image data. The degraded image generation unit 1011 generates degraded image data obtained by degrading the HR image data output from the HR image creation unit 1005 in accordance with the degradation condition input from the degradation condition input unit 1006.

  Degradation conditions are the degradation function (PSF: point spread function) of the optical system of the imaging device 113 indicated by the imaging model information input from the input terminal 1014 and downsampling which is a degradation process due to the limitation on the number of pixels of the imaging device 102. is there. Note that the downsampling degradation conditions handled in this embodiment are given by the reduction ratios 1 / M and 1 / N in downsampling (M and N are natural numbers). The deterioration condition is data measured prior to super-resolution processing or known data.

  The difference calculation unit 1010 calculates the difference between the initial HR image data and the deteriorated image data. Here, the average value of the difference image between the initial HR image data and the deteriorated image data is calculated. The end determination unit 1007 determines whether or not the super-resolution image has been sufficiently generated using the average value of the difference images. The determination threshold at that time is given from the input terminal 1008, and the determination threshold is an arbitrary value from 0 to 255. That is, the end determination unit 1007 compares the average value of the difference image with the determination threshold value, and outputs a determination value '0' if the average value ≦ the determination threshold value, and determines the determination value '1 if the average value> the determination threshold value. 'Is output.

  The correction amount calculation unit 1004 inputs the determination value. When the determination value is “1”, the correction amount calculation unit 1004 calculates the correction amount from the average value of the difference images, and outputs the correction amount to the HR image creation unit 1005. When the determination value is “0”, the correction amount having a value of zero is output to the HR image creation unit 1005.

  The HR image creation unit 1005 outputs HR image data from the output terminal 1003 when the value of the correction amount is zero. If the correction amount> 0, the HR image data is updated by adding the correction amount to the HR image data. That is, the update of the HR image data is repeated until the difference between the input image data and the deteriorated image data becomes equal to or less than the determination threshold value.

[Imaging device deterioration model]
A deterioration model of the image pickup apparatus will be described with reference to FIG. The deterioration process of the imaging system will be described using a linear model.

  The HR image 1101 (vector X) before the degradation process is subjected to degradation by the PSF of the optical system processing of the optical system processing 1102 (hereinafter, optical degradation) (matrix B). In addition, the captured image 1104 (vector Y) is obtained in response to deterioration due to downsampling (hereinafter referred to as downsampling deterioration) (matrix D) in the enlargement processing 1103 in the electronic zoom processing. The condition for downsampling degradation is provided as an image reduction magnification.

If the optical degradation is represented by a linear filter F having a size of (2C + 1) × (2C + 1) (C is a natural number), the image Y _PSF subjected to the optical degradation is represented by the following equation by the convolution calculation.
Y _PSF (i, j) = Σ _p Σ _q X (ip, jq) * F (p, q)… (1)
Where Y _PSF is an image degraded by PSF,
X is HR image 1101 before deterioration,
(i, j) is the pixel coordinates,
F (p, q) is the degradation filter,
* Represents a convolution operation,
Calculation range of sigma _p from p = -C to C,
The calculation range of Σ _q is from q = -C to C.

For convenience in super-resolution processing, the convolution operation of Equation (1) is defined by Equation (2) and changed to the linear operation format. A conversion method from Expression (1) to Expression (2) will be described later. According to the linear calculation format, the deterioration model of the image pickup apparatus is expressed by the following equation.
Y = DB · X (2)
Where X is the vector of the HR image 1101 before degradation,
Y is the captured image 1104,
Matrix B is a square matrix corresponding to optical system degradation,
Matrix D is a matrix corresponding to downsampling degradation.

  The sizes of the matrices B and D vary depending on the size of the input image. The sizes of the matrices B and D will be described with reference to FIG. In FIG. 12 (a), the size of the input image is W for the number of pixels in the horizontal direction, H for the number of pixels in the vertical direction, and the image reduction ratio, which is a condition for downsampling degradation, 1 / M in the horizontal direction and 1 / N in the vertical direction Shows the size of general matrices B and D.

  The relationship between the two-dimensional image and the matrix X will be described with reference to FIG. The size of the two-dimensional image is W × H, and the pixel position is represented by (i, j). That is, the pixel data of the HR image 1101 is scanned in raster order to form a matrix X with one column and HW rows.

  A configuration example of the matrix B will be described with reference to FIG. The size of the linear filter F is (2C + 1) × (2C + 1), and a product-sum operation with the linear filter F is performed on all pixels as shown in Expression (2).

  When the pixel (i, j) is at the center of the linear filter F, the value of the filter is substituted into the corresponding position of the matrix in the matrix i × j row. Assign to 0 except where it is assigned.

  A configuration example of the matrix D will be described with reference to FIG. The matrix D is a matrix for obtaining pixel data obtained by thinning out the pixels of the image using the reduced size. Regarding the matrix values, the matrix value corresponding to the position where the pixel from which the pixel data is to be obtained is 1, and the other matrix values are 0.

  FIG. 12 (b) shows a specific example of the sizes of the matrices B and D. In FIG. 12B, the size of the HR image 1101 is W = 1024, H = 1024, and the reduction magnification, which is a condition for downsampling degradation, is M = 1/2 and N = 1/2. That is, the size of the captured image 1104 is W = 512 and H = 512.

[Generation of super-resolution images]
In this embodiment, the MAP method is used to generate a super-resolution image. The MAP method is a method for estimating a high-resolution image by minimizing an evaluation function obtained by adding probability information of a high-resolution image to a square error. That is, it is a super-resolution processing method that estimates a high-resolution image as an optimization problem that maximizes the posterior probability by using some prior information for the high-resolution image. In this embodiment, a super-resolution image is generated using the following equation.
X ^ = arg min [α || CX ^ || ² + 1 / σ ² || Y-DBX ^ || ² ]… (3)
Where X ^ is the estimated super-resolution image,
Y is a deteriorated captured image,
D and B are matrices representing degradation,
C is a linear filter.

|| CX ^ || ^{2 in} the first term on the right side of Equation (3) is the pixel value of the adjacent pixel, and is a constraint term that takes into account prior information that the pixel of interest often has a value similar to the adjacent pixel. There is an effect of smoothing the entire image. A Laplacian filter is used as the linear filter C. The coefficient alpha, which is a parameter for adjusting the degree of smoothing of || CX ^ || ² Section. To obtain a result with a high degree of smoothing, the value of α is increased.

1 / σ ² || Y-DBX ^ || ² that is the second term on the right side of Equation (3) is a difference term between the input image and an image that has deteriorated the estimated super-resolution image. The coefficient σ is a standard deviation of noise of the captured image Y of the imaging device. Expression (4) is an expression that is used as an evaluation function during super-resolution processing by extracting the right side of Expression (3).
I = α || CX ^ || ² + 1 / σ ² || Y-DBX ^ || ² … (4)

  A processing example of the super-resolution image generation unit 409 will be described with reference to the flowchart of FIG.

  The super-resolution image generation unit 409 inputs image data (corresponding to the captured image Y) (S1701), inputs imaging model information (corresponding to the matrixes B and D representing deterioration) (S1702), and electronic zoom magnification information (Corresponding to the condition of downsampling degradation) is input (S1703). Then, initial HR image data is generated from the LR image data obtained by cropping the image data of the area corresponding to the electronic zoom magnification information at the center of the image represented by the image data (S1704). In other words, the size of the LR image data is expanded M times in the horizontal direction and N times in the vertical direction by linear interpolation.

Next, the super-resolution image generation unit 409 inputs a determination threshold value for determining the degree of achievement of super-resolution processing (S1705), and optical degradation (matrix B) F and down-sampling degradation (matrix D) are added to the HR image data. Is generated (S1706). Expression (5) represents a generation expression of the deteriorated image data.
Y '= DB · X _k (5)
Where Y ′ is the degraded image data,
X _k is the HR image data generated by the kth calculation.

Next, the super-resolution image generation unit 409 calculates a value difference for each pixel of the deteriorated image data Y ′ and the initial HR image data Y by Equation (6), and calculates an average value Δ of these differences (S1707). ).
Δ = [Σ _i Σ _j {Y '(i, j)-Y (i, j)}] / N… (6)
Here, N is the total number of pixels (W / M × H / N).

Next, the super-resolution image generation unit 409 compares the average difference Δ with the determination threshold Th (S1708). For delta> Th, and calculates an update term [Delta] X _k is a correction amount of the HR image data X _k (S1709), it increments the counter k (S1710). Then, the update term ΔX _k-1 is added to the HR image data X _k-1 to update the HR image data X _k (S1711), and the process returns to step S1706. If Δ ≦ Th, the super-resolution image generation unit 409 outputs the HR image data X _k (S1712).

In the present embodiment, the steepest descent method shown in the following equation is used to calculate the update term ΔXk. The steepest descent method is an optimization method using the first derivative of the function to be minimized. The update term (first derivative of the function) ΔXk required for the steepest descent method is calculated by the following formula.
ΔXk = ∂I / (∂Xk) = 2αC ^T CXk + 2 / σ ² (DB) ^T [Y-DBXk]… (6)
Here, T represents a transposed matrix.

  In the above description, the method of including the image data before the electronic zoom processing corresponding to the frame of the video image subjected to the electronic zoom processing, the imaging model information, and the electronic zoom magnification information in the data file of the moving image data has been described. Then, when playing back a moving image, the image data of the electronic zoom processed frame is generated from the image data before the electronic zoom processing by super-resolution processing, and the image data of the electronic zoom processed frame is generated by super-resolution processing. The method of replacing with generated image data has been described. In this way, it is possible to reproduce a higher resolution and higher definition image than an image obtained by electronic zoom.

  In the above description, an example in which the embodiment is applied to a moving image has been described. However, even in the case of a still image, image data that has been subjected to electronic zoom processing and image data that has not been electronically zoomed can be retained. When displaying an image that has been subjected to electronic zoom processing, if the image is displayed using image data obtained by super-resolution processing of the image data before the electronic zoom processing, the resolution is higher than that obtained by the electronic zoom. An image can be displayed.

  In addition, in order to hold the image data that has been subjected to the electronic zoom processing and the image data before the electronic zoom processing, a moving image or still image that has been subjected to the electronic zoom processing is reproduced or displayed on a display device that does not have a super-resolution processing function. And compatibility can be maintained.

[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (6)

Imaging means for imaging image data of a subject;
Generating means for generating zoom image data obtained by enlarging a part of the image represented by the image data output by the imaging means according to the zoom magnification information input;
Storage means for storing image degradation information due to the optical system of the imaging means and image degradation information due to the enlargement processing as imaging model information;
An image processing apparatus comprising: information indicating whether the enlargement processing is on / off, the zoom magnification information, the imaging model information, the image data, and a creation unit that creates a data file storing the zoom image data. Further, super-resolution obtained by enlarging a part of the image represented by the image data according to the zoom magnification information by super-resolution processing from the zoom magnification information, the imaging model information, and the image data stored in the data file. Super-resolution means for generating image image data;
Replacement means for replacing the zoom image data retrieved from the data file with the super-resolution image data,
When the information indicating on / off of the enlargement process stored in the data file indicates that the enlargement process is on, the super-resolution means generates super-resolution image data, and the replacement means performs replacement. 2. The image processing apparatus according to claim 1. Furthermore, it has a control means for controlling the optical zoom function of the imaging means,
3. The control unit according to claim 1, wherein the control unit outputs the zoom magnification information to the generation unit when a zoom magnification indicated by a user reaches a limit of the optical zoom function. Image processing device.   The creation means stores the image data in a storage area of image data used for normal reproduction of the data file when the information indicating on / off of the enlargement processing indicates off, and information indicating on / off of the enlargement processing 4. The image processing apparatus according to claim 1, wherein the zoom image data is stored in a storage area of the image data used for the normal reproduction when “I” indicates ON. . An imaging unit that captures image data of a subject, a generation unit that generates zoom image data obtained by enlarging a part of an image represented by the image data output by the imaging unit, in accordance with input zoom magnification information, and an optical unit of the imaging unit An image processing method for an image processing apparatus having storage means for storing image degradation information by the system and image degradation information by the enlargement processing as imaging model information, and a creation means,
An image processing method, wherein the creation unit creates a data file storing information indicating on / off of the enlargement process, the zoom magnification information, the imaging model information, the image data, and the zoom image data.   6. A program causing an image processing apparatus to execute the image processing according to claim 5.

Images