JP2012133492A - Image processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent impression of blur of image from changing due to magnification processing.SOLUTION: The image processing device for magnifying image comprises: a feature amount extraction part that extracts feature amount relevant to blur from plural partial areas of an image; and an interpolation processing part that magnifies the image by performing interpolation calculation using an interpolation parameter suitable for the extracted feature amount on each of the plural partial areas. The feature amount extraction part may extract the feature amount by normalizing an edge amount extracted from each of plural partial areas in an image area wider than the partial area.

Description

  The present invention relates to an image processing apparatus and a program.

A technique is known in which input image data is enlarged by nearest neighbor interpolation processing and then filtered with a filter having a size corresponding to the enlargement ratio (for example, Patent Document 1).
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP 2002-259960 A

  If the image is scaled uniformly by interpolation processing, the behavior of the edge component of the scaled image will be different from the original image. For example, in the image enlargement process, the enlarged image may be blurred uniformly. Further, in the image reduction process, the reduced image may be uniformly clear. For this reason, there has been a problem that an image that has been uniformly scaled by interpolation processing has an impression that is significantly different from the original image.

  In order to solve the above-described problem, in one aspect of the present invention, an image processing apparatus for scaling an image, the feature amount extraction unit extracting a feature amount related to blur from each of a plurality of partial regions in the image. And an interpolation processing unit that scales the image by performing an interpolation operation on each of the plurality of partial regions using an interpolation parameter corresponding to the feature amount extracted from each of the partial regions.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is a conceptual diagram which shows the scene which changes the magnification of the image which the camera 100 image | photographed. It is a figure which shows the system block diagram of the camera 100 which concerns on this embodiment. It is a figure which shows typically a part of process which produces | generates the reduced image 400. FIG. It is a figure which shows typically a part of process which produces | generates the reduced image 400. FIG. It is a figure explaining an example of the interpolation parameter used for an interpolation calculation. FIG. 6 is a diagram showing a processing flow for generating a reduced image in the camera 100.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a conceptual diagram showing a scene in which an image taken by the camera 100 is scaled. The camera 100 is a digital camera as an example of an imaging device. The captured image data generated by being captured by the camera 100 is displayed on a display unit 110 such as a display panel in order to present a captured result or the like to the user. At this time, the captured image data is converted into image data of the number of pixels used by the display unit 110 for image display. In many cases, the captured image data is reduced and displayed on the display unit 110. In recent years, the number of effective pixels of an image pickup device included in an image pickup apparatus is increasing. However, since the number of pixels included in the display unit 110 has not increased significantly, the reduction ratio for captured image data tends to be high.

  Also, captured image data generated by the camera 100 is output to various output media via various image output devices. For example, the captured image data is output to a digital photo frame 20, a mobile phone terminal 30, a personal computer 42, a television 60, etc. as an example of an image output device, and is output to a display panel included in each image output device. The captured image data is displayed by being reduced or enlarged according to the number of pixels used for display by the display panel. The captured image data is output to the printer 50, and is reduced or enlarged according to the number of pixels corresponding to the print resolution of the print head of the printer 50, and printed on the print medium. Examples of the image output device include a projection device such as a projector in addition to the illustrated device.

  For example, when high-pixel-captured image data captured by a recent imaging apparatus is displayed on the digital photo frame 20 or the large TV 60, the captured image data needs to be greatly reduced and displayed. Further, when displaying image data with a small number of pixels by partially enlarging the image data on the PC monitor 40 using the PC 42, it is necessary to display the image data in a greatly enlarged manner.

  In general, when a scaling process is performed on an image, the appearance of the image may change before and after the scaling process. For example, when an image is reduced, even if there is a blurred area in a part of the image, the area may be sharpened by the reduction process, and the entire image may be uniformly clear. Specifically, a case where a cloud or mountain photographed as a background in an image before zooming is sharpened to the same extent as the main subject in the foreground by the reduction process.

  The change in the appearance impression that occurs before and after zooming tends to be particularly noticeable when the enlargement ratio or reduction ratio is high. In addition, an image output device including the PC monitor 40 and the digital photo frame 20 may receive an image having various numbers of pixels. Therefore, there is a demand for a scaling process that does not change the visual impression not only for a specific scaling ratio but also for a wide range of scaling ratios.

  In the present embodiment, an example of image processing in which the visual impression is less likely to change before and after the scaling process will be described by focusing on image blur. In the present embodiment, an example of the operation of the camera 100 will be described by taking up mainly the case where the captured image data is reduced, such as when the camera 100 reduces the captured image data by interpolation and displays the image on the display unit 110.

  FIG. 2 is a system configuration diagram of the camera 100 according to the present embodiment. The camera 100 includes an optical system 101. The optical system 101 includes a zoom lens, a focus lens, and the like. Subject light enters the optical system 101 along the optical axis, passes through the shutter 102, and forms an image as a subject image on the light receiving surface of the image sensor 103.

  The image sensor 103 is an element that photoelectrically converts an optical image that is a subject image, and for example, a CCD or a CMOS sensor is used. The subject image photoelectrically converted by the image sensor 103 is read out as electric charge, and gain adjustment or the like is performed by the amplifier unit 104. The signal amplified by the amplifier unit 104 is transferred as an analog signal to the A / D converter 105 and converted to a digital signal by the A / D converter 105. Controls such as charge reading of the image sensor 103 and driving of the shutter 102 are controlled by a clock signal supplied by the timing generator 112 that receives synchronization control of the system controller 114.

  The subject image converted into the digital signal is sequentially processed as image data. The image data converted into a digital signal by the A / D converter 105 is delivered to the image processing unit 107 and processed. Specifically, the image data converted into a digital signal by the A / D converter 105 is temporarily stored in the internal memory 108 under the control of the memory control unit 106. The internal memory 108 is a random access memory that can be read and written at high speed, and for example, a DRAM, an SRAM, or the like is used. The internal memory 108 serves as a buffer memory that waits for the order of image processing when image data is continuously generated at high speed in continuous shooting and moving image shooting. In addition, the image processing unit 107 also serves as a work memory in the image processing and compression processing. Therefore, the internal memory 108 has a sufficient memory capacity corresponding to these roles. The memory control unit 106 controls how much memory capacity is allocated to what work.

  The image processing unit 107 converts the image data into image data of a standardized image format according to the set shooting mode and an instruction from the user. For example, when having a still image shooting mode and a moving image shooting mode as part of the shooting mode of the camera 100 and generating a JPEG file as a still image in the still image shooting mode, the image processing unit 107 performs color conversion processing, After performing image processing such as gamma processing and white balance correction, compression processing is performed by performing adaptive discrete cosine transform or the like. Also, for example, an MPEG file as a moving image in the moving image shooting mode, When generating an H.264 file, the image processing unit 107 performs compression processing by performing intra-frame encoding, inter-frame encoding, quantization, entropy encoding, and the like on the generated frame images as continuous still images. I do.

  The converted image data is stored in the internal memory 108 again. Still image data and moving image data processed by the image processing unit 107 are recorded on the recording medium 122 which is a nonvolatile memory from the internal memory 108 via the recording medium IF 111 under the control of the system control unit 114. The recording medium 122 is configured by a flash memory or the like. The recording medium 122 may be a non-volatile memory that can be attached to and detached from the main body of the camera 100.

  The image processing unit 107 generates display image data in parallel with the image data processed for recording. The image processing unit 107 generates display image data by performing a scaling process described later in parallel with the image data processed for recording. The generated display image data is displayed on the display unit 110 configured by a liquid crystal panel or the like under the control of the display control unit 109. In addition, the display control unit 109 can display various menu items related to various settings of the camera 100 on the display unit 110 with or without displaying an image.

  The camera 100 is directly or indirectly controlled by the system control unit 114 including each element in the image processing described above. The system control unit 114 includes a system memory that is an electrically erasable / recordable nonvolatile memory. The system memory is configured by EEPROM (registered trademark) or the like. The system memory included in the system control unit 114 records constants, variables, programs, and the like necessary when the camera 100 is operating so that they are not lost even when the camera 100 is not operating. The system control unit 114 expands constants, variables, programs, and the like as appropriate in the internal memory 108 and uses them for controlling the camera 100.

  The camera 100 includes an operation member 120 that receives an operation from a user. The system control unit 114 detects that the operation member 120 has been operated. The system control unit 114 performs an operation according to the detected operation. For example, when it is detected that the release switch as the operation member 120 is operated, the system control unit 114 performs a series of shooting operations for photoelectrically converting the subject image to generate image data.

  The optical system 101 is controlled by the lens control unit 113. For example, the lens control unit 113 drives the zoom lens in accordance with a user instruction to change the angle of view of the subject image. In addition, the lens control unit 113 drives the focus lens based on the AF information, and focuses the subject image on the light receiving surface of the image sensor 103. The lens control unit 113 may perform autofocus control on a subject designated by the user through the operation member 120 or the like. The lens control unit 113 also calculates the exposure value by analyzing the image data processed by the image processing unit 107 or the output of the photometric sensor. The lens control unit 113 outputs a control signal for the diaphragm of the optical system 101 according to the calculated exposure value.

  3 and 4 schematically show processing until the reduced image 400 is generated by performing the reduction processing on the input image 300. The processing according to FIGS. 3 and 4 is executed by the image processing unit 107 acting as a main subject unless otherwise specified. The input image 300 is captured image data captured by the image sensor 103 and stored in the internal memory 108, for example, and is an image to be subjected to a reduction process. For example, the input image 300 may be image data having a number of pixels exceeding 16 million pixels, such as 4928 × 3264 pixels. On the other hand, the reduced image may be image data having a number of pixels of one million pixels or less, such as 900 × 600 pixels.

  The image processing unit 107 performs an edge amount extraction process 310 on the input image 300 to generate a data string having an edge amount corresponding to each pixel point of the reduced image in the internal memory 108. This data string is called an edge amount image 320. In the edge amount extraction process 310, for example, an edge amount is extracted from a partial area having a predetermined size corresponding to the pixel position of the reduced image. As an example, the edge extraction filter 312 is applied to each of a plurality of partial regions illustrated by solid lines in the input image 300. The illustrated edge extraction filter 312 determines 8 for the center pixel and −1 for the surrounding 8 pixels as the weighting coefficient applied to each pixel in the 3 × 3 pixel block as an example of the partial region. ing. By applying the edge extraction filter 312, the edge amount of each center pixel is extracted. The extracted edge amount of each central pixel is associated with the pixel point illustrated by the broken line in the edge amount image 320. Note that the edge amount extraction method is not limited to this example. The size of the filter for extracting the edge amount is not limited to this example.

  The image processing unit 107 performs a normalization process 330 on the edge amount image 320 to generate a normalized image 340 and generates it in the internal memory 108. The normalized image 340 is a data string having a normalized edge amount corresponding to each pixel point of the reduced image. In the normalization process 330, the average value of the edge amount and the standard deviation σ are calculated from the entire region of the edge amount image 320. Then, the deviation is calculated by subtracting the average value from the edge amount for each pixel point. Then, the normalized image 340 is generated by dividing the deviation for each pixel point by the standard deviation σ. It can be said that the normalized image 340 shows how strong the edge should be at each pixel point of the reduced image.

  The image processing unit 107 performs a smoothing process 350 on the normalized image 340 to generate a smoothed image 360, which is generated in the internal memory 108. The smoothed image 360 is a data string having a normalized and smoothed edge amount corresponding to each pixel point of the reduced image. For example, in the smoothing process 350, the normalized edge amount of each pixel point is smoothed by applying the smoothing filter 352 to each pixel point. The illustrated smoothing filter 352 determines 2 for the target pixel and 1 for the surrounding 8 pixels. By the smoothing process 350, the correlation between the target pixel and the peripheral pixels can be enhanced. The specific processing method of smoothing is not limited to this example. Any processing may be used as long as the behavior of surrounding pixels can be added to the target pixel. When converting to a smoothed edge amount, conversion may be performed using an arbitrary predetermined function or using an LUT.

  With the processing illustrated in FIG. 3, the image processing unit 107 can extract a feature amount related to blur from each of a plurality of partial regions in the input image 300. When simply referred to as a feature value in the description of the present embodiment, it refers to a feature value related to blur unless otherwise specified. In the present embodiment, unless otherwise specified, the feature amount refers to a normalized and smoothed edge amount.

  FIG. 4 schematically shows processing from the smoothed image 360 to the generation of the reduced image 400. As described above, the smoothed image 360 is a data string having a feature amount corresponding to each pixel point of the reduced image. As the filter parameter extraction process 370, the image processing unit 107 converts the feature amount of each pixel point into an interpolation parameter.

Specifically, the image processing unit 107 converts the feature amount of each pixel point into an interpolation parameter according to a table 372 that associates the feature amount with the interpolation parameter. The table 372 may associate data defining a weighting factor for neighboring pixels used in the interpolation calculation with the feature amount. For example, when the pixel value of the n × n pixel block including the target pixel is used for the interpolation calculation, n corresponding to each pixel position (x _{1 to} x _n , y _{1 to} y _n ) of the n × n pixel block. data defining the ^two weighting coefficients are stored in the table 372.

  The table 372 stores data for determining a weighting factor that can transmit a higher spatial frequency component corresponding to a feature value having a larger value. For example, the data stored in the table 372 may include parameters that determine the function shape of the weighting coefficient for cubic interpolation. The table 372 may store data for determining weighting factors corresponding to combinations of feature amounts and reduction ratios. Details of the data for determining the weighting coefficient will be described later.

  An interpolation parameter image 380 is generated in the internal memory 108 by the filter parameter extraction process 370. The interpolation parameter image 380 is a data string having an interpolation parameter corresponding to each pixel point of the reduced image. In the interpolation calculation process 390, the reduced image 400 is generated using the interpolation parameter image 380 and the input image 300 and stored in the internal memory 108. Specifically, the pixel value of each pixel point is calculated by performing interpolation calculation on the pixel value of the input image 300 according to the interpolation parameter.

  According to the camera 100 of this embodiment, an interpolation parameter that transmits a high-frequency region can be applied to an image region having a relatively strong edge amount in the image. In addition, an interpolation parameter that does not transmit a high-frequency region can be applied to an image region having a relatively weak edge amount in the image. Therefore, an image area with relatively small blur in the input image 300 has relatively small blur in the reduced image 400, and an image area with relatively blur is relatively large in the reduced image 400. Become. Therefore, it is possible to prevent the appearance of blurring from changing significantly before and after the reduction process.

  FIG. 5 is a diagram for explaining an example of interpolation parameters used for the interpolation calculation. In this figure, the x-axis indicates the distance from the interpolation position to each pixel position. The coordinate values on the horizontal axis are normalized so that the pixel interval of the image to be scaled is 1. h (x) represents a weighting coefficient used for the interpolation calculation.

The weighting factor 500 indicates a cubic interpolation weighting factor as an example of the weighting factor. In the cubic interpolation, the pixel value of the pixel of interest is calculated by weighted averaging the pixel values of 4 × 4 pixels in the vicinity of the pixel of interest according to a weighting coefficient. In the present embodiment, a weighting coefficient for a pixel having | x | of 1 or less is calculated as h (x) = (a + 2) | x | ³ − (a + 3) | x | ² +1. The weighting coefficient for pixels where | x | is greater than 1 and less than or equal to ² is calculated by h (x) = a | x | ³ −5a | x | ² + 8a | x | −4a. For pixels where | x | is greater than 2, 0 is applied as a weighting factor.

  In the present embodiment, the coefficient a is a parameter that determines the function shape of h (x), and is an example of data that determines the weighting coefficient. In the drawing, h (x) when a = −1 is indicated by a solid line, and h (x) when a = −0.2 is indicated by a dotted line. Compared with the weighting factor determined by a = −1, the weighting factor determined by a = −0.2 can transmit higher frequency components. That is, by selecting a coefficient a having a different value, the transmission degree of the high frequency component can be varied. Therefore, in the table 372 illustrated in FIG. 4, the value of the coefficient a that can transmit a higher frequency component may be associated with a larger feature amount. In the table 372, coefficients a having different values may be associated with two or more feature amounts.

  The weighting factor 510 indicates a weighting factor for linear interpolation as another example of the weighting factor. In linear interpolation, interpolation calculation is performed using pixel values of 2 × 2 pixels in the vicinity of the target pixel. The weighting coefficient for pixels having | x | of 1 or less is calculated by h (x) = 1− | x |. For pixels where | x | exceeds 1, 0 is applied as a weighting factor. In general, linear interpolation has a smaller filter radius and can transmit higher frequency components than cubic interpolation. Therefore, cubic interpolation may be applied to pixel points having a feature amount equal to or less than a predetermined value, and linear interpolation may be applied to pixel points having a feature amount larger than a predetermined value. For example, in the table 372, information for identifying cubic interpolation may be associated with a first feature amount, and information for identifying linear interpolation may be associated with a second feature amount having a large first feature amount. . As described above, in the table 372, information identifying an interpolation function used for the interpolation calculation may be associated with the feature amount as an example of data for determining the weighting coefficient.

  FIG. 6 shows a processing flow for generating a reduced image in the camera 100. This flow is started when the captured image data is stored in the internal memory 108. In this flowchart, the image processing unit 107 operates mainly unless otherwise specified.

  In step S602, a reduction ratio for the captured image data is determined. The reduction ratio is determined by the number of pixels of the captured image data and the number of pixels of the reduced image to be generated. In step S604, an edge amount is extracted from the captured image data. This processing flow corresponds to the edge amount extraction processing 310 described with reference to FIG.

  In step S606, a feature amount corresponding to the reduced pixel is calculated from the captured image data. This processing flow corresponds to the normalization processing 330 and the smoothing processing 350 described with reference to FIG. With this processing flow, data corresponding to the smoothed image 360 described with reference to FIG. 3 is generated in the internal memory 108.

  In step S608, an interpolation parameter is calculated for each pixel point. This processing flow corresponds to the filter parameter extraction processing 370 described with reference to FIG. As described with reference to FIGS. 4 and 5, the interpolation parameter is selected according to the combination of the reduction ratio and the feature amount. For example, as described with reference to FIG. 5 and the like, the interpolation parameter is selected for each pixel point according to the table 372 that associates the interpolation parameter with the reduction ratio and the feature amount. For example, the coefficient a for cubic interpolation corresponding to the reduction ratio and the feature amount is determined for each pixel point. Then, based on the positional relationship between the position of the pixel point and 4 × 4 pixels of the captured image data, a set of weighting coefficients as an example of an interpolation parameter is calculated for each pixel point.

  In step S610, a reduced image is calculated by performing an interpolation operation on the captured image data using the interpolation parameter calculated in step S608. This processing flow corresponds to the interpolation calculation processing 390 described with reference to FIG. For example, the pixel value of each pixel point is calculated by applying the weighted coefficient set calculated for each pixel point in step S608 to the corresponding 4 × 4 pixels of the captured image data and performing weighted averaging.

  In step S612, a reduced image is output. For example, the reduced image is displayed on the display unit 110 under the control of the display control unit 109. The reduced image may be recorded on the recording medium 122 via the recording medium IF111.

  In step S614, it is determined whether to generate an image with another reduction ratio from the captured image data. For example, it is determined whether to generate an image for an external image output device such as the external photo frame 20. In addition, it is determined whether or not to further generate an image in which a partial area designated by the user is enlarged. When generating an image with another reduction ratio, the process proceeds to step S602. The processing from step S602 to step S612 is continued until a predetermined reduced image set is generated. When the generation of all the reduction ratio images is completed, a series of reduced image generation processing ends.

  According to the reduction process described above, it is possible to perform an interpolation operation using different interpolation parameters depending on, in particular, a feature amount related to blur. For this reason, the edge of the partial area where the blur is relatively large is not emphasized and sharpened. In addition, the partial area where the blur is relatively small is not blurred. For this reason, the difference in the degree of blur in the image to be reduced can be properly reflected in the image after reduction.

  In the above, the operation of the camera 100 when performing the reduction process has been mainly described. However, by applying the same process to the enlargement process, it is possible to perform an interpolation operation using different interpolation parameters according to the feature amount related to the blur. Therefore, even in the enlargement process, it is possible to prevent a partial region with relatively small blurring from being blurred compared to a case where uniform interpolation parameters are applied to the entire image. For this reason, the difference in the degree of blur in the image to be enlarged can be accurately reflected in the enlarged image. Therefore, the interpolation processing according to the feature amount related to the blur described above can be applied not only to the reduction processing for reducing the number of pixels but also to the enlargement processing for increasing the number of pixels.

  The image processing unit 107 extracts a feature quantity related to blur from each of a plurality of partial areas in the image, and performs interpolation according to the feature quantity extracted from each of the plurality of partial areas. By performing the interpolation calculation using the parameters, it can function as an interpolation processing unit for scaling the image. In the above description, it has been described that the feature amount is extracted by normalizing the edge amount extracted from each of the plurality of partial regions in the entire image region. However, the image area to be normalized may not be the entire image area. For example, it may be an image region formed from a plurality of pixel points including a plurality of pixel points of the reduced image. That is, the feature amount may be extracted by normalizing the edge amount extracted from each of the plurality of partial regions with an image region wider than the partial region. By normalizing the entire image or a relatively wide image area, for example, even in captured image data in which the image is totally blurred, the interpolation parameter can be varied according to the distribution of the feature amount. The same applies to captured image data that is clear overall. Therefore, the difference in the degree of blur in the image to be scaled can be appropriately reflected in the image after scaling.

  In the above description, the normalized edge amount is calculated using the average value of the edge amount and the standard deviation σ. However, any processing may be used as long as it is an index value that can evaluate the position of each edge amount in the edge amount distribution in the image.

  In the above description, the edge amount is smoothed using an averaging filter that weights and averages the edge amount. When there is a locally steep edge in the captured image data, a large edge amount may be calculated at the corresponding pixel point. When selecting an interpolation parameter, if the extracted edge amount is used without being smoothed, a filter having greatly different interpolation characteristics may be applied between adjacent pixels. However, by smoothing the edge amount as in the present embodiment, the feature amount can be calculated by taking into account the edge amount of the surrounding pixel points, so that the interpolation characteristics of a certain pixel point are completely different from the surroundings. The filter is never applied.

  The process for smoothing the edge amount is not limited to the process using the averaging filter described above, and any process can be used as long as it can extract the feature quantity averaged with the surroundings. May be. For example, by extracting and normalizing the edge amount with a smaller number of pixel points than the number of pixels of the reduced image, and calculating the edge amount corresponding to the pixel point of the reduced image by interpolation or the like, it corresponds to the pixel point of the reduced image The feature amount to be calculated may be calculated. By calculating the edge amount of the pixel point from the edge amount of the small pixel point, a smoothed edge amount as an example of the feature amount can be calculated. As described above, even when the edge amount is extracted corresponding to a smaller number of pixel points than the image after scaling, the feature amount corresponding to the pixel point of the image after scaling can be calculated. In other words, regardless of whether or not the edge amount is extracted at all pixel points, feature amounts corresponding to a plurality of pixels after scaling of the image are extracted. The pixel values of the plurality of pixels can be calculated by performing the interpolation calculation using the interpolation parameters corresponding to the feature amounts respectively corresponding to the plurality of pixels. As another example, the feature amount or the interpolation parameter may be calculated for a smaller number of pixel points than the image after scaling, instead of interpolating the edge amount. Then, the interpolation parameter of the pixel point that has not been calculated may be calculated by interpolating the feature amount of the neighboring pixel point or the interpolation parameter.

  In the above, by selecting either cubic interpolation or linear interpolation according to the feature quantity, or by determining the coefficient a of cubic interpolation according to the feature quantity, the weighting coefficient for the neighboring pixels used for the interpolation calculation is calculated using the feature quantity. It was supposed to be different depending on the situation. However, in addition to cubic interpolation and linear interpolation, an interpolation filter to be applied can be selected from various interpolation filters such as nearest neighbor interpolation and Gaussian interpolation. In other words, the type of interpolation filter used for the interpolation calculation may be varied according to the feature amount. Also, depending on the feature amount, the result of two different filters may be used to calculate the result with the respective weighted sums.

  For example, when the feature amount indicates a blur amount equal to or less than a predetermined value, an interpolation filter based on the first interpolation function is applied, and when the feature amount indicates a blur amount larger than a predetermined value, the first interpolation An interpolation filter based on a second interpolation function that does not transmit a higher spatial frequency component than the function may be applied. “Not transmitting high spatial frequency components” is a concept including not emphasizing or generating high spatial frequency components. Examples of the first interpolation filter include an interpolation filter based on at least one of linear interpolation and nearest neighbor interpolation, and examples of the second interpolation filter include an interpolation filter based on at least one of cubic interpolation and Gaussian interpolation. Can do.

  Further, as a parameter for controlling the shape of the interpolation function, the standard deviation σ of Gaussian interpolation can be exemplified in addition to the coefficient a of cubic interpolation. For example, the standard deviation σ of Gaussian interpolation may be calculated based on the feature amount. Furthermore, a value proportional to the inverse of the feature amount may be applied to the standard deviation σ of Gaussian interpolation. In this case, the filter radius of the Gaussian interpolation becomes large in the portion where the edge is strong and the feature amount is large, and as a result, the pixel value of the corresponding point calculated by the interpolation processing has a large low frequency component.

  Then, the number of neighboring pixels used for the interpolation calculation may be varied according to the feature amount, such as changing the filter radius according to the feature amount. Specifically, when the feature amount indicates a larger blur amount, the number of neighboring pixels used for the interpolation calculation may be increased. As an example, linear interpolation may be applied when the feature amount is equal to or smaller than a predetermined value, and cubic interpolation with a filter radius larger than the linear interpolation may be applied when the feature amount is larger than the predetermined value. As described above, when the feature amount is larger, a pixel value having a larger blur is calculated by applying an interpolation function having a larger filter radius.

  In the above description, it has been described that the interpolation parameter is associated with the reduction ratio and the feature amount. That is, it has been described that the interpolation parameter according to the ratio between the number of pixels of the image to be subjected to the scaling process and the number of pixels after the scaling is applied. As an example, when the reduction ratio is larger, it is preferable to increase the number of neighboring pixels used for the interpolation calculation. Specifically, when the feature amount is larger, an interpolation filter having a larger filter radius may be applied. For example, the filter radius of cubic interpolation may be changed according to the reduction ratio and the feature amount.

  In the above description, it has been described that the pixel value of the image after scaling is calculated by the interpolation calculation according to the feature amount. Examples of pixel values include the intensity of a plurality of color signals such as RGB. That is, with respect to the R signal, the G signal, and the B signal, the intensity of each color signal in the image after scaling may be calculated by performing an interpolation operation for each color according to the feature amount. The feature amount may be extracted for each color from the image to be scaled, or may be extracted based on the luminance signal of the image to be scaled. Moreover, as a pixel value, a luminance signal and a color difference signal can be exemplified in addition to color signals such as RGB. In other words, the intensity of the luminance signal and the color difference signal in the image after scaling may be calculated by performing an interpolation operation on each of the luminance signal and the color difference signal according to the feature amount. The feature amount may be extracted from the image to be scaled for each luminance signal and color difference signal, or may be extracted based on the brightness signal of the image to be scaled. The interpolation process based on the feature amount may be performed only on the luminance signal. That is, an interpolation calculation may be performed on the luminance signal according to the feature amount of the luminance signal, and a predetermined interpolation calculation may be uniformly applied to the color difference signal regardless of the feature amount.

  In the above description, the edge amount is extracted from the two-dimensional pixel block. In addition, the edge amount in the one-dimensional direction may be extracted. For example, the respective edge amounts in the vertical direction and the horizontal direction may be extracted. Then, a feature amount related to blur may be calculated for each of the vertical direction and the horizontal direction. For each of the vertical direction and the horizontal direction, interpolation parameters may be calculated based on the respective feature amounts. For example, an interpolation parameter corresponding to the feature amount may be calculated for each of the vertical direction and the horizontal direction. Further, an interpolation filter having a filter radius corresponding to the feature amount may be applied to each of the vertical direction and the horizontal direction.

  In the above description, the zooming process according to the shooting condition when shooting the captured image data is not specifically mentioned. However, the interpolation parameter may be determined in consideration of shooting conditions. For example, when scaling processing is performed on captured image data captured in the portrait mode, the feature amount of the partial area included in the person image area may be corrected so that the high-frequency component of the person image area is transmitted. . For example, a feature amount extracted from a partial area belonging to a person's image area may be multiplied by a correction coefficient larger than 1. With this correction, it is possible to prevent a person image captured in the portrait mode from being blurred by the scaling process.

  The camera 100 described in the above embodiment can be applied to a mobile phone with a camera function as well as a lens interchangeable single lens reflex camera, a compact digital camera, a mirrorless single lens camera, a video camera, and the like. In the above description, it has been described that the scaling process is mainly performed by the camera 100. However, the scaling process can be performed by the digital photo frame 20, the mobile phone terminal 30, the personal computer 42, the printer 50, and the television 60.

  The camera 100, the digital photo frame 20, the mobile phone terminal 30, the personal computer 42, the printer 50, and the television 60 execute the scaling process by the processor executing a control program that controls the execution of the scaling process. Good. In an electronic information processing apparatus that can read a program from an external recording medium such as the personal computer 42, the control program may be loaded by reading a computer-readable recording medium that stores the control program. .

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

20 digital photo frame, 30 mobile phone terminal, 42 personal computer, 50 printer, 60 TV, 100 camera, 101 optical system, 102 shutter, 103 image sensor, 104 amplifier unit, 105 A / D converter, 106 memory control unit, 107 image processing unit, 108 internal memory, 109 display control unit, 110 display unit, 111 recording medium IF, 112 timing generation unit, 113 lens control unit, 114 system control unit, 120 operation member, 122 recording medium, 300 input image, 310 Edge amount extraction processing, 312 Edge extraction filter, 320 Edge amount image, 330 Normalization processing, 340 Normalization image, 350 Smoothing processing, 352 Smoothing filter, 360 Smoothed image, 370 Filter parameter extraction processing, 372 Buru, 380 interpolation parameter image, 390 interpolation calculation process, 400 the reduced image, 500, 510 weight factors

Claims (14)

An image processing apparatus for scaling an image,
A feature amount extraction unit that extracts a feature amount related to blur from each of the plurality of partial regions in the image;
An image processing apparatus comprising: an interpolation processing unit that scales the image by performing an interpolation operation on each of the plurality of partial regions using an interpolation parameter corresponding to the feature amount extracted from each of the partial regions. The feature amount extraction unit extracts the feature amount by normalizing an edge amount extracted from each of the plurality of partial regions using an edge amount extracted from an image region wider than each partial region. The image processing apparatus according to claim 1. The image processing apparatus according to claim 1, wherein the interpolation processing unit varies the number of neighboring pixels used for the interpolation calculation according to the feature amount. The image processing apparatus according to claim 3, wherein the interpolation processing unit increases the number of neighboring pixels when the feature amount indicates a larger blur amount. 5. The image processing apparatus according to claim 1, wherein the interpolation processing unit varies a weighting coefficient for neighboring pixels used for the interpolation calculation according to the feature amount. 6. The image processing apparatus according to claim 1, wherein the interpolation processing unit adds pixel values calculated by two or more different types of interpolation filters with different weights according to the feature amount. The image processing apparatus according to claim 1, wherein the interpolation processing unit changes a type of an interpolation filter used for the interpolation calculation in accordance with the feature amount. The interpolation processing unit applies a first interpolation filter when the feature amount indicates a blur amount equal to or less than a predetermined value, and when the feature amount indicates a blur amount larger than the predetermined value, The image processing apparatus according to claim 7, wherein a second interpolation filter that does not transmit a higher spatial frequency component than the first interpolation filter is applied. The image processing apparatus according to claim 8, wherein the first interpolation filter is an interpolation filter based on at least one of linear interpolation, nearest neighbor interpolation, and cubic interpolation, and the second interpolation filter is an interpolation filter based on Gaussian interpolation. . The image processing apparatus according to claim 1, wherein the feature amount extraction unit extracts the feature amount averaged with a surrounding partial region. The feature amount extraction unit extracts the feature amount corresponding to each of a plurality of pixels after scaling of the image,
The said interpolation process part calculates the pixel value of each of these pixels by performing each interpolation calculation using the interpolation parameter according to the said feature-value corresponding to each of these pixels. The image processing apparatus according to any one of claims. The image processing apparatus according to claim 1, wherein the interpolation processing unit applies an interpolation parameter in accordance with a ratio between the number of pixels of the image and the number of pixels after scaling. The image processing apparatus according to claim 12, wherein the interpolation processing unit increases the number of neighboring pixels used for the interpolation calculation when the reduction ratio of the image is larger. An image scaling process program for a computer,
A feature amount extraction step for extracting a feature amount related to blur from each of the plurality of partial regions in the image;
A program for executing an interpolation processing step for scaling the image by performing an interpolation operation on each of the plurality of partial regions using an interpolation parameter corresponding to the feature amount extracted from each of the plurality of partial regions.

Images