JP5340021B2 - Image processing apparatus, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To restore original edge intensity in image processing. <P>SOLUTION: The image processing apparatus which applies geometrical conversion along with pixel interpolation, includes: a pixel selection means to select a pixel most similar to a pixel of interest among a plurality of pixels used for interpolation of the pixel of interest in the pixel interpolation; a selected pixel edge intensity acquisition means that selects an area centering on the selected pixel selected by the pixel selection means and applies filter processing to the selected area to acquire the edge intensity of the selected pixel; and an interpolated pixel value update means that reflects the edge intensity acquired by the selected pixel edge intensity acquisition means in the pixel value of the pixel of interest according to a distance between the center of gravity of a plurality of pixels used for the interpolation and the pixel of interest. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium. It is particularly suitable for use when processing digital image data obtained by photographing with a digital camera.

  In recent years, due to the improvement in performance of digital cameras (hereinafter referred to as cameras), the resolution of captured images has become very high, and it has become possible to obtain high-quality photographic images that are not inferior to silver halide photographs. In general, image data obtained by photographing with a camera has various problems such as geometric distortion caused by distortion of a lens and inclination caused by a holding state of the camera at the time of photographing.

  Cameras can process image data relatively easily compared to silver halide photography. For example, an application for correcting geometric distortion (lens aberration correction processing) using information at the time of shooting such as a shooting lens has also appeared. Further, the tilt of the image data caused by a slight tilt of the camera can be easily leveled by rotating the image data (image rotation processing). By performing such lens aberration correction processing and image rotation processing, more uniform and high-quality image data can be obtained.

  Furthermore, when the printing paper size is very large, it is necessary to enlarge the image data. Since the image data photographed by the camera is composed of a set of rectangles called pixels, the enlarged image data is acquired by performing interpolation processing between pixels before enlargement. In general, bilinear interpolation or bicubic interpolation is often used for this interpolation processing.

  Interpolation processing is required not only for enlargement processing, but also for lens aberration correction processing and image rotation processing described above. An example of distortion correction is shown in FIG. As shown in FIG. 2A, image data corrected by performing distortion correction on the image data distorted in a barrel shape and cutting it out is acquired. Further, as shown in FIG. 2B, the corrected image data is obtained by performing distortion correction on the image data distorted in a pincushion shape and cutting out the image data. Distortion correction is correction that involves movement of the pixel position. When there is no pixel that completely matches the corrected coordinate, the pixel value is obtained by interpolating the surrounding pixels of the coordinate. The same applies to image rotation processing of image data.

JP-A-3-157785 JP 2004-153668 A

  However, it is known that the edge component is lost when the interpolation process is simply performed. Here, FIG. 3 shows an example of bilinear interpolation. In FIG. 3, when the pixel of interest is P, in bilinear interpolation, the pixel value of the pixel P is calculated from the surrounding pixels A to D. However, when the position of the pixel P coincides with the gravity center G of the surrounding pixels, the pixel value of the pixel P becomes an average of the surrounding 4 pixels. Therefore, the sharpness of the edge and the noise feeling are greatly reduced.

In order to solve this problem, in Patent Document 1 and Patent Document 2, edge detection is used when determining the pixel value of the target pixel, and weighting is performed in the edge direction when interpolation processing is performed, and resolution A technique for preventing deterioration is disclosed. However, even though the reduction in resolution due to the interpolation process can be reduced, it is difficult to restore the original edge strength as long as the interpolation process is executed.
The present invention has been made in view of the above-described problems, and enables image processing to restore the original edge strength.

The present invention is an image processing apparatus that performs a geometric transformation process accompanied by a pixel interpolation process, and a pixel that selects a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process A selection means, an edge detection means for detecting an edge in a region centered on the selected pixel selected by the pixel selection means, and a filter selection for selecting a filter shape according to the direction of the edge detected by the edge detection means And a value obtained by selecting a region centered on the selected pixel selected by the pixel selecting unit, and applying a filtering process by the filter shape selected by the filter selecting unit to the selected region obtaining the difference between the pixel values of the selected pixel, and the selected pixel edge strength obtaining means for obtaining an edge intensity of the selected pixel, a plurality of using the interpolation Depending on the distance between the center of gravity and the target pixel of pixels, and having an interpolation pixel value updating unit that reflects the edge intensities obtained by the selected pixel edge strength obtaining means to the pixel value of the pixel of interest.

  According to the present invention, by reflecting the edge strength lost by the interpolation processing to the interpolation pixel, it is possible to restore the original edge strength and obtain a more preferable processing result.

It is a figure which shows the outline of an image processing apparatus. It is a figure which shows an example of distortion aberration correction. It is a figure for demonstrating an example of bilinear interpolation. It is a flowchart which shows the flow of a distortion aberration correction process. It is a figure for demonstrating conversion of the image height before and after distortion correction. It is a flowchart which shows the flow of an image size calculation process. It is a figure for demonstrating the content of the image size calculation process. It is a flowchart which shows the flow of a distortion correction pixel value calculation process. It is a figure for demonstrating the content of a distortion correction pixel value calculation process. It is a figure which shows the outline | summary of an edge component restoration process. It is a flowchart which shows the flow of an interpolation pixel value calculation process. It is a figure which shows a response | compatibility of the filter shape for every edge pattern. It is a flowchart which shows the flow of a pixel value calculation process.

Embodiments according to the present invention will be specifically described below with reference to the drawings. In the present embodiment, a method for restoring edge strength using pixel interpolation processing accompanying distortion correction processing will be described.
(First embodiment)
FIG. 1 is a diagram showing an outline of an image processing apparatus. In this embodiment, a personal computer will be described as an example of the image processing apparatus. The present invention can also be implemented inside a single digital camera or video camera. The CPU 101 controls the operation of the entire image processing apparatus 100 and executes a program stored in the primary storage 102. The primary storage 102 is mainly a memory, and reads and stores programs and the like stored in the secondary storage 103. The secondary storage 103 is, for example, a hard disk. In general, the capacity of the primary storage 102 is smaller than the capacity of the secondary storage 103, and programs and data that cannot be stored in the primary storage 102 are stored in the secondary storage 103. Data that must be stored for a long time is also stored in the secondary storage 103. In the present embodiment, when realizing a processing procedure as will be described later, the CPU 101 reads a program stored in the secondary storage 103 into the primary storage 102 and executes it when executing the program.

The input device 104 is, for example, a mouse or a keyboard. A user of the image processing apparatus 100 can transmit an interrupt signal to the CPU 101 that is executing a program or the like via the input device 104.
The output device 105 is, for example, a monitor or a printer.
The reading device 106 reads image data captured by an imaging device such as a digital camera or a digital video camera having a known imaging device (CCD) or the like into a primary storage 102 or a secondary storage 103 via a recording medium.

  The characteristic data supply unit 107 includes a memory (not shown). The characteristic data supply unit 107 stores lens characteristic data corresponding to various lenses, specifically correction coefficients corresponding to the photographing lens, and supplies the correction coefficient to the CPU 101 as necessary. In the distortion correction (distortion correction processing) in the present embodiment, a correction function is determined in advance, and the correction coefficient acquired from the characteristic data supply unit 107 is applied to the correction function, so that the CPU 101 calculates a correction amount and performs correction. This is realized by transforming the image data according to the amount (geometric transformation process). Details of the distortion correction will be described later. The lens characteristic data is not limited to being stored in the characteristic data supply unit 107, and may be stored in the secondary storage 103 or an external storage device (not shown).

(Outline of distortion correction processing)
FIG. 4 is a flowchart showing the flow of distortion correction processing. This process is realized by the CPU 101 executing a program read into the primary storage 102. This process is executed using a correspondence table between the image height before distortion correction and the image height after distortion correction. In this embodiment, as shown in FIG. 5, the conversion from the image height before distortion correction to the image height after distortion correction is a forward conversion, and the conversion from the image height after distortion correction to the image height before distortion correction is performed. Inverse transformation. In addition to lens correction processing, image processing involving coordinate conversion generally performs image processing based on the concept of inverse conversion. The concept of forward conversion is good for continuous graphics, but in the case of a bitmap that is a digital image, the pixel before processing does not always correspond to the pixel after processing.

  First, in step S401, the CPU 101 acquires lens characteristic data from the characteristic data supply unit 107 using shooting time information (lens type, focal length, f value, etc.) of image data to be processed. The lens characteristic data for distortion correction in this embodiment is assumed to be inverse transformation data, that is, data of image height before distortion correction with respect to the image height after distortion correction.

Next, in step S402, the CPU 101 acquires the pixel pitch from the shooting time information of the processing target image data. The pixel pitch is a distance between pixels. In the present embodiment, the pixel pitch is represented by a distance in millimeters on the image sensor. The relationship between the distance lp expressed in pixel units, the pixel pitch pi, and the actual sensor distance lr can be expressed by the following equation.
lp × pi = lr (1)
If the pixel pitch pi is used, the position of the pixel corresponding to the image height expressed in millimeters can also be obtained in pixel units using the dimensions of the effective area of the image sensor.

Next, in step S403, the CPU 101 obtains the image size after distortion correction. Since distortion correction is a process applied by changing the image height of each pixel, the image size may change before and after distortion correction. The image size calculation process after distortion correction will be described later.
Next, in step S404, the CPU 101 calculates the distortion correction lookup table size fs by the following equation (2) using the image size (width fw pixel, height fh pixel) obtained in step S403. fs is the distance in pixels from the image center after distortion correction to each vertex.
fs = √ (((fw / 2) × (fw / 2)) + ((fh / 2) × (fh / 2))) ·· Equation (2)

  Next, in step S405, the CPU 101 performs spline interpolation on the lens characteristic data acquired in step S401 to generate a distortion correction lookup table having the distortion correction lookup table size fs obtained in step S404. Here, the input of the distortion correction lookup table is a distance in pixels from the image center after distortion correction. The output of the distortion correction lookup table is a distance in pixels from the center of the processing target image. In order to generate this distortion correction look-up table, it is necessary to convert the image height (in millimeters) of the lens characteristic data into an image height in units of pixels. For this conversion, the image data to be processed acquired in step S402 is obtained. The pixel pitch is used.

  Next, in step S406, the CPU 101 executes distortion correction pixel value calculation processing for each pixel constituting the image data having the image size obtained in step S402. Details of the distortion correction pixel value calculation process will be described later. After the pixel values are calculated for all the pixels, the process ends.

(Image size calculation process after distortion correction)
FIG. 6 is a flowchart showing the flow of image size calculation processing after distortion correction. This process corresponds to step S403 in the flowchart shown in FIG. Note that description will be made with reference to the drawing for explaining the image size calculation processing shown in FIG. 7 as appropriate.
First, in step S601, the CPU 101 acquires the width and height of an image before distortion correction from image data to be processed. Next, in step S602, the CPU 101 generates a distortion correction lookup table. The generation of this distortion correction lookup table is the same procedure as the processing described in steps S401 to S405 in the flowchart shown in FIG. 4, but the calculation processing in step S403 is omitted. The distortion correction lookup table size ds calculated in step S404 is obtained by the following equation using the width w and height h of the image before distortion correction.
ds = √ (((w / 2) × (w / 2)) + ((h / 2) × (h / 2)))) Equation (3)

In step S603, the CPU 101 obtains the maximum image height of the reference pixel before distortion correction from the distortion correction lookup table. Here, the maximum image height is equal to the maximum value of the output of the distortion correction lookup table. Since the distortion correction lookup table is expressed in units of pixels, the maximum image height is also in units of pixels.
In step S604, the CPU 101 generates a forward conversion lookup table from the distortion correction lookup table using the maximum image height. The forward conversion lookup table size rts is calculated by the following equation using the maximum value maxL of the distortion correction lookup table.
rts = [maxL + 1] .. Formula (4)
[] Is a Gaussian symbol.
Assume that the input of the forward conversion lookup table is an image height in pixel units before distortion correction, and the output of the forward conversion lookup table is an image height in pixel units after distortion correction. The output of the forward conversion lookup table may not be an integer value, but in this case, the distortion correction lookup table is linearly interpolated.

Next, in steps S605 and S606, the CPU 101 calculates the size of the effective area (see FIG. 7). That is, in step S605, when y in FIG. 7 is fixed to h / 2 and x is changed from 0 to w / 2, the distance from the origin is calculated and a forward conversion lookup table is drawn to correct the distortion. Calculate later coordinates. If the smallest value among the x components of the obtained coordinates is hW, the effective area width fw is twice the value of hW.
Next, in step S606, when x in FIG. 7 is fixed to w / 2 and y is changed from 0 to h / 2, a distance from the origin is calculated, a forward conversion lookup table is drawn, and distortion is performed. Calculate the corrected coordinates. Assuming that the smallest value among the y components of the obtained coordinates is hH, the height fh of the effective area is twice the value of hH.
By performing the processing as described above, the image size after the distortion correction can be obtained before the distortion correction is executed.

(Distortion correction pixel value calculation processing)
FIG. 8 is a flowchart showing the flow of distortion correction pixel value calculation processing. This process corresponds to step S406 in the flowchart shown in FIG. FIG. 9 is a diagram for explaining the content of the distortion correction pixel value calculation process.
The distortion correction pixel value calculation process is a process of calculating a pixel value of a designated pixel (i, j) out of display image data having a width fw pixel and a height fh pixel using a distortion correction lookup table. Yes (see FIG. 9).

First, in step S801, the CPU 101 obtains a distance (image height) in pixel units from the image center to the pixel (i, j).
Next, in step S802, the CPU 101 obtains a correction amount (image height before distortion correction) corresponding to the image height using a distortion correction lookup table. In some cases, the image height does not become an integer value. In this case, the correction amount is calculated by linear interpolation.

In step S803, the CPU 101 uses the correction amount obtained in step S802 to calculate coordinates on the image before distortion correction using the following equation.
P ′ (i, j) = d ′ × P (i, j) / d (5)
Here, P (i, j) is a direction vector from the image center to the pixel (i, j) after distortion correction. P ′ (i, j) is a direction vector from the center of the image to the coordinates before the pixel (i, j) is corrected for distortion. d is an image height in pixel units of the pixel (i, j) after distortion correction. d ′ is the image height in pixels of the pixel (i, j) before distortion correction calculated using the distortion correction lookup table. Note that the processing in step S803 corresponds to (1) shown in FIG.

Finally, in step S804, the CPU 101 determines the pixel value of the pixel (i, j) by performing interpolation using the pixel value in the vicinity of the coordinates corresponding to P ′ (i, j) on the image data. . At this time, as shown in FIG. 9 (2), the CPU 101 calculates the pixel value of P ′ (i, j) from the adjacent pixels of P ′ (i, j) by bilinear interpolation.
In step S804, if the CPU 101 simply performs pixel interpolation, the edge strength may decrease or the noise component may decrease. Therefore, in the present embodiment, one pixel is selected from the pixels used for interpolation, and the difference between the average of the surrounding pixels of the selected pixel and the selected pixel itself is regarded as the edge strength and reflected in the interpolation result.

Next, edge component restoration processing in the present embodiment will be described with reference to FIGS. 10 and 11. FIG. 10A is a diagram showing an outline of the edge component restoration process when the pixel value is calculated by bilinear interpolation. FIG. 10B is a diagram illustrating an example of a filter used for edge component restoration processing. FIG. 11 is a flowchart showing the flow of interpolation pixel value calculation processing. The flowchart shown in FIG. 11 corresponds to step S804 of the flowchart shown in FIG.
First, in step S1101, the CPU 101 calculates a pixel value bL by bilinear interpolation from four pixels in the vicinity of the pixel P ′. Specifically, as illustrated in FIG. 10A, the CPU 101 calculates a pixel value bL from the pixels A, B, C, and D that are in the vicinity of the pixel P ′ by bilinear interpolation.

Next, in step S1102, the CPU 101 selects the nearest pixel among the neighboring pixels. Specifically, as illustrated in FIG. 10A, the CPU 101 selects the pixel A. This processing corresponds to an example of processing by the pixel selection unit.
Next, in step S1103, the CPU 101 selects a 3 × 3 pixel area centered on the selected pixel (in this case, pixel A) as the selected area. Specifically, as illustrated in FIG. 10A, the CPU 101 selects a selection region 1000 including the pixels E, G, D, and I with the pixel A as the center. Subsequently, the CPU 101 obtains an output a ′ by applying a filter 1202 as shown in FIG. 10B to the selected region. This processing corresponds to an example of processing by the filter processing unit.
A filter 1202 shown in FIG. 10B calculates an average of pixel values of pixels included in the selected region. The filter indicates that the pixel corresponding to the position indicated by “1” is used. Therefore, since all of the filters 1202 are “1”, the average of the pixel values is calculated using all the pixels A to I in the selected region.

  In step S1104, the CPU 101 calculates a luminance difference diff obtained by subtracting the luminance value obtained from the value a ′ calculated in step S1103 from the luminance value of the selected pixel, and sets this as the edge intensity of the pixel A. This process corresponds to an example of a process performed by the selected pixel edge strength acquisition unit. Specifically, in the case shown in FIG. 10A, the luminance difference diff is a value obtained by subtracting the average luminance of the pixels A to I from the luminance value of the pixel A. The luminance difference diff is a value indicating the degree of variation of the selected pixels in the selected area. In the present embodiment, the edge strength is obtained as a deviation in order to reduce the calculation cost, but it is also conceivable to improve the accuracy by using a standard deviation or the like for the calculation of the edge strength. It is also conceivable to reduce the influence of noise by setting the selected area larger than 3 × 3 pixels and applying a larger filter.

Next, in step S1105, the CPU 101 uses the centroid G of the pixels A to D used in the interpolation processing in step S1101 and the distance lg between the pixels P ′, and uses the following equation to determine the final pixel P ′. The value p is calculated.
p = bL + m × (1−lg ² ) × 2 × diff (Equation 6)
Here, m is a coefficient of how much the edge component is reflected in the interpolation result, and is set to 0.5 in the present embodiment, for example. In the present embodiment, the value of m is a fixed value, but it goes without saying that m may be determined dynamically depending on the states of the pixels A to I, for example. Thus, by reflecting the edge strength of the selected pixel in the pixel P ′, the edge strength lost by the interpolation can be restored. The above-described processing corresponds to an example of processing by the interpolation pixel value update unit.

Note that when the position of the pixel P ′ is near the center of gravity of the four neighboring pixels, that is, approximately equidistant from all the pixels, the CPU 101 randomly selects one pixel from the four pixels in the vicinity of the pixel P ′. Specifically, when the distance between the center of gravity and the pixel P ′ is equal to or less than a predetermined distance (distance t or less), the CPU 101 randomly selects a pixel from among the pixels A to D. In the present embodiment, for example, t = 0.01 (pixels). Therefore, when t = 0, that is, when the center of gravity and the pixel of interest match, the CPU 101 selects a pixel at random.
Here, when the selected pixel is determined with a complete random value, there is a problem that the result changes every time an image is processed. Therefore, a value calculated using coordinates (i, j) is set as a seed value of a random number expression. For example, in this embodiment, the value calculated by the following equation is set as the seed s.
s = i × w + j (7)

Thus, by fixing the seed value for each pixel, it is possible to always obtain the same processing result. In addition, an unnatural pattern is prevented from appearing in the processing result by randomly selecting the selected pixel. For example, if the processing is such that the pixel located at the upper left is always selected when the position of the pixel P ′ completely coincides with the center of gravity, a specific pattern may appear.
By performing the processing as described above, it is possible to prevent a decrease in resolution due to the interpolation processing and obtain an image with a more preferable processing result.

(Second Embodiment)
In the first embodiment, filters having the same shape are always applied regardless of the state of the pixels in the selected region. However, in order to obtain the edge strength with higher accuracy, it is conceivable to perform edge detection on the selected region and change the filter shape in accordance with the detected edge.
In the present embodiment, a case where the size of the selection area is 3 × 3 pixels will be described, as in the first embodiment.

Here, the correspondence between the detected edge pattern and the filter shape for each edge pattern will be described with reference to FIG. This correspondence data is stored in the secondary storage 103, for example, and is referred to by the CPU 101 as necessary.
As shown in FIG. 12, for example, when the edge pixel is an edge pattern 1101 that appears in the selected pixel, the upper left pixel, and the lower right pixel, the CPU 101 applies the filter 1201 corresponding to the edge pattern 1101 with reference to the corresponding data. The filter 1201 is a filter that excludes the edge pixels of the upper left pixel and the lower right pixel in the selected region. That is, the CPU 101 can calculate the edge component with high accuracy by excluding the pixel having a value similar to the selected pixel from the selected region by adopting a filter shape that intersects the detected edge in the vertical direction. If no edge is detected or if an edge is detected but the edge pattern is not simple, the CPU 101 sets the edge pattern to “others” 1102 shown in FIG. 12, and the corresponding first implementation. A filter 1202 similar to the form is applied.

  FIG. 13 is a flowchart showing a flow of pixel value calculation processing according to the present embodiment. In the pixel value calculation process illustrated in FIG. 13, processes in steps S1301 and S1302 are added between steps S1102 and S1103 in the flowchart illustrated in FIG. 11 described in the first embodiment. Note that the processing in other steps is the same as that in the flowchart shown in FIG. 11, and the same step numbers are assigned and description thereof is omitted.

In step S1301, the CPU 101 detects an edge in the selected area. This processing corresponds to an example of processing by the edge detection unit. Subsequently, based on the detected edge, the CPU 101 detects an edge pattern that matches the corresponding data shown in FIG.
Next, in step S1302, the CPU 101 selects a filter shape corresponding to the detected edge pattern. This processing corresponds to an example of processing by the filter selection unit. Thereafter, in step S1103, the CPU 101 acquires the output a ′ by applying the selected filter to the selected region.
As described above, according to the present embodiment, it is possible to reflect the edge intensity on the interpolation pixel with higher accuracy by changing the filter applied to the selected region in consideration of the edge.

(Third embodiment)
In the first embodiment, the distortion correction has been described as the image processing accompanied by the interpolation processing. In distortion correction, the amount of pixel movement generally tends to increase from the center of the image to the periphery. An increase in the amount of movement means that the number of corrected pixels that fall between pixels before correction increases, and the edge strength tends to decrease. Therefore, when the correction processing is distortion correction, it is conceivable that the reflection rate of the edge strength is increased as the image height is higher in consideration of this tendency. Specifically, the function f having the coordinates (i, j) after distortion correction as an argument is added to the equation (6) of the first embodiment, and the following changes are made.
p = bL + m × (1−lg ² ) × 2 × diff × f (i, j) (8)
Here, f (i, j) is a distance d (i, j) from the center of the image to the pixel (i, j) when the movement amount of the pixel by distortion correction is moved in accordance with the square of the image height for the sake of simplicity. j), the pixel pitch pitch, and the maximum image height max_d on the image sensor can be expressed by the following equation.
f (i, j) = 1.0 + v ² , v = d (i, j) × pitch / max_d (9)

By adding f (i, j), the edge strength is greatly reflected. For example, in this embodiment, it is necessary to reduce the value of m in Expression (8). In this embodiment, m = 0.2.
By performing such processing, it is possible to obtain a more appropriate interpolation result by reflecting the edge strength stronger as it is closer to the peripheral portion of the image that is greatly stretched by distortion correction, and weaker as the edge portion is closer to the center portion of the image. it can.
As described above, according to the present embodiment, it is possible to obtain a processing result with higher accuracy by reflecting the edge strength on the interpolation pixel in consideration of the distortion correction characteristic.

(Fourth embodiment)
Instead of applying according to the image height as described in the third embodiment, it is also conceivable to change the reflection rate of the edge strength according to the amount of change in the image height due to distortion correction. Since the amount of change in image height varies depending on the lens, changing the reflection rate of the edge intensity according to the amount of change in image height can obtain a more accurate processing result than using the image height. It becomes possible.
In the present embodiment, the function g having the correction amounts d and d ′ used in the expression (5) as arguments is added to the expression (6) of the first embodiment, and the following changes are made.
p = bL + m × (1−lg ² ) × 2 × diff × g (d, d ′) Equation (10)
Here, g (d, d ′) is expressed by the following equation, for example.
g (d, d ′) = 1.0+ (d−d ′) / max_d (11)
max_d is the maximum movement amount in pixel units by aberration correction. For example, in the present embodiment, the image width w and the image height h before correction are used,
max_d = 0.5 × √ (((w / 2) × (w / 2)) + ((h / 2) × (h / 2)))) Equation (12). In addition, since the edge strength is greatly reflected by adding g (i, j), for example, in the present embodiment, it is necessary to reduce the value of m in Expression (10). In this embodiment, m = 0.1.

By performing such processing, a more appropriate interpolation result can be obtained by reflecting a strong edge strength in a portion greatly expanded by distortion correction, and reflecting a weak edge strength in a portion that does not change much.
As described above, according to the present embodiment, it is possible to obtain a processing result with higher accuracy by reflecting the edge intensity on the interpolation pixel in consideration of the change amount of the image height due to the distortion correction.

(Fifth embodiment)
Lens distortion is corrected by changing the image height. Therefore, each pixel moves along the direction vector of the target pixel from the center of the image. In the method described in the second embodiment, edge detection is performed in a state before distortion correction. However, since the pixel moves along the direction vector of the target pixel by applying distortion correction, an edge in a direction orthogonal to the direction vector is used. Is less reliable. Therefore, when two or more edges are detected during the edge detection process, it is conceivable that the direction vector of the target pixel and the inner product of each detected edge are taken and the edge having the larger inner product is selected.

In the present embodiment, by applying the inner product of the edge direction vector E and the target pixel direction vector D to the application result of the Sobel operator corresponding to eight directions that are diagonally up, down, left and right with the target pixel as the center, an absolute value is obtained. Increase the accuracy of edge detection. For example, the CPU 101 calculates the edge strength δ using the following equation, and determines that there is an edge in that direction when the maximum of the eight strengths in the eight directions is equal to or greater than a predetermined threshold. This processing corresponds to an example of processing by the edge selection unit.
δ = Sobel (dir) × ((E · D) /2+0.5) ·· Equation (13)
Here, Sobel (dir) is the output of the Sobel operator corresponding to 8 directions. By performing such processing, it becomes possible to detect the edge after distortion correction more correctly and correct the image satisfactorily.

Each means in the image processing apparatus and each step of the image processing method in the embodiment of the present invention described above can also be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable recording medium on which this program is recorded are included in the present invention.
In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, or recording medium. Specifically, the present invention may be applied to a system including a plurality of devices.

  The present invention supplies a software program for realizing the functions of the above-described embodiments directly or remotely to a system or apparatus. And the case where it is achieved also by reading and executing the supplied program code by the computer of the system or apparatus is included.

  101 CPU, 107 Lens characteristic data supply unit

Claims (18)

An image processing apparatus that performs a geometric transformation process with a pixel interpolation process,
A pixel selection means for selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
Edge detection means for detecting an edge in a region centered on the selected pixel selected by the pixel selection means;
Filter selection means for selecting a filter shape according to the direction of the edge detected by the edge detection means;
A value obtained by selecting a region centered on the selected pixel selected by the pixel selecting unit and applying a filter process based on the filter shape selected by the filter selecting unit to the selected region and the selected pixel A selected pixel edge strength obtaining means for obtaining a difference between the pixel values of the selected pixels and obtaining an edge strength of the selected pixel;
Interpolated pixel value updating means for reflecting the edge strength acquired by the selected pixel edge strength acquisition means on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. An image processing apparatus. In the case where two or more edges are detected by the edge detection means, an edge selection means for selecting one edge using the inner product of the direction vector from the image center to the target pixel and the direction vector of each edge is further provided. The image processing apparatus according to claim 1 , further comprising: An image processing apparatus that performs a geometric transformation process with a pixel interpolation process,
A pixel selection means for selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition unit that selects a region centered on the selection pixel selected by the pixel selection unit, and acquires edge strength of the selection pixel by applying a filtering process to the selection region;
Interpolated pixel value updating means for reflecting the edge strength acquired by the selected pixel edge strength acquiring means on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
If the center of gravity with the target pixel of a plurality of pixels used for the interpolation are matched, the pixel selecting means, images processing apparatus characterized by selecting a random pixel of a plurality of pixels used for the interpolation. An image processing apparatus that performs a geometric transformation process with a pixel interpolation process,
A pixel selection means for selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition unit that selects a region centered on the selection pixel selected by the pixel selection unit, and acquires edge strength of the selection pixel by applying a filtering process to the selection region;
Interpolated pixel value updating means for reflecting the edge strength acquired by the selected pixel edge strength acquiring means on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
When the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is equal to or less than a predetermined distance, the pixel selection unit randomly selects a pixel from the plurality of pixels used for the interpolation. images it is processing device according to. When a pixel is randomly selected from the plurality of pixels used for the interpolation by the pixel selection unit, the random value used for determining the selected pixel uses a value calculated using the coordinates of the target pixel as a seed value. The image processing apparatus according to claim 3 or 4 . An image processing apparatus that performs a geometric transformation process with a pixel interpolation process,
A pixel selection means for selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition unit that selects a region centered on the selection pixel selected by the pixel selection unit, and acquires edge strength of the selection pixel by applying a filtering process to the selection region;
Interpolated pixel value updating means for reflecting the edge strength acquired by the selected pixel edge strength acquiring means on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
The interpolated pixel value updating means reflects the edge intensity on the pixel value of the target pixel according to the image height of the target pixel in addition to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. images processing device according to claim. An image processing apparatus that performs a geometric transformation process with a pixel interpolation process,
A pixel selection means for selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition unit that selects a region centered on the selection pixel selected by the pixel selection unit, and acquires edge strength of the selection pixel by applying a filtering process to the selection region;
Interpolated pixel value updating means for reflecting the edge strength acquired by the selected pixel edge strength acquiring means on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
When the geometric conversion process is a distortion correction process, the interpolated pixel value update means adds the image height of the target pixel that changes due to the distortion correction in addition to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. images processing apparatus characterized by reflecting the edge intensity on the pixel value of the pixel of interest in accordance with the amount of change. An image processing method for performing geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
An edge detection step of detecting an edge in a region centered on the selected pixel selected by the pixel selection step;
A filter selection step of selecting a filter shape according to the direction of the edge detected by the edge detection step;
A value obtained by selecting a region centered on the selected pixel selected by the pixel selecting step and applying a filter process based on the filter shape selected by the filter selecting step to the selected region and the selected pixel A selected pixel edge strength obtaining step for obtaining a difference between the pixel values of the selected pixels and obtaining an edge strength of the selected pixel;
An interpolated pixel value updating step for reflecting the edge strength acquired by the selected pixel edge strength acquiring step in the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. An image processing method. To a computer that performs geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
An edge detection step of detecting an edge in a region centered on the selected pixel selected by the pixel selection step;
A filter selection step of selecting a filter shape according to the direction of the edge detected by the edge detection step;
A value obtained by selecting a region centered on the selected pixel selected by the pixel selecting step and applying a filter process based on the filter shape selected by the filter selecting step to the selected region and the selected pixel A selected pixel edge strength obtaining step for obtaining a difference between the pixel values of the selected pixels and obtaining an edge strength of the selected pixel;
An interpolation pixel value update step that reflects the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is executed. Program to let you. An image processing method for performing geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step for reflecting the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
An image processing method, wherein, when the centroids of a plurality of pixels used for the interpolation coincide with a target pixel, the pixel selection step randomly selects a pixel from the plurality of pixels used for the interpolation. An image processing method for performing geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step for reflecting the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
When the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is equal to or less than a predetermined distance, the pixel selection step randomly selects a pixel from the plurality of pixels used for the interpolation. An image processing method. An image processing method for performing geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step for reflecting the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
In the interpolation pixel value update step, in addition to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel, the edge strength is reflected in the pixel value of the target pixel according to the image height of the target pixel. A featured image processing method. An image processing method for performing geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step for reflecting the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel. And
When the geometric conversion process is a distortion correction process, in the interpolation pixel value update step, in addition to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel, the image height of the target pixel that changes due to the distortion correction An image processing method, wherein the edge intensity is reflected in a pixel value of a target pixel in accordance with a change amount of. To a computer that performs geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step that reflects the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is executed. Let
A program characterized by randomly selecting a pixel from the plurality of pixels used for the interpolation in the pixel selection step when the center of gravity of the plurality of pixels used for the interpolation coincides with the target pixel. To a computer that performs geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step that reflects the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is executed. Let
When the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is equal to or less than a predetermined distance, the pixel selection step randomly selects a pixel from the plurality of pixels used for the interpolation. Program. To a computer that performs geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step that reflects the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is executed. Let
In the interpolation pixel value update step, in addition to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel, the edge strength is reflected in the pixel value of the target pixel according to the image height of the target pixel. A featured program. To a computer that performs geometric transformation processing with pixel interpolation processing,
A pixel selection step of selecting a pixel closest to the target pixel among a plurality of pixels used for interpolation of the target pixel in the pixel interpolation process;
A selection pixel edge strength acquisition step of selecting a region centered on the selection pixel selected by the pixel selection step and acquiring edge strength of the selection pixel by applying a filtering process to the selection region;
An interpolation pixel value update step that reflects the edge strength acquired in the selected pixel edge strength acquisition step on the pixel value of the target pixel according to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel is executed. Let
When the geometric conversion process is a distortion correction process, in the interpolation pixel value update step, in addition to the distance between the center of gravity of the plurality of pixels used for the interpolation and the target pixel, the image height of the target pixel that changes due to the distortion correction The edge intensity is reflected in the pixel value of the pixel of interest according to the amount of change. The computer-readable recording medium which recorded the program of any one of Claim 9, 14 thru | or 17 .

Images