JP6972089B2 - Image processing equipment, image processing methods, and programs - Google Patents

Description

The present invention relates to a technique for performing geometric transformation on data recorded according to a two-dimensional coordinate position.

Conventionally, in devices that handle digital images such as digital cameras, printers, and projectors, geometric transformation processing by coordinate transformation is often executed for digital images. For example, in a printer, an enlargement process (high resolution process) of an input image is performed according to a print resolution. With a surveillance camera, when shooting from an oblique direction toward the subject, a rectangular subject such as a building is distorted into a trapezoid, so the trapezoid is corrected to be rectangular (perspective correction), and the area of interest in the image is enlarged. There are things to do. In the projector, keystone correction processing is performed so that the projected image is aligned with the projection surface. In the geometric transformation process, linear coordinate transformations such as enlargement / reduction, rotation, skew and translation are called linear transformations, and other coordinate transformations are called non-linear transformations.

Geometric transformations are often implemented in hardware for speed. Linear transformation can be implemented with relatively few arithmetic resources, but nonlinear transformation requires complicated operations such as division, so there is a problem that the circuit scale becomes large. In particular, with the recent increase in image resolution (8K, etc.), the range in which coordinates can be obtained has expanded, and a large-scale arithmetic circuit or the like is required to realize highly accurate coordinate conversion.

Therefore, for example, in Patent Document 1, the image is first divided into a plurality of triangular regions. Then, affine transformation (linear transformation) is performed from the correspondence of the vertices for each divided region. As a result, a transformation that approximates the desired nonlinear transformation is realized for the entire image. Further, in Patent Document 2, a correction table for distortion correction corresponding to all shooting angles is prepared in advance in the memory in the X-ray diagnostic apparatus. Non-linear conversion is realized by setting a correction table corresponding to the current shooting angle in the address generator as needed.

Japanese Unexamined Patent Publication No. 2004-227470 Japanese Unexamined Patent Publication No. 2000-224481

However, in the method described in Patent Document 1, it is necessary to set parameters for affine transformation for each region, and the calculation of the parameters itself becomes complicated. In addition, inconsistency at the boundaries of the area may cause deterioration of image quality. The method described in Patent Document 2 is effective for a device having a limited shooting angle such as an X-ray diagnostic device, but a correction table corresponding to all shooting angles in a device having a high degree of freedom in shooting angle such as a general camera. It is not realistic to prepare.

The present invention has been made in view of the above circumstances, and an object of the present invention is to realize highly accurate geometric transformation with relatively few arithmetic resources.

The image processing apparatus according to the present invention is an image processing apparatus that performs geometric transformation to the recorded data according to the two-dimensional coordinate position, the conversion process from the first coordinate to a second coordinate, the first In order to combine the result of the linear transformation that transforms from the first coordinate to the third coordinate and the result of the nonlinear transformation that transforms from the first coordinate to the fourth coordinate, the linear transformation and the nonlinear control means for determining the transformation parameters of the transformation, and linear transformation means for calculating said third coordinate in the linear conversion based on the conversion parameter of the linear transformation, with the non-linear conversion based on the conversion parameter of the non-linear transformation, the It is characterized by having a non-linear transformation means for calculating a fourth coordinate and a synthesis means for synthesizing the third coordinate and the fourth coordinate to calculate the second coordinate.

According to the present invention, highly accurate geometric transformation can be realized with relatively few arithmetic resources.

It is a figure which shows the structural example of the image processing part in Embodiment 1. FIG. It is a figure which shows the example of the geometric transformation in Embodiment 1. FIG. It is a figure for demonstrating the intermediate quadrangle in Embodiment 1. FIG. It is a figure which shows the coordinate transformation example of the four vertices of a quadrangle in Embodiment 1. FIG. It is a figure for demonstrating the decomposition method of the coordinate transformation process in Embodiment 1. FIG. It is a figure which shows the structural example of the coordinate synthesis part in Embodiment 2. It is a figure for demonstrating the decomposition method of the coordinate transformation process in Embodiment 2. It is a figure which shows the structural example of the image synthesis part in Embodiment 3. FIG. It is a flowchart of conversion parameter calculation in Embodiment 1. It is a figure which shows the example of the geometric transformation in Embodiment 5.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention relating to the scope of claims, and all combinations of features described in the present embodiments are essential for the means for solving the present invention. Is not always.

<Embodiment 1>
(Structure of image processing unit)
FIG. 1 is a diagram showing a configuration example of the image processing unit 100 in the present embodiment. In this embodiment, an example of streaming and outputting the image after geometric transformation to the input image on the memory will be described. In the present embodiment, the coordinates in the image before the geometric transformation are calculated from the coordinates of each pixel in the image after the geometric transformation, and the pixel value corresponding to the calculated coordinates of the image before the geometric transformation is calculated from the input image.

The coordinates of the image after geometric transformation are input as the first coordinates (Ix, Iy) from the coordinate input port 150 in raster order. Here, assuming that the width of the image after geometric transformation is W pixel and the height is H pixel, the image after geometric transformation is composed of pixels of W pixel × H pixel. The first coordinates (Ix, Iy) are input in order while incrementing by 1 from 0.5 to (W-0.5) with Iy = 0.5. When Ix reaches (W-0.5), Iy is incremented by 1, and the value in which Ix is incremented in order is input as in the case of Iy = 0.5. Here, since the position of the center of gravity of the pixel is used as the coordinates, it is incremented by 1 starting from (Ix, Iy) = (0.5, 0.5). The coordinate linear conversion unit 110 and the coordinate nonlinear conversion unit 120 perform predetermined linear conversion processing and nonlinear conversion processing on the first coordinates (Ix, Iy) input from the coordinate input port 150, respectively. That is, the coordinate linear conversion unit 110 performs linear conversion on the first coordinates (Ix, Iy) and outputs the coordinates (Fx, Fy) of the linear conversion result. The coordinate non-linear conversion unit 120 performs non-linear conversion on the first coordinates (Ix, Iy) and outputs the coordinates (Gx, Gy) of the non-linear conversion result. Here, the linear transformation means a linear coordinate transformation such as enlargement / reduction, rotation, skew, and translation, and the non-linear transformation means other transformations.

In the coordinate synthesis unit 130, the coordinates (Fx, Fy) of the linear conversion result output from the coordinate linear conversion unit 110 and the coordinates (Gx, Gy) of the non-linear conversion result output from the coordinate nonlinear conversion unit 120 are combined. , The second coordinates (Hx, Hy) are calculated. The calculated second coordinates are coordinates indicating the position in the input image before the geometric transformation, and are passed to the image input / output unit 140. The conversion parameters used for the linear conversion of the coordinate linear conversion unit 110 and the non-linear conversion of the coordinate non-linear conversion unit 120 are obtained and set by the control unit 180 (CPU or the like), respectively. Details of the conversion parameter settings will be described later.

The image input / output unit 140 can access an external memory (not shown) that temporarily stores the input image, and externally inputs the pixel value located at the coordinates (Hx, Hy) in the input image via the image input port 160. Get from memory. The acquired pixel value is output to the image output port 170 as the pixel value of the first coordinate (Ix, Iy). The coordinate values Hx and Hy calculated by the coordinate synthesizing unit 130 are not necessarily integers. Therefore, when each coordinate value is not an integer, the pixel values of a plurality of pixels near the coordinates (Hx, Hy) in the input image are acquired from the external memory and the interpolation calculation is performed to correspond to the coordinates (Hx, Hy). Find the pixel value. The pixels output to the image output port 170 may be further input to another image processing unit such as filter processing, or may be written back to the external memory as they are. Here, each configuration of the image processing unit 100 other than the control unit 180 is implemented as hardware (circuit).

(Example of geometric transformation)
In this embodiment, an example of perspective correction (tilt correction) will be described as a specific example of geometric transformation. FIG. 2 is a diagram showing perspective correction. FIG. 2A shows a quadrangle before perspective correction, and FIG. 2B shows a quadrangle after perspective correction. The quadrangle 210 before perspective correction has a trapezoidal shape, and the coordinates of the four vertices are (0,0), (1600,200), (1600,1000), and (0,1200), respectively. The rectangle 220 after perspective correction is a rectangle, and the coordinates of the four vertices are (0,0), (1600,0), (1600,1200), and (0,1200), respectively. When a shooting device such as a digital camera shoots with the optical axis tilted with respect to the subject, the obtained shot image becomes an image in which the subject is distorted as shown in FIG. 2A. Therefore, by performing perspective correction on the captured image, the subject is converted into an image captured in parallel from the front as shown in FIG. 2 (b). The captured image itself to be corrected is data in which pixel values are stored in each pixel and is a quadrangle. However, a quadrangle as shown in FIG. 2A, which is a partial area included in the captured image, is shown in FIG. 2B. ), An image with reduced distortion can be obtained. What kind of perspective correction is performed on the shot image to be processed is determined by the mode of the shooting device at the time of shooting and the designation by the user.

In the present embodiment, it is assumed that the image data to be corrected is stored in the external memory. The coordinates of the corrected quadrangle 220 are input to the coordinate input port 150 of FIG. 1 in raster order. That is, the coordinates of the corrected quadrangle 220 are input as the first coordinates (Ix, Iy), and the coordinates of the corresponding uncorrected quadrangle 210 are calculated as the second coordinates (Hx, Hy). In the present embodiment, the non-linear coordinate conversion of the first coordinate (Ix, Iy) ⇒ second coordinate (Hx, Hy) is decomposed into a linear component and a non-linear component, and the coordinate linear conversion unit 110 and the coordinate nonlinear conversion unit 120 are decomposed. Then, linear transformation and non-linear transformation are performed for the first coordinates (Ix, Iy), respectively. Then, the second coordinates (Hx, Hy) are calculated by synthesizing the obtained linear transformation result and the non-linear transformation result.

(Coordinate conversion process)
FIG. 3 is a diagram for explaining a method of realizing a coordinate transformation by a linear transformation and a non-linear transformation in the present embodiment. The control unit 180 first decomposes the coordinate transformation into a linear transformation and a non-linear transformation, and as shown in FIG. 3, the control unit 180 first defines a quadrangle 310 (hereinafter, P3'as intermediate data by P0', P1', P2', and P3'. (Called an intermediate quadrangle) is set. The control unit 180 uses an intermediate quadrangle 310 based on the quadrangle 210 before perspective correction defined by the points P0, P1, P2, and P3 and the quadrangle 320 after perspective correction defined by points Q0, Q1, Q2, and Q3. To decide. The linear conversion process converts the quadrangle after perspective correction to the intermediate quadrangle, and the non-linear conversion process converts the intermediate quadrangle to the quadrangle before perspective correction. It is desirable to perform the main conversion process by the linear conversion process and reduce the number of operations of the nonlinear conversion process. Therefore, the control unit 180 is a quadrangle that is within the range of the linear transformation from the quadrangle 220 that is composed of the first coordinates and that is close to the shape of the quadrangle 210 (P0 / P1 / P2 / P3) that is composed of the second coordinates. To calculate the intermediate quadrangle. The intermediate quadrangle 310 is composed of the linear transformation result of the coordinate linear transformation unit 110.

FIG. 4 is a diagram showing an example of coordinate conversion of the four vertices of a quadrangle. In FIG. 4, the column 410 is a column showing the first coordinates, and shows the coordinates of the four vertices of the quadrangle 220 composed of the first coordinates. Column 430 is a column showing the second coordinates, and shows the coordinates of the four vertices of the quadrangle 210 composed of the second coordinates. In column 420, the linear conversion result is shown in the upper row and the nonlinear transformation result is shown in the lower row for each coordinate of column 410, and the sum of the coordinates in the upper row and the lower row is the four vertex coordinates of the quadrangle 210 shown in column 430. That is, the non-linear conversion of the first coordinate (Ix, Iy) ⇒ the second coordinate (Hx, Hy) is changed to the linear transformation of the first coordinate (Ix, Iy) ⇒ the coordinate after the linear conversion (Fx, Fy). Coordinates of 1 (Ix, Iy) ⇒ Coordinates after non-linear conversion (Gx, Gy) are decomposed into two non-linear conversions.

The configuration of the coordinate linear conversion unit 110 and the coordinate nonlinear conversion unit 120 that realize the above conversion will be described in detail. Here, the coordinate linear transformation unit 110 when the affine transformation, which is a typical linear transformation, is applied, will be described. A transformation that can be expressed by an affine transformation is called a linear transformation, and a transformation that cannot be expressed by an affine transformation is called a non-linear transformation. It is assumed that the coordinate linear conversion unit 110 is a circuit having a resource capable of calculating the affine transformation function shown in the equation (1).

Here, the affine coefficients a, b, c, d, e, and f are conversion parameters set by the control unit 180. To obtain the intermediate square 310 shown in FIG. 3, set a = 1.0, b = 0.0, c = 0.0, d = 0.833, e = 0.0, f = 100.0. This makes it possible to achieve the desired linear transformation.

The coordinate non-linear conversion unit 120 realizes non-linear conversion by UV mapping using bilinear interpolation based on the UV values of the four vertices of the quadrangle. The nonlinear transformation function in this case is shown in Eq. (2).

Here, the parameters (U0, V0), (U1, V1), (U2, V2), (U3, V3) are the four-vertex coordinates (0,0), (W, 0), which are the first coordinates. These are the coordinates when W, H) and (0, H) are non-linearly converted. Each parameter (U0, V0), (U1, V1), (U2, V2), (U3, V3) is a conversion parameter set by the control unit 180, respectively. In the present embodiment, as shown in FIG. 4, since the first coordinates (column 410) are converted into differences (lower row of column 420) by non-linear transformation, the control unit 180 has the following four vertices of the quadrangle. The desired non-linear conversion can be achieved by setting the UV value. That is, (U0, V0) = (0.0, -100.0), (U1, V1) = (0.0, 100.0), (U2, V2) = (0.0, -100.0). ), (U3, V3) = (0.0, 100.0). Further, in the present embodiment, W = 1600 and H = 1200 are set, but the present invention is not limited to this, and division may be realized by a shift operation by limiting W and H to powers of 2 such as 1024 and 4096. ..

In the coordinate synthesizing unit 130, the coordinates (Fx, Fy) of the linear transformation result according to the equation (1) and the coordinates (Gx, Gy) of the nonlinear transformation result according to the equation (2) are added. That is, it is represented by the equation (3).

Next, the calculation accuracy of the coordinate transformation will be described. As described above, the control unit 180 sets the coordinates (Fx, Fy) after the linear transformation obtained by the linear transformation to a value as close as possible to the second coordinates (Hx, Hy). As a result, the range (range) in which the absolute value of the coordinates (Gx, Gy) after the non-linear transformation, which is the difference, can be taken can be made smaller than the coordinates (Fx, Fy) after the linear transformation. For example, in the above perspective correction example, as shown in FIG. 4, the absolute value of the coordinates (Fx, Fy) after the linear transformation is 10 times or more the absolute value of the coordinates (Gx, Gy) after the non-linear transformation. Considering that the calculation error is proportional to the absolute value of the calculation result, the error included in the second coordinates (Hx, Hy) is dominated by the influence of the error caused by the coordinate linear conversion unit 110. Therefore, by providing the calculation accuracy of the coordinate linear conversion unit 110 to be higher than the calculation accuracy of the coordinate nonlinear conversion unit 120, highly accurate coordinate conversion can be realized. Here, the calculation accuracy may be the number of bits after the decimal point if it is a fixed point number, or it may be the number of bits in the mantissa part if it is a floating point number.

(Setting of conversion parameters)
The control unit 180 decomposes the coordinate conversion process into linear transformation and non-linear transformation based on the acquired four-vertex coordinates. FIG. 9A is a flowchart of the process executed by the control unit 180. The control unit 180 is a CPU, and by reading a program capable of realizing the flowchart shown in FIG. 9 from a ROM (not shown), subsequent processing is realized by software. First, in step S901, the control unit 180 acquires four vertices that define the quadrangle before perspective correction and four vertices that define the quadrangle after perspective correction. Here, it is assumed that the building, which is the main subject, is detected from the images to be corrected, and the four vertices of the building are generated so as to be an image without distortion. Therefore, the control unit 180 acquires the coordinates of the four vertices of the building, which is the subject, as the four vertices that define the quadrangle before the perspective correction. Further, the four vertices after the perspective correction are determined in advance according to the mode at the time of shooting, the user designation, the image to be corrected, and the like.

In step S902, the control unit 180 determines an intermediate quadrangle from the two four vertices acquired in step S901. FIG. 9B is a more detailed flowchart of the process in step S902. First, in step S905, the vertices Q0 to Q3 of the quadrangle after perspective correction are set as the initial values of the candidate intermediate quadrangle. Next, in step S906, the control unit 180 calculates the difference between each of the coordinates of the candidate intermediate quadrangle and each of the vertices P0 to P3 of the quadrangle before perspective correction. At this time, the vertices for which the difference is calculated are the vertices at the positions closest to each other.

Next, in step S907, a quadrangle that becomes the next candidate intermediate quadrangle is searched for. Specifically, the operation of bringing the X coordinate or the Y coordinate of at least one vertex of the current candidate intermediate quadrangle closer to the vertex of the corresponding uncorrected quadrangle is performed. At this time, the above operation is performed within the range of the linear transformation. If a new intermediate quadrangle capable of such an operation is found, the process proceeds to step S908, the next candidate intermediate quadrangle is set, and the process returns to step S906. If there is no new candidate intermediate quadrangle, the process proceeds to step S909 assuming that the differences between all the candidate intermediate quadrilaterals have been calculated. When the difference between the vertices P0 to P3 of the quadrangle before perspective correction is calculated for all the candidate intermediate quadrilaterals, the candidate intermediate quadrangle having the minimum difference is determined as the intermediate quadrangle in step S909. In the example of FIG. 3, the coordinates of the four vertices of the intermediate quadrangle 310 are (0,100), (1600,100), (1600,1100), (0,1100).

Returning to FIG. 9A, in step S903, the control unit 180 sets the affine coefficients a, b, c, d, e, and f, which are parameters for being executed by the linear transformation process, to the equation (1) and the intermediate square. It is calculated based on the coordinates, and parameters are set in the coordinate linear transformation unit 110.

Further, in step S904, the control unit 180 calculates the parameters to be executed by the nonlinear conversion process based on the coordinates of the four vertices before the perspective correction and the coordinates of the intermediate quadrangle, and sets the parameters in the coordinate nonlinear conversion unit 110. .. The coordinate non-linear conversion unit 120 performs conversion processing up to the intermediate quadrangle 310 composed of the linear conversion result and the quadrangle 210 composed of the second coordinates. Here, the control unit 180 calculates the parameter from the difference between the intermediate quadrangle 310 composed of the linear transformation result and the quadrangle 210 composed of the second coordinates. In the example of FIG. 3, the difference between the four-vertex coordinates of the intermediate quadrangle 310 and the four-vertex coordinates of the quadrangle 210 is (P0-P0'), (P1-P1'), (P2-P2'), (P3-P3'). ). The parameters are calculated by substituting each difference into (U0, V0), (U2, V2), (U1, V1), and (U3, V3) of the equation (2).

As described above, the decomposition of the conversion process by the control unit 180 is completed. As described above, the control unit 180 sets the decomposition result (conversion parameter) calculated before the start of the coordinate conversion process in the coordinate linear conversion unit 110 and the coordinate nonlinear conversion unit 120, respectively. As described above, it is desirable from the viewpoint of accuracy that the absolute value of the coordinates (Gx, Gy) of the nonlinear conversion result is small. Therefore, in the present embodiment, the absolute value sum of the distances between the four vertices of the intermediate quadrangle and the four vertices of the quadrangle 210 which is the final calculation result (| P0-P0'| + | P1-P1'| + | P2-P2' An intermediate quadrangle 520 having the minimum | + | P3-P3'|) was obtained.

(Effect of Embodiment 1)
Next, the arithmetic resources that realize these coordinate transformation processes will be described. In general, when a non-linear conversion is realized by itself, the non-linear conversion involves complicated operations such as division, and therefore the circuit scale becomes large in order to achieve high accuracy. In the configuration of this embodiment, even if the accuracy of the non-linear conversion is lowered, the influence is small, so that it can be realized on a relatively small circuit scale. In this embodiment, a case where each of the coordinate linear conversion unit 110, the coordinate nonlinear conversion unit 120, the coordinate composition unit 130, and the image input / output unit 140 is realized by a circuit has been described as an example. However, even when it is executed as a program without hardware, the effect can be obtained by performing the same calculation. When the calculation cost for performing a complicated calculation with high accuracy for coordinate conversion is high, an unfavorable situation such as an increase in calculation delay time and an increase in power consumption is caused. Therefore, by applying the present embodiment, it is possible to reduce the calculation cost and improve the above situation.

As described above, in the above embodiment, the desired coordinate conversion process is decomposed into two conversions, the coordinates are converted independently, and then the results of both are combined. The two conversions are different in calculation accuracy, range of calculation result, and calculation function (formula (1) and formula (2)), respectively. This made it possible to achieve both the accuracy of the final calculation result and the circuit scale.

<Modification 1>
In the first embodiment, an example in which the image after geometric transformation is streamed to the input image on the memory has been described, but the present invention is not limited to this. For example, conversely, the image after geometric transformation may be written in the memory for the image of the streaming input. Generally, when either the input image or the output image is streaming (raster order) and the other is random access, the coordinates on the streaming side are input from the coordinate input port 150 and random based on the result of converting the coordinates. It may be configured to access.

In this modification, the case where the pixels of the input image are input from the image input port 160 in the order of raster will be described. In the first embodiment, the coordinates of the output image are input to the coordinate input port 150 as the first coordinates, but in this modification, the coordinates of the input image are input as the first coordinates. In this case, the calculated second coordinate becomes the coordinate of the output image and is passed to the image input / output unit 140. Then, the image input / output unit 140 writes the pixels of the input image to the memory address corresponding to the coordinates of the passed output image. The image input / output unit 140 may be provided with a line buffer for temporarily storing images for several lines.

<Embodiment 2>
In the first embodiment, all of the candidate intermediate quadrilaterals are different from the four vertices before the perspective correction, and the candidate with the smallest difference is determined as the intermediate quadrangle. In the second embodiment, a method of decomposing the coordinate conversion process so that the absolute value of the coordinates (Gx, Gy) of the nonlinear transformation result becomes small by using three vertices out of the four vertices before the perspective correction will be described. FIG. 5A is a diagram for explaining a decomposition method of the coordinate conversion process in the present embodiment. In general, it is possible to perform a coordinate transformation process that aligns any three points within the range of linear transformation. Therefore, the control unit 180 calculates a quadrangle obtained as a result of linear transformation so as to match three points (P0, P1, P3) among the four vertices of the quadrangle 210 composed of the second coordinates as an intermediate quadrangle. Therefore, in the second embodiment, three points (Q0, Q1, Q3) out of the four vertices of the quadrangle 220 composed of the first coordinates are three points (Q0, Q1, Q3) out of the four vertices of the quadrangle 210 composed of the second coordinates. It is linearly converted into an intermediate quadrangle 510 according to P0, P1, P3). The four vertices of the intermediate quadrangle 510 are P0, P1, P2', and P3.

FIG. 9C is a flowchart of the process executed by the control unit 180 in the second embodiment. First, in step S910, the four vertices of the quadrangle 210 composed of the second coordinates are acquired, and any three vertices (here, P0, P1, P3) are selected from the four vertices.

In step S911, the control unit 180 solves the equation obtained by substituting the coordinates of Q0, Q1, Q3 and the coordinates of P0, P1, P3 into (Ix, Iy) and (Fx, Fy) of the equation (1), respectively. By doing so, the affine coefficient is obtained. The control unit 180 sets the calculated affine coefficient in the coordinate linear conversion unit 110. Next, in step S912, the control unit 180 also calculates the coordinates of the remaining one point (P2') of the intermediate quadrangle 510 using the obtained affine coefficient.

In step S913, the control unit 180 uses the conversion parameters of the coordinate non-linear conversion unit 120 ((U0, V0), (U1, V1), (U2, V2), (U3, V3) in the equation (2) representing the non-linear conversion function). ) Is calculated and set based on the coordinates of P2'and P2. Specifically, the coordinates of (P2-P2') are set in (U2, V2) of the equation (2), and the coordinates of (U0, V0), (U1, V1), (U3, V3) of the equation (2) are set. (0.0, 0.0) should be set. As described above, the decomposition process of the coordinate conversion process in the second embodiment is completed. In this case, since only one point (P2'is set to P2) is adjusted by non-linear transformation, the calculation amount of the decomposition itself (that is, the load of the control unit 180) is relatively small.

Further, another decomposition method may be used. In FIG. 5B, focusing on the overlapping area (hatched area), the ratio of the area overlapping with the quadrangle 210, which is the final calculation result, is the maximum (or approximately the maximum) in the total area of the intermediate quadrangle. We are looking for an intermediate quadrangle 530.

The above intermediate quadrangles 510, 520, and 530 are all within the range of linear transformation from the quadrangle 220 shown in FIG. 2 (b). Therefore, the coordinates can be calculated as a linear conversion result by appropriately setting the conversion parameters in the coordinate linear conversion unit 110. The coordinate non-linear conversion unit 120 sets the conversion parameters so that only the difference between the quadrangle 210 and the intermediate quadrangle, which is the final calculation result, is calculated as the non-linear conversion result. In the case of any intermediate quadrangle, the absolute value of the difference can be expected to be relatively small. The decomposition method is not limited to the example described above, and the coordinate conversion process may be performed using the conversion parameters obtained by another method.

<Modification 2>
In the first embodiment, the coordinate linear conversion unit 110 maps the affine transformation and the coordinate nonlinear conversion unit 120 maps the UV value by bilinear interpolation, but other calculations may be used. For example, the coordinate linear conversion unit 110 may perform only simple scaling. At this time, the equation (4) is executed as a linear transformation.

Here, a is a parameter representing the enlargement / reduction ratio. Even in this case, it is sufficient for use cases where there is no rotation or translation, and the circuit scale can be further reduced.

Further, for example, the coordinate nonlinear conversion unit 120 may perform UV mapping in consideration of the depth. At this time, the equation (5) is executed as a non-linear transformation.

Here, Z0, Z1, Z2, and Z3 are the depths (coordinates in the depth direction) of each of the four vertices, and the other parameters are the same as those in the equation (2). By doing so, more accurate UV value mapping becomes possible, but the calculation becomes complicated, such as an increase in division. However, due to the configuration described in the first embodiment, even if the accuracy of the coordinate nonlinear conversion unit 120 is lower than that of the coordinate linear conversion unit 110, the influence on the final result is minor, so that even a complicated calculation is relatively relatively possible. It can be mounted on a small circuit scale.

<Embodiment 3>
In the first embodiment, the coordinate synthesis unit 130 performs addition of the linear transformation result (Fx, Fy) and the non-linear transformation result (Gx, Gy), but other operations may be executed, for example, multiplication or comparison. (Select the larger one, etc.) may be an executable configuration. FIG. 6 shows a configuration example of the coordinate synthesizing unit 130 in this case. As shown in FIG. 6, the configuration is such that one of addition, multiplication, and comparison results is output to the input linear transformation result (Fx, Fy) and nonlinear transformation result (Gx, Gy). The operation selection is controlled from the control unit 180. With such a configuration, more diverse coordinate transformations can be realized.

For example, an example of decomposing the conversion process from the coordinates of the quadrangle 220 to the coordinates of the quadrangle 210 shown in FIG. 2 into multiplication of a linear transformation and a non-linear transformation will be described. This can be achieved, for example, by using the intermediate quadrangle 710 shown in FIG. 7. That is, the coordinates of the intermediate quadrangle 710 are obtained by the coordinate linear conversion unit 110, and the coordinates of the four vertices are (1.0, 1.0), (1.0, 0.67), (1. Non-linear conversion is performed so that 0,0.67) and (1.0,1.0) are obtained. The coordinate composition unit 130 multiplies the linear transformation result and the non-linear transformation result. By doing so, the left end (P0, P3) of the intermediate quadrangle 710 is the same size, shrinks in the Y direction as it goes to the right, and the right end (P1', P2') becomes 2/3 in the Y direction. realizable. As described above, a desired coordinate transformation can be realized by multiplying the linear transformation and the non-linear transformation.

<Embodiment 4>
In this embodiment, an example of applying the geometric transformation described in the above embodiment to the image composition processing is shown. FIG. 8 is a diagram showing a configuration example of the image synthesizing unit 800 in the present embodiment. The image geometric transformation unit 810 reads a plurality of images (frames) constituting the moving image from a memory (not shown), executes a predetermined geometric transformation process, and then sequentially supplies the images (frames) to the composition unit 830. The attribute data geometric transformation unit 820 reads the attribute map from the memory (not shown), executes a predetermined geometric transformation process, and then sequentially supplies the attribute map to the synthesis unit 830. Here, the attribute map records the attribute data (metadata) calculated by the analysis process for each divided area of the image according to the two-dimensional coordinate position of the divided area. For example, the amount of movement between frames is recorded. It is the data to be done. The synthesizing unit 830 sequentially performs synthesizing processing for each pixel and outputs the synthesizing result. During the synthesis process, the processing parameters are adaptively switched according to the attribute data. The image compositing process is, for example, an HDR compositing process, in which a compositing ratio is determined based on the amount of movement (attribute data) for a plurality of images of the same scene with different exposure times, and a blending process is performed.

The image geometric transformation unit 810 executes, for example, a process of correcting blurring of a shooting device when shooting a moving image. Specifically, the four vertices corresponding to the four vertices of the quadrangle set in the first frame are detected from each subsequent frame, and the detected quadrangle is transformed into a predetermined rectangle.
On the other hand, if the attribute map is the result of calculating one movement amount for each 64 × 64 area of the image (image before geometric transformation), the attribute map has a resolution of 1/64 compared to the image. Become. That is, in order to input the attribute data corresponding to the pixels to the composition unit 830, the attribute data geometric transformation unit 820 enlarges the image by 64 times (difference in resolution between the image and the attribute map) in addition to the geometric transformation corresponding to the blur correction. It suffices to execute the conversion that becomes the process of absorption).

In the above configuration, the image geometric transformation unit 810 is implemented by a general projective transformation, and the attribute data geometric transformation unit 820 is implemented by using the image processing unit 100 in the first embodiment. A specific setting method of the attribute data geometric transformation unit 820 will be described. The converted rectangle (first coordinate) is the same as the image geometric transformation unit 810 (predetermined rectangle). The quadrangle (second coordinate) before deformation is used by multiplying the detected coordinates of the four vertices by 1/64.

In the present embodiment, unlike the image processing unit 100 of the first embodiment, what is deformed is to apply geometric transformation to the attribute data instead of the image data, but the data group recorded according to the two-dimensional coordinate position. If so, it can be applied in the same way. In general, in order to realize high-magnification enlargement / reduction processing, high calculation accuracy is required for coordinate calculation, and especially in the case of non-linear transformation, there is a concern that the calculation cost will be high because the calculation is complicated. However, since the 64 times enlargement is a linear transformation by using the attribute data geometric transformation unit 820 of the present embodiment, the coordinate linear transformation unit 110 converts it with high accuracy, and only the residual is obtained by the coordinate nonlinear transformation unit 120. As a result, the calculation cost can be kept relatively low.

<Embodiment 5>
In the first embodiment, the conversion parameter is determined based on the coordinates of the four vertices before and after the conversion, but the present invention is not limited to this. For example, if the image capturing device is equipped with a distance sensor or the like and distance information to the subject is available, the conversion parameter may be obtained based on the distance information. As an example, FIG. 10 will be described with respect to a case where a flat subject such as a building is photographed by tilting from the front to the upper side. As shown in FIG. 10A, the distance to the lower end P1 of the field of view of the camera is d1, and the distance to the upper end P2 of the field of view of the camera is d2. At this time, the captured image becomes an image that is reduced in the horizontal direction toward the upper end as shown in FIG. 10 (b). The dotted line in the figure is an auxiliary line and indicates the vertical direction. This is corrected as shown in FIG. 10 (c). That is, at the lower end P1 of the image, the magnification is the same in the horizontal direction, and at the upper end P2 of the image, the processing is such that the magnification is d2 / d1 in the horizontal direction. Hereinafter, it will be described as R = d2 / d1.

In the case of performing the above deformation, in order to increase the linear transformation component, for example, the entire image is horizontally (R + 1) / 2 times (that is, the average of the magnification of the lower end and the magnification of the upper end) in the linear conversion unit 110. Can be considered. Specifically, the affine coefficient is set to a = (R + 1) / 2, b = 0.0, c = 0.0, d = 1.0, e = W × (R-1) / 4, f = 0. It may be 0 or the like. Here, W is the image width. If the parameters of the linear transformation are determined, the parameters of the non-linear transformation can be obtained from the residual with the final calculation result (FIG. 10 (c)) as in the first embodiment.

As described above, in this embodiment, an example of obtaining a conversion parameter based on the distance information to the subject is shown. The conversion parameter may be calculated by using the value of the gyro sensor, the zoom value of the lens, or the like in addition to the distance information. When the device is fixed and the use cases are limited, a plurality of parameter sets may be stored in advance and selectively switched according to the use cases.

<Other Examples>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Further, the present invention is not limited to the above-described embodiment, and can be variously modified or modified within a range that does not deviate from the gist of the present invention.

110 Coordinate linear transformation unit 120 Coordinate non-linear transformation unit 130 Coordinate composition unit 180 Control unit 810 Image geometric transformation unit 820 Attribute data geometric transformation unit 830 Synthesis unit

Claims (14)

An image processing device that performs geometric transformation on the data recorded according to the two-dimensional coordinate position.
The result of the linear conversion that converts the conversion process from the first coordinate to the second coordinate from the first coordinate to the third coordinate, and the non-linear conversion that converts the first coordinate to the fourth coordinate. A control means for determining the transformation parameters of the linear and nonlinear transformations so that they can be combined with the result.
And linear transformation means for calculating said third coordinate in the linear conversion based on the conversion parameter of the linear transformations,
In the nonlinear conversion based on the conversion parameter of the non-linear transformation, a nonlinear transformation means for calculating the fourth coordinate,
An image processing apparatus comprising: a synthesizing means for synthesizing the third coordinate and the fourth coordinate to calculate the second coordinate. The control means acquires the four-vertex coordinates of the first quadrangle composed of the first coordinates and the four-vertex coordinates of the second quadrangle composed of the second coordinates, and uses the acquired four-vertex coordinates as the acquired four-vertex coordinates. The image processing apparatus according to claim 1, wherein the conversion parameters are determined based on the above. The control means sets the conversion parameter so that three of the four vertices of the third quadrangle composed of the third coordinates coincide with three of the four vertices of the second quadrangle. The image processing apparatus according to claim 2, wherein the image processing apparatus is determined. The control means determines the conversion parameter so that the sum of the absolute values of the distances between the four vertices of the third quadrangle composed of the third coordinates and the four vertices of the second quadrangle is minimized. The image processing apparatus according to claim 2, wherein the image processing apparatus is characterized in that. The control means determines the conversion parameter so that the ratio of the area of the area overlapping the area of the second quadrangle to the area of the third quadrangle composed of the third coordinates is maximized. The image processing apparatus according to claim 2. The control means decomposes the transformation process into addition of the linear transformation and the nonlinear transformation, and the synthesis means adds the third coordinate and the fourth coordinate to calculate the second coordinate. The image processing apparatus according to any one of claims 1 to 5. The control means decomposes the transformation process into multiplication of the linear transformation and the nonlinear transformation, and the synthesis means multiplies the third coordinate by the fourth coordinate to calculate the second coordinate. The image processing apparatus according to any one of claims 1 to 5, wherein the image processing apparatus is characterized. The image processing apparatus according to any one of claims 1 to 7, wherein the linear transformation is a transformation that can be expressed by an affine transformation, and the non-linear transformation is a transformation that cannot be expressed by an affine transformation. The image processing apparatus according to any one of claims 1 to 8, wherein the number of bits of the operation of the non-linear transformation is smaller than the number of bits of the linear transformation. The data recorded according to the two-dimensional coordinate position is image data, and the conversion process is a conversion process from the coordinates of the image after the geometric transformation to the coordinates of the image before the geometric transformation, and is before the geometric transformation. The image processing apparatus according to any one of claims 1 to 9 , further comprising a means for reading the data from an external memory according to the coordinates of the image. The data recorded according to the two-dimensional coordinate position is image data, and the conversion process is a conversion process from the coordinates of the image before the geometric transformation to the coordinates of the image after the geometric transformation, and after the geometric transformation. The image processing apparatus according to any one of claims 1 to 9 , further comprising means for writing the data to an external memory according to the coordinates of the image. An image processing device that synthesizes multiple image data.
Image geometric transformation means for geometrically transforming multiple image data,
Attribute data geometric transformation means for geometrically transforming attribute data,
It has an image synthesizing means for synthesizing the plurality of image data geometrically transformed by the image geometric transformation means based on the attribute data geometrically transformed by the attribute data geometric transformation means.
The attribute data geometric transformation means is
The result of the linear conversion that converts the conversion process from the first coordinate to the second coordinate from the first coordinate to the third coordinate, and the non-linear conversion that converts the first coordinate to the fourth coordinate. A control means for determining the transformation parameters of the linear and nonlinear transformations so that they can be combined with the result.
And linear transformation means for calculating said third coordinate in the linear conversion based on the conversion parameter of the linear transformations,
In the nonlinear conversion based on the conversion parameter of the non-linear transformation, a nonlinear transformation means for calculating the fourth coordinate,
An image processing apparatus comprising: a synthesizing means for synthesizing the third coordinate and the fourth coordinate to calculate the second coordinate. It is an image processing method that performs geometric transformation on the data recorded according to the two-dimensional coordinate position.
The result of the linear conversion that converts the conversion process from the first coordinate to the second coordinate from the first coordinate to the third coordinate, and the non-linear conversion that converts the first coordinate to the fourth coordinate. A control step that determines the transformation parameters of the linear and non-linear transformations so that they can be combined with the results.
A linear transformation step of calculating the third coordinate in the linear transformation based on the transformation parameters of the linear transformation determined in the control process,
In the nonlinear conversion based on the conversion parameter of the non-linear transformation determined in the control step, a nonlinear transformation step of calculating the fourth coordinate,
An image processing method comprising a synthesis step of synthesizing the third coordinate and the fourth coordinate to calculate the second coordinate. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images