JP4483517B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly to an image processing apparatus, method, and program that are optimal for image processing using movement of a subject image.

  For example, when a flower field as shown in FIG. 1 is imaged at a certain elevation angle with a video camera when viewed from directly above, an image that is coarser as the predetermined pattern is closer to the front and becomes finer as it is further away as shown in FIG. . In the following description, a unit of a pattern having a tendency to become coarser toward the front and become finer as it moves away when imaged at a certain elevation angle is referred to as texture, and the degree of the tendency is referred to as texture gradient.

  By the way, in a video camera, recording or the like is usually performed on an image obtained as a result of imaging by performing processing such as noise reduction using the motion of a subject image.

  Specifically, as shown in FIG. 2, the image of the subject imaged by a video camera that moves linearly in a horizontal direction with a certain elevation angle does not have a uniform linear motion. A portion that can be approximated to a constant velocity linear motion is appropriately detected, motion compensation is performed on the portion of the previous image corresponding to that portion, and the detected portion is added.

  However, since the subject image on the image captured at a constant elevation angle does not move at a constant linear velocity, it is impossible to accurately perform motion compensation, and processing such as noise reduction cannot be performed properly. was there.

  The present invention has been made in view of such a situation, and makes it possible to appropriately perform processing using movement of a subject image.

An image processing apparatus according to the present invention includes a blocking unit that blocks an input image obtained by imaging a subject at a predetermined elevation angle by an imaging unit and cuts out a block having a predetermined size from the input image, and a block of the input image From the maximum value, minimum value, or dynamic range of pixel luminance values in a block, maximum value, minimum value, or dynamic range of pixel edge strength, histogram distribution of pixel luminance values, or pixel edge strength The feature amount calculation means for extracting the histogram distribution as the feature amount of the block, and the block so that the difference between the block feature amount and the predetermined reference value is equal to or greater than a predetermined threshold value for each block of the input image The feature amount determination means that changes the size of the block to a smaller size and causes the blocking means to cut out the block of the changed size. For each line of pixels arranged in a predetermined direction on the input image, calculate the average value of the size of the blocks arranged on the line, set the regression line using the average value for each line, and set the vanishing point The vanishing point calculating means for calculating, the elevation angle calculating means for calculating the elevation angle using the vanishing point, and the facing image obtained when the subject is imaged from the front by the imaging means using the elevation angle, the input image is And image processing means for converting .

According to the image processing method or program of the present invention, the blocking unit blocks the input image obtained by imaging the subject at a predetermined elevation angle by the imaging unit and cuts out a block having a predetermined size from the input image. Step and histogram distribution of the maximum, minimum, or dynamic range of pixel brightness values, maximum, minimum, or dynamic range of pixel edge intensity, and pixel brightness values from the input image block Alternatively, for each of the blocks of the input image, a feature amount calculating step for extracting the distribution of the pixel edge intensity histogram as a block feature amount, and for each of the blocks of the input image, a difference between the block feature amount and a predetermined reference value is a predetermined threshold value. Change the size of the block to a smaller size so that it becomes the above, and change to the blocking means For the feature amount determination step that cuts out the block of size, and for each line consisting of pixels arranged in a predetermined direction on the input image, the average value of the size of the block arranged on the line is calculated, and the average value for each line is used A vanishing point calculating step for setting a regression line and calculating a vanishing point, an elevation angle calculating step for calculating an elevation angle using the vanishing point, and an imaging means for imaging the subject from the front using the elevation angle And an image processing step of converting the input image into the facing image obtained in the above.

In the image processing apparatus, method, and program of the present invention, the input image obtained by imaging the subject at a predetermined elevation angle by the imaging means is blocked, and a block having a predetermined size is cut out from the input image. From the block of the input image, the maximum value, minimum value, or dynamic range of the luminance values of the pixels in the block, the maximum value, minimum value, or dynamic range of the pixel edge strength, the histogram distribution of the pixel luminance values, or pixels The edge intensity histogram distribution is extracted as a block feature value, and for each block of the input image, the difference between the block feature value and a predetermined reference value is equal to or greater than a predetermined threshold value. The size is changed to a smaller size, and the block of the changed size is cut out. For each line consisting of pixels lined up, the average value of the size of the blocks arranged on the line is calculated, the regression line is set using the average value for each line, the vanishing point is calculated, and the vanishing point is used. The elevation angle is calculated, and the elevation angle is used to convert the input image into a facing image obtained when the subject is imaged from the front by the imaging means.

  According to the present invention, image processing can be performed using vanishing points.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, although there is an embodiment which is described in the embodiment of the invention but is not described here as corresponding to the invention, it means that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

The image processing apparatus according to the present invention blocks an input image obtained by imaging a subject at a predetermined elevation angle by an imaging unit , and blocks a predetermined size block from the input image (for example, FIG. 49). Blocked memory 121), the maximum value, minimum value, or dynamic range of the luminance values of the pixels in the block from the input image block, the maximum value, minimum value, or dynamic range of the pixel edge strength, and the luminance value of the pixels For each of the feature amount calculation means (for example, the feature amount calculation unit 122 in FIG. 49) that extracts the histogram distribution of the pixel or the edge intensity histogram distribution of the pixel as the block feature amount, and the block of the input image The block size is set to a smaller size so that the difference between the feature amount and the predetermined reference value is equal to or greater than a predetermined threshold. The feature amount determination means (for example, the feature amount determination section 123 in FIG. 49) that causes the block forming means to cut out the block having the changed size, and each line composed of pixels arranged in a predetermined direction on the input image The vanishing point calculating means for calculating the average value of the sizes of the blocks arranged on the line, setting the regression line using the average value for each line, and calculating the vanishing point (for example, vanishing point calculation in FIG. 49) Unit 112), an elevation angle calculation unit (for example, elevation angle calculation unit 23 in FIG. 49) that calculates an elevation angle using the vanishing point, and an imaging unit that captures the subject from the front using the elevation angle. The image processing means (for example, the image conversion part 13 of FIG. 3) which converts an input image into a facing image is provided.

  FIG. 3 shows a configuration example of the image processing apparatus 1 to which the present invention is applied. The image processing apparatus 1 displays, from the front, an image (an image having a certain texture gradient) obtained as a result of being captured by a video camera that moves linearly in a horizontal direction with a certain elevation angle as shown in FIG. The image is converted into a captured image (in the example of FIG. 2, an image captured from directly above (FIG. 1)) (an image having no texture gradient) (hereinafter referred to as a directly-facing image). That is, the subject image is converted so as to have a uniform linear motion even in the image.

  Then, the image processing apparatus 1 performs, for example, a noise reduction process with motion compensation on the entire facing image. That is, since the image of the subject has a uniform linear motion in the directly-facing image, accurate motion compensation can be performed, and as a result, noise reduction and the like can be performed appropriately.

  The image processing apparatus 1 returns the image obtained as a result of the processing to an image having an original texture gradient, and outputs the image to a recording medium or a display unit (not shown), for example.

  The configuration of the image processing apparatus 1 will be described.

  The image input unit 11 inputs an image obtained as a result of being imaged by an imaging unit (not shown). For example, as shown in FIG. 2, an image captured by a video camera that moves linearly in a horizontal direction with a certain elevation angle is input by an imaging unit.

  The image input unit 11 supplies the input image that has been input to the elevation angle calculation unit 12 and the image conversion unit 13.

  The elevation angle calculation unit 12 calculates the elevation angle of the imaging unit at the time of imaging the image from the input image supplied from the image input unit 11 (for example, for each frame), and the image conversion unit 13 and the image conversion unit 15. To supply.

  The elevation angle calculation unit 12 is configured as shown in FIG. The quantization unit 21 quantizes the input image supplied from the image input unit 11 for each frame and supplies the quantized point to the vanishing point calculation unit 22.

  The vanishing point calculating unit 22 calculates the vanishing point (coordinates on the captured image) based on the quantized data supplied from the quantizing unit 21 and supplies the vanishing point to the elevation angle calculating unit 23.

  The vanishing point is a point where the plane of the subject converges to infinity, and in this case, it is obtained from the texture gradient.

  The elevation angle calculation unit 23 uses the coordinates of the vanishing point supplied from the vanishing point calculation unit 22 to calculate the elevation angle of the imaging unit when the input image is captured, and supplies the calculation result to the image conversion unit 13.

  Details of the configuration and processing of the quantizing unit 21 to the elevation angle calculating unit 23 will be described later.

  Returning to FIG. 3, the image conversion unit 13 captures an input image supplied from the image input unit 11 based on the elevation angle of the imaging unit supplied from the elevation angle calculation unit 12 from the front. Image). That is, an image in which the subject image moves linearly at a constant speed is generated. The image conversion unit 13 supplies the converted facing image to the processing unit 14.

  Here, the principle of image conversion of the image conversion unit 13 will be described.

  For example, when an iron plate T having holes at equal intervals as shown in FIG. 5 is imaged at a predetermined elevation angle θ by a camera CA as shown in FIG. 6, a hole pattern (texture) as shown in FIG. An image Da having a gradient that becomes rougher toward the front and becomes finer as it moves away is obtained. Note that the scale M in FIG. 7 represents the position in the vertical direction (texture gradient direction) of the hole pattern.

  When the image Da having such a texture gradient is displayed on the screen SC tilted by the same angle as the elevation angle θ of the camera CA, as shown in FIG. 8, this time, as shown in FIG. Thus, like the actual iron plate T (FIG. 5), the image Db of the iron plate T in which hole patterns are arranged at regular intervals is displayed.

  9 represents the vertical position of the predetermined hole pattern of the image Da, and the tip of the arrow represents the hole pattern of the image Db corresponding to the hole pattern of the original image Da of the arrow. It represents the position in the vertical direction.

  That is, the image conversion unit 13 uses the geometric relationship between the pixel position and the elevation angle (θ) of the facing image (image Db) and the input image (image Da) as shown in FIG. Based on the image, a pixel of the input image corresponding to each pixel constituting the facing image is detected, and the image value of the detected pixel of the input image is set to the pixel of the facing image, thereby generating a facing image. .

  Note that the vanishing point (more precisely, the image of the vanishing point) R exists in the upper part of the drawing in which the vertical interval of the hole pattern disappears in the example of FIG. Also, as shown in FIG. 9, the line connecting the vanishing point R and the focal point Q is parallel to the surface of the subject (the screen surface in the example of FIG. 9), so the elevation angle (the camera optical axis and the surface of the subject) (Angle) can be obtained by calculating the vanishing point. That is, for this reason, the elevation angle calculation unit 12 first obtains the vanishing point (vanishing point calculation unit 22) and calculates the elevation angle using the vanishing point (elevation angle calculation unit 23).

  Returning to FIG. 3, the processing unit 14 performs a noise reduction process with motion compensation on the facing image obtained from the image conversion unit 13, and supplies the image obtained as a result to the image conversion unit 15.

  Specifically, the frame memory 31 with motion compensation delays the facing image supplied from the image conversion unit 13 by a predetermined frame and supplies the delayed image to the motion vector estimation unit 32. The frame memory with motion compensation 31 also performs motion compensation according to the motion vector estimated by the motion vector estimation unit 32 for the input facing image according to the control of the address control unit 33, and stores it. 34.

  The motion vector estimation unit 32 performs block matching or the like using the facing image supplied from the image conversion unit 13 and the facing image (past image) supplied from the frame memory 31 with motion compensation, and performs motion vector Is supplied to the address control unit 33.

  The address control unit 33 shifts the facing image stored in the frame memory with motion compensation 31 according to the motion vector supplied from the motion vector estimation unit 32 (motion compensation).

  The multiplication unit 34 multiplies the facing image (the past facing image) supplied from the frame memory 31 with motion compensation by a coefficient k (0 <k <1) and supplies the result to the adder 36.

  The multiplier 35 multiplies the facing image supplied from the image conversion unit 13 by a coefficient (1-k), and supplies the resulting image to the adder 36. The coefficient k is determined by appropriately performing reliability judgment based on the estimated motion vector.

  The adder 36 adds the image supplied from the multiplier 34 and the image supplied from the multiplier 35, and supplies the resulting image (image from which noise has been removed) to the image conversion unit 15.

  Based on the elevation angle supplied from the elevation angle calculation unit 12, the image conversion unit 15 obtains the same texture gradient as that of the input image based on the elevation image supplied from the processing unit 14. The image is converted into an image (hereinafter referred to as an output image) and supplied to the image output unit 16.

  The image output unit 16 outputs and displays the output image supplied from the image conversion unit 15 on, for example, a display unit (not shown) or outputs and records it on a recording medium.

  Next, the operation of the image processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S1, the elevation angle calculation unit 12 (FIG. 3) calculates the elevation angle of the imaging unit at the time of imaging from the input image input from the image input unit 11. This process will be described with reference to the flowchart of FIG. 11 and FIG.

  In step S11, the quantization unit 21 (feature amount extraction unit 51) of the elevation angle calculation unit 12 extracts the feature amount of each pixel for each frame of the input image. For example, the color difference value and edge strength of each pixel are extracted. When the edge strength is a feature amount, the feature amount extraction unit 51 incorporates a differential filter (not shown) and emphasizes the edge of the input image to extract the edge strength.

  In step S12, the quantization unit 52 of the quantization unit 21 quantizes the input image based on the feature amount of each pixel extracted in step S11.

  For example, when the feature amount is a color difference value, the value of the pixel having the same color difference value as the predetermined reference value is set to 1, and the value of the pixel having the other color difference value is set to 0.

  For example, the input image shown in FIG. 2 is quantized as shown in FIG. In FIG. 12, a white point represents a pixel having a value of 1, and a black point represents a pixel having a value of 0.

  Here, the process for determining the reference value will be described with reference to the flowchart of FIG.

  In step S21, a histogram of predetermined feature amounts (in this example, color difference values) is generated from the input image (for example, one frame).

  In step S22, the top n color difference values in frequency are selected in the histogram generated in step S21.

  In step S23, one color difference value is selected from the n color difference values selected in step S22. In step S24, the position of a pixel having the same color difference value as the selected color difference value is detected.

  Next, in step S25, as in the example shown in FIGS. 14 to 16, the difference between the maximum value and the minimum value of the horizontal position (X-axis coordinates) of the pixel whose position is detected in step S24, and the vertical The difference between the maximum value and the minimum value of the direction position (Y-axis coordinates) is calculated, and the total value is calculated.

  In step S26, it is determined whether all of the n color difference values selected in step S22 have been selected in step S23. If it is determined that there are color difference values that have not yet been selected, the process returns to step S23. The next color difference value is selected, and the processing from step S24 is similarly performed.

  If it is determined in step S26 that all the color difference values have been selected, the process proceeds to step S27, and the color difference value that provides the maximum total value among the total values calculated in step S25 is selected as the reference value. .

  In this way, the reference value is determined. That is, for example, there are relatively many colors in the input image, and the color difference values of the colors distributed over the entire input image become reference values, and such colors are quantized.

  Although the case where the feature amount is a color difference value has been described as an example here, the reference value is similarly determined for other feature amounts such as edge strength.

  Returning to FIG. 11, in step S <b> 13, the elevation angle calculation unit 12 (vanishing point calculation unit 22) calculates the vanishing point from the quantized data obtained in step S <b> 12.

  Specifically, the texture gradient calculation unit 61 of the vanishing point calculation unit 22 determines, for each line, the average value of the white point (texture) intervals (texture gradient) from the quantized data obtained in step S12 (FIG. 12). Data). For example, as shown in FIG. 17, when pixels a1, a2, a3, and a4 having a value of 1 are arranged in a predetermined line, an average value of the intervals is calculated.

  The texture gradient calculating unit 61 sets a regression line as shown in FIG. 19 based on the calculated average value AV (FIG. 18) for each line. That is, this regression line corresponds to the texture gradient.

  Next, the vanishing point calculation unit 62 calculates an intersection point between the set regression line and the Y axis (Y axis on the input image) as a vanishing point.

  Since the vanishing point R is a point where the plane of the subject converges to infinity when the input image (FIG. 2) is viewed, it may exist at a position beyond the input image as shown in FIG. Note that the X-axis coordinate of the vanishing point R is the X-axis coordinate of the center point of the input image.

  Next, in step S14, the elevation angle calculation unit 12 (elevation angle calculation unit 23) calculates the elevation angle using the vanishing point calculated in step S13.

  As shown in FIG. 21, the elevation angle has the same value as the angle formed by the straight line connecting the vanishing point R and the focal point Q and the straight line connecting the central point of the imaging plane Z and the focal point Q. Is calculated as shown in FIG.

  In the equation, p is the distance between the center point of the input image and the vanishing point R (FIG. 20), h is the size (height) in the Y-axis direction (FIG. 20) of the input image, and r is , P / h. K is a predetermined coefficient (described later). The geometric relationship shown in FIG. 21 can also be shown in FIG.

  When the elevation angle is calculated in this way, in step S2 of FIG. 10, the image conversion unit 13 converts the input image into a facing image using the elevation angle.

  Specifically, for each pixel (x1, y1) of the directly-facing image, the corresponding pixel (x0, y0) of the input image is detected (FIG. 23).

The pixel (x1, y1) of the directly-facing image, the pixel (x0, y0) of the input image, and the elevation angle θ have a geometrical relationship as shown in FIG. The pixel (x0, y0) of the input image corresponding to x1, y1) is obtained as shown in equation (2).
Using this, when an input image is given, an actual pattern on the object plane can be estimated.

  Next, proceeding to step S3, the processing unit 14 performs the noise reduction process with motion compensation as described above on the directly-facing image converted by the image conversion unit 13.

  In step S4, the image conversion unit 15 uses the elevation angle calculated in step S1 to convert the facing image subjected to the noise reduction processing by the processing unit 14 into an image having an original texture gradient (output image). Convert to

  Specifically, for each pixel (x0, y0) of the output image, the corresponding pixel of the facing image (x1, y1) is detected (FIG. 25).

  The pixel (x0, y0) of the output image, the pixel (x1, y1) of the directly-facing image, and the elevation angle θ have a geometrical relationship as shown in FIG. , Y0), the pixel (x1, y1) of the directly-facing image corresponding to (0, y0) is obtained as shown in Expression (3). This can be used to know how the image appears on the film.

  In step S5, the image output unit 16 supplies the output image supplied from the image conversion unit 15 to the display unit or the recording medium.

  The above process is repeated for each frame to process the input image.

  When trying to estimate a pattern on a subject plane from an image actually taken by a camera, the angle of view of the camera and the distance from the origin are often unknown. If the correct k and d values are k0, d0, and the estimated values are k1, d1, respectively, the estimated point P (x1, y1) on the subject plane and the point P (X1, Y1) on the actual subject plane The relationship is expressed by equation (17) by substituting k = k1 and d = d1 in equation (2) into k = k0 and d = d0 in equation (3).

  From this, even if the focal length kh of the lens and the distance d from the camera origin are not accurately determined, the pattern drawn on the estimated object plane is the actual pattern on the object plane in the x and y directions. Enlarged or reduced to a constant multiple. In many cases, it can be considered that the object is performing a uniform linear motion on the facing image thus obtained.

  This can be confirmed by simple drawing (FIGS. 27 to 29).

  For example, referring to FIG. 21, the angle of view fov is specified by a coefficient k as shown in Expression (4), and the elevation angle is obtained including the coefficient as shown in Expression (1). The elevation angle changes according to the angle of view. In the examples of FIGS. 27 to 29, if the vertical enlargement ratio is r, Expression (5) is established.

  In the above description, a case where a horizontal subject is imaged has been described as an example. However, the present invention can also be applied to a case where a vertical subject is imaged as shown in FIG.

  A configuration example of the image processing apparatus 1 in this case is shown in FIG. In this image processing apparatus, the image processing apparatus 1 shown in FIG.

  The image rotation unit 71 rotates the input image (an image that may include a vertical subject image) supplied from the image input unit 11 by 90 degrees, and an image obtained as a result (hereinafter referred to as a rotation image). , Supplied to the direction determining unit 72 and the switch 73.

  The direction determination unit 72 controls the switch 73 on the basis of the reliability information supplied from the elevation angle calculation unit 12, and receives the rotation image supplied from the image rotation unit 71 or the input image supplied from the image input unit 11. One of the images is selected and supplied to the elevation angle calculation unit 12 and the image conversion unit 13.

  When calculating the elevation angle, the elevation angle calculation unit 12 first represents reliability indicating whether the elevation angle should be calculated using either the input image itself or an image obtained by rotating the input image by 90 degrees (rotated image). Information is generated and supplied to the direction determining unit 72.

  Specifically, the elevation angle calculation unit 12 calculates a vanishing point from each of the input image and the rotation image of one frame, and applies each texture gradient data (example of FIG. 19) to the regression line (FIG. 19) used at that time. In the case of (1), the distance (average value of the interval of pixels having a value of 1 for each line) is calculated (dispersion or deviation is calculated), and this is used as reliability information (or the inverse thereof).

  That is, the direction determination unit 72 compares the reliability information obtained from the input image with the reliability information obtained from the rotated image, and the image with the higher reliability (the one with the smaller variance or deviation) is the elevation angle calculation unit. 12 and the image converter 13 are controlled.

  The above direction determination process is performed once for a predetermined number of frames.

  As described above, the present invention can be applied to a vertical subject. In principle, the present invention can also be applied to a plane with an arbitrary angle other than horizontal and vertical.

  Further, in the above, when the input image is quantized by the quantization unit 21 (FIG. 4) of the elevation angle calculation unit 12, one feature amount (for example, color difference value or edge strength) is used for each pixel. However, feature quantities of a plurality of features can be used.

  A configuration example of the elevation angle calculation unit 12 in this case is shown in FIG. This elevation angle calculation unit is provided with a quantization unit 81 instead of the quantization unit 21 of the elevation angle calculation unit 12 of FIG. The quantization unit 81 includes n feature amount extraction units 51-1 to 51-n and a feature amount evaluation unit 91. The other parts are the same as those in FIG. 4, and the description thereof will be omitted as appropriate.

  An input image is supplied to each of the n feature amount extraction units 51, and each feature amount extraction unit 51 extracts a predetermined feature amount of each pixel for each frame. For example, the feature quantity extraction unit 51-1 extracts color difference values, and the feature quantity extraction unit 51-2 extracts edge strength. Each feature amount extraction unit 51 supplies the extracted feature amount of each pixel to the feature amount evaluation unit 91.

  The feature amount evaluation unit 91 calculates an evaluation value of each pixel based on the feature amount supplied from the feature amount extraction unit 51 and supplies the evaluation value to the quantization unit 52.

  The quantization unit 52 quantizes the input image based on the evaluation value of each pixel supplied from the feature amount evaluation unit 91.

  In this way, the input image can be quantized using a plurality of feature amounts.

The feature amount evaluation unit 91 calculates an evaluation value E for each feature C. Then, as shown in FIG. 33, a table indicating the weight value W corresponding to the feature C to be used is held, and when the evaluation value E is calculated, weighting is performed as shown in Expression (6) to obtain the evaluation value. E can be calculated.
E = W1 · E1 + W2 · E2 + ... + Wn · En (6)
However, E1 to En are evaluation values based on the feature amounts of C1 to Cn.

Further, the feature amount evaluating unit 91 has weight values W (W1 to Wj) corresponding to predetermined features (all or a part of the features to be used) as shown in FIG. 34 (features C1 to Cj in the example of FIG. 34). And the evaluation value Z is calculated from the feature amount of the feature as shown in Expression (7).
Z = W1 · E1 + W1 · E2 + ... + Wj · Ej (7)

Then, the feature quantity evaluation unit 91, as shown in FIG. 35, determines a predetermined feature C (all or part of the feature to be used (in the example of FIG. 35, when calculating the evaluation value Z). A table indicating the weight values W (W (j + 1) to Wn) of the special quantities C (j + 1) to Cn) that are not used is held, and from the set of the weight values W, Expression (8) The evaluation value E can also be calculated as shown in FIG.
E = W (j + 1) * E (j + 1) + W (j + 2) * E (j + 2) + ... + Wn * En (8)

  FIG. 36 shows another configuration example of the elevation angle calculation unit 12. In this elevation angle calculation unit, a vanishing point calculation unit 101 is provided instead of the quantization unit 21 and the vanishing point calculation unit 22 of the elevation angle calculation unit 12 shown in FIG. The other parts are the same as in FIG.

  The operation of the elevation angle calculation unit 12 will be described with reference to the flowchart of FIG.

  In step S51, the vanishing point calculation unit 101 sets the number of repetitions A.

  In step S52, the vanishing point calculation unit 101 generates random numbers i, j, s.

  In step S53, the vanishing point calculation unit 101, as shown in FIG. 39, has a square with one side length S (hereinafter referred to as a sub image) centered on a predetermined coordinate (i, j) from the input image. Cut out.

The center coordinates (i, j) of the sub-image are determined as shown in Expression (9). In the equation, w is the length of the input image in the X-axis direction, and h is the length of the Y-axis direction (FIG. 38). random () is an output of a random number generator (not shown) that generates a random real number greater than or equal to 0 and less than 1, and takes a different value for each calculation. That is, the random number used for calculating i is different from the random number used for calculating j.
i = w × random ()
j = h × random () (9)

  Further, the length s of one side of the sub-image is calculated by Expression (10). In the formula, log is a natural logarithm, exp is an exponential function, and min (a, b) is a function indicating the smaller of a and b.

  When the sub image protrudes from the input image, i, j, and s are calculated with different random numbers until they do not protrude.

  Next, in step S54, the vanishing point calculation unit 101 applies each pixel (p, q) of the partial image, which is a strip-shaped portion having a width of S above and below cut out from the input image with (i, j) as the center. And matching with sub-images. Specifically, the vanishing point calculation unit 101 cuts out a square of one side length S around each pixel (p, q) of the partial image, and RGB for all pixels between the sub-images. The total value V (p, q) of the absolute values of the pixel value differences is calculated. In this process, a square value is used in addition to the absolute value.

  In step S55, the vanishing point calculation unit 101 specifies the feature amount U (p) for each Y coordinate based on the total value V (p, q) of each pixel calculated in step S54.

  Specifically, the minimum value of the total value V (p, q) for each Y coordinate is set as the feature amount U (p) (formula (11)).

  Further, the average value of the total value V (p, q) for each Y coordinate is defined as the feature amount U (p) (formula (12)).

  Of the total value V (p, q) for each Y coordinate, the value at q = s is defined as the feature amount U (p) (formula (13)).

  In step S56, the vanishing point calculation unit 101 calculates the texture gradient data a from the feature amount U calculated in step S55.

  For example, the activity value of the feature amount U is (formula (14)), which is texture gradient data a.

  The texture gradient data a can also be calculated using the autocorrelation function X (t) of the feature amount U.

  Specifically, t1 satisfying the equation (15) is set as a half value of the autocorrelation (FIG. 40), and this is the texture gradient data a.

  Further, the reciprocal of the number of times that the autocorrelation function X (t) becomes zero (zero cross) is the texture gradient data a (FIG. 41). If the number of zero crossings is 0, a = 0.

  Moreover, the correlation activity calculated | required by Formula (16) is made into the texture gradient data a.

  When the texture gradient data a of the partial image is calculated as described above, the process proceeds to step S57.

  Here, for each trial, a feature value U for each X-axis coordinate is calculated for the partial image, and texture gradient data a is obtained from the feature value U.

  The vanishing point calculation unit 101 plots one point for each divided region using the texture gradient data a as shown in FIG.

  The trial from step SS52 to S57 is defined as one trial. Next, the process proceeds to step S58, where it is determined whether the trial has been repeated a predetermined number of times A. If so, the process proceeds to step S59, and if not, the process proceeds to step S52.

  If it is determined in step S58 that the trial has been repeated a predetermined number of times, in step S59, the vanishing point calculation unit 101 divides the set of plots into n sections in the vertical direction as shown in FIG. The point specified by the coordinates, with the average value or median of the texture gradient data a as A (n) and the intermediate value in the j direction as j (n) Plot.

  Further, the vanishing point calculation unit 101 uses this plot to set a regression line and sets the intersection with the Y axis as the vanishing point.

  In step S60, the elevation angle calculation unit 23 calculates the elevation angle using the vanishing point calculated in step S59.

  44 to 48 show an example of the above processing.

  FIG. 49 shows another configuration example of the elevation angle calculation unit 12. This elevation angle calculation unit is provided with a blocking unit 111 and a vanishing point calculation unit 112 instead of the quantization unit 21 and the vanishing point calculation unit 22 of the elevation angle calculation unit 12 illustrated in FIG. The other parts are the same as in FIG.

  The blocking unit 111 determines a block size of the input image using a texture gradient.

  The configuration of the blocking unit 111 will be described. The block memory 121 cuts out a part (block) of the input image under the control of the address control unit 125 and supplies it to the feature amount calculation unit 122.

  The feature amount calculation unit 122 calculates a predetermined feature amount from the block supplied from the block memory 121. For example, the distribution of luminance values and edge intensity histograms of pixels in a block, the difference between each component of an RGB histogram, or the maximum, minimum, or dynamic range of the luminance values and edge intensity of pixels in a block are calculated as feature quantities. Is done.

  The feature amount determination unit 123 reduces the size of the block extracted by the block memory 121 according to the feature amount of the block supplied from the feature amount calculation unit 122, or stores the block size at that time in the storage unit 124. Remember.

  The address control unit 125 changes the size of the block to be cut out based on the determination result from the feature amount determination unit 123, and cuts out the block from the entire input image.

  The vanishing point calculation unit 112 calculates the vanishing point using the block size specified by the blocking unit 111.

  Next, the operation of the elevation angle calculation unit 12 will be described with reference to the flowchart of FIG.

  In step S71, the address control unit 125 sets the block size to an initial value, and in step S72, controls the block memory 121 to set the block to a predetermined position in the raster order.

  In step S73, the blocked memory 121 cuts out the set block. The block cut out here is supplied to the feature amount calculation unit 122.

  In step S <b> 74, the feature amount calculation unit 122 calculates the feature amount of the block supplied from the block memory 121.

  In step S75, the feature amount determination unit 123 determines that the difference between the feature amount calculated by the feature amount calculation unit 122 and the reference feature amount (the feature amount of the initial block size or the feature amount of the immediately preceding block size) is a predetermined threshold value. It is determined whether or not it is greater than that, and if it is determined that it is less than that, the process proceeds to step S76, and the address controller 125 is notified accordingly. Thereby, the address control unit 125 reduces the size of the block to be cut out by one size.

  Thereafter, the processing returns to step S73, and the subsequent processing is performed.

  If it is determined in step S75 that it is larger than the threshold value, the process advances to step S77, and the feature amount determination unit 123 stores the block size at that time in the storage unit 124.

  Next, in step S78, the address control unit 125 determines whether or not the block has been cut out from the entire input image. If it is determined that the block has not been cut out, the process returns to step S71, sets the block size to the initial size, In S72, the block is moved to the next raster order position. The subsequent processing is performed in the same manner.

  If it is determined in step S78 that the block has been cut out from the entire input image, the process proceeds to step S79.

  In step S79, the vanishing point calculation unit 112 calculates an average value of block sizes for each line for the identified block.

  In step S80, the vanishing point calculation unit 112 sets a regression line using the calculated average value, and calculates the vanishing point.

  In step S81, the elevation angle calculation unit 23 calculates the elevation angle using the vanishing point calculated in step S80.

  Thereafter, the process ends.

  The series of processes described above can be realized by hardware, but can also be realized by software. When a series of processing is realized by software, a program constituting the software is installed in a computer, and the program is executed by the computer, whereby the above-described image processing apparatus 1 is functionally realized. .

  FIG. 51 shows a configuration example of the computer 501. An input / output interface 516 is connected to a CPU (Central Processing Unit) 511 via a bus 515, and the CPU 511 receives a command from an input unit 518 such as a keyboard and a mouse via the input / output interface 516. When input, it is stored in a recording medium such as a read only memory (ROM) 512, a hard disk 514, or a magnetic disk 531, an optical disk 532, a magneto-optical disk 533, or a semiconductor memory 534 mounted in the drive 520. The program is loaded into a RAM (Random Access Memory) 513 and executed. Thereby, the various processes described above are performed. Further, the CPU 511 outputs the processing result to an output unit 517 such as an LCD (Liquid Crystal Display) via the input / output interface 516 as necessary. The program is stored in advance in the hard disk 514 or the ROM 512 and provided to the user integrally with the computer 501 or as a package medium such as the magnetic disk 531, the optical disk 532, the magneto-optical disk 533, and the semiconductor memory 534. Or can be provided to the hard disk 514 via the communication unit 519 from a satellite, a network, or the like.

  In the present specification, the step of describing the program provided by the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

It is a figure which shows the example of a to-be-photographed object. It is a figure which shows the example of a captured image. It is a block diagram which shows the structural example of the image processing apparatus to which this invention is applied. It is a block diagram which shows the structural example of the elevation angle calculation part of FIG. It is a figure which shows the other example of a to-be-photographed object. It is a figure explaining the principle of image conversion. It is a figure which shows the other example of a captured image. It is another figure explaining the principle of image conversion. It is another figure explaining the principle of image conversion. It is a flowchart explaining operation | movement of an image processing apparatus. It is a flowchart explaining the process of step S1 of FIG. It is a figure explaining a texture. It is a flowchart explaining the process which determines the reference value at the time of specifying a texture. It is a figure explaining the process which determines the reference value at the time of specifying a texture. It is another figure explaining the process which determines the reference value at the time of specifying a texture. It is another figure explaining the process which determines the reference value at the time of specifying a texture. It is a figure explaining the method of calculating regression line data. It is another figure explaining the method of calculating regression line data. It is a figure explaining the process which sets a regression line. It is a figure explaining the calculation method of a vanishing point. It is a figure explaining the method of calculating an elevation angle. It is another figure explaining the method of calculating an elevation angle. It is a figure explaining image conversion. It is another figure explaining image conversion. It is another figure explaining image conversion. It is another figure explaining image conversion. It is another figure explaining image conversion. It is another figure explaining image conversion. It is another figure explaining image conversion. It is a figure which shows the other example of a captured image. It is a block diagram which shows the other structural example of the image processing apparatus to which this invention is applied. It is a block diagram which shows the other structural example of an elevation angle calculation part. It is a figure which shows the example of the table in which the weight value was set. It is a figure which shows the example of the other table to which the weight value was set. It is a figure which shows the example of the table in which the weight value was set. It is a block diagram which shows the other structural example of an elevation angle calculation part. It is a flowchart explaining operation | movement of the elevation angle calculation part of FIG. It is a figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is another figure explaining operation | movement of the elevation angle calculation part of FIG. It is a block diagram which shows the other structural example of an elevation angle calculation part. It is a flowchart explaining operation | movement of the elevation angle calculation part of FIG. And FIG. 20 is a block diagram illustrating a configuration example of a computer 501.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 11 Image input part, 12 Elevation angle calculation part, 13 Image conversion part, 14 Processing part, 15 Image conversion part, 16 Image output part, 21 Quantization part, 22 Vanishing point calculation part, 23 Elevation angle calculation part, 51 feature amount extraction unit, 52 quantization unit, 61 texture gradient calculation unit, 62 vanishing point calculation unit, 71 image rotation unit, 72 direction determination unit, 73 switch, 81 quantization unit, 91 feature amount evaluation unit, 101 feature amount Extraction unit, 102 vanishing point calculation unit, 111 blocking unit, 121 blocked memory, 122 feature quantity calculation unit, 123 feature quantity determination unit, 124 storage unit, 125 address control unit, 11 radio unit, 12 code decoding processing unit, 13 control unit, 14 storage unit, 15 database unit, 16 display unit, 17 audio signal generation unit, 8 speaker

Claims (3)

Blocking means for blocking an input image obtained by imaging a subject at a predetermined elevation angle by an imaging means, and cutting out a block of a predetermined size from the input image ;
From the block of the input image, a maximum value, a minimum value, or a dynamic range of luminance values of pixels in the block, a maximum value, a minimum value, or a dynamic range of edge intensity of the pixels, and a histogram of the luminance values of the pixels Or a feature amount calculation means for extracting the distribution of the edge intensity histogram of the pixel as a feature amount of the block;
For each block of the input image, change the block size to a smaller size so that the difference between the feature amount of the block and a predetermined reference value is equal to or greater than a predetermined threshold value, Feature amount determination means for causing the block forming means to cut out the block having the changed size; and
For each line composed of the pixels arranged in a predetermined direction on the input image, an average value of the sizes of the blocks arranged on the line is calculated, and a regression line is set using the average value for each line. a vanishing point calculation means for calculating a vanishing point,
An elevation angle calculating means for calculating the elevation angle using the vanishing point;
An image processing apparatus comprising: an image processing unit that converts the input image into a facing image obtained when the subject is imaged from the front by the imaging unit using the elevation angle . A blocking step in which the blocking means blocks the input image obtained by imaging the subject at a predetermined elevation angle by the imaging means, and cuts out a block of a predetermined size from the input image ; and
From the block of the input image, a maximum value, a minimum value, or a dynamic range of luminance values of pixels in the block, a maximum value, a minimum value, or a dynamic range of edge intensity of the pixels, and a histogram of the luminance values of the pixels Or a feature amount calculating step of extracting the distribution of the edge intensity histogram of the pixel as a feature amount of the block;
For each block of the input image, change the block size to a smaller size so that the difference between the feature amount of the block and a predetermined reference value is equal to or greater than a predetermined threshold value, A feature amount determination step of cutting out the block having the changed size by the blocking unit;
For each line composed of the pixels arranged in a predetermined direction on the input image, an average value of the sizes of the blocks arranged on the line is calculated, and a regression line is set using the average value for each line. a vanishing point calculation step of calculating a vanishing point,
An elevation angle calculating step of calculating the elevation angle using the vanishing point;
An image processing method comprising: an image processing step of converting the input image into a facing image obtained when the subject is imaged from the front by the imaging means using the elevation angle . A blocking step in which the blocking means blocks the input image obtained by imaging the subject at a predetermined elevation angle by the imaging means, and cuts out a block of a predetermined size from the input image ; and
From the block of the input image, a maximum value, a minimum value, or a dynamic range of luminance values of pixels in the block, a maximum value, a minimum value, or a dynamic range of edge intensity of the pixels, and a histogram of the luminance values of the pixels Or a feature amount calculating step of extracting the distribution of the edge intensity histogram of the pixel as a feature amount of the block;
For each block of the input image, change the block size to a smaller size so that the difference between the feature amount of the block and a predetermined reference value is equal to or greater than a predetermined threshold value, A feature amount determining step of cutting out the block having the changed size by the blocking unit;
For each line composed of the pixels arranged in a predetermined direction on the input image, an average value of the sizes of the blocks arranged on the line is calculated, and a regression line is set using the average value for each line. a vanishing point calculation step of calculating a vanishing point,
An elevation angle calculating step of calculating the elevation angle using the vanishing point;
An image processing step of converting the input image into a facing image obtained when the subject is imaged from the front by the imaging means using the elevation angle is executed by a computer. program.

Images