JP2011259342A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of correcting a tilt included in an imaged image.SOLUTION: An image processing apparatus extracts tilt information included in images as an angle histogram to calculate relative tilt angles between two or more images. While a matching is performed by similarity between the angle histograms and a movement is corrected, the angle histograms are added cumulatively, and noise components are suppressed and tilt angle information is sharpened. From this result, a tilt angle is estimated to correct the tilt of the image.

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image correction technique.

  In recent years, a technique for correcting the tilt of an image has been proposed as a camera shake correction technique in a photographing apparatus such as a camera or a movie.

  Japanese Patent Application Laid-Open No. 2004-228688 provides a method in which an acceleration sensor is provided in an imaging apparatus, the inclination of the imaging apparatus with respect to the earth axis is detected by measuring gravitational acceleration, and the inclination of an image captured based on the detected inclination angle is corrected. Is disclosed. Patent Document 2 discloses a method for detecting line segments existing above a captured image, estimating the tilt angle of the entire image from the tilt angles of those line segments, and correcting the tilt of the captured image. . Patent Document 3 discloses a method for improving the accuracy of tilt angle calculation by using a combination of a sensor and image processing.

  Further, in Patent Document 4, when calculating the tilt angle of an image using image processing, the image is divided into small blocks, the directionality indicated by the texture in each block is determined, and the texture is unidirectional. A method for calculating an inclination angle only for a region having a curve is shown. This method is equivalent to extracting the tilt information from only the structure in the image, and contributes to improving the accuracy and stability of the tilt angle calculation.

JP 2006-245726 A Japanese Patent No. 3676360 International Publication No. 2009-001512 International Publication No. 2009-008174

  However, in the method using the sensor disclosed in Patent Document 1, the sensor output value includes fluctuation components such as inertia noise and other-axis sensitivity. Highly accurate tilt correction was difficult. Further, in Patent Document 2, there is a limitation on the composition of the captured image, which is not practical in a general use situation.

  In Patent Document 3 and Patent Document 4, it is proposed to use a sensor and image processing in combination in order to compensate for the above-described drawbacks. However, since these methods aim to calculate the tilt angle from a single image, the area occupied by structures in the image is smaller than that of non-structures such as natural objects and people. The tilt information from the non-structure is a noise component for calculating the tilt angle, and it is impossible to calculate the correct tilt angle. This problem becomes more prominent as the imaging angle of view of the imaging apparatus becomes wider. (A) and (b) of FIG. 1 are examples of images taken in the same direction with different angles of view, and the thick line is a component that contributes to the calculation of the tilt angle. Here, the inclination angle is θ. FIGS. 2A and 2B show angle histograms obtained by accumulating the tilt angle information of FIGS. 1A and 1B. Originally, as shown in FIG. The value is the maximum. However, since the length of the thick line in FIG. 1B is shorter than that in FIG. 1A, the tilt angle information extracted from the natural object as shown in FIG. 2B (the shaded area in FIG. 2 is the natural object). The tilt angle information to be extracted cannot be extracted by the tilt angle information extracted from (1). In order to solve this problem, Patent Document 4 discloses a technique for maintaining a histogram shape by removing natural objects in advance. However, even if the divided small block has directionality in one direction, there may be a subject with a messy direction in other areas. The subject itself has a small size and it is difficult to detect the peak of the angle histogram.

  An object of the present invention is to correct a tilt of an image by suppressing a noise component and enabling tilt angle estimation even when tilt angle information obtained from an image is relatively small.

  An image acquisition unit, an inclination information calculation unit that extracts information for estimating an image inclination with respect to each pixel of the image acquired by the image acquisition unit, and the inclination using the inclination information calculated by the inclination information calculation unit An angle histogram generation unit that generates an angle histogram that is a frequency distribution for each angle, an angle histogram storage unit that stores two or more angle histograms among the angle histograms generated by the angle histogram generation unit, and an angle histogram storage A reading unit that selects one reference angle histogram and one comparison target angle histogram different from the reference angle histogram from two or more angle histograms stored in the unit, and reads out from the angle histogram storage unit; and a reference angle histogram And a matching unit for examining the degree of similarity and movement amount of the comparison target angle histogram, When the determination result of the calculation enable / disable determining unit and the calculation enable / disable determining unit for determining whether the calculation is possible between the reference angle histogram and the comparison target angle histogram using the similarity output by the hatching unit is possible, A calculated angle histogram is generated by performing an operation using the amount of movement between the reference angle histogram and the comparison target angle histogram, and if the determination result is negative, the reference angle histogram is calculated without performing the calculation. An angle histogram calculation unit, a peak detection unit that detects the maximum value or maximum value of the calculated angle histogram, and an inclination angle of the image are estimated from the inclination angle indicated by the maximum value or maximum value detected by the peak detection unit. An image processing apparatus includes an inclination estimation unit and an image correction unit that corrects the inclination of an image using an estimated inclination angle.

  With this configuration, when estimating the tilt angle of a certain image, the tilt angle information of a plurality of images as well as the target image can be used.

  According to the image processing apparatus and method of the present invention, even when an inclination angle of an image is estimated, there is little inclination angle information included in the image, and it is difficult to estimate and correct the inclination angle in the past. The tilt angle can be estimated and corrected.

It is the figure which compared the normal view angle image (a) and the wide angle image (b). It is the figure which showed the influence of each structure and natural thing in the angle histogram of the image of (a) and (b) of FIG. It is a figure which shows the structure of the image processing apparatus in Embodiment 1 of this invention. It is a flowchart which shows the image processing method in Embodiment 1 of this invention. It is a figure explaining the brightness | luminance gradient in the point on a line segment. It is a figure which shows the coefficient matrix used with a Sobel filter. It is a figure explaining the parameter which characterizes the line segment in Hough conversion. It is a figure explaining the flow of an image, an angle histogram, and its accumulation addition. It is a figure which shows the structure of the image processing apparatus in Embodiment 2 of this invention. It is a figure which shows an example of the output value of 3 axes | shafts of an acceleration sensor. It is a figure which shows an example of the output value of the re-x direction and y direction which perform inclination detection using an acceleration sensor. It is a figure which shows an example which produces | generates the vector which points to a perpendicular direction from the output value of an acceleration sensor.

  The configuration and operation of the embodiment of the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 3 is a diagram showing the configuration of the image processing apparatus according to Embodiment 1 of the present invention. FIG. 4 is a flowchart showing the image processing method according to Embodiment 1 of the present invention.

  The image acquisition unit 301 takes the image data acquired by the imaging unit 307 into the image processing apparatus 300 and sends it to the tilt information calculation unit 302 (S401). The inclination information calculation unit 302 calculates inclination information for the captured image (S402). As the method, grayscale conversion is performed on the image, a vertical and horizontal Sobel filter is applied to each pixel, and the result is combined to calculate a gradient vector, or Hough conversion that extracts a line segment from the image However, the present invention is not limited to this method.

  Here, calculation of the gradient vector will be described with reference to FIGS. In FIG. 5, when the pixel value of the image at the position P is I (P) and the contrast changes in the x and y directions at P are dx and dy, respectively.

It can be expressed. At this time, if the direction of contrast change is θ,

And this corresponds to the direction of the gradient vector. The Sobel filter shown in FIG. 6 is used to obtain dx and dy. 6A and 6B are filter elements for measuring contrast changes in the x direction and the y direction, respectively. Next, line segment detection by Hough transform will be described with reference to FIG. In FIG. 7A, a straight line passing through the point (x_i, y_i) satisfies the following equation.

  Here, ρ represents the distance between the straight line and the origin, and θ represents the angle between the perpendicular line from the origin to the straight line and the x axis.

  This means that a group of straight lines passing through the point (x_i, y_i) is associated with one curve in (b) of FIG. On the other hand, when the same correspondence is performed with respect to a plurality of points on the same straight line, a plurality of obtained curves intersect at one point, and this intersection (ρ, θ) indicates a straight line passing through these points. That is, paying attention to a certain θ, the number of line segments of the inclination θ can be acquired by counting the number of ρ.

  The angle histogram generator 303 accumulates the gradient vectors having the same direction component, or adds the lengths of the line segments obtained by the Hough transform having the same direction component. A histogram is generated (S403). This angle histogram is an index that represents "how many edges exist in a certain angle direction". Generally, when only the structure is shown in the image with no tilt (when there is no distortion), the horizontal and vertical line components Therefore, the angle histogram values of 0 degree and 90 degrees have a strong peak. If the image is tilted by θ as shown in FIGS. 1 and 2, an angle histogram obtained by translating the angle histogram when there is no tilt by θ is obtained.

  In the tilt angle estimation by the conventional image processing, the peak detection unit 304 searches for the angle indicating the maximum value of the angle histogram, and the tilt estimation unit 305 directly uses the angle indicating the maximum value of the angle histogram as the image tilt angle to perform image correction. The unit 306 performs processing for correcting the tilt angle. Alternatively, the peak detection unit 304 selects several local maximum points including the maximum value, and the slope estimation unit 305 calculates an envelope that smoothly connects the local maximum values by polynomial interpolation, the least square method, the gradient method, and the like. The maximum value is calculated analytically, and this is used as the tilt angle of the image, and the tilt is corrected by the image correction unit 306. In general, the image correction unit 306 corrects rotation by affine transformation. In the present invention, the same processing is performed for the estimation of the tilt angle from the angle histogram and the correction by the affine transformation.

  The angle histogram storage unit 310 temporarily stores an angle histogram after each frame of the image or movie captured by the imaging unit 307 is processed by the processing blocks 301 to 303 (S411). The reading unit 311 reads the angle histogram of the image to be tilt-corrected as the reference angle histogram, the angle histogram of the image temporally adjacent to the image as the comparison target angle histogram, and reads both from the angle histogram storage unit 310, and the matching unit 312. (S412).

  The matching unit 312 determines the similarity between the two histograms (S413). This process will be described with reference to FIG. In general, when a movie is shot by the image pickup unit 307, the time interval between frames is about 33 ms to 66 ms (30 frames / second to 15 frames / second), which is a major component of temporally adjacent images. Since there is no change, there is inevitably a large difference in their angle histograms. Therefore, the angle histograms of the two images are translated by the relative rotation angle of each other. Since the matching unit 312 only needs to be able to calculate the similarity between two angle histograms and the amount of parallel movement, the cross-correlation function generally used to measure the similarity of images and signal waveforms, and the similarity of histograms Is calculated by a histogram intersection or the like that is often used to measure. The cross-correlation function has the maximum value as the similarity, and the variable that gives the maximum value is the relative movement amount existing between the two histograms. In the histogram intersection, by shifting the variable for calculating the intersection, The maximum histogram intersection value is the similarity, and the shift amount of the variable giving it is the movement amount. The relative movement amount is treated with the reference histogram side as a reference, and is regarded as “the movement amount of the comparison target angle histogram with respect to the reference angle histogram”.

  The calculation availability determination unit 313 compares the similarity calculated by the matching unit 312 with a predetermined reference, and determines whether or not the calculation can be performed between the two angle histograms (S414). For example, when a cross-correlation function is used, the reference value may be set to x% of the auto-correlation function because the maximum value does not exceed the maximum value of the auto-correlation function of the reference angle histogram. Alternatively, when using a histogram intersection, if the maximum value of the intersection between the reference angle histograms is defined as 100%, it may be y% thereof. Here, it is assumed that the angle histograms of temporally adjacent images are compared, and it is assumed that there is no significant change in the configuration of the images, so a relatively large value such as x = 80, y = 75 is set. Better.

  The angle histogram calculation unit 314 receives the determination result of the calculation determination unit 313, the amount of movement of the comparison angle histogram with respect to the reference angle histogram, and two angle histograms, and generates a calculated angle histogram according to the determination result. To do. If the determination result is acceptable, the calculated angle histogram is calculated (added) for each variable between the reference angle histogram and the comparison target angle histogram in which the variable is shifted by the movement amount (S415). When the determination result is negative, the reference angle histogram is set as the post-calculation angle histogram without performing the calculation (S416).

  Here, the inclination angle information calculated by the inclination information calculation unit 302 can be roughly classified into two types. One is tilt angle information having a strong direction extracted from a structure, and the other is tilt angle information having a large variance value of tilt angle information extracted from a non-structure (natural object, person, etc.). Since the tilt angle information from the non-structure has a low contrast as compared with the structure, the tilt angle information is not stable and thus has white noise characteristics. For example, foliage, bark, lawn, etc. fluctuate in the tilt angle information for each image. On the other hand, since the structure has high contrast, the tilt angle information continues to be output in a predetermined direction stably. Although these two components are mixed in the angle histogram, the inclination angle information from the non-structure is averaged by performing the cumulative addition after correcting the movement amount calculated by the matching unit 312. On the other hand, the tilt angle information from the structure is emphasized.

  Therefore, by performing the steps from S401 to S416 between images having appropriate time widths, the influence of non-structures in the post-computation histogram becomes relatively small, and conversely, information from the structures is sharpened. . As a result, the tilt angle information of the structure buried in the non-structure shell tilt angle information can be detected by the peak detection unit 304 (S404). The inclination estimation unit 305 sets the angle that gives the mode value detected by the peak detection unit 304 as the inclination estimation angle, and sends this information to the image correction unit 306 (S405). The image correction unit 306 can correct the amount of rotational movement between images by rotating the entire image by performing, for example, affine transformation. Here, the inclination estimation unit 305 obtains the peak detection unit 304 when the angle that gives the mode value detected by the peak detection unit 304 is significantly different from the inclination angle estimated by the inclination estimation unit 305 immediately before. The angle estimated just before may be used as the tilt angle. Alternatively, an appropriate error allowable range (for example, ± 1 °) is set for the inclination angle estimated immediately before by the inclination estimation unit 305, and the histogram mode value within this range is extracted from the peak detection unit 304. This may be the estimated angle. In either case, the image correction unit 306 finally rotates the entire image to complete the correction of the target image (S406).

  The corrected image may be stored in the storage unit 308 and then displayed on a monitor device (not shown) or saved as a file in an external storage device (not shown). Moreover, you may send out via a network.

  The above is the configuration and operation of the image processing apparatus 300 which is an example of the present invention.

  Note that no processing may be performed on an image that does not have information necessary to estimate the tilt angle. For example, in an image in which a person or a natural object is shown on the entire screen, the tilt information cannot be calculated because there is no structure information. In such a case, the sensitivity to the tilt of the image is low visually (it is difficult to perceive a decrease in quality even if it is tilted), so an unstable and low accuracy tilt estimation angle extracted from a person or a natural object is used. It is better to cancel the correction itself than to use it for correction.

  A buffer that stores the estimated tilt angle may be provided. This buffer stores the estimated tilt angle calculated in time series, and when the tilt angle of a single image cannot be calculated, the result of successful tilt angle estimation before and after that is used. The estimation may be performed by performing interpolation. This is because the inclination angle of an image generally changes continuously in a device that acquires continuous images such as a movie, and by using this property, extrapolation processing can be performed from past inclination angle estimation results. It is also possible to estimate the tilt angle of the target image.

(Embodiment 2)
FIG. 9 is a diagram showing the configuration of the image processing apparatus according to Embodiment 2 of the present invention. 9, the same components as those in FIG. 3 are denoted by the same reference numerals, and description thereof is omitted.

  The sensor unit 901 is installed inside or outside the imaging unit 307, and measures and outputs the movement of the imaging unit 307 at a predetermined cycle. Generally, it is better to measure the movement in synchronization with the image capturing unit 307 capturing an image. The sensor unit 901 includes an acceleration sensor, a gyro sensor, an orientation sensor, and the like. Ideally, the sensor unit 901 is required to have three measurement axes in order to measure all the movements of the imaging unit 307. You don't need an axis. Image data and sensor data are handled so as to be synchronized inside the image processing apparatus 300. The sensor acquisition unit 902 processes the acquired sensor data and calculates auxiliary information for calculating the estimated tilt angle.

  For example, FIG. 10 shows an example of triaxial output values of the acceleration sensor. In the present embodiment, the output value of the sensor itself is not so important, and the period and timing at which the waveform changes are more important than that. In FIG. 10, the waveforms change on all axes at the timings (a), (b), and (c), and imaging is performed at this timing rather than that some movement has occurred about a certain axis. It can be seen that an impact was applied to the entire portion 307. If this impact is caused by, for example, a foot landing during walking, the period from (a) to (b) or (b) to (c) is relatively stable, and the image composition itself is It can be predicted that there is no significant change in comparison with the previous and subsequent images. Therefore, the angle histograms generated from the images taken in the respective periods have similar shapes, and can be cumulatively added. In general, walking is about 2 Hz, running is ˜5 Hz, and in the case of a moving image shot at 30 fps, it is possible to cumulatively add about 6 to 15 images. By inputting such information to the reading unit 311, reading the angle histogram from the angle histogram storage unit 310, performing matching, calculation feasibility determination, and securing histogram calculation (cumulative addition), a more accurate estimated slope The angle can be calculated.

  Although an example with three axes of the acceleration sensor is used here, a gyro sensor may be used. Further, it is not always necessary to have three axes, and it may be more or less than that. However, in order to determine whether the movement of the imaging unit 307 itself is detected or due to a disturbance factor such as an impact during walking, the information should be redundant, and at least two axes It is desirable to provide the above (may be different sensors).

  Further, for example, the inclination angle of the imaging unit 307 can be roughly estimated by a sensor using an acceleration sensor or a gyro sensor, and a constraint condition can be set when the estimated inclination angle is calculated by the inclination estimation unit 305. FIG. 11 is an example showing changes in acceleration in the x and y directions. FIG. 12 shows an acceleration obtained in each direction at the timings (a), (b), and (c) in FIG. 11 and mapped to a two-dimensional plane and vectorized. The vectors of (a), (b), and (c) in FIG. 12 indicate the inclination angle with respect to the vertical direction when the image capturing unit 307 captures the image. Therefore, the image is rotated so that the vector is directed in the vertical direction. Can remove the tilt. By the way, in shooting while walking, the sensor value may not be accurate due to inertia noise, other axis sensitivity, or the like. In that case, correction by the method clarified in the present invention is required, but the effect of the present invention can be further enhanced by inputting the tilt direction measured by the sensor to the tilt estimation unit 305. That is, it is highly likely that the direction indicated by the sensor is the true vertical direction of the image, a predetermined range is set around the direction indicated by the sensor, and the peak detection unit 304 has a range within the set range. The mode value is defined as the tilt angle. The range setting may be a fixed value or a fluctuation value. In the case of a fluctuation value, it may be changed depending on the magnitude of movement, that is, the amplitude or stability of the sensor (a dispersion value within a predetermined time range can be used). . In this case, when the motion is small, it is determined that the error of the sensor output value is small and the range is set narrow. When the motion is large, the error is determined to be large and the range is set wide. The change may be continuous or a discrete value having two or more steps. Alternatively, when the information for calculating the estimated angle cannot be obtained from the image, the estimated angle may be calculated from the past estimation result, the previous and subsequent estimation results, and the sensor value. In that case, the assumption is made that the tilt angle also changes continuously if the images are acquired continuously in time series, and an allowable value is set for the angle that has already been estimated. If the sensor value falls within the range, a method may be considered in which the value is used as a correction value.

  The above is the configuration and operation of the image processing apparatus 300 according to the second embodiment of the present invention.

(Other variations)
Although the present invention has been described based on the above embodiment, it is needless to say that the present invention is not limited to the above embodiment. The following cases are also included in the present invention.

  (1) Each of the above devices is specifically a computer system including a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse, and the like. A computer program is stored in the RAM or hard disk unit. Each device achieves its functions by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

  (2) A part or all of the constituent elements constituting each of the above-described devices may be configured by one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip, and specifically, a computer system including a microprocessor, ROM, RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

  (3) Part or all of the constituent elements constituting each of the above devices may be configured from an IC card that can be attached to and detached from each device or a single module. The IC card or the module is a computer system including a microprocessor, a ROM, a RAM, and the like. The IC card or the module may include the super multifunctional LSI described above. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

  (4) The present invention may be the method described above. Further, the present invention may be a computer program that realizes these methods by a computer, or may be a digital signal composed of the computer program.

  The present invention also provides a computer-readable recording medium such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu-ray Disc). ), Recorded in a semiconductor memory or the like. The digital signal may be recorded on these recording media.

  In the present invention, the computer program or the digital signal may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.

  The present invention may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program, and the microprocessor operates according to the computer program.

  In addition, the program or the digital signal is recorded on the recording medium and transferred, or the program or the digital signal is transferred via the network or the like, and executed by another independent computer system. It is good.

  (5) The above embodiment and the above modifications may be combined.

  The image processing apparatus according to the present invention can be incorporated into a photographing apparatus, an image display apparatus, or a video display apparatus, thereby correcting the inclination of the acquired image and generating an image with the correct orientation. Even for an image that has been difficult to be corrected by a conventional tilt correction apparatus using image processing, tilt information of a desired image can be extracted by integrating a plurality of pieces of image information. The present invention can be applied not only to a photographing apparatus and a display apparatus but also to tilt correction other than electronic media such as a printer or a scanner that handles video.

DESCRIPTION OF SYMBOLS 300 Image processing apparatus 301 Image acquisition part 302 Inclination information calculation part 303 Angle histogram generation part 304 Peak detection part 305 Inclination estimation part 306 Image correction part 307 Imaging part 308 Storage part 310 Angular histogram storage part 311 Reading part 312 Matching part 313 Computability Determination unit 314 Angle histogram calculation unit 901 sensor unit 902 sensor acquisition unit

Claims (17)

An image acquisition unit;
An inclination information calculation unit for extracting information for estimating the inclination of the image for each pixel of the image acquired by the image acquisition unit;
An angle histogram generation unit that generates an angle histogram that is a frequency distribution for each inclination angle using the inclination information calculated by the inclination information calculation unit;
An angle histogram storage unit that stores two or more of the angle histograms among the angle histograms generated by the angle histogram generation unit;
Among the two or more angle histograms stored in the angle histogram storage unit, one reference angle histogram and one comparison target angle histogram different from the reference angle histogram are selected and read out from the angle histogram storage unit. A reading unit;
A matching unit that examines the similarity and movement amount of the shape of the reference angle histogram and the comparison target angle histogram;
A calculation availability determination unit that determines whether calculation is possible between the reference angle histogram and the comparison target angle histogram using the similarity output from the matching unit;
When the determination result of the calculation possibility determination unit is acceptable, calculation is performed using the movement amount between the reference angle histogram and the comparison target angle histogram to generate a calculated angle histogram, and the determination result is If not, an angle histogram calculation unit that sets the reference angle histogram as the post-calculation angle histogram without performing calculation,
A peak detection unit for detecting the maximum value or the maximum value of the angle histogram after the calculation;
An image processing apparatus comprising: an inclination estimation unit that estimates an inclination angle of an image from an inclination angle indicated by the maximum value or the maximum value detected by the peak detection unit. The readout unit selects the post-calculation angle histogram as a reference angle histogram, and selects one angle histogram that is not used for the post-computation angle histogram generation from the angle histogram storage unit as the comparison target angle histogram. Read from the
The image processing apparatus according to claim 1. The reading unit selects and reads one or more angle histograms other than the reference angle histogram and one reference angle histogram from the angle histograms stored in the angle histogram storage unit,
The matching unit examines the degree of similarity and the amount of movement between the two or more comparison target angle histograms and the reference angle histogram,
The calculation possibility determination unit determines whether calculation is possible using each of the similarities,
The angle histogram calculation unit performs the calculation with the reference angle histogram using only the angle histogram in which the determination result of the calculation availability determination unit is possible among the two or more comparison target angle histograms, and after the calculation The image processing apparatus according to claim 1, wherein an angle histogram is generated, and the reference angle histogram is used as the post-calculation angle histogram without performing a calculation when all the determination results are negative. An imaging unit for taking an image;
A sensor unit for sensing information other than an image captured by the imaging unit;
It further includes a sensor acquisition unit that acquires information acquired by the sensor unit, and extracts a feature of the movement of the imaging unit from the information acquired by the sensor unit to set a time-series interval in which the type of movement is the same The image processing apparatus according to claim 2, wherein the angle histogram obtained from the frames included in the section is selected and cumulative addition is performed. The matching unit calculates a cross-correlation function between the reference angle histogram and the comparison target angle histogram, the maximum value of the cross-correlation function is the similarity, and a variable (inclination angle) that gives the maximum value is the movement amount. The image processing apparatus according to claim 1. The similarity in the matching unit is a histogram intersection of the reference angle histogram and the comparison target histogram, and the movement amount is a deviation amount that gives the maximum value of the histogram intersection value obtained by shifting both histogram variables. The image processing apparatus according to claim 1. The image processing according to claim 1, wherein the calculation in the angle histogram calculation unit is performed after correcting the movement amount so that corresponding portions of the shapes of the reference angle histogram and the comparison target angle histogram coincide with each other. apparatus. The image processing apparatus according to claim 6, wherein the calculation in the angle histogram calculation unit is addition of values of corresponding variables (inclination angles). The image processing apparatus according to claim 6, wherein the calculation in the angle histogram calculation unit is an integration of values of corresponding variables (tilt angles). The image processing apparatus according to claim 1, wherein the inclination information calculation unit uses, as inclination information, a vector obtained by combining the vertical and horizontal edge strengths of each pixel. The image processing apparatus according to claim 1, wherein the inclination information calculation unit is a length and direction of a line segment obtained by Hough transform. The image processing apparatus according to claim 1, wherein the inclination estimation unit acquires information other than an image from the sensor acquisition unit, and estimates an inclination angle while constraining an angle range as an inclination angle using the information. The image processing apparatus according to claim 1, further comprising an image correction unit that corrects an image according to the tilt information output by the tilt estimation unit. An image acquisition step;
An inclination information calculation step for extracting information for estimating the inclination of the image for each pixel of the image acquired in the image acquisition step;
Using the inclination information calculated in the inclination information calculation step, an angle histogram generation step for generating an angle histogram that is a frequency distribution for each inclination angle;
An angle histogram storage step of storing two or more of the angle histograms among the angle histograms generated in the angle histogram generation step;
A reading step of selecting and reading one reference angle histogram and one comparison target angle histogram different from the reference angle histogram among the two or more angle histograms stored in the angle histogram storage step;
A matching step for examining the similarity and movement amount of the shape of the reference angle histogram and the comparison target angle histogram;
A calculation possibility determination step for determining whether calculation is possible between the reference angle histogram and the comparison target angle histogram using the similarity output in the matching step;
When the determination result of the calculation possibility determination step is acceptable, calculation is performed using the movement amount between the reference angle histogram and the comparison target angle histogram to generate a calculated angle histogram, and the determination result is If not, an angle histogram calculation step in which the reference angle histogram is set as the post-calculation angle histogram without performing a calculation;
A peak detecting step for detecting a maximum value or a maximum value of the angle histogram after the calculation;
An image processing method comprising: an inclination estimation step for estimating an inclination angle of an image from an inclination angle indicated by the maximum value or the maximum value detected in the peak detection step. An integrated circuit including the image processing apparatus according to claim 1. A program for causing a computer to execute the image processing method according to claim 14. A storage medium storing the program according to claim 16.