JP2017083817A - Deviation amount acquisition device, imaging apparatus, and deviation amount acquisition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly accurately acquire a deviation amount while reducing computational complexity.SOLUTION: A deviation amount acquisition device comprises: correlation operation means for acquiring a first data string that is a data string of a correlation amount between a first signal including an image signal of a plurality of pixels at a predetermined pixel interval based on a first image and a second signal including an image signal of a plurality of pixels at the predetermined pixel interval based on a second image, and acquiring a second data string that is a data string of a correlation amount between a third signal including an image signal that is an image signal based on the second image, of a plurality of pixels at the predetermined pixel interval and corresponding to a signal acquired in a pixel at a different position from the second signal in the case where a displacement amount giving an extremal value of the first data string is smaller than a threshold value, and the first signal; and deviation amount acquisition means for acquiring, as the deviation amount, a displacement amount giving an extremal value of an approximate function representing a relation of a displacement amount calculated from data included in the first data string and the second data string, and a correlation amount.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a deviation amount acquisition device, an imaging device, and a deviation amount acquisition method.

  There is known a method of detecting a distance using image signals taken from a plurality of viewpoints with a digital still camera or a video camera (Non-Patent Document 1). Further, there is known a method of performing distance measurement from an image signal captured by a camera in which distance measurement pixels having a distance measurement function are arranged in some or all pixels of an image sensor (Patent Document 1). It is possible to calculate the amount of deviation of the relative position of the pair of stereo image signals and measure the distance to the subject by triangulation.

  It is known that the accuracy of distance measurement decreases when the amount of deviation between a pair of image signals is different from the interval (pixel interval) between data constituting the image signal. Non-Patent Document 1 proposes a technique for reducing a positional deviation amount calculation error and improving distance measurement accuracy. Specifically, two image signals corresponding to image signals acquired at different positions on the image sensor are generated from one image signal of a stereo image. Then, a deviation amount is calculated from each of the two sets of stereo image signals, and the final deviation amount is calculated by averaging both the deviation amounts.

JP 2002-314062 A US Pat. No. 7,599,512 U.S. Pat. No. 8723926

Masao Shimizu and Masatoshi Okutomi, "Sub-Pixel Estimation Error Cancellation on Area-Based Matching", International Journal of Computer Vision, Vol. 63, No. 3, pp. 207-224, 2005

  The technique described in Non-Patent Document 1 uses that two shift amounts including errors in the opposite directions and the same degree can be acquired by using the above-described two sets of image signals. When the two deviation amounts satisfy this condition, the deviation amount acquisition error is reduced by averaging these deviation amounts.

  However, the error of the shift amount changes according to the shift amount of the pair of image signals. When the deviation amount is an integral multiple or a half integer multiple of the interval between the data constituting the image signal, the error of one of the two deviation amounts is greatly increased. For this reason, the improvement effect due to the average is reduced, the accuracy of obtaining the deviation amount is lowered, and the ranging accuracy is lowered. Further, according to the conventional method, the shift amount is acquired for each of the two sets of image signals, so that the calculation load increases.

  In view of the above problems, an object of the present invention is to provide a shift amount acquisition device and a distance detection device that can acquire a high-precision image shift amount with a small calculation load.

A first aspect of the present invention is a shift amount acquisition device that acquires a shift amount of a relative position between a first image and a second image having different viewpoints acquired by an imaging device including a plurality of pixels. And
Correlation calculation means for acquiring a data string of correlation amounts for the respective displacement amounts when the two image signals are relatively displaced;
A shift amount acquisition means for acquiring the shift amount between the first image and the second image from a data string of correlation amounts;
With
The correlation calculation means includes
An image signal based on the first image, the first signal including an image signal of a plurality of pixels having a predetermined pixel interval, and an image signal based on the second image, wherein the predetermined pixel interval A first data sequence that is a data sequence of the correlation amount between the second signal including image signals of a plurality of pixels
An image signal based on the second image when an amount of displacement giving an extreme value of the first data string is smaller than a threshold value, the image signal including a plurality of pixels at the predetermined pixel interval; A second data string that is a data string of the correlation amount between a third signal corresponding to a signal acquired in a pixel at a position different from the signal of 2 and the first signal;
The deviation amount acquisition means is
When the amount of displacement that gives the extreme value of the first data sequence is smaller than the threshold value, the relationship between the amount of displacement and the correlation amount obtained from the data included in the first data sequence and the second data sequence is expressed. A displacement amount that gives an extreme value of the approximate function is acquired as the deviation amount.
It is characterized by that.

According to a second aspect of the present invention, there is provided a shift amount acquisition device that acquires a shift amount of a relative position between a first image and a second image having different viewpoints acquired by an imaging device including a plurality of pixels. A method for obtaining a deviation amount to be executed,
An image signal based on the first image, the first signal including an image signal of a plurality of pixels having a predetermined pixel interval, and an image signal based on the second image, wherein the predetermined pixel interval Obtaining a first data string that is a data string of the correlation amount between a second signal including image signals of a plurality of pixels, and
An image signal based on the second image when an amount of displacement giving an extreme value of the first data string is smaller than a threshold value, the image signal including a plurality of pixels at the predetermined pixel interval; Obtaining a second data string that is a data string of the correlation amount between a third signal corresponding to a signal acquired in a pixel at a position different from the signal of 2 and the first signal. When,
When the amount of displacement that gives the extreme value of the first data sequence is smaller than the threshold value, the relationship between the amount of displacement and the correlation amount obtained from the data included in the first data sequence and the second data sequence is expressed. Obtaining a displacement amount giving an extreme value of an approximate function as the deviation amount;
It is characterized by including.

  According to the present invention, it is possible to acquire a highly accurate image shift amount with a small calculation load.

Schematic diagram showing an example of an imaging device having a distance detection device according to the present embodiment Diagram explaining sensitivity characteristics and pupil area of ranging pixels The figure which shows an example of the flow of the distance detection method which concerns on this embodiment. The figure which shows an example of the image signal of the distance detection method which concerns on this embodiment The figure which shows an example of the data sequence of the distance detection method which concerns on this embodiment The figure explaining the example of the conventional distance detection method The figure which shows the principle of the distance detection method which concerns on this embodiment. The figure which shows the effect of the distance detection method which concerns on this embodiment Schematic diagram showing an example of an imaging device having a distance detection device according to the present embodiment The figure which shows an example of the flow of the distance detection method which concerns on this embodiment.

  An embodiment of a distance detection device (deviation amount acquisition device) that acquires a displacement amount of a relative position from two images having parallax and detects a distance of a subject based on the displacement amount will be described. In the following description, a digital still camera will be described as an example of an imaging device including the distance detection device according to the embodiment of the present invention. However, the application of the present invention is not limited to this. For example, the distance detection device according to the present invention can be applied to a digital video camera, a digital distance measuring device, and the like.

  In the description with reference to the drawings, even if the drawing numbers are different, in principle, the same reference numerals are given to the portions indicating the same portions, and overlapping descriptions are avoided as much as possible.

<Distance detection device>
FIG. 1A is a schematic diagram of an imaging apparatus having a distance detection device 40 according to the present embodiment. In addition to the distance detection device 40, the imaging device includes an imaging element 10, an imaging optical system 20, and a recording device 30. Furthermore, the imaging apparatus includes a drive mechanism for focusing the imaging optical system 20, a shutter, an ornamental image generation unit, a display such as a liquid crystal for image confirmation, and the like.

  FIG. 1B is a schematic diagram illustrating an example of the image sensor 10. The image sensor 10 has a plurality of ranging pixels 13 including the photoelectric conversion units 11 and 12 (hereinafter, the ranging pixels are also simply referred to as pixels). Specifically, a solid-state imaging device such as a CMOS sensor (a sensor using a complementary metal oxide semiconductor) or a CCD sensor (a sensor using a charge coupled device) can be used as the imaging device 10.

  FIG. 1C is a schematic cross-sectional view illustrating an example of the pixel 13. The photoelectric conversion units 11 and 12 of the pixel 13 are formed in the substrate 14. The pixel 13 has a microlens 15. The microlens 15 is disposed so that the exit pupil 21 and the photoelectric conversion units 11 and 12 are optically conjugate.

  As shown in FIG. 1A, the imaging optical system 20 forms an image of an external subject on the surface of the image sensor 10. The imaging device 10 acquires the light beam transmitted through the exit pupil 21 of the imaging optical system 20 by the photoelectric conversion unit 11 or the photoelectric conversion unit 12 of the pixel 13 via the microlens 15 and converts it into an electric signal. Specifically, the light beam that has passed through the first pupil region 23 (FIG. 2B) of the exit pupil 21 is converted into an electrical signal by the photoelectric conversion unit 11 of each pixel 13. Further, the light beam that has passed through the second pupil region 24 (FIG. 2C) different from the first pupil region of the exit pupil 21 is converted into an electrical signal by the photoelectric conversion unit 12 of each pixel 13. The pixel 13 includes a floating diffusion (FD) portion, a gate electrode, wiring, and the like in order to output an electric signal to the distance detection device 40.

  All pixels of the image sensor 10 may be distance measuring pixels 13. Alternatively, the image sensor 10 may include a pixel having a single photoelectric conversion unit and a distance measuring pixel 13. By adding the signals acquired by the plurality of photoelectric conversion units 11 and 12 of the pixel 13, an image signal equivalent to that having a single photoelectric conversion unit can be created. Alternatively, the imaging device 10 includes only a pixel including only the photoelectric conversion unit 11 that receives the light beam that has passed through the first pupil region 23 and only the photoelectric conversion unit 12 that receives the light beam that has passed through the second pupil region 24. You may have a pixel. Note that the pixels 13 may be arranged discretely in the imaging element 10 or may be arranged at different intervals in the X direction and the Y direction.

The distance detection device 40 corresponds to the first signal S ₁ corresponding to the light beam that has passed through the first pupil region 23 of the exit pupil 21 of the imaging optical system 20 and the light beam that has passed through the second pupil region 24. a second signal S _2, based on, and has a function of calculating the distance to the subject. Specifically, the distance detection device 40 includes functional units such as a signal generation unit 41, a correlation calculation unit 42, a deviation amount calculation unit 43, and a distance calculation unit 44. For example, the signal processing board includes a CPU and a memory, and these functions are realized by the CPU executing a program. The signal processing board can be configured by using an integrated circuit in which semiconductor elements are integrated, and can be configured by an IC, an LSI, a system LSI, a micro processing unit (MPU), a central processing unit (CPU), or the like.

The first signal S ₁ is a set of electrical signals generated by the photoelectric conversion unit 11 of each pixel 13. In the first signal S ₁ , the position of each pixel 13 on the image sensor is associated with each electrical signal generated by the photoelectric conversion unit 11 of each pixel 13. The second signal S ₂ is a set of electrical signals generated by the photoelectric conversion portion 12 of each pixel 13. In the second signal S _2, the electrical signal generated by the position and the photoelectric conversion portion 12 of each pixel on the image pickup element of each pixel 13 and are associated. By adding the signals acquired by the photoelectric conversion units 11 and 12 of the pixel 13, an image signal equivalent to the case of having a single photoelectric conversion unit can be created.

  In the present embodiment, the distance between the imaging optical system 20 and the image sensor 10 is sufficiently long with respect to the size of the pixel 13. Therefore, light beams that have passed through different positions on the exit pupil 21 of the imaging optical system 20 are incident on the surface of the image sensor 10 as light beams having different incident angles. A light beam from a predetermined angle range 22 (FIG. 1A) is incident on the photoelectric conversion units 11 and 12 in accordance with the shape of the exit pupil 21 and the image height (position on the image sensor where the light beam reaches). .

The sensitivity distribution on the exit pupil when the sensitivity characteristics of the photoelectric conversion units 11 and 12 with respect to the incident light beam are projected onto the exit pupil according to the angle is referred to as pupil transmittance distribution. The position of the center of gravity of the pupil transmittance distribution at this time is called the pupil center of gravity. The pupil center of gravity can be calculated by the following equation (1). In Expression (1), r is a coordinate on the exit pupil 21, t represents a pupil transmittance distribution of the photoelectric conversion units 11 and 12, and an integration range is an area on the exit pupil 21.

Of the region on the exit pupil 21 through which the light beam received by the photoelectric conversion unit passes, the region including the center of gravity of the pupil and the sensitivity of the corresponding photoelectric conversion unit higher than a predetermined threshold is referred to as a pupil region. The direction connecting the pupil centroids of the two pupil regions is called the pupil division direction, and the length between the pupil centroids is called the baseline length. In the present embodiment, the pupil division direction is the x direction, which is the first direction, and the y direction perpendicular to the x direction is the second direction. With such a configuration, it is possible to acquire the image signals S ₁ and S ₂ of the subject from different viewpoints. Since the image signals S ₁ and S ₂ have parallax and the position of the subject image is shifted, the distance of the subject can be detected from the amount of deviation.

  In this embodiment, the amount of deviation in the optical axis direction (z direction) between the imaging surface of the imaging optical system 20 and the light receiving surface of the image sensor 10 is referred to as a defocus amount.

  FIG. 2A shows the sensitivity characteristic 16 of the photoelectric conversion unit 11 and the sensitivity characteristic 17 of the photoelectric conversion unit 12 with respect to the incident light beam in the xz plane. The horizontal axis represents the angle formed between the incident light beam and the z axis in the xz plane, and the vertical axis represents the sensitivity.

FIG. 2B is a diagram showing an exit pupil 21 of the imaging optical system 20, a pupil transmittance distribution 25 corresponding to the photoelectric conversion unit 11, a pupil center of gravity 27, and a pupil region 23 (first pupil region). is there. The pupil region 23 is a pupil region that is eccentric in the + x direction (first direction) from the center of the exit pupil 21. The photoelectric conversion unit 11 of each pixel 13 is configured to receive mainly the light flux that has passed through the pupil region 23. With this configuration, the first signal S ₁ corresponding to the light beam that has passed through the pupil region 23 is obtained.

FIG. 2C is a diagram showing an exit pupil 21 of the imaging optical system 20, a pupil transmittance distribution 26 corresponding to the photoelectric conversion unit 12, a pupil center of gravity 28, and a pupil region 24 (second pupil region). is there. The pupil region 24 is a pupil region that is eccentric in the −x direction (second direction) from the center of the exit pupil 21. The photoelectric conversion unit 12 of each pixel 13 is configured to mainly receive the light flux that has passed through the pupil region 24. With this configuration, the second signal S ₂ corresponding to the light beam passing through the pupil region 24 is obtained.

<Distance detection method>
FIG. 3 is an example of a flowchart of a distance detection method for detecting the distance to the subject performed by the distance detection device 40. This distance detection method includes a signal generation step (step S10), a correlation calculation step (step S11), a deviation amount calculation step (step S12), and a distance calculation step (step S13).

4A to 4C are diagrams illustrating an example of each signal. In order to simplify the description, a case will be described in which the signals S ₁ and S ₂ are one-dimensional signals acquired in a certain row 18 of the image sensor 10 in FIG. FIG. 4B is a diagram illustrating an example of the signals S ₁ and S ₂ . In the figure, the horizontal axis represents the position on the image sensor 10, and the vertical axis represents the signal intensity. Each point represents data constituting the signal. n is a data number representing the position of each data on the image sensor 10 and is an arbitrary real number sequentially assigned in the pupil division direction (x direction). The signals S ₁ and S ₂ are electric signals acquired by the photoelectric conversion unit of each pixel 13 in the row 18 and have the same data interval P as the size of the pixel 13.

[Signal generation process]
In the signal generating step (S10), the signal generation unit 41, the signal S ₂ and have the same data interval, the signal S ₃ and S ₂ corresponding to the acquired signal at different positions in the pupil division direction on the imaging device 10 Is generated. Signal _{S 3} is equivalent to the signal obtained in position 19 between the pixels 13 which has acquired the signal _{S 2} (see Figure 4 (a)). Signal S _3, for example according to equation (2) it is created by linear interpolation between the data signal S ₂ at the periphery of each data. In Expression (2), α is a real number greater than 0 and less than 1, and is a coefficient obtained by normalizing the distance in the pupil division direction of the position of each data of the signal S ₃ and the signal S ₂ with the data interval.

FIG. 4 (c) shows an example of the signal _{S 3.} In FIG. 4 (c), the data of the signal _{S 2} is black circle, the data signal _{S 3} are shown by white circles.

The signal S ₃ may be generated by non-linear interpolation between the signal S ₂ data. It may also be interpolated using three or more signal S ₂ of the data surrounding the positions of calculating the signal S _3.

Hereinafter, in the present embodiment, the position of the signal S ₃ is set to exactly the middle of each data of the signal S ₂ (α = 0.5 in equation (2)). That is, the signal S ₃ corresponds to an image signal obtained from the pixel (virtual pixels) from the pixel 13 to generate a signal S ₂ to the position shifted by 0.5 times the pixel spacing in the pupil division direction.

[Correlation calculation process]
The correlation calculation step (S11), the correlation calculation unit 42, (a _{S 1} and _{S 2, S 1} and _{S 3)} 2 sets of signals from, to calculate the data sequence C12, C13 of the correlation amount. The data strings C12 and C13 are data arrays composed of correlation amounts indicating the degree of similarity of a pair of signals at each displacement amount when the relative position (displacement amount) between the signals of each group is changed ( FIG. 5).

Data column C12 is determined by calculating a correlation amount by changing the relative position in the x direction of the signals S ₁ and S _2. Data column C13 is determined by calculating a correlation amount by changing the relative position in the x direction of the signals S ₁ and S _3. The amount of displacement is a multiple of the pixel interval. The direction of displacement is a direction connecting the pupil centroids 27 and 28 corresponding to the photoelectric conversion units 11 and 12, and is the x direction in this embodiment.

  As the correlation amount, for example, SAD (Sum of Absolute Difference) or SSD (Sum of Square Difference) can be used. When SAD or SSD is used as the correlation amount, the smaller the correlation amount, the higher the correlation.

Specifically, the data string C12 can be obtained by Expression (3) or Expression (4). Here, d is a displacement amount (integer) normalized by the data interval, subscripts i and j are data numbers in the x and y directions, respectively, and the range of integration can be arbitrarily selected. The calculation method of the correlation amount is not limited to these methods, and other methods may be used.

The pixel position to acquire the signal S _3, since offset by α multiple of the pixel spacing in the x-direction from the position of the pixel to obtain the acquisition of the signal S1, the data sequence C13 is less in consideration of Equation (2) (3 ′) and (4 ′).

[Deviation calculation process]
In the deviation amount calculating step (S12), the deviation amount calculation unit 43 calculates the shift amount of Na signals _{S 1} and _{S 2} with a data sequence C12 and C13.

The deviation amount calculation unit 43 first synthesizes the data strings C12 and C13 to create a data string C123. The data string C123 is defined as follows with respect to the displacement amounts of the integers n and n + α.
C123 (n) = C12 (n)
C123 (n + α) = C13 (n + α)

FIG. 5 is a diagram illustrating an example of the data string C123. The horizontal axis of the figure represents the displacement amount normalized by the data interval P. The vertical axis represents the correlation amount. In this embodiment, the smaller the correlation amount, the higher the correlation. The black circle in the figure represents the data string C12, and the white circle represents the data string C13. A data string C12 and C13, only the signal _{S 2} and _S are different partial position on the image sensor 10 of each data ₃ are displaced from each other. The data string C123 is a data string in which the data of the data strings C12 and C13 are alternately arranged according to the displacement amount calculated for each data.

  Next, the deviation amount calculation unit 43 calculates the deviation amount using the combined data string C123. Of the data string C123, the correlation amount representing the highest correlation is referred to as an extreme value of the data string C123. The deviation amount calculation unit 43 calculates the deviation amount using a plurality of data including the extreme values of the data string C123. The data used for the deviation amount calculation is preferably data that takes an extreme value and data in the vicinity of the displacement that gives the extreme value among the data included in the data string C123.

Here, the data in the vicinity of the displacement amount that gives the extreme value is data on the displacement amount i that satisfies | i−i ₀ | ≦ th, where i ₀ is the displacement amount that gives the extreme value of the data string C123. (C123 (i)). Here, th is a threshold value for determining the neighborhood range. For example, when th = 0.5 (more generally, max (α, 1−α)), two data adjacent to the data having extreme values become neighboring data. Note that data in the vicinity of the displacement amount that gives the extreme value is also simply referred to as data in the vicinity of the extreme value below.

A known method can be used to calculate the amount of deviation. For example, as described in US Pat. No. 7,599,512, first, a predetermined number of correlation amounts in the vicinity of extreme values (for example, C12 (1), C13 (0.5), C13 (1. 3) of 5) is approximated by a polynomial. The calculated displacement amount giving an extreme value of the polynomial (actual quantity) as a shift amount of Na signals S ₁ and the signal S _2. A function other than a polynomial may be employed as the approximate function.

Note that the deviation amount Na can be calculated by applying the following equation (5) without specifically performing polynomial approximation or extreme value calculation. In Expression (5), C (0) corresponds to the extreme value of the data string C123, C (−1) and C (+1) correspond to the correlation amount adjacent to the extreme value of C (0), and N0 corresponds to C (0). Represents the amount of displacement to be performed.

  Using equation (5), the displacement amount giving the extreme value of the approximate function representing the relationship between the displacement amount and the correlation amount can be obtained as the deviation amount Na.

Another example of the method of calculating the deviation amount Na is a method of calculating from the slope of a straight line connecting an extreme value and data adjacent to the extreme value, as described in US Pat. No. 8723926. For example, the amount of deviation can be calculated by applying Equation (6). C1 represents a correlation value having a higher correlation amount between C (−1) and C (+1).

  The calculation method using Expression (6) is also regarded as obtaining the displacement amount giving the extreme value of the approximate function representing the relationship between the displacement amount and the correlation amount as the deviation amount Na.

  The deviation amount Na may be calculated using another method. The amount of deviation may be calculated by an appropriate method according to the method used for calculating the correlation amount.

  In the above description, the data string C123 is generated. However, it is necessary to actually generate the data string C123 (that is, prepare an array different from the data strings C12 and C13 in the storage device and newly store the data). Absent. The processing described above can be realized by appropriately acquiring and using data from the data strings C12 and C13. For example, the deviation amount Na can be obtained as follows. The extreme values of the data strings C12 and C13 are obtained and the extreme values are compared to obtain the correlation amount corresponding to the extreme value of the data string C123. Then, the relationship between the displacement amount near the extreme value and the correlation is extracted from the data strings C12 and C13, and the deviation amount is calculated by the above-described method.

ここで、ＮａはステップＳ１２で算出したズレ量であり、Ｋａはズレ量を距離に変換するための変換係数である。 [Distance calculation process]
In the distance calculation step (S13), the distance calculation unit (distance acquisition unit) 44 calculates distance information based on the deviation amount Na calculated in step S12. More specifically, the distance calculation unit 44 obtains the defocus amount from the deviation amount Na, and calculates the distance of the subject from the defocus amount and the imaging relationship of the imaging optical system 20. For example, the defocus amount ΔL can be calculated by the following equation (7).

Here, Na is the shift amount calculated in step S12, and Ka is a conversion coefficient for converting the shift amount into a distance.

  Since the defocus amount can be easily changed to the distance to the subject in real space using the imaging relationship of the imaging optical system, the defocus amount can be regarded as distance information to the subject. The distance calculation unit 44 may output a defocus amount as distance information to the subject, or may output a distance in real space. The distance information output by the distance calculation unit 44 may be a distance (relative distance) from the focus position to the subject, or a distance (absolute distance) from the imaging device to the subject at the time of shooting. The absolute distance or relative distance may be either the distance on the image plane side or the distance on the object side. The distance may be expressed as a real space distance, or may be expressed as an amount that can be converted into a real space distance such as a defocus amount or a deviation amount.

  By such a distance detection method, the distance of the subject can be calculated with high accuracy with a small calculation load.

<Principle>
The principle by which the distance of the subject can be calculated with high accuracy with a small calculation load by this method will be described.

In this method, the shift amount of the pair of image signals (S ₁ and S ₂ or S ₁ and S ₃ ) is calculated as follows. First, the relative position (displacement amount) of a pair of image signals is changed, and the correlation amount between both image signals is calculated at each position, whereby a correlation amount data string is calculated. A predetermined function representing the relationship between the displacement amount and the correlation amount is obtained from the data in the vicinity of the extreme value of the correlation amount data string, and the displacement amount that gives the extreme value of this approximate function is estimated as the deviation amount Na. As a result, the amount of deviation is calculated with an accuracy finer than the data interval of the image signal. In the following description, a function used for approximating the correlation amount data string is referred to as an approximate function. Further, the deviation amount of the image signal and the displacement amount in the correlation calculation will be described as an amount normalized by the data interval P of the image signal.

On the other hand, in Non-Patent Document 1, which is a conventional method, the amount of deviation is calculated as follows. First, similarly to the method of the present embodiment, two image signals (corresponding to signals S ₂ and S ₃ ) corresponding to signals acquired at different positions on the image sensor are one image signal of a stereo image. Generated. Then, Non-Patent Document 1, unlike the present method, with each of the image signals (corresponding to S ₂ and S _3), the other stereo image signal (corresponding to S ₁₎ from the amount of correlation for each amount of displacement Data strings C12 and C13 are calculated. Then, the shift amounts Na ₁₂ and Na ₁₃ are estimated independently from the data strings C 12 and C ₁₃ , and the final shift amount Na (= 0.5 × (Na ₁₂ + Na ₁₃ ) is obtained by averaging these two shift amounts. ) Is calculated.

6A, 6B, and 6C show the correlation amount data string C12 and the estimated deviation amount Na calculated when the correct deviation amounts Nc of the pair of image signals are N, N + 0.25, and N + 0.5, respectively. It is the schematic diagram which showed the example of ₁₂ . N is an arbitrary integer. The horizontal axis represents the amount of displacement. The vertical axis represents the correlation amount, and the lower the value, the higher the correlation.

  The black dots in the figure represent the data in the correlation amount data string C12. A solid line L1 represents an approximate function of the data string C12, and a broken line L2 represents a correct function. The correct function is a function obtained using a function having the same form as the approximate function under the condition that the vertex of the function is the correct deviation amount Nc.

Vertical line represents the shift amount Na ₁₂ which is estimated to deviation amount Nc of correct answers respectively. Usually, since the function connecting the data string C12 and the approximate function are functions having different characteristics, the approximate function includes an error. This error increases as the data string C12 does not include data in the vicinity of the correct deviation amount Nc. The error of the approximation function increases as the correct answer amount Nc moves away from the integer multiple N of the data interval P, and the difference between the correct answer amount Nc and the estimated error amount Na ₁₂ increases as shown in FIGS. .

FIGS. 6D, 6 </ b> E, and 6 </ b> F are schematic diagrams illustrating an example of the data string C <b> 13 and the estimated shift amount Na ₁₃ . White dots in the figure represent the data of C13. 6 (d) is N as in FIG. 6 (a), FIG. 6 (e) is N + 0.25 as in FIG. 6 (b), and FIG. 6 (f) is FIG. It is the figure which showed the data row | line | column of the correlation amount calculated when the amount of correct deviation | shift is N + 0.5 similarly to (c).

  As shown in FIGS. 6B and 6E, when the correct deviation amount Nc is N + 0.25, the two estimated misalignment amounts Na include errors in the opposite directions and the same degree with respect to the correct answer amount Nc. When averaged, the calculation error of the estimated deviation amount Na is reduced.

However, as shown in FIGS. 6A, 6D and 6C, when the correct deviation amount Nc is N or N + 0.5, one of the data strings C12 and C13 is correct. Since the data in the vicinity of the quantity Nc is not included, the error of the approximation function increases. Since the error of one of the two deviation amounts Na ₁₂ or Na ₁₃ increases, even if the other data string includes data in the vicinity of the correct deviation amount Nc and there is little error, the deviation amount obtained by averaging these two deviation amounts The calculation error of Na becomes large. In addition, since the shift amounts Na ₁₂ and Na ₁₃ are calculated for the data strings C12 and C13, the calculation load increases.

  The distance detection device 40 according to the present embodiment creates a data sequence C123 by combining the data sequences C12 and C13, calculates an approximate function for the extreme values of the data sequence C123 and surrounding data, and sets the approximate function as the approximate function. Based on this, the deviation amount Na is calculated. 7A, 7B, and 7C are schematic diagrams illustrating examples of the data string C123 and the estimated deviation amount Na calculated when the correct deviation amount Nc is N, N + 0.25, and N + 0.5, respectively. is there.

When the correct answer amount Nc is N + 0.25, the data strings C12 and C13 are data strings that are inverted with respect to the correct answer quantity Nc. When an approximate function for data around extreme values in the data string C123 is calculated, the approximate function becomes a function having characteristics between the data strings C12 and C13. Therefore, the estimated deviation amount Na calculated from the approximate function is a value near the correct deviation amount Nc (FIG. 7B).

  When the correct answer amount Nc is N or N + 0.5, the data string C123 includes data in the vicinity of the correct answer value Nc, and the data string C123 takes an extreme value in the vicinity of the correct answer value Nc. By calculating an approximate function for the data string C123 including this displacement, errors in the approximate function can be reduced. By using this approximate function, a deviation amount calculation error can be reduced (FIGS. 7A and 7C).

Thus, by calculating the estimated amount of deviation Na using the data string C123 that combines data sequence C12, C13, even if the image signals S ₁ and S ₂ has what amount of deviation The amount of deviation can be calculated with higher accuracy than before.

The signal S ₃ is preferably in the pupil division direction of the signal S ₂ is the signal at the intermediate position of each data. That is, it is desirable that α = 0.5 in the formula (2). By doing so, it is possible to reduce the difference between the displacement amount giving the correct deviation amount Nc and the displacement amount of the data closest to the displacement amount in the data string C123. Specifically, this difference can always be made smaller than 0.25 by setting α = 0.5. That is, regardless of the correct deviation amount of the image signal, the data string C123 can easily obtain data on the correlation amount with the displacement amount in the vicinity of the correct answer amount Nc. Therefore, the error of the approximate function can be reduced, and the above-described effect can be most obtained.

  Further, by calculating the shift amount dc using the data sequence C123 obtained by synthesizing the correlation value data sequences C12 and C13, it is possible to reduce the number of calculation of the shift amount compared to the method of Non-Patent Document 1, and to calculate the calculation load. Can be reduced.

  The calculation accuracy of the shift amount can be evaluated by calculating the shift amount from various image signals having the same correct shift amount and obtaining their standard deviation. FIG. 8 is a diagram illustrating a result of calculating a deviation amount for various image signals and evaluating a standard deviation of the calculated deviation amount. In the figure, the horizontal axis represents the set deviation amount, and the vertical axis represents the standard deviation. A broken line L3 is a line representing a result when the method of Non-Patent Document 1 is used, and a solid line L4 is a line representing a result when the method of the present embodiment is used. As shown in FIG. 8, it can be seen that this method reduces the standard deviation and improves the accuracy of calculating the deviation amount.

  For the reasons described above, by using the method of this embodiment, it is possible to calculate the amount of deviation with high accuracy and with a small calculation load.

<Other configurations>
In the present embodiment, as shown in FIG. 9A, the imaging element 10 includes the pixel 51 dedicated to imaging, in which the ranging pixels 13 are arranged only in a part and the other photoelectric converter 50 has a single photoelectric conversion unit 50. An arranged configuration may be used. Further, the imaging element 10 may have a configuration in which pixels including only the photoelectric conversion unit 11 of the pixel 13 and pixels including only the photoelectric conversion unit 12 are arranged as illustrated in FIG. Note that the pixels 13 may be arranged at different intervals in the X direction and the Y direction. Further, as shown in FIG. 9C, the image sensor 10 has the ranging pixels 13 arranged in all the pixels, and the distance detection device 40 detects the distance of the signals acquired by some of the pixels 52 (hatched pixels). You may use for. In any configuration, the signals S ₁ and S ₂ are obtained from the photoelectric conversion units 11 and 12, and the amount of deviation is calculated by the same method as described above, so that the calculation load is small and highly accurate distance detection is possible. Become. By using signals acquired from some pixels, the amount of data used to calculate the amount of deviation can be reduced, and the calculation load can be further reduced.

In the signal generation step S10, the signal generation unit 41, by reading the electrical signals generated by the photoelectric conversion portion 12 of the pixel 13 in a position different from the pixel 13 which has acquired the signal S _2, and generates a signal S ₃ May be. For example, in FIG. 9D, S ₁ and S ₂ are generated from signals acquired by the photoelectric conversion units 11 and 12 of some of the pixels 52 of the pixels 13 and acquired by the photoelectric conversion units 12 of the other pixels 53. S ₃ can be generated from the processed signal. From the signal S ₂ by calculation (interpolation) in comparison with the case of generating the S _3, it is possible to obtain a more accurate signal can detect the distance with higher accuracy.

Note that the image sensor 10 according to the present embodiment is a pixel (R pixel, G pixel, B pixel) having high sensitivity to light having respective wavelengths of red (R), green (G), and blue (B). May be arranged. In this case, the ranging pixels 13 are arranged in each of RGB pixels or a part of the pixels. For example, the image signals S ₁ and S ₂ can be generated by using signals acquired from the pixels 13 arranged in any one of R, G, and B pixels. Alternatively, the signals S ₁ and S ₂ can be generated by adding or converting the signals acquired by the RGB pixels close to each other into luminance information. Regardless of which signal is used, by using the same method as described above, the calculation load is small, the amount of deviation can be calculated with high accuracy, and the distance can be detected with higher accuracy.

<Modification 1 of the distance detection method>
The deviation amount is calculated based on the displacement amount giving the extreme value in the data strings C12 and C13 and data in the vicinity thereof. Therefore, it is not necessary to generate the data string C13 so as to correspond to the entire range of the data string C12. Here, a method of performing distance detection by generating the data string C13 only within a partial range will be described.

In this modification, in the correlation calculation step (S11), the correlation calculation unit 42 calculates the data strings C12 and C13 as follows. First, to calculate the data sequence C12 by calculating a correlation amount by changing the relative position in the x direction of the signals S ₁ and S _2. Next, the correlation calculation unit 42 obtains a displacement amount that gives an extreme value of the data string C12, and determines a nearby range including the displacement amount as a calculation range of the data string C13. Then, the correlation calculation unit 42 calculates the data string C13 within the determined range.

  In the deviation amount calculation step (S12), the deviation amount is calculated using data in the vicinity of the extreme value in the data string C123. For this reason, the data string C13 requires data for the displacement amount near the extreme value. The displacement amount that gives the extreme value of the data string C123 is either the displacement amount that gives the extreme value of the data string C12 or the displacement amount that gives the extreme value of the data string C13, which are almost equal. Therefore, the vicinity range of the extreme value of the data string C12 is substantially equal to the vicinity range of the extreme value of the data string C123. By grasping in advance the amount of displacement with which high correlation is obtained using the data string C12, determining the calculation range of the data string C13 based on this information, and calculating the data string C13, reducing unnecessary correlation calculations, The calculation load can be further reduced.

<Modification 2 of the distance detection method>
Depending correct shift amount (correlation signals S ₁ and S ₂₎ data is also a column C13 is not necessary to consider the data sequence C12 the deviation or distance from only accurately be detected. In this modification, first, the data string C12 is calculated, and based on the result, it is determined whether to perform the remaining steps.

  FIG. 10 shows a flowchart of a distance detection method according to this modification. Hereinafter, a distance detection method according to this modification will be described with reference to FIG. In addition, since the process to which the same code | symbol as FIG. 3 was attached | subjected is the same as that of the said embodiment, description is abbreviate | omitted.

In the first correlation calculation step (step S17), the correlation calculation unit 42 calculates the data string C12 from the signals S ₁ and S ₂ . The specific calculation method is as described above.

Next, in the evaluation step (step S18), the correlation calculation unit 42 evaluates the data string C12 and evaluates whether or not the correct answer amount Nc is close to an integer multiple of the data interval. This evaluation can be performed, for example, by comparing the extreme values of the data string C12 and the size of the data (correlation amount) on both sides thereof. When the correct deviation amount Nc is close to an integral multiple of the data interval, the extreme value of the data string C12 becomes a small value, and the correlation amount on both sides becomes a large value. For example, when the difference between the extreme value and the adjacent data is smaller than the threshold value, it can be determined that the correct deviation amount Nc is a non-integer multiple. Alternatively, when the value obtained by dividing the data adjacent to the extreme value by the extreme value is smaller than the threshold value, it is possible to determine that the correct deviation amount Nc is a non-integer multiple.

  When it is determined in the evaluation step S18 that the correct deviation amount Nc is close to an integral multiple of the data interval, that is, when the difference or ratio between the extreme values in the data string C12 and the values of the adjacent data is equal to or greater than the threshold value, Proceed to the deviation amount calculation step (step S19). The second deviation amount calculation step S19 is a step of calculating the deviation amount Na from the data string C12, and calculates the deviation amount by a known method. Specifically, using the data in the vicinity of the extreme value of the data string C12, the relationship between the displacement amount and the correlation is approximated by an approximate function, and the displacement amount that gives the extreme value of the approximate function may be obtained as the deviation amount Na.

If correct deviation amount Nc in the evaluation step S18 is be determined that the non-integer multiple of the data interval, the flow advances to the signal generating step (step S10), and the signal generator 41 generates a signal S _3. In the second correlation calculation step (step S20), the correlation calculation unit 42 calculates the data string C13, and in the deviation amount calculation step (step S12), the deviation amount calculation unit 43 adds the data strings C12 and C13. Based on this, the amount of deviation is calculated. The distance calculation step S13 is the same as described above.

When the correct deviation amount Nc is an integral multiple of the data interval, the deviation amount can be calculated with high accuracy only from the data string C12 as shown in FIG. Therefore, by providing the evaluation step S18, when the correct deviation amount Nc is close to an integral multiple of the data interval, the deviation amount can be calculated with high accuracy using only the data string C12. Further, generation of signal S _3, it is possible to omit the generation of the data sequence C13, it is possible to further reduce the calculation load. On the other hand, when the correct answer amount Nc is other than an integral multiple of the data interval, the difference amount can be calculated with high accuracy by the method of the present invention.

<Modification 3 of the distance detection method>
In the above modification 2, it is determined whether or not the shift amount obtained from the data string C12 in the evaluation step (step S18) is close to an integral multiple of the data interval, and whether or not the data string C13 is generated is switched. . In this modification, the necessity of generating the data string C13 is determined by an evaluation method different from that in Modification 2. Since the process of this modification is substantially the same as that of the modification 2 (FIG. 10), different parts will be mainly described below.

  In the present modification, the magnitude of the correct deviation amount Nc is evaluated using the data string C12 in the evaluation step (step S18) of FIG. This evaluation can be performed by evaluating the amount of displacement that gives the extreme value of the data string C12. When this displacement amount is larger than the threshold value, the correct deviation amount Nc is evaluated as being large, and when it is smaller than the threshold value, the correct deviation amount Nc is evaluated as being small.

  If it is determined in the evaluation step S18 that the correct deviation amount Nc is large, the process proceeds to the second deviation amount calculation step (step S19), and the deviation amount is calculated based on the data string C12. When it is determined in the evaluation step S18 that the correct deviation amount Nc is small, the process proceeds to the signal generation step (step S10), the second correlation calculation step (step S20), and the deviation amount calculation step (step S12), and the data string C13 Is also used to calculate the amount of deviation.

In the above description (paragraph 0056), since the function connecting the data string C12 and the approximate function are functions having different characteristics, the approximate function includes an error, and this error causes a deviation amount calculation error. Said. The error of the approximation function increases as the spatial frequency increases in the image. When the correct answer amount is large, the defocus amount is large and the possibility that a high-frequency signal is included in the image is low. On the other hand, when the correct answer amount is small, the defocus amount is small, and there is a high possibility that a high-frequency signal is included in the image.

Accordingly, the evaluation step S18 is provided to evaluate the size of the correct answer amount Nc, and when the correct answer amount is large, the error amount can be calculated with high accuracy only from the data string C12. In addition, when the correct answer amount Nc is small, the amount of difference can be calculated with high accuracy using the data strings C12 and C13 as described above. Further, when a large correct shift amount, generation of signal S _3, it is possible to omit the generation of the data sequence C13, it is possible to further reduce the calculation load.

<Modification 4 of the distance detection method>
In the distance calculation step S13, for example, the subject distance L may be directly calculated using a conversion coefficient Kb that links the deviation amount Nc and the subject distance L as shown in Equation (8).

Alternatively, the defocus amount ΔL may be calculated according to Equation (9), and the subject distance may be calculated from the defocus amount ΔL. Here, Kc represents a conversion coefficient, and H represents a distance between the exit pupil 21 and the image sensor 10. By using such an expression, the defocus amount and distance can be calculated with higher accuracy.

<Range finding result>
The distance measurement result of the distance detection apparatus of the present invention can be used, for example, for focus detection of the imaging optical system. With the distance detection device of the present invention, the distance of the subject can be measured at high speed and with high accuracy, and the amount of deviation between the subject and the focal position of the imaging optical system can be known. By controlling the focal position of the imaging optical system, the focal position can be adjusted to the subject at high speed and with high accuracy. An imaging apparatus such as a digital still camera or a digital video camera can be configured with the distance detection apparatus of this embodiment, and the focus detection of the optical system can be performed based on the distance detection result of the distance detection apparatus. In addition, the distance detection device of the present invention can generate a distance map by calculating distances at a plurality of positions on the image sensor 10.

<Other examples>
Note that the image acquisition means for acquiring the stereo image signals S ₁ and S ₂ is not limited to the configuration of the above-described embodiment. For example, the image signals S ₁ and S ₂ may be acquired by shooting from different shooting positions with a plurality of cameras or a single camera, the amount of deviation between these image signals may be calculated, and the distance may be detected. For example, it is possible to employ a configuration in which two imaging optical systems and two imaging elements are provided, and stereos having different viewpoints acquire image signals from the respective imaging elements. Even with such a configuration, a stereo image signal can be obtained, and the amount of deviation can be calculated by the same method as described above, so that the calculation load is small and the amount of deviation can be calculated with high accuracy, and high-precision distance detection is possible. Is possible.

In the above-described embodiment, an example in which the distance to the subject is calculated has been described. However, the present invention can also be provided for a parallax amount detection device that detects a parallax amount (an angle difference in the subject direction) corresponding to a deviation amount. For example, the parallax amount detection device can perform processing such as cutting out a subject near the in-focus position from an image based on the amount of deviation. Note that the amount of parallax may be the amount of deviation between two signals, or a physical amount related to them.

  If this parallax amount detection device is configured to have a parallax amount calculation unit that calculates the parallax amount corresponding to the shift amount of two signals, instead of the distance calculation unit 44 of the distance detection device 40 of the first embodiment, The configuration may be the same as that of the distance detection device 40. Furthermore, the parallax amount detection device may include an extraction unit that extracts a subject having a predetermined parallax amount from the image according to the parallax amount (deviation amount).

  In the parallax amount detection method in the parallax amount detection apparatus, if the parallax amount calculation step is performed instead of the distance calculation step S13 in the flowchart of FIG. 3, other processing steps may be the same as those in FIG. Note that the amount of parallax may be calculated by calculating a signal shift amount or by calculating a physical amount related thereto.

  This parallax amount detection device can also be used as a part of the imaging device, like the distance detection device of the first embodiment.

  The present invention includes a computer program in addition to the distance detection device and the parallax amount detection device. The computer program of the present embodiment causes a computer to execute a predetermined process for calculating a distance or a parallax amount.

  The program of the present embodiment is installed in a computer of an imaging apparatus such as a digital camera equipped with a distance detection apparatus, a parallax amount detection apparatus, or any one thereof. When the installed program is executed by a computer, the above-described function is realized, and high-speed and high-precision distance detection and parallax amount detection can be performed.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

40 Distance detection device 42 Correlation calculation unit 43 Deviation amount calculation unit

Claims (15)

A shift amount acquisition device for acquiring a shift amount of a relative position between a first image and a second image having different viewpoints acquired by an imaging device including a plurality of pixels,
Correlation calculation means for acquiring a data string of correlation amounts for the respective displacement amounts when the two image signals are relatively displaced;
A shift amount acquisition means for acquiring the shift amount between the first image and the second image from a data string of correlation amounts;
With
The correlation calculation means includes
An image signal based on the first image, the first signal including an image signal of a plurality of pixels having a predetermined pixel interval, and an image signal based on the second image, wherein the predetermined pixel interval A first data sequence that is a data sequence of the correlation amount between the second signal including image signals of a plurality of pixels
An image signal based on the second image when an amount of displacement giving an extreme value of the first data string is smaller than a threshold value, the image signal including a plurality of pixels at the predetermined pixel interval; A second data string that is a data string of the correlation amount between a third signal corresponding to a signal acquired in a pixel at a position different from the signal of 2 and the first signal;
The deviation amount acquisition means is
When the amount of displacement that gives the extreme value of the first data sequence is smaller than the threshold value, the relationship between the amount of displacement and the correlation amount obtained from the data included in the first data sequence and the second data sequence is expressed. A displacement amount that gives an extreme value of the approximate function is acquired as the deviation amount.
A shift amount acquisition device characterized by that. The deviation amount acquisition means is
When the displacement amount is larger than the threshold value, the shift amount is acquired from the first data string without using the second data string;
The deviation amount acquisition device according to claim 1, wherein The deviation amount acquisition means uses data in which the correlation amount takes an extreme value and data in the vicinity of the displacement amount that gives the extreme value among the data included in the first data row and the second data row. To obtain the amount of deviation,
The deviation amount acquisition device according to claim 1 or 2. The correlation calculation means obtains the second data string only for a displacement amount giving an extreme value of the first data string and a displacement amount in the vicinity thereof.
The deviation amount acquisition device according to claim 3. The third signal is an image signal corresponding to a signal acquired at a position different from the position of the pixel from which the second signal is acquired by 0.5 times the predetermined pixel interval.
The deviation amount acquisition device according to any one of claims 1 to 4. Signal generation means for generating the third signal by calculation from the second signal;
The deviation amount acquisition device according to any one of claims 1 to 5. The second signal includes an image signal read from some pixels of the image sensor,
The third signal includes an image signal read from a pixel different from the part of the pixels of the image sensor.
The deviation amount acquisition device according to any one of claims 1 to 5. When the difference between the extreme value of the first data string and the correlation amount corresponding to the displacement amount in the vicinity of the extreme value of the first data string is smaller than a threshold value, the correlation calculation means is configured to use the second data string. The deviation amount acquisition means acquires the deviation amount based on the first data string and the second data string,
When the difference between the extreme value of the first data string and the correlation amount corresponding to the displacement amount in the vicinity of the extreme value of the first data string is equal to or larger than the threshold value, the deviation amount acquisition means is The amount of deviation is acquired from the first data string without using the data string of
The deviation amount acquisition device according to any one of claims 1 to 7. When the value obtained by dividing the extreme value of the first data string by the correlation amount corresponding to the amount of displacement in the vicinity of the extreme value of the first data string is smaller than the threshold value, the correlation calculating means is configured to output the second data A shift amount acquisition unit acquires the shift amount based on the first data string and the second data string;
When the value obtained by dividing the extreme value of the first data string by the amount of correlation corresponding to the amount of displacement in the vicinity of the extreme value of the first data string is equal to or greater than the threshold value, the deviation amount acquiring means The amount of deviation is acquired from the first data string without using the data string of 2.
The deviation amount acquisition device according to any one of claims 1 to 7. A distance acquisition unit that acquires distance information of the subject from the amount of deviation;
The deviation amount acquisition apparatus according to any one of claims 1 to 9. Image acquisition means for acquiring the first image and the second image;
The deviation amount acquisition device according to any one of claims 1 to 10,
An imaging apparatus comprising: The image acquisition means includes a first image signal corresponding to a light beam that has passed through the first pupil region of the exit pupil of the imaging optical system, and a second pupil different from the first pupil region of the exit pupil. Having an image sensor composed of pixels that acquire a second image signal corresponding to the light flux that has passed through the region;
The imaging device according to claim 11. The image acquisition unit acquires a first imaging unit including a first imaging optical system and a first imaging element for acquiring the first image signal, and acquires the second image signal. A second imaging optical system and a second imaging means including a second imaging element;
The imaging device according to claim 11. A shift amount acquisition method executed by a shift amount acquisition apparatus, which acquires a shift amount of a relative position between a first image and a second image having different viewpoints acquired by an imaging device including a plurality of pixels. And
An image signal based on the first image, the first signal including an image signal of a plurality of pixels having a predetermined pixel interval, and an image signal based on the second image, wherein the predetermined pixel interval Obtaining a first data sequence that is a data sequence of a correlation amount between the second signal including the image signals of the plurality of pixels, and
An image signal based on the second image when an amount of displacement giving an extreme value of the first data string is smaller than a threshold value, the image signal including a plurality of pixels at the predetermined pixel interval; Obtaining a second data string that is a data string of a correlation amount between a third signal corresponding to a signal acquired in a pixel at a position different from the signal of 2 and the first signal; ,
When the amount of displacement that gives the extreme value of the first data sequence is smaller than the threshold value, the relationship between the amount of displacement and the correlation amount obtained from the data included in the first data sequence and the second data sequence is expressed. Obtaining a displacement amount giving an extreme value of an approximate function as the deviation amount;
A method for obtaining a deviation amount, comprising:   The program for making a computer perform each step of the method of Claim 14.

Images