JP2010081177A - Edge detector, and method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an edge detector which is applied to raw data, high in accuracy and has no limit in the spectral sensitivity and arrangement, of pixels. <P>SOLUTION: The edge detector includes: an imaging data input part 101 for inputting imaging data including the position of an imaging pixel, the spectral sensitivity and a measurement signal amount from each of the plurality of imaging pixels with the different spectral sensitivities that an imaging element has; a local edge detection part 102 for calculating a local edge which is a two-dimensional vector indicating the increase/decrease amount of the measurement signal amount corresponding to the position on the basis of the positions and measurement signal amounts of the respective imaging pixels which are present at the different positions inside a block and have the same spectral sensitivity; and a block edge estimation part 103 for obtaining an edge intensity and an edge direction from an autocorrelation matrix at the local edge. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an edge detection apparatus and method for detecting an edge based on a measurement signal amount from an image sensor.

  The edge detection in the image sensor can be applied to raw data, desirably has high accuracy, and is not limited in the spectral sensitivity and arrangement of pixels. The “raw data” is image data that has not been processed in the digital camera, that is, a measurement signal amount obtained by simply digitizing an electrical signal obtained from the image sensor.

  However, the method of Patent Document 1 can be applied only to the Bayer array. In addition, accuracy is limited because the block size is fixed.

On the other hand, the method of Non-Patent Document 1 can flexibly cope with the number of reference pixels and the arrangement of the pixels, and an improvement in accuracy can be expected by increasing the number of reference pixels. However, it cannot be applied to raw data as it is.
US Pat. No. 5,629,734 A. Cumani, "Edge Detection in Multispectral Images," 1989. H. Takeda et. Al., "Kernel Regression for Image Processing and Reconstruction," 2007.

  As described above, the method of Patent Document 1 can be applied only to the Bayer array and has a problem that accuracy is limited.

  Further, the method of Non-Patent Document 1 can flexibly cope with pixel arrangement change and can be expected to improve accuracy by increasing the number of pixels, but has a problem that it cannot be applied to raw data.

  Therefore, the present invention has been made to solve the above-described problems of the prior art, and is an edge detection apparatus and method that can be applied to raw data, has high accuracy, and has no restrictions on the spectral sensitivity and arrangement of pixels. The purpose is to provide.

  In order to achieve the above object, the edge detection apparatus of the present invention includes imaging data including the position of the imaging pixel, spectral sensitivity, and measurement signal amount from each of a plurality of imaging pixels having different spectral sensitivities of the imaging device. Based on the position of each imaging pixel that is present at different positions and has the same spectral sensitivity and the measurement signal amount, in an arbitrary block of the imaging device, the input imaging data input unit A local edge detector that calculates a local edge that is a two-dimensional vector representing the amount of increase or decrease in the amount of the measurement signal corresponding to, and obtains the edge strength in the block from the eigenvalue in the autocorrelation matrix of the local edge, A block edge estimator that obtains an edge direction from an eigenvector in the matrix.

  According to the present invention, an edge can be detected with high accuracy from raw data of an image sensor having an arbitrary pixel arrangement and spectral sensitivity.

  Hereinafter, an edge detection apparatus according to an embodiment of the present invention will be described with reference to FIGS.

(1) Configuration of Edge Detection Device FIG. 1 is a block diagram showing an edge detection device according to this embodiment.

  The edge detection device includes an imaging data input unit 101 that acquires each position of a plurality of photodiodes included in the imaging element, spectral sensitivity thereof, and a measurement signal amount obtained from the photodiodes, and light of the light in the block from the imaging data. A local edge detection unit 102 that detects an intensity change amount and a block edge estimation unit 103 that integrates the local edges obtained from the local edge detection unit 102 to estimate edge strength and edge direction in the block.

  Note that this edge detection device can also be realized by using, for example, a general-purpose computer device as basic hardware. In other words, the imaging data input unit 101, the local edge detection unit 102, and the block edge estimation unit 103 can be realized by causing a processor mounted on the computer device to execute a program. At this time, the edge detection apparatus according to the present embodiment may be realized by installing the above-described program in a computer device in advance, or may be stored in a storage medium such as a CD-ROM or via a network. You may implement | achieve by distributing a program and installing this program in a computer apparatus suitably.

  Next, the operation of the edge detection apparatus according to this embodiment will be described. FIG. 2 is a flowchart showing the operation of the edge detection apparatus.

(2) Imaging data input unit 101
First, the imaging data input unit 101 acquires, as imaging data, the position of the photodiode included in the imaging element, the spectral sensitivity, and the measurement signal amount obtained from the photodiode. In the present embodiment, each of the photodiodes constitutes an imaging pixel. In addition, examples of the measurement signal amount obtained from the photodiode include raw data of the photodiode. Hereinafter, as illustrated in FIG. 3, an example in which imaging data of a 5 × 5 block including the photodiodes 300 to 324 is acquired from the Bayer array will be described.

Define a coordinate system with the center of the block as the origin. That is, (x ₃₁₂ , y ₃₁₂ ) = (0, 0), and the coordinates of the photodiode 318 are (x ₃₁₈ , y ₃₁₈ ) = (1, 1).

A label c _i (c _i = {R, Gr, Gb, B}) representing the spectral sensitivity of the photodiode and the arrangement of surrounding color filters is defined. Gr and Gb represent the same spectral sensitivity, but the arrangement of surrounding color filters is different. For example, the label of the photodiode 318 is c ₃₁₈ = R, and the label of the photodiode 319 is c ₃₁₉ = Gr.

The measurement signal amount obtained from the photodiode is represented by s. For example, the measurement signal amount obtained from the photodiode 318 is expressed as s ₃₁₈ .

(3) Local edge detection unit 102
Next, the operation of the local edge detection unit 102 will be described with reference to FIG.

  In step 1, two or more photodiodes having the same spectral sensitivity are selected from within the block. For example, assume that 300, 302, and 310 are selected as photodiodes having blue spectral sensitivity.

In step 2, the parameter of the spectral intensity function f (x, y) representing the relationship between the position (x, y) of the photodiode and the measurement signal quantity s which is a real number is estimated. For example, the shape of the spectrum intensity function f (x, y) is represented by the following formula (1).

When using three photodiodes 300, 302, and 310, consider the simultaneous equations shown in the following equation (2).

  Then, by solving the equation (2), the parameters a, b, and c of the spectrum intensity function f (x, y) can be calculated.

  If it demonstrates in FIG. 5, Formula (2) will correspond to calculating | requiring the parameter of the triangle 501 corresponding to the spectrum intensity function f (x, y) in FIG.

  In step 3, the local edge detector 102 spatially differentiates the spectrum intensity function f (x, y), and detects a two-dimensional vector (hereinafter referred to as “local edge”) of the amount of change in light intensity. That is, the local edge represents an increase / decrease amount of the measurement signal amount corresponding to the position of the photodiode.

The local edge obtained when solving the simultaneous equations of Expression (2) is d _j = (dx _j , dy _j ) = (a, b). In FIG. 5, the parameter a represents the inclination of the triangle 501 in the x-axis direction, and the parameter b represents the inclination of the triangle 501 in the y-axis direction.

  The combination and number of photodiodes used for local edge detection must be set in advance by the user. However, the number of photodiodes must be equal to or greater than the number of independent parameters of the spectral intensity function f (x, y).

  Further, a combination in which all photodiodes are arranged in a straight line, such as a combination of photodiodes 300, 312, and 324, cannot be used.

  However, as long as the above-described conditions are satisfied, the photodiodes can be used even if they are not square lattices.

  Hereinafter, an example in which one local edge is detected for each triangle having the smallest area that can be configured by photodiodes having the same color filter in the block of FIG. 3 will be described. In FIG. 3, since colors cannot be represented, red (R) is represented by dots, green (G) is represented by left-up hatching lines, and blue (B) is represented by right-up hatching lines. . The same applies to FIG.

  At this time, 16 local edges can be calculated from the pixel having the B color filter. Similarly, a total of 44 local edges can be calculated, 24 from the G color filter and 4 from the R color filter.

(4) Block edge estimation unit 103
In step 4, after the local edge detection unit 102 detects local edges from all combinations of photodiodes, the block edge estimation unit 103 obtains the edge direction and edge strength in the same procedure as in Non-Patent Document 1. .

First, in step 5, the block edge estimation unit 103 obtains an autocorrelation matrix C _i of local edges shown in the following equation (3).

The sum of equation (3) is calculated as in equation (4) below.

In step 6, the block edge estimation unit 103 estimates the edge strength λ ± and the edge direction θ in the block according to the equation (5). The edge strength λ ± is the first eigenvalue and the second eigenvalue of the autocorrelation matrix C, and the edge direction θ is a rotation angle obtained from the eigenvector of the autocorrelation matrix C.

In addition, a two-dimensional normal distribution N (x, y) represented by Expression (6) defined using the autocorrelation matrix C of Expression (4) is obtained.

  The contour line of the equation (6) becomes an ellipse whose shape changes according to the shape of the edge as shown in FIG. For example, if it is an edge on a line extending from the upper left to the lower right, the shape of the ellipse becomes an ellipse 601 in FIG. 6, and the shape of the ellipse for an edge in the horizontal direction becomes an ellipse 602 in FIG. When the directions of the local edges detected in the block are different, the shape is circular as shown by an ellipse 603 in FIG.

(5) Effect As described above, according to the edge detection apparatus according to the present embodiment, the edge direction and the edge intensity are estimated by integrating the change vectors of the local light amounts calculated from the three photodiodes. It becomes possible.

(Example of change)
Note that this embodiment can also be implemented by changing the following items.

(1) Correction of Number of Local Edges Using Weights In the above embodiment, an example is described in which one local edge is detected for each triangle having the smallest area that can be configured with photodiodes having the same color filter in the block of FIG. did. At this time, a total of 44 local edges can be calculated, 16 from the B color filter, 24 from the G color filter, and 4 from the R color filter.

  However, in the Bayer array demosaicing, a total of four patterns must be considered when the central pixel is B (see FIG. 3), R, Gb (see FIG. 4), and Gr. If the autocorrelation matrix C of Equation (4) is used for all patterns, the output changes according to the color filter pattern even if the same color edge is input.

Therefore, Equation (4) in equation (7) is a weighted sum of the autocorrelation matrix C _i instead the expression of the autocorrelation matrix C (3) is used.

In real w _{(c j)} is determined by the number of local edge detected, for example, RGB respectively _N R from the color _filters, N G, _{if N B} number of local edge is obtained,

w (B) = 1 / N _B ,
w (Gr) = 1 / N _G ,
w (Gb) = 1 / N _G ,
w (R) = 1 / N _R

In this way, a positive real number that decreases as the number of local edges obtained from the block increases.

  By applying a weight that is inversely proportional to the number, the variation in the number of local edges between blocks can be corrected.

(2) Weighting by wavelength In addition, since the Bayer array originally exists at a ratio of R: G: B = 1: 2: 1 and the measurement signal amount of G is larger than other colors, G If the edge is emphasized, the performance may be improved.

Therefore, the target RGB weights are determined in advance as R: G: B = T _R : T _G : T _B ,

w (B) = T _B / N _B ,
w (Gr) = T _G / N _G ,
_{w (Gb) = T G /} N G,
w (R) = T _R / N _R

In this way, it may be redefined so as to be proportional to a predetermined weight.

  In other words, if the number of local edges corresponding to the same spectrum calculated from each block is reduced to the number obtained from the block with the smallest number of color filters corresponding to that spectrum, the multiplication of weights can be eliminated. It is. Similarly, the color weight can be dealt with by calculating more local edges of the color to be emphasized.

  It is also possible to perform edge detection using white balance information. For example, when a subject illuminated with a light source that does not contain much blue (B) light is imaged, the B signal is almost noisy, but by reducing the B weight beforehand, edge detection that is robust to noise is possible. It becomes possible. Whether or not the light is illuminated by a light source that does not include B light can be determined by the magnitude of the coefficient applied to B at the stage of white balance adjustment.

(3) Weighting of local edge using distance from block center When calculating autocorrelation matrix C, weight using distance between photodiode used for calculating local edge and block center is used. It is also possible to do. For example, in the case of FIG. 3, to define the position weight w _l to be multiplied by the autocorrelation matrix C in equation (8).

Then, the autocorrelation matrix C is defined by equation (9).

Then, it becomes possible to detect an edge faithful to the amount of light at the block center. It is assumed that σ _l is a positive real number and is manually adjusted by looking at the image quality of the output image.

The definition of position weight w _l, if positive real number that takes a more smaller value the distance between the photodiode and the block center is increased, need not be of the formula (8).

(4) Exclusion of weight by controlling the number of local edges detected It is also possible to correct the number of local edges without using weights.

  For example, when detecting a local edge using a triangle having the smallest area, the number of local edges obtained from the red color filter is 4 in the block of FIG. 3 and 8 in the block of FIG.

  In the case of correction using weights, when calculating the autocorrelation matrix in the block of FIG. 3, the local edge is multiplied by twice the weight of FIG. 4. However, since multiplication is performed, the circuit scale increases.

  However, when local edge detection is performed in the block of FIG. 4, weight multiplication can be eliminated by using local edges obtained from four combinations including the photodiodes 407 and 417 simultaneously.

(5) Modification Example of Local Edge Calculation Method In the above example, the local edge has the same spectral sensitivity and is calculated from a plurality of three photodiodes that do not exist in the same straight line. It is also possible to use edge detection by filtering for edge calculation.

  Filtering is defined for regularly arranged photodiodes, and the position where the local edge is calculated is uniquely determined when the filter is defined.

  However, if the change in the amount of light in the block is modeled by a differentiable spectral intensity function f (x, y) using the method of Non-Patent Document 2, and the spectral intensity function f (x, y) is differentiated, the block It is possible to obtain a local edge at any position of the photodiode, and any arrangement of the photodiodes can be handled.

  Local edge calculation using equation (2) described in the above example is equivalent to filtering using three photodiodes. This is equivalent to defining the spectral intensity function f (x, y) as a linear function in the method of Non-Patent Document 2.

(6) Weighting of local edge in consideration of frequency characteristics Since the frequency characteristics differ depending on the above-described local edge calculation method, if necessary frequency characteristics are known in advance, weights used for calculating autocorrelation matrix C accordingly Can be changed.

For example, when the local edges dx _j and dy _j are calculated using three photodiodes, a higher frequency can be detected as the area m _{j of the} triangle formed by the three photodiodes becomes smaller. Therefore, an edge having a high frequency can be easily detected by weighting a positive real number that increases as m _j becomes smaller, such as w _m (m _j ) = 1 / m _j .

(7) Local edge weighting considering noise immunity When there are multiple local edge calculation methods with the same frequency characteristics, the more photodiodes used to calculate local edges, the more robust the noise is. . Therefore, when the number of photodiodes used for calculating the local edges dx _j and dy _j is n _j , for example, a positive value that takes a larger value as n _j such as w _n (n _j ) = 1 / n _j increases. It is considered that an edge that is resistant to noise can be detected by using the weight w _n (n _j ) that is a real number.

(8) Others It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

It is a block diagram which shows the structure of the edge detection apparatus concerning embodiment of this invention. It is a flowchart which shows operation | movement of an edge detection apparatus. It is a figure showing the block on an image sensor. It is a figure showing a different thing from FIG. 3 in the block on an image pick-up element. It is a figure which shows the concept of local edge detection. It is a figure which shows the concept of an edge detection result.

Explanation of symbols

101 imaging data input unit 102 local edge detection unit 103 block edge estimation unit

Claims (8)

An imaging data input unit that inputs imaging data including the position of the imaging pixel, spectral sensitivity, and measurement signal amount from each of a plurality of imaging pixels having different spectral sensitivities of the imaging element;
Based on the position of each imaging pixel having the same spectral sensitivity and the amount of the measurement signal in an arbitrary block of the image sensor, the measurement signal amount corresponding to the position A local edge detector that calculates a local edge that is a two-dimensional vector representing the amount of increase or decrease;
A block edge estimator that obtains an edge strength in the block from an eigenvalue in the autocorrelation matrix of the local edge, and obtains an edge direction from an eigenvector in the autocorrelation matrix;
An edge detection device comprising: The local edge detector is
Estimating a parameter of a spectral intensity function representing the relationship between the position of each imaging pixel and the measurement signal amount based on the position and each measurement signal amount,
A slope of a plane determined by the spectrum intensity function, and the local edge representing the increase / decrease amount is calculated from the estimated parameter;
The edge detection apparatus according to claim 1. The block edge estimation unit
Giving a weight that is a positive real number having a larger value to the autocorrelation matrix obtained from the imaging pixel having high spectral sensitivity with respect to light of a predetermined spectrum,
Calculating the edge strength and the edge direction from the autocorrelation matrix given the weight;
The edge detection apparatus according to claim 1. The block edge estimation unit
A weight that is a positive real number having a larger value is assigned to the autocorrelation matrix obtained from the imaging pixel located near the block center,
Calculating the edge strength and the edge direction from the autocorrelation matrix given the weight;
The edge detection apparatus according to claim 1. The block edge estimation unit
A positive real number having a smaller value as the area of a triangle including the imaging pixel used when calculating the local edge is larger, is given as a weight to the autocorrelation function,
Calculating the edge strength and the edge direction from the autocorrelation matrix given the weight;
The edge detection apparatus according to claim 1. The block edge estimation unit
A weight that is a positive real number that increases as the number of the imaging pixels used when calculating the local edge increases is given to the autocorrelation matrix,
Calculating the edge strength and the edge direction from the autocorrelation matrix given the weight;
The edge detection apparatus according to claim 1. An imaging data input step for inputting imaging data including a position of the imaging pixel, spectral sensitivity, and a measurement signal amount from each of a plurality of imaging pixels having different spectral sensitivities of the imaging element;
Based on the position of each imaging pixel having the same spectral sensitivity and the amount of the measurement signal in an arbitrary block of the image sensor, the measurement signal amount corresponding to the position A local edge detection step of calculating a local edge which is a two-dimensional vector representing an increase / decrease amount;
A block edge estimation step for obtaining an edge strength in the block from an eigenvalue in the autocorrelation matrix of the local edge, and obtaining an edge direction from an eigenvector in the autocorrelation matrix;
An edge detection method comprising: An imaging data input function for inputting imaging data including the position of the imaging pixel, spectral sensitivity, and measurement signal amount from each of a plurality of imaging pixels having different spectral sensitivities of the imaging element;
Based on the position of each imaging pixel having the same spectral sensitivity and the amount of the measurement signal in an arbitrary block of the image sensor, the measurement signal amount corresponding to the position A local edge detection function for calculating a local edge which is a two-dimensional vector representing an increase / decrease amount;
A block edge estimation function for obtaining an edge strength in the block from an eigenvalue in the autocorrelation matrix of the local edge, and obtaining an edge direction from an eigenvector in the autocorrelation matrix;
Edge detection program for causing computer to implement

Images