JP5914092B2 - Image processing system and microscope system including the same - Google Patents

Description

  The present invention relates to an image processing system and a microscope system including the image processing system.

  In general, a method for evaluating focusing based on the band contrast of an image is known. In-focus evaluation based on contrast is used for acquiring depth information of a subject, for example, in addition to an autofocus function or the like. The depth information is acquired by, for example, selecting an image focused on each position from the plurality of images after imaging the subject at a plurality of focal positions. In addition, the depth information is obtained by capturing an image of a subject at a plurality of focal positions, selecting an in-focus image for each position of the subject from the plurality of images, and synthesizing the in-focus images. This is used when creating a focus image or a three-dimensional reconstructed image.

  In creating an omnifocal image or a three-dimensional reconstructed image, after selecting the most focused image from a plurality of images having different focal planes for each position in the image and estimating the three-dimensional shape of the subject, Optimization processing is required for the estimated value of the three-dimensional shape. This optimization process may include reducing isolated point estimation errors based on correlation between pixels. It may also include estimating the shape of the sample for the location where the selection could not be made.

  For example, Patent Document 1 discloses a technique related to a microscope system that creates an omnifocal image. Japanese Patent Application Laid-Open No. H10-228707 discloses that processing using a recovery filter is performed after an omnifocal image is created. In general, images have different frequency bands depending on the optical system, magnification, and subject characteristics used to acquire the images. In the technique disclosed in Patent Document 1, the coefficient of the recovery filter is determined in accordance with the setting of the optical system such as the magnification and the numerical aperture of the objective lens in consideration of the band change of the optical system.

JP-A-9-298682

  In the technique according to Patent Document 1, the restoration of the structure and the enhancement of the edge are performed by performing the process using the restoration filter. This is different from the optimization processing for the estimated value of the three-dimensional shape of the sample as described above. Even in the optimization process as described above, an error may be included in the optimization process unless the frequency band of the image is taken into consideration.

  Therefore, an object of the present invention is to provide an image processing system capable of performing an optimization process on an estimated value of a three-dimensional shape of a subject in consideration of an image frequency band, and a microscope system including the image processing system.

To achieve the above object, according to an aspect of the present invention, an image processing system includes: an image acquisition unit that acquires a plurality of images captured at different focal positions for the same subject; and a plurality of images for each pixel of the image. A candidate value estimator for estimating a candidate value of a three-dimensional shape based on the image and a band for calculating a band evaluation value of a band included in the image for each of a plurality of frequency bands A characteristic evaluation unit; an effective frequency determination unit that determines an effective frequency of the pixel based on statistical information of the band evaluation value; and data correction and data interpolation for the candidate value based on the effective frequency A candidate value correction unit that performs at least one and calculates a correction candidate value that represents the three-dimensional shape of the subject, and the candidate value correction unit uses a correlation of values of local regions represented by the candidate value. Said modification Includes a correction candidate value calculation unit that calculates a complement value, the area of the local region, characterized in that it is determined according to the effective frequency.

  Moreover, according to one aspect of the present invention, a microscope system includes a microscope optical system, an imaging unit that acquires an image of a specimen through the microscope optical system as a specimen image, and the specimen image that is acquired as the image. And an image processing system.

  According to the present invention, the effective frequency of the pixel is determined based on the statistical information of the band evaluation value, and at least one of data correction and data interpolation is performed on the three-dimensional shape candidate value according to the effective frequency. Therefore, the optimization process for the estimated value of the three-dimensional shape of the subject can be performed in consideration of the frequency band of the image.

1 is a block diagram illustrating a configuration example of an image processing system according to each embodiment of the present invention. The figure which shows an example of the frequency characteristic of the filter bank which the band process part which concerns on each embodiment has. The figure which shows another example of the frequency characteristic of the filter bank which the band process part which concerns on each embodiment has. 5 is a flowchart illustrating an example of processing in the image processing system according to the first embodiment. 6 is a flowchart illustrating an example of noise / isolated point removal processing according to the first embodiment. It is a figure for demonstrating a coring process, and is a figure showing the example of the original signal corresponded to a shape candidate value. It is a figure for demonstrating a coring process, and is a figure showing the example of a moving average and a threshold value. It is a figure for demonstrating a coring process, and is a figure showing the example of a result of a coring process. 6 is a flowchart illustrating an example of interpolation processing according to the first embodiment. The block diagram which shows the structural example of the microscope system which concerns on 3rd Embodiment.

[First Embodiment]
A first embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows an outline of a configuration example of an image processing system 100 according to the present embodiment. As shown in this figure, the image processing system 100 includes an image acquisition unit 110, a band processing unit 120, a band characteristic evaluation unit 130, an effective frequency determination unit 140, a candidate value estimation unit 150, and a data correction unit 160. And a 3D shape estimation unit 170 and an image composition unit 180. The effective frequency determination unit 140 includes a statistical information calculation unit 142 and a parameter determination unit 144. The candidate value estimation unit 150 includes a contrast evaluation unit 152 and a shape candidate estimation unit 154. The data correction unit 160 includes a data correction unit 162 and a data interpolation unit 164.

  The image acquisition unit 110 includes a storage unit 114. The image acquisition unit 110 acquires a plurality of images captured while changing the focal position with respect to the same subject, and stores the acquired images in the storage unit 114. Here, it is assumed that each of these images includes information related to the focal position of the optical system when the image is acquired, that is, information related to the depth. The image acquisition unit 110 outputs these images in response to requests from the band processing unit 120, the shape candidate estimation unit 154, and the image synthesis unit 180.

  The band processing unit 120 has a filter bank. That is, the band processing unit 120 includes, for example, a first filter 121, a second filter 122, and a third filter 123. The frequency characteristics of the first filter 121, the second filter 122, and the third filter 123 are shown in FIG. As shown in this figure, these filters are low-pass filters, and the cut-off frequency increases in the order of the first filter 121, the second filter 122, and the third filter 123. That is, the frequency band of the signal transmitted by each filter is different. Note that the first filter 121, the second filter 122, and the third filter 123 may be band-pass filters whose frequency characteristics are shown in FIG. In addition, any filter may be used as long as it is a plurality of filters designed to transmit different frequency bands. In the present embodiment, the band processing unit 120 has three filters. However, the number may be any number. The band processing unit 120 acquires an image from the image acquisition unit 110, and for each of a plurality of images having different focal positions, the first filter 121 and the second filter 122 for each region (for example, each pixel) of the image. Filtering processing is performed using each of the third filter 123 and the third filter 123. In the following description, the processing is performed for each pixel. However, the same applies when processing is performed for each region composed of a plurality of pixels. Band processing unit 120 outputs the result of the filtering process to band characteristic evaluation unit 130.

  The band characteristic evaluation unit 130 calculates a band evaluation value for each pixel of the plurality of filtered images. The band evaluation value is obtained, for example, by calculating an integral value of a signal that has passed for each filter. In this way, a band evaluation value is obtained for each pixel and for each frequency band for each image. The band characteristic evaluation unit 130 outputs the calculated band evaluation value to the statistical information calculation unit 142 in the effective frequency determination unit 140 and the contrast evaluation unit 152 in the candidate value estimation unit 150.

  The statistical information calculation unit 142 in the effective frequency determination unit 140 calculates a statistical information value having a relationship with an average value of band evaluation values of a plurality of images having different focal positions for each frequency band. This statistical information will be described later. The statistical information calculation unit 142 outputs the calculated statistical information value to the parameter determination unit 144.

  The parameter determination unit 144 in the effective frequency determination unit 140 calculates the effective frequency based on the statistical information value input from the statistical information calculation unit 142. The parameter determination unit 144 further calculates a correction parameter used by the data correction unit 162 in the data correction unit 160 and an interpolation parameter used by the data interpolation unit 164 in the data correction unit 160 based on the effective frequency. . The calculation of the correction parameter and the interpolation parameter will be described later. The parameter determination unit 144 outputs the calculated correction parameter to the data correction unit 162 in the data correction unit 160 and the interpolation parameter to the data interpolation unit 164 in the data correction unit 160, respectively. Regarding the determination of the frequency, a filter bank as in the present embodiment may be used, or data based on frequency analysis using an orthogonal basis such as wavelet transform may be used.

  The contrast evaluation unit 152 in the candidate value estimation unit 150 evaluates the intensity of the high frequency component for each pixel of the plurality of images based on the band evaluation value input from the band characteristic evaluation unit 130, and calculates a contrast evaluation value. . In calculating the contrast evaluation value, the contrast evaluation unit 152 may use one of the plurality of band evaluation values calculated by the band characteristic evaluation unit 130 or may use a plurality of band evaluation values. . The contrast evaluation unit 152 outputs the calculated contrast evaluation value relating to each pixel of each image to the shape candidate estimation unit 154.

  The shape candidate estimation unit 154 in the candidate value estimation unit 150 evaluates the focus of each pixel in the image with respect to each of the plurality of images based on the contrast evaluation value input from the contrast evaluation unit 152. Further, the shape candidate estimation unit 154 selects the most focused pixel among a plurality of images having different focal positions for each pixel in the image. The shape candidate estimation unit 154 acquires information on the focal position when the most focused image is captured from the image acquisition unit 110, and based on the information, the depth of the subject corresponding to each pixel in the image And a shape candidate value which is information as a candidate value of the three-dimensional shape of the subject is calculated. For pixels for which the depth of the subject cannot be estimated based on the contrast evaluation value, the shape candidate estimation unit 154 sets the shape candidate value corresponding to the pixel as a value indicating that estimation is impossible. The shape candidate estimation unit 154 outputs the calculated shape candidate value to the data correction unit 162 in the data correction unit 160.

  The data correction unit 162 included in the data correction unit 160 performs, for example, noise coring on the shape candidate value input from the shape candidate estimation unit 154 to remove noise from the shape candidate value. The data correction unit 162 uses the correction parameter input from the parameter determination unit 144 when performing noise coring, as will be described in detail later. The data correction unit 162 outputs the noise removal shape candidate value, which is the shape candidate value from which noise has been removed, to the data interpolation unit 164.

  The data interpolation unit 164 included in the data correction unit 160 performs data interpolation on the pixel that is a value indicating that estimation is impossible among the noise removal shape candidate values input from the data correction unit 162. When interpolating data, the data interpolation unit 164 uses the interpolation parameter input from the parameter determination unit 144 as will be described later. The data interpolation unit 164 outputs to the 3D shape estimation unit 170 an interpolation shape candidate value that is a shape candidate value obtained by interpolating a pixel value that cannot be estimated from which noise has been removed.

  The 3D shape estimation unit 170 optimizes the height information based on the interpolation shape candidate value input from the data interpolation unit 164, and determines the estimated value of the three-dimensional shape of the subject. The 3D shape estimation unit 170 outputs the determined three-dimensional shape of the subject to the image composition unit 180. The image synthesis unit 180 synthesizes a plurality of images with different focal positions based on the three-dimensional shape of the subject input from the 3D shape estimation unit 170 and the plurality of images acquired from the image acquisition unit 110. Create This synthesized image is, for example, a three-dimensional reconstructed image or an omnifocal image. The image composition unit 180 outputs the created composite image to the display unit for display, for example, or outputs it to the storage device for storage, for example.

  An example of the operation of the image processing system 100 according to the present embodiment will be described with reference to the flowchart shown in FIG. In step S101, the image acquisition unit 110 acquires a plurality of images captured while changing the focal position of the same subject. These images include information related to depth, such as information related to the focal position of the optical system when the image is acquired. The image acquisition unit 110 stores the acquired image in the storage unit 114.

  In step S102, the band processing unit 120 applies, for example, each of the first filter 121, the second filter 122, and the third filter 123 to each pixel of the plurality of images having different focal positions stored in the storage unit 114. The filtering process is performed using Here, since the number of filters is not limited, the band processing unit 120 will be described as having N filters in the following description. Band processing unit 120 outputs the result of the filtering process to band characteristic evaluation unit 130.

In step S103, the band characteristic evaluation unit 130 calculates a band evaluation value for each band for each region of the plurality of filtered images. That is, for each focal position h (h = 1, 2,..., H) and for each pixel (i, j) (each pixel (i, j) included in the entire area A of the image), that is, data For each of I (h, i, j), a band evaluation value Q (h, f _n , i, j) is calculated for each frequency band f _n (n = 1, 2,..., N). The band evaluation value Q (h, f _n , i, j) is calculated, for example, as an integral value of the signal that has passed through the filter, and is an amount corresponding to the amplitude in the band that passes through the filter. The band characteristic evaluation unit 130 outputs the band evaluation value Q (h, f _n , i, j) to the statistical information calculation unit 142 and the contrast evaluation unit 152.

統計情報算出部１４２は、算出した統計情報値をパラメータ決定部１４４に出力する。 In step S104, the statistical information calculation unit 142 calculates a statistical information value representing an average value of the band evaluation values Q (h, f _n , i, j) for a plurality of images having different focal positions for each frequency band. This statistical information value is represented by the following formula (1) as an average value L (f _n , i, j), for example.

The statistical information calculation unit 142 outputs the calculated statistical information value to the parameter determination unit 144.

ここで、閾値Ｔｈｒは、例えばＮ＝５の場合は０．２といった任意の設計値である。この変数Ｌ_Ｎ（ｆ_ｎ，ｉ，ｊ）を用いて、有効周波数ｆνを決定する。例えば、低周波数側から計数して、すなわち低周波数側からｎを順に大きくして、下記条件式（３）を満たす最小の周波数ｆ_ｍに対するｆ_ｍ−１を有効周波数ｆνとする。

すなわち、下記式（４）の条件を満たす最大の周波数を有効周波数ｆνとする。

なお、上記式（２）及び（３）を用いた処理に限らず、単に閾値Ｔｈｒを超える最大の周波数を有効周波数ｆνとしてもよい。 In step S105, the parameter determining unit 144 determines the effective frequency fν. The effective frequency fν is determined as follows, for example. The variable L _N (f _n , i, j) is set to 1 or 0 depending on whether or not the value relating to the average value L (f _n , i, j) satisfies a predetermined condition. That is, the variable L _N (f _n , i, j) is expressed by, for example, the following formula (2).

Here, the threshold value Thr is an arbitrary design value such as 0.2 when N = 5, for example. The effective frequency fν is determined using the variable L _N (f _n , i, j). For example, counted from the low frequency side, that is, in turn increasing the n from a low frequency side, the f _m-1 and effective frequency fν for minimum frequency f _m which satisfies the following conditional expression (3).

That is, the maximum frequency that satisfies the condition of the following expression (4) is set as the effective frequency fν.

Note that the processing is not limited to the processing using the above formulas (2) and (3), and the maximum frequency exceeding the threshold Thr may be simply set as the effective frequency fν.

In step S106, the parameter determination unit 144 determines the correction parameters m, n, w (k, l) used in the data correction unit 162 and the interpolation parameters σ _k , used in the data interpolation unit 164 based on the effective frequency fν. σ _l is determined. The parameter determination unit 144 stores, for example, a lookup table indicating the relationship between the effective frequency fν and the correction parameters m, n, w (k, l). The parameter determination unit 144 refers to this lookup table and determines the correction parameters m, n, w (k, l) based on the effective frequency fν. Here, the correction parameters m and n become larger as the effective frequency fν is lower. Further, as w (k, l), a function is provided such that the weight does not decrease when the values of m and n in Expression (7) described later are large. The parameter determination unit 144 outputs the determined correction parameters m, n, w (k, l) to the data correction unit 162.

ここで、ｉｎｔは整数化処理であり、Ｃ_１及びＣ_２は任意の係数である。このほかに一般に負の相関を有する関数を使用してもよい。 Further, since the distance dimension and the frequency dimension (for example, the number of periods per unit distance) are inversely related, the parameter determination unit 144 may obtain the correction parameters m and n by the following equation (5). .

Here, int is an integer process, and C ₁ and C ₂ are arbitrary coefficients. In addition, a function generally having a negative correlation may be used.

The parameter determination unit 144 determines the interpolation parameters σ _k and σ _l used by the data interpolation unit 164 based on the effective frequency fν. The parameter determination unit 144 stores a look-up table indicating the relationship between the effective frequency fν and the interpolation parameters σ _k and σ _l , for example. The parameter determination unit 144 refers to this lookup table and determines the interpolation parameters σ _k and σ _l based on the effective frequency fν. Here, the interpolation parameters σ _k and σ _l become larger as the effective frequency fν is lower. The parameter determination unit 144 outputs the determined interpolation parameters σ _k and σ _l to the data interpolation unit 164.

ここで、ｉｎｔは整数化処理であり、Ｃ_３及びＣ_４は任意の係数である。 In addition, since the interpolation parameters σ _k and σ _l represent dispersion and the dispersion has a dimension of distance, the parameter determination unit 144, like the correction parameters m and n, uses the following equation (6) to calculate the interpolation parameter σ You may obtain _| require _k and (sigma) _l .

Here, int is an integerization process, and C ₃ and C ₄ are arbitrary coefficients.

In step S107, the contrast evaluation unit 152 acquires the band evaluation value Q (h, f _n , i, j) from the band characteristic evaluation unit 130, evaluates the intensity of the high frequency component for each pixel of the plurality of images, and performs contrast. An evaluation value is calculated. The contrast evaluation unit 152 outputs the calculated contrast evaluation value relating to each pixel of each image to the shape candidate estimation unit 154.

  In step S108, the shape candidate estimation unit 154 evaluates the focus of each pixel in the image for each of the plurality of images based on the contrast evaluation value input from the contrast evaluation unit 152. The shape candidate estimation unit 154 evaluates that, for example, the higher the contrast, the higher the degree of focus. Furthermore, the shape candidate estimation unit 154 selects the most focused image among a plurality of images having different focal planes for each pixel in the image. The shape candidate estimation unit 154 acquires information on the focal position when the most focused image is captured from the image acquisition unit 110. The shape candidate estimation unit 154 estimates the depth of the subject corresponding to each pixel in the image based on the information acquired from the image acquisition unit 110, and the shape candidate value P (i, j) that is information related to the shape of the subject. Is calculated. The shape candidate value P (i, j) represents, for example, the depth of the subject at the coordinates (i, j). When the depth of the subject cannot be estimated based on the contrast evaluation value, the shape candidate estimation unit 154 sets the shape candidate value P (i, j) corresponding to the pixel as a value indicating that estimation is impossible. The shape candidate estimation unit 154 outputs the calculated shape candidate value P (i, j) to the data correction unit 162 in the data correction unit 160.

  In step S109, the data correction unit 162 performs noise / isolated point removal processing for removing noise or isolated points on the shape candidate value P (i, j). In the present embodiment, the noise / isolated point removal process is performed by a coring process. An example of noise / isolated point removal processing will be described with reference to a flowchart shown in FIG.

  In step S210, the data correction unit 162 reads the shape candidate value P (i, j). In the present embodiment, it is assumed that the image size is p + 1 pixels in the horizontal direction from 0 to p, and q + 1 pixels in the vertical direction from 0 to q. In step S220, the data correction unit 162 reads the correction parameters m, n, w (k, l).

上記式（７）のとおり、基準値Ｐ_ａｖｅ（ｉ，ｊ，ｍ，ｎ）は、この領域における平均値を表す。また、上記式（７）において、パラメータ決定部１４４が決定した補正パラメータｍ，ｎ，ｗ（ｋ，ｌ）が用いられている。すなわち、上記式（７）は、有効周波数ｆνに応じて変化する。 In the present embodiment, as shown in FIG. 5, in steps S231 to S234, the following processing is sequentially performed on the shape candidate values P (i, j) corresponding to all the pixels of the image. In step S231, the data correction unit 162 calculates the reference value P _ave (i, j, m, n) of the region including (i, j) based on the following equation (7).

As shown in the above formula (7), the reference value P _ave (i, j, m, n) represents an average value in this region. In the above equation (7), the correction parameters m, n, w (k, l) determined by the parameter determining unit 144 are used. That is, the above equation (7) changes according to the effective frequency fν.

In step S232, the data correction unit 162 determines whether the difference between the shape candidate value P (i, j) and the reference value P _ave (i, j, m, n) is smaller than a predetermined threshold value. When the difference between the shape candidate value P (i, j) and the reference value P _ave (i, j, m, n) is smaller than the predetermined threshold Th _r−1 , that is, “| P (i, j) − When the determination of P _ave (i, j, m, n) | <Th _r−1 ”is true, the process proceeds to step S234. Here, the threshold value Th _r−1 is determined based on an empirical rule such as a criterion for determining whether or not the error falls within the reference value error range.

On the other hand, when the difference between the shape candidate value P (i, j) and the reference value P _ave (i, j, m, n) is not smaller than the predetermined threshold value, in step S233, the data correction unit 162 determines the shape candidate. It is determined whether the value P (i, j) is an isolated point. When the shape candidate value P (i, j) is an isolated point, the process proceeds to step S234.

Here, for determining whether or not the point is an isolated point, for example, whether or not “| P (i, j) −P _ave (i, j, m, n) |> Th _r-2 ” is true. Is used. Here, Th _r-2 is a threshold value set based on the variance value in a predetermined plurality of pixel regions. More specifically, for example, when the variance value is σ, Th _r−2 is set to ± 2σ.

In step S234, the data correction unit 162 sets the value of the shape candidate value P (i, j) to the reference value P _ave (i, j, m, n). The processes in steps S231 to S234 are performed for all pixels. That is, this process is expressed by the following formula (8) when the predetermined threshold is ΔT and the processed shape candidate value is the noise removal shape candidate value P ′ (i, j).

  The concept of the coring process used in this embodiment will be described with reference to FIGS. 6A, 6B, and 6C. FIG. 6A represents an original signal corresponding to the shape candidate value P (i, j). For this original signal, a moving average corresponding to the average value calculated by the above equation (7) is shown by a one-dot chain line in FIG. 6B. A value obtained by adding or subtracting a threshold corresponding to the predetermined threshold ΔT to the moving average is indicated by a broken line in FIG. 6B. In this case, as represented by the above equation (8), when the original signal is located between two broken lines in FIG. 6B, the original signal is replaced with a moving average indicated by a one-dot chain line. As a result, a result as shown in FIG. 6C is obtained. In this figure, a circle indicates a value replaced with a moving average. As described above, the coring process has an effect of suppressing a fluctuation component that is determined to be a small amplitude signal and eliminating an error.

  The data correction unit 162 performs the noise / isolated point removal processing described with reference to FIG. 5 on the shape candidate value P (i, j), that is, the noise removal shape candidate value P ′ (i, j). ) Is output to the data interpolation unit 164.

  In step S110, the data interpolation unit 164 performs an interpolation process for interpolating data that is a value indicating that estimation is impossible for the noise removal shape candidate value P ′ (i, j) input from the data correction unit 162. Inability to estimate means that, when the shape candidate estimation unit 154 calculates the shape candidate value P (i, j) based on the contrast evaluation value calculated by the contrast evaluation unit 152, the shape candidate estimation unit 154 focuses the image. It means that it could not be identified. In other words, the impossibility of estimation means that the contrast evaluation value of any of the plurality of images for the pixel does not meet the condition indicating a predetermined focus.

  When the periphery of the value indicating the impossibility of estimation for the noise removal shape candidate value P ′ (i, j) is not incapable of estimation, that is, for example, when only one pixel in the region of 5 pixels × 5 pixels cannot be estimated, The data interpolation unit 164 interpolates the data that cannot be estimated using neighboring data. At this time, the data interpolation unit 164 can use, for example, bilinear interpolation or bicubic interpolation for data interpolation.

On the other hand, when there are consecutive values indicating that estimation is not possible for the noise removal shape candidate value P ′ (i, j), the data interpolation unit 164 performs estimation based on a function that represents a correlation with neighboring data. Interpolate data that is disabled. That is, assuming the distribution around the location that cannot be estimated, the value of the location is estimated. In this embodiment, a kernel regression method is used for interpolation. At this time, the data interpolation unit 164 uses the interpolation parameters σ _k and σ _l input from the parameter determination unit 144. An example of this interpolation processing will be described with reference to the flowchart shown in FIG.

ここで、Ｎはサンプリング数を表し、下記式（１０）で表される。

また、Ｃ（ｋ，ｌ）は、下記式（１１）で表される。

上記式（１１）に示すようにＣ（ｋ，ｌ）は、補間パラメータσ_ｋ，σ_ｌに応じて決定される。なお、Ｂは変数である。 In step S310, the data interpolation unit 164 reads the noise removal shape candidate value P ′ (i, j). In step S320, the data interpolation unit 164 reads the interpolation parameters σ _k and σ _l . Subsequently, the data interpolation unit 164 calculates the interpolation data R (i, j). Here, the interpolation data R (i, j) is expressed by the following equation (9).

Here, N represents the number of samplings and is represented by the following formula (10).

C (k, l) is expressed by the following formula (11).

As shown in the above equation (11), C (k, l) is determined according to the interpolation parameters σ _k and σ _l . B is a variable.

ここで、Ｔｈｒは、所定の閾値である。この収束条件を満たしていれば、処理はステップＳ３４０に進む。一方、収束条件を満たしていなければ、所定の回数であるＤ回まで、ステップＳ３３１乃至ステップＳ３３３の処理を繰り返す。すなわち、収束条件を満たすまで、ステップＳ３３１で変数Ｂの値を変更しながら、ステップＳ３３２で各変数Ｂに対する補間データＲ（ｉ，ｊ）を算出し、ステップＳ３３３で算出した補間データＲ（ｉ，ｊ）が収束条件を満たすか否か判定する。 In step S331, the data interpolation unit 164 updates the variable B. In step S332, the data interpolation unit 164 superimposes a Gaussian kernel on the noise removal shape candidate value P ′ (i, j) based on the above equations (9) to (11). In step S333, the data interpolation unit 164 determines whether or not the value obtained in step S332 satisfies a predetermined convergence condition represented by the following formula (12), for example.

Here, Thr is a predetermined threshold value. If this convergence condition is satisfied, the process proceeds to step S340. On the other hand, if the convergence condition is not satisfied, the processing from step S331 to step S333 is repeated up to a predetermined number of times D. That is, until the convergence condition is satisfied, the interpolation data R (i, j) for each variable B is calculated in step S332 while changing the value of the variable B in step S331, and the interpolation data R (i, j, calculated in step S333). It is determined whether j) satisfies the convergence condition.

  If it is determined in step S333 that the interpolation data R (i, j) satisfies the convergence condition, in step S340, the data interpolation unit 164 is based on the interpolation data R (i, j) that satisfies the condition. To generate extended data. In step S350, the data interpolation unit 164 assigns the generated extension data to data that cannot be estimated among the noise removal shape candidate values P ′ (i, j), and the interpolation shape candidate value P ″ (i, j). ) Is generated. The data interpolation unit 164 outputs the generated interpolation shape candidate value P ″ (i, j) to the 3D shape estimation unit 170.

  Returning to FIG. 4, the description will be continued. In step S111, the 3D shape estimation unit 170 optimizes the height information based on the interpolation shape candidate value P ″ (i, j) input from the data interpolation unit 164, and determines the three-dimensional shape of the subject. presume. The 3D shape estimation unit 170 outputs the estimated three-dimensional shape of the subject to the image composition unit 180.

  In step S112, the image composition unit 180 synthesizes a plurality of images with different focal planes based on the three-dimensional shape of the subject input from the 3D shape estimation unit 170 and the plurality of images acquired from the image acquisition unit 110. To create a composite image. For example, if the synthesized image is a three-dimensional reconstructed image, a synthesized image is created by synthesizing a three-dimensional shape and a focused image relating to each part of the three-dimensional shape. For example, if the composite image is an omnifocal image, an image extracted from an image having a focal plane corresponding to the height of each pixel is combined to synthesize an image focused on all the pixels. The image composition unit 180 outputs the created composite image to a display unit or a storage device.

  For example, when a subject having a depth larger than the depth of field is captured using an optical system having a shallow depth of field, such as a microscope image, the image is not easily recognized by the user. On the other hand, according to the three-dimensional reconstructed image or the omnifocal image, an image of a subject having a depth larger than the depth of field can be easily recognized by the user.

  Thus, for example, the image acquisition unit 110 functions as an image acquisition unit that acquires a plurality of images captured at different focal positions for the same subject. For example, the candidate value estimation unit 150 functions as a candidate value estimation unit that estimates a three-dimensional shape candidate value based on a plurality of images for each pixel of the image. For example, the band characteristic evaluation unit 130 functions as a band characteristic evaluation unit that calculates a band evaluation value of a band included in the image for each of a plurality of frequency bands for each pixel of the plurality of images. For example, the effective frequency determination unit 140 functions as an effective frequency determination unit that determines the effective frequency of the pixel based on the statistical information of the band evaluation value. For example, the data correction unit 160 functions as a candidate value correction unit that performs at least one of data correction and data interpolation on the candidate value based on the effective frequency, and calculates a correction candidate value representing the three-dimensional shape of the subject. To do. For example, the data correction unit 162 functions as a correction candidate value calculation unit that calculates a correction candidate value using a correlation between values of local regions represented by the candidate value. For example, the 3D shape estimation unit 170 functions as an omnifocal image creation unit or a 3D reconstructed image creation unit.

According to the present embodiment, as a result of the noise / isolated point removal processing by the data correction unit 162, image noise and errors due to estimation processing are effectively reduced. Here, in the present embodiment, the correction parameters m and n used in the noise / isolated point removal processing are determined based on the effective frequency fν in the image. That is, since the correction parameters m and n are larger as the effective frequency fν is lower, in the above equation (7), the lower the effective frequency fν, the larger the shape candidate value P (i, j). reference value _{P ave (i, j, m} , n) is calculated, shape candidate value narrow enough effective frequency fν region of high P (i, j) to the reference value _P ave (i based, j, m, n ) Is calculated. That is, the optimum reference value P _ave (i, j, m, n) corresponding to the effective frequency fν of the image is calculated. As a result, noise can be reduced more accurately than in the case where the effective frequency fν of the image is not taken into consideration. That is, the shape candidate value P (i, j) is not excessively smoothed, and even when there is a lot of noise, the input signal is not evaluated as an excessively high frequency signal.

Further, in the interpolation processing in the data interpolation unit 164, information on the effective frequency fν of the image is used when the correlation of neighboring data is assumed. That is, a Gaussian kernel optimized according to the frequency band is generated, and the depth value of the subject at a position that cannot be estimated is estimated based on the contrast evaluation value. At this time, since the interpolation parameters σ _k and σ _l are given based on the effective frequency fν, the calculation speed is reduced and the processing speed is increased as compared with the case where the convergence value is searched while changing these values. And it is possible to prevent the calculation result from converging to an incorrect value. Here, since the interpolation parameters σ _k and σ _l become larger as the effective frequency fν is lower, in the above equation (9), the noise removal shape candidate value P ′ (i, j in a wider region is smaller as the effective frequency fν is lower. ) To calculate interpolation data R (i, j). The higher the effective frequency fν, the more the interpolation data R (i, j) is calculated based on the noise removal shape candidate value P ′ (i, j) in a narrow area. Is done. That is, the edge structure is appropriately evaluated without being excessively smoothed, and even when there is a lot of noise, the input signal is not excessively evaluated as a high-frequency signal.

  In addition, each said formula is an example, if the above-mentioned effect is acquired, it will not be limited to these formulas, Of course, another formula may be used. For example, real order polynomials, logarithmic functions, and exponential functions may be used instead of the above equations (5) and (6). Moreover, dispersion | distribution etc. can be used instead of the said Formula (1). In the above-described embodiment, the process is performed for each pixel, but the process may be performed for each region including a plurality of pixels.

[Modification of First Embodiment]
A modification of the first embodiment will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. In the processing by the data interpolating portion 164 in the first embodiment, in the loop process of steps S331 through step S333 described with reference to FIG. 5, sigma _k in the above equation (11), the sigma _l interpolation parameter sigma _k, σ _l is set and these values do not change.

On the other hand, in this modification, the convergence values are searched for while σ _k and σ _l are also changed in step S331. Therefore, in the present modification, the parameter determination unit 144 outputs a possible range or probability density function to the data interpolation unit 164 as the interpolation parameters σ _k and σ _l . Data interpolating unit 164, the loop process of steps S331 through step S333, interpolation parameter sigma _k input from parameter determining section 144, based on the possible range or probability density function of the σ _l, σ _k, σ _l also The convergence value is searched while changing. Other operations are the same as those in the first embodiment.

According to this modification, the processing amount is increased as compared with the first embodiment, but the interpolation data R (i, j) can be converged to a convergence value more suitable than that of the first embodiment. Also in the present modification, the parameter determination unit 144 determines a possible range or probability density function as the interpolation parameters σ _k and σ _l based on the effective frequency fν, so that the same effect as in the first embodiment can be obtained. .

ここで、Ｃ（ｋ，ｌ）は距離相関を規定する因子であり、Ｓ（Ｐ_１−Ｐ_２）は異なる画素間における画素レベルの差に起因する相関を規定する因子である。Ｃ（ｋ，ｌ）及びＳ（Ｐ_１−Ｐ_２）にどのような分布関数を用いるかによって、生成される画像の鮮鋭度、Ｓ／Ｎ比が変化する。 [Second Embodiment]
A second embodiment of the present invention will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. In this embodiment, the data correction unit 162 uses a bilateral filter for noise removal. A formula representing the bilateral filter used in the present embodiment is shown in the following formula (13).

Here, C (k, l) is a factor that defines the distance correlation, and S (P ₁ −P ₂ ) is a factor that defines the correlation caused by the difference in pixel level between different pixels. The sharpness and S / N ratio of the generated image vary depending on what distribution function is used for C (k, l) and S (P ₁ -P ₂ ).

ここで、σ_ｋ，σ_ｌは補正パラメータであり、Ｃ_５は所定の定数である。補正パラメータσ_ｋ，σ_ｌは、第１の実施形態における補間パラメータσ_ｋ，σ_ｌと同じである。また、Ｓ（Ｐ_１−Ｐ_２）を例えば下記式（１５）とする。

ここで、σ_Ｐは補正パラメータであり、Ｃ_６は所定の定数である。本実施形態では、パラメータ決定部１４４は、補正パラメータσ_Ｐをも画像の有効周波数ｆνに基づいてルックアップテーブルを参照して決定する。ここで、補正パラメータσ_Ｐは、有効周波数ｆνが低いほど大きな値となる。 In the present embodiment, for example, a function based on a Gaussian distribution is used for C (k, l) and S (P ₁ −P ₂ ). That is, C (k, l) is, for example, the following formula (14).

Here, σ _k and σ _l are correction parameters, and C ₅ is a predetermined constant. The correction parameters σ _k and σ _l are the same as the interpolation parameters σ _k and σ _l in the first embodiment. Further, S (P ₁ -P ₂ ) is, for example, the following formula (15).

Here, σ _P is a correction parameter, and C ₆ is a predetermined constant. In the present embodiment, the parameter determination unit 144 determines the correction parameter σ _P by referring to the lookup table based on the effective frequency fν of the image. Here, the correction parameter σ _P has a larger value as the effective frequency fν is lower.

In addition, since the correction parameter σ _p and the frequency have a positive correlation, the parameter determination unit 144 uses the M (M> 0) integer polynomial as shown in the following equation (16) to correct the correction parameter σ _p may be obtained.

As in the case of the first embodiment, if information on the effective frequency fν of an image is acquired, the sharpness inherent in the image can be estimated. For example, when the effective frequency fν is low, C (k, l) is set so that long-range correlation is emphasized, and further, based on the assumption that there is no abrupt step in neighboring data. (P ₁ -P ₂ ) is set. Thus, for example, S (P ₁ −P ₂ ) functions as a first correlation that is a correlation between two distant values. For example, C (k, l) functions as a second correlation that is a correlation by distance.

  In the present embodiment, when the correlation of neighboring data is assumed, information on the frequency band that the image originally has is used. A bilateral filter is set based on the correlation of the neighboring data. As a result, according to the present embodiment, the noise removal shape candidate value P ′ (i, j) in which the noise and error of the shape candidate value P (i, j) are effectively reduced can be acquired.

In the present embodiment as well, the correction parameters σ _k , σ _l , and σ _P may be used as the established density variables, as in the first modification of the first embodiment. In this case, the same effect as that of the present embodiment can be obtained.

ここで、ＰΔ（ｉ，ｊ，ｋ，ｌ）は、下記式（１８）で表される。

また、Ｎ（ｉ，ｊ，ｋ，ｌ）は、下記式（１９）で表される。

ここで、Ｕ（ｉ，ｊ）は平滑化した勾配ベクトルを表し、下記式（２０）で表される。

ここで、Ｐ_ｆ（ｉ，ｊ，ｋ，ｌ）は、下記式（２１）で表される。

ここで、Ｕ（ｉ，ｊ）_ｉは勾配の水平成分を表し、Ｕ（ｉ，ｊ）_ｊは勾配の垂直成分を表している。 [First Modification of Second Embodiment]
A first modification of the second embodiment will be described. Here, differences from the second embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. In the present modification, the data correction unit 162 uses a trilateral filter for noise removal. A formula representing the trilateral filter used in the present embodiment is shown in the following formula (17).

Here, PΔ (i, j, k, l) is expressed by the following formula (18).

N (i, j, k, l) is represented by the following formula (19).

Here, U (i, j) represents a smoothed gradient vector and is represented by the following equation (20).

Here, P _f (i, j, k, l) is expressed by the following equation (21).

Here, U (i, j) _i represents the horizontal component of the gradient, and U (i, j) _j represents the vertical component of the gradient.

  This trilateral filter is obtained by applying the bilateral filter used in the second embodiment to the gradient ∇P (i, j). By introducing ∇P (i, j), impulse noise, that is, an isolated fluctuation component can be strongly suppressed.

Also in this modification, C (k, l) and S (P ₁ −P ₂ ) determined according to the effective frequency fν of the image are used as in the case of the second embodiment. As a result, the same effect as in the second embodiment can be obtained.

[Third Embodiment]
A third embodiment of the present invention will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. The present embodiment is a microscope system 200 including the image processing system 100 according to the first embodiment.

  An outline of a configuration example of the microscope system 200 according to the present embodiment is shown in FIG. As shown in this figure, the microscope system 200 includes a microscope 210 and the image processing system 100 according to the first embodiment. The microscope 210 is a digital microscope, for example. The microscope 210 includes an LED light source 211, an illumination optical system 212, an optical path control element 213, an objective lens 214, a specimen surface 215 on a stage (not shown), an observation optical system 218, an imaging surface 219, and an imaging unit 220. And a controller 222. The observation optical system 218 includes a zoom optical system 216 and an imaging optical system 217. In the observation optical path, an objective lens 214, an optical path control element 213, a zoom optical system 216, and an imaging optical system 217 are arranged in this order from the sample surface 215 to the imaging surface 219.

  The illumination light emitted from the LED light source 211 enters the optical path control element 213 via the illumination optical system 212. The optical path control element 213 reflects the illumination light toward the objective lens 214 on the observation optical path. The illumination light is applied to the sample disposed on the sample surface 215 via the objective lens 214.

  When the sample is irradiated with illumination light, observation light is generated from the sample. Here, the observation light is reflected light, fluorescence, or the like. The observation light is incident on the optical path control element 213. Unlike the case of illumination light, the optical path control element 213 transmits observation light and makes the observation light incident on an observation optical system 218 having a zoom optical system 216 and an imaging optical system 217. Thus, the optical path control element 213 is an optical element that reflects or transmits incident light according to the characteristics of the incident light. For the optical path control element 213, for example, a polarization element such as a wire grid or a polarization beam splitter (PBS) that reflects or transmits incident light according to the polarization direction of the incident light can be used. The optical path control element 213 may use, for example, a dichroic mirror that reflects or transmits incident light according to the frequency of incident light.

  The observation optical system 218 collects observation light on the imaging surface 219 and forms an image of the sample on the imaging surface 219. The imaging unit 220 generates an image signal based on the image formed on the imaging surface 219 and outputs the image signal to the image acquisition unit 110 as a microscope image. The controller 222 controls the operation of the microscope 210. In the present embodiment, the microscope 210 acquires a plurality of microscope images captured at different focal planes for the same specimen. For this reason, the controller 222 controls the optical system of the microscope 210 to cause the imaging unit 220 to capture an image of the sample on each focal plane while gradually changing the focal plane. Specifically, for example, the controller 222 causes the imaging unit 220 to capture an image while changing the height of the stage of the microscope 210 or the position of the height of the objective lens. The controller 222 outputs information related to the focal position to the image acquisition unit 110 in association with the image captured by the imaging unit 220.

  An operation of the microscope system 200 according to the present embodiment will be described. The sample is placed on a stage (not shown) so as to have a sample surface 215. The controller 222 controls the microscope 210. For example, the controller 222 gradually changes the position of the sample surface 215 in the optical axis direction, and gradually changes the focal position of the optical system with respect to the sample. Specifically, for example, the controller 222 changes the height of the stage of the microscope 210, the height of the objective lens, or the position of the focus lens. At this time, the controller 222 causes the imaging unit 220 to sequentially capture microscopic images of the specimen at each focal position. The image acquisition unit 110 acquires a microscopic image of the specimen at each focal position from the imaging unit 220. Further, the image acquisition unit 110 acquires the focal position when each image is captured from the controller 222. The image acquisition unit 110 stores the acquired microscope image in the storage unit 114 in association with the focal position.

  The process of creating a composite image by combining a plurality of images with different focal positions based on the microscope image stored in the storage unit 114 is the same as the process of the first embodiment. In the present embodiment, the microscope system 200 creates a composite image related to a microscope image, such as a three-dimensional reconstructed image or an omnifocal image. The image composition unit 180 outputs the created composite image, for example, to the display unit for display or to the storage device for storage. According to the three-dimensional reconstructed image and the omnifocal image, an image of a subject having a depth larger than the depth of field like a general microscope image can be easily recognized by the user.

  Thus, for example, the illumination optical system 212, the optical path control element 213, the objective lens 214, the observation optical system 218, and the like function as a microscope optical system. For example, the imaging unit 220 functions as an imaging unit that acquires a specimen image through a microscope optical system as a specimen image.

  In general, an optical system of a microscope has a larger image enlargement ratio than an optical system of a digital camera. For this reason, in microscope photography, the bandwidth of the optical system of the microscope may not be so high with respect to the sampling bandwidth of the camera image sensor. Further, the bandwidth of the optical system can change depending on the numerical aperture, magnification, etc. of the optical system. For example, when the microscope has a zoom optical system, the band of the optical system changes. According to the present embodiment, the statistical information calculation unit 142 calculates a statistical information value in consideration of the frequency band of the image, and the parameter determination unit 144 calculates a correction parameter and an interpolation parameter based on the statistical information value. Compared with the case where the effective frequency fν of the image is not taken into consideration, noise or the like can be reduced with high accuracy and interpolation can be appropriately performed. As a result, the microscope system 200 can create a three-dimensional reconstructed microscope image and an omnifocal microscope image with high accuracy.

  Note that when the zoom optical system is included in the optical system of the microscope 210, the numerical aperture changes in accordance with the focal length of the zoom optical system and the band of the microscope image changes, so this embodiment is particularly effective. In the above embodiment, the image processing system 100 has been described as related to the first embodiment. However, the second embodiment or a modification thereof may be used.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the problem described in the column of problems to be solved by the invention can be solved and the effect of the invention can be obtained. The configuration in which this component is deleted can also be extracted as an invention. Furthermore, constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 100 ... Image processing system 110 ... Image acquisition part 114 ... Memory | storage part 120 ... Band processing part 121 ... 1st filter 122 ... 2nd filter 123 ... 3rd filter 130 ... Band characteristic evaluation part , 140 ... effective frequency determination unit, 142 ... statistical information calculation unit, 144 ... parameter determination unit, 150 ... candidate value estimation unit, 152 ... contrast evaluation unit, 154 ... shape candidate estimation unit, 160 ... data correction unit, 162 ... data Correction unit, 164 ... Data interpolation unit, 170 ... 3D shape estimation unit, 180 ... Image composition unit, 200 ... Microscope system, 210 ... Microscope, 211 ... LED light source, 212 ... Illumination optical system, 213 ... Optical path control element, 214 ... Objective lens, 215 ... specimen surface, 216 ... zoom optical system, 217 ... imaging optical system, 218 ... observation optical system, 219 ... imaging surface, 220 ... imaging , 222 ... controller.

Claims (13)

An image acquisition unit that acquires a plurality of images captured at different focal positions for the same subject;
A candidate value estimator for estimating a candidate value of a three-dimensional shape based on a plurality of the images for each pixel of the image;
A band characteristic evaluation unit that calculates a band evaluation value of a band included in the image for each of a plurality of frequency bands for each pixel of the plurality of images,
An effective frequency determining unit that determines an effective frequency of the pixel based on statistical information of the band evaluation value;
A candidate value correction unit that performs at least one of data correction and data interpolation on the candidate value based on the effective frequency, and calculates a correction candidate value representing a three-dimensional shape of the subject;
Equipped with,
The candidate value correction unit includes a correction candidate value calculation unit that calculates the correction candidate value using a correlation of values of local regions represented by the candidate value;
The area of the local region is determined according to the effective frequency . The image processing system according to claim 1 , wherein the data correction suppresses a fluctuation component that is determined to be a small amplitude signal by using a correlation between values of a local region represented by the candidate value. The correlation includes a first correlation that is a correlation between two points separated from each other, and a second correlation that is a correlation based on distance.
The first correlation and the second correlation are determined according to the effective frequency.
The image processing system according to claim 1 . The image processing system according to claim 3 , wherein the correction candidate value calculation unit calculates the correction candidate value using a bilateral filter including the first correlation and the second correlation. The image processing system according to claim 3 , wherein the correlation is based on a probability density function. The image processing system according to claim 1 , wherein the data correction includes suppressing an isolated variation component included in the candidate value. The data interpolation using the correlation due to the distance, the image processing system according to any one of claims 1 to 6, characterized in that it comprises interpolating the values could not be determined from among the candidate values . The image processing according to any one of claims 1 to 7 , further comprising an omnifocal image creating unit that creates an omnifocal image based on the correction candidate value and the plurality of images. system. Based on said plurality of images and the correction candidate value, to any one of claims 1 to 7, characterized by further comprising a three-dimensional reconstructed image creating unit that creates a three-dimensional reconstructed image The image processing system described. The image processing system according to claim 1 , wherein the area is determined so as to increase as the effective frequency decreases. The image processing system according to claim 7 , wherein the correlation based on the distance is determined such that the weight of the value in a wide area becomes higher as the effective frequency is lower. Microscope optics,
An imaging unit that obtains an image of a specimen through the microscope optical system as a specimen image;
The image processing system according to any one of claims 1 to 11 , wherein the sample image is acquired as the image.
A microscope system comprising: The microscope system according to claim 12 , wherein the microscope optical system includes a variable magnification optical system.

Images