JP5868758B2 - Image processing system and microscope system including the same - Google Patents

Description

  The present invention relates to an image processing system and a microscope system including the image processing system.

  In general, a method for evaluating focusing based on the band contrast of an image is known. In-focus evaluation based on contrast is used for acquiring depth information of a subject, for example, in addition to an autofocus function or the like. The depth information is acquired by, for example, selecting an image focused on each position from the plurality of images after imaging the subject at a plurality of focal positions. In addition, the depth information is obtained by capturing an image of a subject at a plurality of focal positions, selecting an in-focus image for each position of the subject from the plurality of images, and synthesizing the in-focus images. This is used when creating a focus image or a three-dimensional reconstructed image.

  For example, Patent Document 1 discloses a technique for determining in-focus based on contrast and creating an omnifocal image based on the focused image. Patent Document 1 discloses a technique related to controlling the characteristics of a filter that limits a high frequency in order to obtain a predetermined contrast even in an unfocused region.

JP 2010-166247 A

  In general, an image has different frequency bands depending on the optical system used to acquire the image, the magnification, the characteristics of the subject, and the like. In in-focus determination based on contrast, more accurate in-focus evaluation is more easily realized when the high-frequency band is used for contrast evaluation. However, on the other hand, if focus evaluation is performed based on information having a frequency higher than the band of the image, inappropriate focus evaluation is performed based on factors unrelated to the shape of the subject such as noise. There is a fear.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing system that performs focusing evaluation based on contrast in consideration of a frequency band of an image, and a microscope system including the image processing system.

  To achieve the above object, according to an aspect of the present invention, an image processing system includes an image acquisition unit that acquires a plurality of images captured at different focal positions for the same subject, and a predetermined region in the plurality of the images. On the other hand, for each of a plurality of frequency bands, a band characteristic evaluation unit that calculates a band evaluation value of a band included in the image, and statistical information using at least two of the band evaluation values of the focal position, at least for each frequency band Based on the statistical information calculation unit, a load coefficient calculation unit that calculates a load coefficient for the band evaluation value for each frequency band based on the statistical information, and the band evaluation value and the load coefficient A contrast evaluation unit for calculating a contrast evaluation value for each of the regions in the plurality of images, and a plurality of the Characterized in that it comprises a and a focus evaluation unit selects the area being focused among each of the areas of the image.

  Moreover, according to one aspect of the present invention, a microscope system includes a microscope optical system, an imaging unit that acquires an image of a specimen through the microscope optical system as a specimen image, and the specimen image that is acquired as the image. And an image processing system.

  According to the present invention, since the load coefficient is calculated for each frequency band and the contrast evaluation value is calculated based on the load coefficient and the band evaluation value, the focus evaluation based on the contrast is performed in consideration of the frequency band of the image. An image processing system to perform and a microscope system including the image processing system can be provided.

1 is a block diagram illustrating a configuration example of an image processing system according to a first embodiment of the present invention. The figure which shows an example of the frequency characteristic of the filter bank which the band process part which concerns on 1st Embodiment has. The figure which shows another example of the frequency characteristic of the filter bank which the band process part which concerns on 1st Embodiment has. 5 is a flowchart illustrating an example of processing in the image processing system according to the first embodiment. The figure for demonstrating wavelet transformation. 6 is a flowchart illustrating an example of processing in an image processing system according to a modification of the first embodiment. The block diagram which shows the structural example of the microscope system which concerns on 2nd Embodiment.

[First Embodiment]
A first embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows an outline of a configuration example of an image processing system 100 according to the present embodiment. As shown in this figure, the image processing system 100 includes an image acquisition unit 110, a band processing unit 120, a band characteristic evaluation unit 130, a statistical information calculation unit 140, a load coefficient calculation unit 150, and a contrast evaluation unit 160. A focus evaluation unit 170, a 3D shape estimation unit 180, and an image composition unit 190.

  The image acquisition unit 110 includes a storage unit 114. The image acquisition unit 110 acquires a plurality of images captured while changing the focal position of the same subject, and stores the acquired images in the storage unit 114. Here, it is assumed that each of these images includes information related to the focal position of the optical system when the image is acquired, that is, information related to the depth of the in-focus position. The image acquisition unit 110 outputs these images in response to requests from the band processing unit 120 and the image composition unit 190.

  The band processing unit 120 has a filter bank. That is, the band processing unit 120 includes, for example, a first filter 121, a second filter 122, and a third filter 123. The frequency characteristics of the first filter 121, the second filter 122, and the third filter 123 are shown in FIG. As shown in this figure, these filters are low-pass filters, and the cut-off frequency increases in the order of the first filter 121, the second filter 122, and the third filter 123. That is, the frequency band of the signal transmitted by each filter is different. Note that the first filter 121, the second filter 122, and the third filter 123 may be band-pass filters whose frequency characteristics are shown in FIG. In addition, any filter may be used as long as it is a plurality of filters designed to transmit different frequency bands. In the present embodiment, the band processing unit 120 has three filters. However, the number may be any number. The band processing unit 120 acquires an image from the image acquisition unit 110, and for each of a plurality of images having different focal positions, the first filter 121 and the second filter 122 for each region (for example, each pixel) of the image. Filtering processing is performed using each of the third and third filters 123. Band processing unit 120 outputs the result of the filtering process to band characteristic evaluation unit 130.

  The band characteristic evaluation unit 130 calculates a band evaluation value for each region of the plurality of filtered images. The band evaluation value is obtained, for example, by calculating an integral value of a signal that has passed for each filter. In this way, for each image, a band evaluation value is obtained for each region and for each frequency band. The band characteristic evaluation unit 130 outputs the calculated band evaluation value to the statistical information calculation unit 140 and the contrast evaluation unit 160.

  The statistical information calculation unit 140 calculates a statistical information value related to the average value of the band evaluation values of a plurality of images having different focal positions for each frequency band. This statistical information will be described later. The statistical information calculation unit 140 outputs the calculated statistical information value to the load coefficient calculation unit 150. The load coefficient calculation unit 150 calculates a value related to weighting for each frequency band, that is, a load coefficient, based on the statistical information value input from the statistical information calculation unit 140. This load coefficient will be described later. The load coefficient calculation unit 150 outputs the calculated load coefficient to the contrast evaluation unit 160.

  The contrast evaluation unit 160 calculates the contrast evaluation value by multiplying the band evaluation value input from the band characteristic evaluation unit 130 by the load coefficient corresponding to each band input from the load coefficient calculation unit 150. The contrast evaluation unit 160 outputs the calculated contrast evaluation value to the focus evaluation unit 170. The focus evaluation unit 170 evaluates the focus of each region in the image for each of a plurality of images having different focal positions based on the contrast evaluation value input from the contrast evaluation unit 160. The focus evaluation unit 170 selects an in-focus image for each region, and estimates depth information corresponding to each region in the image based on information on a focal position when the image is captured. . The focus evaluation unit 170 outputs the depth information regarding each region in the image to the 3D shape estimation unit 180.

  The 3D shape estimation unit 180 optimizes the depth information based on the depth information input from the focus evaluation unit 170, and estimates the three-dimensional shape of the subject. The 3D shape estimation unit 180 outputs the estimated three-dimensional shape of the subject to the image composition unit 190. The image synthesis unit 190 synthesizes a plurality of images with different focal positions based on the three-dimensional shape of the subject input from the 3D shape estimation unit 180 and the plurality of images acquired from the image acquisition unit 110. Create an image. This synthesized image is, for example, a three-dimensional reconstructed image or an omnifocal image. The image composition unit 190 outputs the created composite image, for example, to a display unit for display or to a storage device for storage.

  An example of the operation of the image processing system 100 according to the present embodiment will be described with reference to the flowchart shown in FIG. In step S101, the image acquisition unit 110 acquires a plurality of images captured while changing the focal position of the same subject. These images include information related to depth (for example, information related to the focal position of the optical system when the image is acquired). The image acquisition unit 110 stores the acquired image in the storage unit 114.

  In step S102, the band processing unit 120 applies, for example, the first filter 121, the second filter 122, and the like to each predetermined region (for example, each pixel) in a plurality of images having different focal positions stored in the storage unit 114. A filtering process is performed using each of the third filters 123. Here, since the number of filters is not limited, the band processing unit 120 will be described as having N filters in the following description. Band processing unit 120 outputs the result of the filtering process to band characteristic evaluation unit 130.

In step S103, the band characteristic evaluation unit 130 calculates a band evaluation value for each band for each region of the plurality of filtered images. That is, for each focal position k (k = 1, 2,..., K) and for each region (i, j) (each region (i, j) included in the entire region A of the image), that is, data For each of I (k, i, j), a band evaluation value Q (k, f _n , i, j) is calculated for each frequency band f _n (n = 1, 2,..., N). The band evaluation value Q (k, f _n , i, j) is calculated, for example, as an integral value of the signal that has passed through the filter, and is an amount corresponding to the amplitude in the band that passes through the filter. The band characteristic evaluation unit 130 outputs the band evaluation value Q (k, f _n , i, j) to the statistical information calculation unit 140.

In step S104, the statistical information calculation unit 140 calculates a statistical information value related to the average value of the band evaluation values Q (k, f _n , i, j) for a plurality of images having different focal positions for each frequency band. . As will be described later, various values calculated by various methods can be used as the statistical information value. The statistical information calculation unit 140 outputs the calculated statistical information value to the load coefficient calculation unit 150.

  In step S105, the load coefficient calculation unit 150 calculates a load coefficient corresponding to the band based on the statistical information value input from the statistical information calculation unit 140. As the load coefficient, various values calculated by various methods as described later are used. The load coefficient calculation unit 150 outputs the calculated load coefficient to the contrast evaluation unit 160.

In step S <b _> 106, the contrast evaluation unit 160 corresponds to the band evaluation value Q (k, f _n , i, j) input from the band characteristic evaluation unit 130 and the frequency corresponding to the load coefficient input from the load coefficient calculation unit 150. The contrast evaluation value is calculated by multiplying the band load coefficient. The contrast evaluation unit 160 outputs the calculated contrast evaluation value to the focus evaluation unit 170.

  In step S107, the focus evaluation unit 170 evaluates the focus based on the contrast evaluation value acquired from the contrast evaluation unit 160. For example, the focus evaluation unit 170 identifies, as the focused region, a region having a contrast evaluation value higher than a predetermined threshold for each of a plurality of images having different focal positions. The in-focus evaluation unit 170 corresponds to a region from a region in focus among a plurality of images having different focal positions and information on the focal position when an image including the region is acquired. Estimate depth information about a point. Here, the depth information is, for example, a value representing the position in the depth direction of the area. The focus evaluation unit 170 outputs the depth information regarding each region to the 3D shape estimation unit 180.

  In step S108, the 3D shape estimation unit 180 optimizes depth information such as smoothing based on the depth information input from the focus evaluation unit 170, and estimates the three-dimensional shape of the subject. The 3D shape estimation unit 180 outputs the estimated three-dimensional shape of the subject to the image composition unit 190.

  In step S <b> 109, the image composition unit 190 composes a plurality of images having different focal positions based on the three-dimensional shape of the subject input from the 3D shape estimation unit 180 and creates a composite image. For example, if the synthesized image is a three-dimensional reconstructed image, a synthesized image is created by synthesizing a three-dimensional shape and a focused image relating to each part of the three-dimensional shape. For example, if the synthesized image is an omnifocal image, images extracted from images having a focal position corresponding to the depth of each region are combined to synthesize an image focused on all the regions. The image composition unit 190 outputs the created composite image to a display unit or a storage device.

  For example, when a subject having a depth larger than the depth of field is captured using an optical system having a shallow depth of field, such as a microscope image, the image is not easily recognized by the user. On the other hand, according to the three-dimensional reconstructed image or the omnifocal image, an image of a subject having a depth larger than the depth of field can be easily recognized by the user.

  Thus, for example, the image acquisition unit 110 functions as an image acquisition unit that acquires a plurality of images captured at different focal positions for the same subject. For example, the band characteristic evaluation unit 130 functions as a band characteristic evaluation unit that calculates a band evaluation value of a band included in an image for each of a plurality of frequency bands with respect to a predetermined region in the plurality of images. For example, the statistical information calculation unit 140 functions as a statistical information calculation unit that calculates statistical information at least for each frequency band using band evaluation values of at least two focal positions. For example, the load coefficient calculation unit 150 functions as a load coefficient calculation unit that calculates the load coefficient for the band evaluation value at least for each frequency band based on the statistical information. For example, the contrast evaluation unit 160 functions as a contrast evaluation unit that calculates a contrast evaluation value for each region in a plurality of images based on the band evaluation value and the load coefficient. For example, the focus evaluation unit 170 functions as a focus evaluation unit that selects the focused region from the regions of the plurality of images based on the contrast evaluation value. For example, the image composition unit 190 functions as an omnifocal image creation unit or a three-dimensional reconstructed image creation unit.

  According to the present embodiment, the band characteristic evaluation unit 130 performs the filtering process, and the contrast evaluation unit 160 calculates the contrast evaluation value based on the band evaluation value obtained as a result. In general, when a filter having a high spectrum with respect to a high frequency is used, a contrast evaluation value indicating a more accurate contrast evaluation can be obtained. On the other hand, when the contrast evaluation value is calculated based on information having a frequency higher than the frequency band of the image, an inappropriate contrast evaluation value that evaluates a factor unrelated to the structure of the subject, such as noise, is obtained. . In the present embodiment, the statistical information calculation unit 140 calculates a statistical information value considering the frequency band of the image, and the load coefficient calculation unit 150 calculates a load coefficient based on the statistical information value. That is, the load coefficient is determined in consideration of the frequency band of the image. The contrast evaluation unit 160 determines the contrast evaluation value based on the band evaluation value calculated by the band characteristic evaluation unit 130 and the load coefficient calculated by the load coefficient calculation unit 150, and therefore does not consider the frequency band of the image. Compared to the case, a more accurate contrast evaluation value can be determined. As a result, the image processing system 100 can create a highly accurate three-dimensional reconstructed image or omnifocal image. The image processing system 100 is particularly effective when used for a microscope image taken by a microscope having a shallow depth of field.

  Next, specific examples of the statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 will be described.

[First example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the first example will be described. In this example, the statistical information value is the band evaluation value Q (k, f _n , i, j) at all focal positions (k = 1, 2,..., K) for each region and each frequency band. Average value. That is, the average value L (f _n , i, j) is calculated by the following formula (1).

The load coefficient is a value obtained by dividing the average value by the sum of the average values of all frequency bands for each region and each frequency band. That is, the load factor _{L N (f m, i,} j) is calculated by the following formula (2).

すなわち、コントラスト評価値Ｄ（ｋ，ｉ，ｊ）は、周波数帯域のそれぞれに係る帯域評価値Ｑ（ｋ，ｆ_ｎ，ｉ，ｊ）と荷重係数Ｌ_Ｎ（ｆ_ｍ，ｉ，ｊ）との積を足し合わせたものである。 Band evaluation value Q based on _(k, f n, i, j) and the load factor _{L N (f m, i,} j) and the contrast evaluation value D (k, i, j) is represented by the following formula (3) Is calculated by

That is, the contrast evaluation value D (k, i, j), the bandwidth evaluation value Q of the respective frequency bands _{(k, f n, i,} j) and the load factor _{L N (f m, i,} j) and the It is the sum of products.

  In step S107, the focus evaluation unit 170 estimates the depth information by selecting, for example, k having the highest contrast evaluation value D (k, i, j) for each region (i, j).

In this example, in calculating the load coefficient L _N (f _m , i, j), the average value is divided by the sum of the average values in all frequency bands as in the above equation (2). Alternatively, the average value may be obtained by dividing the average value by the sum of the average values of some frequency bands instead of the entire frequency band.

[Second example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the second example will be described. In this example, the statistical information value is an average value of the band evaluation values Q (k, f _n , i, j) at all focal positions for each region and each frequency band, as in the first example. is there. That is, the average value L (f _n , i, j) is calculated by the above equation (1). In this example, an average value L (f _n , i, j) is used as the load coefficient. Therefore, the contrast evaluation value D (k, i, j) is calculated by the following equation (4).

  In step S107, the focus evaluation unit 170 estimates the depth information based on the contrast evaluation value D (k, i, j).

ここで、帯域ｆ_０は、ｎ＝１〜Ｎのうち何れを用いてもよいが、例えば最も低い帯域が用いられる。コントラスト評価値Ｄ（ｋ，ｉ，ｊ）は、この荷重係数Ｌ_Ｎ（ｆ_ｍ，ｉ，ｊ）を用いて、上記式（３）で算出される。 [Third example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the third example will be described. In this example, the statistical information value is an average value of the band evaluation values Q (k, f _n , i, j) at all focal positions for each region and each frequency band, as in the first example. is there. That is, the average value L (f _n , i, j) is calculated by the above equation (1). In this example, the relative value of the average value L (f _n , i, j) with respect to the predetermined frequency band f ₀ is used as the load coefficient. That is, the weighting coefficient L _N (f _n , i, j) is calculated by the following equation (5).

Here, the band f ₀ may be any of n = 1 to N, but the lowest band is used, for example. Contrast evaluation value D (k, i, j) is the weighting factor _{L N (f m, i,} j) using, is calculated by the equation (3).

  In step S107, the focus evaluation unit 170 estimates the depth information based on the contrast evaluation value D (k, i, j).

ここで、閾値Ｔｈｒは、例えばＮ＝３の場合は０．２というように任意の設計値である。コントラスト評価値Ｄ（ｋ，ｉ，ｊ）は、第１の例と同様に上記式（３）で算出される。 [Fourth example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the fourth example will be described. In this example, the statistical information value is an average value of all focal positions for each region and each band, as in the first example. That is, the average value L (f _n , i, j) is calculated by the above equation (1). In this example, the load coefficient is 1 or 0 depending on whether or not a predetermined condition is satisfied. That is, whether or not to use the band evaluation value Q (k, f _n , i, j) is determined depending on whether or not the condition is satisfied. In this example, the load coefficient is calculated by the following equation (6).

Here, the threshold value Thr is an arbitrary design value such as 0.2 when N = 3, for example. The contrast evaluation value D (k, i, j) is calculated by the above equation (3) as in the first example.

In step S107, the focus evaluation unit 170 estimates the depth information based on the contrast evaluation value D (k, i, j). The determination in determining the load coefficient is not limited to using the average value L (f _n , i, j) divided by the sum of the average values of all frequency bands as in the above formula (6), but the average value L (F _n , i, j) itself or a value obtained by dividing the average value L (f _n , i, j) by the sum of the average values of arbitrary frequency bands or the average value of arbitrary frequency bands may be used.

According to the first to fourth examples, since the load coefficient L _n (f _m , i, j) is calculated for each region, it is particularly effective when the band characteristics are not constant for each region in the image. Further, according to these examples, the average value L (f _n , i, j) as the statistical information value is calculated for each frequency band. When the average value L (f _n , i, j) is low, information necessary for evaluating the contrast is not included in the band evaluation value Q (k, f _n , i, j), or the band evaluation It is conceivable that noise is included in the value Q (k, f _n , i, j). According to the first to fourth examples, the weight of the band evaluation value Q (k, f _n , i, j) that does not include such necessary information or includes noise is reduced. Thus, it is possible to prevent the band evaluation value Q (k, f _n , i, j) that does not include necessary information from affecting the contrast evaluation value. As a result, a highly accurate band evaluation value Q (k, f _n , i, j) is generated, and based on this band evaluation value Q (k, f _n , i, j), estimation of depth information with high accuracy is performed. Is realized.

[Fifth Example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the fifth example will be described. In this example, the statistical information value is the band evaluation value Q (k, f _n , i, j) at all focal positions (k = 1, 2,..., K) for each region and each frequency band. It is a fluctuation amount. In this example, variance is used as an example of the variation. That is, the variance S (f _n , i, j) is calculated by the following equation (7) using, for example, the average value L (f _n , i, j) calculated by the above equation (1).

The load coefficient is a value obtained by dividing the variance for each region and each frequency band by the sum of the variances for all frequency bands. That is, weighting factors _{S N (f m, i,} j) is calculated by the following equation (8).

The contrast evaluation value D (k, i, j) is calculated by the following equation (9) using the band evaluation value Q (k, f _n , i, j) and the load coefficient S _N (f _m , i, j). Is done.

In step S107, the focus evaluation unit 170 estimates the depth information based on the contrast evaluation value D (k, i, j). In this example, in calculating the load coefficient S _N (f _m , i, j), the average value is divided by the sum of the average values in all frequency bands as in the above equation (8). Alternatively, the average value may be obtained by dividing the average value by the sum of the average values of some frequency bands instead of the entire frequency band.

[Sixth example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the sixth example will be described. In this example, the statistical information value is the average of the variances of the band evaluation values Q (k, f _n , i, j) at all focal positions for each region and each frequency band, as in the fifth example. Value. That is, the variance S (f _n , i, j) is calculated by the above equation (7). In this example, the variance S (f _n , i, j) is used as the load coefficient. Accordingly, the contrast evaluation value D (k, i, j) is calculated by the following equation (10).

  In step S107, the focus evaluation unit 170 estimates the depth information based on the contrast evaluation value D (k, i, j).

[Seventh example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the seventh example will be described. In this example, the statistical information value is the variance of the band evaluation values Q (k, f _n , i, j) at all focal positions for each region and each frequency band, as in the fifth example. . That is, the variance S (f _n , i, j) is calculated by the above equation (7). In this example, a relative value for the predetermined frequency band f ₀ of the variance S (f _n , i, j) is used as the load coefficient. That is, the load coefficient S _N (f _n , i, j) is calculated by the following equation (11).

Here, band f ₀ may use any of n = 1 to N. The contrast evaluation value D (k, i, j) is calculated by the above equation (9) using this load coefficient S _N (f _n , i, j).

  In step S107, the focus evaluation unit 170 estimates the depth information based on the contrast evaluation value D (k, i, j).

According to the fifth to seventh examples, since the load coefficient L _n (f _m , i, j) is calculated for each region, it is particularly effective when the band characteristics are not constant for each region in the image. Further, according to these examples, the variance S (f _n , i, j) as the statistical information value is calculated for each frequency band. When the variance S (f _n , i, j), that is, the amount of variation is low, information necessary for evaluating the contrast is not included in the band evaluation value Q (k, f _n , i, j), or The band evaluation value Q (k, f _n , i, j) includes noise, and it is considered that the fluctuation is relatively small. According to the fifth to seventh examples, the weight of the band evaluation value Q (k, f _n , i, j) that does not include such necessary information or includes noise is reduced. Thus, it is possible to prevent the band evaluation value Q (k, f _n , i, j) that does not include necessary information from affecting the contrast evaluation value. As a result, a highly accurate band evaluation value Q (k, f _n , i, j) is generated, and based on this band evaluation value Q (k, f _n , i, j), estimation of depth information with high accuracy is performed. Is realized.

Even in the case of using dispersion, the load coefficient may be set to 1 or 0 depending on whether or not a predetermined condition is satisfied as in the case of the fourth example. That is, whether or not to use the band evaluation value Q (k, f _n , i, j) is determined depending on whether or not the condition is satisfied. Even if it does in this way, the effect similar to the 5th thru | or 7th example is acquired.

[Eighth Example]
The statistical information value calculated in step S104, the load coefficient calculated in step S105, and the contrast evaluation value calculated in step S106 in the eighth example will be described. In the first to fourth examples, the average value L (f _n , i, j) is determined for each region as the statistical information value. On the other hand, in the eighth example, the statistical information value is an average value for the entire image A for each band. That is, the average value L (f _n ) is calculated by the following formula (12).

The load coefficient is a value obtained by dividing the average value L (f _n ) by the sum of the average value L (f _n ) of all frequency bands. That is, the load factor _L N _{(f m)} is calculated by the following equation (13).

The contrast evaluation value D (k, i, j) is calculated by the following formula (14) using the band evaluation value Q (k, f _n , i, j) and the load coefficient L _N (f _m ).

  In step S107, the focus evaluation unit 170 estimates the depth information based on the contrast evaluation value D (k, i, j).

According to this example, when the difference for each band characteristic is small, the amount of calculation is small, which is effective. Also in the case of this example, as shown in the above equation (13), it may be divided not by the entire frequency band but by the sum of the average values of some frequency bands or the average value of a specific frequency band. In the above formula (14), L (f _m ) may be used instead of L _N (f _m ). Further, L _N (f _m ) may take 1 or 0. Further, as in the fifth to seventh examples, an average of variances in the entire area A of the image may be used.

[Modification of First Embodiment]
A modification of the first embodiment will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. In the image processing system 100 according to this modification, the band processing unit 120 executes wavelet conversion instead of having a filter bank.

  In the wavelet transform, an original image as shown on the left in FIG. 5 is subjected to a filtering process having a specific direction, and images A and B after band separation as shown on the right in FIG. And C. Then, the filtering process having a specific directionality is again performed on the image obtained by reducing the residual image of the filter to obtain images D, E, and F. Thereafter, such processing is repeated, and images G, H and I, and images J, K, L and M are acquired. By performing such conversion processing, image data expressed in multi-resolution as shown on the right side of FIG. 5 is created. By such wavelet transform, an amount corresponding to a gain in a specific band is associated with each region of the image, as in the first embodiment.

  An example of processing in the image processing system 100 according to this modification is shown in the flowchart of FIG. In step S <b> 201, the image acquisition unit 110 acquires a plurality of images captured while changing the focal position of the same subject, and stores the images in the storage unit 114. In step S202, the band processing unit 120 performs wavelet conversion on a plurality of images having different focal positions stored in the storage unit 114. Band processing unit 120 outputs the conversion result to band characteristic evaluation unit 130. In step S203, the band characteristic evaluation unit 130 calculates an evaluation value for each region (p, q) of the plurality of images subjected to the wavelet transform. That is, the coefficient of the wavelet stage number n is set to the band evaluation value Q (k, n, p, q) for each region (p, q), that is, for each of the data I (k, p, q). . Band characteristic evaluation unit 130 outputs band evaluation value Q (k, n, p, q) to statistical information calculation unit 140.

ここで、Ｆ_ｎはウエーブレットの段数ｎのときの画像の大きさを示している。
統計情報算出部１４０は、算出した統計情報値を荷重係数算出部１５０に出力する。 In step S204, the statistical information calculation unit 140 calculates a statistical information value. In this modification, the average value of all focal positions k = 1, 2,..., K of the band evaluation values Q (k, n, p, q) of each band is defined as a statistical information value L (f _n ). That is, the statistical information value L (f _n ) is calculated by the following equation (15).

Here, F _n indicates the size of the image when the number of wavelets is n.
The statistical information calculation unit 140 outputs the calculated statistical information value to the load coefficient calculation unit 150.

荷重係数算出部１５０は、算出した荷重係数Ｌ_Ｎ（ｆ_ｍ）をコントラスト評価部１６０に出力する。 In step S205, the load coefficient calculation unit 150 calculates a load coefficient corresponding to the band based on the statistical information value L (f _n ) input from the statistical information calculation unit 140. In this modification, the load factor _L N _{(f m)} is calculated by the following equation (16).

The load coefficient calculation unit 150 outputs the calculated load coefficient L _N (f _m ) to the contrast evaluation unit 160.

In step S206, the contrast evaluating unit 160 corresponds to the band evaluation value Q (k, p, q) input from the band characteristic evaluating unit 130 and the corresponding load coefficient L _N (for each band input from the load coefficient calculating unit 150). The contrast evaluation value D (k, i, j) relating to each region (i, j) of the image before wavelet transform is calculated by multiplying f _m ) and performing inverse transform. The contrast evaluation unit 160 outputs the calculated contrast evaluation value D (k, i, j) to the focus evaluation unit 170.

  Thereafter, as in the first embodiment, the focus evaluation unit 170 evaluates the focus based on the contrast evaluation value D (k, i, j) in step S207, and sets the depth information regarding each pixel in a 3D shape. It outputs to the estimation part 180. In step S208, the 3D shape estimation unit 180 optimizes depth information such as smoothing based on the depth information, estimates the three-dimensional shape of the subject, and sends the estimated three-dimensional shape of the subject to the image composition unit 190. Output. In step S209, the image composition unit 190 composes a plurality of images with different focal positions based on the three-dimensional shape of the subject and the plurality of images to create a composite image.

  Also by this modification, the same effect as the first embodiment can be obtained.

[Second Embodiment]
A second embodiment of the present invention will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. The present embodiment is a microscope system 200 including the image processing system 100 according to the first embodiment.

  An outline of a configuration example of the microscope system 200 according to the present embodiment is shown in FIG. As shown in this figure, the microscope system 200 includes a microscope 210 and the image processing system 100 according to the first embodiment. The microscope 210 is a digital microscope, for example. The microscope 210 includes an LED light source 211, an illumination optical system 212, an optical path control element 213, an objective lens 214, a sample surface 215 placed on a stage (not shown), an observation optical system 218, and an imaging surface 219. , An imaging unit 220 and a controller 222. The observation optical system 218 includes a zoom optical system 216 and an imaging optical system 217. In the observation optical path, an objective lens 214, an optical path control element 213, a zoom optical system 216, and an imaging optical system 217 are arranged in this order from the sample surface 215 to the imaging surface 219.

  The illumination light emitted from the LED light source 211 enters the optical path control element 213 via the illumination optical system 212. The optical path control element 213 reflects the illumination light toward the objective lens 214 on the observation optical path. The illumination light is applied to the sample disposed on the sample surface 215 via the objective lens 214.

  When the sample is irradiated with illumination light, observation light is generated from the sample. Here, the observation light is reflected light, fluorescence, or the like. The observation light is incident on the optical path control element 213. Unlike the case of illumination light, the optical path control element 213 transmits observation light and makes the observation light incident on an observation optical system 218 having a zoom optical system 216 and an imaging optical system 217. Thus, the optical path control element 213 is an optical element that reflects or transmits incident light according to the characteristics of the incident light. For the optical path control element 213, for example, a polarization element such as a wire grid or a polarization beam splitter (PBS) that reflects or transmits incident light according to the polarization direction of the incident light can be used. The optical path control element 213 may use, for example, a dichroic mirror that reflects or transmits incident light according to the frequency of incident light.

  The observation optical system 218 collects observation light on the imaging surface 219 and forms an image of the sample on the imaging surface 219. The imaging unit 220 generates an image signal based on the image formed on the imaging surface 219 and outputs the image signal to the image acquisition unit 110 as a microscope image. The controller 222 controls the operation of the microscope 210. In the present embodiment, the microscope 210 acquires a plurality of microscope images captured at different focal planes for the same specimen. For this reason, the controller 222 controls the optical system of the microscope 210 to cause the imaging unit 220 to capture an image of the sample on each focal plane while gradually changing the focal plane. Specifically, for example, the controller 222 causes the imaging unit 220 to capture an image while changing the height of the stage of the microscope 210 or the position of the height of the objective lens. The controller 222 outputs information related to the focal position to the image acquisition unit 110 in association with the image captured by the imaging unit 220.

  An operation of the microscope system 200 according to the present embodiment will be described. The sample is placed on a stage (not shown) so as to have a sample surface 215. The controller 222 controls the microscope 210. For example, the controller 222 gradually changes the position of the sample surface 215 in the optical axis direction, and gradually changes the focal position of the optical system with respect to the sample. Specifically, for example, the controller 222 changes the height of the stage of the microscope 210, the height of the objective lens, or the position of the focus lens. At this time, the controller 222 causes the imaging unit 220 to sequentially capture microscopic images of the specimen at each focal position. The image acquisition unit 110 acquires a microscopic image of the specimen at each focal position from the imaging unit 220. Further, the image acquisition unit 110 acquires the focal position when each image is captured from the controller 222. The image acquisition unit 110 stores the acquired microscope image in the storage unit 114 in association with the focal position.

  The process of creating a composite image by combining a plurality of images with different focal positions based on the microscope image stored in the storage unit 114 is the same as the process of the first embodiment. In the present embodiment, the microscope system 200 creates a composite image related to a microscope image, such as a three-dimensional reconstructed image or an omnifocal image. The image composition unit 190 outputs the created composite image, for example, to a display unit for display or to a storage device for storage. According to the three-dimensional reconstructed image and the omnifocal image, an image of a subject having a depth larger than the depth of field like a general microscope image can be easily recognized by the user.

  Thus, for example, the illumination optical system 212, the optical path control element 213, the objective lens 214, the observation optical system 218, and the like function as a microscope optical system. For example, the imaging unit 220 functions as an imaging unit that acquires a specimen image through a microscope optical system as a specimen image.

  In general, an optical system of a microscope has a larger image enlargement ratio than an optical system of a digital camera. For this reason, in microscope photography, the bandwidth of the optical system of the microscope may not be so high with respect to the sampling bandwidth of the camera image sensor. Further, the band of the optical system can change depending on the numerical aperture, magnification, and the like of the optical system. For example, when the microscope has a zoom optical system, the band of the optical system changes. According to the present embodiment, the statistical information calculation unit 140 calculates a statistical information value in consideration of the frequency band of the image, and the load coefficient calculation unit 150 calculates a load coefficient based on the statistical information value. That is, since the contrast evaluation unit 160 determines the contrast evaluation value based on the evaluation value calculated by the band characteristic evaluation unit 130 and the load coefficient calculated in consideration of the frequency band of the image, an accurate contrast evaluation value is obtained. Can be determined. As a result, the microscope system 200 can create an accurate three-dimensional reconstructed microscope image and an all-focus microscope image. In the case where the optical system of the microscope 110 includes a zoom optical system, the numerical aperture changes in accordance with the focal length of the zoom optical system and the band of the microscope image changes, so this embodiment is particularly effective.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the problem described in the column of problems to be solved by the invention can be solved and the effect of the invention can be obtained. The configuration in which this component is deleted can also be extracted as an invention. Furthermore, constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 100 ... Image processing system, 110 ... Image acquisition part, 114 ... Memory | storage part, 115 ... Sample surface, 120 ... Band processing part, 121 ... 1st filter, 122 ... 2nd filter, 123 ... 3rd filter, 130 DESCRIPTION OF SYMBOLS ... Band characteristic evaluation part 140 ... Statistical information calculation part 150 ... Load coefficient calculation part 160 ... Contrast evaluation part 170 ... Focus evaluation part 180 ... 3D shape estimation part 190 ... Image composition part 200 ... Microscope system , 210 ... Microscope, 211 ... LED light source, 212 ... Illumination optical system, 213 ... Optical path control element, 214 ... Objective lens, 215 ... Sample surface, 216 ... Zoom optical system, 217 ... Imaging optical system, 218 ... Observation optical system 219 ... Imaging surface, 220 ... Imaging unit, 222 ... Controller.

Claims (13)

An image acquisition unit that acquires a plurality of images captured at different focal positions for the same subject;
A band characteristic evaluation unit that calculates a band evaluation value of a band included in the image for each of a plurality of frequency bands for a predetermined region in the plurality of images,
A statistical information calculation unit that calculates statistical information for at least each frequency band using the band evaluation values of at least two focal positions;
Based on the statistical information, a load coefficient calculation unit that calculates a load coefficient for the band evaluation value at least for each frequency band;
A contrast evaluation unit that calculates a contrast evaluation value for each of the regions in the plurality of images based on the band evaluation value and the load coefficient;
A focus evaluation unit that selects the region in focus among the regions of each of the plurality of images based on the contrast evaluation value;
An image processing system comprising:   The image processing system according to claim 1, wherein the band evaluation value is an amount corresponding to an amplitude in each of the frequency bands. The statistical information calculation unit further calculates the statistical information for each region,
The load coefficient calculation unit further calculates the load coefficient for each region,
The contrast evaluation unit calculates the contrast evaluation value based on the band evaluation value for each region and the load coefficient for each region.
The image processing system according to claim 1, wherein the image processing system is an image processing system.   4. The image processing system according to claim 1, wherein the statistical information is an average value of the band evaluation values corresponding to each of the plurality of images having different focal positions. 5. .   The statistical information is a relative value obtained by dividing an average value of the corresponding band evaluation values of each of the plurality of images having different focal positions by a sum of the average values for at least one of the frequency bands. The image processing system according to claim 1, wherein the image processing system is one of the first to third aspects.   The statistical information is a relative value obtained by dividing the average value of the corresponding band evaluation values of each of the plurality of images having different focal positions by the sum of the average values for all the frequency bands. The image processing system according to any one of claims 1 to 3.   4. The image processing system according to claim 1, wherein the statistical information is a fluctuation amount of the band evaluation value corresponding to each of the plurality of images having different focal positions. 5. .   The statistical information is a relative value obtained by dividing the variation amount of the corresponding band evaluation value of each of the plurality of images having different focal positions by the sum of the variation amounts for at least one of the frequency bands. The image processing system according to claim 1, wherein the image processing system is one of the first to third aspects.   The statistical information is a relative value obtained by dividing the variation amount of the corresponding band evaluation value of each of the plurality of images having different focal positions by the sum of the variation amounts for all the frequency bands. The image processing system according to any one of claims 1 to 3.   2. The omnifocal image creation unit that creates an omnifocal image based on the image corresponding to each of the focused areas selected by the focus evaluation unit. 10. The image processing system according to any one of items 9 to 9.   Three-dimensional reconstructed image based on the image corresponding to each of the focused areas selected by the focus evaluation unit and the focal position corresponding to each of these areas The image processing system according to claim 1, further comprising a reconstructed image creating unit. Microscope optics,
An imaging unit that obtains an image of a specimen through the microscope optical system as a specimen image;
The image processing system according to any one of claims 1 to 11, wherein the sample image is acquired as the image.
A microscope system comprising:   The microscope system according to claim 12, wherein the optical system includes a variable magnification optical system.

Images