JP5210571B2 - Image processing apparatus, image processing program, and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus , an image processing program, and an image processing method for estimating a spectral characteristic of a subject using Wiener estimation based on a pixel value of a target image obtained by illuminating illumination light and imaging the subject.

  One of the physical quantities representing physical properties inherent to the subject is a spectral transmittance spectrum. Spectral transmittance is a physical quantity that represents the ratio of transmitted light to incident light at each wavelength. Unlike color information that depends on changes in illumination light such as RGB values, object-specific information whose value does not change due to external influences. It is. For this reason, the spectral transmittance is used in various fields as information for reproducing the color of the subject itself. For example, in the field of pathological diagnosis using a biological tissue specimen, particularly a pathological specimen, a spectral transmittance estimation technique is used to analyze an image obtained by imaging the specimen.

  In pathological diagnosis, block specimens obtained by organ excision and pathological specimens obtained by needle biopsy are sliced to several microns in thickness, and are then widely observed using a microscope to obtain various findings. ing. In particular, transmission observation using an optical microscope is one of the most popular observation methods because the equipment is relatively inexpensive and easy to handle, and has been performed for a long time. In this case, since the sliced specimen hardly absorbs and scatters light and is nearly colorless and transparent, it is general to dye with a dye prior to observation.

  Various dyeing methods have been proposed, and the total number thereof reaches 100 or more. Particularly, regarding pathological specimens, hematoxylin-eosin staining using two of blue-violet hematoxylin and red eosin as pigments ( Hereinafter, “H & E staining” is used as a standard.

  Hematoxylin is a natural substance collected from plants and itself has no dyeability. However, its oxide, hematin, is a basophilic dye and binds to a negatively charged substance. Since deoxyribonucleic acid (DNA) contained in the cell nucleus is negatively charged by a phosphate group contained as a constituent element, it binds to hematin and is stained blue-violet. As described above, it is not hematoxylin that has a staining property but hematin, which is an oxide thereof. However, since it is common to use hematoxylin as a name of a pigment, the following is followed.

  Eosin is an acidophilic dye that binds to positively charged substances. Whether amino acids or proteins are charged positively or negatively is affected by the pH environment, and the tendency to be positively charged under acidic conditions becomes stronger. For this reason, acetic acid may be added to the eosin solution. Proteins contained in the cytoplasm are stained from red to light red by binding to eosin.

  In a specimen (stained specimen) after H & E staining, cell nuclei, bone tissue, etc. are stained blue-purple, and cytoplasm, connective tissue, erythrocytes, etc. are stained red, so that they can be easily visually recognized. As a result, the observer can grasp the size and positional relationship of the elements constituting the tissue such as the cell nucleus and can morphologically determine the state of the stained specimen.

  The stained specimen is observed not only by an observer's visual observation, but also by displaying this stained specimen on a display screen of an external device after performing multiband imaging. When displaying on the display screen, processing to estimate the spectral transmittance of each point of the sample from the captured multiband image, and processing to estimate the amount of dye that is staining the sample based on the estimated spectral transmittance Then, processing for correcting the color of the image based on the estimated amount of pigment is performed, and variations in camera characteristics, staining state, and the like are corrected, and an RGB image of the specimen for display is synthesized. FIG. 15 is a diagram illustrating an example of a combined RGB image. If the amount of pigment is appropriately estimated, it is possible to correct a darkly stained sample or a lightly stained sample to an image having a color equivalent to that of a properly stained sample. Therefore, estimating the spectral transmittance of the stained specimen with high accuracy leads to higher accuracy such as estimation of the amount of dye fixed to the stained specimen and correction of staining variation.

  As a method of estimating the spectral transmittance of each point of the sample from the multiband image of the sample, for example, an estimation method by principal component analysis (for example, refer to Non-Patent Document 1) or an estimation method by Wiener estimation (for example, Non-Patent Document 2).

  Wiener estimation is widely known as one of the linear filter methods for estimating the original signal from the observation signal with superimposed noise, and minimizes the error by taking into account the statistical properties of the observation target and the characteristics of the observation noise. It is a technique to do. Since some noise is included in the signal from the camera, the Wiener estimation is extremely useful as a method for estimating the original signal.

  Here, a method for estimating the spectral transmittance of each point of the sample from the multiband image of the sample by the Wiener estimation method will be described.

  First, a multiband image of a specimen is taken. For example, using the technique disclosed in Patent Document 1, a multiband image is picked up in a frame sequential manner while 16 band pass filters are switched by rotating with a filter wheel. Thereby, a multiband image having 16-band pixel values at each point of the sample is obtained. Dye is originally distributed three-dimensionally in a stained specimen that is the object of observation, but it cannot be captured as a three-dimensional image as it is in a normal transmission observation system, and illumination light transmitted through the specimen is taken as a camera. It is observed as a two-dimensional image projected on the image sensor. Therefore, each point here means a point on the sample corresponding to each pixel of the projected image sensor.

λは波長、ｆ（ｂ，λ）はｂ番目のフィルタの分光透過率、ｓ（λ）はカメラの分光感度特性、ｅ（λ）は照明の分光放射特性、ｎ（ｂ）はバンドｂにおける撮像ノイズをそれぞれ表す。ｂはバンドを識別する通し番号であり、ここでは１≦ｂ≦１６を満たす整数値である。 The camera response system between the pixel value g (x, b) in the band b and the spectral transmittance t (x, λ) of the corresponding point on the sample for the position x of the captured multiband image. The following equation (1) is established.

λ is the wavelength, f (b, λ) is the spectral transmittance of the b-th filter, s (λ) is the spectral sensitivity characteristic of the camera, e (λ) is the spectral radiation characteristic of the illumination, and n (b) is in the band b. Each represents imaging noise. b is a serial number for identifying a band, and here is an integer value satisfying 1 ≦ b ≦ 16.

In actual calculation, the following formula (2) obtained by discretizing the formula (1) in the wavelength direction is used.
G (x) = FSET (x) + N (2)
If the number of sample points in the wavelength direction is D and the number of bands is B (here, B = 16), G (x) is a matrix of B rows and 1 column corresponding to the pixel value g (x, b) at the position x. is there. Similarly, T (x) is a D × 1 matrix corresponding to t (x, λ), and F is a B × D matrix corresponding to f (b, λ). On the other hand, S is a diagonal matrix of D rows and D columns, and the diagonal elements correspond to s (λ). Similarly, E is a diagonal matrix of D rows and D columns, and the diagonal element corresponds to e (λ). N is a matrix of B rows and 1 column corresponding to n (b). In Expression (2), since the expressions related to a plurality of bands are aggregated using a matrix, the variable b representing the band is not explicitly described. In addition, the integration with respect to the wavelength λ is replaced with a matrix product.

Here, in order to simplify the notation, a matrix H defined by the following equation (3) is introduced. H is also called a system matrix.
H = FSE (3)

Next, the spectral transmittance at each point of the sample is estimated from the captured multiband image using Wiener estimation. The estimated value T ^ (x) of the spectral transmittance can be calculated by the following equation (4). T ^ indicates that a hat (^) representing an estimated value is attached on T.

Ｒ_SSは、Ｄ行Ｄ列の行列であり、標本の分光透過率の自己相関行列を表す。Ｒ_NNは、Ｂ行Ｂ列の行列であり、撮像に使用するカメラのノイズの自己相関行列を表す。このように、ウィナー推定行列は、システム行列Ｈと、観測対象の統計的性質を表す項Ｒ_SSと、観測ノイズの特性を表す項Ｒ_NNから構成され、それぞれの特性を高精度に表すことが、分光透過率の推定精度の向上に繋がる。 Here, W is called a Wiener estimation matrix and is expressed by the following equation (5).

R _SS is a matrix of D rows and D columns, and represents an autocorrelation matrix of the spectral transmittance of the sample. R _NN is a matrix of B rows and B columns, and represents an autocorrelation matrix of camera noise used for imaging. As described above, the Wiener estimation matrix is composed of the system matrix H, the term R _SS indicating the statistical property of the observation target, and the term R _NN indicating the characteristic of the observation noise, and each characteristic can be expressed with high accuracy. This leads to an improvement in estimation accuracy of spectral transmittance.

JP-A-7-120324 “Development of support systems for pathology using spectral transmittance-The quantification method of stain conditions”, Proceedings of SPIE, Vol.4684, 2002, p.1516-1523 “Color Correction of Pathological Images Based on Dye Amount Quantification”, OPTICAL REVIEW, Vol.12, No.4, 2005, p.293-300

When estimating the spectral transmittance of each sample point from a multiband image using Wiener estimation according to the method disclosed in Non-Patent Document 1, based on the relationship of the camera response system shown in Equation (1). , The spectral transmittance F of the optical filter shown in the expression (3), the spectral sensitivity characteristic S of the camera, and the spectral emission characteristic E of the illumination, and the autocorrelation matrix R _SS representing the statistical properties of the observation target shown in the expression (5) _{, And} each spectral characteristic estimation parameter of the camera noise autocorrelation matrix R _NN must be acquired in advance. Here, the camera noise autocorrelation matrix R _NN acquires a multiband image without a sample, obtains a variance of pixel values for each band of the obtained multiband image, and uses this as a diagonal component. It is obtained by generating a matrix and represents the observed noise characteristics.

  By the way, among the spectral characteristic estimation parameters, the spectral radiation characteristic E of illumination is a measured value obtained by measuring the spectral radiation characteristic of illumination in a predetermined range region (measurement region) composed of a plurality of pixels using a spectrometer. Is used as a representative value. That is, the measurement value of the spectral radiation characteristic of illumination in a predetermined measurement region is uniformly applied to pixels within the same field of view. However, illumination light has non-uniform illumination radiation distribution, and it is technically difficult to irradiate the same field of view with uniform illumination intensity.

In order to solve this problem, conventionally, as a pre-processing when measuring the spectral radiation characteristics of illumination, shading correction to the measurement center of the measurement area is performed to make the illumination radiation distribution non-uniformity uniform. After that, the spectral radiation characteristics of the illumination were measured by a spectrometer. In addition, a correction coefficient is obtained from the pixel value of each pixel obtained by capturing an image without a specimen, and the multiband image of the specimen is obtained by performing shading correction processing on the multiband image of the specimen using the correction coefficient. It can be converted into image data with uniform brightness. However, the shading correction process is a so-called gain adjustment process for making the illumination radiation distribution of the illumination light uniform, and has an effect of adjusting the gain for the observation noise included in each pixel. For this reason, there is a problem that the observation noise of each pixel increases or decreases with the shading correction processing. Therefore, the observation noise of each pixel after the shading correction processing may be different from the camera noise autocorrelation matrix R _NN acquired in advance as the spectral characteristic estimation parameter, which causes a problem that the estimation accuracy of the spectral transmittance decreases. was there.

  The present invention has been made in view of the above-described conventional problems, reduces the estimation error of the spectral characteristic value caused by the change in the characteristic of the observation noise characteristic accompanying the shading correction process, and estimates the spectral characteristic of the subject. An object is to provide an image processing apparatus and an image processing program capable of improving accuracy.

To solve the above problems and achieve an object, an image processing apparatus according to the present invention estimates the spectral characteristic before Symbol object based on the pixel values of the target image obtained by imaging a subject by irradiating the illuminating light In the image processing apparatus, a correction coefficient calculation unit that calculates a correction coefficient when correcting a pixel value based on a pixel value of an illumination image obtained by imaging a background in a state where the illumination light is irradiated; and the correction coefficient calculation unit Correction means for performing a shading correction process on the target image, and an observation noise characteristic of a pixel constituting the target image subjected to the shading correction process as the correction coefficient for the pixel. and noise characteristic determination means for determining based on the noise characteristics the observation noise characteristics determined by the determining means calculates based on estimation matrix, the calculated estimated matrix There are characterized by and a spectral characteristic estimation means for estimating a spectral characteristic values of the object position corresponding to the pixel.

The image processing program according to the present invention, the computer, the pixel values of the target image by irradiating illuminating light to image a subject an image processing program for estimating a spectral characteristic before Symbol subject based on The correction coefficient calculation step for calculating a correction coefficient when correcting the pixel value based on the pixel value of the illumination image obtained by imaging the background with the illumination light applied, and the correction coefficient calculation step. A correction step for performing a shading correction process on the target image using the correction coefficient, and an observation noise characteristic of a pixel constituting the target image subjected to the shading correction process are determined based on the correction coefficient for the pixel. a noise characterization step, calculated based on estimate matrix the observation noise characteristics determined by the noise characterization step , Characterized in that to execute, the spectral characteristic estimation step of estimating the spectral characteristic values of the object position corresponding to the pixel by using the calculated estimated matrix.

  According to the present invention, it is possible to estimate the spectral characteristics of a subject using observation noise characteristics that take into account the influence of shading correction processing. Therefore, it is possible to reduce the estimation error of the spectral characteristic value caused by the characteristic change of the observation noise characteristic accompanying the shading correction process, and to improve the estimation accuracy of the spectral characteristic of the subject.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case will be described in which a spectral characteristic value of spectral transmittance is estimated as a spectral characteristic value from a multiband image obtained by imaging a pathological specimen using an H & E stained pathological specimen as a subject.

  FIG. 1 is a schematic diagram illustrating the configuration of the image processing apparatus according to the present embodiment. As shown in FIG. 1, the image processing apparatus 1 includes an image acquisition unit 110 and is connected to a computer 10 such as a personal computer.

  The image acquisition unit 110 captures an H & E-stained specimen for spectral transmittance estimation (hereinafter referred to as “target specimen”) and acquires a six-band multiband image. The image acquisition unit 110 includes an RGB camera 111 including an imaging device such as a CCD, a sample holding unit 113 on which the target sample S is placed, an illumination unit 115 that transmits and illuminates the target sample S on the sample holding unit 113, An optical system 117 for condensing the transmitted light from the target sample S to form an image, a filter unit 119 for limiting the wavelength band of the imaged light to a predetermined range, and the like are provided.

  The RGB camera 111 is widely used in digital cameras and the like, and has RGB color filters arranged in a mosaic pattern on a monochrome image sensor. The RGB camera 111 is installed so that the center of the image to be captured is positioned on the optical axis of the illumination light. FIG. 2 is a diagram schematically illustrating a color filter array example and a pixel array of each RGB band. In this case, each pixel can image only one of R, G, and B components, but by using neighboring pixel values, the insufficient R, G, and B components are interpolated. This technique is disclosed in, for example, Japanese Patent No. 3510037. If a 3CCD type camera is used, R, G and B components in each pixel can be acquired from the beginning. In this embodiment, any imaging method may be used. In the following, it is assumed that R, G, and B components can be acquired in each pixel of an image captured by the RGB camera 111.

  The filter unit 119 includes two optical filters 1191 a and 1191 b each having different spectral transmittance characteristics, and these are held by a rotary optical filter switching unit 1193. 3A is a diagram illustrating a spectral transmittance characteristic of one optical filter 1191a, and FIG. 3B is a diagram illustrating a spectral transmittance characteristic of the other optical filter 1191b. For example, first, first imaging is performed using the optical filter 1191a. Next, the optical filter to be used is switched to the optical filter 1191b by the rotation of the optical filter switching unit 1193, and the second imaging is performed using the optical filter 1191b. By the first imaging and the second imaging, a 3-band image is obtained, and a 6-band multiband image is obtained by combining both results. Note that the number of optical filters is not limited to two, and three or more optical filters can be used. The acquired multiband image is held in the storage unit 140 described later as a target specimen image.

  In the image acquisition unit 110, the illumination light irradiated by the illumination unit 115 passes through the target sample S placed on the sample holding unit 113. Then, the transmitted light that has passed through the target specimen S passes through the optical system 117 and the optical filters 1191a and 1191b, and then forms an image on the image sensor of the RGB camera 111. The filter unit 119 including the optical filters 1191 a and 1191 b may be installed at any position on the optical path from the illumination unit 115 to the RGB camera 111. FIG. 4 shows an example of the spectral sensitivity of each of the R, G, and B bands when the illumination light from the illumination unit 115 is imaged by the RGB camera 111 via the optical system 117.

  In the present embodiment, the image acquisition unit 110 is used to capture an illumination image, which is a reflected image of illumination light, by capturing an image of a background without a sample in a state where illumination light is irradiated. The acquired illumination image is held in the storage unit 140. This illumination image can be acquired by taking an image without placing the sample on the sample holding unit 113. Note that when acquiring an illumination image, the optical filters 1191a and 1191b are not necessary, and thus, for example, the filter unit 119 is removed to perform imaging.

  FIG. 5 is a block diagram illustrating a functional configuration of the image processing apparatus 1. In the present embodiment, the image processing apparatus 1 controls the image acquisition unit 110, the input unit 120, the display unit 130, the storage unit 140, the image processing unit 150, and each unit described with reference to FIG. And a control unit 160.

  The input unit 120 is realized by, for example, a keyboard, a mouse, a touch panel, various switches, and the like, and outputs an operation signal corresponding to the operation input to the control unit 160.

  The display unit 130 is realized by a display device such as an LCD or an ELD, and displays various screens based on a display signal input from the control unit 160.

  The storage unit 140 is realized by various IC memories such as ROM and RAM such as flash memory that can be updated and stored, information storage media such as a built-in or data communication terminal, a hard disk connected by a data communication terminal, a CD-ROM, and a reading device thereof. A program relating to the operation of the image processing apparatus 1, a program for realizing various functions of the image processing apparatus 1, data relating to execution of these programs, and the like are stored. For example, the illumination image, the image data of the target specimen image, the region division result of the correction coefficient image, and the like are stored. Further, the observation noise characteristic for each divided area is determined based on the pixel value of the correction coefficient image created based on the shading correction coefficient data 141, the observation noise characteristic data 143 for each divided area, and the illumination image, and determined. An image processing program 145 for estimating the spectral characteristics of the target specimen using the observed noise characteristics is stored. The shading correction coefficient data 141 stores the correction coefficient of each pixel constituting the illumination image as image data of a correction coefficient image obtained by imaging the correction coefficient of each pixel as a pixel value. The observation noise characteristic data 143 for each divided area stores observation noise characteristics for each divided area.

The image processing unit 150 is realized by hardware such as a CPU. The image processing unit 150 includes a correction coefficient calculation unit 151 as a correction coefficient calculation unit, a distribution data generation unit 152 as a distribution image generation unit and a coefficient distribution calculation unit, an area division unit, a division determination unit, and a gradation conversion division. A region dividing unit 153 as a means, a binarizing unit and a labeling processing unit, a representative point extracting unit 154 as a representative point extracting unit, an observation noise characteristic determining unit 155 as a noise characteristic determining unit and a noise calculating unit, and correction A shading correction unit 156 as a means and a spectral characteristic estimation unit 157 as a spectral characteristic estimation means are included. The correction coefficient calculation unit 151 calculates a correction coefficient for each pixel based on the pixel value of the illumination image. The distribution data creation unit 152 creates correction coefficient distribution data for each pixel calculated by the correction coefficient calculation unit 151. Specifically, a correction coefficient image is created based on the correction coefficient of each pixel, and a histogram that is a correction coefficient distribution chart is created. The area dividing unit 153 divides the correction coefficient image into a plurality of divided areas. The representative point extracting unit 154 extracts one or a plurality of representative points (divided region representative points) from each divided region divided by the region dividing unit 153. The observation noise characteristic determination unit 155 determines the observation noise characteristic in each divided area based on the noise autocorrelation matrix R _NN of the RGB camera 111 as the reference noise amount and the correction coefficient α at the divided area representative point. The shading correction unit 156 performs a shading correction process on the target sample image using the correction coefficient of each pixel calculated by the correction coefficient calculation unit 151. The spectral characteristic estimation unit 157 estimates the spectral transmittance of the target sample point corresponding to the predetermined pixel constituting the target sample image using the observation noise characteristic at the divided region representative point of the divided region to which the predetermined pixel belongs. .

  The control unit 160 is realized by hardware such as a CPU. The control unit 160 is connected to each unit constituting the image processing apparatus 1 based on an operation signal input from the input unit 120, image data input from the image acquisition unit 110, a program and data stored in the storage unit 140, and the like. Instruction, data transfer, etc., and overall operation of the image processing apparatus 1 is controlled. In addition, the control unit 160 controls the operation of the image acquisition unit 110 to acquire an illumination image, and the specimen image acquisition that acquires the target specimen image by controlling the operation of the image acquisition unit 110. And a control unit 163.

  Next, a processing procedure in the image processing apparatus 1 will be described. The image processing apparatus 1 according to the present embodiment includes a process for calculating an observation noise characteristic for each divided region (hereinafter referred to as “observation noise characteristic calculation process”) and a process for estimating a spectral characteristic of a target sample (hereinafter, referred to as “observation noise characteristic calculation process”). (Referred to as “spectral characteristic estimation processing”). Note that the processing described here is realized by the operation of each unit of the image processing apparatus 1 according to the image processing program 145 stored in the storage unit 140.

  First, the procedure of the observation noise characteristic calculation process will be described. FIG. 6 is a flowchart showing the procedure of the observation noise characteristic calculation process. In this observation noise characteristic calculation processing, first, the illumination image acquisition control unit 161 controls the operation of the image acquisition unit 110 to acquire an illumination image obtained by capturing a background without a subject in a state where illumination light is irradiated (step S101). ).

Next, the correction coefficient calculation unit 151 calculates a correction coefficient for each pixel based on the illumination image (step S103). The calculated correction coefficient for each pixel is stored in the storage unit 140. Here, the spectral emission characteristic E of illumination, which is one of the spectral characteristic estimation parameters, is obtained by measuring the spectral emission characteristic of illumination in a predetermined measurement region using a spectrometer, and the correction coefficient is the predetermined coefficient. It is calculated based on the pixel value of the point X ₀ which is the measurement center of the measurement region. Here, assuming that the pixel value of the point X ₀ is W ₀ and the pixel value of the arbitrary point X is W _x , the correction coefficient α _x of the point X is expressed by the following equation (6).

Subsequently, the distribution data creation unit 152 creates a correction coefficient image obtained by imaging the correction coefficient α _x of each pixel calculated in step S103 as a pixel value, and stores the correction coefficient image as shading correction coefficient data 141 in the storage unit 140 (step S100). S105). The distribution data creation unit 152 creates a histogram based on the correction coefficient image, with the horizontal axis representing the correction coefficient α and the vertical axis representing the number of pixels (step S107). The generated histogram data is stored in the storage unit 140. FIG. 7 shows an example of the created histogram. If the histogram is created, the region dividing unit 153 determines whether or not to execute the region dividing process according to the distribution width of the pixel values obtained from the histogram. That is, the region dividing unit 153 calculates the distribution width of the pixel values of the correction coefficient image from the created histogram (step S109). Then, the region dividing unit 153 compares the calculated distribution width with a reference distribution width set as a threshold value in advance, and determines whether or not to execute the region dividing process.

If the calculated distribution width is smaller than the reference distribution width (step S111: No), the region dividing unit 153 determines not to execute the region dividing process. In this case, the area dividing unit 153 sets the entire correction coefficient image as one divided area, and, for example, sets the measurement center X ₀ of the measurement area of the spectrometer when the spectral radiation characteristic E of illumination is acquired as the divided area representative. As a point, the process proceeds to step S115.

  On the other hand, if the distribution width is equal to or larger than the reference distribution width (step S111: Yes), the region dividing unit 153 proceeds to step S113 and executes region dividing processing. It should be noted that the histogram created in step S107 may be visually presented to the user, and it may be determined whether or not to execute the region division process according to the user operation. In this case, the control unit 160 performs control to display the histogram created by the region dividing unit 153 as illustrated in FIG. 7 on the display unit 130 and determines whether to divide the image into divided regions. It performs control to display the notification of the request and functions as a coefficient distribution display control unit and a determination input request unit. FIG. 8 is a diagram illustrating an example of a notification screen for determining whether to divide a correction coefficient image. On the notification screen W10, a message for requesting determination of whether or not to perform division is displayed, and buttons B11 and B13 for selecting whether to perform division or not are arranged. The user performs a pressing operation of the button B11 or the button B13 via the input unit 120, and inputs a division instruction or a non-division instruction. In accordance with the response to the notification of the determination request, the area dividing unit 153 determines whether or not to execute the area dividing process.

  Here, the region division processing in step S113 will be described. FIG. 9 is a flowchart showing the procedure of the area division process. In this area dividing process, the area dividing unit 153 first performs a gradation conversion process to divide the correction coefficient image into a plurality of divided areas (step S201). At this time, data for specifying each divided region obtained as a result of the gradation conversion process is held in the storage unit 140.

  This gradation conversion processing is performed based on division parameters specified in advance. In the present embodiment, a correction coefficient interval (specifically, an intensity width of a pixel value) I is used as a division parameter. Specifically, the region dividing unit 153 calculates the maximum value and the minimum value of the pixels based on the histogram created in step S107. Next, the area dividing unit 153 performs gradation conversion of each pixel included between the calculated maximum value and minimum value based on the correction coefficient interval I to obtain a gradation conversion image. For example, the maximum value is M, the minimum value is m, and each pixel is categorized by a pixel value width [m, M−nI], [M−2I + 1, M−1],... [M−I + 1, M]. To do. Then, using the median value of the pixel value width as a conversion value, the pixels classified into each category are subjected to gradation conversion. In this manner, the region dividing unit 153 divides the correction coefficient image into a plurality of divided regions by performing tone conversion on the correction coefficient image for each pixel value width corresponding to the correction coefficient interval I. FIG. 10 is a diagram illustrating an example of the division result of the correction coefficient image. In the example shown in FIG. 10, the correction coefficient image G10 is divided into three divided areas A11, A13, and A15.

  Next, the region dividing unit 153 performs binarization processing based on the gradation conversion result, and acquires a binary image for each divided region (step S203). Subsequently, the region dividing unit 153 performs a labeling process on each obtained binary image for each divided region, and assigns a unique value (label) to the connected component (connected pixel group) in each binary image. (Step S205). By this labeling process, at least one binary image divided region can be obtained from each divided region. At this time, data for specifying each binary image division region obtained as a result of the binarization process and the labeling process is held in the storage unit 140. Note that the specific processing method of the binarization process in step S203 and the labeling process in step S205 need not be limited from the features of the present invention, and can be implemented by using a known process. FIG. 11 is a diagram illustrating an example of a binary image division result. FIG. 11 shows a binary image divided region obtained as a result of the binarization process and the labeling process for the divided region A11 of the correction coefficient image G10 shown in FIG. 10, and according to this binary image G20. For example, four binary image divided areas A21, A23, A25, and A27 are acquired for the divided area A11. After the labeling process, the process returns to step S113 in FIG. 6 and then proceeds to step S115.

  In step S115, the representative point extracting unit 154 extracts one or a plurality of divided region representative points from the divided regions obtained as a result of the region dividing process in step S113. Specifically, the representative point extraction unit 154 performs the following processing for each divided region obtained by dividing the correction coefficient image. First, the binary image divided area having the maximum area is selected from the binary image divided areas acquired from the divided areas to be processed. By this processing, for example, in the example of FIG. 11, the binary image division region A23 is selected. Next, based on the pixel value of each pixel of the correction coefficient image included in the selected binary image division region, a histogram is created with the horizontal axis of the image as the pixel value and the vertical axis as the number of pixels. Next, based on the created histogram, pixel values whose number is equal to or less than a predetermined threshold set in advance are removed as noise, and based on the minimum and maximum pixel values other than the removed pixel values, Calculate the center value. Then, one or a plurality of pixels having the calculated center value as the pixel value are randomly selected from the pixels included in the selected binary image divided region, and the divided region representative points in the divided region to be processed are To do. The divided area representative points for each divided area extracted in this way are stored in the storage unit 140. As a result, from the pixels in the divided area, pixels having an average correction coefficient in the divided area can be extracted as the divided area representative points. Any number of segmented area representative points may be extracted. However, when a plurality of segmented area representative points are extracted, pixels existing within a predetermined range centered on the segmented area representative points that have already been extracted are excluded from the extraction target. A plurality of divided region representative points are not extracted from the range region.

  Note that the result of the area division process may be visually presented to the user, and the divided area representative points may be extracted according to the user operation. In this case, the control unit 160 performs control to display the division region of the correction coefficient image as illustrated in FIG. 10 on the display unit 130, and displays a notification of the selection request for the division region representative point for each division region. It functions as a divided area display control means and a representative point selection request means. FIG. 12 is a diagram illustrating an example of a notification screen for requesting selection of representative points of divided areas. The notification screen W20 displays a message M20 requesting selection of a divided area representative point for each divided area. The user selects a divided region representative point for each divided region by designating the position of one or a plurality of divided region representative points from each divided region via the input unit 120. Further, the designation operation is canceled by pressing the cancel button B21, or the designation operation is confirmed by pressing the enter button B23. In this notification screen W20, the selected position is identified and displayed in accordance with the designated operation of the divided area representative points. In the example of FIG. 12, the divided area representative points selected from the divided areas A11 are displayed. The position P10 is identified and displayed. In accordance with the response to the notification of the selection request, the representative point extraction unit 154 extracts the divided region representative points of each divided region. Alternatively, control is performed to display the binary image divided area acquired from the divided area as illustrated in FIG. 11 on the display unit 130 for each divided area, and the division in the divided area from the binary image divided areas having the largest area is performed. An area representative point may be selected.

Next, the observation noise characteristic determination unit 155 determines the observation noise characteristic in each divided region based on the noise autocorrelation matrix R _NN of the RGB camera 111 and the correction coefficient α at the divided region representative point (step S117). Here, the noise autocorrelation matrix R _NN of the RGB camera 111 is acquired in advance. That is, a multiband image is acquired by the image acquisition unit 110 in the absence of a sample, the variance of pixel values is obtained for each band of the obtained 6-band multiband image, and a matrix having this as a diagonal component is generated. Can be obtained. Also, assuming that the divided region representative point is a point X, the correction coefficient α _X of the point X is obtained from the pixel value of the point X in the correction coefficient image. When there is one divided region representative point extracted from one divided region, the observation noise characteristic determining unit 155 _adds the correction coefficient at the divided region representative point to the noise autocorrelation matrix R _NN of the RGB camera 111. The observation noise characteristic in each divided region is calculated by multiplying by the square of α. On the other hand, when there are a plurality of divided region representative points extracted from one divided region, an average value is calculated from the correction coefficient α at each divided region representative point, and the noise autocorrelation matrix R of the RGB camera 111 is calculated. The observation noise characteristic in each divided region is calculated by multiplying _NN by the square of the calculated average value. Then, if the observation noise characteristic determination unit 155 calculates the observation noise characteristics of each divided region, the observation noise characteristic determination unit 155 associates these with the identification information of the corresponding divided region, and stores them in the storage unit 140 as observation noise characteristic data 143 for each divided region. .

Incidentally, it is determined not to execute the dividing process in step S111, when the measurement center X ₀ of the measurement region of the spectrometer when acquiring the spectral emission characteristic E of illumination is a divided area representative point, point X _Since the calculated value at ₀ is the observation noise characteristic, the observation noise characteristic in this case is the noise autocorrelation matrix R _NN of the RGB camera 111. As a result, depending on the degree of non-uniformity of the correction coefficient distribution, it is uniformly used for estimating the spectral transmittance within the same field of view, as in the prior art.

  Next, the procedure of spectral characteristic estimation processing will be described. FIG. 13 is a flowchart illustrating a procedure of spectral characteristic estimation processing. In this spectral characteristic estimation process, first, the sample image acquisition control unit 163 controls the operation of the image acquisition unit 110 to perform multiband imaging of a target sample to be estimated for spectral transmittance and acquire a target sample image ( Step S301).

  Subsequently, the shading correction unit 156 uses the correction coefficient α of each pixel calculated in step S103 of FIG. 6 (the pixel value of the correction coefficient image imaged in step S105) to obtain the target sample image acquired in step S301. Shading correction processing is performed (step S303). Specifically, the pixel value of each pixel constituting the target specimen image is multiplied by the correction coefficient α, which is the pixel value of the pixel of the corresponding correction coefficient image, and each obtained value is imaged as the pixel value of each pixel. The obtained shading correction image is obtained.

  Subsequently, the spectral characteristic estimation unit 157 estimates the spectral transmittance at the target sample point corresponding to the pixel at the arbitrary point x of the target sample image. That is, the spectral characteristic estimation unit 157 first identifies the divided region to which the pixel at the point x belongs, and reads the observation noise characteristics in this divided region from the observation noise characteristic data 143 for each divided region stored in the storage unit 140. The observation noise characteristic used for estimating the spectral transmittance is acquired (step S305).

The spectral characteristic estimation unit 157 also includes spectral transmittance F of the optical filters 1191a and 1191b, spectral sensitivity characteristic S of the RGB camera 111, spectral radiation characteristic E of illumination, and autocorrelation matrix R _SS of the spectral transmittance of the specimen. A spectral characteristic estimation parameter is acquired (step S307).

Here, the spectral transmittance F of the optical filters 1191a and 1191b, the spectral sensitivity characteristic S of the RGB camera 111, and the spectral emission characteristic E of the illumination are measured in advance using a spectrometer or the like after selecting the equipment to be used. . Here, the spectral transmittance of the optical system 117 is approximated to 1.0. However, when the deviation from the approximate value 1.0 cannot be allowed, the spectral transmittance of the optical system 117 is also measured in advance. The spectral radiation characteristic E of illumination may be multiplied. Also, the autocorrelation matrix R _SS of the spectral transmittance of the sample is acquired in advance. The R _SS is obtained by preparing a typical specimen stained with H & E, and measuring the spectral transmittance of a plurality of points with a spectrometer to obtain an autocorrelation matrix.

Then, the spectral characteristic estimation unit 157 calculates a Wiener estimation matrix according to the following equation (7) based on the spectral characteristic estimation parameters acquired in step S305 and step S307, and the object corresponding to the point x of the shading correction image. The spectral transmittance at the sample point is estimated by Wiener estimation (step S309).

In other words, in the present embodiment, the estimated value T ^ (x) of the spectral transmittance at the target sample point corresponding to the point x of the shading correction image is the matrix representation G (x) of the pixel value at the point x of the shading correction image. From the above, estimation is performed according to the following equation (8) using the Wiener estimation matrix represented by equation (7). The obtained spectral transmittance estimated value T ^ (x) is stored in the storage unit 140.

  The spectral transmittance estimated by the image processing apparatus 1 is used, for example, for estimating the amount of the dye that is staining the target specimen. Then, the color of the image is corrected based on the estimated amount of pigment, the characteristics of the camera, the variation in the staining state, and the like are corrected, and the RGB image for display is synthesized. This RGB image is displayed on the screen of the display unit 130 and used for pathological diagnosis.

According to the present embodiment, an illumination image obtained by capturing a background without a subject in a state where illumination light is irradiated is acquired, and a correction obtained by imaging the correction coefficient α of each pixel calculated based on the illumination image as a pixel value The observation noise characteristic can be determined for each divided region obtained by dividing the coefficient image based on the correction coefficient interval I. Specifically, the observation noise characteristics in each divided region are determined based on the noise autocorrelation matrix R _NN of the RGB camera 111 and the correction coefficient α at each divided region representative point extracted from each divided region. Can do. Then, each pixel of the target sample image is subjected to shading processing using the correction coefficient α of the corresponding pixel, and the spectral transmittance of the target sample point corresponding to the pixel constituting the corrected shading correction image is Wiener estimated. In this case, it is possible to use the observation noise characteristic at the divided region representative point of the divided region to which this pixel belongs. Therefore, the nonuniformity of the spectral radiation distribution of illumination can be removed by the shading correction process. In addition, the problem of reduced estimation accuracy due to non-uniformity of correction coefficients within the same field of view becomes a problem of reduced estimation accuracy due to non-uniformity of correction coefficients within the same divided region. The estimation error of the spectral characteristic value caused by the characteristic change can be reduced, and the estimation accuracy of the spectral characteristic of the subject can be improved.

  Note that the correction coefficient interval I, which is a division parameter for dividing the correction coefficient image into regions, may be set according to a user operation. In this case, for example, the control unit 160 performs control to display the histogram created by the region dividing unit 153 illustrated in FIG. 7 on the display unit 130 and also notifies the input of the correction coefficient interval I. FIG. 14, which performs display control and functions as a parameter input request unit, is a diagram illustrating an example of a notification screen for an input request for the correction coefficient interval I. On the notification screen W30, an input box B30 for receiving an input operation of the correction coefficient interval I is arranged. The user designates the correction coefficient interval I by inputting a desired correction coefficient interval I value into the input box B30 via the input unit 120. The area dividing unit 153 performs gradation conversion processing using the value input in response to the notification of the input request as the correction coefficient interval I. As described above, if the region division processing is performed based on the correction coefficient interval I designated by the user operation, the user can designate the tolerance of the nonuniformity of the correction coefficient, and the nonuniformity of the correction coefficient. The influence on the estimation accuracy of the spectral characteristic due to the characteristic change of the observation noise characteristic caused by the characteristics can be reduced within an allowable range. Therefore, the user can appropriately adjust the balance between the reduction of the spectral characteristic estimation error due to the non-uniformity of the correction coefficient and the increase of the spectral characteristic estimation processing time due to the region division according to the value of the designated correction coefficient interval I. For example, if the correction coefficient interval I is specified to be narrow, the estimation processing time increases, but the spectral characteristic estimation error decreases, and the spectral characteristic estimation accuracy can be improved. On the other hand, when the correction coefficient interval I is specified widely, the estimation accuracy of spectral characteristics decreases to some extent, but the estimation processing time can be shortened.

  In the embodiment described above, the correction coefficient interval I is used as the division parameter and the correction coefficient image is divided. However, the correction coefficient image may be divided using the division number as the division parameter. In this case, the division number becomes the gradation number after gradation conversion. For example, the correction coefficient interval I is determined according to the division number, and the correction coefficient image is divided into a plurality of divided regions according to the determined correction coefficient interval I. Further, the number of divisions may be set according to a user operation.

In the above-described embodiment, the spectral transmission of the target sample point corresponding to the predetermined pixel constituting the target sample image that has been subjected to the shading correction process is determined for each divided region obtained by dividing the correction coefficient image. In estimating the rate, the observation noise characteristic corresponding to the divided region to which the pixel belongs is used, but the observation noise characteristic may be determined for each pixel. In this case, based on the noise autocorrelation matrix R _NN of the RGB camera 111 and the correction coefficient α in the predetermined pixel, the observation noise characteristic is calculated for the predetermined pixel, and the spectral transmittance at the corresponding target sample point is calculated. Used to estimate

  In the above-described embodiment, the case where the spectral feature value of the spectral transmittance is estimated from the multiband image obtained by imaging the pathological specimen has been described. However, the spectral feature value of the spectral reflectance is estimated as the spectral characteristic value. The same applies to the above.

  In the above embodiment, the case where the pathological specimen stained with H & E is observed through transmission has been described. However, the present invention can also be applied to a biological specimen stained using another staining method. The present invention can be similarly applied not only to observation of transmitted light but also to observation of reflected light, fluorescence, and light emission.

It is a figure which shows the structure of an image processing apparatus. It is a figure which shows typically the example of a color filter arrangement | sequence, and the pixel arrangement | sequence of each RGB band. It is a figure which shows the spectral transmittance characteristic of one optical filter. It is a figure which shows the spectral transmittance characteristic of another optical filter. It is a figure which shows the example of the spectral sensitivity of each band of R, G, B. It is a block diagram which shows the function structure of an image processing apparatus. It is a flowchart which shows the procedure of an observation noise characteristic calculation process. It is a figure which shows an example of the histogram produced based on the correction coefficient image. It is a figure which shows an example of the notification screen of the determination request | requirement of whether to divide | segment a correction coefficient image. It is a flowchart which shows the procedure of an area division process. It is a figure which shows an example of the division area | region which divided | segmented the correction coefficient image. It is a figure which shows an example of the binary image division area | region which divided | segmented the binary image. It is a figure which shows an example of the notification screen of the selection request | requirement of a division | segmentation area | region representative point. It is a flowchart which shows the procedure of a spectral characteristic estimation process. It is a figure which shows an example of the notification screen of the input request of the correction coefficient space | interval I. It is a figure which shows an example of an RGB image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 110 Image acquisition part 120 Input part 130 Display part 140 Storage part 141 Shading correction coefficient data 143 Observation noise characteristic data classified by division area 145 Image processing program 150 Image processing part 151 Correction coefficient calculation part 152 Distribution data creation part 153 area | region Division unit 154 Representative point extraction unit 155 Observation noise characteristic determination unit 156 Shading correction unit 157 Spectral characteristic estimation unit 160 Control unit 161 Illumination image acquisition control unit 163 Sample image acquisition control unit

Claims (13)

In the image processing apparatus that estimates the spectral characteristics of the subject based on the pixel value of the target image obtained by illuminating the illumination light and capturing the subject,
Correction coefficient calculation means for calculating a correction coefficient when correcting the pixel value based on the pixel value of the illumination image obtained by imaging the background in a state where the illumination light is irradiated;
Correction means for performing a shading correction process on the target image using the correction coefficient calculated by the correction coefficient calculation means;
Noise characteristic determining means for determining an observation noise characteristic of a pixel constituting the target image subjected to the shading correction processing based on the correction coefficient for the pixel;
A spectral characteristic estimation unit that calculates an estimation matrix based on the observation noise characteristic determined by the noise characteristic determination unit and estimates a spectral characteristic value of a subject position corresponding to the pixel using the calculated estimation matrix;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the noise characteristic determination unit determines an observation noise characteristic of the pixel by multiplying a predetermined reference noise amount of the observation noise by the correction coefficient for the pixel. . Distribution image creating means for creating a correction coefficient distribution image based on the correction coefficient for the pixel;
Area dividing means for executing an area dividing process for dividing the correction coefficient distribution image created by the distribution image creating means into a plurality of divided areas;
Noise calculating means for calculating an observation noise characteristic for each divided area based on the correction coefficient for the pixels in each divided area divided by the area dividing means;
With
The image processing apparatus according to claim 1, wherein the noise characteristic determining unit sets the observation noise characteristic of the pixel as an observation noise characteristic of a divided region to which the pixel belongs. Representative point extracting means for extracting at least one pixel as a divided area representative point from each divided area divided by the area dividing means,
The noise calculating means converts the observation noise characteristics of the pixels of each divided area representative point based on the correction coefficient for the pixels of the divided area representative points extracted by the representative point extracting means to the observed noise characteristics for each divided area. As
The image processing apparatus according to claim 3, wherein the noise characteristic determination unit uses the observation noise characteristic of the pixel as an observation noise characteristic of the pixel at the divided region representative point in the divided region to which the pixel belongs. . Coefficient distribution calculating means for calculating a distribution of correction coefficients for each pixel,
4. The area dividing unit includes a division determining unit that determines whether or not to execute the area dividing process based on a correction coefficient distribution created by the coefficient distribution calculating unit. 5. The image processing apparatus according to 4. Coefficient distribution display control means for performing control to display the distribution of correction coefficients calculated by the coefficient distribution calculation means on a display unit;
Determination input requesting means for requesting determination of whether or not to divide the correction coefficient distribution image into the divided regions;
With
The division determination unit determines to execute the region division process when a division instruction is input in response to a determination request from the determination input request unit, and when the non-division instruction is input, the region determination unit 6. The image processing apparatus according to claim 5, wherein it is determined that the division process is not executed.   7. The area dividing means according to claim 3, wherein the correction coefficient interval and / or the number of divisions are used as division parameters, and the correction coefficient distribution image is divided into the division areas using the division parameters. The image processing apparatus according to one. Comprising a parameter input requesting means for requesting input of the divided parameters;
8. The image processing according to claim 7, wherein the area dividing unit divides the correction coefficient distribution image into regions using a division parameter input in response to an input request from the parameter input request unit. apparatus. The region dividing means includes
Gradation conversion dividing means for converting the gradation of the correction coefficient distribution image for each gradation section according to the division parameter, and dividing the correction coefficient distribution image into the divided regions based on a gradation conversion result;
Binarization means for performing binarization processing based on the gradation conversion result and obtaining a binary image for each of the divided regions;
Based on the binarization result by the binarization means, labeling processing means for performing a labeling process on each of the binary images for each of the divided regions;
Have
9. The image processing apparatus according to claim 7, wherein the representative point extracting unit extracts the divided region representative points from the divided regions based on a result of the labeling processing unit. A divided area display control means for performing control to display a result of the area dividing process on a display unit;
Representative point selection requesting means for requesting selection of the divided region representative points for each divided region;
With
The representative point extracting unit extracts the divided region representative point in each divided region based on the divided region representative point selected in response to the selection request by the representative point selection requesting unit. The image processing apparatus according to claim 3.   The image processing apparatus according to claim 1, wherein the spectral characteristic is a spectral characteristic value of spectral transmittance or spectral reflectance. An image processing program for causing a computer to estimate spectral characteristics of a subject based on a pixel value of a target image obtained by irradiating illumination light to image the subject,
A correction coefficient calculating step for calculating a correction coefficient when correcting the pixel value based on the pixel value of the illumination image obtained by imaging the background in a state where the illumination light is irradiated;
A correction step of performing a shading correction process on the target image using the correction coefficient calculated by the correction coefficient calculation step;
A noise characteristic determination step of determining an observation noise characteristic of a pixel constituting the target image subjected to the shading correction processing based on the correction coefficient for the pixel;
A spectral characteristic estimation step of calculating an estimation matrix based on the observation noise characteristic determined by the noise characteristic determination step, and estimating a spectral characteristic value of a subject position corresponding to the pixel using the calculated estimation matrix;
An image processing program for executing   An image processing method for estimating spectral characteristics of a subject based on a pixel value of a target image obtained by illuminating illumination light and imaging the subject,
  Calculating a correction coefficient when correcting the pixel value based on the pixel value of the illumination image obtained by imaging the background in a state where the illumination light is irradiated; and
  Applying a shading correction process to the target image using the calculated correction coefficient;
  Determining an observation noise characteristic of a pixel constituting the target image subjected to the shading correction processing based on the correction coefficient for the pixel;
  Calculating an estimation matrix based on the determined observation noise characteristic, and estimating a spectral characteristic value of a subject position corresponding to the pixel using the calculated estimation matrix;
  An image processing method comprising:

Images