JP2010156612A - Image processing device, image processing program, image processing method, and virtual microscope system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To estimate a spectrum of a dyeing sample accurately without preparing beforehand a teacher data table wherein a dyeing state is assumed. <P>SOLUTION: A spectrum estimation part 142 estimates as a first spectrum, a spectrum (spectral transmittance) at a sample point on an object sample image corresponding to an estimation object pixel constituting the object sample image, by using standard teacher data 154 which are teacher data prepared beforehand. An adaptive teacher data generation part 143 generates adaptive teacher data based on the first spectrum. The spectrum estimation part 142 estimates as a second spectrum, a spectrum at the sample point on the object sample image corresponding to the estimation object pixel, by using the adaptive teacher data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing program, an image processing method, and a virtual microscope system that estimate a spectrum of a stained specimen from a stained specimen image obtained by imaging the stained specimen.

  One of the physical quantities representing physical properties inherent to the subject is a spectral transmittance spectrum. Spectral transmittance is a physical quantity that represents the ratio of transmitted light to incident light at each wavelength. Unlike color information that depends on changes in illumination light such as RGB values, object-specific information whose value does not change due to external influences. It is. For this reason, the spectral transmittance is used in various fields as information for reproducing the color of the subject itself. For example, in the field of pathological diagnosis using a biological tissue specimen, particularly a pathological specimen, spectral transmittance is used as an example of a spectral characteristic value for analysis of an image obtained by imaging the specimen.

  In pathological diagnosis, block specimens obtained by organ excision and pathological specimens obtained by needle biopsy are sliced to several microns in thickness, and are then widely observed using a microscope to obtain various findings. ing. In particular, transmission observation using an optical microscope is one of the most popular observation methods because the equipment is relatively inexpensive and easy to handle, and has been performed for a long time. In this case, since the sliced specimen hardly absorbs and scatters light and is almost colorless and transparent, it is general to stain with a dye prior to observation.

  Various dyeing methods have been proposed, and the total number thereof reaches 100 or more. Particularly, regarding pathological specimens, hematoxylin-eosin staining using two of blue-violet hematoxylin and red eosin as pigments ( Hereinafter, “H & E staining” is used as a standard.

  Hematoxylin is a natural substance collected from plants and itself has no dyeability. However, its oxide, hematin, is a basophilic dye and binds to a negatively charged substance. Since deoxyribonucleic acid (DNA) contained in the cell nucleus is negatively charged by a phosphate group contained as a constituent element, it binds to hematin and is stained blue-violet. As described above, it is not hematoxylin that has a staining property but hematin, which is an oxide thereof. However, since it is common to use hematoxylin as a name of a pigment, the following is followed. On the other hand, eosin is an acidophilic dye and binds to a positively charged substance. Whether amino acids or proteins are charged positively or negatively is affected by the pH environment, and the tendency to be positively charged under acidic conditions becomes stronger. For this reason, acetic acid may be added to the eosin solution. Proteins contained in the cytoplasm are stained from red to light red by binding to eosin.

  In a specimen (stained specimen) after H & E staining, cell nuclei, bone tissue, etc. are stained blue-purple, and cytoplasm, connective tissue, erythrocytes, etc. are stained red, so that they can be easily visually recognized. As a result, the observer can grasp the size and positional relationship of the elements constituting the tissue such as the cell nucleus, and can judge the state of the specimen morphologically.

  In addition to visual observation by an observer, the specimen is observed by performing multiband imaging of the specimen and displaying it on a display screen of an external device. When displaying on the display screen, the process of estimating the spectral transmittance of the sample from the captured multiband image, the process of estimating the amount of dye dyeing the sample based on the estimated spectral transmittance, A process for correcting the color of the image based on the estimated amount of pigment is performed, and the characteristics of the camera, the variation in the staining state, and the like are corrected, and a display image that is an RGB image of the display sample is synthesized. If the amount of pigment is appropriately estimated, it is possible to correct a darkly stained sample or a lightly stained sample to an image having a color equivalent to that of a properly stained sample.

  Examples of methods for estimating the spectral transmittance of a specimen from a multiband image of the specimen include, for example, an estimation method based on principal component analysis (see, for example, Non-Patent Document 1) and an estimation method based on Wiener estimation (eg, non-patent). Reference 2). Wiener estimation is widely known as one of the linear filter methods for estimating the original signal from the observed signal with superimposed noise, and takes into account the statistical properties of the observation target and the characteristics of the noise (observation noise). This is a method for minimizing. Since some noise is included in the signal from the camera, the Wiener estimation is extremely useful as a method for estimating the original signal.

  Here, a method for synthesizing a display image from a multiband image of a specimen will be described. The detailed method is disclosed in, for example, Patent Document 1 and Patent Document 2. First, a multiband image of a specimen is taken. For example, using the technique disclosed in Patent Document 1, a multiband image is picked up in a frame sequential manner while 16 band pass filters are switched by rotating with a filter wheel. Thereby, a multiband image having 16-band pixel values at each point on the sample is obtained. The dye is originally distributed three-dimensionally in the specimen to be observed. However, in a normal transmission observation system, it cannot be regarded as a three-dimensional image as it is, and the illumination light transmitted through the specimen is not captured by the camera. It is observed as a two-dimensional image projected on the image sensor. Therefore, each point here means a point on the sample (sample point) corresponding to each pixel of the projected image sensor.

λは波長、ｆ（ｂ，λ）はｂ番目のフィルタの分光透過率、ｓ（λ）はカメラの分光感度特性、ｅ（λ）は照明の分光放射特性、ｎ（ｂ）はバンドｂにおける観測ノイズをそれぞれ表す。ｂはバンドを識別する通し番号であり、ここでは１≦ｂ≦１６を満たす整数値である。 For any sample point (pixel) x of the captured multiband image, between the pixel value g (x, b) in band b and the spectral transmittance t (x, λ) of the corresponding sample point, The relationship of the following formula (1) based on the camera response system is established.

λ is the wavelength, f (b, λ) is the spectral transmittance of the b-th filter, s (λ) is the spectral sensitivity characteristic of the camera, e (λ) is the spectral radiation characteristic of the illumination, and n (b) is in the band b. Represents each observation noise. b is a serial number for identifying a band, and here is an integer value satisfying 1 ≦ b ≦ 16.

In actual calculation, the following formula (2) obtained by discretizing the formula (1) in the wavelength direction is used.
G (x) = FSET (x) + N (2)
If the number of sample points in the wavelength direction is D and the number of bands is B (here, B = 16), G (x) is a matrix of B rows and 1 column corresponding to the pixel value g (x, b) at the position x. is there. Similarly, T (x) is a D × 1 matrix corresponding to t (x, λ), and F is a B × D matrix corresponding to f (b, λ). On the other hand, S is a diagonal matrix of D rows and D columns, and the diagonal elements correspond to s (λ). Similarly, E is a diagonal matrix of D rows and D columns, and the diagonal element corresponds to e (λ). N is a matrix of B rows and 1 column corresponding to n (b). In Expression (2), since the expressions related to a plurality of bands are aggregated using a matrix, the variable b representing the band is not explicitly described. In addition, the integration with respect to the wavelength λ is replaced with a matrix product.

Here, in order to simplify the notation, a matrix H defined by the following equation (3) is introduced. H is also called a system matrix.
H = FSE (3)

Therefore, Expression (3) is replaced with the following Expression (4).
G (x) = HT (x) + N (4)

Next, the spectral transmittance at the sample point on the specimen is estimated from the captured multiband image using Wiener estimation. The estimated value T ^ (x) of the spectral transmittance can be calculated by the following equation (5). T ^ indicates that a symbol "^ (hat)" representing an estimated value is attached on T.

Ｒ_SSは、Ｄ行Ｄ列の行列であり、標本の分光透過率の自己相関行列を表す。また、Ｒ_NNは、Ｂ行Ｂ列の行列であり、撮像に使用するカメラのノイズの自己相関行列を表す。 Here, W is expressed by the following equation (6), and is called “a winner estimation matrix” or “an estimation operator used for winner estimation”. In the following description, W is simply referred to as “estimated operator”.

R _SS is a matrix of D rows and D columns, and represents an autocorrelation matrix of the spectral transmittance of the sample. R _NN is a matrix of B rows and B columns, and represents an autocorrelation matrix of camera noise used for imaging.

  Once the spectral transmittance T ^ (x) is estimated in this way, the dye amount at the sample point on the corresponding sample is estimated based on T ^ (x). There are three types of pigments to be estimated: hematoxylin, eosin stained with cytoplasm, eosin stained with erythrocytes, or original pigments of unstained erythrocytes, and are abbreviated as pigment H, pigment E, and pigment R, respectively. Strictly speaking, erythrocytes have their own unique color even without staining, and after H & E staining, the color of erythrocytes and the color of eosin changed in the staining process are superimposed. Observed. For this reason, the combination of both is called dye R.

ｋ（λ）は波長に依存して決まる物質固有の値、ｄは物質の厚さをそれぞれ表す。 In general, in a material that transmits light, a Lambert bale (Lambert) expressed by the following equation (7) between the intensity I ₀ (λ) of incident light and the intensity I (λ) of emitted light for each wavelength λ. -Beer) law is known to hold.

k (λ) is a material-specific value determined depending on the wavelength, and d represents the thickness of the material.

Here, the left side of the equation (7) means the spectral transmittance t (λ), and the equation (7) is replaced by the following equation (8).

The spectral absorbance a (λ) is expressed by the following equation (9).

Therefore, Expression (8) is replaced with the following Expression (10).

ｋ_H（λ），ｋ_E（λ），ｋ_R（λ）は、それぞれ色素Ｈ，色素Ｅ，色素Ｒに対応するｋ（λ）を表し、例えば、標本を染色している各色素の色素スペクトル（以下、「基準色素スペクトル」と称す。）である。またｄ_H，ｄ_E，ｄ_Rは、マルチバンド画像の各画像位置に対応する標本点における色素Ｈ，色素Ｅ，色素Ｒの仮想的な厚さを表す。本来色素は、標本中に分散して存在するため、厚さという概念は正確ではないが、標本が単一の色素で染色されていると仮定した場合と比較して、どの程度の量の色素が存在しているかを表す相対的な色素量の指標となる。すなわち、ｄ_H，ｄ_E，ｄ_Rはそれぞれ色素Ｈ，色素Ｅ，色素Ｒの色素量を表しているといえる。なお、ｋ_H（λ），ｋ_E（λ），ｋ_R（λ）は、色素Ｈ，色素Ｅ，色素Ｒを用いてそれぞれ個別に染色した標本を予め用意し、その分光透過率を分光計で測定することによって、ランベルト・ベールの法則から容易に求めることができる。 When the H & E-stained specimen is stained with three types of dyes, dye H, dye E, and dye R, the following equation (11) is established at each wavelength λ according to Lambert-Beer's law.

k _H (λ), k _E (λ), and k _R (λ) represent k (λ) corresponding to the dye H, the dye E, and the dye R, respectively. For example, the dye of each dye that stains the specimen Spectrum (hereinafter referred to as “reference dye spectrum”). Further, d _H , d _E , and d _R represent virtual thicknesses of the dye H, the dye E, and the dye R at the sample point corresponding to each image position of the multiband image. Since the dye is inherently dispersed in the specimen, the concept of thickness is not accurate, but how much dye is compared to the assumption that the specimen is stained with a single dye. It is an indicator of the relative amount of pigment that indicates whether or not the selenium is present. That is, it can be said that d _H , d _E , and d _R represent the dye amounts of the dye H, the dye E, and the dye R, respectively. For k _H (λ), k _E (λ), and k _R (λ), specimens individually dyed with the dye H, the dye E, and the dye R are prepared in advance, and the spectral transmittance is measured by the spectrometer. Can be easily obtained from the Lambert-Beer law.

Here, when the spectral transmittance at the position x is t (x, λ) and the spectral absorbance is a (x, λ), the equation (9) is replaced by the following equation (12).

Then, assuming that the estimated spectral transmittance at the wavelength λ of the spectral transmittance T ^ (x) estimated using the equation (5) is t ^ (x, λ) and the estimated absorbance is a ^ (x, λ), Expression (12) is replaced with the following expression (13). Here, t ^ indicates that a symbol "^" is attached on t, and a ^ indicates that a symbol "^" is attached on a.

Since there are three unknown variables d _H , d _E , and d _R in equation (13), these equations can be solved if equation (13) is combined for at least three different wavelengths λ. In order to further improve the accuracy, the multiple regression analysis may be performed by simultaneous equations (13) for four or more different wavelengths λ. For example, when Equation (13) is made simultaneous for _three wavelengths λ ₁ , λ ₂ , and λ ₃ , the matrix can be expressed as the following Equation (14).

波長方向のサンプル点数をＤとすれば、Ａ＾（ｘ）は、ａ＾（ｘ，λ）に対応するＤ行１列の行列であり、Ｋは、ｋ（λ）に対応するＤ行３列の行列、ｄ（ｘ）は、位置ｘにおけるｄ_H，ｄ_E，ｄ_Rに対応する３行１列の行列である。なお、Ａ＾は、Ａの上に記号「＾」が付いていることを示す。 Here, the equation (14) is replaced with the following equation (15).

Assuming that the number of sample points in the wavelength direction is D, A ^ (x) is a matrix of D rows and 1 column corresponding to a ^ (x, λ), and K is D rows 3 corresponding to k (λ). A matrix of columns, d (x), is a 3 × 1 matrix corresponding to d _H , d _E , d _R at position x. A ^ indicates that the symbol "^" is attached on A.

ｄ＾（ｘ）は、推定された色素量である。 Then, the dye amounts d _H , d _E , and d _R are calculated using the least square method according to the equation (15). The least square method is a method of determining d (x) so as to minimize the sum of squares of errors in a single regression equation, and can be calculated by the following equation (16).

d ^ (x) is the estimated pigment amount.

Dye H, the dye E, the dye amount d ^ _H, d ^ _E, the d ^ _R estimates for dye R, by substituting the equation (12), restored restored spectral absorbance ^a ~ (x, lambda) is the following formula It is calculated | required by (17). In addition, d ^ is, indicates that with a symbol "^" on top of the d, a ^~ shows that with a symbol "~" on top of a.

Here, the estimated error e in the dye amount estimation (lambda), the estimated spectral absorbance a ^ (x, lambda) and restoring spectral absorbance a ^{~ (x,} lambda) and on the basis of obtained by the following equation (18) .

The estimated spectral absorbance a ^ (x, λ) can be expressed by the following equation (19) from the equations (17) and (18).

  Lambert-Beer's law formulates the attenuation of light transmitted through a translucent object, assuming no refraction or scattering. However, refraction and scattering can occur in an actual specimen. For this reason, when the attenuation of light by the sample is modeled only by the Lambert-Beer law, errors due to refraction and scattering occur. However, it is extremely difficult to construct a model including refraction and scattering. Therefore, by taking into account an estimation error e (λ) that is a modeling error including the effects of refraction and scattering, it is possible to prevent unnatural color variation due to the physical model.

If the dye amounts d ^ _H , d ^ _E , d ^ _R are obtained as described above, it is possible to simulate changes in the dye amount in the sample by correcting them. Here, the dye amounts d ^ _H and d ^ _E stained by the staining method are corrected. D ^ _R, which is the original color of red blood cells, is not corrected. That is, corrected dye amount ^{_{d ^ H *, d ^ E}} * , the following equation using an appropriate dye-amount correction coefficient alpha _H, alpha _E (20), obtained by (21).

By substituting the calculated corrected dye amounts d ^ _H ^* and d ^ _E ^* into the equation (12), the spectral absorbances a ^{to *} (x, λ) are obtained by the following equation (22). Note that a˜ ^* indicates that the symbol “˜” is added on a ^* .

When the estimation error e (λ) is included, a new spectral absorbance a ^ ^* (x, λ) is obtained by the equation (23). Here, a ^ ^* indicates that the symbol “^” is attached on a ^* .

By substituting this spectral absorbance a˜ ^* (x, λ) or spectral absorbance a ^ ^* (x, λ) into equation (10), a new spectral transmittance t ^* (x, λ) is obtained by equation (24). Is obtained. The spectral absorbance a ^* (x, λ) is one of the spectral absorbances a ^{to *} (x, λ) or the spectral absorbance a ^ ^* (x, λ).

Then, by substituting equation (24) into equation (1), a new pixel value g ^* (x, b) can be obtained by the following equation (25). In this case, the observation noise n (b) may be calculated as zero.

Here, the equation (4) is replaced with the following equation (26).
G ^* (x) = HT ^* (x) (26)
G ^* (x) is a B × 1 matrix corresponding to g ^* (x, b), and T ^* (x) is a D × 1 matrix corresponding to t ^* (x, λ). is there. Thereby, the pixel value G ^* (x) of the sample in which the pigment amount is virtually changed can be synthesized.

  As described above, the amount of dye at an arbitrary position x in the multiband image is estimated, the amount of dye at the sample point on the sample is virtually adjusted, and the image of the sample after adjustment is synthesized, The amount of dye can be corrected. At this time, it is also possible to automatically perform color normalization processing and adjust the dye amount at each point of the sample to an appropriate staining state. In addition, if an appropriate user interface is prepared, the user can manually adjust the dye amount. The display image synthesized for display is displayed on a screen of a display device, for example, and used for pathological diagnosis by a doctor or the like. According to this, even if there is a staining variation in the specimen, it is possible to observe an image adjusted to an appropriate staining state.

By the way, when estimating the spectral transmittance (spectrum) of each point of the sample from the multiband image using the estimation operator W according to the method disclosed in Non-Patent Document 1, the system matrix H shown in Expression (5) is calculated. The spectral transmittance F of the optical filter, the spectral sensitivity characteristic S of the camera, and the spectral emission characteristic E of the illumination, the term R _SS representing the statistical properties of the observation object, and the term R _NN representing the characteristics of the imaging noise It is necessary to get to. Among these, the autocorrelation matrix R _SS , which is a term representing the statistical properties of the observation target, is prepared by preparing a typical specimen (standard stained specimen) that is standardly stained with, for example, hematoxylin and eosin, It is obtained by measuring the spectrum of a point to obtain an autocorrelation matrix.

However, it is difficult to uniformly stain the specimen, and even with the same staining method, the staining state (degree of staining) may vary depending on the facility that stains the specimen or the technician who performs the staining. Moreover, the staining state may change depending on the thickness of the specimen. For this reason, there is a problem that the staining state of the stained sample for estimating the spectral characteristics is different from the staining state of the standard stained sample used when calculating the autocorrelation matrix R _SS , and the spectrum estimation accuracy is lowered. there were.

  As a technique for solving this type of problem, for example, in Patent Document 3, statistical data of a plurality of spectral reflectances used when estimating the color of a subject is prepared in advance and estimated according to the subject photographing signal. A method of switching statistical data of spectral reflectance used in the above is disclosed.

JP-A-7-120324 JP 2008-51654 A JP 2001-8220 A “Development of support systems for pathology using spectral transmittance-The quantification method of stain conditions”, Proceedings of SPIE, Vol.4684, 2002, p.1516-1523 “Color Correction of Pathological Images Based on Dye Amount Quantification”, OPTICAL REVIEW, Vol.12, No.4, 2005, p.293-300

If the technique of Patent Document 3 is applied and teacher data for calculating the autocorrelation matrix R _SS is generated for each staining state in advance, the teacher data can be switched for each pixel for estimating the spectrum. The spectrum can be estimated with high accuracy. However, in order to generate teacher data, it is necessary to prepare standard stained specimens dyed in the corresponding staining state, so it is difficult to prepare the teacher data in consideration of all staining states. is there. For this reason, there arises a problem that the staining state of the stained sample and the staining state of the standard stained sample used when generating the teacher data are different, and the estimation accuracy of the spectrum is lowered.

  The present invention has been made in view of the above-described conventional problems, and an image processing apparatus capable of accurately estimating a spectrum of a stained specimen without preparing a teacher data table assuming a staining state in advance, An object is to provide an image processing program, an image processing method, and a virtual microscope system.

  In order to solve the above-described problems and achieve the object, an image processing apparatus according to an aspect of the present invention is an image processing apparatus that estimates a spectrum of the stained specimen from a stained specimen image obtained by imaging the stained specimen, Based on the pixel value of the estimation target pixel constituting the stained sample image, the spectrum at the sample point on the stained sample corresponding to the estimation target pixel is set as the first spectrum using preset standard teacher data. Based on the first spectrum estimating means for estimating, the teacher data generating means for generating adaptive teacher data adapted to the estimation target pixel based on the first spectrum, and the pixel value of the estimation target pixel And a second spectrum estimation means for estimating a spectrum at the sample point as a second spectrum using the adaptive teacher data.

  According to the image processing apparatus according to this aspect, the sample point on the stained sample corresponding to the estimation target pixel using the preset standard teacher data based on the pixel value of the estimation target pixel constituting the stained sample image 1 can be estimated, and adaptive teacher data adapted to the estimation target pixel can be generated based on the estimated first spectrum. Then, based on the pixel value of the estimation target pixel, the second spectrum at the sample point can be estimated using adaptive teacher data. Therefore, based on the pixel value of the estimation target pixel, it is possible to dynamically generate teacher data appropriate for adaptation to the estimation target pixel each time. According to this, it is possible to reduce the estimation error of the spectrum without preparing teacher data assuming all staining states of the stained specimen in advance, and it is possible to estimate the spectrum of the stained specimen accurately and easily. There is an effect.

  An image processing program according to another aspect of the present invention is an image processing program for causing a computer to estimate a spectrum of the stained specimen from a stained specimen image obtained by imaging the stained specimen, and configures the stained specimen image. A first spectrum for estimating, as a first spectrum, a spectrum at a sample point on the stained sample corresponding to the estimation target pixel using preset standard teacher data based on the pixel value of the estimation target pixel An estimation step; a teacher data generation step for generating adaptive teacher data adapted to the estimation target pixel based on the first spectrum; and the adaptive teacher data based on a pixel value of the estimation target pixel. And a second spectrum estimation step for estimating a spectrum at the sample point as a second spectrum. Characterized in that to execute the data.

  An image processing method according to another aspect of the present invention is an image processing method for estimating a spectrum of the stained specimen from a stained specimen image obtained by imaging a stained specimen, and includes an estimation target pixel constituting the stained specimen image. A first spectrum estimation step of estimating a spectrum at a sample point on the stained sample corresponding to the estimation target pixel as a first spectrum using standard teacher data set in advance based on the pixel value; A teacher data generation step for generating adaptive teacher data adapted to the estimation target pixel based on the first spectrum, and the sample points using the adaptive teacher data based on the pixel value of the estimation target pixel And a second spectrum estimation step of estimating the spectrum at as a second spectrum.

  In addition, a virtual microscope system according to another aspect of the present invention uses a microscope to capture an image of a stained specimen and acquire a stained specimen image, and a pixel value of an estimation target pixel constituting the stained specimen image. First, a first spectrum estimation unit that estimates a spectrum at a sample point on the stained sample corresponding to the estimation target pixel as a first spectrum using preset standard teacher data, and the first spectrum Based on the spectrum, teacher data generation means for generating adaptive teacher data adapted to the estimation target pixel, and based on the pixel value of the estimation target pixel, the spectrum at the sample point is obtained using the adaptive teacher data. And a second spectrum estimation means for estimating as a second spectrum.

  According to the present invention, it is possible to accurately estimate the spectrum of a stained specimen without preparing in advance a teacher data table that assumes a stained state.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In this embodiment, multiband imaging is performed using a biological tissue specimen (stained specimen) that has been subjected to H & E staining as an imaging target, and the spectrum of each point (specimen point) of the stained specimen is spectroscopic based on the obtained multiband image. A case where the transmittance is estimated will be described as an example. Note that the present invention is not limited to the embodiments. Moreover, in description of drawing, the same code | symbol is attached | subjected and shown to the same part.

  1 to 3 are diagrams for explaining the spectrum estimation principle of the stained specimen in the present embodiment. In the spectrum estimation according to the present embodiment, first, as shown in FIG. 1, a spectrum is prepared by using standard teacher data 12 prepared in advance based on a 6-band multiband image 11 obtained by imaging a stained specimen. Estimation is performed to estimate the first spectrum 13 of the stained specimen.

Subsequently, as shown in FIG. 2, the amount of dye is estimated based on the first spectrum, and an estimated amount of dye 14 of the dye H, dye E, and dye R staining the stained specimen is obtained. Subsequently, one or more neighboring pigment amounts are calculated based on the estimated pigment amount. For example, a value in a predetermined range indicated by an arrow A _H centering on an estimated dye amount value of the dye H is set as a neighboring dye amount for the dye H, and indicated by an arrow A _E centering on the estimated dye amount value of the dye E. the value of the predetermined range and the vicinity dye amounts for pigments E, the value of the predetermined range indicated by the arrow a _R around the value of the estimated dye amount of the dye R and vicinity dye amounts for R dye. Then, the adaptive teacher data 15 is dynamically generated by synthesizing the spectrum based on the calculated neighboring pigment amount.

  Subsequently, as shown in FIG. 3, spectrum estimation is performed again using the generated adaptive teacher data 15 based on the 6-band multiband image 11 obtained by multiband imaging of the stained specimen, and the second of the stained specimen is obtained. The spectrum 16 is estimated.

(Embodiment 1)
First, the first embodiment will be described. FIG. 4 is a block diagram illustrating a functional configuration of the image processing apparatus 1 according to the first embodiment. The image processing apparatus 1 includes a computer such as a personal computer, and includes an image acquisition unit 110 that acquires a multiband image of a stained specimen.

  The image acquisition unit 110 captures a stained sample (hereinafter referred to as “target sample”) whose spectrum (spectral transmittance) is to be estimated, and acquires a 6-band multiband image. FIG. 5 is a schematic diagram illustrating the configuration of the image acquisition unit 110. As illustrated in FIG. 5, the image acquisition unit 110 performs an image acquisition operation to capture the target specimen S, and acquires a 6-band multiband image. The image acquisition unit 110 includes an RGB camera 111 including an imaging device such as a CCD, a sample holding unit 113 on which the target sample S is placed, an illumination unit 115 that transmits and illuminates the target sample S on the sample holding unit 113, An optical system 117 for condensing the transmitted light from the target sample S to form an image, a filter unit 119 for limiting the wavelength band of the imaged light to a predetermined range, and the like are provided.

  The RGB camera 111 is widely used in digital cameras and the like, and has RGB color filters arranged in a mosaic pattern on a monochrome image sensor. The RGB camera 111 is installed so that the center of the image to be captured is positioned on the optical axis of the illumination light. FIG. 6 is a diagram schematically illustrating an arrangement example of color filters and a pixel arrangement of each RGB band. In this case, each pixel can image only one of R, G, and B components, but by using neighboring pixel values, the insufficient R, G, and B components are interpolated. This technique is disclosed in, for example, Japanese Patent No. 3510037. If a 3CCD type camera is used, R, G and B components in each pixel can be acquired from the beginning. In this embodiment, any imaging method may be used. In the following, it is assumed that R, G, and B components can be acquired in each pixel of an image captured by the RGB camera 111.

  The filter unit 119 includes two optical filters 1191 a and 1191 b each having different spectral transmittance characteristics, and these are held by a rotary optical filter switching unit 1193. FIG. 7 is a diagram showing the spectral transmittance characteristics of one optical filter 1191a, and FIG. 8 is a diagram showing the spectral transmittance characteristics of the other optical filter 1191b. For example, first, first imaging is performed using the optical filter 1191a. Next, the optical filter to be used is switched to the optical filter 1191b by the rotation of the optical filter switching unit 1193, and the second imaging is performed using the optical filter 1191b. By the first imaging and the second imaging, a 3-band image is obtained, and a 6-band multiband image is obtained by combining both results. Note that the number of optical filters is not limited to two, and three or more optical filters can be used. The acquired multiband image of the target specimen S (hereinafter referred to as “target specimen image”) is stored in the storage unit 150 of the image processing apparatus 1 as multiband image data 152.

  In the image acquisition unit 110, the illumination light irradiated by the illumination unit 115 passes through the target sample S placed on the sample holding unit 113. The transmitted light that has passed through the target specimen S passes through the optical system 117 and the optical filters 1191a and 1191b, and then forms an image on the image sensor of the RGB camera 111. The filter unit 119 including the optical filters 1191 a and 1191 b may be installed at any position on the optical path from the illumination unit 115 to the RGB camera 111. An example of spectral sensitivities of each of the R, G, and B bands when the illumination light from the illumination unit 115 is imaged by the RGB camera 111 via the optical system 117 is shown in FIG.

  As shown in FIG. 4, the image processing apparatus 1 includes an input unit 120, a display unit 130, a calculation unit 140, a storage unit 150, and a control unit 160 that controls each unit of the apparatus.

  The input unit 120 is realized by various input devices such as a keyboard, a mouse, a touch panel, and various switches, and outputs an input signal corresponding to an operation input to the control unit 160. The display unit 130 is realized by a display device such as an LCD, an EL display, or a CRT display, and displays various screens based on display signals input from the control unit 160.

  The calculation unit 140 is realized by hardware such as a CPU. The calculation unit 140 includes an estimation operator calculation unit 141 as an estimation operator calculation unit, a spectrum estimation unit 142 as a first spectrum estimation unit and a second spectrum estimation unit, and adaptive teacher data generation as a teacher data generation unit. Part 143 and spectrum evaluation part 147 as spectrum evaluation means. The estimation operator calculation unit 141 calculates the estimation operator W using standard teacher data that is teacher data prepared in advance or adaptive teacher data that is adaptively generated each time. The spectrum estimation unit 142 estimates the spectral transmittance as the spectrum at the sample point on the target sample image corresponding to the estimation target pixel constituting the target sample image. The adaptive teacher data generation unit 143 includes a pigment amount estimation unit 144 as a pigment amount estimation unit, a neighborhood pigment amount calculation unit 145 as a neighborhood pigment amount calculation unit, and a spectrum synthesis unit 146 as a spectrum synthesis unit. Generate teacher data. The pigment amount estimation unit 144 estimates the pigment amount (estimated pigment amount) of the corresponding sample point based on the first spectrum estimated for the estimation target pixel. The near dye amount calculation unit 145 calculates the near dye amount of each of the dye H, the dye E, and the dye R based on the estimated dye amount. The spectrum synthesizer 146 synthesizes a spectrum based on the amount of dye in the vicinity of the dye H, the dye E, and the dye R. Hereinafter, the spectrum synthesized by the spectrum synthesizing unit 146 is referred to as a “synthesized spectrum”. The spectrum evaluation unit 147 evaluates the estimation accuracy of the spectrum (second spectrum) performed using the adaptive teacher data.

  The storage unit 150 is realized by various IC memories such as ROM and RAM such as flash memory that can be updated and stored, a built-in hard disk connected by a data communication terminal, an information storage medium such as a CD-ROM, and a reading device thereof. It is. The storage unit 150 stores a program for operating the image processing apparatus 10 and realizing various functions of the image processing apparatus 10, data used during execution of the program, and the like.

  For example, the storage unit 150 stores an image processing program 151, multiband image data 152, and spectrum estimation data 153. The image processing program 151 is a program for realizing processing for estimating the spectrum of the target specimen from the target specimen image. The multiband image data 152 is image data of the target specimen image acquired by the image acquisition unit 110. The spectrum estimation data 153 includes standard teacher data 154, filter spectral transmittance (F) 155, camera spectral sensitivity characteristic (S) 156, illumination spectral radiation characteristic (E) 157, reference dye spectrum (dye H, E, R) 158.

  The standard teacher data 154 stores spectrum data and color data of a standard stained specimen obtained by preparing a typical specimen (standard stained specimen) that is standardly stained with hematoxylin and eosin. The spectrum data is data of a spectrum (spectral transmittance) measured with a spectrometer at, for example, a plurality of points on the standard stained specimen. The color data is, for example, color data acquired from a standard specimen image obtained by multiband imaging of a standard stained specimen, and may be one pixel value, or pixel values of a plurality of pixels, for example, the plurality of points described above. May be an average value, a maximum value, a minimum value, spectral information, or the like of pixel values of pixels corresponding to. In addition, the storage unit 150 stores adaptive teacher data generated by the adaptive teacher data generation unit 143 and the like.

  The control unit 160 is realized by hardware such as a CPU. The control unit 160 is a component that configures the image processing apparatus 10 based on input signals input from the input unit 120, image data input from the image acquisition unit 110, programs and data stored in the storage unit 150, and the like. The operation of the image processing apparatus 10 as a whole is controlled in an integrated manner. The control unit 160 includes a multiband image acquisition control unit 161. The multiband image acquisition control unit 161 acquires the target specimen image by controlling the operation of the image acquisition unit 110.

  FIG. 10 is a flowchart illustrating a processing procedure performed by the image processing apparatus 1 according to the first embodiment. Note that the processing described here is realized by the operation of each unit of the image processing apparatus 1 in accordance with the image processing program 151 stored in the storage unit 150. Further, in the following, an arbitrary position x of the target specimen image is set as an estimation target pixel and described as processing focusing on the estimation target pixel. However, when estimating the spectrum for all the pixels on the target specimen image, Processing may be performed sequentially with pixels as pixels to be estimated.

  As shown in FIG. 10, first, the multiband image acquisition control unit 161 controls the operation of the image acquisition unit 110 to perform multiband imaging of the target sample and acquire the target sample image (step s1). The acquired image data of the target specimen image is stored in the storage unit 150.

Subsequently, the estimation operator calculation unit 141 calculates the autocorrelation matrix R _SS using the standard teacher data 154 (step s3). Here, when there is one spectrum included in a certain staining state, the autocorrelation matrix R _SS can be calculated according to the following equation (27). The determinant V represents the average vector of the spectrum. T represents transposition of the determinant.

Actually, the standard teacher data 154 is, for example, spectrum (spectral transmittance) data measured by a spectrometer at a plurality of points on the standard stained specimen, and includes a plurality of spectra. Therefore, a weighted average vector V ′ shown in the following equation (28) is calculated based on the similarity d _i between the staining state of the standard staining sample and the staining state of the target sample. i represents the identification number of the spectrum included in the standard teacher data 154.

Here, the similarity d _i is calculated as follows, for example. That is, first, the color data of the target specimen image is acquired. For example, the pixel value of the estimation target pixel is acquired as color data. Then, the color data of the standard teacher data 154 and the acquired color data of the target specimen image are mapped to the feature space, and the distance between the mapping points is calculated to obtain the similarity d _i .

Subsequently, the autocorrelation matrix R _SS is calculated according to the following equation (29) using the calculated weighted average vector V ′.

When the autocorrelation matrix R _SS is calculated as described above, as shown in FIG. 10, the estimation operator calculation unit 141 calculates the calculated self based on the pixel value of the position x (spectrum estimation target pixel). An estimation operator W is calculated using the correlation matrix R _SS (step s5). Specifically, the estimation operator W is calculated according to the following equation (6) shown in the background art.

Here, as shown in the background art, a system matrix H defined by the following equation (3) is introduced.
H = FSE (3)

Note that the values of the spectral transmittance F of the optical filters 1191a and 1191b, the spectral sensitivity characteristic S of the RGB camera 111, and the spectral radiation characteristic E (^) of illumination per unit time are used for each part of the image acquisition unit 110. After selecting the equipment, measure using a spectrometer. The filter spectral transmittance (F) 155, camera spectral sensitivity characteristic (S) 156, and illumination spectral radiation characteristic (E) 157 are stored in the storage unit 150. Also, the noise autocorrelation matrix R _NN of the RGB camera 111 is calculated using the pixel value of the estimation target pixel. That is, a multiband image is acquired by the image acquisition unit 110 without a sample in advance and without a sample. Then, for each band of the obtained multiband image, the variance of the pixel value with respect to the pixel value is obtained, and an approximate expression is calculated from the pixel value and the variance of the pixel value. Then, the noise autocorrelation matrix R _NN is calculated by _obtaining the variance of the pixel value with respect to the pixel value of the estimation target pixel using the calculated approximate expression and generating a matrix having this as a diagonal component. However, it is assumed that there is no noise correlation between the bands. The estimated operator W calculated here is stored in the storage unit 150 as the first estimated operator.

Subsequently, as shown in FIG. 10, the spectrum estimation unit 142 estimates the first spectrum at the sample point on the target sample corresponding to the estimation target pixel using the first estimation operator calculated in step s5. (Step s7). Specifically, according to the following equation (5) shown in the background art, from the matrix representation G (x) of the pixel value of the pixel at the position x of the target sample image that is the estimation target pixel, the spectrum T ^ at the corresponding sample point (X) is calculated. The obtained spectrum T ^ (x) is stored in the storage unit 150 as the first spectrum.

  Subsequently, the process proceeds to calculation of adaptive teacher data. That is, as illustrated in FIG. 10, first, the pigment amount estimation unit 144 of the adaptive teacher data generation unit 143 measures the pigment H and pigment E that are measured in advance and stored in the storage unit 150 based on the first spectrum. The amount of pigment in the target sample is estimated using the reference pigment spectrum of the pigment R (step s9). Here, the reference dye spectra of the dye H, the dye E, and the dye R are calculated in advance. Then, it is stored in the storage unit 150 as a reference dye spectrum (dyes H, E, R) 158.

Specifically, based on the first spectrum at the position x of the stained specimen image that is the estimation target pixel, the dye amounts (estimated dye amounts) of the dye H, the dye E, and the dye R that are fixed to the corresponding sample points. ). The pigments to be estimated here are hematoxylin (dye H), eosin stained with cytoplasm (dye E), and unstained red blood cell color (dye R). That is, d ^ _H , d ^ _E , d ^ _R are solved according to the following equation (16) shown in the background art. The estimated estimated pigment amounts d ^ _H , d ^ _E , d ^ _R are stored in the storage unit 150.

  In the present embodiment, since dyed specimens subjected to H & E staining are targeted, the dye amounts of dye H, dye E, and dye R are estimated here. On the other hand, even when a specimen stained with another staining pigment is used as an object, the amount of the pigment can be estimated by applying the same processing. The unique color of the specimen itself can be handled in the same way. For example, here, dye R, which is a color of red blood cells, corresponds.

Subsequently, as shown in FIG. 10, the neighboring pigment amount calculation unit 145 calculates the neighboring pigment amount based on the estimated pigment amounts d ^ _H , d ^ _E , d ^ _R estimated in step s9 (step s11). ). For example, for each of the dye H, the dye E, and the dye R, a value in a predetermined range centered on the estimated dye amounts d ^ _H , d ^ _E , d ^ _R is calculated as the neighboring dye amount. Specifically, the near dye amount can be calculated as a corrected dye amount using a predetermined dye amount correction coefficient. That is, the dye amount d ^ _H, d ^ _E and the dye amount correction coefficient alpha _H, and a alpha _E, the following equation described in the background art (20), corrected dye amount of the dye H, the dye E in accordance with (21) (i.e. near dye amount) d ^ _H ^*, and calculates a d ^ _E ^*, and a dye amount d ^ _R and the dye amount correction coefficient alpha _R, the neighborhood dye amount d ^ _R ^* dye R according to the following equation (30) calculate.

More specifically, at this time, the nearby dye amount calculation unit 145 calculates the dye amount correction coefficients α _H , α _E , and α _R according to the following equations (31), (32), and (33).

stepα _H, stepα _E, stepα _E, each dye-amount correction coefficient alpha _H, alpha _E, represents the step size of the alpha _R, is set, for example, are "0.1". The step size may be set in advance as a fixed value, or may be set variably in accordance with a user operation. Also, indexα _H , indexα _E , and indexα _E represent identification numbers for identifying each data of neighboring pigment amounts. For example, {−3, −2, −1, 0, 1, 2, 3} Is set. The index α _H , index α _E , and index α _E may be set as appropriate according to the number of neighboring pigment amounts to be calculated. Here, seven index α _H , d ^ _E , and d ^ _R are used as the centers. (Including estimated pigment amounts d ^ _H , d ^ _E , d ^ _R ). The calculated neighborhood pigment amounts d ^ _H ^* , d ^ _E ^* , d ^ _R ^* are stored in the storage unit 150.

The calculation of the neighboring pigment amounts d ^ _H ^* , d ^ _E ^* , d ^ _R ^* is performed by estimating the estimated pigment amounts d ^ _H , d ^ _E , d ^ shown in the equations (20), (21), and (30). _{The present} invention is not limited to the form in which _R is multiplied by the dye amount correction coefficients α _H , α _E , and α _R, and may be calculated by adding or subtracting them. Further, the subsequent processing described below may be performed using the estimated pigment amounts d ^ _H , d ^ _E , d ^ _R as the neighboring pigment amounts.

Subsequently, as shown in FIG. 10, the spectrum synthesizer 146 synthesizes a spectrum (synthetic spectrum) using the reference dye spectra of the dye H, the dye E, and the dye R for each of the nearby dye amounts calculated in step s11. (Step s13). Specifically, first, the following expression (34), which is a calculation expression similar to the expression (22) shown in the background art, based on the neighboring dye amounts d ^ _H ^* , d ^ _E ^* , d ^ _R ^*. The spectral absorbances a ^{to *} (x, λ) are obtained according to k _H (λ), k _E (λ), and k _R (λ) are reference dye spectra of the dye H, the dye E, and the dye R, respectively.

Subsequently, a new spectral transmittance t ^* (x, λ) at the position x is obtained from the obtained spectral absorbances a ^{to *} (x, λ) according to the following equation (24) shown in the background art.

Then, the spectral transmittance t ^* (x, λ) is obtained D times in the wavelength direction to obtain a combined spectrum T ^* (x). The combined spectrum T ^* (x) is a D × 1 matrix corresponding to t ^* (x, λ).

Then, as shown in FIG. 10, the adaptive teacher data generation unit 143 generates adaptive teacher data based on the combined spectrum T ^* (x) synthesized in step s13 (step s15). The generated adaptive teacher data is stored in the storage unit 150. For example, the adaptive teacher data generation unit 143 acquires color data from the combined spectrum T ^* (x) for each neighboring pigment amount. Then, the combined spectrum T ^* (x) for each adjacent pigment amount is set as spectrum data, and this spectrum and the acquired color data are set as adaptive teacher data. The color data is obtained by, for example, performing a process of converting the combined spectrum T ^* (x) for each neighboring pigment amount into an RGB value and calculating a pixel value. Specifically, a new pixel value G ^* (x) is calculated according to the following equation (26) shown in the background art, and is set as color data corresponding to each combined spectrum T ^* (x).
G ^* (x) = HT ^* (x) (26)

Subsequently, as illustrated in FIG. 10, the estimation operator calculation unit 141 calculates the autocorrelation matrix R _SS using the adaptive teacher data generated in step s15 (step s17). The calculation procedure is performed in the same manner as in step s3. That is, the weighted average vector V ′ shown in the following equation (28) is calculated. i represents the identification number of the combined spectrum T ^* (x) included in the adaptive teacher data.

The degree of similarity d _{i at} this time is calculated as follows, for example. That is, first, the pixel value of the estimation target pixel is acquired as the color data of the target specimen image. Then, the color data of the adaptive teacher data and the acquired color data of the target specimen image are mapped to the feature space, and the distance between the mapping points is calculated to obtain the similarity d _i .

Subsequently, the autocorrelation matrix R _SS is calculated according to the following equation (29) using the calculated weighted average vector V ′.

When the autocorrelation matrix R _SS is calculated as described above, the estimation operator calculation unit 141 subsequently calculates based on the pixel value at the position x (spectrum estimation target pixel) as shown in FIG. The estimated operator W is calculated using the autocorrelation matrix R _SS (step s19). Specifically, the estimation operator W is calculated according to the following equation (6) shown in the background art. The estimated operator W calculated here is stored in the storage unit 150 as the second estimated operator.

Subsequently, as shown in FIG. 10, the spectrum estimation unit 142 estimates the second spectrum at the sample point on the target sample corresponding to the estimation target pixel using the second estimation operator calculated in step s19. (Step s21). Specifically, according to the following equation (5) shown in the background art, the spectrum at the corresponding sample point is obtained from the matrix representation G (x) of the pixel value of the pixel at an arbitrary position x of the target sample image that is the estimation target pixel. T ^ (x) is calculated. The obtained spectrum T ^ (x) is stored in the storage unit 150 as the second spectrum.

Subsequently, the spectrum evaluation unit 147 evaluates the estimation accuracy of the second spectrum based on the difference between the first spectrum and the second spectrum (step s23), and the evaluation result of the estimation accuracy of the second spectrum Whether or not satisfies the termination condition (step s25). For example, first, the difference T _diff between the first spectrum and the second spectrum is calculated according to the following equation (34).

Subsequently, according to the following equation (35), a fluctuation amount of the difference T _diff between the calculated first spectrum and the second spectrum is obtained, and it is determined whether or not this value has converged. Diff _threshold represents a _threshold value, and an appropriate value is set. Note that the value of the threshold diff _threshold may be set as a fixed value, or may be set variably according to a user operation, for example. Further, n represents the number of times (number of trials) that the second spectrum is estimated.

As shown in FIG. 10, the spectrum evaluation unit 147 satisfies the termination condition when the expression (35) is satisfied, that is, when the difference T _diff between the first spectrum and the second spectrum has converged. (Step s25: Yes), and the process ends. On the other hand, when Expression (35) is not satisfied, that is, when the difference T _diff between the first spectrum and the second spectrum has not converged, it is determined that the termination condition is not satisfied (step s25: No). In this case, the calculation unit 140 updates the first spectrum with the second spectrum (step s27), and causes the process from step s9 to be performed again as a re-estimation processing unit. Through this processing, adaptive teacher data is newly generated from the updated first spectrum, and the second spectrum is estimated again based on the generated adaptive teacher data.

  Here, it is determined whether to regenerate adaptive teacher data and estimate the second spectrum based on the evaluation result of the second spectrum. However, the number of repetitions of the process is set in advance. A configuration may be adopted in which adaptive teacher data generation and second spectrum estimation are repeated according to the number of repetitions. Further, the processing here is not essential, and the second spectrum may not be evaluated or re-estimated. That is, it is good also as a structure which does not perform the process of step s23-step s27 of FIG.

  The spectrum (spectral transmittance) estimated as described above is used, for example, for estimating the amount of the dye that is staining the target specimen. Then, the color of the image is corrected based on the estimated pigment amount, the camera characteristics and the variation in the staining state are corrected, and the display RGB image is synthesized. This RGB image is displayed on the screen of the display unit 130 and used for pathological diagnosis.

According to the first embodiment, first, the autocorrelation matrix R _SS is calculated using standard teacher data prepared in advance, and the first estimation operator is calculated. Then, spectrum estimation is performed using the first estimation operator, and adaptive teacher data can be generated using the obtained spectrum as the first spectrum. Then, an autocorrelation matrix R _SS is calculated using the generated adaptive teacher data, and a second estimation operator is calculated. Then, spectrum estimation can be performed using the second estimation operator. Therefore, based on the pixel value of the estimation target pixel, it is possible to dynamically generate teacher data suitable for adaptation to the estimation target pixel each time and use it for spectrum estimation. According to this, it is possible to reduce the estimation error of the spectrum without preparing teacher data assuming all staining states of the stained specimen in advance, and it is possible to estimate the spectrum of the stained specimen accurately and easily. There is an effect.

  In addition, since the second spectrum is evaluated and adaptive teacher data generation and second spectrum estimation can be repeatedly performed until the evaluation result satisfies the termination condition, the adaptive teacher data can be optimized, and the spectrum estimation accuracy can be optimized. Can be further improved.

  In the first embodiment described above, the case where the second spectrum is evaluated by obtaining the amount of change in the difference between the first spectrum and the second spectrum is exemplified. However, the amount of pigment estimated from the first spectrum It is also possible to evaluate the second spectrum by obtaining the amount of change in the difference between the amount of the pigment estimated from the second spectrum and the pigment amount estimated from the second spectrum.

  FIG. 11 is a flowchart illustrating a processing procedure performed by the image processing apparatus 1 in the present modification. In FIG. 11, the same reference numerals are assigned to the same processing steps as those in the first embodiment.

  In the second embodiment, as shown in FIG. 11, after the spectrum estimation unit 142 estimates the first spectrum in step s7, the pigment amount estimation unit 144 performs the first spectrum in the same manner as in the first embodiment. Based on the above, the amount of pigment in the target specimen is estimated to be the first estimated pigment amount (step s10). Then, it moves to step s11.

  Then, after the spectrum estimation unit 142 estimates the second spectrum in step s21, the pigment amount estimation unit 144 subsequently estimates the pigment amount of the target sample based on the second spectrum, and the second estimation pigment. The amount is set (step s22). The estimation method of the pigment amount can be realized by the same process as the estimation of the first pigment amount (estimation of the pigment amount described in the first embodiment shown in step s9 in FIG. 10).

  Subsequently, the spectrum evaluation unit 147 evaluates the estimation accuracy of the second spectrum based on the difference between the first estimated dye amount and the second estimated dye amount (step s24), and estimates the second spectrum. It is determined whether or not the accuracy evaluation result satisfies the termination condition (step s26). For example, the difference between the first estimated pigment amount and the second estimated pigment amount is calculated, the amount of variation in the difference between the calculated first estimated pigment amount and the second estimated pigment amount is obtained, and this value converges It is determined whether or not.

  Then, the spectrum evaluation unit 147 determines that the end condition is satisfied when the difference between the first estimated dye amount and the second estimated dye amount has converged (step s26: Yes), and ends the process. On the other hand, when the difference between the first estimated pigment amount and the second estimated pigment amount has not converged, it is determined that the termination condition is not satisfied (step s26: No). In this case, the calculation unit 140 updates the first spectrum with the second spectrum (step s27) and causes the process from step s10 to be performed again as a re-estimation processing unit. Through this processing, adaptive teacher data is newly generated from the updated first spectrum, and the second spectrum is estimated again based on the generated adaptive teacher data.

  In addition, in this modification, the case where the variation amount of the difference between the first estimated dye amount and the second estimated dye amount is obtained and the second spectrum is evaluated is illustrated. In the first embodiment described above, the case where the second spectrum is evaluated by obtaining the amount of change in the difference between the first spectrum and the second spectrum is exemplified, but the present invention is not limited to this. For example, the difference between the first spectrum and the second spectrum may be thresholded to evaluate the estimation accuracy of the second spectrum, or the first estimated dye amount and the second estimated dye amount It is good also as a structure which evaluates the estimation precision of a 2nd spectrum by threshold-processing a difference. Alternatively, other values that are secondarily calculated based on the second spectrum can be appropriately used for the evaluation. For example, a configuration may be adopted in which a square difference between the first spectrum and the second spectrum is obtained, threshold processing is performed, and the estimation accuracy of the second spectrum is evaluated.

(Embodiment 2)
Next, a second embodiment will be described. FIG. 12 is a block diagram illustrating a functional configuration of the image processing apparatus 10b according to the second embodiment. In addition, the same code | symbol is attached | subjected about the structure same as the structure demonstrated in Embodiment 1. FIG. As shown in FIG. 12, the image processing apparatus 10b includes an image acquisition unit 110, an input unit 120, a display unit 130, a calculation unit 140b, a storage unit 150b, and a control unit 160 that controls each unit of the apparatus. .

  The calculation unit 140b includes an estimation operator calculation unit 141, a spectrum estimation unit 142, an adaptive teacher data generation unit 143b, and a spectrum evaluation unit 147. In the second embodiment, the adaptive teacher data generation unit 143b includes a pigment amount estimation unit 144, a neighboring pigment amount calculation unit 145, and a spectrum synthesis unit 146b. The spectrum synthesis unit 146b uses a difference as a difference spectrum calculation unit. A spectrum calculation unit 148b is included. The difference spectrum calculation unit 148b calculates a difference spectrum between the first spectrum and a synthesized spectrum synthesized based on the amount of the nearby pigment.

  FIG. 13 is a flowchart illustrating a processing procedure performed by the image processing apparatus 1b according to the second embodiment. In FIG. 13, the same reference numerals are assigned to the same processing steps as those in the first embodiment.

In the second embodiment, after the neighboring dye amount calculation unit 145 calculates the neighboring dye amounts d ^ _H ^* , d ^ _E ^* , d ^ _R ^* in step s11, the difference spectrum calculation unit 148b of the spectrum synthesis unit 146b Spectral absorbance is calculated from neighboring dye amounts d ^ _H ^* , d ^ _E ^* , d ^ _R ^*, and a difference spectrum from the estimated absorbance of the first spectrum calculated in step s7 is calculated using the standard teacher data 154. (Step s131).

That is, first, the difference spectrum calculation unit 148b obtains the spectral absorbances a ^{to *} (x, λ) according to the following equation (34) based on the neighboring dye amounts d ^ _H ^* , d ^ _E ^* , d ^ _R ^*. .

Subsequently, the difference spectrum calculation unit 148b calculates the estimated absorbance a ^ of the first spectrum according to the following equation (35), similarly to the calculation procedure of e (λ) described as an estimation error in the background art. A difference spectrum from the spectral absorbance a ^{to *} (x, λ) is calculated. Hereinafter, e (λ) is referred to as “difference spectrum”.

Here, the estimated absorbance a ^ (x, λ) of the first spectrum is calculated according to the following equation (13) shown in the background art.

And as shown in FIG. 13, the spectrum synthetic | combination part 146b synthesize | combines a spectrum using the difference spectrum calculated for every adjacent pigment | dye amount as mentioned above (step s133), and moves to step s15. That is, first, the spectrum synthesizer 146b adds the difference spectrum e (λ) calculated in step s131 according to the following equation (36) to calculate the spectral absorbances a ^{to *} (x, λ) again.

Thereafter, as in the first embodiment, the spectrum synthesizer 146b performs a new spectral analysis at the position x from the obtained spectral absorbances a ^{to *} (x, λ) according to the following equation (24) shown in the background art. The transmittance t ^* (x, λ) is obtained.

Then, the spectrum synthesis unit 146b repeats D times in the wavelength direction to obtain the spectral transmittance t ^* (x, λ), and obtains the synthesized spectrum T ^* (x). Composite spectrum T ^* (x) is t ^* (x, λ) D rows and one column corresponding to the matrix.

  According to the second embodiment, the same effects as those of the first embodiment are obtained. Also, a difference spectrum between the first spectrum and the combined spectrum can be calculated, and adaptive teacher data can be generated using the difference spectrum.

  In Embodiment 1 described above, the amount of pigment is estimated based on the first spectrum, the spectrum is synthesized, and adaptive teacher data is generated. In the second embodiment, a difference spectrum between the first spectrum and the synthesized spectrum is calculated, and adaptive teacher data is generated by adding the difference spectrum to the synthesized spectrum. On the other hand, adaptive teacher data may be generated by adding randomly generated noise to all bands of the first spectrum to obtain spectrum data. According to this, it is not necessary to estimate the pigment amount, and the processing load can be reduced.

(Embodiment 3)
When observing a specimen using a microscope, the range (field of view range) that can be observed at a time is mainly determined by the magnification of the objective lens. Here, as the magnification of the objective lens is increased, a high-definition image can be obtained, but the field of view is narrowed. In order to solve this type of problem, conventionally, a sample image is taken for each part using a high-magnification objective lens while moving the visual field range by moving an electric stage on which the specimen is placed, etc. A high-definition and wide-field specimen image is generated by joining the images, and this is called a virtual microscope system. Hereinafter, a specimen image generated by the virtual microscope system is referred to as a “VS image”. According to this virtual microscope system, observation can be performed even in an environment where no specimen actually exists.

  In the third embodiment, the present invention is applied to the virtual microscope system described above. FIG. 14 is a schematic perspective view showing an example of an overview of the virtual microscope system 2 of the third embodiment. As shown in FIG. 14, the virtual microscope system 2 is configured by connecting a microscope apparatus 20 and a host system 40 so that data can be transmitted and received. The host system 40 includes an input unit 41 such as a keyboard and a mouse, and a display unit 42.

  FIG. 15 is a schematic diagram illustrating an example of the overall configuration of the virtual microscope system 2 according to the third embodiment. FIG. 16 is a block diagram illustrating a functional configuration of the host system 40 according to the third embodiment. Here, the optical axis direction of the objective lens 27 shown in FIG. 15 is defined as the Z direction, and a plane perpendicular to the Z direction is defined as the XY plane.

  The microscope apparatus 20 has an electric stage 21 on which the target specimen S is placed and a substantially U-shaped side view, and supports the electric stage 21 and holds an objective lens 27 via a revolver 26. A light source 28 disposed behind the bottom of the microscope main body 24 (to the right in FIG. 15), and a lens barrel 29 placed on the top of the microscope main body 24. Further, a binocular unit 31 for visually observing a sample image of the target sample S and a TV camera 32 for capturing the sample image of the target sample S are attached to the lens barrel 29. The target specimen S is a biological tissue specimen (stained specimen) that has been subjected to H & E staining, as in the first embodiment.

  The electric stage 21 is configured to be movable in the XYZ directions. That is, the electric stage 21 is movable in the XY plane by the motor 221 and the XY drive control unit 223 that controls the driving of the motor 221. Under the control of the microscope controller 33, the XY drive control unit 223 detects a predetermined origin position on the XY plane of the electric stage 21 by an XY position origin sensor (not shown), and the driving amount of the motor 221 with this origin position as a base point. Is controlled to move the observation location on the target specimen S. Then, the XY drive control unit 223 outputs the X position and Y position of the electric stage 21 during observation to the microscope controller 33 as appropriate. The electric stage 21 is movable in the Z direction by a motor 231 and a Z drive control unit 233 that controls driving of the motor 231. Under the control of the microscope controller 33, the Z drive control unit 233 detects a predetermined origin position in the Z direction of the electric stage 21 by an origin sensor (not shown) of the Z position, and the driving amount of the motor 231 is based on this origin position. Is controlled to move the target sample S to an arbitrary Z position within a predetermined height range. Then, the Z drive control unit 233 appropriately outputs the Z position of the electric stage 21 at the time of observation to the microscope controller 33.

  The revolver 26 is rotatably held with respect to the microscope main body 24, and the objective lens 27 is disposed above the target sample S. The objective lens 27 is interchangeably mounted together with other objective lenses having different magnifications (observation magnifications) with respect to the revolver 26, and is inserted into the optical path of the observation light according to the rotation of the revolver 26, so The objective lens 27 used for observation is selectively switched.

  The microscope main body 24 has an illumination optical system for transmitting and illuminating the target specimen S at the bottom. The illumination optical system includes a collector lens 251 that collects illumination light emitted from the light source 28, an illumination system filter unit 252, a field stop 253, an aperture stop 254, and an optical path of the illumination light along the optical axis of the objective lens 27. A bending mirror 255 to be deflected, a condenser optical element unit 256, a top lens unit 257, and the like are arranged at appropriate positions along the optical path of the illumination light. Illumination light emitted from the light source 28 is irradiated onto the target specimen S by the illumination optical system and enters the objective lens 27 as observation light.

  The microscope main body 24 has a filter unit 30 in the upper part thereof. The filter unit 30 rotatably holds two or more optical filters 303 for limiting the wavelength band of light to be imaged as a specimen image to a predetermined range, and these optical filters 303 are appropriately observed after the objective lens 27. Insert into the light path. The filter unit 30 is configured similarly to the filter unit 119 described with reference to FIG. Here, the case where the optical filter 303 is disposed at the subsequent stage of the objective lens 27 is illustrated, but the present invention is not limited to this, and is disposed at any position on the optical path from the light source 28 to the TV camera 32. That's good. Observation light that has passed through the objective lens 27 enters the lens barrel 29 via the filter unit 30.

  The lens barrel 29 includes a beam splitter 291 that switches the optical path of the observation light that has passed through the filter unit 30 and guides it to the binocular unit 31 or the TV camera 32. The sample image of the target sample S is introduced into the binocular unit 31 by the beam splitter 291 and visually observed by the spectroscope through the eyepiece 311. Alternatively, the image is taken by the TV camera 32. The TV camera 32 includes an image sensor such as a CCD or a CMOS that forms a sample image (specifically, a sample image in the field of view of the objective lens 27), images the sample image, and outputs image data of the sample image. Output to the host system 40.

  The microscope apparatus 20 includes a microscope controller 33 and a TV camera controller 34. Under the control of the host system 40, the microscope controller 33 comprehensively controls the operation of each part constituting the microscope apparatus 20. For example, the microscope controller 33 rotates the revolver 26 to switch the objective lens 27 arranged on the optical path of the observation light, dimming control of the light source 28 according to the magnification of the switched objective lens 27, and various optical elements. Switching, or an instruction to move the electric stage 21 to the XY drive control unit 223 or the Z drive control unit 233, etc., and adjustment of each part of the microscope apparatus 20 accompanying observation of the target specimen S, and the state of each part is appropriately set in the host system 40. Notify Under the control of the host system 40, the TV camera controller 34 drives the TV camera 32 by performing automatic gain control ON / OFF switching, gain setting, automatic exposure control ON / OFF switching, exposure time setting, and the like. Then, the imaging operation of the TV camera 32 is controlled.

  On the other hand, as shown in FIG. 16, the host system 40 includes an input unit 41, a display unit 42, a calculation unit 43, a storage unit 50, and a control unit 54 that controls each unit of the apparatus. 16 shows the functional configuration of the host system 40, the actual host system 40 includes a CPU, a video board, a main storage device such as a main memory (RAM), and an external storage device such as a hard disk and various storage media. It can be realized by a known hardware configuration including an output device such as a communication device, a display device or a printing device, an input device, an interface device for connecting each part or an external input, such as a workstation or a personal computer. A general-purpose computer can be used.

  Here, the virtual microscope system 2 of the third embodiment is configured based on the configuration of the image processing apparatus 1 of the first embodiment, and the calculation unit 43 includes an estimation operator calculation unit 45, a spectrum estimation unit, and the like. 46, an adaptive teacher data generation unit 47, a spectrum evaluation unit 48, and an VS image generation unit 44 including an image synthesis unit 49. Note that the modification of the first embodiment and the configuration of the second embodiment can also be applied.

  The VS image generation unit 44 processes each of a plurality of target specimen images obtained by the multi-band imaging of the target specimen S by the microscope apparatus 20, and generates a VS image. Here, the VS image is an image generated by connecting one or more images taken by the microscope device 20 with multiband imaging. The VS image referred to in Embodiment 3 is an image generated by connecting a plurality of high-resolution images obtained by capturing the target specimen S for each part using, for example, a high-magnification objective lens. A wide-field and high-definition multiband image that reflects the entire area.

  The estimation operator calculation unit 45 calculates the estimation operator W using standard teacher data or adaptive teacher data in the same manner as the estimation operator calculation unit 141 of the first embodiment. The spectrum estimation unit 46 estimates the spectral transmittance as the spectrum at the sample point on the target sample image corresponding to each pixel constituting the target sample image in the same manner as the spectrum estimation unit 142 of the first embodiment. The adaptive teacher data generation unit 47 includes a pigment amount estimation unit 471, a neighboring pigment amount calculation unit 472, and a spectrum synthesis unit 473, and is similar to the adaptive teacher data generation unit 143b of Embodiment 1 in the manner of adaptation teacher data. Is generated. That is, the pigment amount estimation unit 471 estimates the pigment amount (first estimated pigment amount) at each sample point based on the first spectrum estimated for each pixel, and based on the second spectrum. The amount of pigment at each sample point corresponding to (second estimated pigment amount) is estimated. The near dye amount calculation unit 472 calculates the near dye amount of each of the dye H, the dye E, and the dye R based on the estimated dye amount. The spectrum synthesizer 473 synthesizes a spectrum based on the amount of the pigment near the pigment H, the pigment E, and the pigment R or the second estimated pigment amount. The spectrum evaluation unit 48 evaluates the estimation accuracy of the spectrum (second spectrum) performed using the adaptive teacher data. In addition, the image composition unit 49 calculates a pixel value using a spectrum synthesized for each pixel based on the second estimated pigment amount, and displays an RGB image (display image) for displaying the target specimen image (VS image). Is synthesized.

  The storage unit 50 is realized by various IC memories such as ROM and RAM such as flash memory that can be updated and stored, information storage media such as a built-in or data communication terminal, a hard disk connected by a data communication terminal, a CD-ROM, and a reading device thereof. It is. The storage unit 50 stores a program for operating the host system 40 and realizing various functions of the host system 40, data used during execution of the program, and the like.

  For example, the storage unit 50 stores a VS image generation program 51 including an image processing program 511, VS image data (multiband image data) 52, and spectrum estimation data 53. The VS image generation program 51 is a program for realizing a process of generating a VS image of the target specimen, and the image processing program 511 is a program for realizing a process of estimating the spectrum of the target specimen from the target specimen image. is there. The VS image data is, for example, image data of the entire area of the target specimen S generated by connecting the image data of the target specimen images acquired for each part by the microscope apparatus 20. The spectrum estimation data 53 includes standard teacher data 531, filter spectral transmittance (F) 532, camera spectral sensitivity characteristic (S) 533, illumination spectral radiation characteristic (E) 534, and reference dye spectrum (dye H, E, R) 535.

  The control unit 54 is realized by hardware such as a CPU. The control unit 54 receives input signals input from the input unit 41, states of each unit of the microscope apparatus 20 input from the microscope controller 33, image data input from the TV camera 32, programs recorded in the storage unit 50, Based on the data and the like, instructions to each part constituting the host system 40, data transfer, etc., or operation instructions of each part of the microscope apparatus 20 to the microscope controller 33 and the TV camera controller 34 are performed, and the entire virtual microscope system 2 is controlled. Control the overall operation. The control unit 54 includes a multiband image acquisition control unit 55 as image acquisition means. The multiband image acquisition control unit 55 issues an operation instruction for each part of the microscope apparatus 20 and acquires a target sample image obtained by imaging the target sample S for each part.

  According to the third embodiment, a virtual microscope system that can achieve the same effects as the first embodiment can be realized.

  In the above embodiment, the case where the spectral feature value of the spectral transmittance is estimated from the multiband image obtained by imaging the stained specimen has been described, but the case where the spectral feature value such as the spectral reflectance and the absorbance is estimated is also described. The same applies.

  In the above-described embodiment, the H & E-stained stained sample has been described as a spectrum estimation target. However, the present invention can be similarly applied to a case where a stained sample stained with another staining pigment is handled.

It is a figure explaining the spectrum estimation principle of a stained specimen. It is another figure explaining the spectrum estimation principle of a stained specimen. It is another figure explaining the spectrum estimation principle of a stained specimen. 2 is a block diagram illustrating a functional configuration of the image processing apparatus according to Embodiment 1. FIG. It is a schematic diagram which shows the structure of an image acquisition part. It is a figure which shows typically the example of a color filter arrangement | sequence of the RGB camera which comprises an image acquisition part, and the pixel arrangement | sequence of each RGB band. It is a figure which shows the spectral transmittance characteristic of one optical filter which comprises an image acquisition part. It is a figure which shows the spectral transmittance characteristic of the other optical filter which comprises an image acquisition part. It is a figure which shows the example of the spectral sensitivity of each band of R, G, B. 3 is a flowchart illustrating a processing procedure performed by the image processing apparatus according to the first embodiment. It is a flowchart which shows the process sequence which the image processing apparatus of a modification is performed. 6 is a block diagram illustrating a functional configuration of an image processing apparatus according to Embodiment 2. FIG. 10 is a flowchart illustrating a processing procedure performed by the image processing apparatus according to the second embodiment. FIG. 10 is a schematic perspective view illustrating an example of an overview of a virtual microscope system according to a third embodiment. FIG. 10 is a schematic diagram illustrating an example of the overall configuration of a virtual microscope system according to a third embodiment. FIG. 10 is a block diagram illustrating a functional configuration of a host system according to a third embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10,10b Image processing apparatus 110 Image acquisition part 120 Input part 130 Display part 140,140b Operation part 141 Estimation operator calculation part 142 Spectrum estimation part 143,143b Adaptive teacher data generation part 144 Dye quantity estimation part 145 Neighboring pigment quantity estimation part 146 , 146b Spectrum synthesis unit 147 Spectrum evaluation unit 148b Difference spectrum calculation unit 150, 150b Storage unit 151, 151b Image processing program 152 Multiband image data 153 Spectrum estimation data 154 Standard teacher data 155 Filter spectral transmittance (F)
156 Camera spectral sensitivity characteristics (S)
157 Illumination spectral radiation characteristics (E)
158 Reference dye spectrum (Dye H, E, R)
160 Control Unit 161 Multiband Image Acquisition Control Unit 2 Virtual Microscope System 20 Microscope Device 40 Host System

Claims (14)

An image processing apparatus for estimating a spectrum of the stained specimen from a stained specimen image obtained by imaging the stained specimen,
Based on the pixel value of the estimation target pixel constituting the stained sample image, the spectrum at the sample point on the stained sample corresponding to the estimation target pixel is set as the first spectrum using preset standard teacher data. First spectrum estimating means for estimating;
Teacher data generation means for generating adaptive teacher data adapted to the estimation target pixel based on the first spectrum;
Second spectrum estimation means for estimating a spectrum at the sample point as a second spectrum using the adaptive teacher data based on the pixel value of the estimation target pixel;
An image processing apparatus comprising:   The teacher data generation means includes dye amount estimation means for estimating the dye amount at the sample point based on the first spectrum, and the adaptive teacher data is obtained from the dye amount estimated by the dye amount estimation means. The image processing apparatus according to claim 1, wherein the image processing apparatus generates the image processing apparatus.   The teacher data generation unit includes a spectrum synthesis unit that synthesizes a spectrum based on the pigment amount estimated by the pigment amount estimation unit, and uses the spectrum synthesized by the spectrum synthesis unit as the adaptive teacher data. The image processing apparatus according to claim 2. The teacher data generating means includes a neighboring dye amount calculating means for calculating one or more neighboring dye amounts based on the dye amount estimated by the dye amount estimating means,
The image processing apparatus according to claim 3, wherein the spectrum synthesizing unit synthesizes a spectrum for each of the neighboring pigment amounts to form the adaptive teacher data.   The teacher data generation means includes difference spectrum calculation means for calculating a difference spectrum between the first spectrum and the combined spectrum, and adds the difference spectrum to the combined spectrum to obtain the adaptive teacher data. The image processing apparatus according to claim 3 or 4, wherein the image processing apparatus is characterized in that:   Re-estimation processing means for replacing the second spectrum with the first spectrum, and re-establishing generation of the adaptive teacher data by the teacher data generation means and estimation of the second spectrum by the second spectrum estimation means The image processing apparatus according to claim 1, further comprising: Spectrum evaluation means for evaluating the estimation accuracy of the second spectrum;
The re-estimation processing unit determines whether or not to generate the adaptive teacher data and estimate the second spectrum again based on an evaluation result by the spectrum evaluation unit. 6. The image processing apparatus according to 6.   The image processing apparatus according to claim 7, wherein the spectrum evaluation unit evaluates estimation accuracy of the second spectrum based on a difference between the first spectrum and the second spectrum.   The image according to claim 7, wherein the spectrum evaluation unit evaluates an estimation accuracy of the second spectrum based on a variation amount of a difference between the first spectrum and the second spectrum. Processing equipment.   The image processing apparatus according to claim 7, wherein the spectrum evaluation unit evaluates the estimation accuracy of the second spectrum based on the pigment amount estimated by the pigment amount estimation unit.   The second spectrum estimation means includes an estimation operator calculation means for calculating an autocorrelation matrix using the adaptive teacher data and calculating an estimation operator based on the calculated autocorrelation matrix, and calculating the estimation operator The image processing apparatus according to claim 1, wherein the second spectrum is estimated using an estimation operator calculated by a unit. An image processing program for causing a computer to estimate a spectrum of the stained specimen from a stained specimen image obtained by imaging the stained specimen,
Based on the pixel values of the estimation target pixels constituting the stained specimen image, using standard teacher data set in advance, the spectrum at the sample point on the stained specimen corresponding to the estimation target pixel is set as the first spectrum. A first spectral estimation step to estimate;
Teacher data generation step for generating adaptive teacher data adapted to the estimation target pixel based on the first spectrum;
A second spectrum estimation step of estimating a spectrum at the sample point as a second spectrum using the adaptive teacher data based on the pixel value of the estimation target pixel;
An image processing program for causing the computer to execute. An image processing method for estimating a spectrum of the stained specimen from a stained specimen image obtained by imaging the stained specimen,
Based on the pixel values of the estimation target pixels constituting the stained specimen image, using standard teacher data set in advance, the spectrum at the sample point on the stained specimen corresponding to the estimation target pixel is set as the first spectrum. A first spectral estimation step to estimate;
Teacher data generation step of generating adaptive teacher data adapted to the estimation target pixel based on the first spectrum;
A second spectrum estimation step of estimating a spectrum at the sample point as a second spectrum using the adaptive teacher data based on a pixel value of the estimation target pixel;
An image processing apparatus comprising: An image acquisition means for acquiring a stained specimen image by imaging a stained specimen using a microscope;
Based on the pixel value of the estimation target pixel constituting the stained sample image, the spectrum at the sample point on the stained sample corresponding to the estimation target pixel is set as the first spectrum using preset standard teacher data. First spectrum estimating means for estimating;
Teacher data generation means for generating adaptive teacher data adapted to the estimation target pixel based on the first spectrum;
Second spectrum estimation means for estimating a spectrum at the sample point as a second spectrum using the adaptive teacher data based on the pixel value of the estimation target pixel;
A virtual microscope system comprising:

Images