JP2006126374A - Image display method, program, and scanning confocal microscope - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional image display method or the like permitting easy recognition by a simple operation. <P>SOLUTION: A CPU 10 detects information on the brightness of a brightest pixel among those pixels which constitute a plurality of sliced images obtained while repeating relative movement between a sample 5 and an objective lens 4 and have the same coordinates showing the pixel positions in the sliced images, and further detects information on the height of the sample face based on a distance between the sample 5 and the objective lens 4 when the maximum brightness is obtained. Then, image processing selected according to the shape of the sample face is applied to the information of the detected height, and the three-dimensional image of the sample 5 is produced and displayed on a display part 13 based on the image-processed height information and the detected brightness information. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique of a scanning confocal microscope, and more particularly to a technique of displaying the image in a scanning confocal microscope that can display a three-dimensional image of a sample.

  A scanning confocal microscope uses a point light source to scan the sample surface. Of the reflected light from the sample surface, only the light that has passed through the pinhole is converted into an electrical signal by the photodetector, and the information on the sample surface is obtained. The three-dimensional information on the sample surface can be obtained by sequentially acquiring the information on the sample surface while moving either the sample or the objective lens in the optical axis direction. .

As a method for displaying such three-dimensional information as a three-dimensional image that can be easily recognized by an observer, for example, Patent Document 1 discloses color information at the time of three-dimensional display assigned according to luminance information as height information. A technique for setting and displaying as a three-dimensional display image is disclosed.
JP 2001-56348 A

  However, in the three-dimensional image displayed in this way, noise may appear when the amount of reflected light at the time of detection is small, for example, on the slope of the sample shape, and the information on the sample surface is recognized as it is. There is a problem that it is difficult.

  Conventionally, it has been necessary for an observer to combine and execute individual image processing in order to cleanly remove noise from an image including such noise. Furthermore, since the optimum type and combination of image processing differ depending on the sample shape, the observer is forced to perform a complicated operation in order to obtain a desired result.

  The present invention has been made in view of the above-described problems, and a problem to be solved is to provide a three-dimensional image that can be easily recognized by a simple operation.

  An image display method according to one aspect of the present invention includes moving at least one of a sample and an objective lens in the optical axis direction, and irradiating the sample with spot light through the objective lens. A method of displaying the image in a scanning confocal microscope that generates an image of the sample based on reflected light from the sample surface received when light is scanned on the sample surface of the sample, Among pixels constituting a plurality of slice images obtained by repeating relative movement between the sample and the objective lens and having the same coordinates indicating the positions of the pixels in the slice images, those having the maximum luminance The information indicating the brightness is detected, and the height information of the sample surface is calculated based on the distance between the sample and the objective lens when the maximum brightness is obtained. The detected height information is subjected to image processing selected according to the shape of the sample surface, and the height information and the detection after the image processing are performed. Based on the luminance information, a three-dimensional image of the sample is generated and displayed on the display unit. This feature solves the above-described problem.

  In the image display method according to the present invention described above, a designation unit for designating the shape of the sample surface is displayed on the display unit, and in advance according to an instruction from the designation unit for designating the shape of the sample surface. Image processing that is optimal for the set shape of the sample surface may be selected.

  In the image display method according to the present invention described above, the selection according to the shape of the sample surface is made in advance, and when the detection of the luminance information and the height information is completed, the selection is immediately made according to the result of the selection. The image processing may be started continuously.

  Further, in the image display method according to the present invention described above, luminance enhancement processing is performed on the detected luminance information, and in the generation of the three-dimensional image of the sample, the luminance information subjected to the luminance enhancement processing is applied. Based on this, the three-dimensional image may be generated.

Note that a program for causing a computer to perform the above-described image display method according to the present invention also relates to the present invention.
In addition, a scanning confocal microscope according to another aspect of the present invention moves at least one of a sample and an objective lens in the optical axis direction, and transmits spot light via the objective lens. A scanning confocal microscope that generates an image of the sample based on reflected light from the sample surface received when the spot light is scanned on the sample surface of the sample while irradiating Among pixels constituting a plurality of slice images obtained by repeating relative movement between the sample and the objective lens and having the same coordinates indicating the positions of the pixels in the slice images, those having the maximum luminance First detection means for detecting information indicating the brightness, and the height of the sample surface based on the distance between the sample and the objective lens when the maximum brightness is obtained. Second detection means for detecting information, image processing means for performing image processing selected according to the shape of the sample surface for the detected height information, and after the image processing has been performed Three-dimensional image generation means for generating a three-dimensional image of the sample based on the height information and the detected luminance information and displaying the three-dimensional image on the display unit, and This feature solves the aforementioned problems.

  The present invention has an effect that it is possible to provide a three-dimensional image that can be easily recognized by a simple operation.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows the configuration of a scanning confocal microscope embodying the present invention.
In the scanning confocal microscope (hereinafter referred to as “this apparatus”) shown in FIG. 1, the light source 1 is a point light source such as a laser beam. Spot light from the light source 1 is guided to the two-dimensional scanner 2. The two-dimensional scanner 2 is used for two-dimensionally scanning the spot light from the light source 1 on the sample 5, and a mirror for X-axis direction scanning and a Y-axis direction that is orthogonal to the X-axis. And a scanning mirror.

  The spot light that has passed through the two-dimensional scanner 2 is changed in the traveling direction by the dichroic mirror 3, passes through the objective lens 4, is irradiated onto the sample 5 on the stage 6, and is scanned two-dimensionally on the sample 5. The stage 6 is configured to be movable in the Z-axis direction, and the sample 5 can be moved in the optical axis direction of the spot light. The stage 6 may be fixed and the objective lens 4 side may be movable with respect to the Z-axis direction.

  Reflected light from the sample 5 is returned to the two-dimensional scanner 2 by the dichroic mirror 3 through the objective lens 4, passes through the two-dimensional scanner 2, passes through the pinhole 7, and is guided to the photodetector 8.

  The photodetector 8 converts the reflected light into an electrical signal corresponding to the amount of light. The electrical signal output from the photodetector 8 is converted into digital data by the A / D converter 9 and input to the CPU 10.

  The CPU 10 in the processing device 20 is connected to the two-dimensional scanner 2 and the stage 6 which are the targets of operation control, and further includes a memory 11 used as a work storage area for control processing, and a generated image. A frame memory 12 for temporarily storing data is connected. A display unit 13 is connected to the frame memory 12.

  A color camera 14 is also connected to the CPU 10. The color camera 14 obtains color information of the sample by detecting reflected light from the sample 5 irradiated with light from a white light source (not shown).

  The CPU 10 executes various control processes by executing a control program prepared in advance, for example, scanning operation of the laser beam with respect to the two-dimensional scanner 2 and scanning of the stage 6 in the Z-axis direction (height direction). Operation control processing or acquisition processing of luminance information of a pixel with the maximum light amount and height information at the maximum light amount based on the amount of reflected light from the sample 4 detected by the photodetector 8, Based on the generation process of the three-dimensional image based on it.

  The memory 11 stores in advance a color table in which a predetermined color is assigned according to the light amount (luminance) in advance while storing the peak light amount information and height information obtained by the CPU 10.

The display unit 13 reads the image data generated by the CPU 10 and stored in the frame memory 12 and displays a three-dimensional image represented by the image data.
As the processing device 20, a computer system having a very standard configuration can be used. The stage 6 is moved with respect to the scanning of the objective lens 4 and the sample 5 in the Z-axis direction, but the objective lens 4 may be moved.

  Next, various control processes performed by the CPU 10 will be described. The control related to various processes described below is realized by causing the CPU 10 to execute a predetermined control program.

FIG. 2 is a flowchart showing the contents of the 3D display information setting process.
Explaining the variables used in the figure, i and j represent the X and Y coordinates (i = 1, 2, 3,..., Ni, j = 1, 2, 3,..., Nj) of the pixel, respectively. k represents the position in the height direction of the stage 6 (k = 1, 2, 3,..., Nk), and a (i, j, k) represents light amount data for each pixel corresponding to each coordinate of the slice image. P (i, j) represents the luminance information of the pixel with the maximum light amount. Further, h (i, j) is distance information between the sample 5 and the objective lens 4 when the light amount is maximum, and here, the height direction of the stage 6 when the light amount is maximum. Location information.

  First, in S201, an initialization process is performed. By this processing, each of the light amount data a (0, 0, 0) of the slice image, the luminance information p (0, 0) of the pixel with the maximum light amount, and the height position information h (0, 0) at the maximum light amount. 0 is substituted as an initial value.

  In S202, processing for determining whether or not the height position k of the current stage 6 is the maximum position, specifically, processing for determining whether or not k = Nk is performed. Here, when it is determined that the height position k is not the maximum position (when the determination result is No), the process proceeds to S203, and when it is determined that the height position k is the maximum position (the determination result is If yes), the process proceeds to S207.

  In S203, a process of acquiring light amount data a (i, j, k) corresponding to the electrical signal output from the photodetector 8 is performed for all the pixels of the slice image of the sample 5 at the height position k.

  In S204, the light intensity data a (i, j, k) of all the pixels of the slice image of the sample 5 at the height position k is the luminance of the pixel when the light intensity is the maximum for each pixel of the same coordinate. Compared with the information p (i, j), a process is performed to determine whether there is a material that satisfies a (i, j, k)> p (i, j). Here, only when it is determined that the inequality is satisfied (only when the determination result is Yes), the process proceeds to S205, and the luminance information p (i, j) is converted into the maximum light amount data a ( In addition to updating to the contents of i, j, k), a process of storing the height position k of the stage 6 at this time as height information h (i, j) at the maximum light amount is performed.

  In S206, the height position k of the stage 6 is updated, that is, the value of k is incremented (the result of adding 1 to the value of k is substituted for the variable k), and then the process proceeds to S202. The same operation is repeated for the slice image at the next height position of the stage 6.

  The above process is repeated until the height position k of the stage 6 is determined to be the maximum position in the determination process of S202 (until the determination result is Yes), and each of the plurality of slice images acquired by this repetition. As for all the pixels constituting the pixel, luminance information p (i, j) of the pixel when the light amount is maximum and height position information h (i, j) when the light amount is maximum are acquired.

  After that, if k = Nk in S202 and the determination result is Yes, the process proceeds to S207, and the three-dimensional display information is set based on the luminance information p and the height position information h obtained by the operation so far. Processing is performed.

  Specifically, as shown in FIG. 3, for the pixel located at the coordinates (x, y), the height position information h (x, y) at this pixel position (x, y) is extracted, and this height is obtained. Position information h (x, y) is set as height information z (x, y) at the time of three-dimensional display. Subsequently, luminance information p (x, y) at this pixel position (x, y) is taken out, and color information c (x, y, z) at the time of three-dimensional display set in advance is based on this information. To set. That is, the color information corresponding to p (x, y) is set as c (x, y, z). Here, a color table to which predetermined color information is assigned in association with the luminance information p is prepared in advance in the memory 11, and color information c (x, y, z at the time of three-dimensional display is used by using this table. ) Is set.

  In S208, a process for determining whether or not the setting of the three-dimensional display information by the process of S207 has been performed for all the pixels constituting the image is performed, and when it is determined that the setting has been performed for all the pixels (determination result) 2 is ended). On the other hand, when it is determined that an unset pixel remains (when the determination result is No), the process returns to S207 and the process of setting the 3D display information for the remaining pixels is repeated. .

The above processing is the 3D display information setting processing.
Next, FIG. 4 will be described. This figure is an example of a process execution button used to obtain an image processing selection instruction to be described from the user of this apparatus. A GUI (Graphical User Interface) displayed on the display unit 13 by the CPU 10 is shown in FIG. One of them.

  As can be seen from FIG. 4, these processing execution buttons each represent the shape of the sample surface of the sample 5. For example, the leftmost button is effective when the surface of the sample 5 is uneven. An image processing sequence is assigned, the second button from the left is an image processing sequence that is effective when the surface of the sample 5 is saw-shaped, and the third button from the left is the surface of the sample 5 Image processing sequences that are effective in the case of a planar shape are incorporated. The rightmost button is a button for returning the image processing sequence to an unselected state.

  When the user of the present apparatus operates an input unit such as a mouse (not shown) provided in the processing apparatus 20 and any one of the left three of these process execution buttons is operated, the above-described tertiary For the original display height information z (x, y), an image processing sequence as shown in the flowchart in FIG. 5 is selected and executed.

  FIG. 5 is a first example of an image processing sequence in the case where the surface of the sample 5 has an uneven shape, and is performed by the CPU 10 when the leftmost button of the processing execution buttons shown in FIG. 4 is operated. It is processing.

  First, in S301, a process for performing low-luminance correction processing on the height information z (x, y) is performed. The low luminance portion correction process is a process for removing height noise. The contents of this process will be described with reference to FIG.

  In FIG. 6, (a) shows an example of luminance information p (x, y) of the pixel having the maximum light quantity obtained from the sample 5, and x and p (x, y) when y = C (constant). This shows the relationship with y). At this time, if no correction processing is performed, the luminance information of the pixels in the noise region (pixels whose luminance is lower than the noise level) is so large that the noise component included in the measurement value cannot be ignored. Will become bigger. Then, the height information z (x, y) obtained by the above-described processing of FIG. 2 becomes discontinuous in the noise region as shown in (b), and the generated three-dimensional image of the sample 5 is obtained. Will be unnatural.

  Therefore, for the height information z (x, y) at a position where the luminance is equal to or lower than the noise level, the luminance information exceeds the noise level among the pixels at the surrounding positions (for example, adjacent eight coordinate positions). Use the height information of what you have. This is the low luminance portion correction process.

  An example of the height information z (x, y) obtained by performing this process on the measurement data shown in FIG. 6A is shown in FIG. In this way, by applying the low-luminance correction process, a three-dimensional image of a region that is optically low-reflective, such as a sloped portion of the sample 5 that has a concavo-convex shape, whose luminance can be lower than the noise level The shape can be made appropriate.

  In S302, a process of further applying the first smoothing filter is performed on the height information z (x, y) after the low luminance portion correction process is performed. The first smoothing filter is a smoothing filter that retains the contour of the shape, for example, a median (median) filter. Here, the filter size (range in which data to be input to the filter is located) is set to a larger size ( For example, a 5 × 5 range centered on coordinates (x, y). By this processing, it is possible to remove wrinkle-like noise that may occur in the flat portion of the three-dimensional image of the sample 5 that has an uneven shape.

  In S303, a process of further applying the second smoothing filter is performed on the height information z (x, y) after the first smoothing filter is applied. The second smoothing filter is an average filter, and here, the filter size is a small size (for example, a 3 × 3 range centered on coordinates (x, y)). By this processing, the state of the slope portion of the three-dimensional image of the sample 5 having the uneven shape can be made smoother.

  The CPU 10 uses the height information z (x, y) obtained by executing the processing of FIG. 5 and the previously set color information c (x, y, z) to render the three-dimensional image. Is displayed on the display unit 13 to obtain a three-dimensional image of the sample 5 in which the noise component is well suppressed.

  By the way, the contents of the process in each step shown in FIG. 5 are changed depending on the selection of the process execution button described above, that is, the shape of the sample 5. For example, for the sample 5 having a saw shape, a filter that emphasizes isolated point removal in the process of S301, for example, when the height information of the target pixel protrudes from the average of the height information of adjacent pixels. A filter that performs processing for replacing the height information of the target pixel with the average value is applied. For the sample 5 having a flat surface, for example, the process of S302 is skipped and the process of the average filter of S303 is executed.

FIG. 7B shows an image display example that is three-dimensionally displayed on the display unit 13 in this way, compared with FIG. 7A that is an image display example when the processing of FIG. 5 is not executed.
In the above-described embodiment, the three processes from S301 to S303 are executed. However, the number of processes is not limited to three, and may be increased or decreased. Further, the number of processing execution buttons shown in FIG. 4 may be increased so that image processing according to other shapes of the sample surface of the sample 5 may be performed.

  In the above-described embodiment, the CPU 10 executes the image processing sequence selected by the processing execution button for each shape of the sample 5 arranged on the GUI as shown in FIG. In addition, the selection of the shape of the sample 5 is registered in advance in the processing apparatus 20 of FIG. 1, and after the luminance information p and the height information z are acquired, the image processing sequence is automatically selected based on the registered contents. A selection may be made and executed. That is, for example, the button shown in FIG. 4 is operated as a switching button, and the user of this apparatus selects one of the sample shapes in advance before starting the processing of FIG. To leave. When the user gives an instruction to execute image capture, the CPU 10 that has acquired the instruction first executes the process shown in FIG. 2, and then immediately starts executing the process shown in FIG. By doing so, it is possible to reduce the burden on the operator when continuously observing a plurality of samples 5 having the same type of shape.

  In the above-described embodiment, the image processing sequence shown in FIG. 5 is performed only on the height information z. However, the luminance information p is also processed to improve the quality of the three-dimensional image of the sample 5. It can also be improved.

FIG. 8 is a second example of an image processing sequence in the case where the surface of the sample 5 has an uneven shape, and the luminance information p is also subjected to processing for improving the quality of the three-dimensional image of the sample 5.
The sequence of FIG. 8 is also performed by the CPU 10 when the leftmost button of the process execution buttons shown in FIG. 4 is operated.

In FIG. 8, the processing from S401 to S403 is the same as that from S301 to S303 shown in FIG.
In S404, processing for applying the image enhancement filter is performed on the luminance information p (x, y), and the color information corresponding to p (x, y) after application is set as c (x, y, z). To do.

  Note that the types of image enhancement filters applied by the processing of S404 include, for example, a sharp filter, an edge enhancement filter, a contrast enhancement filter, and the like. Further, these filters may be arbitrarily combined.

  The CPU 10 performs a three-dimensional image rendering process using the height information z (x, y) obtained by executing the processing of FIG. 8 and the color information c (x, y, z) set in advance. Is displayed on the display unit 13 to obtain a three-dimensional image of the sample 5 in which the noise component is well suppressed. In addition, since the luminance information is emphasized in the three-dimensional image displayed on the display unit 13 at this time, it is combined with the effect of removing noise on the sample 5 having the same reflectance. Visual recognition of the surface shape is facilitated.

FIG. 9B shows an image display example displayed three-dimensionally on the display unit 13 in this way, compared with FIG. 9A, which is an image display example when the processing of FIG. 8 is not executed.
As described above, according to the present embodiment, the three-dimensional image of the sample 5 from which noise has been removed can be displayed with a simple operation, and the sample surface can be observed in an optimal state that is easy to recognize. It becomes possible.

  Incidentally, as described above, in order for the CPU 10 to perform the control processing shown in FIGS. 2, 5, and 8, it is necessary to cause the CPU 10 to execute a predetermined control program. For this purpose, for example, the control program may be stored in advance in the memory 11 of the processing device 20, and the control program may be read from the storage unit and executed. Instead, the control program is recorded on a recording medium that can be read by the computer system that is the processing device 20, and the control program is stored in the data reading device (not shown) of the processing device 20 from the recording medium. It may be read and executed. Here, as a recording medium from which the recorded control program can be read by the processing device 20, a portable type such as an FD (flexible disk), an MO (magneto-optical disk), a CD-ROM, a DVD-ROM or the like can be used. Recording media can be used.

  The recording medium may be a storage device provided in a computer functioning as a program server connected to a computer system that is the processing device 20 via a communication network that is a data transmission medium. In this case, a transmission signal obtained by modulating a carrier wave with a data signal representing the control program is transmitted from the program server through a communication network, and the processing device 20 demodulates the received transmission signal. The control program can be executed by reproducing the original control program.

  In addition, the present invention is not limited to the above-described embodiments, and various improvements and changes can be made without departing from the scope of the present invention.

It is a figure which shows the structure of the scanning confocal microscope which implements this invention. It is a flowchart which shows the processing content of a three-dimensional display information setting process. It is a figure explaining the symbol used by the process of FIG. It is a figure which shows the example of a process execution button. It is a flowchart which shows the 1st example of an image processing sequence. It is a figure explaining the processing content of a low-intensity part correction process. It is a figure which shows the example of the three-dimensional image of the sample when not performing the image processing sequence of FIG. It is a figure which shows the example of the three-dimensional image of the sample at the time of performing the image processing sequence of FIG. It is a flowchart which shows the 2nd example of an image processing sequence. It is a figure which shows the example of the three-dimensional image of the sample when not performing the image processing sequence of FIG. It is a figure which shows the example of the three-dimensional image of the sample at the time of performing the image processing sequence of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Light source 2 Two-dimensional scanner 3 Dichroic mirror 4 Objective lens 5 Sample 6 Stage 7 Pinhole 8 Photo detector 9 A / D converter 10 CPU
11 Memory 12 Frame Memory 13 Display 14 Color Camera 20 Processing Device

Claims (9)

When at least one of the sample and the objective lens is moved in the optical axis direction, and the spot light is scanned on the sample surface of the sample while irradiating the sample with the spot light through the objective lens A method of displaying the image in a scanning confocal microscope that generates an image of the sample based on reflected light from the sample surface that is received,
Among pixels constituting a plurality of slice images obtained by repeating relative movement between the sample and the objective lens, and having the same luminance indicating the position of the pixel in the slice image, the luminance is maximum Detection of information indicating the brightness of
Detecting the height information of the sample surface based on the distance between the sample and the objective lens when the maximum luminance is obtained;
For the detected height information, perform image processing selected according to the shape of the sample surface,
Generating a three-dimensional image of the sample based on the height information after the image processing and the detected luminance information and displaying them on a display unit;
An image display method characterized by the above. Display a designation part for designating the shape of the sample surface on the display part,
In accordance with an instruction from the designation unit for designating the shape of the sample surface, an image processing optimal for the shape of the sample surface set in advance is selected.
The image display method according to claim 1.   Selection according to the shape of the sample surface is made in advance, and when the detection of the luminance information and the height information is completed, preset image processing is immediately started according to the result of the selection. The image display method according to claim 1. A luminance enhancement process is performed on the detected luminance information,
In the generation of the three-dimensional image of the sample, the three-dimensional image is generated based on the luminance information subjected to the luminance enhancement processing.
The image display method according to claim 1. When at least one of the sample and the objective lens is moved in the optical axis direction, and the spot light is scanned on the sample surface of the sample while irradiating the sample with the spot light through the objective lens A program for causing a computer to display the image obtained by a scanning confocal microscope that generates an image of the sample based on the reflected light from the sample surface that is received,
Among pixels constituting a plurality of slice images obtained by repeating relative movement between the sample and the objective lens, and having the same luminance indicating the position of the pixel in the slice image, the luminance is maximum Detecting information indicating the brightness of
A process for detecting height information of the sample surface based on a distance between the sample and the objective lens when the maximum luminance is obtained;
A process of performing image processing selected according to the shape of the sample surface with respect to the detected height information;
A process of generating a three-dimensional image of the sample based on the height information after the image processing and the detected luminance information and displaying the three-dimensional image on a display unit;
A program that causes a computer to perform Further causing the computer to perform a process of displaying on the display unit a designation unit for designating the shape of the sample surface,
In accordance with an instruction from a designation unit for designating the shape of the sample surface, an image processing optimal for the shape of the sample surface set in advance is selected.
The program according to claim 5.   Selection according to the shape of the sample surface is made in advance, and when the detection of the luminance information and the height information is completed, preset image processing is immediately started according to the result of the selection. The program according to claim 5. Causing the computer to further perform a process of performing a brightness enhancement process on the detected brightness information;
In the generation of the three-dimensional image of the sample, the three-dimensional image is generated based on the luminance information subjected to the luminance enhancement processing.
The program according to claim 5. When at least one of the sample and the objective lens is moved in the optical axis direction, and the spot light is scanned on the sample surface of the sample while irradiating the sample with the spot light through the objective lens A scanning confocal microscope that generates an image of the sample based on reflected light from the sample surface that is received,
Among pixels constituting a plurality of slice images obtained by repeating relative movement between the sample and the objective lens, and having the same luminance indicating the position of the pixel in the slice image, the luminance is maximum First detection means for detecting information indicating the brightness of
Second detection means for detecting height information of the sample surface based on a distance between the sample and the objective lens when the maximum luminance is obtained;
Image processing means for performing image processing selected according to the shape of the sample surface with respect to the detected height information;
3D image generating means for generating a 3D image of the sample based on the height information after the image processing and the detected luminance information and displaying the 3D image on a display unit;
A scanning confocal microscope characterized by comprising: