JP2005331488A - Magnified observation device, magnified image observation method, magnified observation operation program, and computer-readable storage medium or storage device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a magnified observation device and the like, capable of easily recognizing the height relationship in comparison between a plurality of three-dimensional images. <P>SOLUTION: This magnified observation device is provided with an allocation part 512, which measures the distribution of height information included in a three-dimensional image, associates an expression pattern of a pixel expressible by a display part with the distribution of the height information, and allocates correlation between the height and the expression pattern for expressing a change in the height information by the change in the expression pattern, and an expression pattern image generating part allocating each expression pattern, based on the correlation by the allocation part 512 for each height information included in the thee-dimensional image and creating a three-dimensional expression pattern image that matches the height. The display part can display a plurality of three-dimensional images, at the same time, and in a plurality of three-dimensional images, the correlation between the height and the expression pattern in creation of each three-dimensional expression pattern image is shared among all the three-dimensional images in the expression pattern image generating part. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a magnification observation apparatus such as a microscope or a microscope that captures and displays an enlarged image, a magnified image observation method, an operation program for magnification observation, and a computer-readable recording medium or recorded apparatus. The present invention relates to an improvement in a technique for visually displaying a three-dimensional shape of a sample from data.

  Today, an optical microscope using an optical lens, a digital microscope, and the like are used as a magnifying observation device that magnifies and displays a minute object. The microscope includes a light receiving element such as a CCD that electrically reads reflected light or transmitted light from the observation target fixed to the observation target fixing unit incident via the optical system for each pixel arranged in a two-dimensional manner. Prepare. An image electrically read using a CCD is displayed on a display unit such as a display (see, for example, Patent Document 1).

  In a magnifying observation device, in general, when the magnifying power increases, the depth of focus becomes shallower and the in-focus area becomes narrower. Therefore, it is difficult to observe the whole if there are irregularities or height differences in the sample (workpiece) to be observed. Become. For this reason, a technique is used in which an image focused on the entire image is created by depth synthesis. In depth synthesis, the lens or sample is moved in the height direction, the relative distance in the optical axis direction is changed, multiple images are taken, and the in-focus part is extracted and synthesized, resulting in a deep focus depth. Take an image. In addition, when shooting a plurality of images, if the amount of movement of the lens or the observation target is recorded at the same time, the height information of the observation target can be obtained when creating the composite image. It is also possible to construct as three-dimensional data. By reproducing a three-dimensional image on the display unit of the magnifying observation device based on the three-dimensional data, it is possible to observe and evaluate the image three-dimensionally from various viewpoints and angles, and to measure the surface shape such as the height. it can.

  In addition, the three-dimensional image obtained in this way may be difficult to grasp only by looking at the difference in height depending on the pattern and shape of the captured surface. For this reason, in order to make it easy to understand the height of a three-dimensional image sensuously, a technique has been developed in which different colors are colored according to the height and the difference in height is indicated by a change in color (for example, a patent). Reference 2). In this display device, in order to visually grasp the difference in the height of the sample to be observed, the height range is divided by a plurality of threshold values, and the display colors are respectively displayed for the divided height ranges. In addition, the sample is colored and displayed in a one-to-one correspondence with the display density. For example, by coloring the three-dimensional display of the surface shape using a dark color for the low part and a bright color for the high part, and coloring with a shade pattern or color gradation according to the height. The sample surface can be colored and displayed in a state color-coded by the pattern. This makes it easy for the user to image the height relationship of each part, and thus the surface profile, from the color pattern of the sample.

  On the other hand, the applicant of the present application has previously developed a magnifying observation apparatus that can display a plurality of three-dimensional images on the same screen and easily compare them (Japanese Patent Application No. 2003-329024). By applying a coloring process corresponding to the height of each of the plurality of three-dimensional images displayed by this apparatus, it is possible to compare them while recognizing the height information of the plurality of three-dimensional images.

However, in the conventional apparatus, an independent coloring process is individually performed for each three-dimensional image. In order to increase the color dynamic range as much as possible to increase the expressive power due to the difference in color, the coloring process usually has the highest and lowest positions as the upper and lower limits of the height information contained in the three-dimensional image. A height range is set, and the colors are equally assigned to the height range. In this method, when observing one three-dimensional image, all colors that can be used are assigned to the entire height range, and display with high expressive power due to the difference in colors is possible. However, when such a process is performed for each image when displaying a plurality of three-dimensional images, the height to which the color is assigned differs for each image. In this case, although the individual height change for each three-dimensional image can be grasped, there is a problem that in comparison between different three-dimensional images, since the heights indicated by the same color are different, it is difficult to compare.
JP 2000-214790 A Japanese Patent Laid-Open No. 11-287627

  The present invention has been made to solve such problems. A main object of the present invention is to provide a magnification observation apparatus, a magnification image observation method, a magnification observation operation program, and a magnified observation operation program capable of displaying each height relatively easily when comparing a plurality of three-dimensional images. An object is to provide a computer-readable recording medium or a recorded device.

  In order to achieve the above object, a magnification observation apparatus of the present invention includes an imaging unit for imaging an observation target, a control unit that generates a three-dimensional observation image based on a signal acquired by the imaging unit, A display unit for displaying a three-dimensional observation image generated by the control unit. This magnified observation device is a pixel in which a three-dimensional image includes height information, and further has a certain variable range that can measure the distribution of height information of three-dimensional image data and can be expressed by a display unit. Each of the height information of the three-dimensional image, and an assigning unit that assigns a correspondence relationship between each height and the expression pattern so that the expression pattern can be associated with the distribution of the height information and the change in the height information can be expressed by the change in the expression pattern And an expression pattern image generation unit that assigns each expression pattern based on the correspondence of the assignment unit and generates a three-dimensional expression pattern image corresponding to the height. Furthermore, the display unit can simultaneously display a plurality of three-dimensional images. In addition, with respect to a plurality of three-dimensional images, the correspondence relationship between each height and the expression pattern when the expression pattern image generation unit generates each three-dimensional expression pattern image is common to all three-dimensional images. Thereby, since the expression pattern of each three-dimensional image is displayed with the same expression pattern for the same height, it is possible to easily grasp the relative height relationship when comparing a plurality of three-dimensional images.

  Further, another magnifying observation apparatus of the present invention is generated by an imaging unit for imaging an observation target, a control unit that generates a three-dimensional observation image based on a signal acquired by the imaging unit, and a control unit. And a display unit for displaying a three-dimensional observation image. In this magnifier, the 3D image contains height information, and based on the distribution of the height information of the 3D image data, the maximum and minimum heights can be detected and expressed on the display unit. Pixel representation pattern, a range of expression patterns with a maximum value and a minimum value is associated with the distribution of height information, and the maximum height and minimum height are represented by the maximum and minimum values of the expression pattern. Corresponding to each of the heights and the expression patterns, and assigning each expression pattern based on the correspondence of the assignment unit for each height information of the three-dimensional image. An expression pattern image generation unit that generates a three-dimensional expression pattern image to be generated. The display unit can simultaneously display a plurality of three-dimensional images. Furthermore, for a plurality of three-dimensional images, the corresponding relationship between each height and the expression pattern when the expression pattern image generation unit generates each three-dimensional expression pattern image is common to all three-dimensional images. Thereby, since the expression pattern of each three-dimensional image is displayed with the same expression pattern for the same height, the relative height relationship can be easily grasped when comparing a plurality of three-dimensional images. By making the maximum height and minimum height correspond to the maximum value and minimum value of the expression pattern, it is possible to effectively use the range of the expression pattern and enhance the expressive power.

  Furthermore, another magnifying observation device according to the present invention is based on a distribution of height information included in a reference three-dimensional image among a plurality of three-dimensional images when the assigning unit assigns the correspondence of the expression patterns. The maximum height and the minimum height are detected, the expression pattern in a certain range is associated with the distribution of the height information, and the maximum height and the minimum height included in one three-dimensional image are set as the maximum value of the expression pattern. A correspondence relationship between each height and the expression pattern is assigned so as to correspond to each minimum value, this correspondence relationship is executed for all three-dimensional images, and the expression pattern image generation unit performs the three-dimensional expression according to the unified correspondence relationship. A pattern image is generated. Thus, since the correspondence is determined based on the height distribution of one reference three-dimensional image, the same height can be displayed in the same expression pattern based on the reference three-dimensional image.

  Furthermore, in another magnifying observation device of the present invention, the reference three-dimensional image is a three-dimensional image having a maximum difference between the maximum height and the minimum height among a plurality of three-dimensional images. As a result, the expression pattern is assigned in accordance with the three-dimensional image having the highest difference in height, so that in the relative height measurement, all the heights can be expressed by the expression pattern, and reliable contrast observation can be performed.

  Still further, in another magnifying observation device of the present invention, when the assigning unit assigns the correspondence of the expression pattern, the maximum height and the minimum height are based on the distribution of all the height information included in the plurality of three-dimensional images. Is detected, and the expression pattern in a certain range is made to correspond to the distribution of the height information, and the maximum height and the minimum height among the heights included in all the three-dimensional images are respectively set to the maximum value and the minimum value of the expression pattern. The correspondence between each height and the expression pattern is assigned so as to correspond, and this correspondence is executed for all three-dimensional images, and the expression pattern image generation unit generates a three-dimensional expression pattern image according to the unified correspondence. To do. As a result, since the correspondence is determined based on all the height distributions of the three-dimensional images to be compared, all the heights can be expressed in the expression pattern even in the absolute height measurement, and the same height is the same. Can be displayed as an expression pattern.

  Furthermore, in another magnified observation device of the present invention, the expression pattern is at least one of color and luminance. Thereby, since the height of each three-dimensional image is displayed with the same color or brightness for the same height, it is possible to easily grasp the relative height relationship when comparing a plurality of three-dimensional images.

  Furthermore, another magnifying observation device of the present invention is generated by an imaging unit for imaging an observation target, a control unit that generates a three-dimensional observation image based on a signal acquired by the imaging unit, and a control unit. And a display unit for displaying the obtained three-dimensional observation image. This magnification observation device detects the maximum height and the minimum height of a three-dimensional image based on a storage unit that holds three-dimensional image data including height information and a distribution of height information of the three-dimensional image data. The color palette that expresses the height by displaying the gradation of colors in the predetermined range is associated with the distribution of height information, and the maximum height and minimum height are the maximum and minimum colors in the color palette range. An assigning unit for assigning the correspondence between each height and the color so as to correspond to each color, and for each height information of the three-dimensional image, a color is assigned based on the correspondence between the assigning units and corresponds to the height. A colored image generation unit that generates a three-dimensional colored image. Also, for a plurality of three-dimensional images, the assigning unit determines a correspondence relationship based on the distribution of the height information based on the reference three-dimensional image as a reference. A plurality of three-dimensional colored images that are executed and generated in accordance with the unified correspondence are configured to be displayed on the display unit. As a result, the height is expressed by the color, and each three-dimensional colored image displays the same height in the same color, so it is easy to grasp the relative height relationship when comparing a plurality of three-dimensional images. it can.

  Furthermore, according to another magnifying observation apparatus of the present invention, a color gauge indicating a distribution of colors assigned to the three-dimensional image is assigned to each color in a display area that displays each three-dimensional image in the display unit. It can be displayed together with the representative height value. Thereby, the range of the color palette used for each three-dimensional image is displayed together with the three-dimensional colored image, and the user can easily understand the height distribution in the three-dimensional image. Also, the height is displayed in the color palette, so the height assigned to each color is known, and the relationship between the color and the height can be visually understood.

  Furthermore, in another magnifying observation apparatus of the present invention, the display unit can display a three-dimensional colored image superimposed on each three-dimensional image, and the three-dimensional colored image displayed superimposed on each three-dimensional image is A transmissivity adjusting unit capable of providing translucency and capable of adjusting the transmissivity of the three-dimensional colored image is provided. As a result, the original three-dimensional image and the height information by color can be displayed at the same time, and the ratio can be changed to an appropriate ratio, so that the display state can be adjusted according to the user's preference and observation purpose.

  Furthermore, in another magnification observation apparatus of the present invention, the three-dimensional image data further includes information regarding the size of the three-dimensional image, and further, based on information regarding the size of the reference three-dimensional image. And a magnification adjustment unit for adjusting the display magnification of the three-dimensional image to be compared to the same magnification. Accordingly, the display magnification of the three-dimensional image to be compared with the reference three-dimensional image is enlarged or reduced, and a process for aligning the size reference is performed in advance, so that a more accurate comparison can be performed. .

  Furthermore, when another magnifying observation apparatus of the present invention displays a plurality of three-dimensional images on the display unit, the three-dimensional image to be compared is based on the viewpoint of the object displayed in the reference three-dimensional image. A viewpoint adjustment unit is provided for adjusting the viewpoint of the object displayed in the image so as to coincide with that of the reference three-dimensional image. This provides an advantage that different three-dimensional images can be observed in a state where they are matched to the same viewpoint.

  Furthermore, another magnifying observation apparatus of the present invention is configured such that when a plurality of three-dimensional images are superimposed by the viewpoint adjustment unit, a height that is a reference for superposition can be designated. Accordingly, the reference position in the height direction of the three-dimensional image to be superimposed can be adjusted, and the image can be easily superimposed even on an image in which a positional deviation or offset occurs in the height direction.

  Furthermore, another magnification observation apparatus of the present invention assigns the maximum color and the minimum color of the color palette to the maximum value and the minimum value included in each three-dimensional image for each of the plurality of three-dimensional images. A normal mode in which the colored image generating unit generates a three-dimensional colored image based on an independent correspondence for each three-dimensional image and displays it on the display unit, and a reference three-dimensional image among a plurality of three-dimensional images. An absolute value comparison mode in which the maximum color and the minimum color of the included height are assigned to the maximum and minimum colors in the color palette, and the colored image generating unit generates a three-dimensional colored image based on a common correspondence and displays it on the display unit And a mode switching unit for switching between. This makes it possible to observe a wide dynamic range in the normal mode when observing one three-dimensional image, and switch to the absolute value comparison mode when observing a plurality of three-dimensional images in comparison. , All heights can be displayed in the same color, and observation suitable for the purpose can be performed.

  The magnified observation method of the present invention is a magnified observation method capable of capturing a plurality of three-dimensional observation images, displaying them on a display unit, and observing them in comparison. In this method, a plurality of three-dimensional image data are constructed and stored including height information in the data, a plurality of image data to be compared are selected, displayed side by side on a display unit, and three-dimensional image data Based on the distribution of height information, the maximum height and the minimum height are detected, and a color palette that expresses the height by displaying gradations in a predetermined range of colors is associated with the distribution of height information. Assigning the correspondence between each height and color so that the maximum height and the minimum height correspond to the maximum color and the minimum color in the range of the color palette, and assigning for each height information of the three-dimensional image Assigning colors based on the correspondence between the parts, generating a three-dimensional colored image corresponding to the height, and superimposing the three-dimensional colored image on the surface of each three-dimensional image displayed on the display unit With steps to display To have. Further, the step of assigning the correspondence between each height and color determines the correspondence based on the distribution of height information included in the reference three-dimensional image as a reference among the plurality of three-dimensional images. This is also performed on the three-dimensional image. As a result, a three-dimensional colored image is generated in accordance with a unified correspondence, so that the same height is expressed in the same color, and the difference in height and shape is displayed as a difference in color. It becomes easy to recognize the difference between images.

  Further, the magnified observation operation program of the present invention is a magnified observation operation program that captures a plurality of three-dimensional observation images, displays them on the display unit, and can compare them with each other. 3D image data, a function for storing the data including height information, a function for selecting a plurality of image data to be compared and displaying them side by side on the display unit, and a reference 3D image data Based on the distribution of height information, the maximum height and minimum height are detected, and a color palette that expresses the height by displaying gradations in a predetermined range of colors is associated with the distribution of height information, and the maximum height A function for assigning the correspondence between each height and color so that the height and the minimum height correspond to the maximum color and the minimum color in the range of the color palette, respectively, and assignment for each height information of a plurality of three-dimensional images. Based on the correspondence between departments A function of assigning each color and generating a three-dimensional colored image corresponding to the height; a function of displaying the three-dimensional colored image on the surface of each three-dimensional image displayed on the display unit; Based on the function of adjusting the transmittance of the three-dimensional colored image displayed superimposed on the three-dimensional image and the information on the size of the reference three-dimensional image, the display magnification of the three-dimensional image to be compared is the same magnification And the function of adjusting to be realized. As a result, a three-dimensional colored image is generated in accordance with a unified correspondence, so that the same height is expressed in the same color, and the difference in height and shape is displayed as a difference in color. It becomes easy to recognize the difference between images.

  The computer-readable recording medium or device of the present invention is a recording medium for a magnified observation operation program. Recording media include CD-ROM, CD-R, CD-RW, flexible disk, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, Blu-ray, HD A medium that can store a program such as a magnetic disk such as a DVD (AOD), an optical disk, a magneto-optical disk, a semiconductor memory, or the like is included. The above-mentioned program includes a form that can be downloaded via a network.

  According to the above magnification observation apparatus, magnification image observation method, magnification observation operation program, and computer-readable recording medium or recorded apparatus according to the present invention, when comparing a plurality of three-dimensional images, comparison with a reference image is performed. Differences in image height and shape can be easily recognized as differences in display patterns such as colors. In particular, since the same height is expressed by the same expression pattern on a plurality of three-dimensional images, it is easy to visually grasp the difference in height, which is suitable for comparison.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below exemplify a magnification observation apparatus, a magnification image observation method, a magnification observation operation program, and a computer-readable recording medium or recorded device for embodying the technical idea of the present invention. However, the present invention does not specify a magnification observation apparatus, a magnification image observation method, a magnification observation operation program, and a computer-readable recording medium or recorded device as follows.

  Further, the present specification by no means specifies the members shown in the claims to the members of the embodiments. In particular, the dimensions, materials, shapes, relative arrangements, and the like of the component parts described in the embodiments are not intended to limit the scope of the present invention unless otherwise specified, but are merely described. It's just an example. Note that the size, positional relationship, and the like of the members shown in each drawing may be exaggerated for clarity of explanation. Furthermore, in the following description, the same name and symbol indicate the same or the same members, and detailed description thereof will be omitted as appropriate. Furthermore, each element constituting the present invention may be configured such that a plurality of elements are constituted by the same member and the plurality of elements are shared by one member, and conversely, the function of one member is constituted by a plurality of members. It can also be realized by sharing.

The connection between the magnification observation apparatus used in the embodiment of the present invention and the computer, printer, external storage device and other peripheral devices for operation, control, display, and other processing connected thereto is, for example, IEEE 1394, RS-232x, RS-422, serial connection such as USB, parallel connection, or communication via network such as 10BASE-T, 100BASE-TX, 1000BASE-T, etc. Do. The connection is not limited to a physical connection using a wire, but may be a wireless connection using a wireless LAN such as IEEE 802.1x or OFDM, radio waves such as Bluetooth, infrared rays, optical communication, or the like. Furthermore, a memory card, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used as a recording medium for exchanging data or storing settings. In this specification, the term “magnification observation apparatus” is used to include not only the magnification observation apparatus body but also an imaging system in which peripheral devices such as a computer and an external storage device are combined.
[First Embodiment]

  The magnification observation apparatus according to the first embodiment of the present invention will be described below with reference to FIGS. As shown in FIG. 1, the magnification observation apparatus includes an illumination unit 60 for illuminating a sample to be observed, an imaging unit 10 for imaging the sample illuminated by the illumination unit 60, and an enlarged image captured by the imaging unit 10. The information processing apparatus 50 which has the display part 52 which displays is provided. Further, the magnified observation apparatus of FIG. 1 includes a sample fixing unit (stage 30 on which the sample S is mounted) for fixing the sample, and reflected light from the sample S fixed to the sample fixing unit incident via the optical system 11 or An image sensor (CCD 12) that electrically reads transmitted light, and a focus adjustment unit (stage elevator 20) that adjusts the focal point by changing the relative distance between the sample fixing unit and the optical system 11 in the optical axis direction. Furthermore, as shown in FIG. 2, the information processing apparatus 50 displays the focal length information about the relative distance in the optical axis direction between the sample fixing unit and the optical system 11 when the focal point is adjusted by the focal point adjusting unit. A focal length information storage unit (memory 53) that stores the two-dimensional position information of the sample in a vertical plane, a display unit 52 that displays an image read by the image sensor, and one of the images displayed by the display unit 52 An area setting unit (operating unit 55, pointing device 55A) that can set at least one area of the area and a focal length information storage unit related to a part or all of the sample S corresponding to the area set by the area setting unit An arithmetic unit (control) that calculates an average height in the optical axis direction of the sample S corresponding to the region set by the region setting unit based on the focal length information that has been set 51) and a. This magnifying observation apparatus uses an image sensor that electrically reads reflected light or transmitted light from a sample fixed to a sample fixing portion incident via an optical system, and uses the optical axis of the sample corresponding to a specified region. The average height (depth) in the direction can be calculated.

  As shown in FIG. 2, the imaging unit 10 is optically applied to a stage 30 that is one form of a sample fixing unit on which the sample S is placed, a stage elevator 20 that moves the stage 30, and a sample fixed to the stage 30. A CCD 12 as one form of an imaging device that electrically reads reflected light or transmitted light incident through the system for each pixel arranged two-dimensionally, and a CCD control circuit 13 that drives and controls the CCD 12 Prepare. Furthermore, the information processing apparatus 50 which is a magnification observation apparatus main body is connected to the imaging part 10. FIG. The information processing apparatus 50 displays an image based on the memory 53 as one form of an image data storage unit that stores image data electrically read by the image sensor and image data electrically read by the image sensor. A display unit 52 such as a display and a monitor, an operation unit 55 that performs input and other operations based on a screen displayed on the display unit 52, and image processing and various other processes based on information input by the operation unit 55 The control part 51 which performs is provided. The display constituting the display unit 52 is a monitor capable of high-resolution display, and a CRT, a liquid crystal panel, or the like is used.

  The operation unit 55 is connected to the computer by wire or wirelessly, or is fixed to the computer. Examples of the general operation unit 55 include various pointing devices such as a mouse, keyboard, slide pad, track point, tablet, joystick, console, jog dial, digitizer, light pen, numeric keypad, touch pad, and accu point. Further, these operation units 55 can be used for operations of the magnification observation apparatus itself and its peripheral devices in addition to the operations of the magnification observation operation program. Furthermore, using a touch screen or touch panel on the display itself that displays the interface screen, the user can directly input and operate the screen by hand, or use voice input or other existing input means, Or these can also be used together. In the example of FIG. 1, the operation unit 55 includes a pointing device 55A such as a mouse 55a.

  FIG. 1 shows an external view of a magnification observation apparatus according to an embodiment of the present invention. A camera 10 a having an optical system and an image sensor is attached to a camera attachment portion 43 fixed to a support column 42 extending in the vertical direction from the stand base 41. On the stand base 41, a stage elevator 20 on which a stage 30 on which the sample S is placed is attached is arranged. The camera 10a and the stage elevator 20 are connected to the information processing apparatus 50 and controlled. The information processing apparatus 50 includes a display unit 52 and an operation unit 55 such as a mouse 55a. An observation image is displayed on the display unit 52.

  Further, a computer 70 can be connected to the magnification observation apparatus which is the information processing apparatus 50, and a magnification observation operation program can be separately installed in the computer 70 so that the magnification observation apparatus can be operated from the computer 70 side. In this specification, the operation program for magnifying observation that operates the magnifying observation apparatus using a computer is an operation program installed in a general purpose or dedicated computer externally connected to the magnifying observation apparatus, as well as the above-described magnifying observation apparatus. It also includes an operation program incorporated in the information processing apparatus 50 as a control unit. The magnifying observation apparatus has a built-in operation function or operation program for operating the magnifying observation apparatus in advance. This operation program can be installed or updated in the magnifying observation apparatus in the form of rewritable software, firmware, or the like. Accordingly, the computer that executes the operation program for magnification observation in this specification includes the magnification observation apparatus itself.

FIG. 2 is a block diagram of the magnification observation apparatus according to the embodiment of the present invention. The information processing apparatus 50 includes a display unit 52, a memory 53 that stores control programs, focal length information, received light data, two-dimensional information, and the like, and the information processing apparatus 50 communicates data with the camera 10 a and the stage elevator 20. Interface 54, and an operation unit 55 for an operator to perform operations related to the magnification observation apparatus. The stage elevator 20 includes, for example, a stepping motor 21 and a motor control unit 22 that controls the elevation of the stepping motor 21. The imaging unit 10 is a light-receiving element such as a CCD 12 as an imaging element, a CCD control circuit 13 that drives and controls the CCD 12, and transmission of light irradiated from the illumination unit 60 to the sample S placed on the stage 30. And an optical system 11 that forms an image of light and reflected light on the CCD 12.
[Pixel shifting means]

  Furthermore, the imaging unit 10 can include pixel shifting means for obtaining a high resolution that is higher than the resolution of the CCD 12 by pixel shifting. Pixel shift is to increase the resolution by combining an image taken by shifting the subject by half the pixel pitch and an image before shifting. As a typical image shifting mechanism, there are a CCD driving method for moving an image sensor, an LPF tilting method for tilting an LPF, a lens moving method for moving a lens, and the like. In FIG. 2, the incident light path of reflected light or transmitted light incident on the CCD 12 from the sample S fixed to the stage 30 via the optical system 11 is set in at least one direction, and the interval of one pixel of the CCD 12 in that direction. An optical path shift unit 14 that optically shifts at a smaller distance is provided. In one embodiment of the present invention, the mechanism and method for realizing pixel shifting are not limited to the above configuration, and a known method or a method developed in the future can be used as appropriate.

  The information processing device 50 inputs control data related to the control of the stepping motor 21 to the motor control circuit 22, thereby allowing the stage 30 as the sample fixing unit, the camera 10 a having the optical system 11 and the CCD 12 as the imaging device, and The relative distance in the optical axis direction, here the height in the z direction is changed. Specifically, the information processing device 50 controls the rotation of the stepping motor 21 by inputting control data necessary for controlling the stage elevator 20 to the motor control circuit 22, and the height z (z direction) of the stage 30. Up and down). The stepping motor 21 generates a rotation signal corresponding to the rotation. The information processing apparatus 50 stores the height z of the stage 30 as information regarding the relative distance between the sample fixing unit and the optical system 11 in the optical axis direction based on the rotation signal input via the motor control circuit 22. In the present embodiment, the example in which the relative distance in the optical axis direction between the sample fixing unit and the optical system is changed by changing the height of the stage 30 is shown. The height, for example, the height of the camera 10a may be changed.

  The CCD 12 can electrically read the amount of received light for each pixel arranged two-dimensionally in the x and y directions. The image of the sample S formed on the CCD 12 is converted into an electric signal according to the amount of received light at each pixel of the CCD 12 and further converted into digital data in the CCD control circuit 13. The information processing apparatus 50 uses the digital data converted by the CCD control circuit 13 as received light data D in a plane (x and y directions in FIG. 2) substantially perpendicular to the optical axis direction (z direction in FIG. 2). The information is stored in the memory 53 together with pixel arrangement information (x, y) as two-dimensional position information of the sample. Here, the in-plane substantially perpendicular to the optical axis direction does not need to be a plane that is exactly 90 ° with respect to the optical axis, and is such that the shape of the sample can be recognized in the resolution of the optical system and the image sensor. Any observation surface may be used as long as it is within the tilt range.

  In the above description, an example in which the sample is placed on the stage is shown as an example of the sample fixing unit. However, for example, an arm may be attached instead of the stage and the sample may be fixed to the tip. Further, the camera 10a can be used by being mounted on the camera mounting portion 43, and can be detachably disposed at a desired position and angle by a hand-held method.

The illumination unit 60 shown in FIG. 1 includes an epi-illumination 60A for irradiating the sample with epi-illumination light and a transmission illumination 60B for irradiating the sample with transmitted light. These lights are connected to the information processing apparatus 50 via the optical fiber 61. The information processing apparatus 50 includes a connector 62 for connecting the optical fiber 61 and a light source (not shown) for sending light to the optical fiber 61 via the connector 62. A halogen lamp or the like is used as the light source.
[Control unit 51]

The control unit 51, which is a control unit, controls the captured observation image to be converted into a resolution that can be displayed on the display unit 52 and displayed. In the magnification observation apparatus of FIG. 1, the imaging unit 10 displays an observation image obtained by imaging the sample S with the CCD 12 on the display unit 52. In general, the performance of an image sensor such as a CCD often exceeds the display capability of the display unit. Therefore, in order to display the captured observation image on one screen, the resolution can be displayed on one screen by thinning the image. Reduced to size and reduced. If the reading resolution when reading by the imaging unit 10 is the first resolution, the display unit 52 displays a second resolution lower than the first resolution.
[Second Embodiment]

  Next, a magnification observation apparatus according to a second embodiment of the present invention will be described with reference to FIG. In the magnification observation apparatus according to the second embodiment, the camera serving as the imaging unit transmits the reflected light of the light from the first light source (laser 101) irradiated to the sample S via the first optical system 100. The second optical system reflects the reflected light of the first imaging unit that receives light by the first light receiving element (photodiode 112) and the second light source (white lamp 201) irradiated to the sample S. And a second imaging unit that receives light by the second light receiving element (CCD 212) via 200.

  First, the first imaging unit will be described. The first optical system 100 includes a laser 101 that irradiates a sample S with monochromatic light (for example, laser light), a first collimating lens 102, a polarizing beam splitter 103, a quarter-wave plate 104, a horizontal deflection device 105, and vertical deflection. A device 106, a first relay lens 107, a second relay lens 108, an objective lens 109, an imaging lens 110, a pinhole plate 111, and a photodiode 112 are included.

  For example, a semiconductor laser 101 that emits red laser light is used as the first light source. The laser light emitted from the laser 101 driven by the laser driving circuit 115 passes through the first collimating lens 102, the optical path is changed by the polarization beam splitter 103, and passes through the quarter wavelength plate 104. Thereafter, the light is deflected in the horizontal (lateral) direction and the vertical (longitudinal) direction by the horizontal deflection device 105 and the vertical deflection device 106, and then passes through the first relay lens 107 and the second relay lens 108, and the objective lens 109. Is condensed on the surface of the sample S placed on the stage 30.

  The horizontal deflection device 105 and the vertical deflection device 106 are each configured by a galvanometer mirror, and scan the surface of the sample S with the laser light by deflecting the laser light in the horizontal and vertical directions. The stage 30 is driven in the z direction (optical axis direction) by the stage elevator 20. As a result, the relative distance between the focal point of the objective lens 109 and the sample S in the optical axis direction can be changed.

  The laser beam reflected by the sample S follows the above optical path in reverse. That is, it passes through the objective lens 109, the second relay lens 108, and the first relay lens 107, and again passes through the quarter wavelength plate 104 via the horizontal deflection device 105 and the vertical deflection device 106. As a result, the laser light passes through the polarization beam splitter 103 and is collected by the imaging lens 110. The condensed laser light passes through the pinhole of the pinhole plate 111 disposed at the focal position of the imaging lens 110 and enters the photodiode 112. The photodiode 112 converts the amount of received light into an electrical signal. An electric signal corresponding to the amount of received light is input to the A / D converter 113 via an output amplifier and a gain control circuit (not shown), and converted into digital data. Here, an example in which a photodiode is used as the first light receiving element is shown, but a photomultiplier or the like may be used. Further, the laser 101 is not limited to a red laser, and a blue or ultraviolet laser may be used. By using such a short wavelength laser, high-resolution height data can be obtained.

  The height (depth) information of the sample S can be obtained by the first imaging unit configured as described above. The principle will be briefly described below. As described above, when the stage 30 is driven in the z direction (the optical axis direction) by the stepping motor 21 and the motor control circuit 22 of the stage elevator 20, the relative focus between the focal point of the objective lens 109 and the sample S in the optical axis direction. The distance changes. When the focal point of the objective lens 109 is focused on the surface of the sample S (surface to be measured), the laser light reflected by the surface of the sample S is condensed by the imaging lens 110 through the optical path described above, and almost all. All the laser beams pass through the pinholes of the pinhole plate 111. Accordingly, at this time, the amount of light received by the photodiode 112 is maximized. On the contrary, in the state where the focus of the objective lens 109 is deviated from the surface (surface to be measured) of the sample S, the laser beam condensed by the imaging lens 109 is focused at a position deviated from the pinhole plate 111. Only a part of the laser beam can pass through the pinhole. As a result, the amount of light received by the photodiode 112 is significantly reduced.

  Therefore, if the received light amount of the photodiode 112 is detected at any point on the surface of the sample S while driving the stage 30 in the z direction (optical axis direction), the height of the stage 30 when the received light amount becomes maximum. You can ask for it.

  Actually, each time the stage 30 is moved by one step, the surface of the sample S is scanned by the horizontal deflecting device 105 and the vertical deflecting device 106 to obtain the amount of light received by the photodiode 112. FIG. 4 shows changes in the received light data D with respect to the height z of the stage 30 at one point (pixel). When the stage 30 is moved in the z direction from the lower end to the upper end of the measurement range, the light reception data D that changes in accordance with the height z as shown in FIG. 4 is obtained for a plurality of points (pixels) in the scanning range. It is done. Based on the received light data D, the maximum received light amount and the focal length Zf at that time are obtained for each point (pixel). The height z of the stage 30 corresponding to the maximum value of the received light data D is the focal length Zf. Accordingly, the distribution of the surface height of the sample S on the xy plane is obtained based on the focal length Zf. This processing is performed by the control unit 51 based on the pixel arrangement information (x, y) and the height information z based on the light reception data D of the CCD 12 input through the interface 53 and stored in the memory 53.

  The obtained surface height distribution can be displayed on the display unit 52 by several methods. For example, the height distribution (surface shape) of the sample can be displayed three-dimensionally by three-dimensional display. Alternatively, it can be displayed as a two-dimensional distribution of brightness by converting the height data into luminance data. Alternatively, the height distribution may be displayed as a color distribution by converting the height data into color difference data.

  Also in the second embodiment, as in the first embodiment, two points on the image of the display unit 52 are designated by the pointing device 55A or the like based on the height data obtained by the first imaging unit. By doing so, the area is set in a rectangular shape, and the average height in the area and the relative height between the areas can be calculated and displayed on the display unit 52.

  Further, a surface image (black and white image) of the sample w is obtained from a luminance signal that uses the received light amount obtained for each point (pixel) in the xy scanning range as luminance data. If a luminance signal is generated using the maximum amount of light received at each pixel as luminance data, a confocal image having a very deep depth of field can be obtained at each point having a different surface height. In addition, when the height (z-direction position) at which the maximum light reception amount is obtained at an arbitrary target pixel is fixed, the light reception amount of the pixel of the target pixel portion and the portion having a large difference in height is significantly reduced. A bright image is obtained only at the same height.

  Next, the second imaging unit will be described. The second optical system 200 includes a second light source 201 for irradiating the sample S with white light (illumination light for color image shooting), a second collimating lens 202, a first half mirror 203, and a second half mirror. 204, a CCD 212 as a second light receiving element. The second optical system 200 shares the objective lens 109 of the first optical system 100, and the optical axes of both the optical systems 100 and 200 are the same.

  For example, a white lamp is used as the second light source 201, but a dedicated light source may not be provided, and natural light or room light may be used. The white light emitted from the second light source 201 passes through the second collimating lens 202, the optical path is bent by the first half mirror 203, and is collected on the surface of the sample S placed on the stage 30 by the objective lens 109. To be lighted.

  The white light reflected by the sample S passes through the objective lens 109, the first half mirror 203, and the second relay lens 108, is reflected by the second half mirror 204, and enters the CCD 212 that can receive light in color. To form an image. The CCD 212 is provided at a position that is conjugate or close to the conjugate with the pinhole of the pinhole plate 111 of the first optical system 100. A color image captured by the CCD 212 is read by the CCD control circuit 213 and converted into digital data. The color image thus obtained is displayed on the display unit 52 as an enlarged color image for observing the sample S.

  Also, by combining a confocal image with a deep depth of field obtained by the first imaging unit and a normal color image obtained by the second imaging unit, the object field in which all pixels are in focus A deep color confocal image can also be generated and displayed. For example, a color confocal image is simply generated by replacing the luminance signal constituting the color image obtained by the second imaging unit with the luminance signal of the confocal image obtained by the first optical system 100. be able to.

  Here, a magnification observation apparatus including a first imaging unit having a first optical system 100 that is a confocal optical system and a second imaging unit having a second optical system 200 that is a non-confocal optical system is shown. However, a configuration including only the first imaging unit may be employed.

  Further, like the magnification observation apparatus according to the first embodiment, the light receiving element is a two-dimensional imaging element (for example, CCD) that reads the amount of received light for each pixel arranged two-dimensionally, and the focus adjustment unit is an area. When the focal point is adjusted based on the sum of received light amounts corresponding to part or all of the sample corresponding to the region set by the setting unit, a complicated configuration like a confocal optical system is required. The height of the sample can be measured with a simple configuration. In particular, in this magnification observation apparatus, the maximum value of the light reception data is determined from the change in the light reception data with respect to the relative distance of the pixel, not the pixel unit, but the area unit set by the operator, and at that time Since the average height is calculated based on the average focal length, even when white light is used as a light source and a CCD is used as a light receiving element, variation in the variation of the received light data with respect to the focal length can be reduced. The average height can be measured. Further, when a color CCD is used as the two-dimensional imaging device, the light reception data of the pixel may be calculated based on the light reception data of RGB, or the light reception data of one or two colors of RGB may be used. The light reception data of the pixel may be used.

  In addition, if the area set by the area setting unit is larger than the sample size and includes the entire sample, the part other than the sample, that is, the upper surface of the stage, should be excluded from the calculation of the average height. Is preferred. This is because a more accurate sample height can be calculated. In this case, whether or not it is the upper surface of the stage can be determined based on whether or not the height difference between the pixel and the pixel adjacent to the pixel is a predetermined height or more. Of course, even if the region set by the region setting unit is a part of the sample, if the upper surface of the stage is included in the region, it is preferable to exclude it from the target for calculating the average height.

In the above embodiments, the example in which the reflected light from the sample fixed to the sample fixing portion is electrically read has been shown. However, the transmitted light is electrically read by irradiating light from the back surface of the sample. You may comprise as follows. In the above description, an example in which the sample is placed on the stage is shown as an example of the sample fixing unit. However, for example, an arm may be attached instead of the stage and the sample may be fixed to the tip.
[3D image]

  In addition, the magnification observation apparatus can display not only a two-dimensional image but also a three-dimensional image. By reproducing the 3D image on the display unit 52 of the magnification observation apparatus based on the 3D data, the image can be observed and evaluated in three dimensions from various viewpoints and angles, and the surface shape and profile such as the height can be measured. It can be carried out. In order to generate and display a three-dimensional image, for example, the relative distance in the optical axis direction, that is, the distance between the sample to be observed and the lens is changed, and a plurality of images are captured, and the movement amount at the time of image capture is recorded simultaneously. Keep it. As a result, the height information of the observation target can also be obtained when creating a composite image, and thus the acquired image can be constructed as three-dimensional data. Note that processing such as generation of three-dimensional data and alignment described later is performed by the control unit 51. The members such as the control unit 51 can be realized by hardware or software such as a predetermined gate array (FPGA, ASIC), or a mixture thereof. In addition, each component is not necessarily the same as the configuration shown in FIG. 2 or FIG. 3, the function is substantially the same, and one element is a plurality of components in the configuration shown in FIG. 2 or FIG. 3. What has the function of an element is included in this invention.

  A general three-dimensional data imaging method will be described with reference to FIG. First, in step S1, the imaging unit 10, the stage elevator 20 and the information processing apparatus 50 are initialized. In step S2, the moving amount and moving range of the lens are set. The movement range of the lens is a range of the lens height. For example, the user designates a rough height as the movement start position. The amount of movement of the lens is the distance of movement of the lens per one time. The finer the setting, the more detailed images can be taken, but the longer it takes to generate, so the value is set to an appropriate value according to the purpose of observation. As the specification method, the user specifies the fineness of the image, and the magnification distance is automatically set by the magnification observation device, or the user directly specifies the movement distance with a numerical value, or the default default value. There are methods such as using.

In step S3, the lens is set at the movement start position specified above. In step S4, imaging is started. After imaging, the lens is moved by the movement amount set in step S2 in step S5, and it is determined whether or not the movement range has reached the end position in step S6. If the end position has not been reached, the process returns to step S4 and the imaging and moving steps are repeated. As a result of the movement, when the movement range set in step S2 is finished, the process proceeds to step S7, and a two-dimensional captured image is synthesized. In step S8, 3D image data is constructed based on the plurality of captured 2D image data and displayed on the display unit 52.
[Create 3D image data]

  For the construction of 3D data, a profile that changes in the height direction can be calculated from the position of the lens and the image captured at that time, so multiple 2D image data captured at different heights can be combined. By doing so, a three-dimensional shape can be constructed. For example, a continuous profile is synthesized by complementing a discrete profile obtained from a captured two-dimensional image. This process can be performed at high speed in hardware.

  A procedure for synthesizing a three-dimensional image will be described below based on the flowchart of FIG. Here, a plurality of two-dimensional images are used as input images, three-dimensional image data is generated based on the input images, and a two-dimensional composite image is generated by combining the three-dimensional images into a focused state. First, in step S8-1, each unit is initialized, and in step S8-2, a composite image region prepared as a region for creating a composite image is initialized. In step S8-3, frequency analysis is performed on the two-dimensional image data that is the input image. For example, the frequency component of the image data can be detected by two-dimensional discrete Fourier transform or two-dimensional discrete wavelet transform. In step S8-4, the composite image and the input image are compared for a certain area. The fixed area is obtained by dividing the composite image area into predetermined reference areas, and the composite image area is scanned while comparing the composite image and the input image in units of the reference area.

  In step S8-5, it is determined whether or not the input image is in focus from the synthesized image. The degree of focus can be determined from the result of the frequency analysis in step S8-3. As a result of the determination, if the input image is in focus, the process proceeds to step S8-6, and the composite image data is replaced with the input image data for a certain region. Further, the height information at the time of imaging the replaced input image data is also stored. Then, the process proceeds to step S8-7. On the other hand, if the subject is not in focus, step S8-6 is skipped and the process jumps to step S8-7. In the first loop, there is no input of composite image data, so the input image is input as it is as a composite image. Then, while repeating the loop of step S8-7, the focused image is sequentially replaced, and finally a focused two-dimensional composite image is acquired.

  In step S8-7, it is determined whether or not the next input image exists. If it exists, the process returns to step S8-3 to repeat the above process for all input images. If it does not exist, the process ends. Through the above steps, a focused two-dimensional composite image is obtained from a plurality of two-dimensional image data, and height information at the focused position is obtained at the same time. Dimensional image data is constructed.

  The three-dimensional image constructed as described above can be displayed by freely changing the observation viewpoint or enlarging or reducing the image. The change of the observation viewpoint includes processing such as rotation, inversion, and deformation of the image in an arbitrary direction. The change operation is performed with a pointing device 55A such as a mouse 55a. The image is rotated in the dragged direction by operating the mouse 55a, moving the mouse pointer 55e while selecting and dragging the image. For enlargement / reduction, a screen enlargement / reduction function can be assigned to the scroll button 55d of the wheel mouse. Not limited to these methods, operation buttons and tools are arranged on the software screen for operation, and operation functions are assigned to specific keys on the keyboard, or dedicated hardware for operation is used as the operation unit You can also.

  As such a three-dimensional image acquisition method and a method for changing the display of the acquired three-dimensional data, a known method or a method developed in the future can be used as appropriate. For example, an API such as Open GL (Open Graphics Library) can be used.

  With respect to the acquired three-dimensional data, reference information regarding the imaging position acquired in the above imaging process can be held together. The reference information regarding the position is information serving as a reference in the X, Y, and Z directions at the time of imaging. As described above, the construction of the three-dimensional data uses the lens height at the time of imaging and a two-dimensional image captured at that height, and synthesizes a plurality of two-dimensional image data captured at different heights. Thus, since the three-dimensional image data is constructed, the movement amount in the height direction at the time of image pickup such as the movement amount and movement range of the lens described above is recorded. This information is used as reference information in the height direction, that is, the Z direction. For the X and Y directions, for example, information on the actual dimensions such as how long each pixel of a two-dimensional image used for constructing three-dimensional image data actually corresponds can be used. By holding these pieces of reference information, it can be used for magnification adjustment and alignment when comparing a plurality of three-dimensional images. Note that the information necessary here does not necessarily need to be information corresponding to the actual size of the captured image, and may be information that can be relatively compared with other three-dimensional image data. Resolution, number of dots, etc. can also be used. In this way, by providing reference information to the three-dimensional image data, the reference three-dimensional image is based on the reference data in the X, Y, and Z directions of the respective three-dimensional images to be compared as will be described later. The display magnification of the three-dimensional image to be compared according to the data is enlarged or reduced.

Further, in addition to the reference information, magnification information and information regarding photographing conditions can also be held. For example, arbitrary data such as illumination intensity, photographing time, and sample name can be stored together with the three-dimensional image data. As a data storage format, there can be used a method in which a dedicated data area for holding such information is provided in a three-dimensional image data file and data is written in this area. Alternatively, data stored in individual files can be stored in association with each other.
[Indicated Area]

  Comparison can be easily performed by displaying a plurality of the three-dimensional images thus captured side by side on the same screen. FIG. 7 shows an interface screen of the magnification observation operation program according to the embodiment of the present invention. The interface screen shown in this figure is provided with two display areas A and B, with the left side of the screen being the display area A and the right side being the display area B. As described above, in addition to the configuration in which the display area (three-dimensional image) is arranged at an arbitrary position on one screen, for example, a boundary line may be displayed on the screen and divided into a plurality of display areas. Alternatively, the display area can be configured by another window, and a plurality of windows can be reduced and displayed side by side on one screen.

  Different display areas A and B can display different three-dimensional images. The user selects a three-dimensional image to be displayed in each area. For example, a saved file is selected and opened from the “Open File” dialog box. Each three-dimensional image can be individually changed in viewpoint, rotated, enlarged or reduced. For example, only one image can be rotated or enlarged or reduced while one image is fixed. It is also possible to change the viewpoint, enlarge / reduce, etc. in conjunction with a plurality of three-dimensional images. It is possible to switch between a display change mode in which each image is linked and an independent individual display change mode. FIG. 8 shows a state in which the viewpoint of the 3D image in the display area B is changed from the state in FIG. 7 while the 3D image in the display area A is fixed. FIG. 9 shows a state in which the display of the 3D image in the display area A is changed while the 3D image in the display area B is fixed from the state of FIG. Further, FIG. 10 shows an example in which the three-dimensional images in both the display areas A and B are changed simultaneously from the state of FIG. In the above example, the viewpoint change of each image is assigned to the left and right buttons 55b and 55c of the mouse 55a. If the mouse 55a is moved while the left button 55b of the mouse 55a is pressed, the three-dimensional image of the display area A on the left side of the screen is rotated in that direction (FIG. 9). Similarly, when the right button 55c of the mouse 55a is operated, A three-dimensional image of the display area B arranged on the right side of the screen can be operated (FIG. 8). Further, when the left and right buttons 55b and 55c of the mouse 55a are simultaneously pressed and operated, the left and right three-dimensional images can be operated simultaneously and rotated in the same manner (FIG. 10). Further, in this example, when the scroll button 55d of the mouse 55a is rotated, the two display areas are simultaneously enlarged / reduced (FIGS. 11 to 13). When the scroll button 55d is rotated from the front to the back, the screen is enlarged and the details can be confirmed as shown in FIGS. If the rotation is reversed, the screen is reduced and switched from FIG. 13 to FIG. 12 and FIG. The present invention is not limited to this example, and one magnification can be fixed and only the other three-dimensional image can be enlarged and reduced. Needless to say, different observation targets can be displayed in each display area, and the same observation target can be displayed from different viewpoints in each display area.

  In this way, two images are displayed on the same screen, and the display mode can be freely changed, so that the images can be easily compared. It is also possible to fix an observation target displayed in an arbitrary display area and display the observation target displayed in another display area by replacing it with another image. Further, the image to be replaced may be automatically switched and displayed. This function can be suitably used for operations such as discrimination between non-defective products and defective products. For example, the same reference type can be displayed by displaying the reference reference product and the inspection target product at the same time, displaying the reference product in one display area, and sequentially switching the images of the inspection target product in the other display area. Can be sequentially compared and inspected. Further, as will be described later, the reference product and the inspection product can be compared and observed more efficiently by displaying each image in a display state such as the same posture and angle. Furthermore, the image to be replaced may be automatically updated like a slide show. This makes it easy to find defective products from the same type of inspection product.

Of course, only one image can be displayed on the display unit 52. FIG. 16 shows an example in which the target image is displayed on a different interface screen as another embodiment. As described above, the control unit 51 can switch and display the mode for displaying only one image on the display unit and the mode for displaying a plurality of display areas.
[Image alignment]

  Furthermore, the image groups displayed in the plurality of display areas can be adjusted so that each is displayed with the same posture, position, angle, and inclination. This adjustment can be performed automatically by the user or automatically. For example, as shown in FIG. 14, the initial display states of the two observation objects W1 and W2 may be shifted without matching. If the initial position is deviated, the subsequent comparison becomes difficult. Therefore, the viewpoint and inclination of the image are corrected so that the two works W1 and W2 are displayed in the same posture as shown in FIG. In this example, image recognition is performed so that the work W1 in the left display area A is fixed as a reference and the work W2 displayed in the right display area B is displayed in the same posture as the work W1 in the display area A. And automatic adjustment using pattern matching technology. For example, feature points are extracted from each image to identify the same location, and either or both images are rotated so as to have the same posture with reference to this position. When both images are rotated, a predetermined display form as a reference is set in advance, such as a front view and a plan view in a horizontal posture, and a perspective view seen from the upper right 45 °, and both images are Match the posture. Alternatively, the display state in the other display region can be adjusted so that the viewpoint and the enlargement ratio are matched with the display state in one display region at an arbitrary time. For example, on the interface screen shown in FIG. 9 or FIG. 10, from the state where similar workpieces W3 and W4 are displayed from different viewpoints, one is rotated to align both postures, and one workpiece is rotated and enlarged or reduced. Synchronously with this, the display state of the other workpiece can be changed in the same manner.

  A specific method for aligning the workpieces W1 and W2 will be described based on the flowchart of FIG. First, in step S'1, each part is initialized. Next, an image composition process for the work 1 is performed in step S'2, and an image composition process for the work 2 is performed in step S'3. The image composition processing is the same processing as the creation of the three-dimensional image data described above with reference to FIG. Here, the two-dimensional images in focus are positioned. Next, in step S′4, pattern matching is performed between the two-dimensional composite images of the work 1 and the work 2. Further, in step S′5, based on the pattern matching result of the images of the work 1 and the work 2, the rotation angle and the enlargement ratio at which the images are displaced are calculated. Based on the calculation result, the position of one image is corrected in step S′6. That is, the rotation angle and the enlargement ratio are changed and displayed so that the display is substantially the same as the reference image. Thereby, the alignment of the two-dimensional image is realized.

  In the above procedure, image alignment is performed for a two-dimensional image obtained by displaying a three-dimensional image in a specific posture. The present invention is not limited to this example, and the alignment of the three-dimensional image data can also be performed. For example, three-dimensional pattern matching or calculating the center of gravity of a stereoscopic image from each three-dimensional image data and adjusting the posture of the three-dimensional image on the basis of the axis of the center of gravity makes the three-dimensional image coincide with the same posture. be able to. For the calculation of the center of gravity axis, moment calculation or the like can be used. As a result, more accurate alignment than two-dimensional data is realized using three-dimensional image data.

  By displaying a plurality of images in a display mode having the same posture, angle, etc., it is possible to easily recognize a common part between the two and grasp a difference. Furthermore, by using the automatic alignment adjustment function, it is possible to save the trouble of manually matching the display state of the captured image, and for example, it is possible to simplify the positioning operation at the time of image capturing and to easily capture the image. .

  This alignment can be executed at an arbitrary timing. For example, it can be automatically performed as an initial state when an image is displayed on the screen. Alternatively, after performing processing such as rotation and enlargement / reduction on each image, these operations can be reset to return the rotation angle, enlargement ratio, etc. to the same posture, initial state, or set state. it can. As the set state, an arbitrary rotation angle and posture can be set. For example, a display posture of a plan view or a perspective view can be designated. In particular, by setting the automatically adjusted state as the initial state, the angle and the enlargement ratio can be returned to a state where the comparison can always be made in the same posture, which is convenient for re-operation and confirmation work. The set posture and the function to return to the initial state can be applied to only any display area in addition to being uniformly executed for all display areas. In order to execute such an alignment function, by preparing a dedicated alignment button, the function can be easily called and the user can return to a predetermined posture at an arbitrary timing. It is also possible to execute the alignment function or reset to the initial state by a method such as double-clicking on the screen or assigning a specific shortcut.

In the above example, an example in which a three-dimensional image is displayed in two display areas A and B is shown, but it is needless to say that three or more three-dimensional images can be displayed. Further, it is possible to display a two-dimensional image or the like without being limited to the three-dimensional image. Furthermore, the same observation target can be displayed on a plurality of screens. For example, it is effective for displaying images with different viewpoints, such as checking the front and back of the same sample at the same time.
[Calibration function]

  Next, a specific example of image alignment will be described in detail. FIG. 18 is an image diagram showing a user interface screen of the controller unit 300 of the magnification observation operation program for performing operations such as image alignment. First, a description will be given of a calibration function for correcting image alignment, that is, when displaying the same target object as a plurality of images, so that the size of the target object displayed in each image matches. The calibration function, for example, images the same sample at different magnifications, so that when the size of the obtained three-dimensional image is different, the size of the sample displayed in these three-dimensional images is aligned on the screen, This is a function for correcting one to enlarge / reduce to match the other. The calibration function is realized by a magnification adjustment unit 522 described later with reference to FIG. In FIG. 18, first, 3D image data serving as a reference (reference) is selected from the “file selection” field 302 in the lower row. In the example shown in the figure, the selection is made from the pull-down menu, but it goes without saying that the selection can also be made from the “open file” dialog screen or the like. When a file is selected and the “display 3D” button 304 is pressed, a 3D image viewer screen as shown in FIG. 19 is opened as the display unit 52B, and the selected 3D image is displayed. In this state, only one three-dimensional image is displayed, and the user can perform enlargement / reduction, rotation, movement, and the like of the image as described above. When the “close 3D” button 306 at the bottom is pressed, the viewer screen in FIG. 19 is closed.

  Next, the comparison 3D image data to be compared with the reference 3D image is selected from the “file selection” column 302 in FIG. When a “comparison mode” button 312 in the “3D file comparison” field 310 provided in the upper part of the “file selection” field 302 is pressed after selecting a file, the display unit 52B switches to a screen as shown in FIG. It is divided into an area A and a display area B. Here, the reference three-dimensional image already displayed is displayed in the left display area A, and the selected comparison three-dimensional image is displayed side by side in the right display area B. In addition, when replacing the comparison three-dimensional image while retaining the reference three-dimensional image, the reference three-dimensional image to be newly selected is similarly selected from the “file selection” field 302 in FIG. When a “sub switch” button 313 provided at the top of the “file selection” field 302 is pressed, the left display area A in FIG. 20 is left as it is, and the newly selected comparison three-dimensional image is displayed in the right display area B. Are displayed side by side.

The image data shown in the example of FIG. 20 is obtained by photographing the same or similar target, but the magnification at the time of imaging is different and the sizes of the displayed images do not match. In this state, when the “linked mode” button 314 in the “3D file comparison” column 310 of FIG. 18 is pressed, the display magnification of the comparison 3D image is changed as shown in FIG. 21, and the reference 3D image on the left side is changed. Are reduced to the same size as The adjustment of the enlargement / reduction ratio is performed based on the reference information regarding the imaging position stored together with the three-dimensional data as described above. Here, the magnification at the time of capturing each image is compared, the magnification of the reference three-dimensional image on the left is used as a reference, and the enlargement / reduction ratio is automatically calculated to match the magnification of the right comparison three-dimensional image, The display magnification is changed based on the calculated value. As a result, images can be compared side by side on the same scale, which is very convenient for observation in comparison. Note that the magnification adjustment unit 522 can automatically adjust the enlargement / reduction ratio of the display, but can also be manually performed. In order to change the display magnification manually, for example, after selecting a target three-dimensional image with the mouse 55a, if the mouse 55a is moved downward while the scroll button 55d is being clicked, the image is reduced and displayed. Enlarged display.
[Align image viewpoint]

  Next, a procedure for adjusting the observation viewpoint will be described. Similarly to the above, the reference three-dimensional image is selected and displayed from the controller unit 300 in FIG. 18 and adjusted to a desired viewpoint. The viewpoint can be adjusted by dragging the mouse as described above. Here, the viewpoint of the reference three-dimensional image of FIG. 19 is changed as shown in FIG. Next, when a comparison three-dimensional image to be compared is selected from the controller unit 300 in FIG. 18 and the “comparison mode” button 312 is pressed in the same manner as described above, the reference three-dimensional image is displayed on the left side as shown in FIG. In the area A, the comparison three-dimensional image is displayed in the display area B on the right side. When the “synchronization set” button 316 in the “3D file comparison” column 310 of FIG. 18 is pressed from this state, the posture of the comparison three-dimensional image becomes the same viewpoint as the reference three-dimensional image on the left as shown in FIG. Will be changed as follows. The above-described image recognition and pattern matching techniques are used for the viewpoint changing process. A rotation angle, an offset position, and the like of each image are calculated and processed so that the display posture of the comparison three-dimensional image matches the reference three-dimensional image. In this case, reference information regarding the imaging position stored together with the above-described three-dimensional data may be used. As described above, since two images can be compared and observed from the same viewpoint, the user does not need to manually adjust the viewpoint and can be used conveniently.

  Furthermore, when the “interlocking mode” button 314 in the “3D file comparison” column 310 in FIG. 18 is pressed from this state, the display magnification of the comparative three-dimensional image is changed while maintaining the same viewpoint as shown in FIG. Are displayed at the same viewpoint and magnification as the reference three-dimensional image on the left side. In this way, by performing both magnification and viewpoint coincidence, an excellent environment in which various images can be easily observed is realized.

The user can also save the display state of the three-dimensional image being displayed as a two-dimensional image. When a “Save” button 322 provided below the “Height Adjustment” slider 320 at the top of the controller unit 300 screen of FIG. 18 is pressed, a dialog box for saving as a two-dimensional image is opened. A file format or the like is specified, and the three-dimensional image is saved as the currently displayed two-dimensional image. A “reset” button 324 provided below the “save” button 322 is used to return to the initial state after rotating or moving the three-dimensional image. Return to the initial posture. Further, a “height adjustment” slider 320 disposed above the “save” button 322 adjusts the stereoscopic effect of the three-dimensional image. As the “height adjustment” slider 320 is moved downward from the intermediate position, the stereoscopic effect is emphasized. Conversely, as the slider 320 is moved upward, the stereoscopic effect is weakened and approaches a plane. It should be noted that the functions and operation methods of the controller unit 300 described here are merely examples, and it goes without saying that the user interface screen and operation method of the program can be changed as appropriate. In addition, hardware corresponding to the controller unit can be installed in the magnification observation apparatus.
[3D expression pattern image]

Next, a procedure for generating a three-dimensional expression pattern image based on the height of the three-dimensional image will be described. Although the three-dimensional image captured by the imaging unit has height information, it may be difficult to determine the height depending on the surface state. In general, a three-dimensional image is a pseudo-reproduced image on a display screen, such as pasting image data (texture) obtained by imaging a surface on a three-dimensional image using a computer graphic (CG) technique. In some cases, such as when the surface pattern is complex, it may be difficult to distinguish the height difference. Therefore, the surface of a three-dimensional image is colored so as to be color-coded according to the height difference, and the height is expressed by the color difference. For example, from high to low positions, the colors are changed from red to orange to yellow to green to water to blue to black, and the colors are separated by heights of a certain width, or the color changes gradually by color gradation. For example, a color palette having a predetermined range of color changes is created, and a height is assigned to each color in the color palette. The color palette determines the color classification within the range of the number of colors that can be displayed on the display unit. For example, classification such as 8 colors, 16 colors, and 256 colors is performed. The assignment unit 512 automatically assigns the divided colors and heights included in the color palette. Based on the correspondence determined by the assigning unit 512, the color corresponds to 1: 1 for each height or height range. Note that the color corresponding to the height may be displayed as it is on the three-dimensional colored image, or a gradation is applied between colors adjacent to each other in the height direction (Z direction) on the three-dimensional image, and the boundary between the colors. You may make it display a part smoothly.
[Block Diagram of Control Unit 51]

FIG. 26 is a block diagram illustrating a configuration of the control unit 51 that performs various processes on the generated three-dimensional image. As shown in this figure, the control unit 51 includes a storage unit 53A for holding or storing a three-dimensional image, a display unit 52 for displaying a generated and adjusted three-dimensional image or a three-dimensional colored image, a user Is connected to an operation unit 55 for performing various operations. As the storage unit 53A, a memory that temporarily stores the three-dimensional image generated by the above procedure, a secondary storage medium such as a hard disk that stores data as a data file, or the like can be used. As described above, the three-dimensional image data includes information regarding the height and reference data in the X, Y, and Z directions regarding the size. The operation unit 55 is an input device for the user to operate the magnification observation apparatus as described above. The control unit 51 illustrated in FIG. 26 includes an allocation unit 512, a colored image generation unit 514, an image adjustment unit 516, and a mode switching unit 518. The image adjustment unit 516 further includes a transmittance adjustment unit 520, a magnification adjustment unit 522, and a viewpoint adjustment unit 524. Note that the image adjustment unit 516 can perform various adjustments not only on the three-dimensional colored image but also on the original three-dimensional image as described later.
[Allocation unit 512]

  The assigning unit 512 first examines the distribution of height information included in the three-dimensional image data, and detects the maximum height and the minimum height. On the other hand, a color palette that expresses the height by displaying gradations in a predetermined range of colors is preset, and the maximum and minimum colors located at the end of the range of this color palette are set to the maximum and minimum heights. Assign the correspondence between height and color. Allocation is performed evenly, and with the height and the end value of the color range matched, the height difference is equally divided by the number of color divisions, and the height for each divided color is calculated. The height of the three-dimensional image can be easily understood by calculating the relative height difference so that the minimum value is zero. Further, the color division is determined by determining the maximum color and the minimum color which are the end values of the range, and assigning an increase in the division between them to a value with good separation such as 5, 10, 100, etc. The display at 52 is easy to see and is preferable.

In addition to the method based on the end values such as the maximum and minimum heights, the allocation method by the allocation unit 512 is a method of allocating on the basis of an average or a center value based on, for example, height dispersion or standard deviation, or a histogram A method of assigning a region having a large distribution of height from the center or a method of eliminating the maximum value or the minimum value as an error and assigning based on the next point data can be used as appropriate.
[Colored image generating unit 514]

When the correspondence is determined in this way, a color is assigned for each surface height of the three-dimensional image based on this, and the three-dimensional colored image is generated by the colored image generation unit 514. The three-dimensional colored image is created by scanning the surface of the three-dimensional image and replacing the height information at each position with color information. In this example, the process is simplified by coloring only the surface position of the three-dimensional image. For coloring, for example, based on a wire frame model, texture data of a color assigned to each height is pasted on the surface. However, a surface model or solid model can be used for more detailed observation, or it can be converted into voxel (3D pixel) data. As an example, FIG. 28 shows an example in which the three-dimensional image in FIG. 27 is converted into a three-dimensional colored image. In order to display a three-dimensional colored image, a “height / color” button 326 is pressed in the controller unit 300 shown in FIG. As a result, the three-dimensional image is switched to a three-dimensional colored image, and is displayed by being color-coded according to the height, so that the shape, particularly the height image, can be easily grasped regardless of the surface pattern of the sample. .
[Color gauge 402]

Further, a color gauge 402 indicating the correspondence between color and height is displayed on a part of the display unit 52. In the example of FIG. 28, the color gauge 402 is displayed on the left side of the screen. However, the color gauge 402 may be arranged at an arbitrary position, and display / non-display may be switched. The color gauge 402 shown in this figure displays a bar-shaped color palette in a vertical orientation and also shows representative values of heights assigned to colors. The color gauge 402 allows the user to understand which color corresponds to what height, and by displaying a numerical value of the height, the display unit 52 can also display a scale of the length and actual dimensions. . In this example, the relative height with the minimum height as a reference, that is, 0 is shown, and the peak height of the sample can be understood by displaying the maximum height at the top of the color gauge 402. Note that the representative value of the height may be a constant increment such as 5, 10, 100, or the like. Further, in this example, the relative height with the minimum height being 0 is shown, but any value can be used as the reference value for the relative height, or an absolute height can be expressed. By displaying the color gauge 402, the user can confirm the correspondence between the color and the height, so there is no need to store this specific relationship or value one by one, Specific values can be easily observed with reference to the color gauge 402.
[Scale grid 404]

Furthermore, a scale grid 404 indicating the sizes of the three-dimensional image in the X, Y, and Z directions can be displayed. FIG. 29 shows an example in which the scale grid 404 is displayed on the three-dimensional colored image of FIG. As shown in this figure, the scale grid 404 displays orthogonal coordinate axes in the X, Y, and Z directions of the three-dimensional image, and scales are written on each coordinate axis. The scale notation is consistent with the scale notation of the color gauge 402, but it may be a representative value of a certain increment such as 5, 10, 100, etc. By displaying the scale grid 404, the actual dimensions such as the depth and width of the three-dimensional image can be grasped. The scale grid 404 is displayed along with the three-dimensional image, and when the three-dimensional image is rotated, the scale grid 404 is also rotated together. For this reason, the orthogonal coordinates also function as a reference line of a three-dimensional image, which is useful for checking the rotation angle and viewpoint. The display of the scale grid 404 can be toggled ON / OFF by pressing the “scale” button 330 of the controller unit 300 shown in FIG.
[Transmissivity adjuster 520]

Further, the three-dimensional colored image generated by the colored image generation unit 514 can be displayed on the display unit 52 so as to be superimposed on the original three-dimensional image. Moreover, you may display independently of the original three-dimensional image. For example, a 3D colored image can be displayed side by side with the original 3D image, or only a 3D colored image can be displayed. Furthermore, the three-dimensional colored image can be made transparent, and when the three-dimensional colored image is superimposed on the three-dimensional image and displayed, the surface pattern of the three-dimensional image can be seen through from below the three-dimensional colored image. Can be adjusted. FIG. 30 shows a state in which the transmittance is set to 50% in the three-dimensional colored image of FIG. FIG. 30 is displayed in a state where the original three-dimensional image of FIG. 27 and the three-dimensional colored image of FIG. 28 are superimposed, and thereby the height and surface state of the sample to be observed displayed in the three-dimensional image. Can be confirmed at the same time. The transmittance is adjusted by the transmittance adjusting unit 520 included in the image adjusting unit 516 in FIG. In the example of the controller unit 300 shown in FIG. 18, the transmittance can be continuously adjusted from 0 to 100% by operating the transmittance adjustment slider 328 provided under the “height / color” button 326. Accordingly, the original three-dimensional image and the height information by color can be displayed at a desired ratio at the same time, and can be adjusted according to the user's preference, observation purpose, and the like. Further, the transmittance adjusting unit 520 can obtain the same effect by adjusting the transmittance of the three-dimensional image instead of the three-dimensional colored image.
[Magnification adjustment unit 522]

Furthermore, the magnification adjustment unit 522 included in the image adjustment unit 516 in FIG. 26 is a member for executing the above-described calibration function for performing image alignment. The magnification adjustment unit 522 can automatically align the sample sizes when displaying a plurality of three-dimensional images on the screen of the display unit 52, as will be described later.
[Viewpoint adjustment unit 524]

Furthermore, the viewpoint adjustment unit 524 included in the image adjustment unit 516 in FIG. 26 realizes the function of matching the viewpoints of the above-described plurality of three-dimensional images. The viewpoint adjustment unit 524 performs adjustment so that the viewpoint of the object displayed as the reference three-dimensional image matches the viewpoint of the object displayed as the comparison target three-dimensional image. Specifically, as described above, the rotation angle and offset amount of the reference three-dimensional image are detected by a method such as image recognition and pattern matching, and the posture of the comparative three-dimensional image is changed based on this. This viewpoint adjustment unit 524 can be applied to both a three-dimensional image and a three-dimensional colored image. The viewpoint adjustment unit 524 automatically adjusts the viewpoints of the three-dimensional images so that the viewpoints of the three-dimensional images and the three-dimensional colored images are manually changed by the user. It can also be adjusted. When the user manually adjusts the viewpoint, the three-dimensional images are rotated and moved using the mouse 55a as described above and adjusted so that their postures coincide. In this example, when the left button 55b of the mouse 55a is dragged, the selected three-dimensional image is rotated, and when the right button 55c is dragged, the three-dimensional image can be moved. If the mouse 55a is moved up and down while the scroll button 55d is being clicked, the display magnification of the selected three-dimensional image is changed. If the left and right buttons 55b and 55c of the mouse 55a are simultaneously clicked, it is possible to return to the initial posture when reading the three-dimensional image. By such an operation, the user can adjust the three-dimensional image to a desired viewpoint.
[Specify height reference position]

Further, the viewpoint adjustment unit is not limited to a three-dimensional colored image, and when a plurality of three-dimensional images are superimposed, a height serving as a reference for superimposition can be specified, so that the three-dimensional images between the three-dimensional images to be superimposed can be specified. Even if there is a deviation in height, superposition can be easily performed. In different three-dimensional images, even if they contain the same shape, there may be positional deviation or offset in the height direction. In such images, the reference height before comparison or overlaying is performed as it is. It is possible to perform these operations more appropriately by aligning the sizes. The reference height can be easily specified by clicking a desired position on the three-dimensional image with a pointing device such as a mouse. In this way, even in a three-dimensional image having different height-direction standards, it is possible to easily perform superposition by aligning the standards of the respective images based on the designated standards.
[Mode switching unit 518]

  The above has described an example in which a three-dimensional colored image is generated and displayed for a single three-dimensional image. In the present embodiment, as shown in FIGS. 31 and 32, a plurality of three-dimensional images are displayed. Similarly, a three-dimensional colored image can be generated and displayed. At this time, the correspondence between height and color can be set independently for each image, or a unified correspondence can be applied to all images. The magnification observation apparatus according to the present embodiment includes a normal mode in which a correspondence relationship is set independently for each three-dimensional image, and a correspondence relationship set based on the three-dimensional image as a reference three-dimensional image. Is applied to all three-dimensional images. These modes are switched by a mode switching unit 518 included in the control unit 51 in FIG. In the mode switching unit 518, the user inputs a command through the operation unit 55, and switches the mode based on the command.

31 and 32 show three-dimensional images obtained by imaging samples having similar shapes and different sizes in the display areas A and B of the display unit 52, respectively. In this example, the 3D image shown in the display area A is a reference 3D image, and the 3D image shown in the display area B is a comparative 3D image. In the display examples of the plurality of three-dimensional colored images shown in these drawings, FIG. 31 shows the display in the normal mode, and FIG. 32 shows the display in the absolute value comparison mode.
[Normal mode]

In the normal mode, as described above, the maximum color and the minimum color of the color palette are assigned to the maximum value and the minimum value included in each three-dimensional image, and an independent correspondence is set for each three-dimensional image. The colored image generation unit 514 generates a three-dimensional colored image and displays it on the display unit 52. In the example of FIG. 31, since the three-dimensional colored image is generated by assigning all the colors of the color palette to each three-dimensional image in each of the display areas A and B, it is possible to display with a wide dynamic range and high expressive power. Therefore, it is suitable for observing each three-dimensional colored image independently. In the normal mode, the above-described display ON / OFF of the scale grid 404 and the change of the transmittance can be used. FIG. 33 shows an example in which the original three-dimensional image is displayed with the transmittance of the three-dimensional colored image set to 0% by the transmittance adjustment slider 328 in the normal mode of FIG. 31, and FIG. 34 shows the transmittance of FIG. An example in which a three-dimensional colored image is superimposed on the original three-dimensional image and displayed as 50% is shown. On the other hand, in this state, the scale feeling of the color patterns colored in the respective three-dimensional colored images is not unified, so that it is not suitable for comparing the actual sizes.
[Absolute value comparison mode]

  On the other hand, in the absolute value comparison mode shown in FIG. 32, the correspondence relationship between the color palettes determined based on the three-dimensional image data of the display area A, which is the reference three-dimensional image, is compared with the comparison Taisho 3D of the display area B. It is also applied to images. As a result of coloring based on the common correspondence as described above, the same height is displayed in the same color, which is suitable for observation of height comparison. In the example of FIG. 32, the calibration function is executed at the same time, and the display magnification of the right comparison 3D image is changed and displayed on the same scale as the left reference 3D image. As a result, not only the height but also the size is unified, and a more accurate comparison is possible. Of course, since each image can be independently enlarged and reduced, rotated, moved, etc., it is possible to easily enlarge and observe a necessary part. In addition, as described above, the viewpoint can be changed by enlarging / reducing, rotating, or moving in conjunction with each other. Furthermore, the transmittance of the three-dimensional colored image can be adjusted by the transmittance adjustment slider 328. For example, when it is desired to check the surface state of the sample at the same scale, only the original three-dimensional image can be displayed with the transmittance of the three-dimensional colored image set to 0% as shown in FIG. Thus, the original three-dimensional image and the three-dimensional colored image can be displayed in a superimposed state with a transmittance of 50%, and the surface state and height can be observed simultaneously. Further, as described above, the color gauge 402 can be displayed in each display area, and the display of the scale grid 404 can be turned ON / OFF. In particular, since the viewpoint can be changed independently for each image, if the scale grid 404 is rotated while the scale grid 404 is displayed, the scale grid 404 is also rotated and the position of the current viewpoint is determined from the display state of the scale grid 404. Since it can confirm, the advantage which can grasp | ascertain the position of the viewpoint in each image is acquired. Thus, the absolute value comparison mode can be suitably used for applications in which a plurality of samples are compared and observed.

In the above example, two examples of the reference three-dimensional image and the one comparison three-dimensional image are displayed. Needless to say, three or more images can be displayed. The plurality of images are displayed in a list by dividing the display unit 52 into a plurality of display areas. Alternatively, a separate window may be opened and displayed individually for each window.
[Expression pattern]

  As described above, the three-dimensional colored image is generated by expressing the height with the color. However, the height is not limited to the color, and the height can be expressed using other expression patterns that can be expressed by the display unit 52. . For example, as an expression pattern, it is possible to use a method of changing the surface pattern such as brightness, brightness, shading or texture or hatching instead of color, or adding contour lines having different colors or hatching patterns for each predetermined height. In particular, when luminance is used as an expression pattern, the height can be expressed in light and dark when the display screen cannot be expressed in monochrome. On the contrary, in the case of the display unit 52 having excellent color reproducibility, the expression pattern can be set so as to change any one of brightness, hue, and saturation, and a plurality of expression patterns can be combined. it can. In this way, even when the expression pattern image generation unit generates a three-dimensional expression pattern image instead of the coloring image generation unit instead of the coloring image generation unit, the height can be expressed by various expression patterns on the display unit. .

  The magnified observation apparatus, magnified image observation method, magnified observation operation program, and computer-readable recording medium or recorded apparatus of the present invention are used in a microscope or a digital microscope, and a plurality of samples to be observed are displayed on the same screen. Display and contrast.

1 is an external view of a magnification observation apparatus according to an embodiment of the present invention. It is a block diagram of the magnification observation device concerning a 1st embodiment of the present invention. It is a block diagram of the magnification observation apparatus which concerns on the 2nd Embodiment of this invention. It is a graph which shows the change of the received light data with respect to height z. It is a flowchart which shows an example of the imaging method of three-dimensional data. It is a flowchart explaining one procedure which produces | generates a synthesized image from several two-dimensional image data. It is an image figure which shows the interface screen of the operation program for magnification observation which concerns on one embodiment of this invention. It is an image figure which shows the state which changed the viewpoint of the three-dimensional image of the display area B from the state of FIG. It is an image figure which shows the state which changed the viewpoint of the three-dimensional image of the display area A from the state of FIG. It is an image figure which shows the state which changed the viewpoint of the three-dimensional image of the display areas A and B simultaneously from the state of FIG. It is an image figure which shows the state which changed the expansion ratio of the three-dimensional image of the display areas A and B. FIG. It is an image figure which shows the state which changed the expansion ratio of the three-dimensional image of the display areas A and B. FIG. It is an image figure which shows the state which changed the expansion ratio of the three-dimensional image of the display areas A and B. FIG. It is an image figure which shows the example which the display state of two images does not correspond. It is an image figure which shows the example which made the display state of the image of FIG. 14 correspond. It is an image figure which shows the interface screen of the operation program for magnification observation which concerns on other embodiment of this invention. It is a flowchart explaining one procedure which aligns a workpiece | work. It is an image figure which shows an example of the user interface screen of the controller part of the operation program for magnification observation. FIG. 19 is an image diagram for displaying a reference three-dimensional image based on the operation of FIG. 18. It is an image figure which displays the reference | standard 3D image of FIG. 19, and the 3D image of a comparison object. It is an image figure which shows the example which changes and displays a comparison 3D image by the same magnification as a reference | standard 3D image from the screen of FIG. It is an image figure which shows the example which changed the viewpoint which displays the reference | standard 3D image of FIG. It is an image figure which displays the reference | standard 3D image of FIG. 22, and the 3D image of a comparison object. It is an image figure which shows the example which changes and displays the viewpoint of a comparison 3D image in the same way as a reference | standard 3D image from the screen of FIG. FIG. 25 is an image diagram illustrating an example in which the comparison three-dimensional image is further changed to the same magnification as the reference three-dimensional image and displayed from the screen of FIG. 24. It is a block diagram which shows the structure of the control part which produces | generates a three-dimensional coloring image. It is an image figure which shows an example of a three-dimensional image. It is an image figure which shows an example which converted the three-dimensional image of FIG. 27 into the three-dimensional coloring image. It is an image figure which shows the example which displayed the scale grid on the three-dimensional coloring image of FIG. It is an image figure which shows the example which set the transmittance | permeability of the three-dimensional coloring image of FIG. 29 to 50%. It is an image figure which shows the example which displays a some three-dimensional coloring image in normal mode. It is an image figure which shows the example which displays a some three-dimensional coloring image in absolute value comparison mode. FIG. 32 is an image diagram showing an example in which the original three-dimensional image is displayed with the transmittance of the three-dimensional colored image in FIG. 31 set to 0%. It is an image figure which shows the example which made the transmittance | permeability 50% of the three-dimensional coloring image of FIG. It is an image figure which shows the example which displayed the original three-dimensional image by making the transmittance | permeability of the three-dimensional coloring image of FIG. 32 0%. It is an image figure which shows the example which made the transmittance | permeability of the three-dimensional coloring image of FIG. 32 50%.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Imaging part; 10a ... Camera 11 ... Optical system 12, 212 ... CCD; 13, 213 ... CCD control circuit 14 ... Optical path shift part 20 ... Stage elevator 21 ... Stepping motor 22 ... Motor control circuit 30 ... Stage 41 ... Stand base 42 ... Stand 43 ... Camera mounting part 50 ... Information processing device 51 ... Control part 52, 52B ... Display unit 53 ... Memory; 53A ... Storage unit 54 ... Interface 55 ... Operating unit; 55A ... Pointing device 55a ... Mouse; 55b ... Left button; 55c ... right button 55d ... scroll button; 55e ... mouse pointer 60 ... illumination unit; 60A ... epi-illumination; 60B ... transmitted illumination 61 ... light Eber 62 ... Connector 70 ... Computer 100 ... First optical system 101 ... Laser 102 ... First collimating lens 103 ... Polarizing beam splitter 104 ... 1/4 wavelength plate 105 ... Horizontal deflection device; 106 ... Vertical deflection device 107 ... First relay lens; 108 ... Second relay lens 109 ... Objective lens 110 ... Imaging lens 111 ... Pinhole plate 112 ... photodiode 113 ... A / D converter 115 ... laser driving circuit 200 ... second optical system 201 ... white lamp 202 ... second collimating lens 203 ... first half mirror; 204 ... second half mirror 300 ... controller unit 302 ... "file selection" column 304 ... "display 3D" button 306 ... "Close 3D" button 310 ... "3D file comparison" field 312 ... "Comparison mode"button; 313 ... "Sub-switch" button 314 ... "Interlocking mode" button 316... "Synchronize set" button 320... "Height adjustment" slider 322... "Save"button; 324 ... "Reset" button 326 ... "Height / color" button 328 ... Transmittance adjustment slider 330 ... "scale" button 402 ... color gauge 404 ... scale grid 512 ... allocation unit 514 ... colored image generation unit 516 ... image adjustment unit 518 Mode switching unit 520 ... Transmittance adjustment unit 522 ... Magnification adjustment unit 524 ... View point adjustment unit S ... Sample; A, B ... Display area; W1, W2 ... Workpiece

Claims (16)

To display an imaging unit for imaging an observation target, a control unit that generates a three-dimensional observation image based on a signal acquired by the imaging unit, and a three-dimensional observation image generated by the control unit A magnifying observation apparatus comprising:
The three-dimensional image includes height information, a distribution of height information included in the three-dimensional image data is measured, and an expression pattern of pixels having a constant changeable range that can be expressed by the display unit is An assigning unit that assigns a correspondence between each height and the expression pattern so that the height information can be represented by a change in the expression pattern in correspondence with the distribution of the height information,
An expression pattern image generation unit that assigns an expression pattern to each of the height information of the three-dimensional image based on a correspondence relationship of the assignment unit, and generates a three-dimensional expression pattern image corresponding to the height;
With
The display unit can simultaneously display a plurality of three-dimensional images,
In addition, for a plurality of three-dimensional images, the corresponding relationship between each height and the expression pattern when the expression pattern image generation unit generates each three-dimensional expression pattern image is common to all three-dimensional images. A magnifying observation device. To display an imaging unit for imaging an observation target, a control unit that generates a three-dimensional observation image based on a signal acquired by the imaging unit, and a three-dimensional observation image generated by the control unit A magnifying observation apparatus comprising:
The three-dimensional image includes height information, and based on the distribution of height information included in the three-dimensional image data, the maximum and minimum heights are detected, and the pixel expression pattern that can be expressed by the display unit The expression pattern of a certain range having the maximum value and the minimum value is made to correspond to the distribution of the height information, and the maximum height and the minimum height are made to correspond to the maximum value and the minimum value of the expression pattern, respectively. An assigning unit for assigning a correspondence between each height and the expression pattern,
An expression pattern image generation unit that assigns an expression pattern to each of the height information of the three-dimensional image based on a correspondence relationship of the assignment unit, and generates a three-dimensional expression pattern image corresponding to the height;
With
The display unit can simultaneously display a plurality of three-dimensional images,
In addition, for a plurality of three-dimensional images, the corresponding relationship between each height and the expression pattern when the expression pattern image generation unit generates each three-dimensional expression pattern image is common to all three-dimensional images. A magnifying observation device. The magnification observation apparatus according to claim 2,
When the assignment unit assigns the correspondence of the expression pattern, the maximum height and the minimum height are detected based on the distribution of height information included in the reference three-dimensional image among the plurality of three-dimensional images, The expression pattern in a certain range is made to correspond to the distribution of the height information, and the maximum height and the minimum height included in one three-dimensional image are made to correspond to the maximum value and the minimum value of the expression pattern, respectively. Assign the correspondence between each height and expression pattern,
An enlargement observation apparatus, wherein the correspondence relationship is executed for all three-dimensional images, and the expression pattern image generation unit generates a three-dimensional expression pattern image according to a unified correspondence relationship. The magnification observation apparatus according to claim 3,
The magnification observation apparatus, wherein the reference three-dimensional image is a three-dimensional image having a maximum difference between a maximum height and a minimum height among a plurality of three-dimensional images. The magnification observation apparatus according to claim 2,
When the assigning unit assigns the correspondence between the expression patterns, based on the distribution of all the height information included in the plurality of three-dimensional images, the maximum and minimum heights are detected, and the expression pattern in the certain range is Corresponding to the distribution of height information, among the heights included in all three-dimensional images, the maximum height and the minimum height are made to correspond to the maximum value and the minimum value of the expression pattern, respectively. Assign a correspondence
An enlargement observation apparatus, wherein the correspondence relationship is executed for all three-dimensional images, and the expression pattern image generation unit generates a three-dimensional expression pattern image according to a unified correspondence relationship. The magnification observation device according to claim 1,
The magnification observation apparatus, wherein the expression pattern is at least one of color and luminance. To display an imaging unit for imaging an observation target, a control unit that generates a three-dimensional observation image based on a signal acquired by the imaging unit, and a three-dimensional observation image generated by the control unit A magnifying observation apparatus comprising:
A storage unit that holds three-dimensional image data including height information regarding the three-dimensional image;
Based on the distribution of height information of the 3D image data, the maximum height and the minimum height are detected,
A color palette that expresses the height by displaying gradations in a predetermined range of colors is associated with the distribution of the height information, and the maximum height and minimum height are the maximum and minimum colors in the range of the color palette. An assigning unit for assigning a correspondence between each height and color so that each corresponds to
For each height information of the three-dimensional image, a colored image generating unit that assigns a color based on the correspondence relationship of the assigning unit and generates a three-dimensional colored image corresponding to the height;
With
For a plurality of three-dimensional images, based on a reference three-dimensional image serving as a reference, the assigning unit determines a correspondence relationship based on the distribution of the height information, and this correspondence relationship is also applied to other three-dimensional images. A magnifying observation apparatus configured to display a plurality of three-dimensional colored images generated according to a unified correspondence relationship on a display unit. The magnification observation apparatus according to claim 7,
In the display unit, a color gauge indicating a distribution of colors assigned to the three-dimensional image can be displayed together with a representative value of the height assigned to each color in a display area for displaying each three-dimensional image. A magnifying observation device. The magnification observation apparatus according to claim 7,
The display unit can display a 3D colored image superimposed on each 3D image,
And the three-dimensional colored image displayed superimposed on each three-dimensional image can have transparency,
A magnification observation apparatus comprising a transmittance adjustment unit capable of adjusting a transmittance of a three-dimensional colored image. The magnification observation apparatus according to claim 9,
The 3D image data further includes information on the size of the 3D image;
Further, when displaying a plurality of three-dimensional images on the display unit, the size of the object displayed in the three-dimensional image to be compared is determined based on the information about the size of the reference three-dimensional image. A magnification observation apparatus comprising a magnification adjustment unit for adjusting to match that of a dimensional image. The magnification observation apparatus according to claim 9, further comprising:
When displaying a plurality of 3D images on the display unit, based on the viewpoint of the object displayed as the reference 3D image, the viewpoint of the object displayed as the comparison target 3D image is set as the reference 3D. A magnification observation apparatus comprising a viewpoint adjustment unit for adjusting the image to match that of an image. The magnification observation apparatus according to claim 1, further comprising:
A magnification observation apparatus configured to be able to designate a height as a reference for superposition when a plurality of three-dimensional images are superposed by the viewpoint adjustment unit. The magnification observation apparatus according to any one of claims 7 to 12, further comprising: for a plurality of three-dimensional images, a maximum color and a maximum value included in each three-dimensional image with a maximum color of a color palette. And a normal mode in which a colored image generating unit generates a three-dimensional colored image and displays it on the display unit based on an independent correspondence for each three-dimensional image.
Among the plurality of three-dimensional images, the maximum and minimum values of the color palette are assigned to the maximum and minimum heights included in the reference three-dimensional image, and the colored image generator is three-dimensional based on a common correspondence. An absolute value comparison mode for generating a colored image and displaying it on the display unit;
A magnification observation apparatus comprising a mode switching unit for switching between the two. A magnified observation method capable of capturing a plurality of three-dimensional observation images and displaying them on a display unit, and comparing them with each other,
Constructing a plurality of three-dimensional image data and storing the data including height information;
Selecting a plurality of image data to be compared and displaying them side by side on a display unit;
Based on the distribution of height information of the 3D image data, the maximum height and the minimum height are detected,
A color palette that expresses the height by displaying gradations in a predetermined range of colors is associated with the distribution of the height information, and the maximum height and minimum height are the maximum and minimum colors in the range of the color palette. Assigning the correspondence between each height and color so that each corresponds to
Assigning each color based on the correspondence of the assigning unit for each height information of the three-dimensional image, and generating a three-dimensional colored image corresponding to the height;
Displaying each three-dimensional colored image superimposed on the surface of each three-dimensional image displayed on the display unit;
Have
The step of assigning a correspondence relationship between each height and color determines a correspondence relationship based on a distribution of height information included in a reference three-dimensional image serving as a reference among a plurality of three-dimensional images. A magnified observation method, which is also performed on a three-dimensional image. An operation program for magnifying observation that allows a plurality of three-dimensional observation images to be captured and displayed on the display unit, and to be compared with each other. The ability to hold information including
A function for selecting a plurality of image data to be compared and displaying them side by side on the display;
Based on the distribution of height information included in the reference three-dimensional image data, the maximum height and the minimum height are detected, and a color palette that expresses the height by displaying gradations in a predetermined range of colors is displayed on the height palette. A function for assigning correspondence between each height and color so that the maximum height and the minimum height correspond to the maximum color and the minimum color in the range of the color palette, corresponding to the distribution of information,
A function of assigning each color based on the correspondence of the assigning unit for each height information of a plurality of three-dimensional images, and generating a three-dimensional colored image corresponding to the height;
A function of superimposing and displaying each three-dimensional colored image on the surface of each three-dimensional image image displayed on the display unit;
A function of adjusting the transmittance of a three-dimensional colored image displayed superimposed on each three-dimensional image;
A function for adjusting the display magnification of the three-dimensional image to be compared to the same magnification based on information on the size of the reference three-dimensional image;
An operation program for magnifying observation characterized by realizing the above.   A computer-readable recording medium or a recorded device on which the magnified observation operation program according to claim 15 is recorded.