JP4266333B2 - Magnification observation apparatus, image file generation apparatus, image file generation program, and computer-readable recording medium - Google Patents

Description

  The present invention relates to a magnification observation apparatus such as a microscope or a microscope that captures and displays an enlarged image, an image file generation apparatus, an image file generation program, and a computer-readable recording medium.

  Today, an optical microscope using an optical lens, a digital microscope, and the like are used as a magnifying observation device that magnifies and displays a minute object. The microscope includes a light receiving element such as a CCD that electrically reads reflected light or transmitted light from the observation target fixed to the observation target fixing unit incident via the optical system for each pixel arranged in a two-dimensional manner. Prepare. An image electrically read using a CCD is displayed on a display unit such as a display (for example, Patent Document 1).

  In a magnifying observation device, in general, when the magnifying power increases, the depth of focus becomes shallower and the in-focus area becomes narrower. Therefore, it is difficult to observe the whole if there are irregularities or height differences in the sample (workpiece) to be observed. Become. For this reason, a technique is used in which an image focused on the entire image is created by depth synthesis. In depth synthesis, the lens or sample is moved in the height direction, the relative distance in the optical axis direction is changed, multiple images are taken, and the in-focus part is extracted and synthesized, resulting in a deep focus depth. Take an image. In addition, when shooting a plurality of images, if the amount of movement of the lens or the observation target is recorded at the same time, the height information of the observation target can be obtained when creating the composite image. It is also possible to construct as three-dimensional data. By reproducing the 3D image on the display unit of the magnification observation apparatus based on the three-dimensional data, the image can be observed and evaluated three-dimensionally from various viewpoints and angles, and the surface shape such as the height can be measured. .

  When the three-dimensional image data constructed in this way is stored for reuse, it is generally stored as a separate file from the two-dimensional image. As a result, it is necessary to manage a plurality of files according to the purpose for the same observation target. In particular, since these three-dimensional image data files are not normally associated with the two-dimensional image data files, it is necessary to devise such as giving the same file name on the user side, which makes management difficult. In addition, when the number of files increases, it becomes difficult to grasp the correspondence between files, and there is a possibility that necessary files cannot be found immediately, or files may be scattered. In addition, file storage and opening are required for each of the two-dimensional image data and the three-dimensional image data, and the operation becomes complicated.

In addition, the standardized general-purpose data format is widely used as the format of the two-dimensional image data. For example, application software capable of displaying two-dimensional image data such as jpeg and tiff is widely used. Can be handled easily. On the other hand, application software that can use three-dimensional data is not common, and there are many unique file formats. For this reason, 3D data cannot be used in many environments, and a user who does not use dedicated application software that can handle 3D data cannot handle this.
JP 2000-214790 A

  The present invention has been made to solve such problems. A main object of the present invention is to provide a magnification observation apparatus, an image file generation apparatus, an image file generation program, and a computer-readable recording medium that can easily manage three-dimensional data.

In order to achieve the above object, a magnifying observation apparatus according to claim 1 of the present invention includes an imaging unit for imaging an observation target and acquiring two-dimensional observation image data, and a plurality of 2 acquired by the imaging unit. A control unit for generating three-dimensional observation image data based on the three-dimensional observation image data and height information at the time of imaging, and a display for displaying the three-dimensional observation image data generated by the control unit A unit, an output unit for embedding three-dimensional image data displayed on the display unit in arbitrary two-dimensional image data, and outputting it as one image data file, and an image data file output by the output unit An image data storage unit for storing the 2D image data into which the 3D image data is embedded in the output unit is acquired as a desired display state of the 3D image displayed on the display unit 2 Dimensional painting And wherein the Oh Rukoto in the data.

  A magnification observation apparatus according to a second aspect is the magnification observation apparatus according to the first aspect, wherein information relating to the three-dimensional image data embedded in the two-dimensional image data by the output unit is in a two-dimensional image data format. It records in the area which can be used by the user.

  Furthermore, the magnification observation apparatus according to claim 3 is the magnification observation apparatus according to claim 2, wherein three-dimensional image data is used as information regarding the three-dimensional image data embedded in the two-dimensional image data by the output unit. Information including at least one of flag data indicating that the images are combined, a combined area address of the three-dimensional image data, and a size of the three-dimensional image data is recorded in an area available to the user in the two-dimensional image data format. It is characterized by that.

Furthermore, the magnified observation device according to claim 4 is the magnified observation device according to any one of claims 1 to 3 , wherein the output unit is a data region of the three-dimensional image data embedded in the two-dimensional image data. Is compressed.

Furthermore, the magnification observation apparatus according to claim 5 is the magnification observation apparatus according to any one of claims 1 to 4 , wherein the format of the two-dimensional image data output from the output unit is jpeg or tiff. It is either.

The image file generation device according to claim 6 is an input unit for acquiring two-dimensional image data and height information at the time of capturing the two-dimensional image data, and a plurality of two-dimensional observations acquired by the input unit. A control unit for generating three-dimensional image data based on the image data and the relative height information, and the three-dimensional image data generated by the control unit is embedded in arbitrary two-dimensional image data, so that one image An output unit for outputting as a data file, and an image data storage unit for storing the image data file output by the output unit, and the two-dimensional image data into which the three-dimensional image data is embedded by the output unit but wherein the Oh Rukoto the three-dimensional image displayed on the display unit a two-dimensional image data obtained as a desired display state.

The image file generation device according to claim 7 is the image file generation device according to claim 6 , wherein information relating to the three-dimensional image data embedded in the two-dimensional image data by the output unit is obtained from the two-dimensional image data. It is characterized in that it is recorded in an area that can be used by the user during formatting.

Furthermore, an image file generation device according to claim 8 is the image file generation device according to claim 7 , wherein the output file includes three-dimensional information as information relating to the three-dimensional image data embedded in the two-dimensional image data. Area in which information including at least one of flag data indicating that image data is combined, 3D image data combining area address, and 3D image data size is available to the user in the 2D image data format It is characterized by recording.

Furthermore, an image file generation device according to claim 9 is the image file generation device according to any one of claims 6 to 8 , wherein the output unit stores the three-dimensional image data embedded in the two-dimensional image data. The data area is compressed.

Furthermore, an image file generation device according to claim 10 is the image file generation device according to any one of claims 6 to 9 , wherein the format of the two-dimensional image data output from the output unit is jpeg or It is one of tiff.

An image file generation program according to claim 11 has a function of acquiring two-dimensional image data and relative height information at the time of imaging the two-dimensional image data, and a plurality of acquired two-dimensional observation image data and relative height. A function for generating three-dimensional image data based on information, a function for embedding the generated three-dimensional image data in arbitrary two-dimensional image data, and outputting it as one image data file, and output image data to realize a function to save the file to the computer, two-dimensional image data of the previous embedding the three-dimensional image data, characterized Oh Rukoto a two-dimensional image data acquired three-dimensional image data as a desired display state And

Furthermore, an image file generation program according to claim 12 is the image file generation program according to claim 11 , wherein information relating to the three-dimensional image data embedded in the two-dimensional image data is received by the user in the two-dimensional image data format. Is recorded in an available area.

Furthermore, an image file generation program according to claim 13 is the image file generation program according to claim 12 , wherein flag data indicating that three-dimensional image data is combined as information about the three-dimensional image data. Information including at least one of the combined area address of the three-dimensional image data and the size of the three-dimensional image data is recorded in an area usable by the user in the two-dimensional image data format.

Furthermore, an image file generation program according to claim 14 is the image file generation program according to any one of claims 11 to 13 , wherein the data area of the three-dimensional image data embedded in the two-dimensional image data is compressed. It is characterized by doing.

Furthermore, the image file generation program according to claim 15 is the image file generation program according to any one of claims 11 to 14, wherein the format of the two-dimensional image data is either jpeg or tiff. It is characterized by.

A computer-readable recording medium according to a sixteenth aspect stores the image file generation program according to any one of the eleventh to fifteenth aspects. Recording media include CD-ROM, CD-R, CD-RW, flexible disk, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, Blu-ray disk, etc. Media that can store programs such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, and the like. The above-mentioned program includes a form that can be downloaded via a network.

  According to the magnification observation apparatus, image file generation apparatus, image file generation program, and computer-readable recording medium of the present invention, three-dimensional image data can be embedded in two-dimensional image data that is generally easy to use. As a result, the user only has to save one file, and it is not necessary to save and manage the three-dimensional image data and the two-dimensional image data separately, and the handling of the file becomes easy. In addition, since 2D image data and 3D image data are stored in one file, viewing of 2D image data is a general-purpose display even if the user does not have application software that can handle 3D image data. This is possible using software. If there is dedicated application software that can use 3D image data, the embedded 3D image data can be used.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiment described below exemplifies a magnification observation device, an image file generation device, an image file generation program, and a computer-readable recording medium for embodying the technical idea of the present invention, The present invention does not specify a magnification observation apparatus, an image file generation apparatus, an image file generation program, and a computer-readable recording medium as follows.

  Further, the present specification by no means specifies the members shown in the claims to the members of the embodiments. In particular, the dimensions, materials, shapes, relative arrangements, and the like of the component parts described in the embodiments are not intended to limit the scope of the present invention unless otherwise specified, and are merely explanations. It is just an example. Note that the size, positional relationship, and the like of the members shown in each drawing may be exaggerated for clarity of explanation. Furthermore, in the following description, the same name and symbol indicate the same or the same members, and detailed description thereof will be omitted as appropriate. Furthermore, each element constituting the present invention may be configured such that a plurality of elements are constituted by the same member and the plurality of elements are shared by one member, and conversely, the function of one member is constituted by a plurality of members. It can also be realized by sharing.

The connection between the magnification observation apparatus used in the embodiment of the present invention and the computer, printer, external storage device and other peripheral devices for operation, control, display, and other processing connected thereto is, for example, IEEE 1394, RS-232x, RS-422, serial connection such as USB, parallel connection, or communication by electrical, magnetic, or optical connection via a network such as 10BASE-T, 100BASE-TX, 1000BASE-T Do. The connection is not limited to a physical connection using a wire, but may be a wireless connection using radio waves such as IEEE802.1x, OFDM, etc., Bluetooth (registered trademark) , infrared rays, optical communication, or the like. Furthermore, a memory card, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used as a recording medium for exchanging data or storing settings. In this specification, the term “magnification observation apparatus” is used to include not only the magnification observation apparatus body but also an imaging system in which peripheral devices such as a computer and an external storage device are combined.

  Further, in this specification, the magnification observation apparatus, the image file generation apparatus, and the image file generation program are a system itself that generates an image file including three-dimensional image data, and input / output, display, calculation, and communication related to image file generation. The present invention is not limited to an apparatus or method that performs other processing in hardware. An apparatus and method for realizing processing by software are also included in the scope of the present invention. For example, a general-purpose circuit or computer that incorporates software, programs, plug-ins, objects, libraries, applets, compilers, modules, macros that operate on specific programs, etc., and enables image generation itself or related processing And the system also correspond to the magnification observation apparatus, the image file generation apparatus, and the image file generation program of the present invention. In this specification, the computer includes a general-purpose or dedicated computer, a workstation, a terminal, a portable electronic device, PDC, CDMA, W-CDMA, FOMA (registered trademark), GSM, IMT2000, fourth generation, and the like. Mobile phones, PHS, PDAs, pagers, smartphones and other electronic devices. Further, in the present specification, the program is not limited to a program that is used alone, an aspect that functions as a part of a specific computer program, software, service, etc., an aspect that is called and functions when necessary, an environment such as an OS, etc. It can also be used as a mode provided as a service, a mode that operates resident in the environment, a mode that operates in the background, and other support programs.

[First Embodiment]
The magnification observation apparatus according to the first embodiment of the present invention will be described below with reference to FIGS. As shown in FIG. 1, the magnification observation apparatus includes an illumination unit 60 for illuminating a sample to be observed, an imaging unit 10 for imaging the sample illuminated by the illumination unit 60, and an enlarged image captured by the imaging unit 10. The information processing apparatus 50 which has the display part 52 which displays is provided. Further, the magnified observation apparatus of FIG. 1 includes a sample fixing unit (stage 30 on which the sample S is mounted) for fixing the sample, and reflected light from the sample S fixed to the sample fixing unit incident via the optical system 11 or An image sensor (CCD 12) that electrically reads transmitted light, and a focus adjustment unit (stage elevator 20) that adjusts the focal point by changing the relative distance between the sample fixing unit and the optical system 11 in the optical axis direction. Furthermore, as shown in FIG. 2, the information processing apparatus 50 displays the focal length information about the relative distance in the optical axis direction between the sample fixing unit and the optical system 11 when the focal point is adjusted by the focal point adjusting unit. A focal length information storage unit (memory 53) that stores the two-dimensional position information of the sample in a vertical plane, a display unit 52 that displays an image read by the image sensor, and one of the images displayed by the display unit 52 A region setting unit (operation unit 55, pointing device 55A) capable of setting at least one region of the unit, and a focal length information storage unit regarding a part or all of the sample S corresponding to the region set by the region setting unit An arithmetic unit (control) that calculates an average height in the optical axis direction of the sample S corresponding to the region set by the region setting unit based on the focal length information that has been set 51) and a. This magnifying observation apparatus uses an image sensor that electrically reads reflected light or transmitted light from a sample fixed to a sample fixing portion incident via an optical system, and uses the optical axis of the sample corresponding to a specified region. The average height (depth) in the direction can be calculated.

  As shown in FIG. 2, the imaging unit 10 is optically applied to a stage 30 that is one form of a sample fixing unit on which the sample S is placed, a stage elevator 20 that moves the stage 30, and a sample fixed to the stage 30. A CCD 12 as one form of an imaging device that electrically reads reflected light or transmitted light incident through the system for each pixel arranged two-dimensionally, and a CCD control circuit 13 that drives and controls the CCD 12 Prepare. Furthermore, the information processing apparatus 50 which is a magnification observation apparatus main body is connected to the imaging part 10. FIG. The information processing apparatus 50 displays an image based on the memory 53 as one form of an image data storage unit that stores image data electrically read by the image sensor and image data electrically read by the image sensor. A display unit 52 such as a display and a monitor, an operation unit 55 that performs input and other operations based on a screen displayed on the display unit 52, and image processing and various other processes based on information input by the operation unit 55 The control part 51 which performs is provided. The display constituting the display unit 52 is a monitor capable of high-resolution display, and a CRT, a liquid crystal panel, or the like is used.

  The operation unit 55 is connected to the computer by wire or wirelessly, or is fixed to the computer. Examples of the general operation unit 55 include various pointing devices such as a mouse, keyboard, slide pad, track point, tablet, joystick, console, jog dial, digitizer, light pen, numeric keypad, touch pad, and accu point. Further, these operation units 55 can be used for operations of the magnification observation apparatus itself and its peripheral devices in addition to the operations of the magnification observation operation program. Furthermore, using a touch screen or touch panel on the display itself that displays the interface screen, the user can directly input and operate the screen by hand, or use voice input or other existing input means, Or these can also be used together. In the example of FIG. 1, the operation unit 55 includes a pointing device 55A such as a mouse 55a. The operation unit 55 functions as a changing unit for changing an observation viewpoint and an enlargement / reduction ratio, which will be described later.

  FIG. 1 shows an external view of a magnification observation apparatus according to an embodiment of the present invention. A camera 10 a having an optical system and an image sensor is attached to a camera attachment portion 43 fixed to a support column 42 extending in the vertical direction from the stand base 41. On the stand base 41, a stage elevator 20 on which a stage 30 on which the sample S is placed is attached is arranged. The camera 10a and the stage elevator 20 are connected to the information processing apparatus 50 and controlled. The information processing apparatus 50 includes a display unit 52 and an operation unit 55 such as a mouse 55a. An observation image is displayed on the display unit 52.

  Further, a computer 70 can be connected to the magnification observation apparatus which is the information processing apparatus 50, and a magnification observation operation program can be separately installed in the computer 70 so that the magnification observation apparatus can be operated from the computer 70 side. In this specification, the operation program for magnifying observation that operates the magnifying observation apparatus using a computer is an operation program installed in a general purpose or dedicated computer externally connected to the magnifying observation apparatus, as well as the above-described magnifying observation apparatus. It also includes an operation program incorporated in the information processing apparatus 50 as a control unit. The magnifying observation apparatus has a built-in operation function or operation program for operating the magnifying observation apparatus in advance. This operation program can be installed or updated in the magnifying observation apparatus in the form of rewritable software, firmware, or the like. Accordingly, the computer that executes the operation program for magnification observation in this specification includes the magnification observation apparatus itself.

  FIG. 2 is a block diagram of the magnification observation apparatus according to the embodiment of the present invention. The information processing apparatus 50 includes a display unit 52, a memory 53 that stores control programs, focal length information, received light data, two-dimensional information, and the like, and the information processing apparatus 50 communicates data with the camera 10 a and the stage elevator 20. Interface 54, and an operation unit 55 for an operator to perform operations related to the magnification observation apparatus. The stage elevator 20 includes, for example, a stepping motor 21 and a motor control unit 22 that controls the elevation of the stepping motor 21. The imaging unit 10 is a light-receiving element such as a CCD 12 as an imaging element, a CCD control circuit 13 that drives and controls the CCD 12, and transmission of light irradiated from the illumination unit 60 to the sample S placed on the stage 30. And an optical system 11 that forms an image of light and reflected light on the CCD 12.

[Pixel shifting means]
Furthermore, the imaging unit 10 can include pixel shifting means for obtaining a high resolution that is higher than the resolution of the CCD 12 by pixel shifting. Pixel shift is to increase the resolution by combining an image taken by shifting the subject by half the pixel pitch and an image before shifting. As a typical image shifting mechanism, there are a CCD driving method for moving an image sensor, an LPF tilting method for tilting an LPF, a lens moving method for moving a lens, and the like. In FIG. 2, the incident light path of reflected light or transmitted light incident on the CCD 12 from the sample S fixed to the stage 30 via the optical system 11 is set in at least one direction, and the interval of one pixel of the CCD 12 in that direction. An optical path shift unit 14 that optically shifts at a smaller distance is provided. In one embodiment of the present invention, the mechanism and method for realizing pixel shifting are not limited to the above configuration, and a known method or a method developed in the future can be used as appropriate.

  The information processing device 50 inputs control data related to the control of the stepping motor 21 to the motor control circuit 22, thereby allowing the stage 30 as the sample fixing unit, the camera 10 a having the optical system 11 and the CCD 12 as the imaging device, and The relative distance in the optical axis direction, here the height in the z direction is changed. Specifically, the information processing device 50 controls the rotation of the stepping motor 21 by inputting control data necessary for controlling the stage elevator 20 to the motor control circuit 22, and the height z (z direction) of the stage 30. Up and down). The stepping motor 21 generates a rotation signal corresponding to the rotation. The information processing apparatus 50 stores the height z of the stage 30 as information regarding the relative distance between the sample fixing unit and the optical system 11 in the optical axis direction based on the rotation signal input via the motor control circuit 22. In the present embodiment, the example in which the relative distance in the optical axis direction between the sample fixing unit and the optical system is changed by changing the height of the stage 30 is shown. The height, for example, the height of the camera 10a may be changed.

  The CCD 12 can electrically read the amount of received light for each pixel arranged two-dimensionally in the x and y directions. The image of the sample S formed on the CCD 12 is converted into an electric signal according to the amount of received light at each pixel of the CCD 12 and further converted into digital data in the CCD control circuit 13. The information processing apparatus 50 uses the digital data converted by the CCD control circuit 13 as received light data D in a plane (x and y directions in FIG. 2) substantially perpendicular to the optical axis direction (z direction in FIG. 2). The information is stored in the memory 53 together with pixel arrangement information (x, y) as two-dimensional position information of the sample. Here, the in-plane substantially perpendicular to the optical axis direction does not need to be a plane that makes an angle of 90 ° with respect to the optical axis. Any observation surface may be used as long as it is within the tilt range.

  In the above description, an example in which the sample is placed on the stage is shown as an example of the sample fixing unit. However, for example, an arm may be attached instead of the stage and the sample may be fixed to the tip. Further, the camera 10a can be used by being mounted on the camera mounting portion 43, and can be detachably disposed at a desired position and angle by a hand-held method.

  The illumination unit 60 shown in FIG. 1 includes an epi-illumination 60A for irradiating the sample with epi-illumination light and a transmission illumination 60B for irradiating the sample with transmitted light. These lights are connected to the information processing apparatus 50 via the optical fiber 61. The information processing apparatus 50 includes a connector 62 for connecting the optical fiber 61 and a light source (not shown) for sending light to the optical fiber 61 via the connector 62. A halogen lamp or the like is used as the light source.

[Control unit 51]
The control unit 51, which is a control unit, controls the captured observation image to be converted into a resolution that can be displayed on the display unit 52 and displayed. In the magnification observation apparatus of FIG. 1, the imaging unit 10 displays an observation image obtained by imaging the sample S with the CCD 12 on the display unit 52. In general, the performance of an image sensor such as a CCD often exceeds the display capability of the display unit. Therefore, in order to display the captured observation image on one screen, the resolution can be displayed on one screen by thinning the image. Reduced to size and reduced. If the reading resolution when reading by the imaging unit 10 is the first resolution, the display unit 52 displays a second resolution lower than the first resolution.

[Second Embodiment]
Next, a magnification observation apparatus according to a second embodiment of the present invention will be described with reference to FIG. In the magnification observation apparatus according to the second embodiment, the camera serving as the imaging unit transmits the reflected light of the light from the first light source (laser 101) irradiated to the sample S via the first optical system 100. The second optical system reflects the reflected light of the first imaging unit that receives light by the first light receiving element (photodiode 112) and the second light source (white lamp 201) irradiated to the sample S. And a second imaging unit that receives light by the second light receiving element (CCD 212) via 200.

  First, the first imaging unit will be described. The first optical system 100 includes a laser 101 that irradiates a sample S with monochromatic light (for example, laser light), a first collimating lens 102, a polarizing beam splitter 103, a quarter-wave plate 104, a horizontal deflection device 105, and vertical deflection. A device 106, a first relay lens 107, a second relay lens 108, an objective lens 109, an imaging lens 110, a pinhole plate 111, and a photodiode 112 are included.

  For example, a semiconductor laser 101 that emits red laser light is used as the first light source. The laser light emitted from the laser 101 driven by the laser driving circuit 115 passes through the first collimating lens 102, the optical path is changed by the polarization beam splitter 103, and passes through the quarter wavelength plate 104. Thereafter, the light is deflected in the horizontal (lateral) direction and the vertical (longitudinal) direction by the horizontal deflection device 105 and the vertical deflection device 106, and then passes through the first relay lens 107 and the second relay lens 108, and the objective lens 109. Is condensed on the surface of the sample S placed on the stage 30.

  The horizontal deflection device 105 and the vertical deflection device 106 are each configured by a galvanometer mirror, and scan the surface of the sample S with the laser light by deflecting the laser light in the horizontal and vertical directions. The stage 30 is driven in the z direction (optical axis direction) by the stage elevator 20. As a result, the relative distance between the focal point of the objective lens 109 and the sample S in the optical axis direction can be changed.

  The laser beam reflected by the sample S follows the above optical path in reverse. That is, it passes through the objective lens 109, the second relay lens 108, and the first relay lens 107, and again passes through the quarter wavelength plate 104 via the horizontal deflection device 105 and the vertical deflection device 106. As a result, the laser light passes through the polarization beam splitter 103 and is collected by the imaging lens 110. The condensed laser light passes through the pinhole of the pinhole plate 111 disposed at the focal position of the imaging lens 110 and enters the photodiode 112. The photodiode 112 converts the amount of received light into an electrical signal. An electric signal corresponding to the amount of received light is input to the A / D converter 113 via an output amplifier and a gain control circuit (not shown), and converted into digital data. Here, an example in which a photodiode is used as the first light receiving element is shown, but a photomultiplier or the like may be used. Further, the laser 101 is not limited to a red laser, and a blue or ultraviolet laser may be used. By using such a short wavelength laser, high-resolution height data can be obtained.

  The height (depth) information of the sample S can be obtained by the first imaging unit configured as described above. The principle will be briefly described below. As described above, when the stage 30 is driven in the z direction (the optical axis direction) by the stepping motor 21 and the motor control circuit 22 of the stage elevator 20, the relative focus between the focal point of the objective lens 109 and the sample S in the optical axis direction. The distance changes. When the focal point of the objective lens 109 is focused on the surface of the sample S (surface to be measured), the laser light reflected by the surface of the sample S is condensed by the imaging lens 110 through the optical path described above, and almost all. All the laser beams pass through the pinholes of the pinhole plate 111. Accordingly, at this time, the amount of light received by the photodiode 112 is maximized. On the contrary, in the state where the focus of the objective lens 109 is deviated from the surface (surface to be measured) of the sample S, the laser beam condensed by the imaging lens 109 is focused at a position deviated from the pinhole plate 111. Only a part of the laser beam can pass through the pinhole. As a result, the amount of light received by the photodiode 112 is significantly reduced.

  Therefore, if the received light amount of the photodiode 112 is detected at any point on the surface of the sample S while driving the stage 30 in the z direction (optical axis direction), the height of the stage 30 when the received light amount becomes maximum. You can ask for it.

  Actually, each time the stage 30 is moved by one step, the surface of the sample S is scanned by the horizontal deflecting device 105 and the vertical deflecting device 106 to obtain the amount of light received by the photodiode 112. FIG. 4 shows changes in the received light data D with respect to the height z of the stage 30 at one point (pixel). When the stage 30 is moved in the z direction from the lower end to the upper end of the measurement range, the light reception data D that changes in accordance with the height z as shown in FIG. 4 is obtained for a plurality of points (pixels) in the scanning range. It is done. Based on the received light data D, the maximum received light amount and the focal length Zf at that time are obtained for each point (pixel). The height z of the stage 30 corresponding to the maximum value of the received light data D is the focal length Zf. Accordingly, the distribution of the surface height of the sample S on the xy plane is obtained based on the focal length Zf. This processing is performed by the control unit 51 based on the pixel arrangement information (x, y) and the height information z based on the light reception data D of the CCD 12 input through the interface 53 and stored in the memory 53.

  The obtained surface height distribution can be displayed on the display unit 52 by several methods. For example, the height distribution (surface shape) of the sample can be displayed three-dimensionally by three-dimensional display. Alternatively, it can be displayed as a two-dimensional distribution of brightness by converting the height data into luminance data. The height distribution may be displayed as a color distribution by converting the height data into color difference data.

  Also in the second embodiment, as in the first embodiment, two points on the image of the display unit 52 are designated by the pointing device 55A or the like based on the height data obtained by the first imaging unit. By doing so, the area is set in a rectangular shape, and the average height in the area and the relative height between the areas can be calculated and displayed on the display unit 52.

  Further, a surface image (black and white image) of the sample w is obtained from a luminance signal that uses the received light amount obtained for each point (pixel) in the xy scanning range as luminance data. If a luminance signal is generated using the maximum amount of light received at each pixel as luminance data, a confocal image having a very deep depth of field can be obtained at each point having a different surface height. In addition, when the height (z-direction position) at which the maximum light reception amount is obtained at an arbitrary target pixel is fixed, the light reception amount of the pixel of the target pixel portion and the portion having a large difference in height is significantly reduced. A bright image is obtained only at the same height.

  Next, the second imaging unit will be described. The second optical system 200 includes a second light source 201 for irradiating the sample S with white light (illumination light for color image shooting), a second collimating lens 202, a first half mirror 203, and a second half mirror. 204, a CCD 212 as a second light receiving element. The second optical system 200 shares the objective lens 109 of the first optical system 100, and the optical axes of both the optical systems 100 and 200 are the same.

  For example, a white lamp is used as the second light source 201, but a dedicated light source may not be provided, and natural light or room light may be used. The white light emitted from the second light source 201 passes through the second collimating lens 202, the optical path is bent by the first half mirror 203, and is collected on the surface of the sample S placed on the stage 30 by the objective lens 109. To be lighted.

  The white light reflected by the sample S passes through the objective lens 109, the first half mirror 203, and the second relay lens 108, is reflected by the second half mirror 204, and enters the CCD 212 that can receive light in color. To form an image. The CCD 212 is provided at a position that is conjugate or close to the conjugate with the pinhole of the pinhole plate 111 of the first optical system 100. A color image captured by the CCD 212 is read by the CCD control circuit 213 and converted into digital data. The color image thus obtained is displayed on the display unit 52 as an enlarged color image for observing the sample S.

  Also, by combining a confocal image with a deep depth of field obtained by the first imaging unit and a normal color image obtained by the second imaging unit, the object field in which all pixels are in focus A deep color confocal image can also be generated and displayed. For example, a color confocal image is simply generated by replacing the luminance signal constituting the color image obtained by the second imaging unit with the luminance signal of the confocal image obtained by the first optical system 100. be able to.

  Here, a magnification observation apparatus including a first imaging unit having a first optical system 100 that is a confocal optical system and a second imaging unit having a second optical system 200 that is a non-confocal optical system is shown. However, a configuration including only the first imaging unit may be employed.

  Further, like the magnification observation apparatus according to the first embodiment, the light receiving element is a two-dimensional imaging element (for example, CCD) that reads the amount of received light for each pixel arranged two-dimensionally, and the focus adjustment unit is an area. When the focal point is adjusted based on the sum of received light amounts corresponding to part or all of the sample corresponding to the region set by the setting unit, a complicated configuration like a confocal optical system is required. The height of the sample can be measured with a simple configuration. In particular, in this magnification observation apparatus, the maximum value of the light reception data is determined from the change in the light reception data with respect to the relative distance of the pixel, not the pixel unit, but the area unit set by the operator, and at that time Since the average height is calculated based on the average focal length, even when white light is used as a light source and a CCD is used as a light receiving element, variation in the variation of the received light data with respect to the focal length can be reduced. The average height can be measured. Further, when a color CCD is used as the two-dimensional imaging device, the light reception data of the pixel may be calculated based on the light reception data of RGB, or the light reception data of one or two colors of RGB may be used. The light reception data of the pixel may be used.

  In addition, if the area set by the area setting unit is larger than the sample size and includes the entire sample, the part other than the sample, that is, the upper surface of the stage, should be excluded from the calculation of the average height. Is preferred. This is because a more accurate sample height can be calculated. In this case, whether or not it is the upper surface of the stage can be determined based on whether or not the height difference between the pixel and the pixel adjacent to the pixel is a predetermined height or more. Of course, even if the region set by the region setting unit is a part of the sample, if the upper surface of the stage is included in the region, it is preferable to exclude it from the target for calculating the average height.

  In the above embodiments, the example in which the reflected light from the sample fixed to the sample fixing portion is electrically read has been shown. However, the transmitted light is electrically read by irradiating light from the back surface of the sample. You may comprise as follows. In the above description, an example in which the sample is placed on the stage is shown as an example of the sample fixing unit. However, for example, an arm may be attached instead of the stage and the sample may be fixed to the tip.

[3D image]
In addition, the magnification observation apparatus can display not only a two-dimensional image but also a three-dimensional image. By reproducing the 3D image on the display unit 52 of the magnification observation apparatus based on the three-dimensional data, the image is observed and evaluated stereoscopically from various viewpoints and angles, and the surface shape and profile such as the height are measured. be able to. In order to generate and display a three-dimensional image, for example, the relative distance in the optical axis direction, that is, the distance between the sample to be observed and the lens is changed, and a plurality of images are captured, and the movement amount at the time of image capture is recorded simultaneously. Keep it. As a result, the height information of the observation target can also be obtained when creating a composite image, and thus the acquired image can be constructed as three-dimensional data.

  A general three-dimensional data imaging method will be described with reference to FIG. First, in step S1, the imaging unit 10, the stage elevator 20 and the information processing apparatus 50 are initialized. In step S2, the moving amount and moving range of the lens are set. The movement range of the lens is a range of the lens height. For example, the user designates a rough height as the movement start position. The amount of movement of the lens is the distance of movement of the lens per one time. The finer the setting, the more detailed images can be taken, but the longer it takes to generate, so the value is set to an appropriate value according to the purpose of observation. As the specification method, the user specifies the fineness of the image, and the magnification distance is automatically set by the magnification observation device, or the user directly specifies the movement distance with a numerical value, or the default default value. There are methods such as using.

  In step S3, the lens is set at the movement start position specified above. In step S4, imaging is started. After imaging, the lens is moved by the movement amount set in step S2 in step S5, and it is determined whether or not the movement range has reached the end position in step S6. If the end position has not been reached, the process returns to step S4 and the imaging and moving steps are repeated. As a result of the movement, when the movement range set in step S2 is finished, the process proceeds to step S7, and a two-dimensional captured image is synthesized. In step S8, 3D image data is constructed based on the captured 2D data and displayed on the display unit 52. For the construction of 3D data, a profile that changes in the height direction can be calculated from the position of the lens and the image captured at that time, so multiple 2D image data captured at different heights can be combined. By doing so, a three-dimensional shape can be constructed. For example, a continuous profile is synthesized by complementing a discrete profile obtained from a captured two-dimensional image. This process can be performed at high speed in hardware.

  The three-dimensional image constructed as described above can be freely displayed by changing the viewpoint or rotating, inverting, deforming, enlarging or reducing the image. The changing operation is performed by a pointing device 55A such as a mouse 55a which is a form of the changing unit. The image is rotated in the dragged direction by operating the mouse 55a, selecting the image, and moving the mouse pointer while dragging. For enlargement / reduction, a screen enlargement / reduction function can be assigned to the scroll button 55d of the wheel mouse. Not limited to these methods, operation buttons and tools can be placed on the software screen for operation as a change unit, operation functions are assigned to specific keys on the keyboard, or dedicated hardware for operation is used as the change unit You can also.

  As such a three-dimensional image acquisition method and a method for changing the display of the acquired three-dimensional data, a known method or a method developed in the future can be used as appropriate. For example, an API such as Open GL (Open Graphics Library) can be used.

[Save two-dimensional image data]
The three-dimensional image data constructed as described above is displayed on the display unit, and the data is stored in an internal memory which is an aspect of the image data storage unit. In storing a three-dimensional image, only three-dimensional image data can be stored in a specific three-dimensional image file format, or can be stored in a two-dimensional image file format in a form embedded in an arbitrary two-dimensional image. For example, a 3D image is displayed under conditions such as a desired posture, magnification, light source, and transmission, and the displayed 3D image is acquired as a 2D image, and then the original 3D image data is added to the 2D image data. You can also embed and save. The user adjusts the viewpoint and magnification of the three-dimensional image using a mouse 55a and a tool that are one form of the changing unit, and displays the observation target in a desired posture. At this time, an image displayed on the display unit is captured and captured as a two-dimensional image.

  The data format of the two-dimensional image is preferably a general format. Examples of the image format include JPEG, JPEG2000, TIFF, BMP, PNG, GIF, WMF, EMF, and the like. Of these, a format having an area that can be freely set by the user is desirable, and JPEG and TIFF are particularly suitable. In addition, the user here refers to the person who designs the apparatus, not the user of the apparatus.

  The data file in JPEG or TIFF format has areas available to the user as shown in FIGS. FIG. 6 shows the basic structure of JPEG compressed data. The compressed data file is recorded in accordance with the JPEG Baseline DCT format defined in ISO / IEC10918-1, and an application marker segment (APP1) is inserted into the compressed data file. If necessary, a plurality of APP2 can be recorded continuously immediately after APP1. The inside of APP1 is composed of an APP1 marker, an Exif identification code, and an attached information body. The size of APP1 including all of these is 64 kbytes or less according to the JPEG standard. The attached information has a TIFF structure including a file header, and can record a maximum of two IFDs (0th IFD, 1st IFD). Among these, the 0th IFD records auxiliary information regarding the compressed two-dimensional image. A thumbnail image can be recorded in the 1st IFD. On the other hand, the inside of APP2 is composed of an APP2 marker, an FPXR (FlashPix Ready) identification code, a content list for recording extended data for FlashPix, or stream data. APP2 also has a limit of 64 kbytes or less like APP1, and when more data is recorded, a plurality of APP2 are recorded continuously. FIG. 7 is a list of attached information (field name and code) recorded in the Exif IFD. As shown in this figure, the user can add various information as tag information.

[Integration of 3D image data and 2D image data]
By embedding the original three-dimensional image data in the two-dimensional image data acquired by displaying the three-dimensional image in a desired state in this way and integrating them into one file, two-dimensional data relating to the same observation object And 3D data can be used in one file without having to be saved separately. This process can be performed by a magnified observation apparatus or a dedicated image file generation apparatus. The image file generation apparatus is an apparatus that does not have image capturing hardware such as an image capturing unit and processes acquired images. Hereinafter, a procedure for embedding a three-dimensional image in a two-dimensional image with the magnification observation apparatus will be described with reference to FIG.

  First, after the apparatus is initialized in step S1, two-dimensional images having different heights are photographed (step S2). Here, a plurality of slice images to be observed are acquired by the imaging unit while changing the relative height between the imaging unit and the observation target. In step S3, 3D processing and display are performed. Specifically, a three-dimensional stereoscopic image is constructed from a plurality of slice images and relative height information, and the created three-dimensional image is displayed on the display unit. Next, in step S4, it is determined whether or not to save the created three-dimensional image. If not saved, the process ends. If saved, the process proceeds to step S5. In step S5, the filer is displayed. The filer displays the two-dimensional image stored in the magnification observation apparatus in a list format such as a thumbnail. Then, the user selects the two-dimensional image into which the three-dimensional image data is embedded from the filer (step S6). Then, the magnification observation apparatus combines the selected 2D image data and 3D image data (step S7). Here, based on the image format of the two-dimensional image data, two-dimensional image data in a form in which the three-dimensional image data is embedded is generated.

As described above, since there is an area that can be used by the user in JPEG or TIFF format data, information relating to 3D image data is recorded in this area. Here, the flag data indicating that the three-dimensional image data is combined, the combined region address of the three-dimensional image data, and the size of the three-dimensional image data are written in the region. Further, two-dimensional image data is created by combining the three-dimensional image data with the designated area. An example of the configuration of the file generated in this way is shown in FIG. As shown in this figure, information relating to 3D image data is recorded in the header information of the 2D image data file, and 3D image data is added after the 2D image data.

  Although this file embeds three-dimensional image data, the file format conforms to a general-purpose two-dimensional image file format. Therefore, it is possible to read the image file with software that can display and display the two-dimensional image data portion. As software, for example, image processing editing software such as photo retouching software, Internet browser, viewer software, and the like can be used. If the 2D image data is processed with software that does not support 3D data, the 2D image data is rewritten into a normal 2D image data format, and the 3D image data is lost. Even if the file is opened, the data structure of the original file is maintained as long as it is not read and written, so that the three-dimensional data is also retained.

When this file is read with dedicated software that supports browsing of 3D data, 3D image data can be displayed together with 2D image data. When reading files in the dedicated display software, to check the binding flags of the three-dimensional image data recorded in the header information. As a result, three-dimensional image data is extracted, and a three-dimensional image can be displayed. Further, as described above, it is also possible to change the display viewpoint of the 3D image data and change the enlargement ratio for display. Furthermore, 2D image data and 3D image data can be displayed at the same time on the same software screen, and can be processed and edited. Furthermore, it is possible to set the three-dimensional image to a desired angle and rewrite the already recorded two-dimensional image data portion with a new two-dimensional image. In addition, it is possible to release the combination of the three-dimensional image data so that only the two-dimensional image data is present, or to embed the three-dimensional image data in new two-dimensional image data.

  In addition to the apparatus and program having the above-described image data generation and editing functions, a simple browsing apparatus or browsing program that can be browsed by omitting such functions can be provided. In other words, the software is limited so that it is possible to change the display, viewpoint, and enlargement ratio of the three-dimensional data and the two-dimensional data, but cannot generate the three-dimensional data. For example, by sending or distributing a browsing program together with 2D image data with 3D data, 3D image data can be displayed without a dedicated image file generation program. For example, if the browsing program can be downloaded from a specific home page and distributed free of charge, anyone can use the created three-dimensional image data. In addition, since 3D image data cannot be generated by browsing software, the demand for image file generation programs increases, and further expansion of use can be expected.

  As described above, the three-dimensional image data embedded in the two-dimensional image data can be compressed to reduce the data capacity. The data area of the three-dimensional image data is compressed with a known compression algorithm to reduce the data size. In particular, since the data size of 3D image data tends to increase, compression does not compress the storage area and facilitates transfer.

  Further, the three-dimensional image data can be encrypted and recorded. Encrypting data can increase security and reduce the risk of information leakage. As the encryption algorithm, a known algorithm can be used as appropriate.

  Needless to say, the two-dimensional image data in which the three-dimensional image data is embedded is not limited to an image acquired based on the three-dimensional image data, and any image can be used. For example, any one piece is selected from a plurality of two-dimensional image data used as a reference when generating the three-dimensional image data, and the selected two-dimensional image data is set as an embedding destination of the three-dimensional image data. Alternatively, it is possible to embed 3D image data in a 2D image unrelated to the 3D image data. For example, when transmitting 3D image data that requires confidentiality by e-mail or the like, it can be hidden by a third party by embedding it in an intentionally unrelated 2D image. .

  The magnification observation apparatus, image file generation apparatus, image file generation program, and computer-readable recording medium of the present invention generate a two-dimensional image including three-dimensional image data using a microscope, a digital microscope, and a digital camera. And browse.

1 is an external view of a magnification observation apparatus according to an embodiment of the present invention. It is a block diagram of the magnification observation device concerning a 1st embodiment of the present invention. It is a block diagram of the magnification observation apparatus which concerns on the 2nd Embodiment of this invention. It is a graph which shows the change of the received light data with respect to height z. It is a flowchart which shows an example of the imaging method of three-dimensional data. It is a conceptual diagram which shows the basic structure of JPEG compression data. It is a list which shows the attached information recorded on Exif IFD of FIG. It is a flowchart which shows an example of the preservation | save method of three-dimensional data. It is a conceptual diagram which shows an example of the file structure produced | generated by the image file production | generation apparatus which concerns on embodiment of this invention.

DESCRIPTION OF SYMBOLS 10 ... Imaging part 10a ... Camera 11 ... Optical system 12, 212 ... CCD
DESCRIPTION OF SYMBOLS 13, 213 ... CCD control circuit 14 ... Optical path shift part 20 ... Stage elevator 21 ... Stepping motor 22 ... Motor control circuit 30 ... Stage 41 ... Stand stand 42 ... Support column 43 ... Camera mounting unit 50 ... Information processing device 51 ... Control unit 52 ... Display unit 53 ... Memory 54 ... Interface 55 ... Operation unit 55A ... Pointing device 55a ... Mouse 55b ... Left button 55c ... Right button 55d ... Scroll button 60 ... Illumination unit 60A ... Epi-illumination 60B ... Transmitted illumination 61 ... Optical fiber 62 ... Connector 70 ... Computer 100 ... First optical system 101 ... Laser 102 ... First collimating lens 103 ... Polarization beam Splitter 104 ... 1/4 wavelength plate 105 ... Horizontal deflection device 106 ... Vertical deflection device 107 ... First relay lens 108 ... Second relay lens 109 ... Objective lens 110 · ··· Imaging lens 111 ··· Pinhole plate 112 ··· Photodiode 113 ··· A / D converter 115 · · · Laser drive circuit 200 · · · Second optical system 201 · · · White lamp 202 · · · ..Second collimating lens 203 ... first half mirror 204 ... second half mirror S ... sample

Claims (16)

An imaging unit for imaging an observation target and acquiring two-dimensional observation image data;
A plurality of two-dimensional observation image data acquired by the imaging unit and a control unit for generating three-dimensional observation image data based on height information at the time of imaging;
A display unit for displaying the three-dimensional observation image data generated by the control unit;
An output unit for embedding the three-dimensional image data displayed on the display unit in arbitrary two-dimensional image data and outputting it as one image data file;
An image data storage unit for storing the image data file output by the output unit ,
Expanding the two-dimensional image data to which the embedding three-dimensional image data by the output unit, characterized in Oh Rukoto a two-dimensional image data acquired a three-dimensional image to be displayed as desired display state on the display unit Observation device. The magnification observation device according to claim 1,
An enlarged observation apparatus, wherein information relating to 3D image data embedded in 2D image data by the output unit is recorded in an area available to a user in a 2D image data format. The magnification observation apparatus according to claim 2,
As the information related to the 3D image data embedded in the 2D image data in the output unit, flag data indicating that the 3D image data is combined, a combined area address of the 3D image data, and the 3D image data An enlargement observation apparatus, wherein information including at least one of sizes is recorded in an area available to a user in a two-dimensional image data format. The magnification observation apparatus according to any one of claims 1 to 3 ,
A magnification observation apparatus that compresses a data area of 3D image data embedded in 2D image data by the output unit. The magnification observation apparatus according to any one of claims 1 to 4 ,
A magnification observation apparatus, wherein a format of two-dimensional image data output from the output unit is either jpeg or tiff. An input unit for acquiring two-dimensional image data and height information at the time of capturing the two-dimensional image data;
A control unit for generating three-dimensional image data based on a plurality of two-dimensional observation image data and relative height information acquired by the input unit;
An output unit for embedding the three-dimensional image data generated by the control unit in arbitrary two-dimensional image data and outputting it as one image data file;
An image data storage unit for storing the image data file output by the output unit;
Equipped with a,
2-dimensional image data to which the embedding three-dimensional image data by said output unit, an image characterized by Oh Rukoto a three-dimensional image by the two-dimensional image data obtained as a desired display state displayed on the display unit File generator. The image file generation device according to claim 6 ,
An image file generating apparatus, wherein information relating to 3D image data embedded in 2D image data by the output unit is recorded in an area available to a user in a 2D image data format. The image file generation device according to claim 7 ,
As the information related to the 3D image data embedded in the 2D image data in the output unit, flag data indicating that the 3D image data is combined, a combined area address of the 3D image data, and the 3D image data An image file generation apparatus, wherein information including at least one of sizes is recorded in an area available to a user in a two-dimensional image data format. The image file generation device according to any one of claims 6 to 8 ,
An image file generation apparatus, wherein the output unit compresses a data area of 3D image data embedded in 2D image data. The image file generation device according to any one of claims 6 to 9 ,
The image file generating apparatus, wherein the format of the two-dimensional image data output from the output unit is either jpeg or tiff. A function of acquiring two-dimensional image data and relative height information at the time of imaging the two-dimensional image data;
A function of generating three-dimensional image data based on a plurality of acquired two-dimensional observation image data and relative height information;
A function of embedding the generated 3D image data in arbitrary 2D image data and outputting it as one image data file;
The computer has a function to save the output image data file ,
The 2-dimensional image data to which the embedding three-dimensional image data, the image file creation program characterized Oh Rukoto a two-dimensional image data acquired three-dimensional image data as a desired display state. An image file generation program according to claim 11 ,
An image file generation program for recording information on 3D image data embedded in 2D image data in an area available to a user in a 2D image data format. An image file generation program according to claim 12 ,
As information about the three-dimensional image data, information including at least one of flag data indicating that the three-dimensional image data is combined, a combined area address of the three-dimensional image data, and a size of the three-dimensional image data is two-dimensional. An image file generation program for recording in an area available to a user in an image data format. An image file generation program according to any one of claims 11 to 13 ,
An image file generation program for compressing a data area of 3D image data embedded in 2D image data. The image file generation program according to any one of claims 11 to 14,
An image file generation program characterized in that the format of the two-dimensional image data is either jpeg or tiff. A computer-readable recording medium storing the image file generation program according to any one of claims 11 to 15 .

Images