JP4434376B2 - Microscope image transfer system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a microscope image transfer system that transmits an observation image obtained by a microscope as a still image, and is particularly used in a telepathology system that remotely observes a pathological specimen.
[0002]
[Prior art]
Conventionally, a microscope still image transmission system is used for remote pathological diagnosis using a microscope TV image of a pathologist, for example. Then, when observing a tissue specimen in pathological diagnosis or biology with a microscope still image observation system, first, what size, shape, and color observation object is placed on which position on the slide glass It is said that grasping is important for efficient observation without oversight.
[0003]
In order to grasp the whole specimen image on the slide glass in this way, it is common to observe it with the naked eye or a magnifier before entering the microscope. Or use a visual image capturing means.
[0004]
Japanese Patent Laid-Open No. 6-3597 proposes that a pathologist (observer) 's observation technique under a microscope is incorporated in a TV observation system, and a macroscopic image photographing means for photographing a whole specimen and pointing for designating an image area. A microscope still image observation system has been devised in which an entire image of a specimen photographed by a macroscopic image photographing means is blocked by a pointing means, an electric stage of a microscope is set, and an electric stage of the microscope is sequentially photographed.
[0005]
In Japanese Patent Laid-Open No. 6-222281, a plurality of frames (blocks) corresponding to a microscope low magnification field of view are designated in the entire specimen image, and control is performed so as to capture a microscope image at a location (stage position) designated by the frame (block). . There has been a proposal that can specify the image capture designated position on the remote observer side (pathologist side). The observer side (pathologist side) receives the entire specimen image (macro image) on the slide glass as a still image from the client side terminal, and then divides this macro image equally to indicate the enlargement designation frame. Has also been proposed. Here, instructing an enlargement designation frame by equally dividing the macro image is referred to as mesh division designation. It has also been proposed to use a microscope having an autofocus function in order to eliminate the manual work of the client. In the present proposal, a single frame indicating an enlargement designation frame is represented as a “spot”, and a macro image divided equally to designate a plurality of enlargement designation frames as a “mesh”.
[0006]
That is, in a conventional microscope still image transmission system, a sample image on a microscope stage is taken into a TV camera, a computer image is taken with a capture board, the image is digitized, and a remote computer is transmitted through a public line such as ISDN. A microscope image transfer system that transfers data to and displays an image is known. You can also operate the microscope from a remote computer to change the magnification and move the stage.
[0007]
[Problems to be solved by the invention]
The point of attention in the present invention is that when a desired microscope image is obtained by specifying a plurality of enlarged designation frames corresponding to a microscope low magnification field of view from a whole specimen image (macro image) of a slide glass. It is in.
[0008]
In the above-described conventional techniques, the whole specimen image (macro image) on the slide glass is photographed by means for photographing a macroscopic image, and a plurality of rectangular frames (frames (blocks)) are set at desired positions by the pointing means ( In order to eliminate the manual operation of the client by controlling the microscopic image of the location designated by the rectangular frame (frame) and specifying the mesh division, it is described. However, the time until the entire mesh designated image is captured and the correctness of the focus position with respect to the captured image are not touched.
[0009]
In remote pathological diagnosis, especially during surgery, an image of a level that can be diagnosed by the observer (pathologist) is transferred from the client side (facility without pathologist) to the observer side (pathologist side) in the shortest time. There must be. Accordingly, there is a problem that it is too time-consuming to manually obtain the in-focus positions for a plurality of mesh images in the intraoperative diagnosis. In addition, it is conceivable to perform autofocus on any one of a plurality of mesh images, and image all the remaining images using the first obtained focus position. It was difficult to acquire an image with the correct in-focus position due to differences in Z stage displacement, specimen thickness, and the like.
[0010]
The present invention has been made in view of the above circumstances, and its purpose is that a remote observer side (pathologist side) performs mesh division designation from the entire specimen image (macro image) of the slide glass. A microscope that can acquire the correct image of the in-focus position of all mesh-designated images in a short time when the client side (one without a pathologist) is instructed to take a microscope image of the mesh-designated frame position. An image transfer system is provided.
[0011]
[Means for Solving the Problems]
2. A microscope image transfer system capable of designating a magnified image to be captured at a desired magnification on a still image according to claim 1, wherein a luminance information storage means for storing luminance information of an enlarged designated frame region on the still image, and a microscope XY Z-direction position correcting means for correcting the position in the Z-direction according to the displacement of the stage, auto-focus execution enable / disable determining means for determining whether or not auto-focus can be executed before taking an enlarged image on a still image, and the auto When the focus execution enable / disable determining means determines that autofocus can be executed, the XYZ position storage means for storing the XY position of the microscope XY stage and the Z position of the microscope XY stage, and the autofocus execution enable / disable determining means determines that autofocus cannot be executed. When determined, the Z position of the previous microscope XY stage stored in the XYZ position storage means And having a Z position restoration means for restoring the.
[0012]
Further, in the microscope image transfer system capable of designating a magnified image to be captured at a desired magnification on the still image according to claim 2, means for moving the microscope XY stage to XY to a position where autofocus is possible, and the microscope XY Means for controlling execution of autofocus after XY movement of the stage, means for restoring the position of the XY stage to the position before movement after execution of autofocus, and image after restoring the position of the previous microscope XY stage to the position before movement And means for controlling capturing.
[0013]
Further, in the microscope image transfer system capable of specifying an enlarged image capture at a desired microscope magnification on a still image according to claim 3, the location where autofocus is first executed automatically from each of a plurality of image capture specification frames is automatically set. It has the means to select to, It is characterized by the above-mentioned.
[0014]
According to the microscope image transfer system of the present invention, the microscope XY stage is controlled in the XYZ direction by the Z-direction position correcting means accompanying the displacement of the microscope XY stage, and the position of the microscope XY stage and the Z position of the microscope XY stage are set to XYZ. Stored by the position storage means. Next, the means for restoring the Z position of the microscope XY stage calls the previous position information and moves the microscope XY stage. Luminance information storage means for storing the luminance information of the enlarged designated frame region on the still image stores the luminance information of the enlarged designated frame region on the still image. Before taking a magnified image on a still image, it is determined whether or not autofocus can be executed by the autofocus execution enable / disable determining unit. When it is determined that autofocus can be executed, autofocus is executed at an optimum position. Further, when the autofocus execution enable / disable determining means determines that autofocus cannot be executed, the previous Z position of the microscope XY stage stored in the XYZ position storage means is restored by the Z position storage means, and error recovery is automatically performed. Reduce the probability of autofocus errors.
[0015]
Furthermore, after the microscope XY stage is moved to the position where the specimen exists, the microscope XY stage is moved XY by means of restoring the position of the microscope XY stage to the position before the movement after executing autofocus even if there is no specimen on the optical axis. Thus, an image at the optimum Z position can be obtained by means for controlling execution of autofocus, so that an image at the optimum Z position can be obtained for any mesh enlargement designated image.
[0016]
The operation on the observer side (pathologist) can be simplified by means for automatically selecting a place where autofocus is first executed from a plurality of enlarged image capture designation frames (mesh designation frames).
[0017]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0018]
FIG. 1 is a schematic view of a microscope image transfer system according to the present invention. The microscope 907 includes an electric stage 909 and an electric revolver 908, and further includes a video camera 906. Further, the microscope 907 can be provided with an auto focus unit (not shown), a light control function, an electric diaphragm function, and the like. In the example shown in FIG. 1, a microscope operating unit 913 is connected to a requesting terminal 905 realized by a personal computer or the like, and operation data is transferred from the requesting terminal 905 to the microscope operating unit 913, so that an electric part of the microscope can be used. It enables certain objective control, auto focus (AF) control, fine movement in the Z direction, dimming control, motorized aperture function, and the like. It may be a manually operated microscope that does not have an electric function. The requesting terminal 905 has a video capture function (not shown), and also includes a terminal connected to the image output of the video camera 906. The observation-side terminal 901 composed of the request-side terminal 905 and a personal computer is provided with a storage medium for storing information such as images. A recording device such as an MO may be provided as a peripheral device of the request side terminal 905 or the observation side terminal 901. A monitor 904 is connected to the requesting terminal 905, and a microscope image and a macro photographing device image can be observed on the monitor.
[0019]
Furthermore, line connection devices 910a and 910b for transmitting information such as image information through public lines are provided, and each of the line connection devices 910a and 910b has an interface circuit with the observation side terminal 901 and the request side terminal 905, respectively. The request side terminal 905 and the observation side terminal 901 are connected to the line connection devices 910a and 910b via a public line 903 such as ISDN. A local area network (LAN) may be used instead of the public line.
[0020]
In a telepathology system that performs pathological diagnosis remotely, a terminal of a facility without a pathologist (observer) is set as a request-side terminal 905, and a terminal of a facility with a pathologist (observer) is set as an observation-side terminal 901. The client is usually provided with a macro imaging device 911 that captures the entire specimen image on the slide glass and a microscope 907 that performs magnified observation from the entire specimen image. Normally, a macro photographing device, a microscope, and the like are not necessary on the observer side, but they may be connected to the observation side terminal 901.
[0021]
FIG. 2 is a flowchart showing an operation sequence in the telepathology system. First, a client side with a microscope takes a whole image (hereinafter referred to as a macro image) of a specimen on a slide glass that the observer side wants to inspect using the macro imaging device 911 (S1001). At that time, the observer side starts up the observation side terminal 901 so that the macro image transmitted from the client side can be received (S1009). If a macro image can be observed with a low-magnification objective of a microscope, a slide glass may be placed under the microscope and an image may be taken with an optimum microscope objective magnification without using a macro imaging device. It is also possible to create a macro image by setting the microscope objective to the minimum magnification under the microscope, moving the stage in consideration of the field of view, and sequentially taking images and pasting these images together.
[0022]
Macro images captured by the video camera 914 of the macro photographing device 911 are sequentially stored in a frame memory via a video capture board (not shown) in the requesting terminal 905 and displayed on the monitor 904 of the requesting terminal 905. . The trigger for capturing the macro image is performed by an arbitrary switch (SW), and the SW button for operation provided on the application software displayed on the monitor 904 of the requesting terminal 905 is recognized by an event such as a mouse click, This is done by selecting with the SW of the external operation panel (not shown). When the request side terminal 905 captures the macro image, the request side terminal 905 issues a line connection request to the observation side terminal 901 (S1002). This line connection request transfers digital data to the observation side terminal 901 via the public line 903 such as ISDN via the line connection devices 910a and 910b. Upon receiving the line connection request, the observation side terminal 901 performs connection request processing (S1010). This connection process recognizes the connection partner, and if there is no problem, returns connection permission to the connection partner. When the line connection process is established, the requesting terminal 905 transmits a macro image and initial setting data to the observation terminal 901 (S1003). The initial setting file includes hardware information (microscope, macro device, TV camera type, etc.) connected to the requesting terminal 905. After the observation side terminal 901 receives these data (S1011), the operation right is transferred from the request side terminal 905 to the observation side terminal 901 (S1004, S1012). The operation right indicates a right to perform operations such as control of a microscope and a stage and designation of image capture. This operation right may be changed when the line connection is automatically established, or the client clicks the operation button on the application software displayed on the terminal monitor at any time. It may be switched by, for example. In the example of FIG. 1, the operation right is passed to the observation side terminal 901 after the request side terminal 905 has taken a macro image.
[0023]
An observer (pathologist) observes with a low-magnification objective of a microscope in order to find a position of interest from an image sent as a macro image (S1013, S1014). Usually, in order to observe the entire region of the macro image, the mesh capture designation is selected as the enlargement designation frame method (S1015). The mesh capture designation is to divide a sample on a lattice and designate an enlargement position as shown in FIG. In the example of FIG. 3, M0 to M8 are designations for capturing meshes, and the captured images are specified so as to overlap in consideration of stage movement accuracy. The magnification designation and mesh capture designation may be assigned to any keyboard (not shown) of the observation side terminal 901.
[0024]
The mesh capture designation is performed by confirming a still image on the monitor 902 of the observation side terminal 901 and determining the mesh designation area by specifying the mesh capture start point and end point by operating the mouse, and enlarging the area within this area. The frame is automatically positioned and displayed as an overlay on the still image. Alternatively, the position where the sample is present may be automatically recognized, and the mesh capture designation frame may be efficiently overlaid on the still image. Here, when a mesh is selected in the enlargement designation frame method selection, mesh capture designation processing is performed (S1017). When the enlargement designation frame information is received by the requesting terminal 905, the requesting terminal 905 performs the processing based on this information. The mesh capture designation frame is displayed as an overlay on the still image (macro image) displayed on the monitor 904 (S1018, S1005). The same enlargement designation frame can be displayed on the monitor 902 of the observation side terminal 901 and the monitor 904 of the request side terminal 905, and the same screen can be shared.
[0025]
When no mesh is selected in the enlargement designation frame method selection in (S1015), spot shooting designation as shown in (S1016) is performed (see S0 in FIG. 3). When spot photographing designation processing is performed on the monitor 902 of the observation side terminal 901 (S1017), this enlarged designation information is sent from the observation side terminal 901 to the request side terminal 905 via the line connection device 910a, ISDN 903, and line connection device 910b. Transmit (S1018). Upon receiving this enlargement designation frame information, the request side terminal 905 displays an overlay of the spot shooting designation frame on the still image of the monitor 904 of the request side terminal 905 based on this information (S1005). In the spot designation, an enlargement is designated at one place around an arbitrary position. In the spot processing, the enlarged position is designated by operating the mouse while confirming the still image on the monitor 902 of the observation terminal 901 as in the mesh processing. The spot designation and mesh designation enlargement designation frame can be moved as shown in FIG. FIG. 4 shows an example in which the enlargement designation frame frame once set is moved from (S0a) to (S0b). This enlargement frame designation position information is recorded on the recording media of the request side terminal 905 and the observation side terminal 901.
[0026]
The enlargement designation frame can be added (S1019). If it is to be added, the flow is returned, the processing from (S1013) to (S1015) is repeated, the position is determined, and the enlargement designation rate is also designated. In addition, the enlargement designation frame method is selected.
[0027]
If the observer (pathologist) determines the position and magnification to be enlarged next and there is no need to add an enlargement designation frame (S1019), an image capture request is sent from the observation side terminal 901 to the request side terminal 905. Is transmitted (S1020).
[0028]
Upon receiving the image capture request, the requesting terminal 905 performs stage movement, objective change, AF execution, and the like as microscope operations (S1006). In the stage movement, the position coordinate on the monitor 904 of the request side terminal 905 designated by the mesh or spot is converted into the stage coordinate position, and the control data is transferred from the observation side terminal 901 to the XY stage control unit 912 via the request side terminal 905. The electric stage 909 is moved by transferring. Similarly, the objective change and AF execution are performed by sending control data from the observation side terminal 901 to the microscope operation unit 913 via the request side terminal 905.
[0029]
When the movement of the electric stage 909 and the magnification of the objective lens are changed by controlling the electric revolver 908, the sample image information is captured by the video camera 906, and this image information is input to the video capture board in the requesting terminal 905, Make an image a still image. The still image may be recorded on the recording medium as it is, but the image is compressed in a format such as JPEG (Joint Photographic Coding Experts Group) in order to transfer the image information. The compressed image information is recorded on a recording medium. The image information recorded on the recording medium is transferred from the request side terminal 905 to the line connection device 910b, and is transferred to the observation side terminal 901 via the public line 903 and the line connection device 910a (S1007).
[0030]
When the observation terminal 901 receives the image information transferred from the request side terminal 905 (S1021), the observation side terminal 901 displays the received image information on the monitor 902 of the observation side terminal 901. When the image information is compressed in a format such as JPEG, the image information is expanded and displayed.
[0031]
When the request-side terminal 905 finishes transferring one image to the observation-side terminal 901, the request-side terminal 905 determines whether the next image capture designated position remains (S1008). If it remains, the requesting terminal 905 performs the microscope operation again (S1006), captures an image, and transmits the image data to the observation terminal 901. This operation is performed until all the image designation positions are eliminated. When the observation-side terminal 901 has received all the images, the observer (pathologist) performs a remote observation while observing the image displayed on the monitor 902 of the observation-side terminal 901 (S1022). At this time, image linkage, mouse position information, and the like are transmitted from the observation side terminal 901 having the operation right to the request side terminal 905 via the line connection device 910a, the public line 903, and the line connection device 910b. The requesting terminal 905 that has received the image linkage information and the mouse position information displays an image on the monitor 904 of the requesting terminal 905 based on the information, and also coordinates information such as the mouse position (S1024). When the diagnosis is completed and the line is disconnected from the observation terminal 901, a line disconnect request is transmitted to the request terminal 905 (S1025). Upon receiving the line disconnection request, the requesting terminal 905 performs line disconnection processing (S1026). The observation side terminal 901 determines the end after the remote observation (S1023). Further, when it is desired to continue the diagnosis by enlarging the image at a high magnification, the logic returns to the sequence of S1013 and is repeated again from the magnification designation (S1014).
[0032]
Normally, since the remote diagnosis is performed from the observation side terminal 901, the operation right is on the observer side, but this operation right can be arbitrarily switched to the request side terminal 905.
[0033]
FIG. 5 is a flowchart of an example in which the operation right is switched from the observation side terminal 901 to the request side terminal 905 to perform the enlargement designation and image capture.
[0034]
That is, in FIG. 5, the observation side terminal 901 transmits an operation right change request to the request side terminal 905 (S1101). The requesting terminal 905 receives this operation right change request and obtains the operation right (S1106). Hereinafter, steps S1107 to S1113 are the same as S1013 to S1019 of FIG. Further, steps S1114 to S1119 are the same as S1006 to S1026 in FIG.
[0035]
On the other hand, step S1102 on the observation side is the same as step S1005 on the requester side in FIG. In FIG. 5, steps S1103 to S1105 on the observation side are the same as S1021 to S1025 in FIG.
[0036]
The above is the schematic sequence operation of the telepathology system. The present invention relates to the microscope operation (S1006) and image capture (S1007) in the mesh designation of S1017 in this sequence operation, and aims to capture the mesh designated image at the optimum focus position.
[0037]
The operation of the present invention will be described below with reference to FIGS.
[0038]
FIG. 6 shows a diagram when mesh designation processing (S1017) is performed in a still image state after macro shooting. The mesh designation is usually used when the entire sample region or the sample partial region is divided into a lattice shape within a range that can be observed with a low-magnification objective lens of a microscope after capturing a macro image. When observing the entire specimen region throughout, if AF is performed on all mesh-designated positions, it takes time to capture all images. If one AF location is provided and image data at all mesh designated positions is acquired at the Z position, the time from the image capture designation to the end of image capture is shortened. However, when there is one AF location, the image is not always in focus at all mesh designation positions. The cause may be a stage surface accuracy problem or an error due to the thickness of the specimen. Here, it is assumed that the slide glass and the microscope stage are mounted in parallel and are fixed accurately. In order to solve the problem that the stage surface accuracy is not in focus, the following is performed. The stage is moved to XY in accordance with a change in the mesh capture designated position, and fine movement in the Z direction is performed in accordance with the amount of XY movement.
[0039]
In order to perform fine movement in the Z direction, Z variation data associated with stage movement is previously stored for each objective lens of the microscope. As shown in FIG. 7, an adjustment component is prepared by drawing lines in a lattice pattern on the surface of the slide glass. The interval between the gratings may be finely and arbitrarily set, but is set according to the magnification objective field of the microscope to be adjusted. First, the microscope objective lens is changed to a magnification for acquiring data. Next, the slide glass of FIG. 7 is placed on the electric stage 909 of the microscope 906, and the stage is moved so that X = 0 and Y = 0 on the slide coordinates. When X = 0 and Y = 0, it becomes the edge of the glass, so it is assumed that a line is drawn on the inside of the slide glass by about several mm. Fine movement in the Z direction is performed by manually operating the focusing unit of the microscope or controlling the microscope operation unit 913 from the requesting terminal 905 so that the focus is achieved at this position. The focus position is confirmed based on an image displayed as a moving image on the monitor 904 of the requesting terminal 905 input via the video camera 906 via a capture board (not shown) in the requesting terminal 905. When the focus position can be adjusted, the Z position information of the microscope is taken into the requesting terminal 905 via the microscope operation unit 913. The microscope Z position at the slide coordinate position of X = 0 and Y = 0 is stored as an initial value (Z0) in a memory (not shown) in the requesting terminal 905.
[0040]
Next, the stage is moved to the stage coordinates of X = 0, Y = 1, and the line on the slide glass is focused.
[0041]
After confirming the focus position, the microscope Z position information is acquired, compared with the Z initial value Z0, and the comparison result data is stored in a memory (not shown) in the requesting terminal 905. In this way, while moving the coordinates on the slide glass, the focus is confirmed, and the Z position at the focused position is compared with the Z initial value data (Z0) to obtain comparison data ΔZ for each coordinate. The data is stored in the memory in the requesting terminal 905. The result is the table shown in FIG. This data holds data for each objective.
[0042]
As described above, Z deviation data with respect to XY fluctuation is stored in the memory of the requesting terminal 905 with the slide glass placed on the microscope stage. Based on this correction data, Z correction accompanying movement of the XY stage is performed, and this will be described with reference to the flowchart of FIG.
[0043]
First, the entire specimen image (macro image) on the slide glass is photographed and displayed on the monitor (S501). The macro image is usually taken by operating the requesting terminal 905 and using the macro photographing device 911 or the microscope 907. . The photographed macro image is stored in the memories of the request side terminal 905 and the observation side terminal 901 as luminance information for each of RGB in the still image state (S502). The memory is stored in a two-dimensional array (for example, in the case of a VGA size image, stored in R (iX, iY) (iX = 0 to 639, iY = 0 to 479), so that data can be easily retrieved later. The mesh capture designation area is designated, and the mesh capture designation is performed by the observation side terminal 901 after connecting to the normal line, of course, the request side terminal 905 can also designate mesh capture even if the line is not connected. In the line connection state, the requesting terminal 905 can also specify mesh shooting, and this area specification is displayed on the still image while being confirmed on each monitor 902, 904 (S503). The mesh position displayed as an overlay on the macro still image has its coordinate information stored in the memory of the requesting terminal 905 and the observation terminal 901. (S504) This coordinate information only needs to store the center of each mesh frame, and requests not only the center coordinates but also the magnification information of the still image and the magnification information of the mesh enlargement designation for each mesh enlargement designation frame. If it is stored in the memory of the side terminal 905 and the observation side terminal 901, it can be reproduced later.
[0044]
Next, the position of the mesh enlargement designation frame for executing AF first is designated on the monitor (S505). The mesh frame on the macro still image displayed on the monitor 902 or 904 of the terminal (request side terminal 905 or observation side terminal 901) having the operation right is also designated for the position of the mesh enlargement designation frame for executing this AF. By clicking the mouse. Of course, this mouse click information is shared by both the request-side terminal 905 and the observation-side terminal 901. Assume that the line thickness or line color of the enlargement designation frame is changed from other mesh enlargement designation frames so that it can be seen which of the mesh enlargement designation frames on the macro image has been selected. When the position for designating AF is first determined, image capturing is started. This image capture is started by pressing an operation button on an application (not shown) or an arbitrary key on the keyboard of the terminal from a terminal having an operation right.
[0045]
When the requesting terminal 905 recognizes an instruction to start image capture, the requesting terminal 905 first moves the stage to a mesh position designated to execute AF (S506). In this stage movement, the requesting terminal 905 sends a movement command to the XY stage control unit 912 to move the electric stage 909 of the microscope 907 to a designated position. The stage movement amount is calculated from the mesh expansion designation frame position on the monitor by actually converting data with the stage coordinate system.
[0046]
After moving to the stage position where the AF position is executed first, AF is executed (S507). The AF execution is realized by the AF unit (not shown) of the microscope 907 operating when the requesting terminal 905 sends AF execution data to the microscope operation unit 913. Here, it is sufficient that the AF operates normally after the AF is performed, but an AF error may occur when the specimen does not exist near the microscope objective optical axis (near the center of the screen). Therefore, it is checked whether or not an AF error occurs (S508). This AF error is output from an AF unit (not shown). When an AF error occurs, the microscope operation unit 913 sends an AF error notification to the requesting terminal 905. When the requesting terminal 905 recognizes this error, an error display indicating that an AF error has occurred is displayed on the monitor 904 of the requesting terminal 905. The error display may be displayed as an error window. Also, the client who operates the requesting terminal 905 is instructed to move the Z stage fine movement handle of the microscope to the in-focus position by manual operation. This determines the first focus position. The Z position is stored in the memory of the requesting terminal 905, and the XY stage position coordinates storing the Z position are also stored. In step S508, if the AF operates normally, the Z position and XY stage position coordinates are stored without being manipulated (S510).
[0047]
When the movement of the stage XYZ is finished, a still image is captured (captured) at this position (S511). The image capturing is performed by capturing an image of the microscope 907 through the video camera 906 by controlling the capture board mounted on the requesting terminal 905 after the requesting terminal 905 performs the XY stage and AF confirmation. . After taking the image, the image data is stored in a recording medium (not shown) such as an HD of the requesting terminal 905 in an arbitrary image file format. The captured image is transferred from the request-side terminal 905 to the observation-side terminal 901 through the line connection devices 910b and 910a and the public line 903, and the same image is displayed on the monitor 902.
[0048]
After first capturing a mesh enlargement designation image at a location where AF is executed, it is checked whether or not a next mesh enlargement designation frame exists (S512). If there is no next mesh enlargement designation frame, the process ends. If the next mesh enlargement designation frame exists, the XY stage is moved to the next location (S513).
[0049]
The XY stage is moved, and the Z position is finely moved in accordance with the movement of XY (S514). This fine movement of the Z position is performed based on the stage coordinates and ΔZ data in the table of FIG. That is, the coordinate position in the table of FIG. 8 corresponds to the previous stage position, and ΔZ at that time is expressed as ΔZ _i-1 Similarly, the current stage position is the coordinate position and ΔZ (ΔZ in the table of FIG. _i ). Previous ΔZ _i-1 And ΔZ _i Difference ((ΔZi−1) − (ΔZ _i )) Is the Z fine movement amount accompanying the movement of the XY stage. In Z fine movement control, the microscope Z motor is driven by sending Z fine movement amount data from the requesting terminal 905 to the microscope operation unit 913, and the Z position changes.
[0050]
When the requesting terminal 905 confirms that the XY movement and the Z fine movement control by the electric stage 909 have been completed, the requesting terminal 905 determines whether or not to perform the AF operation (S515). In order to absorb not only the Z shift of the stage but also the Z error due to the thickness of the specimen, it is determined whether or not to arbitrarily AF. Details of this determination processing will be described later with reference to the flowchart shown in FIG. If it is determined that AF should be executed (S516), AF is executed (S517). If it is not necessary to execute AF (S516), the process returns to the image capturing process (S511). If there is no error after executing AF (S517) (S518), the process returns to the process of storing the Z position and XY coordinate values (S510). If an AF error has occurred after executing AF (S518), the error is notified from the microscope operation unit 913 to the requesting terminal 905. When this error is recognized by the requesting terminal 905, the Z position of the position where the previous image was taken is called from the memory (not shown) of the requesting terminal 905, and this Z position data is transferred to the microscope operation unit 913. Thus, the best Z position is automatically restored. At this time, the requesting terminal 905 does not request an error display and manual Z fine movement operation. After automatically moving to the previous Z position (S519), the process returns to the image capturing process (S511).
[0051]
Next, a flowchart for determining whether or not to perform AF will be described with reference to FIG.
[0052]
First, if there is a movement difference between the previous stage position and the current stage position, the movement difference reference data (XYrefth) for determining whether to perform AF is set as a threshold (S601). This movement difference reference data (XYrefth) is arbitrarily determined by the initial setting. Further, the movement difference reference data (XYrefth) is obtained by calculating the depth of focus from the numerical aperture (NA) of the objective lens, and using the XY movement amount at the time of Z displacement accompanying XY movement that cannot cover this depth of focus as preset data. You may have it. The movement difference reference data (XYrefth) is stored in a memory in the requesting terminal 905. Next, the difference between the stage XY coordinate position of the current mesh enlargement designation frame and the stage XY coordinate position where the previous AF was executed is taken, and the difference is taken as XYref (S602). The stage XY coordinates at which the previous AF was executed are stored in the memory of the requesting terminal 905. Comparison between XYref and XYrefth is performed (S603). If the difference XYref with respect to the stage XY coordinate position where AF was executed last time is smaller than the movement difference reference data (XYrefth), AF need not be executed (S619). On the other hand, if the difference XYref from the stage coordinate position previously executed is larger than the movement difference reference data (XYrefth), the process proceeds to a process for confirming whether or not AF can be executed. To check whether AF can be executed, it is determined whether or not a sample exists near the center of the image (near the center of the optical axis). This is because AF may not operate normally when there is no specimen near the center of the optical axis. The flowcharts after step S605 represent processing for determining whether or not a sample exists near the center of the optical axis at the coordinate position to be expanded next in the still image state. Even in the moving image state, it can be determined that there is a specimen near the center of the optical axis, but in the present invention, before capturing a magnified image in a state where a mesh expansion designation frame of a still image before obtaining the magnified image is determined, Predict whether a sample exists near the axis.
[0053]
First, as a reference for determining whether or not a sample exists, RGB luminance information of a still image is used. The luminance information of each RGB in the still image state is already stored in the memory of the requesting terminal 905 in step S502. In step S604, a luminance threshold (Gth) recognized as a sample is determined, and in step S605, the sample image recognition check counter is cleared. The luminance threshold for sample recognition is stored in an arbitrary file as initial data, and when used, data is extracted from this file and stored in the memory of the requesting terminal 905. Next, since it cannot be judged with only one pixel to recognize it as a sample image, it is necessary to determine the range of pixel data near the center and compare it with the luminance threshold (Gth). In steps S606 and S607, the range of XY coordinate values to be checked in the still image state is determined. First, the range for checking the X coordinate is determined. FIG. 10 shows a state in which a mesh enlargement designation frame is displayed as an overlay on the macro image still image after taking the entire specimen image (macro image) on the slide glass. ImageWidth and ImageHeight in FIG. 10 indicate the width and height of the still image. In the VGA size, the width is 640 × 480 pixels, and (ImgXmax, ImgYmax) = (639,479). Frames (1) to (18) are mesh enlargement designation frames, and their width (FrameWidth) and height (FrameHeight) are determined by the current display magnification and the next enlargement designation magnification.
[0054]
FrameWidth = ImageWidth x (current display magnification / next specified magnification)
FrameHeight = ImageHeight x (current display magnification / next specified magnification)
FIG. 12 is an enlarged view of the frame (10) in the mesh enlargement designation frame in FIG. (Xmin, Ymin) to (Xmax, Ymax) in FIG. 12 are regions for evaluating whether a sample is present near the center even when AF is executed. (Xic, Yic) in FIG. 12 represents the center coordinates of the mesh expansion designation frame (10) when the upper left of the image in FIG. 10 is (0, 0).
[0055]
FIG. 13 is a flowchart for determining the X coordinate area, and FIG. 14 is a flowchart for determining the Y coordinate area.
[0056]
First, a flowchart for determining the X coordinate area will be described. First, the center coordinates (Xic, Yic) of the mesh enlargement designation frame viewed from the entire image of FIG. 10 are detected (S701). The center coordinates of each mesh enlargement designation frame are extracted from the memory (not shown) of the requesting terminal 905. In order to check whether or not a sample exists from the center coordinates, the X region is designated based on the following equation (S702).
[0057]
Xmin = Xic-FrameWidth / n
Xmax = Xic + FrameWidth / n
The value of n can be arbitrarily changed in the initial setting.
[0058]
Similarly, in the Y region, the center coordinates (Xic, Yic) of the mesh enlargement designation frame viewed from the entire image of FIG. 10 are detected (S801), and the following formula is used to check whether or not a sample exists: The Y area is designated based on (S802).
[0059]
Ymin = Yic-FrameHeight / m
Ymax = Yic + FrameHeight / m
The value of m can be arbitrarily changed in the initial setting.
[0060]
As described above, an area for checking whether or not a sample exists is determined. Whether or not a sample exists is determined by comparing the luminance information of each pixel with the luminance threshold (Gth) determined in step S604. Here, there are three types of luminance information for each pixel, but in the example of the flowchart of FIG. 9, only G information is used. Of course, it is also possible to have the luminance threshold values Rth, Gth, and Bth using the three luminance information of RGB and compare them to make a comprehensive judgment.
[0061]
The Gdata (X, Y) luminance information of each pixel in the region for determining whether or not a sample exists is compared with the luminance threshold (Gth) determined in step S604, and finally AF is executed. A flowchart of the process for determining whether or not the process is acceptable is shown after step S608.
[0062]
First, initial coordinates are set (S608). Here, X and Y are variables, and are stored in a memory (not shown) of the client side terminal 905. The initial values are the initial coordinates of the check coordinate area described above, and X = Xmin and Y = Ymin.
[0063]
The G luminance information (Gdata (X, Y)) at the X and Y coordinates is stored in a memory (not shown) of the requesting terminal 905. The initial values are the initial coordinates of the check coordinate area described above, and X = Xmin and Y = Ymin.
[0064]
The G luminance information (Gdata (X, Y)) at the X and Y coordinates is extracted from the memory (not shown) of the requesting terminal 905 (S609). The luminance information (Gdata (X, Y)) is compared with the luminance threshold (Gth) (S610).
[0065]
If the value of the G luminance information (Gdata (X, Y)) is smaller than the value of the luminance threshold Gth (S611), it is determined that a sample image exists, and the sample image recognition check counter cleared in step S605 is incremented ( S612), and stores it in the memory (not shown) of the requesting terminal 905. If the value of the G luminance information (Gdata (X, Y)) is larger than the value of the luminance threshold (Gth) (S611), it is determined that there is no sample image, and the sample image recognition check counter (ChkCounter) changes. do not do. Next, the X coordinate is incremented by one pixel (S613). At this time, it is determined whether or not the X coordinate is outside the region determined in S606 (S614). If the X coordinate is within the region, the process returns to the process of comparing the luminance information and the luminance threshold value in S609. When the X coordinate is out of the region (S614), the X coordinate is set to an initial value (X = Xmin), and the Y coordinate is incremented by one pixel (S615). After incrementing the Y coordinate, it is determined whether or not the Y coordinate is outside the region (S616). If the Y coordinate does not fall outside the region, the luminance information in S609 is again compared with the luminance threshold value to determine whether a sample image exists. When the Y coordinate is incremented and the Y coordinate is out of the area, the check of all areas for determining whether or not a sample image exists is completed. Next, the value of the comparison data (ChkCounterMin) is set based on the following formula (S617).
[0066]
ChkCounterMin = ((Xmax-Xmin) × (Ymax-Ymin)) / 2
In the above formula, it is divided by 2 to make it half of the area for determining whether or not a sample exists, but it may be an arbitrary value.
[0067]
Next, it is determined whether or not a sample image exists near the final center (S618). If the sample image recognition counter (ChkCounter) is equal to or greater than the comparison data (ChkCounterMin), it can be determined that the sample image exists in that region. If it is equal to or greater than the comparison data (ChkCounterMin), it can be determined that a sample image exists in that region.
[0068]
The sample image recognition counter (ChkCounter) is compared with the comparison data (ChkCounterMin). If the sample image recognition counter is larger, it is determined that AF can be performed (S620), and if it is smaller, it is determined that AF cannot be performed (S619). .
[0069]
As described above, it can be determined whether or not the sample image exists near the center of the image, and it can be determined whether or not AF can be performed in the still image state.
[0070]
Next, a second embodiment of the present invention will be described.
[0071]
In the first embodiment, it is checked whether or not a sample image exists near the center of the mesh enlargement designation frame, and it is determined whether AF may be executed. However, as shown in FIG. Even if no image exists, a sample image may exist over the entire region other than the vicinity of the center. For example, as shown in FIG. 15, a case where a hole 1503 is opened when the sample 1501 is sliced is considered. In such a case, AF execution is impossible in the example of the flowchart of FIG. 11 of the first embodiment (S619). In the second embodiment, the AF execution check process is performed again after determining that the AF execution is impossible.
[0072]
AFChkAria (a) in FIG. 15 is an area in which it is checked whether or not a sample exists near the center of the enlargement designation frame in the flowchart in FIG. There is certainly no sample near the center, but if AF is performed with AFChkAria (b) as the center of the image (optical axis center), no AF error should occur.
[0073]
The AF execution recheck after recognizing that AF execution is impossible in S619 will be described using the flowchart of FIG. First, a luminance threshold (Gth) for a reference recognized as a sample image is set and stored in a memory in the requesting terminal 905 (S1501). Then, initial values of coordinates to be checked are set (S1502). (Xic, Yic) is the center coordinates of an arbitrary mesh enlargement designation frame when the mesh enlargement designation frame is displayed from the macro image as shown in FIG. FrameWidth and FrameHeight are the width and height of the mesh enlargement designation frame.
[0074]
Next, a check counter for determining whether or not a sample image exists is cleared (S1503). Next, while changing the XY coordinates, G luminance information (Gdata (X, Y)) in individual pixels is acquired, and the luminance information is compared with the luminance threshold value (Gth) determined in S1501 (S1504). If it is smaller than the luminance threshold (Gth), the check counter is incremented (S1505). That is, generally, the brightness level is the highest in white. If the sample 1501 has a hole 1503, the G luminance information (Gdata (X, Y)) is larger than the luminance threshold (Gth). Conversely, if there is a sample, the G luminance information (Gdata (X, Y)) is smaller than the luminance threshold (Gth). Then, the X coordinate is incremented and the next coordinate is prepared (S1506). If it is determined in step S1507 that the X coordinate is within the area within the mesh enlargement designation frame, the process returns to step S1504. On the contrary, if the mesh enlargement designation frame is obtained after the X coordinate is incremented in step S1507, the Y coordinate is incremented and the X coordinate is returned to the initial value (S1508). After the Y coordinate is incremented, it is checked whether it is outside the mesh enlargement designation frame area (S1509). If it is within the region, the process returns to step S1504.
[0075]
When all the XY coordinates have been checked, the ratio of the number of pixels in which the sample exists within the mesh enlargement designation frame is calculated based on the following equation (S1510).
[0076]
PicRate = (ChkCounter / FrameWidth × FrameHeight) × 100
If PicRate is 60% or less, it is determined that there is no sample near the center, but there is a sample in a region other than the center (S1511). This 60% number can be changed arbitrarily. After confirming that the sample exists in a region other than the center, the stage is moved to a position where the sample exists (S1512).
[0077]
The position where the sample exists can be determined by searching whether the sample image exists continuously while moving the central coordinates by several pixels.
[0078]
By moving the stage, the specimen image moves to a position near the center of the optical axis, and AF is executed at this position (S1513). If an AF error has occurred, the Z position is returned to the Z position where the previous mesh enlargement designation frame was captured. When the Z movement is completed, the current Z position and XY coordinate values are stored in the memory of the requesting terminal 905 (S1514), and then the stage position is returned to the original position (Xic, Yic) (S1515).
[0079]
As described above, even if it is determined that there is no sample image near the center and AF execution cannot be performed, and it is determined that there are many regions where the sample exists in regions other than the center region, the sample image exists at the position where the sample image exists. By moving the stage, performing AF at this position, and then returning it to the original stage position again, it is possible to acquire a mesh capture designation image with the correct in-focus position.
[0080]
Next explained is the third embodiment of the invention.
[0081]
In the first embodiment, the example in which the mesh designation frame position where AF is first executed is indicated on the monitors 902 and 904 in step S506 in the flowchart of FIG. 9 is shown. In the third embodiment, AF is first executed. An example of automatically specifying the mesh specification frame position to be performed is shown.
[0082]
FIG. 17 is a flowchart showing the processing of the third embodiment of the present invention.
[0083]
After capturing a macro image, a mesh enlargement designation frame is designated on the still image. When this instruction is completed, a mesh enlargement designation frame for executing AF first is automatically determined.
[0084]
First, the number of mesh enlargement designation frames is checked (S1601). In order to have sample image recognition pixel number data for each mesh enlargement designation frame, a sample image check counter is provided as array data (S1602), and the contents of the array data are cleared. The sample image existence check is performed for each mesh enlargement designation frame (S1603). As described above, the sample image check is determined by comparing the luminance threshold data with the luminance information of each pixel. The number of pixels recognized as a sample image is stored in ChkCounter (i). i indicates an arbitrary mesh enlargement designation frame number (numbers (0) to (19) in FIG. 10). When the sample image check of all the mesh enlargement designation frames is completed, the mesh image check counters are arranged in descending order (S1604). The order of arrangement is stored as another variable. Next, it is determined whether or not there is a sample near the center in descending order of the sample image check counter value (S1605). Whether or not a sample exists near the center can be determined by examining the array data stored in step S1602 corresponding to the center coordinates. If it is determined that there is no sample image near the center (S1606), the process returns to S1605 again, and a mesh enlargement designation frame with the next largest sample image check counter value is taken out, and whether or not a sample image exists near the center. Judge. If there is a sample image near the center, it is stored as the position where AF is first executed (S1607).
[0085]
As described above, a portion having the number of pixels that can be recognized most as a sample image from among a plurality of mesh enlargement designation frames and a sample image in the vicinity of the center can be automatically recognized as the first AF position.
[0086]
【The invention's effect】
According to the present invention, in the remote observation of a microscope image, a microscope image transfer system capable of shortening an image capture time after specifying an observation region (mesh division) in an initial observation image and observing an image with a correct in-focus position is provided. Can do.
[Brief description of the drawings]
FIG. 1 is a schematic diagram of a microscope image transfer system to which the present invention is applied.
FIG. 2 is a flowchart showing basic operations of the telepathology system.
FIG. 3 is an explanatory diagram of a mesh shooting designation frame and a spot shooting designation frame.
FIG. 4 is a diagram for explaining movement of an enlargement designation frame.
FIG. 5 is a flowchart showing a basic operation when the operation right of the telepathology system is on the requesting side.
FIG. 6 is a diagram when mesh designation processing is performed in a still image state after macro shooting in the embodiment of the present invention.
FIG. 7 is a diagram for explaining acquisition of Z correction data associated with stage XY movement in the embodiment of the present invention.
FIG. 8 is a table showing comparison data ΔZ for each coordinate obtained by comparing the Z position at the focused position with the Z initial value data in the embodiment of the present invention.
FIG. 9 is a flowchart showing mesh image capturing processing in the embodiment of the present invention.
FIG. 10 is a diagram illustrating setting of a mesh division position as a specimen image position in the embodiment of the present invention.
FIG. 11 is a flowchart showing a check process of whether or not an autofocus operation can be executed in the embodiment of the present invention.
FIG. 12 is a diagram illustrating an area for determining AF execution from a mesh division screen in the embodiment of the present invention.
FIG. 13 is a flowchart showing a process for determining an X region to be determined for autofocus execution in the embodiment of the present invention.
FIG. 14 is a flowchart showing processing for determining a Y region for performing autofocus execution in the embodiment of the present invention.
FIG. 15 is a diagram for explaining the change of the autofocus check area in the second embodiment of the present invention.
FIG. 16 is a flowchart for rechecking autofocus in the second embodiment of the present invention;
FIGS. 17A and 17B are diagrams illustrating locations where autofocus is executed in the third embodiment of the present invention. FIGS.
[Explanation of symbols]
901 ... Observation terminal
902 ... Monitor
903 ... ISDN
904 ... Monitor
905 ... Requesting terminal
906 ... Video camera
907 ... Microscope
908 ... Electric revolver
909 ... Electric stage
910a, 910b ... line connection device
911 ... Macro photography device
912 ... XY stage control unit
913 ... Microscope operation unit

Claims (3)

In a microscope image transfer system that can specify the capture of an enlarged image at a desired magnification on a still image,
Luminance information storage means for storing the luminance information of the enlargement designation frame area on the still image;
Z-direction position correcting means for correcting the position in the Z-direction accompanying the displacement of the microscope XY stage
Autofocus execution enable / disable determining means for determining whether or not autofocus can be executed before capturing an enlarged image on a still image;
XYZ position storage means for storing the XY position of the microscope XY stage and the Z position of the microscope XY stage when the autofocus execution enable / disable determination means determines that autofocus execution is possible;
A Z position restoring means for restoring the previous Z position of the microscope XY stage stored in the XYZ position storage means when the autofocus execution possibility judging means judges that autofocus execution is impossible;
A microscope image transfer system comprising: Means for moving the microscope XY stage to XY at a position where autofocusing is possible;
Means for controlling execution of autofocus after XY movement of the microscope XY stage;
Means for restoring the position of the XY stage to the position before the movement after the execution of autofocus;
Means for controlling image capture after restoring the position of the previous microscope XY stage to the position before movement;
The microscope image transfer system according to claim 1, further comprising: 2. The microscope image transfer system according to claim 1, further comprising means for automatically selecting a place where autofocus is first executed from a plurality of image capture designation frames.

Images