JP5059816B2 - Slide image data creation device - Google Patents

Description

  The present invention relates to a slide image data creation device and slide image data. In particular, in a histopathological examination, a hematology examination, a genetic examination, a cytology examination, etc., a preparation (slide glass) containing a sample is set on a microscope to increase the height. The present invention relates to a slide image data creation device that shoots at a magnification, generates and manages slide image data, and generated slide image data.

  In recent years, in microscopic examinations of histopathology, hematology, genetic testing, etc., the collected pathological tissues and samples such as cells and blood are photographed with an imaging device equipped with a microscope, and the images are taken into a computer and displayed. Image data that can be observed above is created. For example, Patent Document 1 below discloses the above-described technique.

JP 2003-222801 A

In the conventional system described above, it takes a long time to take an image of a sample with a microscope, and a large amount of memory is required to open and process a large number of image files when the image is taken or viewed. There were problems such as taking time. In addition, since basically only two-dimensional image information can be captured, there is a problem that the structure of the sample in the Z-axis direction parallel to the optical axis of the microscope cannot be determined from the captured image.
The present invention solves the problems in the prior art as described above, and can generate slide image data including a plurality of images obtained by photographing a necessary part of a sample at a high magnification with a microscope at high speed and automatically. It is an object to provide a data creation device and generated slide image data.

  The slide image data creation device of the present invention is provided in a microscope, and controls a digital photographing means for photographing a high-magnification image, a sample conveying means that holds a sample and is movable in a three-dimensional direction, and controls the sample conveying means. Then, the digital photographing means is controlled to photograph a plurality of high-magnification images while sequentially moving the sample in a direction perpendicular to the optical axis of the microscope so that there is a marginal part where the image overlaps with another image. A control unit that recognizes the margin portion by image processing from a plurality of high-magnification images, and generates a combination information of the plurality of high-magnification images; and a plurality of high-magnification images together with the combination information The main feature is that it includes image file generation means for storing in one file.

  In addition, the slide image data creation apparatus described above is characterized in that a plurality of images are taken at predetermined intervals while being finely moved in a direction parallel to the optical axis of the microscope. In addition, the slide image data generating apparatus described above is characterized in that it includes a focus position detecting unit that detects a position where the microscope is focused in a direction parallel to the optical axis of the microscope.

  Further, the slide image data creation apparatus described above is further characterized in that it includes thumbnail image generation means for generating a thumbnail image that is a reduced image of the entire image obtained by pasting together the high-magnification images. In addition, the slide image data creation apparatus described above is further characterized in that it includes external digital camera means. In addition, the slide image data creation device described above is characterized in that images taken by the external digital camera, the plurality of high-magnification images, and the pasting information are collectively stored in one moving image file. .

In addition, the slide image data creation device described above further includes a photographing area automatic recognition means that automatically determines a region for photographing a high-magnification image from an image photographed by the external digital camera. . In the above slide image data creation device, a list of images taken by the external digital camera for a plurality of slide glasses together with the shooting areas recognized by the shooting area automatic recognition means is displayed, and the user takes the shooting. It is also characterized in that it has a photographing area semi-automatic recognition means that can correct the area.
Further, the slide image data creating apparatus described above is characterized in that it further includes an image publishing unit that registers a captured image file in a database and can search and refer to an image via a network.
The main feature of the slide image data of the present invention is that a plurality of high-magnification image data and image pasting information are stored together in one moving image file.

  According to the present invention, it is possible to shoot only a region necessary for photographing with a microscope at high speed, and it is not necessary to open and process a large number of image files at the time of browsing by storing image data in one file. In addition, a large amount of memory is not required and processing and display can be performed at high speed. In addition, according to the present invention, the three-dimensional structure of the sample can be determined by scanning the focal position in the Z-axis direction and taking an image. For example, it is possible to determine whether the shape of the cell nucleus necessary for diagnosis is a true sphere or an elliptic sphere. Furthermore, image data can be managed by a database, and a virtual slide image and its attribute information can be browsed through a network or the Web.

It is a block diagram which shows the structure of the whole system containing the slide image data production apparatus of this invention. It is a top view which shows the example of the slide glass used in this invention. It is a top view which shows the structure of the slide glass imaging device of this invention. It is a flowchart which shows the content of the fully automatic slide creation process by this invention. It is a flowchart which shows the content of the high magnification image imaging process of S16. It is a flowchart which shows the content of the Z-axis position adjustment process of S33. It is a flowchart which shows the content of the viewer process by this invention. It is explanatory drawing which shows the example of the structure of a file, and the image displayed. It is a flowchart which shows the content of the semi-automatic slide creation process of 2nd Example of this invention.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the entire system including the slide image data creation device of the present invention. The slide image data creation device of the present invention is roughly composed of a slide glass photographing device including a microscope 10 and a personal computer 13. The personal computer 13 is connected to a DB, a Web server 50, a DB (database) 51, and a terminal (PC) 52 via a LAN 54. The DB and the Web server 50 are further connected to an external terminal (PC) via a router 53 and the Internet 55. ) 56.

  FIG. 3 is a plan view showing the configuration of the slide glass photographing apparatus of the present invention. The slide glass photographing apparatus includes a microscope 10 and a digital (external) camera 12 fixed on a base 20, a drive device 31 for a slide glass storage magazine 30, and three-dimensional movement mechanisms 21 to 28 for holding and moving the slide glass. The motor drive device 16 and the like are provided.

  The microscope 10 has a binocular eyepiece lens for observing the sample with the naked eye, and has a built-in digital microscope camera 11 that can be controlled from the personal computer 13, and digitally captures a high-magnification image of the sample taken by the microscope camera 11. Data can be taken into the personal computer 13 and the image data can be edited to form a database as described later, and a desired portion can be displayed on the display device 14.

  The microscope 10 is fixed at a predetermined position by a fixed attachment attached on the base 20, and a plurality of types of fixed attachments having different shapes so as to be compatible with microscopes of a plurality of manufacturers. It is prepared. In this embodiment, the optical axis of the microscope 10 is the vertical direction.

  The digital (external) camera 12 is also fixed on the base 20 and outputs normal color still image data to the personal computer 13 based on control from the personal computer 13. It is used for photographing a loupe image that is an entire image of the sample portion of the slide glass 17 and reading a two-dimensional barcode 42 and the like.

  The three-dimensional moving mechanisms 21 to 28 include a first movable frame 23 that can be moved in the front-rear direction by the stepping motor 22, a second movable frame 25 that can be moved in the left-right direction by the stepping motor 24, and a vertical movement by the stepping motor 26. A movable third movable frame 27 and a slide glass holding hand 28 fixed to the third movable frame 27 are provided.

  Then, each stepping motor is driven by the motor driving device 16 based on the control from the personal computer 13 so that the slide glass 17 is held by the slide glass holding hand 28 and moved to any position in the front / rear, left / right and up / down directions. It is configured to be possible.

  The three-dimensional movement mechanisms 21 to 28 take out the slide glass 17 from the slide glass storage magazine 30 and move it below the digital camera 12 and between the objective lens 18 of the microscope 10 and a condenser lens (not shown) below. In addition, the operation of returning to the original slide glass storage magazine 30 is automatically performed.

  When shooting a high-magnification image, it is necessary to divide a single sample into a number of images, so that the position before and after shooting should be adjusted in the horizontal direction, and the vertical direction centered on the in-focus position. Regarding the range, a plurality of images are taken while moving the third movable frame 27 by a minute length. In order to prevent a collision with the objective lens 18 from moving up and down, the movable range is set within 2 millimeters, and the focus is adjusted within the range.

  The drive device 31 of the slide glass storage magazine 30 moves the slide glass storage magazine 30 up and down by the stepping motor 32 being driven by the motor drive device 16 based on the control from the personal computer 13. A slide glass storage magazine 30 is detachably attached to the drive device 31. The slide glass storage magazine 30 is provided with a plurality of slide glass holding grooves arranged in parallel, and is configured to accommodate a plurality of slide glasses 17 arranged in a shelf shape.

  The personal computer 13 is a personal computer equipped with well-known standard hardware, and is connected to the display device 14, the keyboard 15, a mouse, a LAN 54, and the like. Further, for example, the microscope camera 11, the digital camera 12, the motor driving device 16 and the like are connected by a USB interface. The personal computer 13 executes processing to be described later based on a user instruction.

  FIG. 4 is a flowchart showing the contents of the fully automatic slide creation process according to the present invention. This process is executed by the personal computer 13. When this processing is performed, first, a slide glass on which an image is to be taken is packed and set in the slide glass storage magazine 30 from below. In S10, the information on the number of slide glasses input by the user is read. In S <b> 11, the unprocessed slide glass is taken out by controlling the three-dimensional moving mechanisms 21 to 28 and set under the digital camera 12.

  FIG. 2 is a plan view showing an example of a slide glass used in the present invention. A sample is sandwiched between the slide glass 17 and the cover glass 45, and a two-dimensional barcode 42 such as a QR code in which an identification number of the slide glass is recorded is attached to the upper left of the slide glass 17. . In S <b> 12, the two-dimensional barcode 42 is read using the digital camera 12.

  In S <b> 13, a loupe image is taken using the digital camera 12. The loupe image is an image in a range indicated by a dotted line 46 in FIG. 2 and includes the entire sample region. A loupe image is extracted and displayed on the right side of FIG.

  In S14, the photographing area is automatically recognized. Since the high-magnification microscope camera 11 has a small shooting area, it is not possible to take the entire sample area with a single image. Therefore, as shown on the right side of FIG. 2, the loupe image is divided into a plurality of regions 47 in a grid pattern (6 × 6 in FIG. 2), and a region slightly wider than each of the divided regions 47 is a high magnification microscope camera. 11 to shoot.

  When sample tissues are scattered on the slide glass, only the mark area in which only necessary portions are automatically extracted is taken in order to increase the processing speed and reduce the file capacity. For example, in the diagram on the right side of FIG. 2, when it is recognized by image processing that there is no sample in the divided image in the leftmost column, the leftmost column is not photographed. Alternatively, only a region with a specimen may be extracted from the loupe image, and only the range covering the extracted region may be photographed with the high-power microscope camera 11.

In S <b> 15, the three-dimensional moving mechanisms 21 to 28 are controlled to move the slide glass 17 below the objective lens 18 of the microscope 10. In S16, a high-magnification image photographing process described later is performed. At this time, a thumbnail image that is a reduced image of the entire image obtained by pasting the high-magnification images is also generated.
In S17, the loupe image + thumbnail image + high-magnification image is collected into one moving image file, and is registered and stored in the DB and the DB 50 provided in the Web server 50 together with the slide glass identification number. It is assumed that other information such as a patient name corresponding to the slide glass identification number is already registered in the DB 50.

  FIG. 8 is an explanatory diagram illustrating an example of a file structure and a displayed image. The moving image file 60 includes a plurality of frame data, and an information frame 61 is stored in the first frame. The information frame 61 stores information such as system information, machine name, version information, shooting magnification, system configuration, and camera name. A loupe image 63 is stored in the second frame, and shooting position information between frames is also stored in this frame. A thumbnail image 70 is stored in the final frame, and a high-magnification image is stored from the third frame to the frame immediately before the final frame.

  Returning to FIG. 4, in S <b> 18, the three-dimensional moving mechanisms 21 to 28 are controlled to house the slide glass in the original position. In S19, it is determined whether or not there is an unprocessed slide. If the determination result is affirmative, the process proceeds to S11, but if not, the process ends.

  FIG. 5 is a flowchart showing the contents of the high-magnification image capturing process in S16. In S30, desired parameters for the exposure time, shutter speed, color balance and the like are set in the microscope camera 11. In S31, it is moved to the initial position of the Y axis (left and right direction). For example, when all the areas of the loupe image are photographed, the slide glass 17 is moved to a position where the uppermost divided image of the loupe image enters the photographing area of the microscope camera 11.

  In S32, it is moved to the initial position of the X axis. For example, when all the areas of the loupe image are photographed, the slide glass 17 is moved to a position where the divided image in the leftmost column of the loupe image enters the photographing area of the microscope camera 11. In S33, a Z-axis position adjustment process described later is performed to detect the Z-axis position of the focus.

  In S34, it is moved to the initial position of the Z axis. In the present invention, a high-magnification image is taken while changing the position at equal intervals in a predetermined range centered on the focal position in the Z-axis direction. Accordingly, in S34, the slide glass 17 is moved downward by a predetermined length from the focal position.

In S35, a high-magnification color image is taken by the microscope camera 11, and image data is read. In S36, if there are other high-magnification images on the top, bottom, left, and right (XY axis direction), overlap information with these images is generated. The overlap information is information indicating, for example, how many pixels overlap another image, that is, which pixel is to be pasted. The overlap information can be obtained, for example, by correlating the edge images while changing the number of overlapping pixels.

  The obtained overlap information is added to the image data and stored. The format of the file to be saved may be, for example, a moving image file format compliant with JPEG2000. Note that the overlap information is incorporated into the high-magnification image itself. In addition to overlapping information, system information such as AF (autofocus) information, hardware identification information, shooting software version information, and X, Y, and Z position information is stored as incorporated information. The number of pixels of one high-magnification image may be, for example, 1280 × 1024. Further, at this time, a thumbnail image that is a reduced image of the entire mark area formed by joining the high-quality images is also generated.

  In S37, the Z axis is moved by one step. The width of one step may be 0.5 microns, for example. In S38, it is determined whether or not the Z-axis processing is completed depending on whether or not the Z-axis has reached the final position. If the determination result is negative, the process proceeds to S35. Migrate to

In S39, the X axis is moved by one step. The width of one step varies depending on the magnification. For example, at 40 times, the width is about 0.18 millimeters. In S40, it is determined whether or not the processing of the X axis is completed depending on whether or not the X axis has reached the final position. If the determination result is negative, the process proceeds to S35, but if the determination is positive, the process proceeds to S39. Migrate to

In S41, the Y axis is moved one step. In S42, it is determined whether or not the processing of the Y axis is completed depending on whether or not the Y axis has reached the final position. If the determination result is negative, the process proceeds to S35, but if the determination is positive, the process is performed. Exit.

FIG. 6 is a flowchart showing the contents of the Z-axis position adjustment process in S33. In S50, it is determined whether or not the Z-axis position adjustment process is executed for the first time. If the determination result is negative, the process proceeds to S51, but if the determination is affirmative, the process proceeds to S55. In S51, a high-magnification image is read at the previous Z-axis center position. In S52, FFT (Fast Fourier Transform) processing is performed, and the frequency spectrum of the image is calculated.

In S53, the peak power value of the frequency spectrum is compared with the stored value. In S54, it is determined whether or not the peak power value is, for example, less than 90% of the stored value. If the determination result is negative, the Z-axis position is not changed and the previous focal position is used as it is, so the processing ends. However, if the determination is affirmative, the process proceeds to S55 to redetect the focal position.

In S55, high-magnification images are sequentially read while scanning a predetermined range in the Z-axis direction. In S56, FFT (Fast Fourier Transform) processing is performed on all images, and the frequency spectrum of the images is calculated. In S57, the Z position with the maximum peak power value is detected. In S58, the peak power value is stored, and this Z position is set as the Z-axis center position. In S59, three images at the center and arbitrarily designated vertical positions are acquired.

  FIG. 7 is a flowchart showing the contents of the viewer process according to the present invention. This process is executed in the personal computer 13 or the terminal (PC) 52, for example. In S70, the slide selection information to be displayed input by the user is input. The user may search the DB 51 based on a keyword such as a patient name. In S71, additional information text files such as a moving image file, diagnostic information storage text file, marking information and comment information storage text file corresponding to the request are read from the DB. In S72, a loupe image that is an entire image of the slide is displayed.

  FIG. 8 is an explanatory diagram illustrating an example of a file structure and a displayed image. The loupe image 63 includes the entire sample of the slide glass. The loupe image 63 is automatically or manually set, and mark areas 64 and 65 where a high-magnification image is taken are displayed. Here, the user clicks in the mark area 64, for example.

  In S73, a thumbnail image 70 that is a reduced image of the entire mark area is displayed. The thumbnail image 70 in the middle of FIG. 8 is obtained by pasting and reducing, for example, 4 × 4 = 16 high-magnification images, and dotted lines 71 and 72 are boundaries between the high-magnification images that are not displayed. Here, the user clicks, for example, a portion of the thumbnail image 70 to be viewed (the center of the region 73).

  In S74, a plurality of high-magnification images necessary for displaying the designated position at an arbitrary magnification designated by the user are determined, and the images are read out. For example, in order to display the region 73, four high-magnification images are required. In S75, the read high-magnification image is pasted based on the embedded overlap information.

  In S76, the high-magnification image 75 and the marking information, comments, and diagnosis information that have been pasted together are shaped and displayed according to the screen size. Note that the display area can be moved up, down, left, and right while the high-magnification image 75 is displayed. Further, the movement in the Z-axis direction is displayed by pasting the image after movement based on an instruction to move upward (front side of the screen) or downward (back side of the screen).

  In S77, it is determined whether or not there is a marking request. If the determination result is negative, the process proceeds to S79, but if the determination is affirmative, the process proceeds to S78. In S78, a marking information recording process is performed. Marking is a function that enables marking (color, shape, etc.) on an image. The contents of the marking information include, for example, a marking number, X-axis position, Y-axis position, Z-axis position, marking shape, color, line thickness, and the like. The marking information is stored as layer information without changing the original image.

  In S79, it is determined whether or not there is a comment request. If the determination result is negative, the process proceeds to S81, but if the determination is affirmative, the process proceeds to S80. In S80, a comment information recording process is performed. The user writes a comment in a comment box displayed on another screen. Comment information is also saved as layer information.

  In S81, it is determined whether or not to return to the display of the entire image. If the determination result is negative, the process proceeds to S82, but if the determination is affirmative, the process proceeds to S73 or S72. In S82, it is determined whether or not to end the process. If the determination result is negative, the process proceeds to S77, but if the determination is affirmative, the process ends.

  When accessing from the external terminal 56 via the Internet, the viewer program described above may be installed in the terminal 56 and necessary files may be downloaded from the DB 51, but the same function as the viewer program is executed in the Web server 50. In the terminal 56, a known Web browser may be activated to access the Web server. The Web server 50 generates screen data to be displayed based on operation information from the browser and transmits it to the terminal 56. In this way, it is not necessary to install a dedicated program, and images can be viewed from any terminal connected to the Internet.

  FIG. 9 is a flowchart showing the contents of the semi-automatic slide creation process of the second embodiment of the present invention. In the first embodiment, an example of automatically recognizing a mark area for photographing a high-magnification image is disclosed. However, if all areas of a sample are photographed at a high magnification, it takes a very long time and the amount of data is very large. turn into. Therefore, first, only the loupe images of all the slide glasses are photographed and displayed in a list, and the user can mark only the portions where high magnification images are necessary for each loupe image. It can be shortened and the data volume can be reduced.

  In S100, the slide glass number information input by the user is read. In S <b> 101, the unprocessed slide glass 17 is taken out by controlling the three-dimensional moving mechanisms 21 to 28 and set under the digital camera 12. In S102, the two-dimensional barcode is read using the digital camera 12.

  In S103, a loupe image is taken using the digital camera 12, and the slide glass is stored in the original storage position. In S104, the shooting area is automatically recognized. In S105, it is determined whether or not there is an unprocessed slide. If the determination result is affirmative, the process proceeds to S101, but if not, the process proceeds to S106.

  In S106, loupe images and automatic recognition areas of all the slide glasses are displayed. Here, the user marks only the portion of the loupe image whose auto-recognition area is desired to be changed that requires high-magnification imaging. In S107, it is determined whether or not there has been a change operation by the user. If the determination result is affirmative, the process proceeds to S108, but if it is negative (change operation complete), the process proceeds to S109. In S108, the input shooting area information is read, the shooting area information is updated, and the process proceeds to S106.

  In S <b> 109, the three-dimensional moving mechanisms 21 to 28 are controlled to take out an unprocessed slide glass 17 and set it under the digital camera 12. In S110, the digital camera 12 is used to read the two-dimensional barcode. In S111, a loupe image is taken using the digital camera 12.

  In S112, the read two-dimensional bar code is checked to confirm the slide glass, and the positional deviation between the loupe image photographed in S103 and the loupe image photographed in S111 is detected, and the three-dimensional movement mechanism is based on the detected value. The movement positions 21 to 28 are corrected.

  In S113, the slide glass is moved below the objective lens 18 of the microscope 10. In S114, the above-described high-magnification image photographing process is performed. At this time, a thumbnail image that is a reduced image of the entire image obtained by pasting the high-magnification images is also generated. In S115, the loupe image + thumbnail image + high-magnification image are collected and stored in the DB 50 as one moving image file.

In S116, the three-dimensional moving mechanisms 21 to 28 are controlled to store the slide glass in the original position. In S117, it is determined whether or not there is an unprocessed slide glass. If the determination result is negative, the process ends. If the determination is affirmative, the process proceeds to S109.

  Although the embodiments have been described above, the following modifications may be considered in the present invention. In the embodiment, the inclination and rotation of the image are not taken into consideration, but the relative inclination (rotation) of the image is detected and corrected between the magnifier image and the high magnification image or between the adjacent high magnification images. You may do it. Further, lens aberration correction, color correction, and the like may be performed.

  The slide image data creation apparatus of the present invention can be widely used in all fields using a microscope, such as medicine (pathology, hematology, parasitology, etc.), agriculture, pharmacy, science, education, etc.

DESCRIPTION OF SYMBOLS 10 Microscope 11 Microscope camera 12 Digital camera 13 Personal computer 16 Motor drive device 17 Slide glass 18 Objective lens 20 Base 21-28 Three-dimensional movement mechanism 30 Slide glass storage magazine

Claims (3)

An external digital camera means for capturing an entire image of the sample portion of the slide glass;
Photographing area automatic recognition means for automatically determining a region for photographing a high-magnification image from the whole image photographed by the external digital camera means;
A digital photographing means provided in a microscope for photographing a high-magnification image;
A sample transport means that holds the sample and is movable in a three-dimensional direction;
High-magnification images are sequentially read while scanning a predetermined range in the Z-axis direction, fast Fourier transform processing is performed on all images, the frequency spectrum of the image is calculated, and the Z position with the maximum peak power value is the center of the Z-axis by the position, and means for detecting the position in focus of the microscope in parallel to the optical axis Z direction of the microscope, it said means high magnification image while scanning a predetermined range in the Z axis direction Before sequential reading, the high-magnification image is read at the Z-axis center position at the time of the previous high-magnification image shooting, fast Fourier transform processing is performed, the frequency spectrum of the image is calculated, and the peak power value of the frequency spectrum is set to the previous peak. Compare with power value. When the peak power value is equal to or greater than a predetermined ratio of the previous peak power value, the Z-axis center position is not used as the Z-axis center position without sequentially reading a high-magnification image while scanning a predetermined range in the Z-axis direction. A focus position detecting means that uses the axial center position as it is;
The sample is controlled so that there is a marginal part where the image overlaps with other images only in the region where the high magnification image determined by the imaging region automatic recognition unit is controlled by controlling the sample transport unit. While sequentially moving in the XY direction perpendicular to the optical axis of the microscope, the sample is moved in the Z direction for a predetermined range centered on the Z position of the focus detected by the focus position detecting means at each XY position. An imaging control unit that controls imaging of a plurality of high-magnification images in three-dimensional directions of XYZ by performing an imaging operation of imaging the high-magnification image by controlling the digital imaging unit while changing positions at equal intervals.
Bonding information generating means for generating bonding information of the plurality of high-magnification images;
Thumbnail image generating means for generating a thumbnail image that is a reduced image of the entire image obtained by pasting the plurality of high-magnification images;
An image file generating means for storing the whole image, the plurality of high-magnification images, the pasting information, and the thumbnail image in one file.   Furthermore, a shooting area semi-automatic recognition means for displaying a list of images taken by the external digital camera for a plurality of slide glasses together with the shooting areas recognized by the shooting area automatic recognition means, and allowing the user to correct the shooting area. The slide image data creation device according to claim 1, further comprising:   2. The slide image data creating apparatus according to claim 1, further comprising image publishing means for registering a captured image file in a database and searching and referencing the image via a network.

Images