JP6338730B2 - Apparatus, method, and program for generating display data - Google Patents

Description

The present invention relates to an apparatus for generating display data, Methods relates及 beauty program.

  In recent years, in the field of pathology, as an alternative to an optical microscope that is a tool for pathological diagnosis, a virtual slide that enables pathological diagnosis on a display by imaging a test sample (specimen) placed on a slide and digitizing the image The system is drawing attention. By digitizing a pathological diagnosis image using a virtual slide system, a conventional optical microscope image of a test sample can be handled as digital data. As a result, it is expected that merits such as speeding up of remote diagnosis, explanation to patients using digital images, sharing of rare cases, efficiency of education and practical training, etc. will be obtained.

In order to realize an operation equivalent to that of an optical microscope using a virtual slide system, it is necessary to digitize the entire test sample on the slide. By digitizing the entire test sample, the digital data created by the virtual slide system is converted to a PC (Personal Computer).
And viewer software that runs on a workstation. When the entire test sample is digitized, the number of pixels is usually a very large data amount of several hundred million to several billion pixels. The amount of data created by the virtual slide system is enormous. Therefore, it is possible to observe from the micro (detailed enlarged image) to the macro (overall bird's-eye view) by performing the enlargement / reduction processing with the viewer. Provide convenience. By acquiring all necessary information in advance, it is possible to immediately display the resolution and magnification required by the user from a low-magnification image to a high-magnification image.

  Also, an image processing apparatus has been proposed in which an annotation is added to a medical image when a medical image (ultrasound image) is acquired, and the medical image is searched using a comment in the annotation as a search key (patent) Reference 1).

  An information processing apparatus has been proposed that retains a display magnification and a display position when an annotation is added to an electronic document and displays the electronic document on a screen based on the retained display magnification and display position (patent) Reference 2).

JP-A-11-353327 JP 2010-61311 A

  When an annotation is added to the virtual slide image, it is difficult for the user to understand the magnification of the virtual slide image when the annotation is added. That is, it is difficult for the user to understand the difference between the magnification at the time of observation and the magnification at the time of annotation addition. In addition, when the magnification at the time of adding a plurality of annotations is different, it is difficult for the user to understand the difference between the magnification at the time of adding each annotation and the magnification at the time of observation.

The same applies to the case where the virtual slide image is a depth image (Z stack image), and it is difficult for the user to know the focal position (Z position) of the virtual slide image when the annotation is added. That is, it is difficult for the user to understand the difference between the focus position at the time of observation and the focus position at the time of annotation addition. In addition, when the focus positions at the time of adding a plurality of annotations are different, it is difficult for the user to understand the difference between the focus position at the time of adding each annotation and the focus position at the time of observation.

  Therefore, an object of the present invention is to allow a user to easily grasp the magnification and the focal position of a virtual slide image when an annotation is added when the annotation is displayed.

The present invention is an apparatus for generating display data for displaying an image of a subject on a display device ,
An additional information acquiring unit that acquires information of the display magnification of the image when an annotation is added to the image,
Designated information acquisition means for acquiring display magnification information of the image specified by the user ;
Anda generating means for generating said display data for displaying said annotation with the image of the designated display magnification,
Said generating means, in the table示倍rate when said additional said the case where the designated display magnification is different, the and the added display magnification and the designated display magnification when the same case as the user The display data different from each other is generated so that it is possible to determine which is the case.
It is the apparatus characterized by this.

The present invention is a method of generating display data for displaying an image of a subject on a display device ,
An additional information acquiring step of acquiring the information of the display magnification of the image when an annotation is added to the image,
A designated information obtaining step for obtaining information on the display magnification of the image designated by the user ;
Anda generating step of generating the display data for displaying said annotation with the image of the designated display magnification,
In the generating step, the display magnification when the additional and the in the case where the designated display magnification is different, the and the added display magnification and the designated display magnification when the same case as the user Is a method of generating the display data different from each other so that it is possible to determine which is the case .

The present invention is a program for causing a computer to execute each step of the above-described method .

  According to the present invention, the user can easily grasp the magnification and the focal position of the virtual slide image when the annotation is added when the annotation is displayed.

Overall view of apparatus configuration of image processing system according to embodiment Functional block diagram of imaging device in image processing system according to embodiment Functional block diagram of an image processing apparatus according to an embodiment 1 is a hardware configuration diagram of an image processing apparatus according to an embodiment. The figure explaining the concept of the hierarchy image prepared beforehand for every different magnification Flow chart showing the flow of annotation addition and presentation Flow chart showing the detailed flow of adding annotation Flow chart showing the detailed flow of annotation presentation Example of display screen of image processing system of the present invention Overall view of apparatus configuration of image processing system according to embodiment 2 The figure explaining the concept of the depth image prepared beforehand for every different focus position of Example 2. The flowchart which shows the flow of annotation addition of Example 2. Flowchart showing annotation presentation processing of the second embodiment Flowchart of annotation data display control processing of embodiment 3 Flowchart of annotation data display control processing of embodiment 4 One-dimensional schematic diagram of depth image data of Examples 3 and 4

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
The image processing apparatus of the present invention can be used in an image processing system including an imaging device and a display device. This image processing system will be described with reference to FIG.

(Device configuration of image processing system)
FIG. 1 shows an image processing system using an image processing apparatus according to the present invention, which includes an imaging apparatus (microscope apparatus or virtual slide scanner) 101, an image processing apparatus 102, and a display apparatus 103. This image processing system is a system having a function of acquiring and displaying a two-dimensional image of a specimen (test sample, subject) to be imaged. The imaging apparatus 101 and the image processing apparatus 102 are connected by a dedicated or general-purpose I / F cable 104, and the image processing apparatus 102 and the display apparatus 103 are connected by a general-purpose I / F cable 105. .

  The imaging apparatus 101 captures a plurality of two-dimensional images having different positions in the two-dimensional plane direction and different positions in the depth direction perpendicular to the two-dimensional plane (that is, different focal positions), and outputs a digital image. A virtual slide device having a function can be used. A solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) is used to acquire a two-dimensional image. Note that, instead of the virtual slide device, the imaging device 101 can be configured by a digital microscope device in which a digital camera is attached to an eyepiece of a normal optical microscope.

  The image processing apparatus 102 is an apparatus having a function of generating display data to be displayed on the display apparatus 103 from a plurality of pieces of imaging data acquired from the imaging apparatus 101 in response to a request from the user. The image processing apparatus 102 is configured by a general-purpose computer or workstation including hardware resources such as various I / Fs including a CPU (Central Processing Unit), a RAM, a storage device, and an operation unit. The storage device is a large-capacity information storage device such as a hard disk drive, and stores programs, data, OS (operating system) and the like for realizing each processing described later. Each function described above is realized by the CPU loading a necessary program and data from the storage device to the RAM and executing the program. The operation unit includes a keyboard 106, a mouse 107, and the like, and is used by an operator to input various instructions.

  The display device 103 is a display that displays an observation image that is a result of the arithmetic processing performed by the image processing device 102, and is configured by a CRT, a liquid crystal display, or the like.

  In the example of FIG. 1, the imaging system is configured by three devices, that is, the imaging device 101, the image processing device 102, and the display device 103, but the configuration of the present invention is not limited to this configuration. For example, an image processing device integrated with a display device may be used, or the function of the image processing device may be incorporated in the imaging device. The functions of the imaging device, the image processing device, and the display device can be realized by a single device. Conversely, the functions of the image processing apparatus and the like may be divided and realized by a plurality of apparatuses.

(Functional configuration of imaging device)
FIG. 2 is a block diagram illustrating a functional configuration of the imaging apparatus 101.
The imaging apparatus 101 is generally configured by an illumination unit 201, a stage 202, a stage control unit 205, an imaging optical system 207, an imaging unit 210, a development processing unit 219, a pre-measurement unit 220, a main control system 221, and a data output unit 222. Is done.

  The illumination unit 201 is means for uniformly irradiating the preparation 206 disposed on the stage 202, and includes a light source, an illumination optical system, and a light source drive control system. The stage 202 is driven and controlled by a stage control unit 205, and can move in three directions of XYZ. The preparation 206 is a member in which a section of tissue to be observed and a smeared cell are attached on a slide glass and fixed together with an encapsulant under the cover glass.

  The stage control unit 205 includes a drive control system 203 and a stage drive mechanism 204. The drive control system 203 receives the instruction from the main control system 221 and performs drive control of the stage 202. The moving direction, moving amount, and the like of the stage 202 are determined based on the position information and thickness information (distance information) of the sample measured by the pre-measurement unit 220 and, if necessary, instructions from the user. The stage drive mechanism 204 drives the stage 202 in accordance with instructions from the drive control system 203.

  The imaging optical system 207 is a lens group for forming an optical image of the specimen of the preparation 206 on the imaging sensor 208.

The imaging unit 210 includes an imaging sensor 208 and an analog front end (AFE) 209. The imaging sensor 208 is a one-dimensional or two-dimensional image sensor that changes a two-dimensional optical image into an electrical physical quantity by photoelectric conversion. For example, a CCD or a CMOS device is used. In the case of a one-dimensional sensor, a two-dimensional image is obtained by scanning in the scanning direction. The imaging sensor 208 outputs an electrical signal having a voltage value corresponding to the light intensity. When a color image is required as a captured image, for example, a single plate image sensor to which a Bayer array color filter is attached may be used. The imaging unit 210 captures a divided image of the specimen by driving the stage 202 in the XY axis direction.

  The AFE 209 is a circuit that converts an analog signal output from the image sensor 208 into a digital signal. The AFE 209 includes an H / V driver, a CDS (Correlated double sampling), an amplifier, an AD converter, and a timing generator, which will be described later. The H / V driver converts a vertical synchronization signal and a horizontal synchronization signal for driving the image sensor 208 into potentials necessary for driving the sensor. CDS is a double correlation sampling circuit that removes noise of fixed patterns. The amplifier is an analog amplifier that adjusts the gain of an analog signal from which noise has been removed by CDS. The AD converter converts an analog signal into a digital signal. When the output at the final stage of the imaging apparatus is 8 bits, the AD converter converts the analog signal into digital data quantized from about 10 bits to about 16 bits and outputs it in consideration of subsequent processing. The converted sensor output data is called RAW data. The RAW data is developed by a subsequent development processing unit 219. The timing generator generates a signal for adjusting the timing of the image sensor 208 and the timing of the development processing unit 219 in the subsequent stage.

  When a CCD is used as the image sensor 208, the AFE 209 is indispensable. However, in the case of a CMOS image sensor capable of digital output, the function of the AFE 209 is included in the sensor. Although not shown, there is an imaging control unit that controls the imaging sensor 208, and controls the operation timing and control of the imaging sensor 208, such as the shutter speed, frame rate, and ROI (Region Of Interest). Do.

  The development processing unit 219 includes a black correction unit 211, a white balance adjustment unit 212, a demosaicing processing unit 213, an image composition processing unit 214, a resolution conversion processing unit 215, a filter processing unit 216, a γ correction unit 217, and a compression processing unit 218. Composed. The black correction unit 211 performs a process of subtracting the black correction data obtained at the time of shading from each pixel of the RAW data. The white balance adjustment unit 212 performs a process of reproducing a desired white color by adjusting the gain of each RGB color according to the color temperature of the light of the illumination unit 201. Specifically, white balance correction data is added to the RAW data after black correction. When handling a monochrome image, the white balance adjustment process is not necessary. The development processing unit 219 generates hierarchical image data to be described later from the divided image data of the specimen imaged by the imaging unit 210.

  The demosaicing processing unit 213 performs processing for generating image data of each color of RGB from RAW data in the Bayer array. The demosaicing processing unit 213 calculates the value of each RGB color of the target pixel by interpolating the values of peripheral pixels (including pixels of the same color and other colors) in the RAW data. In addition, the demosaicing processing unit 213 also performs a defective pixel correction process (interpolation process). Note that when the imaging sensor 208 does not have a color filter and a single color image is obtained, the demosaicing process is not necessary.

The image composition processing unit 214 performs processing for generating large-capacity image data in a desired imaging range by connecting image data acquired by dividing the imaging range by the imaging sensor 208. In general, since the existence range of the specimen is wider than the imaging range that can be acquired by one imaging with an existing image sensor, one piece of two-dimensional image data is generated by joining the divided image data. For example, assuming that a 10 mm square area on the slide 206 is imaged with a resolution of 0.25 μm, the number of pixels on one side is 40,000 pixels of 10 mm / 0.25 μm, and the total number of pixels is 1.6 billion, which is the square of the number of pixels. It becomes a pixel. In order to acquire image data of 1.6 billion pixels using the image sensor 208 having 10M (10 million) pixels, it is necessary to divide the area into 160 areas of 1.6 billion / 10,000,000 and perform imaging. In addition, as a method of joining a plurality of image data, a method of joining by aligning based on position information of the stage 202, a method of joining corresponding points or lines of a plurality of divided images, and divided image data There is a method of connecting based on the position information of the. At the time of joining, it can be smoothly connected by interpolation processing such as zero-order interpolation, linear interpolation, and high-order interpolation. In the present embodiment, it is assumed that a single large-capacity image is generated. However, as a function of the image processing apparatus 102, a configuration may be adopted in which images acquired in a divided manner are connected when display data is generated.

  The resolution conversion processing unit 215 performs processing for generating a magnification image corresponding to the display magnification in advance by resolution conversion in order to display the large-capacity two-dimensional image generated by the image composition processing unit 214 at high speed. A plurality of stages of image data from low magnification to high magnification are generated and configured as image data having a hierarchical structure. Details will be described later with reference to FIG.

  The filter processing unit 216 is a digital filter that realizes suppression of high-frequency components contained in an image, noise removal, and resolution enhancement. The γ correction unit 217 executes processing for adding an inverse characteristic to an image in accordance with the gradation expression characteristic of a general display device, or adjusts to the human visual characteristic by gradation compression or dark part processing of a high luminance part. Or perform tone conversion. In the present embodiment, in order to acquire an image for the purpose of morphological observation, gradation conversion suitable for the subsequent synthesis processing and display processing is applied to the image data.

  The compression processing unit 218 is a compression encoding process performed for the purpose of improving the efficiency of transmission of large-capacity two-dimensional image data and reducing the storage capacity. As a still image compression technique, standardized encoding methods such as JPEG (Joint Photographic Experts Group) and JPEG 2000 and JPEG XR, which are improved and evolved from JPEG, are widely known.

  The pre-measurement unit 220 is a unit that performs pre-measurement for calculating the position information of the specimen on the slide 206, the distance information to the desired focal position, and the parameter for adjusting the amount of light caused by the specimen thickness. By acquiring information by the pre-measurement unit 220 before the main measurement, it is possible to perform image pickup without waste. A two-dimensional image sensor having a lower resolving power than the image sensor 208 is used to acquire position information on the two-dimensional plane. The pre-measurement unit 220 acquires the position of the sample on the XY plane from the acquired image. For obtaining the distance information and the thickness information, a laser displacement meter or a Shack-Hartmann measuring instrument is used.

  The main control system 221 is a function for controlling the various units described so far. The control functions of the main control system 221 and the development processing unit 219 are realized by a control circuit having a CPU, a ROM, and a RAM. That is, the program and data are stored in the ROM, and the functions of the main control system 221 and the development processing unit 219 are realized by the CPU executing the program using the RAM as a work memory. For example, a device such as an EEPROM or a flash memory is used as the ROM, and a DRAM device such as DDR3 is used as the RAM. Note that the function of the development processing unit 219 may be replaced with an ASIC implemented as a dedicated hardware device.

  The data output unit 222 is an interface for sending the RGB color image generated by the development processing unit 219 to the image processing apparatus 102. The imaging apparatus 101 and the image processing apparatus 102 are connected by an optical communication cable. Alternatively, a general-purpose interface such as USB or Gigabit Ethernet (registered trademark) is used.

(Functional configuration of image processing apparatus)
FIG. 3 is a block diagram showing a functional configuration of the image processing apparatus 102 of the present invention.
The image processing apparatus 102 includes an outline, an image data acquisition unit 301, a memory 302, a user input information acquisition unit 303, a display device information acquisition unit 304, a link information generation unit 305, a link information table 306, a display data generation control unit 307, an annotation The data generation unit 308, the hierarchical image data acquisition unit 309, the display data generation unit 310, and the display data output unit 311 are configured.

  The image data acquisition unit 301 acquires image data captured by the imaging device 101.

  Image data acquired from an external device is taken into the memory 302 via the image data acquisition unit 301, stored, and held. As the image data stored and held in the memory 302, there is one piece of two-dimensional image data obtained by combining RGB color division image data obtained by dividing and imaging a specimen. In addition, there is image data configured hierarchically by a plurality of image data (hierarchical image data) having different magnifications and a plurality of image data (depth image data) having different focal positions.

  The user input information acquisition unit 303 acquires information on a virtual slide image display state change instruction and added annotation information input by the user via an operation unit such as a mouse or a keyboard. Examples of the instruction to change the display state of the virtual slide image include scrolling (changing the display position), scaling (changing the display magnification), rotation (changing the display angle), and the like. The annotation information is information on the position of the region of interest (attention region) designated by the user and comments (annotations).

  The display device information acquisition unit 304 acquires information on the size of the display area (screen resolution, number of pixels), information on the display magnification of the currently displayed virtual slide image, and the like from the display device 103.

  The link information generation unit 305 generates link information from the annotation position information obtained from the user input information acquisition unit 303 and the virtual slide image display magnification at the time of annotation addition obtained from the display device information acquisition unit 304. The link information is information that associates the position information of the annotation with the converted position information obtained by converting the position information of the annotation into the position in each of a plurality of layer image data constituting the image data, and the magnification of each layer image data. Generated for each annotation added to. Details will be described later with reference to FIG.

  The link information table 306 is a table that stores the link information generated by the link information generation unit 305.

  The display data generation control unit 307 controls the generation of display data based on the instruction to change the display state of the virtual slide image input by the user and the annotation information in accordance with the instruction from the user input information acquisition unit 303. The display data mainly consists of virtual slide image data and annotation display data. The display data generation control unit 307 instructs the hierarchical image data acquisition unit 309 to generate virtual slide image data, and instructs the annotation data generation unit 308 to generate annotation display data.

  The annotation data generation unit 308 generates annotation display data based on the annotation information according to the control of the display data generation control unit 307.

  The hierarchical image data acquisition unit 309 acquires hierarchical image data necessary for displaying the virtual slide image from the memory 302 according to the control of the display data generation control unit 307.

The display data generation unit 310 generates display data to be displayed on the display device 103 from the annotation display data generated by the annotation data generation unit 308 and the hierarchical image data acquired by the hierarchical image data acquisition unit 309. The display data generation unit 310 generates a virtual slide image from the hierarchical image data in accordance with a display state change instruction from the user, and generates display data by superimposing the virtual slide image on the annotation display data.

  The display data output unit 311 outputs the display data generated by the display data generation unit 310 to the display device 103 which is an external device.

(Hardware configuration of image forming apparatus)
FIG. 4 is a block diagram illustrating a hardware configuration of the image processing apparatus according to the present exemplary embodiment. For example, a PC (Personal Computer) is used as the image processing apparatus.
The image processing apparatus includes a CPU (Central Processing Unit) 401 and a RAM (Random Access).
Memory) 402, a storage device 403, a data input / output I / F 405, and an internal bus 404 for connecting them together.

The CPU 401 appropriately accesses the RAM 402 or the like as necessary, and performs overall control of the entire blocks of the PC while performing various arithmetic processes. The RAM 402 is used as a work area for the CPU 401, and temporarily holds the OS, various programs being executed, and various data to be processed such as display data generation, which is a feature of the present invention. The storage device 403 is an auxiliary storage device that can record and read information, and stores firmware such as an OS, a program, and various parameters that are executed by the CPU 401 in a fixed manner. As the storage device 403, a magnetic disk drive such as an HDD (Hard Disk Drive) or SSD (Solid State Disk) or a semiconductor device using a flash memory is used.

  An image server 1101 is connected to the data input / output I / F 405 via the LAN I / F 406, the display device 103 is connected via the graphics board 407, and the imaging device 101 is connected via the external device I / F 408. Is done. The imaging device 101 is a virtual slide device or a digital microscope. In addition, a keyboard 410 and a mouse 411 are connected to the data input / output I / F 405 via an operation I / F 409.

  The display device 103 is a display device using, for example, liquid crystal, EL (Electro-Luminescence), CRT (Cathode Ray Tube), or the like. In the present embodiment, the configuration in which the display device 103 is connected as an external device to the image processing device is illustrated, but the image processing device of the present invention may be configured as a notebook PC integrated with the display device. good.

  Although the keyboard 410 and the mouse 411 are illustrated as connection devices with the operation I / F 409, a configuration in which other input devices such as a touch panel are connected may be used. When a touch panel is used as the input device, the display device 103 connected to the graphics board 407 and the input device connected to the operation I / F 409 are integrated.

(Concept of hierarchical images prepared for each magnification)
FIG. 5 is a conceptual diagram of image data composed of a plurality of hierarchical image data with different magnifications. Here, hierarchical image data generated by the resolution conversion processing unit 215 of the imaging apparatus 101 illustrated in FIG. 2 will be described.

  Hierarchical image data 501, 502, 503 and 504 are two-dimensional image data prepared in accordance with the display magnification and having different resolutions in stages. Here, the relationship of the one-dimensional resolution (number of pixels) between hierarchical image data of different magnifications is as follows: the hierarchical image 503 is 1/2 of the hierarchical image 504, the hierarchical image 502 is 1/2 of the hierarchical image 503, and the hierarchical Assume that the image 501 is ½ of the hierarchical image 502. What magnification hierarchical image data is prepared is arbitrary, and is not limited to the example of FIG.

  The imaging data acquired by the imaging device 101 is high-resolution image data consisting of billions of pixels, and resolution conversion processing such as enlargement / reduction in response to a request to change the display magnification of the virtual slide image was performed each time. Then, it is thought that processing is not in time. Therefore, a plurality of image data with different magnifications are generated as hierarchical image data from the high-resolution imaging data in advance. Then, when there is a request to change the display magnification, hierarchical image data having a magnification close to the requested display magnification is selected from the plurality of hierarchical image data, and the hierarchical image data is adjusted to the requested display magnification. Display data of the virtual slide image is generated by performing resolution conversion. It is preferable in terms of image quality to generate display data from higher-magnification image data.

  Each hierarchical image data is generated by reducing high-resolution imaging data by resolution conversion. As a resolution conversion method, a bilinear method which is a two-dimensional linear interpolation process, a bicubic method using a cubic interpolation formula, or the like can be used.

  Each hierarchical image data has a two-dimensional axis of X and Y axes. In FIG. 5, a P-axis described as an axis in a direction orthogonal to the XY axis represents a magnification.

  In FIG. 5, one piece of hierarchical image data 502 includes a plurality of divided image data 505. As described above, high-resolution two-dimensional image data is generated by joining image data obtained by divided imaging. Here, the divided image data 505 is image data obtained by imaging a range that can be captured at once by the imaging sensor 208. Note that the size of the divided image data 505 is not limited to this, and may be a size obtained by arbitrarily dividing image data obtained by imaging the range that can be imaged at one time by the imaging sensor 208, or an arbitrary number of the image data. It may be a combined size.

  Image data for pathological diagnosis that is assumed to be observed at various display magnifications by enlargement / reduction as described above is image data having a hierarchical structure composed of hierarchical image data having a plurality of different magnifications as shown in FIG. It is desirable to generate and hold as As a format of such image data, there is a format in which a plurality of hierarchical image data can be collectively handled as one image data. Alternatively, each hierarchical image data may be independent image data, and information indicating the relationship between each hierarchical image data and the display magnification may be held separately. Here, the following description will be made assuming a single image data composed of a plurality of hierarchical image data.

(Annotation addition and presentation method)
The flow of annotation addition and presentation in the image processing apparatus of the present invention will be described using the flowchart of FIG.

  In step S <b> 601, the display device information acquisition unit 304 acquires information on the size (screen resolution, number of pixels) of the display area of the display device 103 and information on the display magnification of the currently displayed virtual slide image. The information on the size of the display area is used when determining the size (number of pixels) of display data generated by the display data generation unit 310. The display magnification information is used when the hierarchical image data acquisition unit 309 selects hierarchical image data from the memory 302, and is used when the link information generation unit 305 generates link information of the annotation. The generation of link information will be described later.

  In step S <b> 602, the hierarchical image data acquisition unit 309 acquires hierarchical image data corresponding to the display magnification of the virtual slide image currently displayed on the display device 103 from the memory 302. Hierarchical image data corresponding to a specified magnification may be acquired.

In step S <b> 603, the display data generation unit 310 generates display data to be output to the display device 103 based on the hierarchical image data acquired by the hierarchical image data acquisition unit 309.
If the display magnification of the virtual slide image designated by the user is different from the magnification of the acquired hierarchical image data, resolution conversion processing is performed. The generated display data is output to the display device 103, and an image based on the display data is displayed on the display device 103.

  In step S604, the display data generation control unit 307 determines whether an instruction to change the display state of the virtual slide image has been input by the user based on the information acquired from the user input information acquisition unit 303. Specifically, it is an instruction to move the display position (scroll) or change the display magnification. The instruction to move the display position is an instruction in particular that the display range of the virtual slide image after the movement of the display position goes outside the display range of the current virtual slide image. When a display state change instruction is input and the virtual slide image needs to be updated, the display data generation control unit 307 returns to step S602. Thereafter, the process of updating the virtual slide image by acquiring the hierarchical image data and generating the display data is repeated. When the display state change instruction is not input, the display data generation control unit 307 proceeds to step S605.

  In step S <b> 605, the display data generation control unit 307 determines whether an instruction to add an annotation has been input by the user based on information acquired from the user input information acquisition unit 303. When an instruction to add an annotation is input, the display data generation control unit 307 proceeds to step S606. If no instruction for adding an annotation has been input, the display data generation control unit 307 proceeds to step S607.

  In step S606, various processes for adding annotations to the image data are performed. Processing contents include acquisition of annotation information (annotation content and position information input by an input device such as the keyboard 410) by the user input information acquisition unit 303, and generation of link information by the link information generation unit 305. Details will be described later with reference to FIG.

  In step S607, the display data generation control unit 307 determines whether there is a request for presenting the added annotation. When an annotation presentation request is input, the display data generation control unit 307 proceeds to step S608. When the annotation presentation request is not input, the display data generation control unit 307 returns to step S604 and repeats the above process. Although the description has been given in time series for convenience of description of the flow, the execution timing of the display position and display magnification change request reception, annotation addition, and annotation presentation is not limited to the above example, and may be simultaneous. The sequential order may be different from the order described above.

  In step S608, the display data generation control unit 307 receives an annotation presentation request and performs an annotation presentation process. Details will be described later with reference to FIG.

(Add annotation)
FIG. 7 is a flowchart showing a detailed flow of processing for adding an annotation shown in step S606 of FIG. FIG. 7 illustrates a flow of generating link information based on the position information of the added annotation and the display magnification of the virtual slide image when the annotation is added.

  In step S701, the display data generation control unit 307 acquires position information of the added annotation. The display data generation control unit 307 acquires the absolute position information (coordinates) of the annotation by performing processing for converting the relative annotation position in the virtual slide image being displayed into the position relative to the entire area of the image data. To do.

In step S <b> 702, the display data generation control unit 307 acquires information on the content of the annotation input using the keyboard 410 or the like. The acquired annotation content information is used when the annotation is presented.

  In step S <b> 703, the display data generation control unit 307 acquires information on the display magnification of the virtual slide image displayed on the display device 103. This display magnification is the display magnification of the virtual slide image when the annotation is added. In the present embodiment, the configuration in which the display data generation control unit 307 acquires display magnification information from the display device 103 is exemplified. However, since the image processing device 102 generates display data, the image processing device 102 The configuration may be such that the information of the display magnification held inside is acquired.

  In step S704, the link information generation unit 305 generates link information based on the annotation position information acquired in step S701 and the display magnification information at the time of annotation addition acquired in step 703. By referring to the link information, it becomes possible to know the position (coordinates) of the annotation in the hierarchical image data at a magnification different from the magnification at the time of adding the annotation, so that the annotation information added in S701 can be used in any hierarchical image data. It becomes like this. For example, it is assumed that an annotation is added at the position of coordinates (100, 100) in a virtual slide image with a display magnification of 20 times. The position of the coordinates (100, 100) is a position where the distance (number of pixels) in the X direction and the Y direction from the origin (0, 0) of the entire area of the virtual slide image is 100 pixels. The annotation position is P1 (200, 200) for a high-magnification image with a display magnification of 40 times, and the coordinate is P2 (50, 50) for a low-magnification image with a display magnification of 10 times. The coordinate of the annotation position in the hierarchical image data of an arbitrary display magnification is obtained by multiplying the ratio between the display magnification and the display magnification at the time of annotation addition acquired at S703 by the coordinate at the time of annotation addition acquired at S701. It is calculated by.

  In step S705, it is determined whether or not annotation has been added for the first time since the start of observation of the virtual slide image. If the annotation is added for the first time, the process proceeds to step S707. If the annotation has been added even once in the past, the process proceeds to step S706.

  In step S706, the link information stored in the link information table is updated using the link information generated in step S704. The link information table will be described later. The value of the table for storing the link information generated when the annotation addition described in step S707 is first performed is updated.

  In step S707, a link information table is generated. The link information table stores the link information generated in step S704. The link information is information on the correspondence between the position information of the added annotation, the position information obtained by converting the position information for hierarchical image data of a plurality of different magnifications, and the display magnification when the annotation is added. It is. Here, it is assumed that text information indicating the content of the annotation is also stored in the link information. The link information is calculated based on the display magnification when the annotation is added, the position in the image, and the display magnification corresponding to the resolution of each hierarchical image. This is information in which the position is associated with the annotation information.

(Presentation of annotation)
FIG. 8 is a flowchart showing a detailed flow of processing for presenting an annotation. FIG. 8 illustrates a flow of generating display data for presenting an annotation based on link information.

  In step S801, the display data generation control unit 307 determines whether there is a request for changing the display state of the virtual slide image (movement of the display position, change of the display magnification) from the user. In general, screening is performed at a display magnification of about 5 to 10 times, and detailed observation is performed at a display magnification of 20 or 40 times. As described above, the display magnification of the virtual slide image when an annotation is added varies depending on the annotation. Therefore, the display magnification suitable for confirming the positions of the plurality of annotations added to the image data depends on the plurality of annotations added to the image data. In step S801, the user can request that the display state of the virtual slide image be changed to a display state suitable for presentation of a plurality of annotations added to the image data. If there is a display state change request, the display data generation control unit 307 proceeds to step S802. If there is no display state change request, the display data generation control unit 307 proceeds to step S803.

  In step S <b> 802, the display data generation control unit 307 selects appropriate hierarchical image data so that the display state of the virtual slide image suitable for the presentation of the annotation is obtained in response to the display state change request. When a plurality of annotations are added to the image data, the display data generation control unit 307 includes all the positions of the plurality of annotations so that the positions of all the added annotations can be displayed in the virtual slide image. Determine the display range. Then, the display data generation control unit 307 selects appropriate hierarchical image data according to the determined display range. For example, when annotation positions are distributed over a wide range and a virtual slide image with a display magnification of 40 times cannot cover all annotation positions, a hierarchical image with a magnification of 20 times is used to generate display data for a virtual slide image with a display magnification of 20 times. Data is selected.

  In step S803, the display data generation control unit 307 determines whether an instruction to change the annotation display method has been input. The annotation display method is a setting of text decoration, frame image color, transparency with respect to a background image, etc. when an annotation is presented. For example, if the display magnification of the virtual slide image at the time of annotation presentation is different from the display magnification of the virtual slide image at the time of annotation addition, the text color, font, frame image color, etc. A display mode can be set. Details will be described later. When an instruction to change the annotation display method is input, the display data generation control unit 307 proceeds to step S804, and otherwise, the display data generation control unit 307 proceeds to step S805.

  In step S804, the display data generation control unit 307 changes the annotation display method in response to the input annotation display method change request.

  In step S <b> 805, the display data generation control unit 307 adopts an initial setting set in advance as a setting for the annotation display method in response to a request for changing the annotation display method not being input.

In step S806, the display data generation control unit 307 determines whether the number of annotations to be presented is too large for the size of the display area of the virtual slide image. The display data generation control unit 307 calculates the ratio of the size of the annotation display area to the size of the display area of the virtual slide image when all annotations are displayed on the virtual slide image according to the display method determined in step S804 or S805. . When this ratio is equal to or greater than the threshold value, the display data generation control unit 307 determines that the number of annotations is too large. When the number of annotations is too large, if all annotations are displayed, the virtual slide image that is the background of the annotations is covered with the annotations, which makes it difficult to observe. The user can arbitrarily set the threshold value used for the determination according to the degree of coverage that does not hinder the observation of the virtual slide image. If it is determined that the number of annotations is too large, the display data generation control unit 307 presents annotations in the pointer display mode. The pointer display mode is a mode in which only the position information of the annotation is displayed on the virtual slide image using an icon or the like, and the text or the frame image indicating the content of the annotation is not displayed. In the pointer display mode, for example, text information indicating the content of the annotation is displayed only for the specific annotation selected by the user. When it is not determined that the number of annotations is too large, the display data generation control unit 307 presents annotations in the annotation display mode. The annotation display mode is a mode in which position information and content information of an annotation are always displayed using icons, text, frame images, etc. for all annotations. Note that the user may set whether to switch between the pointer display mode and the annotation display mode according to the number of annotations.

  In step S807, the annotation data generation unit 308 generates annotation display data for presenting the annotation in the pointer display mode. A display example of the virtual slide image in which the annotation is presented in the pointer display mode will be described later with reference to FIG.

  In step S808, the annotation data generation unit 308 generates annotation display data for presenting the annotation in the annotation display mode. A display example of the virtual slide image on which the annotation is presented in the annotation display mode will be described later with reference to FIG.

  In step S809, the display data generation unit 310 generates display data for the virtual slide image based on the hierarchical image data selected in step S802 and the annotation display data generated in step S807 or step S808.

  In step S810, the display data output unit 311 outputs the display data generated in step 809 to the display device 103.

  In step S811, the display device 103 displays an image based on the display data output from the display data output unit 311 on the screen.

  In step S812, the display data generation control unit 307 determines whether the annotation presentation mode is the pointer display mode. In the case of the pointer display mode, the display data generation control unit 307 proceeds to step S813. In the annotation display mode, the display data generation control unit 307 proceeds to step S815.

  In step S813, the display data generation control unit 307 has selected whether the user has selected the pointer indicating the annotation position displayed on the virtual slide image or holding the mouse cursor over the pointer with the mouse or the keyboard. Judge whether or not. When the pointer indicating the annotation position is selected or the mouse cursor is held over the pointer, the display data generation control unit 307 proceeds to step S814. In other cases, the display data generation control unit 307 performs annotation presentation processing. finish.

In step S814, the display data generation control unit 307 generates annotation display data such that the text content of the annotation added to the position of the selected pointer is displayed in a pop-up. In the pointer display mode, when the selection of the pointer is canceled or the mouse cursor is removed from the pointer, the display data generation control unit 307 generates annotation display data in which the pop-up display of the annotation content is deleted. Note that once the pointer is selected, the annotation content display may be continued until an instruction to turn off the annotation display is input.

  In step S815, the display data generation control unit 307 determines whether the user has performed an operation of selecting the annotation displayed on the virtual slide image with the mouse or the keyboard. When an annotation is selected, in the following processing, the display data generation control unit 307 generates display data that reproduces the display magnification and display position of the virtual slide image when the selected annotation is added. . If an annotation is selected on the virtual slide image, the display data generation control unit 307 proceeds to step S816, and if not selected, ends the annotation presentation process.

  In step S816, the hierarchical image data acquisition unit 309 selects hierarchical image data based on the position information and display magnification at the time of annotation addition stored in the link information.

  In step S817, the display data generation unit 310 generates display data using the annotation display data generated by the annotation data generation unit 308 corresponding to the annotation selected in step S815 and the hierarchical image data selected in step S816. .

  Output of the display data in step S818 and display of the image based on the display data in step S819 by the display device 103 are the same as in steps S810 and S811, respectively.

(Display screen layout)
FIG. 9 is an example when display data generated by the image processing apparatus 102 is displayed on the display apparatus 103. FIG. 9 describes the determination of the annotation display method, the difference between the pointer display mode and the annotation display mode, and the reproduction of the display position and display magnification when the annotation is added.

  FIG. 9A shows a basic configuration (layout) of a virtual slide image viewer window displayed on the display device 103. The window of the viewer includes an information area 902 indicating the status of display and operation and various types of image information, and a thumbnail image 903 that roughly displays the entire specimen to be observed. Also, in the entire window 901, a frame 904 indicating the display range of the virtual slide image in the thumbnail image, a display area 905 of the virtual slide image, and a display 906 indicating information on the display magnification of the virtual slide image displayed in the display area 905 are displayed. is there. The window configuration of the viewer may be a single document interface in which windows for displaying various images and information are arranged in the entire window 901, or a multi-document interface in which windows for displaying various images and information are independent. The thumbnail image 903 displays a frame 904 that displays the position and size of an area corresponding to the display range of the virtual slide image displayed in the display area 905 in the entire image of the specimen. The position and size of the frame 904 can be changed by a user instruction via an input device such as a mouse or a keyboard. Further, the position and size of the frame 904 are changed in conjunction with the user's operation for instructing to change the display range (moving the display position or changing the display magnification) for the virtual slide image displayed in the display area 905. The A virtual slide image is displayed in the display area 905, and the user performs image diagnosis and adds annotations while observing the virtual slide image. The user inputs instructions for changing the display position (moving the display range) and changing the display magnification (enlarging / reducing) by operating the mouse or keyboard so that a virtual slide image suitable for observation is displayed. Thus, the display state of the virtual slide image can be changed.

  FIG. 9B is an example of a screen display for performing an operation for adding an annotation. In the example of FIG. 9B, the display magnification 906 is set to 20 times. A user designates a region of interest (region of interest) from the virtual slide image in the display region 905 and inputs annotation information, thereby adding an annotation. The flow of operations and processes for adding annotations is roughly as follows. This will be described with reference to FIG. First, when the user operates the mouse or the like to specify the position 907 where the annotation is to be added, the mode shifts to a mode for inputting the annotation content (text), where the user operates the keyboard or the like to enter the annotation content (text). Enter 908. At this time, the image processing apparatus 102 acquires the position information to which the annotation is added and the display magnification information (20 times in the case of FIG. 9B) of the virtual slide image to which the annotation has been added as a set. .

  FIG. 9C shows an example of an annotation display method setting screen. The annotation display method setting screen 909 may be configured to be displayed when an annotation is added, or may be configured to be called and displayed from a menu in advance or at an appropriate time. Here, when the user performs an annotation addition operation, the annotation display method setting screen 909 is displayed for the first time in the information area 902 of FIG. The annotation display method is an appearance feature when an annotation is presented. In the present embodiment, the annotation display method according to the magnification of the virtual slide when the annotation is added and the difference between the magnification of the virtual slide image between the magnification at the time of annotation addition and the magnification at the time of annotation presentation Can be different. In this embodiment, as an annotation display method, an example is shown in which three types of items are set: an annotation content (text) display method, an annotation frame display method, and an entire annotation display method. However, the present invention is not limited to this. Absent. The setting items of the annotation content (text) display method include text color, brightness, font type, and modification (bold, italic). The setting of the display method of the annotation frame includes the frame color, line type (solid line, broken line), shape (speech balloon, rectangle, etc.), background color, and the like. The setting of the display method of the entire annotation includes the transmittance when performing alpha blending on the virtual slide image as the background image, the blinking speed when blinking, and the like.

  The GUI component 910 is a check box for the user to select a favorite display method from among a plurality of display methods. The GUI component 911 indicates the setting item name of the annotation display method. The GUI component 912 is a button for displaying a color setting window 913 in which a plurality of color patches 914 and a selected display color 915 are displayed, and a slider for changing brightness and transmittance values. For example, when the brightness of the text can be set with an 8-bit value, the brightness value can be set from 0 to 255 according to the slide position. Although not shown, a GUI component capable of directly inputting a numerical value may be displayed for setting a luminance value.

FIG. 9D is a screen display example when an annotation is presented in the annotation display mode. In the annotation display mode, each annotation is presented using an icon 917 indicating the position information of the annotation, and an image 916 including text indicating the content information of the annotation, a balloon, and a frame. FIG. 9D shows an example in which three annotations are presented. When the position where the annotation is added covers a wide range, the display magnification is changed so that the positions of all the annotations can be displayed. The display magnification may be changed automatically based on the annotation position information, or may be manually adjusted by the user to adjust the display range and display magnification. In the example of FIG. 9D, the display magnification is five times. Each annotation has a different display magnification of the virtual slide image when added. For example, it is assumed that annotation 1 is added in a virtual slide image having a display magnification of 10 times, annotation 2 is 20 times, and annotation 3 is 40 times. In the present embodiment, as shown in FIG. 9D, the style of the balloon and the frame of each annotation is changed according to the magnification of the virtual slide image when the annotation is added. Therefore, the user can determine that the display magnifications when the annotations are added are different.

  FIG. 9E is a screen display example when an annotation is presented in the pointer display mode. In the pointer display mode, each annotation is presented by an icon 918 indicating annotation position information. When any of the icons 918 indicating the position information is selected or when the mouse is over, the content of the annotation corresponding to the icon is displayed in a pop-up 919 as shown in FIG. FIG. 9E shows an example in which seven annotations are presented. Each annotation is assumed to have a mixture of the same and different display magnifications of the virtual slide image at the time of addition. In the present embodiment, as shown in FIG. 9 (e), the mode of the icon 918 indicating the position information of each annotation is varied according to the magnification of the virtual slide image when the annotation is added. Therefore, the user can discriminate whether the display magnification when the annotation is added is different or the same depending on the difference in the icon 918 indicating the position information of the annotation. In addition, the pop-up 919 displayed when the icon 918 indicating the position information is selected or when the mouse is moved over is also the display of the virtual slide image when the annotation is added, as in FIG. 9D. It depends on the magnification. Thereby, the user can easily select a desired annotation from many annotations.

  FIG. 9F is a screen display example that reproduces the position and display magnification of the annotation in the virtual slide image when the annotation is added. When any annotation is selected by the user in the annotation display mode or the pointer display mode, the display data generation control unit 307 performs the following processing. That is, the display data generation control unit 307 generates display data so as to reproduce the magnification of the virtual slide image and the position of the annotation in the image when the annotation is added with reference to the link information. At this time, in the thumbnail image display area 903, the position information of all the annotations displayed in FIG. 9D or 9E is displayed in a frame 921 indicating a displayable range and the virtual slide image being displayed. A frame 922 indicating the range being displayed is displayed.

(Effect of Example)
In this embodiment, when an annotation is added to a virtual slide image, link information is generated based on the annotation position information and the virtual slide image magnification information. The link information is obtained for each added annotation, and each of a plurality of hierarchical image data having different magnifications constituting the image data, and converted position information obtained by converting the position of the annotation into a position in each hierarchical image data. This is information indicating a correspondence relationship. When presenting a plurality of annotations added to the image data, the display mode of each annotation is changed according to the display magnification of the virtual slide image when the annotation is added. Thereby, the user can easily grasp the difference in magnification of the virtual slide image when added for each annotation.

[Second Embodiment]
An image processing system according to a second embodiment of the present invention will be described with reference to the drawings.
In the second embodiment, a plurality of annotations added to image data composed of a plurality of hierarchical image data having different focal positions are made easier for the user to grasp the difference in the focal position of the virtual slide image when added. The example to present is shown. Hereinafter, the configuration different from the first embodiment will be mainly described, and the same reference numerals and names as those of the first embodiment are used for the same configuration as the first embodiment, and detailed description thereof will be omitted.

(Device configuration of image processing system)
FIG. 10 is an overall view of an apparatus constituting an image processing system according to the second embodiment of the present invention.
In FIG. 10, the image processing system using the image processing apparatus of the present invention includes an image server 1101, an image processing apparatus 102, and a display apparatus 103. The image processing apparatus 102 can acquire image data obtained by imaging the specimen from the image server 1101 and generate display data for display on the display apparatus 103. The image data here includes high-resolution two-dimensional image data obtained by synthesizing the imaging data obtained by the divided imaging described in the first embodiment, and a plurality of hierarchical image data at different magnifications for high-speed display. And a plurality of depth image data imaged at different focal positions. Details of the depth image will be described later with reference to FIG. The image server 1101 and the image processing apparatus 102 are connected by a general-purpose I / F LAN cable 1003 via a network 1002. The image server 1101 is a computer including a large-capacity storage device that stores image data captured by an imaging device (not shown; equivalent to the imaging device 101 of the first embodiment) that is a virtual slide device. The image server 1101 may store image data (depth image data) at different focal positions that are hierarchized in a local storage connected to the image server 1101 as one unit. Alternatively, it is possible to divide each and store the entity of each depth image data and reference information to the entity separately for a server group (cloud server) existing somewhere on the network. The depth image data itself does not need to be stored in one server, and may be stored in a distributed manner. The image processing apparatus 102 and the display apparatus 103 are the same as the image processing system of the first embodiment.

  In the example of FIG. 10, the image processing system is configured by three devices, the image server 1101, the image processing device 102, and the display device 103, but the present invention is not limited to this configuration. For example, an image processing device integrated with a display device may be used, or some of the functions of the image processing device 102 may be incorporated in the image server 1101. Conversely, the functions of the image server 1101 and the image processing apparatus 102 may be divided and realized by a plurality of apparatuses.

(Concept of hierarchical image prepared in advance for each focal position)
FIG. 11 is a conceptual diagram of image data including depth image data at a plurality of different focal positions. By imaging the stage 202 of the imaging apparatus 101 in the depth direction (Z direction in FIG. 2, direction perpendicular to the stage, optical axis direction) a plurality of times, a plurality of imaging data with different focal positions can be obtained.

FIG. 11A is a conceptual diagram of image data having a hierarchical structure in which two-dimensional planar image data captured at different focal positions are stacked in the depth direction.
A two-dimensional image 1102 imaged at an arbitrary focal plane of a specimen to be observed is referred to as depth image data. The image data group 1100 is configured by stacking a plurality of depth image data 1102 in the depth direction (Z-axis direction) perpendicular to the two-dimensional plane (XY plane). FIG. 11A shows an example in which image data is composed of depth image data at ten different focal positions.

  One piece of depth image data 1102 includes a plurality of pieces of divided image data 1103. As described above, the generation of large-capacity high-resolution image data is generated by synthesizing a plurality of pieces of divided image data. The divided image data 1103 may be image data having the same size as the image data that has been divided and picked up, or may be a set of image data that has been picked up and divided, or image data that further divides the divided image data. It may be. That is, the method of dividing the depth image data 1102 is arbitrary, and the unit of division may be the same as or different from the unit of divided imaging.

  The depth image data at each focal position has two axes X and Y constituting a two-dimensional plane. Further, the depth image data at different focal positions in the Z-axis direction (depth direction) perpendicular to the XY axis is a hierarchical data format.

  The imaging optical system of the virtual slide apparatus has a large NA (numerical aperture) for high resolution and generally has a shallow depth of field. The thickness of the specimen to be observed is about 3 to 5 μm for histological examination and about 100 μm for cytological examination, but the depth of field is much narrower and about 1 μm. Therefore, it is difficult to generate a focused image on the entire specimen surface. In addition, even if the specimen is thin, structures such as cell nuclei may be included inside, so that detailed observation of the specimen is desired while changing the focal position. Acquisition and generation of image data composed of a plurality of depth image data is based on such needs.

  Also, a plurality of hierarchical image data with different magnifications (resolutions) generated for the purpose of speeding up the display described in the first embodiment, and depth image data of a plurality of different focal positions described in the present embodiment. Can be combined to generate hierarchical image data. The configuration of such image data will be described with reference to FIG.

  In FIG. 11B, each of the depth image data groups 1104, 1105, and 1106 is a set of a plurality of image data having the same magnification and different focal positions. That is, image data belonging to the same depth image data group is image data having the same magnification and different focal positions, and image data belonging to different depth image data groups have different magnifications.

  The display data of the virtual slide image is generated from the image data of the magnification and the focal position selected according to the request from these hierarchical image data.

  Image data for pathological diagnosis assuming that observation is performed with different focal positions is generated and held as image data having a hierarchical structure composed of depth image data of a plurality of different focal positions as shown in FIG. It is desirable to do. As a format of such image data, there is a format in which a plurality of depth image data can be collectively handled as one image data. Alternatively, each depth image data may be independent image data, and information indicating the relationship between each depth image data and the focal position may be separately held. Here, the following description will be made assuming a single image data composed of a plurality of depth image data.

(Add annotation)
FIG. 12 is a flowchart showing a flow of processing for adding an annotation. FIG. 12 illustrates a flow of generating link information based on the position information of the added annotation and the focus position information of the virtual slide image when the annotation is added.

  Steps S701 to S703 are the same as the annotation adding process described in the first embodiment with reference to FIG. In FIG. 12, the display magnification information is acquired in step S703. However, since it is not an essential item in the configuration of the present embodiment, it may be skipped.

  In step S1201, information on the focal position of the virtual slide image when the annotation is added is acquired. The focal position information is information indicating the number of depth image data among the plurality of layered depth image data described with reference to FIG. The focal position information may be obtained from the display device 103 as in the first embodiment, or may be obtained from information relating to display data generation held internally in the image processing device 102.

  In step S1202, the link information generation unit 305 generates link information based on the focus position information acquired in step S1201 and the position information of the added annotation acquired in step S701. The link information is information associating the converted position information obtained by converting the position information of the annotation into the position in each of the plurality of depth image data constituting the image data, and the focal position of each depth image data. Generated for each annotation added to. Link information is calculated based on the focus position when the annotation is added and the position in the image, and associates the position when the annotation is displayed over each depth image and the annotation information. Information. If the display magnification information is acquired in step S703, the converted position information obtained by converting the position information of the added annotation into the position in each of the plurality of layer image data, and each layer described in the first embodiment. Link information that associates the magnification of the image data is also generated.

  In step S705, it is determined whether or not annotation has been added since the observation of the virtual slide image was started. Since this processing content is also the same as that of the first embodiment, description thereof is omitted.

  In step S1203, the link information stored in the link information table is updated using the link information generated in step S1202.

  In step S1204, a link information table is generated. The link information table stores the link information generated in step S1202. The link information is the correspondence between the position information of the added annotation, the position information obtained by converting the position information for depth image data of a plurality of different focal positions, and the focal position when the annotation is added. Information. If the link information of the correspondence relationship between the position information and the magnification is generated in step S1202, the correspondence information between the position information to which the annotation converted for each layer image data is added and the magnification of the layer image data is displayed. You may store together.

(Presentation of annotation)
FIG. 13 is a flowchart showing a flow of processing for presenting an annotation. In FIG. 13, a flow of generating display data for presenting an annotation based on link information will be described.

  The flow of the initial annotation presentation process is almost the same as that described in FIG. 8 of the first embodiment. The difference is that the selection of the display method for indicating the difference in display magnification is changed to the selection of the display method for indicating the difference in focus position. Hereinafter, a description will be given of the processing for changing the annotation presentation when the display magnification is changed or the focus position is changed after the initial annotation presentation. Here, a case where image data having a hierarchical structure as shown in FIG. 11B configured by a plurality of depth image data having different focal positions and a plurality of hierarchical image data having different magnifications will be described as an example. That is, in the image display based on the image data, it is possible to switch the magnification at a high speed during image observation (switching without resolution conversion each time) and change the focal position by a user operation.

  In step S <b> 1301, the display data generation control unit 307 determines whether there is a display magnification change request from the user. If there is a display magnification change request, the display data generation control unit 307 proceeds to step S1302, and if there is no display magnification change request, the display data generation control unit 307 proceeds to step S1303.

  In step S1302, the hierarchical image data acquisition unit 309 acquires hierarchical image data having a magnification that matches the magnification change request from the plurality of hierarchical image data.

  In step S1303, the display data generation control unit 307 determines whether or not the user has requested to change the focal position. If there is a request for changing the focal position, the display data generation control unit 307 proceeds to step S1303. If there is no request for changing the focal position, the display data generation control unit 307 ends the process.

  In step S1304, the hierarchical image data acquisition unit 309 acquires depth image data of the focal position that matches the focal position change request from the plurality of depth image data.

  In step S1305, the annotation data generation unit 308 updates the annotation display data. In the annotation display mode, there is annotation display data that displays the positions and contents of a plurality of annotations added to the image data in different modes depending on the display magnification and the focal position of the virtual slide image when the annotation is added. Generated. In the pointer display mode, annotation display data is generated so that the positions of a plurality of annotations added to the image data are displayed in different modes depending on the display magnification and the focal position of the virtual slide image when the annotation is added. The Annotation display that is set by the user, such as the color, brightness, font, etc. of the text that makes up the annotation display data, the shape and color of the annotation display frame, the background color within the frame, the transparency of the annotation display area, the presence or absence of blinking display, etc. Determined according to the method.

  In step S1306, the display data generation unit 310 generates display data for screen display from the hierarchical image data selected in step S1302 or the depth image data selected in step S1304 and the annotation display data generated in step S1305. Generate.

  In step S1307, the display data output unit 311 outputs the display data generated in step S1306 to the display device 103.

  In step S1308, the display device 103 displays an image based on the display data input from the display data output unit 311 on the screen.

(Effect of Example)
In this embodiment, when an annotation is added to a virtual slide image, link information is generated based on the annotation position information and the focal position information of the virtual slide image. Link information is determined for each added annotation, each of a plurality of different depth image data constituting the image data, converted position information obtained by converting the position of the annotation to a position in each depth image data, This is information indicating the correspondence relationship. When presenting a plurality of annotations added to image data, the display mode of each annotation is made different according to the focal position of the virtual slide image when the annotation is added. Thereby, the user can easily grasp the difference in the focal position of the virtual slide image when added for each annotation.

[Third Embodiment]
In the present embodiment, a case will be described in which display control is performed using annotation data attached to a depth image whose focal position is different from the depth image being displayed according to the display magnification and the display focal position.

In general, since the depth of field becomes shallow in high-magnification observation, it is often necessary to observe the specimen by changing the focal position. Therefore, when the display magnification is a predetermined magnification or higher, if the annotation attached to the depth image whose focal position is different from the depth image being displayed is displayed, information on the previous and next focal positions can be obtained without switching the focal position. This is useful information for detailed observation.

  In the present embodiment, the above-described function is realized by inserting an annotation data control process (not shown) corresponding to the display magnification and the display focus position immediately before S1305 in FIG. 13 described in the second embodiment.

  FIG. 14 is a flowchart of annotation data display control processing according to display magnification and display focus position.

  First, in S1401, the display data generation control unit 307 determines whether or not an annotation exists in the depth image at any focal position in the display area. If it does not exist, the display data generation control unit 307 ends the process. If an annotation is present in the depth image at any focal position in the display area, the display data generation control unit 307 proceeds to S1402 and determines whether the display magnification is greater than or equal to a predetermined magnification. The predetermined magnification can be set to an arbitrary magnification. In this embodiment, the following description will be made assuming that the predetermined magnification is 20 times.

If the display magnification is less than 20, the display data generation control unit 307 ends the process. When the display magnification is 20 times or more in S1402, the display data generation control unit 307 proceeds to S1403 and changes the display setting of the annotation attached to the depth image whose focal position is different from the depth image being displayed.
For example, if the annotation attached to a depth image whose focal position is different from the depth image being displayed is set not to be displayed, the annotation attached to the depth image whose focal position is different from that of the displayed depth image is also displayed. Change to the setting you want. At this time, it is set so that the annotation of the depth image whose focal position is different from the depth image being displayed can be visually identified. For example, the color and transparency are changed so as to be different from the original annotation (annotation of the depth image being displayed).

  FIG. 16A shows a one-dimensional schematic diagram of five depth images having a magnification of 20 times, and annotations 1601 and 1602 are added to locations considered to be abnormal locations with Z = 1 and 4, respectively. Yes.

  When observing this depth image at a display magnification of 20 times and a focal position Z = 3, the display control processing for annotation data described above is applied, and the display magnification is 20 times or more. Although two annotations are annotations attached to a depth image having a focus position different from that of the displayed depth image, they are determined and displayed as annotations to be displayed.

  Therefore, even when observing the depth image at the focal position Z = 3, the contents of the annotation attached to the depth images at the front and rear focal positions are displayed. As a result, the user can know that there is an abnormal part in the vicinity without switching the focal position, and can perform detailed observation consciously.

  On the other hand, when observing at a display magnification of 5 × and a focal position Z = 1, since the display magnification is less than 20 ×, only the depth image annotation at the focal position Z = 1 is displayed. In the case of low-magnification display, since the depth of field is deep, it is rarely necessary to observe a plurality of depth image data while switching, and there is little need to switch the focus position. Therefore, it is sufficient for the user to display the contents of the annotation attached to the depth image of the focal position to be displayed.

At high magnification, the depth of field is shallow. Therefore, annotations existing in the depth image of the focal position in the adjacent range predetermined with respect to the focal position of the depth image being displayed are not subjected to the annotation existing in the depth image of all the focal positions. Only the processing target may be used.

  As described above, the processing shown in FIG. 14 makes it possible to control the display of annotations attached to a depth image having a focal point position different from the depth image being displayed according to the display magnification and the display focal point position. The effect which improves a user's convenience is acquired.

[Fourth Embodiment]
In this embodiment, when an annotation is added to a depth image whose focal position is different from that of the displayed depth image, the display of the annotation is controlled according to the degree of similarity between the images in the vicinity of the annotation in both depth images. Examples will be described.

FIG. 15 is a flowchart of annotation data display control in this embodiment.
First, in step S <b> 1501, the display data generation control unit 307 determines whether an annotation exists in a depth image having a focus position different from that of the depth image being displayed. If it does not exist, the display data generation control unit 307 ends the process, and if it does exist, the display data generation control unit 307 proceeds to S1502. In S1502, the display data generation control unit 307 acquires an image near the annotation position in the depth image where the annotation exists and the depth image being displayed, and the process proceeds to S1503.

  The area of the image near the annotation position may be defined in advance by an annotation, or may be an area inside a predetermined rectangle centered on the position of the annotation.

  In step S1503, the display data generation control unit 307 calculates the similarity between the images near the annotation position and the post-shift coordinates in the acquired depth image being displayed and the depth image where the annotation exists. For the calculation of similarity and post-shift coordinates, block matching, which is a general method for finding the corresponding position between images, can be used, and for the internal calculation processing, residual sum of squares, normalized cross-correlation, etc. can be used. .

  The similarity is the maximum correlation value between two images obtained by shifting the image near the annotation position in the depth image where the annotation exists relative to the image near the annotation position in the displayed depth image. The post-shift coordinates are coordinates determined based on the shift amount when the correlation value is maximized. Rather than calculating the similarity for the entire display area, the processing can be speeded up by performing matching within a predetermined range near the annotation position.

  In step S1504, the display data generation control unit 307 determines whether the similarity calculated in step S1503 is equal to or greater than a predetermined threshold value. If it is less than the threshold value, the display data generation control unit 307 interprets that the image in the vicinity of the annotation position has changed between the depth image in which the annotation exists and the depth image being displayed, and proceeds to S1305.

  On the other hand, if it is equal to or greater than the threshold value, the display data generation control unit 307 interprets that the depth image in which the annotation exists and the depth image being displayed are substantially the same in the vicinity of the annotation position, and the process advances to step S1505.

  In S1505, information on a new annotation to be added to the displayed depth image is generated.

  The position of the new annotation is set to the post-shift coordinates obtained in S1503. The comment (annotation) is used as it is without being changed.

However, in order to clearly indicate that the new annotation is an estimation based on image processing, it is preferable to display the new annotation in a manner different from the original annotation. For example, this can be realized by changing the color or transparency of the annotation. In order to notify the similarity, information corresponding to the above-described similarity may be added to the annotation.

  In this way, by moving the annotation information attached to the depth images at the focal positions before and after the display focal position to an appropriate position and displaying them, the user can confirm the annotation by switching the focal position during observation as in the third embodiment. This saves you time and effort. In addition, the user can save the trouble of adding annotations to a plurality of focal positions.

Hereinafter, the processing results obtained in FIG. 15 will be described using the schematic diagram shown in FIG.
FIG. 16B is an example of image data composed of depth image data at five focal positions, and an annotation 1603 is added to the depth image at the focal position Z = 3. It is assumed that the site with this annotation is a cavity in the sample tissue.

  When the display focal position is Z = 1, the cavity is not in the same XY position as Z = 3, but a similar cavity exists at the shifted position. Therefore, in the above-described annotation data control process, when a depth image at the focal position Z = 1 is displayed, a new annotation is created at the post-shift coordinates obtained in S1503. At this time, the new annotation is different in display mode such as color and transparency from the original annotation 1603.

  Similarly, FIG. 16C is an example of image data including depth image data at five focal positions, and an annotation 1604 is added to the depth image at the focal position Z = 3. It is assumed that the part to which the annotation is added is a certain structure in the specimen tissue.

  An image similar to the image showing the structure in the vicinity of the annotation 1604 does not exist in the depth image at any focal position. Therefore, when displaying a depth image with Z = 1, the similarity obtained in S1503 does not exceed the threshold value in S1503, and thus no new annotation is generated.

  At high magnification, the depth of field is shallow. Therefore, in S1501 to S1503, the annotations existing in the depth images of all the focal positions are not processed, but the depth images of the focal positions in the adjacent range predetermined with respect to the focal position of the depth image being displayed. Only the processing target may be used.

  As described above, with the configuration shown in FIG. 15, even when an annotation is added to a depth image having a focal position different from that of the currently displayed depth image, when the images in the vicinity of the annotation in both depth images are similar, different depths are used. It is possible to display the annotation. Thereby, the effect which improves the convenience of the user at the time of observation is acquired. In particular, since the focal position is different due to the structure of the tissue, the added annotation can be used effectively even when the position on the two-dimensional image is different while showing the same tissue.

[Other Embodiments]
The object of the present invention may be achieved by the following. That is, a recording medium (or storage medium) in which a program code of software that realizes all or part of the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  When the computer executes the read program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. The case where the functions of the above-described embodiments are realized by the processing can also be included in the present invention.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Then, based on the instruction of the program code, the CPU or the like provided in the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. It can be included in the invention.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

  In addition, the configurations described in the first, second, third, and fourth embodiments can be combined with each other. For example, the focus position explicit processing of the second embodiment may be applied to the system of the first embodiment, or the image processing device is connected to both the imaging device and the image server and used for processing. You may make it the structure which can acquire an image from any apparatus. In addition, configurations obtained by appropriately combining various techniques in the above embodiments also belong to the category of the present invention.

301: Image data acquisition unit 303: User input information acquisition unit 307: Display data generation control unit 308: Annotation data generation unit 310: Display data generation unit

Claims (6)

An apparatus for generating display data for displaying an image of a subject on a display device ,
An additional information acquiring unit that acquires information of the display magnification of the image when an annotation is added to the image,
Designated information acquisition means for acquiring display magnification information of the image specified by the user ;
Anda generating means for generating said display data for displaying said annotation with the image of the designated display magnification,
Said generating means, in the table示倍rate when said additional said the case where the designated display magnification is different, the and the added display magnification and the designated display magnification when the same case as the user The display data different from each other is generated so that it is possible to determine which is the case.
A device characterized by that .   The additional information acquisition unit acquires position information of the annotation.
  The apparatus according to claim 1.   The image data includes data of a plurality of hierarchical images of the subject having different resolutions,
  The generation unit generates the display data using the hierarchical image data having a resolution corresponding to the designated display magnification.
  The apparatus according to claim 2.   Link information generating means for generating link information for determining the position information of the annotation in each of the plurality of hierarchical images based on the display magnification when added and the position information of the annotation
  The apparatus according to claim 3.   A method of generating display data for displaying an image of a subject on a display device,
  An additional information acquisition step of acquiring information of a display magnification of the image when an annotation is added to the image;
  A designated information obtaining step for obtaining information on the display magnification of the image designated by the user;
  Generating the display data for displaying the image and the annotation at the designated display magnification, and
  In the generation step, the user can use the display magnification when the addition is different from the designated display magnification, and the case where the display magnification when the addition is the same as the designated display magnification. Generate different display data so that you can determine which case
  A method characterized by that.   The program for making a computer perform each process of the method of Claim 5.

Images