JP2013152701A - Image processing device, image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device allowing a user to efficiently find a position of interest while reducing the time and effort of the user.SOLUTION: The image processing device includes: an adding unit that adds an annotation to a diagnostic image obtained by taking an image of an object; a storage that stores a piece of attribute information which is a piece of predetermined attribute information as the information on the annotation in a storage along with annotation; a searching unit that searches a position of interest as a position interested by the user from the plural positions added with the annotation in the diagnostic image; and a display unit that displays the search result by the searching unit on the display section. The searching unit searches the position of interest using a word or a piece of attribute information included in the annotation.

Description

  The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program.

  In recent years, in the field of pathology, as an alternative to an optical microscope, which is a tool for pathological diagnosis, a virtual slide that enables pathological diagnosis on a display by imaging a test sample (subject) placed on a slide and digitizing the image The system is drawing attention. By digitizing a pathological diagnosis image using a virtual slide system, an optical microscope image of a subject can be handled as digital image data. As a result, it is expected that merits such as quick remote diagnosis, explanation to patients using digital images, sharing of rare cases, and efficiency of education / practice will be obtained.

In order to realize an operation equivalent to that of an optical microscope with a virtual slide system, it is necessary to digitize the entire image of the subject on the slide. By digitizing the image of the entire subject, the entire subject can be observed with viewer software (viewer) operating on a PC (Personal Computer) or a workstation. The number of pixels when an image of the entire subject is digitized is usually a very large data amount of several hundred million to several billion pixels. Thus, the amount of digital image data created by the virtual slide system is enormous. However, it is possible to observe from the micro (detailed enlarged image) to the macro (overall bird's-eye view image) by performing enlargement / reduction processing of the image with the viewer, and it is possible to provide various conveniences. . By acquiring all necessary information in advance, it is possible to immediately display an image having the resolution and magnification required by the user. For example, it is possible to immediately display from a low magnification image to a high magnification image.
Also, an image processing apparatus has been proposed in which an annotation is added to a medical image when a medical image (ultrasonic image) is acquired, and the medical image is searched using a comment in the annotation as a search key (Patent Document) 1).

JP-A-11-353327

  In a diagnostic image such as a virtual slide image, compared to other medical images, there are many positions (attention positions; attention sites; regions of interest) that are noticed by the diagnosis doctor. When only annotations (comments) as annotations are added in order to search these attention positions, the search target is limited to the annotations, and there is a limitation in finding a desired attention position. In addition, if all the added annotations are checked and the attention position is searched for, the time required for diagnosis by the diagnostician is increased because the time required for the diagnostician (pathologist) increases.

  Accordingly, an object of the present invention is to provide a technique that enables a user to efficiently find a position of interest, and thus reduces the user's trouble.

The image processing apparatus according to the present invention stores an addition unit that adds an annotation to a diagnostic image obtained by imaging a subject, and attribute information that is information about a predetermined attribute as information related to the annotation, together with the annotation. A recording unit for recording in the unit, a search unit for searching for a target position that is a position of interest to the user from a plurality of positions added with annotations in the diagnostic image, and a search result of the search unit on the display unit Display means for displaying, wherein the search means searches for the position of interest using a word or attribute information included in the annotation as a key.

  An image processing system according to the present invention includes the image processing apparatus and the display unit.

  The image processing method of the present invention includes an adding step in which a computer adds an annotation to a diagnostic image obtained by capturing an image of a subject, and an attribute that is information of a predetermined attribute as information related to the annotation. A recording step of recording information in the storage unit together with the annotation, a search step in which the computer searches a position of interest, which is a position of interest by the user, from a plurality of positions to which the annotation is added in the diagnostic image; And a display step of displaying a search result of the search step on a display unit. In the search step, the attention position is searched using a word or attribute information included in the annotation as a key.

  A program according to the present invention causes a computer to execute each step of the image processing method.

  According to the present invention, it becomes possible for the user to efficiently find the position of interest, thereby reducing the user's trouble.

1 is a diagram illustrating a configuration of an image processing system according to a first embodiment. FIG. 1 is a block diagram illustrating a functional configuration of an imaging apparatus according to a first embodiment. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment. 1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to a first embodiment. Conceptual diagram of hierarchical image prepared in advance for each magnification 1 is a flowchart showing the overall processing flow of the image processing apparatus according to the first embodiment. 10 is a flowchart showing the flow of annotation addition processing according to the first embodiment. 7 is a flowchart illustrating a flow of attention position search processing according to the first embodiment. 7 is a flowchart showing the flow of search result display processing according to the first embodiment. Example of display image according to embodiment 1 An example of annotation list configuration FIG. 10 is a diagram illustrating a configuration of an image processing system according to the second embodiment. 10 is a flowchart illustrating a flow of search result display processing according to the second embodiment. Example of display image according to embodiment 2 Example of diagnostic criteria setting screen and diagnostic classification screen according to embodiment 1 Example of diagnosis support data list configuration

Example 1
Hereinafter, an image processing system according to a first embodiment of the present invention will be described with reference to the drawings.
The image processing apparatus according to the first embodiment of the present invention can be used in an image processing system including an imaging device and a display device.
(Device configuration of image processing system)
An image processing system according to the present embodiment will be described with reference to FIG.
FIG. 1 is a diagram illustrating a configuration of an image processing system using the image processing apparatus according to the present embodiment. The image processing system according to this embodiment includes an imaging device (microscope device or virtual slide scanner) 101, an image processing device 102, a display device 103, and the like, and a two-dimensional image of a subject (test sample) to be imaged. It has the function to acquire and display. The imaging apparatus 101 and the image processing apparatus 102 are connected to each other by a dedicated or general-purpose I / F cable 104, and the image processing apparatus 102 and the display apparatus 103 are connected to each other by a general-purpose I / F cable 105. .

As the imaging device 101, for example, a virtual slide device having a function of capturing a plurality of two-dimensional images with different positions in a two-dimensional plane direction and outputting a digital image can be used. A solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) is used to acquire a two-dimensional image. As the image pickup apparatus 101, a digital microscope apparatus in which a digital camera is attached to an eyepiece part of a normal optical microscope can be used.
The image processing apparatus 102 has a function of generating data (display data) to be displayed on the display apparatus 103 from digital image data (original image data) acquired from the imaging apparatus 101 in response to a request from the user. The image processing apparatus 102 is configured by a general-purpose computer or workstation having hardware resources such as various I / Fs including a CPU (Central Processing Unit), a RAM, a storage device, and an operation unit. The storage device is a large-capacity information storage device such as a hard disk drive, and stores programs, data, OS (operating system) and the like for realizing each processing described later. Each function of the image processing apparatus 102 is realized by the CPU loading a necessary program and data from the storage device to the RAM and executing the program. The operation unit includes a keyboard, a mouse, and the like, and is used by the user to input various instructions.
The display device 103 is a display (display unit) that displays an image based on display data that is a result of arithmetic processing performed by the image processing device 102. As the display device 103, a display device using EL (Electro-Luminescence), liquid crystal, CRT (Cathode Ray Tube), or the like can be used.

  In the example of FIG. 1, the image processing system is configured by three devices of the imaging device 101, the image processing device 102, and the display device 103, but the configuration of the present invention is not limited to this configuration. For example, an image processing device integrated with a display device may be used, or the function of the image processing device may be incorporated in the imaging device. In addition, the functions of the imaging device, the image processing device, and the display device can be realized by a single device. Conversely, the function of one device may be realized by a plurality of devices. For example, the function of the image processing device may be realized by a plurality of devices.

(Functional configuration of imaging device)
FIG. 2 is a block diagram illustrating a functional configuration of the imaging apparatus 101.
The imaging apparatus 101 is generally configured by an illumination unit 201, a stage 202, a stage control unit 205, an imaging optical system 207, an imaging unit 210, a development processing unit 219, a pre-measurement unit 220, a main control system 221, and a data output unit 222. Is done.

The illumination unit 201 is means for uniformly irradiating light to a preparation 206 (subject) disposed on the stage 202, and includes a light source, an illumination optical system, and a light source drive control system. The stage 202 is driven and controlled by a stage control unit 205, and can move in three directions of XYZ. The preparation 206 is a member in which a subject to be observed (a tissue slice or smeared cells) is attached on a slide glass and fixed together with an encapsulant under the cover glass.
The stage control unit 205 includes a drive control system 203 and a stage drive mechanism 204. The drive control system 203 receives the instruction from the main control system 221 and performs drive control of the stage 202. The moving direction, moving amount, and the like of the stage 202 are determined based on the subject position information and thickness information (distance information) measured by the pre-measurement unit 217 and, if necessary, instructions from the user. The stage drive mechanism 204 drives the stage 202 in accordance with instructions from the drive control system 203.
The imaging optical system 207 is a lens group for forming an optical image of the subject of the preparation 206 on the image sensor 208.

The imaging unit 210 includes an imaging sensor 208 and an analog front end (AFE) 209.
The imaging sensor 208 is a one-dimensional or two-dimensional image sensor that changes a two-dimensional optical image into an electrical physical quantity by photoelectric conversion. As the imaging sensor 208, for example, a CCD or a CMOS device is used. When the imaging sensor 208 is a one-dimensional sensor, a two-dimensional image (two-dimensional captured image) is obtained by scanning in the scanning direction. The imaging sensor 208 outputs an electrical signal (analog signal) having a voltage value corresponding to the light intensity. When a color image is desired as the captured image, for example, a single-plate image sensor to which a Bayer array color filter is attached may be used. The imaging unit 210 captures a divided image (a plurality of divided images having different imaging areas) of the subject by driving the stage 202 in the XY axis direction.
The AFE 209 is a circuit that converts an analog signal output from the image sensor 208 into a digital signal. The AFE 209 includes an H / V driver, a CDS (Correlated double sampling), an amplifier, an AD converter, and a timing generator, which will be described later. The H / V driver converts a vertical synchronization signal and a horizontal synchronization signal for driving the image sensor 208 into potentials necessary for driving the sensor. CDS is a double correlation sampling circuit that removes noise of fixed patterns. The amplifier is an analog amplifier that adjusts the gain of an analog signal from which noise has been removed by CDS. The AD converter converts an analog signal into a digital signal. When the output at the final stage of the imaging apparatus is 8 bits, the AD converter converts the analog signal into digital data quantized from about 10 bits to about 16 bits and outputs it in consideration of subsequent processing. The converted sensor output data is called RAW data. The RAW data is developed by a subsequent development processing unit 219. The timing generator generates a signal for adjusting the processing timing of the image sensor 208 and the processing timing of the development processing unit 219 in the subsequent stage.
When a CCD is used as the image sensor 208, the AFE 209 is indispensable. However, in the case of a CMOS image sensor capable of digital output, the function of the AFE 209 is included in the sensor. Although not shown, there is an imaging control unit that controls the imaging sensor 208, and the operation timing and control such as the operation control of the imaging sensor 208, the shutter speed, the frame rate, and the ROI (Region Of Interest) are combined. Done.

The development processing unit 219 includes a black correction unit 211, a white balance adjustment unit 212, a demosaicing processing unit 213, an image composition processing unit 214, a resolution conversion processing unit 215, a filter processing unit 216, a γ correction unit 217, and a compression processing unit 218. Composed.
The black correction unit 211 performs a process of subtracting the black correction data obtained at the time of shading from each pixel of the RAW data.
The white balance adjustment unit 212 performs a process of reproducing a desired white color by adjusting the gain of each RGB color according to the color temperature of the light of the illumination unit 201. Specifically, white balance correction data is added to the RAW data after black correction. When handling a monochrome image, the white balance adjustment process is not necessary.
The demosaicing processing unit 213 performs processing for generating image data of each color of RGB from RAW data in the Bayer array. The demosaicing processing unit 213 calculates the value of each RGB color of the target pixel by interpolating the values of peripheral pixels (including pixels of the same color and other colors) in the RAW data. The demosaicing processing unit 213 also executes defective pixel correction processing (interpolation processing). Note that when the imaging sensor 208 does not have a color filter and a single color image is obtained, the demosaicing process is not necessary.
The image composition processing unit 214 performs a process of generating large-capacity image data in a desired imaging range by joining the divided image data acquired by dividing the imaging range by the imaging sensor 208. In general, since the existence range of the subject is wider than the imaging range that can be acquired by one imaging with an existing image sensor, one piece of two-dimensional image data (large-capacity image data) is generated by joining the divided image data. For example, assuming that a 10 mm square area on the slide 206 is imaged with a resolution of 0.25 μm, the number of pixels on one side is 40,000 pixels of 10 mm / 0.25 μm, and the total number of pixels is 1.6 billion, which is the square of the number of pixels. It becomes a pixel. In order to acquire image data of 1.6 billion pixels using the image sensor 208 having 10M (10 million) pixels, it is necessary to divide the area into 160 areas of 1.6 billion / 10,000,000 and perform imaging. In addition, as a method of joining a plurality of image data, a method of joining by aligning based on the position information of the stage 202, a method of joining by matching points or lines corresponding to a plurality of divided images, There is a method of connecting based on position information of image data. At the time of joining, a plurality of divided images can be smoothly connected by interpolation processing such as zero-order interpolation, linear interpolation, and high-order interpolation. In the present embodiment, it is assumed that one large-capacity image is generated. However, when the image processing apparatus 102 acquires a plurality of divided image data and generates display data, the divided images are joined together. May be.
The resolution conversion processing unit 215 performs processing for generating a plurality of images with different magnifications in advance by resolution conversion in order to display the large-capacity two-dimensional image generated by the image composition processing unit 214 at high speed. The resolution conversion processing unit 215 generates image data of a plurality of magnifications from low magnification to high magnification, and generates data having a hierarchical structure in which these image data are collected together. Details of the data having the hierarchical structure will be described later with reference to FIG.
The filter processing unit 216 is a digital filter that realizes suppression of high-frequency components contained in an image, noise removal, and resolution enhancement. The γ correction unit 217 executes processing for adding an inverse characteristic to an image in accordance with the gradation expression characteristic of a general display device, or adjusts to the human visual characteristic by gradation compression or dark part processing of a high luminance part. Or perform tone conversion. In this embodiment, in order to acquire an image for the purpose of morphological observation, gradation conversion suitable for composition processing and display processing is applied to image data.
The compression processing unit 218 performs encoding processing for improving the efficiency of transmission of large-capacity two-dimensional image data and reducing (compressing) the capacity when storing. As a still image compression technique, standardized encoding methods such as JPEG (Joint Photographic Experts Group) and JPEG 2000 and JPEG XR, which are improved and evolved from JPEG, are widely known.

  The pre-measurement unit 220 is a unit that performs pre-measurement for calculating the position information of the subject on the slide 206, the distance information to the desired focal position, and the parameter for adjusting the amount of light caused by the thickness of the subject. By acquiring information by the pre-measurement unit 220 before the main measurement, it is possible to perform image pickup without waste. A two-dimensional image sensor having a lower resolving power than the image sensor 208 is used to acquire position information on the two-dimensional plane. The pre-measurement unit 220 grasps the position of the subject on the XY plane from the acquired image. For obtaining distance information and thickness information, a laser displacement meter or a Shack-Hartmann measuring instrument is used.

The main control system 221 is a function for controlling the various units described so far. The control functions of the main control system 221 and the development processing unit 219 are realized by a control circuit having a CPU, a ROM, and a RAM. That is, the program and data are stored in the ROM, and the functions of the main control system 221 and the development processing unit 219 are realized by the CPU executing the program using the RAM as a work memory. For example, a device such as an EEPROM or a flash memory is used as the ROM, and a DRAM device such as DDR3 is used as the RAM. Note that the main control system 221 may be replaced with a function obtained by developing the function of the development processing unit 219 into an ASIC as a dedicated hardware device.

  The data output unit 222 is an interface for sending the image data generated by the development processing unit 219 to the image processing apparatus 102 as diagnostic image data. The imaging device 101 and the image processing device 102 are connected to each other by an optical communication cable. Alternatively, a general-purpose interface such as USB or Gigabit Ethernet (registered trademark) is used.

(Functional configuration of image processing apparatus)
FIG. 3 is a block diagram illustrating a functional configuration of the image processing apparatus 102 according to the present embodiment.
The image processing apparatus 102 includes an outline, an image data acquisition unit 301, a storage holding unit (memory) 302, a user input information acquisition unit 303, a display device information acquisition unit 304, an annotation data generation unit 305, an annotation list storage unit 306, and an annotation search. The processing unit 307, the display data generation control unit 308, the display image data acquisition unit 309, the display data generation unit 310, and the display data output unit 311 are configured.

  The image data acquisition unit 301 acquires image data captured by the imaging apparatus 101 (diagnosis image (image of a diagnosis target) obtained by imaging a subject). The diagnostic image data referred to here is RGB color divided image data obtained by dividing and imaging a subject, one piece of two-dimensional image data (high resolution image data) obtained by combining the divided image data, and It is at least one of the image data (hierarchical image data) of each magnification generated based on the high resolution image data. The divided image data may be monochrome image data.

  The storage holding unit 302 captures, stores, and holds image data acquired from an external device (imaging device 101) via the image data acquisition unit 301.

  The user input information acquisition unit 303 changes the display position in the diagnostic image, changes the display magnification of the diagnostic image (magnification of the tomographic image to be displayed; enlargement rate or reduction rate), etc., via an operation unit such as a mouse or a keyboard. The update instruction of the display image data (image data of the area where the diagnostic image is displayed) is acquired. In addition, the user input information acquisition unit 303 acquires input information to a display application used when adding an annotation to a position of interest of a diagnostic image via an operation unit such as a mouse or a keyboard. An annotation is information given as an annotation to image data, and may be only information indicating that there is an annotation, or may include annotation content (text data).

  The display device information acquisition unit 304 acquires display magnification information (display magnification information) of the currently displayed image, in addition to display area information (screen resolution) of the display held by the display device 103.

The annotation data generation unit 305 adds an annotation to the position specified by the user in the diagnostic image. In addition, when an annotation is added, the annotation data generation unit 305 records not only text information as annotation content but also attribute information as information related to the annotation together with the text information in the storage unit (annotation list storage unit 306). To do. As will be described later, the attribute information is used to narrow down annotations that an observer (doctor, engineer, etc.) should pay attention from among a large number of annotations added to a diagnostic image. Therefore, any type of information can be used as attribute information as long as it is useful for narrowing down (searching) annotations. For example, attribute information includes information such as when and who (whether added by a person or computer automatically), information indicating the annotation, purpose, intention, or viewpoint Can be used. Details of the attribute information will be described later.
The annotation data generation unit 305 receives the position coordinates (coordinates of the position specified by the user (position where the annotation is added) in the display screen) in the display screen (screen of the display device 103) from the user input information acquisition unit 303. Get information. The annotation data generation unit 305 acquires display magnification information from the display device information acquisition unit 304. Then, the annotation data generation unit 305 converts the position coordinates in the display screen into position coordinates in the diagnostic image using the information. Thereafter, the annotation data generation unit 305 generates annotation data including text information (text data) input as an annotation, information on position coordinates in the diagnostic image, display magnification information, and attribute information. The generated annotation data is recorded in the annotation list storage unit 306. Details of the annotation addition processing will be described later with reference to FIG.

  The annotation list storage unit 306 stores a reference table (annotation list) in which the annotation data generated by the annotation data generation unit 305 is listed. The structure of the annotation list will be described later with reference to FIG.

  The annotation search processing unit 307 searches an attention position, which is a position that the user pays attention, from a plurality of positions to which the annotation is added. Details of the attention position search processing will be described later with reference to FIG.

The display data generation control unit 308 controls display data generation in accordance with instructions from the user input information acquisition unit 303. The display data mainly includes display image data and annotation image data (annotation image data).
The display image data acquisition unit 309 acquires diagnostic image data (display image data) necessary for display from the storage holding unit 302 in accordance with an instruction from the display data generation control unit 308.

Display data is generated by the display data generation unit 310 and the display data output unit 311 and is output to the display device 103. Thereby, an image based on the display data is displayed on the display device 103. When the attention position is searched, the display data generation unit 310 and the display data output unit 311 display the search result by the annotation search processing unit 307 on the display device 103.
Specifically, the display data generation unit 310 uses the annotation data generated by the annotation data generation unit 305 and the diagnostic image data acquired by the display image data acquisition unit 309 to display on the display device 103. Generate display data.
The display data output unit 311 outputs the display data generated by the display data generation unit 310 to the display device 103 which is an external device.

(Hardware configuration of image forming apparatus)
FIG. 4 is a block diagram illustrating a hardware configuration of the image processing apparatus according to the present embodiment. As an image processing apparatus that performs information processing, for example, a PC (Personal Computer) is used.
The PC includes a CPU (Central Processing Unit) 401, a RAM (Random Access Memory).
) 402, a storage device 403, a data input / output I / F 405, and an internal bus 404 for connecting them together.
The CPU 401 accesses the RAM 402 and the like as necessary, and comprehensively controls each block of the PC while performing various arithmetic processes.
The RAM 402 is used as a work area for the CPU 401, and temporarily stores the OS, various programs being executed, and various data used in processing such as annotation search and display data generation, which are features of the present invention.
The storage device 403 is an auxiliary storage device in which firmware such as an OS, a program, and various parameters to be executed by the CPU 401 is fixedly stored. As the storage device 403, a magnetic disk drive such as an HDD (Hard Disk Drive) or a semiconductor device (SSD (Solid State Disk) or the like) using a flash memory is used.
The data input / output I / F 405 is connected to the image server 1101 via the LAN I / F 406.
However, the display device 103 via the graphics board 407, the imaging device 101 represented by a virtual slide device and a digital microscope via the external device I / F 408, the keyboard 410 and the mouse 411 via the operation I / F 409, Each is connected.

In this embodiment, a PC to which the display device 103 is connected as an external device is assumed. However, a PC integrated with the display device may be used. For example, a notebook PC corresponds to this.
In the present embodiment, a pointing device such as a keyboard 410 and a mouse 411 is assumed as an input device connected via the operation I / F 409. However, when the screen of the display device 103 is a touch panel. The touch panel may be used as an input device. In that case, the touch panel can be integrated with the display device 103.

(Concept of data hierarchized for each magnification)
As described above, the image processing apparatus 102 may receive data hierarchized for each magnification. FIG. 5 is a conceptual diagram of an image (hierarchical image; an image generated by the resolution conversion processing unit 215 of the imaging device 101) prepared in advance for each magnification.
The hierarchical image has a two-dimensional axis of axis X and axis Y. An axis P in a direction orthogonal to the axes X and Y is an axis for showing a plurality of hierarchical images in a pyramid format.
Reference numerals 501, 502, 503, and 504 are two-dimensional images (hierarchical images) having different magnifications and hierarchical images having different resolutions. In order to simplify the description, the resolution of each one-dimensional direction (X direction, Y direction) of the hierarchical image 503 is ½ that of the hierarchical image 504. The resolution in each one-dimensional direction of the hierarchical image 502 is ½ of the hierarchical image 503. The resolution in each one-dimensional direction of the hierarchical image 501 is ½ of the hierarchical image 502.
Reference numeral 505 indicates an area of the size of the divided image.

  The image data obtained by imaging with the imaging device 101 is desired to be high resolution and high resolution image data for the purpose of diagnosis. However, as described above, when displaying a reduced image of image data consisting of billions of pixels, if the resolution conversion is performed each time a display request is made, the process will not be in time. Therefore, it is desirable to prepare several hierarchical images with different magnifications in advance, select an image with a magnification close to the display magnification from the prepared hierarchical images, and perform resolution conversion of the selected image according to the display magnification. . Thereby, the processing amount of resolution conversion can be reduced. In general, it is more preferable in terms of image quality to generate a display magnification image from a higher magnification image.

  Since the image data obtained by imaging is high-resolution image data, hierarchical image data of other magnifications is generated by reducing the image data by a resolution conversion method. As a resolution conversion method, bicubic using a cubic interpolation formula is widely known in addition to bilinear which is a two-dimensional linear interpolation process.

  Thus, when diagnosing and observing while changing the display magnification of a diagnostic image, it is desirable to prepare a plurality of hierarchical image data having different magnifications as shown in the figure as diagnostic image data. A plurality of hierarchical image data may be collected and handled as one data (file), and each hierarchical image data is prepared as independent image data and information indicating the relationship between the magnification and the image data is prepared. May be.

(How to add annotations, search attention positions, and display search results)
The flow of annotation addition, attention position search, and search result display in the image processing apparatus according to the present embodiment will be described with reference to the flowchart of FIG.

  In step S601, the display device information acquisition unit 304 acquires size information (screen resolution) of the display area (screen) of the display device 103 and information on the magnification of the currently displayed diagnostic image (display magnification information). To do. The size information of the display area is used when determining the size of display data to be generated. The display magnification is used when selecting a hierarchical image and generating annotation data. The generation of annotation data will be described later.

  In step S <b> 602, the display image data acquisition unit 309 acquires diagnostic image data corresponding to the display magnification acquired in step S <b> 601 from the storage holding unit 302. Specifically, the display image data acquisition unit 309 stores the diagnostic image data having the magnification closest to the display magnification acquired in step S601 (or larger than the display magnification and closest to the display magnification). Get more. When no diagnostic image is displayed, the display image data acquisition unit 309 acquires diagnostic image data corresponding to a predetermined display magnification (initial value) from the storage holding unit 302.

  In step S603, the display data generation unit 310 generates display data using the diagnostic image data acquired in step S602. Specifically, when the display magnification is equal to the magnification of the diagnostic image data acquired in step S602, the display data generation unit 310 generates display data using the acquired diagnostic image data as it is. When the display magnification is different from the magnification of the diagnostic image data acquired in step S602, the display data generation unit 310 performs resolution conversion of the acquired diagnostic image data so that the magnification becomes the display magnification, and the resolution Display data is generated using the converted image data. The generated display data is displayed on the display device 103. At this time, when the display position (display position) in the diagnostic image is instructed, a part or all of the diagnostic image data at the display magnification is displayed so that the position of the diagnostic image at the display magnification is displayed. Extraction and display data are generated.

  In step S604, the user input information acquisition unit 303 determines whether or not an instruction to update display image data has been given by the user. Specifically, it is determined whether or not there has been an instruction to change the display area of the diagnostic image, such as changing the display position or changing the display magnification. If there is an instruction to update the display image data, the process returns to step S602, and diagnostic image data acquisition and screen update (display image update) processing by display data generation are performed. If there is no instruction to update the display image data, the process proceeds to step S605.

  In step S605, the user input information acquisition unit 303 determines whether an annotation addition instruction and request has been received from the user. If there is an instruction to add an annotation, the process proceeds to step S606. If there is no instruction to add an annotation, the annotation adding process is skipped and the process proceeds to step S607. For example, the user input information acquisition unit 303 determines that there is an instruction for adding an annotation when the user specifies a position to add the annotation.

  In step S606, various processes accompanying the addition of the annotation are performed. As processing contents, in addition to acquiring text data input with the keyboard 410 or the like as annotations, acquisition of attribute information, which is a feature of this embodiment, can be mentioned. Details of step S606 will be described later with reference to FIG.

  In step S <b> 607, the user input information acquisition unit 303 determines whether an instruction and request for searching for a target position has been received from the user. If there is an instruction to search for the target position, the process proceeds to step S608. If there is no instruction to search for the position of interest, the process ends.

  In step S608, various processes associated with the search for the target position are performed. In this step, the attention position is searched using the word or attribute information included in the annotation as a key. Specifically, by searching for a target position, candidate positions that are candidates for the target position are detected from a plurality of positions with annotations added. Details of the processing in step S608 will be described later with reference to FIG. If no position is detected as a result of the search, an image or message indicating that there is no position corresponding to the input key is displayed by the display data generation unit 310 and the display data output unit 311. Is displayed on the display device 103.

  In step S <b> 609, the user input information acquisition unit 303 determines whether an instruction and a request for displaying the search result of the target position have been received from the user. If there is an instruction to display the search result, the process proceeds to step S610. If there is no instruction to display the search result, the process ends.

  In step S610, processing for displaying the search result is performed. Details of the processing in step S610 will be described later with reference to FIG.

  Note that FIG. 6 shows a case where a screen update request that is a request for changing the display position and display magnification, an annotation addition, and a search for a target position are sequentially performed. Not limited to. These processes may be performed simultaneously, and may be performed not only in the order of FIG. 6 but at any timing.

(Add annotation)
FIG. 7 is a flowchart showing a detailed flow of processing for adding an annotation shown in step S606 of FIG. Hereinafter, the flow of processing for generating annotation data based on the position where the annotation is added, the display magnification, and the attribute information will be described with reference to FIG.

  In step S <b> 701, the annotation data generation unit 305 acquires information of position coordinates (coordinates of positions to which annotations are added designated by the user) from the user input information acquisition unit 303. Since the information acquired here is information on the position (relative position) in the display screen of the display device 103, the annotation data generation unit 305 stores the position represented by the acquired information in the storage holding unit 302. Convert to the position (absolute position) in the stored diagnostic image.

  In step S <b> 702, the annotation data generation unit 305 acquires text data (annotation) input by the user using the keyboard 410 from the user input information acquisition unit 303. When there is an instruction to add an annotation, an image prompting the user to input text (annotation) is displayed, and the user inputs text information as annotation content in accordance with the display of the image.

  In step S <b> 703, the annotation data generation unit 305 acquires the current display magnification information (when an annotation is instructed) from the display device information acquisition unit 304. Here, the display magnification information is acquired from the display device 103, but the image processing device 102 generates display data, so that the configuration may use the display magnification data held internally. .

In step S704, the annotation data generation unit 305 acquires various attribute information for facilitating the user's search for annotations. The position information converted in step S701 and the display magnification acquired in step S702 are also attribute information in a broad sense. Position information and display magnification are information indicating the state of observation when an annotation is added, whereas the attribute information acquired in step S704 reflects the environment at the time of pathological diagnosis and the intention of the user It is.
Specifically, the attribute information includes date and time information, user information, diagnostic information, diagnostic criteria information, and the like.
The date / time information is, for example, the date / time when the corresponding annotation is added, for example, the date / time when the addition of the annotation is instructed or the date / time when the text as the annotation is input. The date / time information may be the date / time when the diagnosis image was observed (diagnosis).
The user information is information for identifying the user to which the annotation is added, and includes, for example, a user name, an identifier for identifying the user, a user attribute, and the like. In a workflow in pathological diagnosis, multiple users (for example, engineers, pathologists, clinicians, computers (automatic diagnosis software)) have different purposes (viewpoints, roles) or different methods (for example, automatic addition by image analysis, visual inspection) Add annotations to the same image in order. The user attribute is information indicating the purpose (viewpoint, role) or method when each user adds an annotation. For example, “pathologist”, “engineer”, “clinician”, “automatic diagnosis”, etc. It is done. In this way, by associating user attributes with annotations as one piece of user information and making it possible to search for each user attribute, it is possible to grasp the nature of each annotation information and to select information at each stage of the pathological diagnosis workflow. It becomes easy and the pathological diagnosis work can be facilitated.
The diagnostic information is information representing the diagnostic content of the diagnostic image. Diagnostic information includes, for example, importance information indicating the purpose of adding annotations, the degree of disease progression, and whether or not to compare for objective (relative) observation Information.
The diagnostic criteria information is information that summarizes the diagnostic classification for each organ, and is in a form in accordance with the actual situation for each country or region. The diagnostic classification indicates the stage (stage) for each organ. For example, in the case of gastric cancer, the diagnostic classification specified in the cancer classification code α, which is a diagnostic standard in one area, may be different from the diagnostic classification specified in the cancer classification code β, which is a diagnostic standard in another area. is there. Therefore, information representing the diagnostic criteria / diagnosis classification used when the user diagnoses the diagnostic image is added to the attribute information as diagnostic criteria information. The diagnostic criteria and diagnostic classification will be described later with reference to FIG.
In this embodiment, it is assumed that the attribute information is information selected by the user from a plurality of options (categories).
The attribute information may be automatically generated or input by the user. Some attribute information may be automatically generated, and other attribute information may be input (selected) by the user. For example, date information can be automatically generated. When the user inputs attribute information, for example, when there is an instruction to add an annotation, an image prompting the user to input attribute information is displayed, and the user inputs attribute information according to the display of the image. . Note that the input timing of attribute information may be the same as or different from the input timing of text as an annotation.

In step S705, the data including the position coordinate information converted in step S701 (absolute position information), the text information acquired in step S702, the display magnification information acquired in step S703, and various attribute information acquired in step S704. Is generated as annotation data.
The absolute position coordinates in the diagnostic image to which the annotation is added can be converted into the position coordinates in the hierarchical image having a magnification different from that of the display diagnostic image at the time of adding the annotation as follows. For example, it is assumed that an annotation is added at the position of a point P (100, 100) where the distance (number of pixels) from the image origin (X = Y = 0) is 100 pixels at a display magnification of 20 times. In this case, in the 40 × high-magnification image, the position coordinate to which the annotation is added is P1 (200, 200). In addition, in the case of a 10 × low-magnification image, the position coordinates where the annotation is added are P2 (50, 50). Here, in order to simplify the description, the description has been given using a display magnification with a sharpness. However, when the display magnification is 25 times and an annotation is added at the position of the point P (100, 100), 40 is used. In the double high-magnification image, the position coordinates to which the annotation is added are P3 (160, 160). Thus, by multiplying the value of the coordinate (the coordinate of the point P) by the ratio between the magnification of the hierarchical image and the display magnification, the absolute position coordinate in the diagnostic image to which the annotation has been added can be obtained when the annotation is added. Can be converted into position coordinates in a hierarchical image having a magnification different from that of the displayed diagnostic image. By performing such conversion, it is possible to indicate the annotation addition position even when displaying a hierarchical image having a different magnification from the display diagnostic image at the time of annotation addition. In this embodiment, the absolute position in each hierarchical image is calculated in this step, and information on the calculated position is included in the annotation data.

  In step S706, the annotation data generation unit 305 determines whether an annotation has been added since the diagnosis (display) of the diagnostic image is started. If an annotation is added for the first time, the process proceeds to step S708. If an annotation has been added even once in the past, the process proceeds to step S707.

  In step S707, the annotation data generation unit 305 updates the annotation list created in step S708. Specifically, the annotation data generation unit 305 adds the annotation data generated in step S705 to the already recorded annotation list.

  In step S708, the annotation data generation unit 305 generates an annotation list. Specifically, the annotation data generation unit 305 generates an annotation list including the annotation data generated in step S705. The structure of the annotation list will be described later with reference to FIG.

(Search attention position)
FIG. 8 is a flowchart showing a detailed flow of the process of searching for the target position shown in step S608 of FIG. Hereinafter, the flow of processing for searching for a position of interest based on an annotation list and generating a search result list will be described with reference to FIG.

  In step S801, the annotation search processing unit 307 determines whether the search for the target position is a search using a word included in the annotation as a key. If a search is performed using a word included in the annotation as a key, the process proceeds to step S802. If a search is performed using attribute information as a key, the process proceeds to step S805.

In step S <b> 802, the annotation search processing unit 307 acquires a word (keyword) that is a search key from the user input information acquisition unit 303. For example, the user inputs a keyword using a keyboard, a mouse, or the like, or selects a keyword from past search history. In accordance with the user operation, a keyword is sent from the user input information acquisition unit 303 to the annotation search processing unit 307.
In step S803, the annotation search processing unit 307 acquires text data (annotation) stored in the annotation list generated and updated in step S707 or step S708.
In step S804, the annotation search processing unit 307 searches the plurality of text data acquired in step S803 with the keyword acquired in step S802. Here, it is possible to use a general keyword search method such as exact matching with a keyword or matching with some words in the keyword.

In step S <b> 805, the annotation search processing unit 307 acquires attribute information that is a search key from the user input information acquisition unit 303. The attribute information as the search key is selected from a plurality of options. The attribute information as the search key may be input (selected) in the same manner as the keyword.
The configuration of the display image when setting the search key will be described later with reference to FIG.
In step S806, the annotation search processing unit 307 acquires attribute information stored in the annotation list.
In step S807, the annotation search processing unit 307 searches the attribute information acquired in step S806 using the attribute information (search key) acquired in step S805. When attribute information is selected from a plurality of choices (categories) at the time of annotation addition, a general method such as detecting attribute information whose category matches the attribute information that is a search key may be used.
In both steps S804 and S807, the search method is not limited to the above method, and a widely known search method according to the purpose can be used.

  In step S808, the annotation search processing unit 307 lists the search results of steps S804 and S807. For example, a list (search result list) of annotation data including annotation data including text data detected in step S804 and annotation data including attribute information detected in step S807 is created.

(Display of search results for attention positions)
FIG. 9 is a flowchart showing a detailed flow of the process of displaying the search result of the target position shown in step S610 of FIG. Hereinafter, the flow of processing for representing a search result based on the search result list will be described with reference to FIG. In this embodiment, a diagnostic image is displayed on the display device 103 as a search result, and candidate positions are indicated in the diagnostic image. That is, as a search result, an image showing candidate positions in the diagnostic image is displayed.

  In step S901, the display data generation control unit 308 acquires the search result list generated in step S808 from the annotation search processing unit 307.

  In step S902, the display data generation control unit 308 determines the range (display range) of the diagnostic image to be displayed on the screen based on the position information (that is, the position information of the candidate position) of the annotation data included in the acquired search result list. calculate. In the present embodiment, when a plurality of candidate positions are detected by the search, a display range (display position and display magnification) that includes all candidate positions is calculated. Specifically, a minimum display range that includes all candidate positions is calculated.

  In step S903, the display data generation control unit 308 determines whether the display magnification when displaying the search result is different from the current display magnification, and the display position when displaying the search result is different from the current display position. Whether or not it is necessary to update the display image data. In general, the display magnification (about 5 to 10 times) in screening that comprehensively observes the entire image data, the display magnification (20 to 40 times) in detailed observation, and the search result of the target position are confirmed. It is expected that it is different from the display magnification (the magnification calculated in step S902). Therefore, it is necessary to perform this step. If the display image data needs to be updated, the process proceeds to step S904. If it is not necessary to update the display image data, the process proceeds to step S905.

  In step S904, the display image data acquisition unit 309 acquires a diagnostic image corresponding to the display magnification (display magnification calculated in step S902) when displaying the search result according to the determination result in step S903.

  In step S905, the display data generation control unit 308 determines whether the number of candidate positions is greater than a predetermined number. The threshold value (predetermined number) used for determination can be arbitrarily set. If the number of candidate positions is greater than the predetermined number, the process proceeds to step S906, and if the number of candidate positions is equal to or less than the predetermined number, the process proceeds to step S907.

  In step S906, the display data generation control unit 308 selects a pointer display mode. The pointer display mode is a mode in which a candidate position is indicated in a diagnostic image using an icon image. Then, the display data generation unit 310 generates pointer display data according to the selection. The pointer display data is image data in which the pointer is located at the candidate position.

  In step S907, the display data generation control unit 308 selects an annotation display mode. The annotation display mode is a mode in which candidate positions are indicated in a diagnostic image using a corresponding annotation (text) image. Then, the display data generation unit 310 generates annotation display data according to the selection. The annotation display data is image data in which a corresponding annotation image is located at a candidate position.

  In step S908, the display data generation unit 310 combines the display data (pointer display data and annotation display data) generated in step S906 or S907 and the diagnostic image data of the display magnification calculated in step S902. Then, display data as a search result is generated. Specifically, image data is generated by superimposing a pointer display image or an annotation display image on the diagnostic image having the display magnification calculated in step S902.

In step S909, the display data output unit 311 outputs the display data generated in step 908 to the display device 103.
In step S910, the display device 103 updates the screen (display image) so that an image based on the display data output in step S909 is displayed.

That is, in this embodiment, when the number of candidate positions is larger than a predetermined number, an image in which the candidate positions are indicated in the diagnostic image using the icon image is displayed as a search result. An example of a display image in the pointer display mode will be described later with reference to FIG. Further, when the number of candidate positions is equal to or less than a predetermined number, an image in which the candidate positions are indicated in the diagnostic image using the corresponding annotation image is displayed as a search result.
When the ratio of the annotation image area to the screen display area is large, it is difficult to observe the diagnostic image. By adopting the above configuration, it is possible to suppress the annotation image from obstructing the observation of the diagnostic image. In this embodiment, the display mode is selected according to the number of candidate positions. However, a configuration in which the user can select the display mode may be used.

  In step S911, the display data generation control unit 308 determines whether the current display mode is the annotation display mode or the pointer display mode. If the current display mode is the pointer display mode, the process proceeds to step S912. If the current display mode is the annotation display mode, the process proceeds to step S914.

In step S912, the display data generation control unit 308 selects the icon (icon image) displayed on the screen based on the information from the user input information acquisition unit 303, or the user uses the mouse. Determine whether the cursor is over the icon. When the icon is selected or when the mouse cursor is held over the icon, the process proceeds to step 913. If not, the process ends.
In step S913, the display data generation unit 310 displays an image of the annotation (text) added to the corresponding icon position (candidate position) in a pop-up according to the determination result in step S912. The annotation image displayed in a pop-up may be deleted (hidden) when the mouse cursor moves off the pointer, or may continue to be displayed until deletion is instructed by a user operation or the like.
By performing the processes of steps S912 and S913, the user can check the content of the annotation (content of the comment) in the pointer display mode.

  In step S914, whether or not the display data generation control unit 308 has selected a candidate position (an annotation image in this embodiment) indicated in the diagnostic image based on information from the user input information acquisition unit 303 or the like. Determine whether. If a candidate position is selected, the process proceeds to step S915. If no candidate position is selected, the process ends.

  In step S915, the display image data acquisition unit 309 selects diagnostic image data having the same magnification as the display magnification when an annotation is added to the selected candidate position, according to the determination result in step S914.

  In step S916, the display data generation unit 310 generates display data based on the annotation data at the candidate position selected in step S914 and the diagnostic image data selected in step S915. Specifically, display data is generated so that the diagnostic image with the annotation is displayed at the display position and display magnification when the annotation is added to the candidate position selected in step S914. By performing this processing, it is possible to perform display that reproduces the display image when the annotation is added.

  In step S <b> 917, the display data output unit 311 outputs the display data generated in step 916 to the display device 103. In step S918, the display device 103 updates the display screen (display image) so that an image based on the display data output in step S917 is displayed.

(Display image layout)
FIG. 10A to FIG. 10E are examples of images (display images) based on display data generated by the image processing apparatus 102 according to the present embodiment.
FIG. 10A shows a basic configuration of a layout of a display image when observing a diagnostic image. In the example of FIG. 10A, the display image is an image in which an information area 1002, a thumbnail image area 1003, and an observation image display area 1005 are arranged in the entire window 1001.
The information area 1002 displays display and operation statuses, information on various images, search images (images for setting search), and search results.
In the thumbnail image area 1003, thumbnail images of the subject to be observed are displayed. In the thumbnail image area 1003, a detailed display area frame 1004 indicating the area being observed is displayed. By displaying the detailed display area frame 1004, it is possible to grasp the position and size of the area under observation (the position and size of an image for detailed observation described later in the thumbnail image).
In the observation image display area 1005, an image for detailed observation is displayed. Specifically, a part or all of the diagnostic image is displayed at a set display magnification as an image for detailed observation. Further, the display magnification of the image for detailed observation is displayed in an area 1006 of the observation image display area 1005. The area of the subject for which detailed observation is performed can be set and updated by a user instruction from an externally connected input device such as a touch panel or a mouse 411, for example. It can also be set and updated by moving the displayed image or by enlarging / reducing (changing the display magnification).
In addition, the configuration may be such that the display area of the entire window 1001 is divided by the single document interface to obtain each of the above-described areas, or the above-described areas may be configured as different window areas by the multi-document interface. Good.

  FIG. 10B is an example of a display image when an annotation is added. In FIG. 10B, the display magnification is set to 20 times. When the user selects a position in the image displayed in the observation image display area 1005, the position is determined as a position of interest, and an annotation is added to the position. For example, when the user designates the position using the mouse pointer, the mode shifts to the annotation input mode, and the user is prompted to input text. Then, when the user inputs text using the keyboard 410, an annotation is added at the designated position. At that time, the image processing apparatus 102 acquires the position information with the annotation added and the display magnification from the display apparatus 103. In addition, attribute information can be set together when an annotation is added. Reference numeral 1009 denotes an image for setting attribute information, for example, an image of a list of settable attribute information. When the user selects and designates desired attribute information from the attribute information list 1009, the selected attribute information is associated with the annotation to be added. FIG. 10B shows a state in which the mouse cursor is confirmed at a position 1007 and text 1008 is input as an annotation.

FIG. 10C shows an example of a display image (image for setting search; setting image) at the time of attention position search.
The setting image 1010 may be displayed in the information area 1002 when searching for the attention position, or may be displayed as a new image. Here, it is assumed that the setting image 1010 is displayed in the information area 1002 when an annotation is added. However, the configuration is not limited to this, and the setting image 1010 may be displayed when the first annotation is added during one diagnosis.
The search for the attention position is performed using a word included in the text as the annotation or attribute information as a search key. Note that both the word included in the text as the annotation and the attribute information may be used as the search key, or one of them may be used as the search key. FIG. 10C shows an example in which information of four types of attributes such as diagnosis information, progress, date and time, and a diagnostician can be set as attribute information that is a search key. However, the types of attributes are not limited to this. Absent.

The user can input a word (keyword) included in the text as an annotation in the text box 1011 as a search key. The user may directly enter a keyword. Alternatively, a list of keywords used in the past may be displayed as a separate window or dialog, and the user may select a word as a search key from the list.
The plurality of radio buttons 1012 correspond to a plurality of attributes, respectively. The user can select the attribute corresponding to the selected radio button as the attribute of the attribute information as the search key by selecting at least one radio button 1012.
Reference numeral 1014 denotes an area in which attribute information used as a search key is displayed. However, even if the attribute information is displayed in the area 1014, if the radio button corresponding to the attribute is not selected, the attribute information is not used as a search key in the search. When attribute information is displayed in the area 1014 and a radio button corresponding to the attribute is selected, the attribute information is used as a search key in the search.

The selection list button 1013 is a button for displaying a list of attribute information (options) of the corresponding attribute. For example, when the user selects a selection list button 1013 corresponding to diagnostic information, an image 1015 of a list of diagnostic information options is displayed as a separate window or dialog. The image 1015 includes a plurality of radio buttons 1016 corresponding to a plurality of options. The user can select and change the search key by selecting and changing one or more radio buttons 1016. For example, a search key can be selected from “attention required” for searching for a point requiring attention, “normal” for searching for a normal point, and “reference comparison” for searching for a point for comparison. Similarly, a search key can be set for the degree of progress. Specifically, a search key can be selected from a plurality of options representing the degree of progression of disease to a cell or tissue (progression level).

When the date / time information is used as a search key, the user may directly input the date / time (annotation addition date, diagnosis date, etc.) in a text box corresponding to the attribute “date / time”. The date / time information list 1017 included in the stored annotation data may be displayed as a separate window or dialog, and the user may select a date / time as a search key from the list. A plurality of dates and times may be used as search keys, or a certain period may be used as a search key.
When user information is used as a search key, the user may directly input a user name or the like in a text box corresponding to the attribute “diagnostic”. A list of registered users may be displayed as a separate window or dialog, and the user may be selected as a search key from the list.

  When a keyword is input and a search button (“Search” in the figure) is selected, a search using a keyword is performed. When attribute information as a search key is set (specifically, when radio button 1012 is selected and corresponding attribute information is set) and a search button is selected, attribute information is used as a key. A search is performed.

FIG. 10D is an example of a display image when the search result is displayed in the pointer display mode. When more annotations are added, the annotation (comment) is hidden as shown in FIG. 10D, and only the icon image 1018 is displayed at the candidate position, so that the candidate position is not hindered. Can be confirmed. In the example of FIG. 10D, a different icon image is shown for each attribute information. With such a configuration, the relationship between the candidate position and the attribute information can be shown, and a desired position (attention position) can be easily found from among the plurality of candidate positions. When an arbitrary icon image is selected, an annotation 1019 is displayed. Thereby, the user can confirm the annotation of the candidate position. Note that the icon image may be different between the candidate position extracted by the search using the keyword and the candidate position extracted by the search using the attribute information.
In the annotation display mode, an annotation 1019 is displayed at each candidate position. In the annotation display mode, the icon image 1018 may be displayed or may not be displayed.

  FIG. 10E is an example of a display image when the candidate position shown in the diagnostic image is selected. When a desired candidate position is selected in annotation display mode or pointer display mode, refer to the annotation list (specifically, the display position and display magnification included in the annotation data at the candidate position) and display when adding annotation Is reproduced. As a result, the annotation 1020 is presented at a position in the screen at the time of addition. In addition, the area of the reproduced and displayed image is displayed as a reproduction range 1022 in the thumbnail image, and the area of the diagnostic image displayed when displaying the search result is displayed as a display frame 1021 in the thumbnail image. Thereby, it is possible to determine the positional relationship between the area that is reproduced and displayed and the area of the diagnostic image that is displayed when the search result is displayed.

(Annotation list example)
FIG. 11 shows the configuration of the annotation list generated by the image processing apparatus 102 according to the present embodiment.
The annotation list is a list of annotation data. An ID number indicating the order in which annotations are added to each annotation data is assigned. As described above, the annotation data includes position information, display magnification, annotation (comment; “annotation content” in the figure), attribute information, and the like, and such information is shown for each ID number in the annotation list. ing. A search using keywords is performed on annotation content, and a search based on attribute information is performed on attribute information. The position information and the display magnification are used to reproduce the display when the annotation is added. The attribute information may be predetermined attribute information, or a new attribute defined by the user may be additionally set.

(Diagnosis criteria, diagnosis classification, warning screen)
FIG. 15A and FIG. 15B are diagrams illustrating an example of a diagnostic criteria setting screen and a diagnostic classification screen.
FIG. 15A is an example of a diagnostic criteria setting screen. For example, it is assumed that the diagnostic criteria can be changed / set by a menu operation in the entire window 1001. Diagnostic criteria include International Codes / International Indicators, International Classification of Cancer Classification I, International Classification of Cancer Classification II, Japanese Regulations / Indicator of Cancer Classification I, Japanese Classification I, Cancer Regulations / Indices The cancer classification US index I and the cancer classification US index II are shown. Cancer Code Japanese Code I is further classified into organs such as stomach cancer, colon cancer, liver cancer, lung cancer, breast cancer, esophageal cancer, thyroid cancer, bladder cancer, prostate cancer and so on. By menu operation, a diagnostic criterion 1501, a list of rules / indexes 1502, and a list of rules / indexes for each organ 1503 are presented. Here, a case where the user selects diagnostic criteria, cancer classification Japanese code I, and gastric cancer by menu operation is shown.
FIG. 15B is an example of a diagnostic classification screen. Cancer classification The Japanese classification I of gastric cancer diagnosis classification 1504 is roughly divided into two, a depth of advancement and a degree of progression. The depth of penetration is an index indicating how far the stomach cancer exceeds the stomach wall, and is classified into four stages from AI to AIV. AI indicates that gastric cancer remains on the surface of the stomach, and AIV indicates that gastric cancer infiltrates to other organs. The degree of progression is an index indicating how far the stomach cancer has metastasized, and is classified into five stages from BI to BV. BI has no gastric cancer metastasis, BV has lymph node metastasis, and so on. In the diagnosis of gastric cancer according to the Japanese classification I of cancer classification, the depth and progression are diagnosed from information such as specimen images. The diagnosis classification 1504 is displayed in the information area 1002 or another window, and the user can operate display / non-display as necessary.
The diagnostic criteria and diagnostic classification shown in FIGS. 15A and 15B are updated as needed, and are not unified by country or region. When a user reviews a sample diagnosed with old diagnostic criteria again, or when checking a sample from another country with a different diagnostic criteria for research, which diagnostic criteria the annotation content described as the basis of diagnosis depends on The reason for specifying the diagnostic criteria and diagnostic classification is to enable easy identification.

(Effect of this embodiment)
As described above, according to this embodiment, when an annotation is added, attribute information that can be used as a search key is stored together with the annotation. Thereby, searching according to various purposes of pathological diagnosis becomes possible, and the user can efficiently find the target position. As a result, the trouble of the user (pathologist) can be reduced.
Further, in this embodiment, when a large number of candidate positions are extracted by the search, it is possible to easily find a desired position (attention position) by specifying the relationship between the candidate position and the attribute information. .

(Another example of annotation)
In the above-described embodiment, an example in which a comment input by a user and attribute information related thereto is added as an annotation to a diagnostic image has been described. However, the information added as an annotation is not limited to that in the above embodiment, and any information may be added as an annotation as long as it is information related to a diagnostic image or a diagnostic work. For example, when a computer (image processing apparatus) has a function (automatic diagnosis software) for automatically detecting various information for supporting diagnosis by analyzing a diagnostic image, based on the detection result Annotations can be automatically generated and added. Even in this case, processing such as annotation recording, retrieval, and display can be performed in the same manner as in the above embodiment.
An example of information automatically generated by a computer is information on a lesion site. Hereinafter, an example of the diagnosis support function of the automatic diagnosis software will be described, and an example of adding information on a lesion site obtained by the diagnosis support function as an annotation to the diagnosis image will be described. In the following, in order to make the difference from the above embodiment easy to understand, an annotation generated based on information automatically detected by the diagnosis support function is referred to as “diagnosis support data”.

(Description of automatic detection of lesion site by diagnosis support function and example of automatic detection algorithm)
Diagnosis support is a function that supports diagnosis by a pathologist, and includes automatic detection of a lesion area in prostate cancer, for example. Prostate cancer tends to have irregular gland duct sizes as the degree of malignancy increases. By detecting the duct and classifying the structure pattern, it is possible to automatically detect the lesion area and determine the malignancy of prostate cancer. For example, texture analysis or the like is used for detection of the ducts. For example, local feature amounts at each spatial position can be extracted by a filter operation using a Gabor filter to detect a ductal region. The classification of the structure pattern can be used as an index of malignancy by calculating the complexity using the cytoplasmic area, lumen area, etc. of the gland duct as the shape feature amount. The lesion area and lesion information (malignancy determination) automatically detected by the diagnosis support are an example of a diagnosis support data list.

  Another example of diagnosis support is automatic detection of ER (estrogen receptor) positive rate in breast cancer. The ER positive rate is an important guideline in deciding the treatment policy for breast cancer. When the IHC (immunohistochemical staining) method is used, nuclei in which ER expression is strongly observed are stained deeply. The ER positive rate is determined by automatic detection of the nucleus and the degree of staining. In this case, image data specifying the automatically detected nuclei, numerical data thereof (number of positive nuclei, ratio, etc.), and positive rate become the diagnosis support data list.

(Example of diagnosis support data list)
FIG. 16 illustrates a configuration of a diagnosis support data list generated by the image processing apparatus 102 according to the present embodiment.
The data (diagnosis support data) related to the lesion site automatically detected by the diagnosis support function includes image data (diagnosis support image data) indicating the lesion area, and therefore the data amount tends to be enormous. Therefore, it is desirable that the image data indicating the lesion area be held separately from the diagnosis support data list. In this case, a pointer to image data indicating a lesion area is described (recorded) in the diagnosis support data list. This pointer is information indicating a position to be noted in the diagnostic image, and is information corresponding to position information in the annotation data list of FIG. In the diagnosis support data list, the position information of the lesion area in the diagnosis image may be recorded instead of the pointer.
The diagnosis support data list is a list of diagnosis support data. An ID number indicating the order in which diagnosis support data is added to each diagnosis support data is assigned. As described above, the diagnosis support data includes an image data pointer, lesion information, and the like, and the information is shown for each ID number in the diagnosis support data list. Search by word is performed on lesion information, and search based on attribute information is performed on attribute information. The attribute information includes the addition date and time, the type / version of the diagnosis support software, and the like. Note that the “lesion information” may be information selected from a predetermined list in the same way as the “diagnosis information” and “progress” in FIG. 10C (that is, the lesion information is included in the attribute information). May be.)

(Search for diagnosis support data)
The search for the diagnosis support data can be performed by the same method as the search for the target position in FIG. “Annotation” in FIG. 8 and the description thereof may be read as “diagnosis support data”.

(Display of diagnosis support data)
The display of the diagnosis support data is a display of the diagnosis support image data indicating the lesion area, which is different from the display image data. However, it is possible to superimpose and display an image showing a lesion area on the display image data by performing automatic detection of the lesion area by the diagnosis support function in more detail. For example, in the automatic diagnosis of prostate cancer, if the malignancy determination for each region such as the malignancy determination V region, IV region, etc. is performed together with the detection of the malignant region, the malignancy determination V of the display image data A region can be displayed in a superimposed manner.

(Effects of diagnosis support data of this embodiment)
As described above, according to this embodiment, when adding diagnosis support data, attribute information that can be used as a search key is stored together with lesion information. As a result, searching according to various purposes of pathological diagnosis becomes possible, and the user can efficiently find a target position such as a lesion area. As a result, the trouble of the user (pathologist) can be reduced.

(Example 2)
Hereinafter, an image processing system according to a second embodiment of the present invention will be described with reference to the drawings.
In the first embodiment, as a search result, an image in which candidate positions are shown in the diagnostic image is displayed on the display device. In this embodiment, a list indicating attribute information corresponding to each candidate position is created as a search result and displayed on the display device. Thereby, it becomes possible to easily grasp the attribute information of the candidate position. Also, when the user selects a candidate position from the list, an image showing the selected candidate position is displayed on the display device. Thereby, it becomes possible to easily grasp the relationship between the candidate position and the attribute information. Hereinafter, differences from the first embodiment will be described in detail, and descriptions of the same configurations and processes as those of the first embodiment will be omitted as much as possible.

(Device configuration of image processing system)
FIG. 12 is a diagram illustrating the configuration of the image processing system according to the present embodiment.
The image processing system according to this embodiment includes an image server 1201, an image processing device 102, a display device 103, and the like. The image processing apparatus 102 acquires diagnostic image data obtained by imaging a subject from the image server 1201 and generates display data to be displayed on the display apparatus 103. The image server 1201 and the image processing apparatus 102 are connected to each other using a general-purpose I / F LAN cable 1203 via a network 1202. The image server 1201 is a computer including a large-capacity storage device that stores diagnostic image data generated by the imaging device 101. In this embodiment, the image server 1201 collects a plurality of diagnostic image data (hierarchical hierarchical image data) having the same subject and different magnifications as a single unit in a local storage connected to the image server 1201. Saved.
The diagnostic image data may be stored in a server group (cloud server) existing somewhere on the network. For example, the diagnostic image data may be divided into a plurality of divided image data and stored in a cloud server. In that case, information for restoring the original image data and information for obtaining a plurality of diagnostic image data having the same subject and different magnifications are generated and held in the image server 1201 as link information. Some of the plurality of diagnostic image data having the same subject and different magnifications may be stored in a server different from other data.
The general functions of the image processing apparatus 102 are the same as those in the first embodiment. The function of the display device 103 is the same as that of the first embodiment.

In the example of FIG. 12, the image server 1201, the image processing apparatus 102, and the display apparatus 103
The image processing system is configured by these three devices, but the configuration of the present invention is not limited to this configuration. For example, an image processing device integrated with a display device may be used, or a part or all of the functions of the image processing device 102 may be incorporated in the image server 1201. Further, the functions of the image server 1201, the image processing apparatus 102, and the display apparatus 103 can be realized by one apparatus. Conversely, the function of one device may be realized by a plurality of devices. For example, the function of the image server 1201 may be realized by a plurality of devices. The function of the image processing apparatus 102 may be realized by a plurality of apparatuses.
In the first embodiment, the image processing apparatus 102 adds an annotation to the diagnostic image data imaged and output by the imaging apparatus 101, and searches the attention position from the diagnostic image data. The position of interest is searched from one diagnostic image (here, a plurality of hierarchical diagnostic images are also regarded as one diagnostic image). In this embodiment, an annotation is added to the diagnostic image stored in the image server 1201, and the position of interest is searched from the diagnostic image stored in the image server 1201. Therefore, the position of interest can be searched from a plurality of diagnostic images. For example, the position of interest can be searched from a plurality of diagnostic images obtained from one patient. Thereby, it is possible to easily perform follow-up observation of one patient, comparison of states at various positions of the same lesion, and the like. Further, by searching for a target position from a plurality of diagnostic images, it is possible to easily grasp annotations that match similar cases and conditions.

(Display of search results for attention positions)
FIG. 13 is a flowchart showing a flow of processing for displaying the search result of the target position in step S610 of FIG. FIG. 13 illustrates a flow in which candidate positions are displayed as a list and the candidate positions are displayed based on the list contents selected by the user.

  In step S901, the display data generation control unit 308 acquires the search result list generated in step S808 from the annotation search processing unit 307.

  In step S1301, the display data generation control unit 308 generates a list (attribute information list) indicating attribute information corresponding to each candidate position, using the search result list acquired in step S901. Then, the display data generation unit 310 generates display data including the attribute information list. The display area of the attribute information list is, for example, the information area 1002 shown in FIG. However, the display area of the attribute information list may be set in addition to this. The display area of the attribute information list may be an independent window area. An example of the display image when displaying the attribute information list will be described later with reference to FIGS. 14 (a) to 14 (c).

  In step S1302, the display data output unit 311 outputs the display data generated in step S1301 to the display device 103, and the display device 103 displays a display image based on the display data.

In this embodiment, the attribute information list is a sortable list.
In step S1303, the display data generation control unit 308 determines whether there is a request for sorting the attribute information list based on information from the user input information acquisition unit 303 or the like. If there is a sort request, the process advances to step S1304. If there is no sort request, the process advances to step S1307.

In step S1304, the display data generation control unit 308 sorts the attribute information list. For example, the attribute information list is sorted in the order of the date and time represented by the date and time information according to the user operation. Specifically, when there is a user operation for sorting in the order of the date and time represented by the date and time information, the display data generation control unit 308 selects the date and time item of the attribute information list in accordance with the operation, and the date and time information Sort attribute information so that is in ascending or descending order.
In step S1305, the display data generation unit 310 displays the attribute information list in step S1.
The display data is updated so that the attribute information list after sorting in 304 is obtained.
In step S1306, the display data output unit 311 outputs the display data updated in step S1305 to the display device 103, and the display device 103 displays a display image based on the display data.

In step S1307, the display data generation control unit 308 determines whether a candidate position has been selected from the displayed attribute information list based on information from the user input information acquisition unit 303 and the like. Only one candidate position may be selected, or a plurality of candidate positions may be selected. If a candidate position is selected, the process proceeds to step S902. If no candidate position is selected, the process ends.
Thereafter, the processes of steps S902 to S904 are performed as in the first embodiment, the processes of steps S907 to S910 are performed as in the first embodiment, and the processes of steps S914 to S918 are performed as in the first embodiment.

(Display image layout)
FIG. 14A to FIG. 14C are examples of images (display images) based on display data generated by the image processing apparatus 102 according to the present embodiment.
FIG. 14A shows an example of an attribute information list displayed as a search result. The attribute information list 1401 includes attribute information for each candidate position. The attribute information list 1401 includes a check box 1402 for selecting a corresponding candidate position from the list, and a sort button 1403 for sorting based on the attribute information. The user can select one or more candidate positions by selecting the check box 1402. In addition, by setting the priority of attribute information, it is possible to perform a sort operation using a plurality of attribute information.

FIG. 14B shows an example of a display image when the candidate position selected using the check box 1402 is displayed in the annotation display mode. Here, an example in which three candidate positions are selected is shown. In the present embodiment, a diagnostic image and an image 1405 of an annotation (an annotation corresponding to the selected candidate position) are displayed at a display position and a display magnification at which all selected candidate positions are displayed. An annotation image 1405 is displayed at the corresponding candidate position 1406. In the example of FIG. 14B, the diagnostic image and the annotation image are displayed at a display magnification of 5 times, which is a low magnification. An annotation image is displayed in a different form for each attribute information (for example, display magnification at the time of annotation addition). For example, assume that the display magnification when annotation 1 is added is 10 times, the display magnification when annotation 2 is added is 20 times, and the display magnification when annotation 3 is added is 40 times. By changing the display method of the annotation image for each display magnification at the time of adding an annotation, the difference in display magnification can be determined. Here, the attribute information is distinguished by the line type of the annotation display frame, but may be distinguished by the color or shape of the display frame.
In this embodiment, when a candidate position is selected using the check box 1402, the selected candidate position is specified in the annotation display mode. However, the selected candidate position is displayed in the pointer display mode. May be specified. As in the first embodiment, the display mode may be switched according to the number of candidate positions to be displayed.

FIG. 14C is an example of an image displayed when a candidate position (annotation image or icon image) shown in the diagnostic image is selected. FIG. 14C shows an example when four candidate positions are selected. In FIG. 14C, the same reproduction display as in FIG. 10E is displayed for each candidate position. Of the four annotations 1 to 4 indicated by reference numeral 1405, the display magnification when annotations 1, 3, and 4 are added is 20 times, and the display magnification when annotation 2 is added is only 40 times. Yes. The display magnification when an annotation is added to each candidate position can be confirmed from the display magnification displayed in the area 1404. In addition, the display magnification difference may be clearly indicated by the color of the frame of the display area of each diagnostic image.
Further, the positional relationship between the diagnostic image area displayed when the candidate position is selected from the list (for example, the diagnostic image area displayed as shown in FIG. 14B) and each area that is reproduced and displayed is It can be determined in the same manner as in Example 1. Specifically, the diagnostic image area displayed when the candidate position is selected from the list is displayed as a display frame 1407 in the thumbnail image, and each reproduced area is displayed as the reproduction range 1408 in the thumbnail image. Is done. The correspondence between the reproduction range 1407 and the image that is reproduced and displayed can be determined by the color of the frame line, the line type, and the like. In addition, by selecting one of a plurality of reproduced images and a plurality of reproduction ranges, an image with a display magnification corresponding to the selected image (reproduction range) is displayed on the entire observation image display area. You may transfer to the display mode to display.

(Effect of this embodiment)
As described above, according to this embodiment, when an annotation is added, attribute information that can be used as a search key is stored together with the annotation. Thereby, searching according to various purposes of pathological diagnosis becomes possible, and the user can efficiently find the target position. As a result, the trouble of the user (pathologist) can be reduced.
In the present embodiment, a list of attribute information for each candidate position is displayed as a result of searching for the target position, and a diagnostic image showing the candidate position selected from the list is displayed. Thereby, it becomes possible to grasp the search result of the attention position more specifically.

(Other examples)
The object of the present invention may be achieved by the following. That is, a recording medium (or storage medium) in which a program code of software that realizes all or part of the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiments, and the recording medium on which the program code is recorded constitutes the present invention.
In addition, when the computer executes the read program code, an operating system (OS) operating on the computer performs part or all of the actual processing based on the instruction of the program code. The case where the functions of the above-described embodiments are realized by the processing can also be included in the present invention.
Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU of the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. It can be included in the invention.
When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.
Further, the configurations described in the first and second embodiments can be combined with each other. For example, the reproduction display processing of a plurality of positions of interest of the second embodiment may be applied to the system of the first embodiment, and the image processing device is connected to both the imaging device and the image server, and the image used for the processing May be obtained from any device. In addition, configurations obtained by appropriately combining various techniques in the above embodiments also belong to the category of the present invention.

DESCRIPTION OF SYMBOLS 102 Image processing apparatus 305 Annotation data generation part 306 Annotation list memory | storage part 307 Annotation search processing part 310 Display data generation part 311 Display data output part

Claims (27)

An adding means for adding an annotation to a diagnostic image obtained by imaging a subject;
Recording means for recording attribute information, which is information of a predetermined attribute, as information related to the annotation in the storage unit together with the annotation;
Search means for searching for a position of interest, which is a position to which the user pays attention, from a plurality of positions with annotations added in the diagnostic image;
Display means for displaying a search result of the search means on a display unit;
With
The image processing apparatus, wherein the search means searches the attention position using a word or attribute information included in the annotation as a key. The image processing apparatus according to claim 1, wherein the attribute information includes diagnostic information representing a diagnostic content of the diagnostic image. Further comprising automatic diagnostic means for automatically detecting information for supporting diagnosis by analyzing the diagnostic image;
The image processing apparatus according to claim 1, wherein the annotation includes an annotation added based on a detection result of the automatic diagnosis unit. The annotation added based on the detection result of the automatic diagnosis means includes information for supporting the diagnosis as text information,
The image processing apparatus according to claim 3, wherein the search unit performs a search using a word included in information for supporting the diagnosis as a key. The image processing apparatus according to claim 1, wherein the attribute information includes information indicating a diagnostic criterion used when diagnosing the diagnostic image. 6. The image processing according to claim 1, wherein the attribute information includes date and time information indicating a date and time when a corresponding annotation is added, or a date and time when the diagnostic image is observed. apparatus. The image processing apparatus according to claim 1, wherein the attribute information includes user information that identifies a user who has added an annotation. A plurality of users sequentially add annotations to the diagnostic image for different purposes or different methods,
The user information includes a user attribute indicating a purpose or a method when the user adds an annotation,
The image processing apparatus according to claim 7, wherein the search unit searches the attention position using the user attribute as a key. The image processing apparatus according to claim 1, 2, 5, 6, 7, or 8, wherein the attribute information is information input by a user who has added an annotation. The search means detects a candidate position that is a candidate for the target position from a plurality of positions added with annotations by the adding means by performing a search,
The image processing apparatus according to claim 1, wherein the display unit displays a list indicating attribute information corresponding to each candidate position on the display unit as the search result. The attribute information includes date and time information indicating the date and time when the corresponding annotation was added, or the date and time when the diagnostic image was observed,
The image processing apparatus according to claim 10, wherein the attribute information list is sortable in order of date and time represented by date and time information. The display means displays the diagnostic image on the display unit and indicates the selected candidate position in the diagnostic image when a user selects a candidate position from the list. The image processing apparatus according to 10 or 11. When the user selects a plurality of candidate positions from the list, the display means displays the diagnostic image on the display unit at a display position and a display magnification that include all the selected candidate positions, The image processing apparatus according to claim 10, wherein the selected candidate position is indicated in the diagnostic image. The search means detects a candidate position that is a candidate for the target position from a plurality of positions added with annotations by the adding means by performing a search,
10. The display device according to claim 1, wherein the display unit displays the diagnostic image on the display unit as the search result, and indicates the candidate position in the diagnostic image. 11. Image processing device. When a plurality of candidate positions are detected by the search means, the display means displays the diagnostic image on the display unit at a display position and display magnification that include all candidate positions, and the diagnostic image The image processing apparatus according to claim 14, wherein the candidate position is indicated within the image. The image processing apparatus according to claim 12, wherein the display unit indicates the candidate position using a corresponding annotation image. The image processing apparatus according to claim 12, wherein the display unit indicates the candidate position using an icon image. The display means includes
When the number of candidate positions to be displayed is a predetermined number or less, the candidate position is indicated using a corresponding annotation image;
18. The image processing apparatus according to claim 12, wherein when the number of candidate positions to be displayed is greater than a predetermined number, the candidate positions are indicated using an icon image. The image processing apparatus according to claim 17, wherein the icon image is different for each attribute information. The image processing apparatus according to claim 16, wherein the annotation image is different for each piece of attribute information. The display unit displays the diagnostic image on the display unit at a display position and a display magnification when an annotation is added to the candidate position when the indicated candidate position is selected by a user. The image processing apparatus according to claim 14 or 15. The image processing apparatus according to any one of claims 1 to 21, wherein the search unit searches the attention position from one diagnostic image. The image processing apparatus according to any one of claims 1 to 21, wherein the search unit searches the attention position from a plurality of diagnostic images.   24. The image processing apparatus according to claim 23, wherein the search unit searches the attention position from a plurality of diagnostic images obtained from one patient.   An image processing system comprising: the image processing apparatus according to claim 1; and the display unit. An adding step in which a computer adds an annotation to a diagnostic image obtained by imaging a subject;
A recording step in which the computer records attribute information, which is information of a predetermined attribute, as information related to the annotation in the storage unit together with the annotation;
A search step in which a computer searches for a target position that is a position that a user pays attention to from among a plurality of positions that are annotated in a diagnostic image;
A computer displaying a search result of the search step on a display unit;
Including
In the search step, the attention position is searched using a word or attribute information included in the annotation as a key.   A program causing a computer to execute each step of the image processing method according to claim 26.

Images