JP2018120227A - Display method, information processing device and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display method, an information processing device and a storage medium for helping avoid the risk of overlooking an object when observed with a microscope by a user.SOLUTION: Provided is a method of displaying a medical image. The method includes: displaying the medical image in a first region; displaying at least a portion of the medical image in a second region as a partial display region; when the partial display region is displayed with prescribed observation magnifying power or higher, displaying a mask image at a pixel position of the medical image displayed in the first region that corresponds to the partial display region displayed in the second region; and, when the partial display region is displayed with less than the prescribed observation magnifying power, displaying a pixel of the medical image displayed in the first region that corresponds to the partial display region displayed in the second region with the same color as before being displayed.SELECTED DRAWING: Figure 14

Description

  The present technology relates to a method for displaying an image obtained by a microscope, an information processing apparatus, and a storage medium in fields such as medical treatment, pathology, living organisms, and materials.

  In the field of medical treatment or pathology, etc., images of living cells, tissues, organs, etc. obtained by an optical microscope are digitized, and based on the digital images, doctors and pathologists examine the tissues, etc. A system for diagnosing the problem has been proposed.

  For example, in the method described in Patent Document 1, an image optically obtained by a microscope is digitized by a video camera equipped with a CCD (Charge Coupled Device), and the digital signal is input to a control computer system for monitoring. Is visualized. The pathologist performs an examination or the like by looking at the image displayed on the monitor (see, for example, paragraphs [0027] and [0028] and FIG. 5 of Patent Document 1).

  Further, as a means for preventing a pathologist from observing an observation in a pathological image, a technique for recording a history of observing a pathological image has been disclosed (for example, Patent Document 2).

JP 2009-37250 A JP 2011-112523 A

  Generally, the observation area of a microscope becomes narrower with respect to the entire observation object as the observation magnification is higher. For example, a pathologist often observes an entire observation target by using a microscope and inspects the target by observing a part of the entire observation target at a particularly high magnification. In such an examination, if there is a disease in an area that the pathologist does not see among the observation objects, that is, if there is an oversight, a serious problem may occur later.

  In view of the circumstances as described above, an object of the present invention is to provide a display method, an information processing apparatus, and a storage medium for avoiding the danger of oversight of an observation object by a microscope by a user. .

In order to achieve the above object, a display method according to an aspect of the present technology is as follows.
A method for displaying medical images,
Display the medical image in the first area,
Displaying at least a part of the medical image as a partial display area in the second area;
When the partial display area is displayed at a predetermined observation magnification or higher, mask the pixel positions corresponding to the partial display area displayed in the second area of the medical image displayed in the first area. Display an image,
When the partial display area is displayed at a magnification less than a predetermined observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is changed. It is the same as before the display.

  The second area of the medical image displayed in the first area when the partial display area is displayed at a predetermined observation magnification or more and the display range of the medical image is constant for a predetermined time or more. A mask image may be displayed at a pixel position corresponding to the partial display area displayed on the screen.

  When the partial display area is displayed at a magnification lower than the predetermined observation magnification, the medical image displayed in the first area is displayed in the second area regardless of the display time of the partial display area. The color of the pixel corresponding to the partial display area may be the same as that before the display.

  The medical image may be a pathological image of at least a part of a living tissue.

  The medical image may be a pathological image acquired from an image management server capable of storing a pathological image.

  The partial display area displayed in the second area is superimposed on the medical image with a frame indicating which part of the medical image displayed in the first area is enlarged. Also good.

  The partial display area displayed in the second area may be changed in conjunction with the movement of the frame in the first area based on an instruction from the user.

  The observation magnification at which the partial display area is displayed in the second area may be changed in conjunction with a change in the size of the frame in the first area based on an instruction from the user.

  When the partial display area is displayed at a predetermined observation magnification or higher, the position information indicating which part of the medical image is displayed as the partial display area, or the partial display area is displayed in the second area. The time that is being recorded may be stored.

  The mask image may be an image representing an observation magnification at which the partial display area is displayed.

  The mask image may be displayed as a gradation according to the observation magnification at which the partial display area is displayed.

  The mask image may be cumulatively combined with the medical image, and the combined result may be generated as a trajectory image indicating a movement trajectory of the range displayed as the partial display area.

In order to achieve the above object, a display method according to an aspect of the present technology is as follows.
A method for displaying medical images,
Display the medical image in the first area,
Displaying at least a part of the medical image as a partial display area in the second area;
When the partial display area is displayed at a predetermined observation magnification or higher, history is recorded at a pixel position corresponding to the partial display area displayed in the second area of the medical image displayed in the first area. Display an image,
When the partial display area is displayed at a magnification less than a predetermined observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is changed. It is the same as before the display.

  The history image may be an image indicating which portion of the medical image is displayed as the partial display region, the positional information thereof, or the observation magnification of the partial display region.

  Based on an instruction from the user, the position information and the observation magnification at which the partial display area is displayed may be periodically recorded.

  Based on an instruction from the user, periodic recording of the position information and the observation magnification at which the partial display area is displayed may be stopped.

In order to achieve the above object, an information processing apparatus according to an aspect of the present technology is provided.
A medical image is displayed in the first area of the screen of the display unit, at least a part of the medical image is displayed as a partial display area in the second area of the screen, and the partial display area is greater than or equal to a predetermined observation magnification. When displayed, a mask image is displayed at a pixel position corresponding to the partial display area displayed in the second area of the medical image displayed in the first area, and the partial display area is predetermined. When the display magnification is less than the observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is the same as that before the display. An arithmetic processing circuit is provided.

In order to achieve the above object, a storage medium according to an embodiment of the present technology is provided.
On the computer,
Display the medical image in the first area of the display screen,
Displaying at least a part of the medical image as a partial display area in the second area of the screen;
When the partial display area is displayed at a predetermined observation magnification or higher, mask the pixel positions corresponding to the partial display area displayed in the second area of the medical image displayed in the first area. Display an image,
When the partial display area is displayed at a magnification less than a predetermined observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is changed. A program for executing the same thing as before the display is stored.

  As described above, according to the present technology, it is possible to avoid a risk of an oversight of an observation target by a user with a microscope.

It is a figure showing the typical use environment of the viewer computer 500 which concerns on this technique. It is a block diagram showing the hardware constitutions of viewer computer 500 concerning this art. 3 is a diagram showing functional blocks of an image management server 400. FIG. It is a figure which shows the functional block of the viewer computer. It is a figure which shows the example of a viewer screen. It is a sequence diagram explaining the flow which records / reproduces the history of screen display by viewer operation. It is a figure which shows the locus | trajectory image by which the pathology whole image and the display locus | trajectory were synthesize | combined. It is a figure which shows a mode that the pathology whole image A and the mask image B are another images, and have overlapped them. It is a graph showing how to increase the alpha value according to an example. It is a graph which shows a mode that the increase amount of an alpha value is changed when observing a fixed place for a long time. It is a figure which shows the example recorded on the mask image in the state which the rectangle R corresponding to the observation area | region 62 rotated. It is a flowchart explaining the flow of the process which produces | generates a locus | trajectory image. It is a figure showing the process in which a user browses while displaying the sample SPL imaged in the observation area. It is a figure which shows the example which recorded on the whole pathology image in the thumbnail map 61 as a display locus | trajectory the process in which the user browses while image | photographing the specimen SPL displayed on the observation area | region 62. FIG. It is a figure which shows the example of the locus | trajectory image synthesize | combined with the mask image converted into the binary image. It is an example of a display history list screen which a display history control part 52 presents to a user.

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.
<First Embodiment>

[Viewer computer usage environment]
First, in a pathological diagnosis, an overall image of an environment in which a pathologist makes a diagnosis using a virtual slide image (pathological image) obtained by photographing a specimen using a microscope will be described. The pathologist uses a viewer on a viewer computer to observe a pathological image and perform image diagnosis. FIG. 1 is a diagram illustrating a typical usage environment of a viewer computer 500 according to the present technology.

  A scanner 100 including a microscope 10 and a scanner computer 20 is installed in a histology laboratory HL in a hospital. The RAW image photographed by the microscope 10 is subjected to image processing such as development processing, shading processing, color balance correction, gamma correction, and 8-bit processing on the scanner computer 20. Thereafter, the image is divided into tiles of 256 pixels in length and width, converted into JPEG (Joint Photographic Experts Group) images, compressed, and stored on the hard disk HD1.

  The JPEG image stored in the hard disk HD1 of the scanner computer 20 is then uploaded via the network 300 to the hard disk HD2 on the image management server 400 in the data center DC in the same hospital.

  The pathologist who is an observer is stored in the hard disk HD2 of the image management server 400 using the viewer computer 500 connected to the image management server 400 and the network 300 in the pathology room PR in the hospital or the building EX outside the hospital. Observe the JPEG image.

  In addition, the pathologist who is an observer instructs the viewer computer 500 to display, as a display history, how the display of the JPEG image has changed on the viewer screen due to the operation performed when observing the JPEG image. It can be recorded. The recorded display history is sent to the image management server 400 via the network 300 and stored.

  Furthermore, the pathologist can call the display history stored in the image management server 400 by instructing the viewer computer 500, and reproduce the state of observation of the JPEG image performed previously on the viewer.

[Outline of this technology]
Next, an outline of the present technology will be described. Until now, it has been performed to record a trajectory obtained by observing a pathological image using the viewer computer 500 as an image superimposed on the pathological image. However, it has been desired that the pathologist observes which part of the pathological image it took and how much time it has been observed as a locus on the pathological image in an easy-to-understand manner.

  Therefore, in this technique, as a recording method of the trajectory, a transparent mask image is overlaid on the entire pathological image, and the transparency of the mask image is changed by the time when the pathologist displays a specific location or the observation magnification, The track is recorded. In the pathological image, the transparency of the displayed part is reduced (by the color of the mask image), the position of the locus is recorded, and the display time is expressed by the degree of transparency reduction (increase in the color of the mask image).

  By performing such recording, it can be verified later whether the pathological image has been overlooked, and the observation position and the observation time when the pathological image is observed can be easily understood.

[Configuration of Viewer Computer 500]
Next, the hardware configuration of the viewer computer 500 will be described.
FIG. 2 is a block diagram illustrating a hardware configuration of the viewer computer 500 according to the present technology.

  The viewer computer 500 receives a CPU (Central Processing Unit) 21 that performs arithmetic control, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23 that is a work memory of the CPU 21, and instructions according to user operations. An operation input unit 24 (input unit) is provided. The viewer computer 500 further includes an interface unit 25, an output unit 26 (display unit), a storage unit 27, a network interface unit 28, and a bus 29 for connecting them together.

  The ROM 22 stores programs for executing various processes. A controller 30 and a camera 31 are connected to the interface unit 25. The controller 30 is provided with various buttons and sticks and can accept various inputs by the user.

  The controller 30 has a built-in acceleration sensor and tilt sensor, and can accept an instruction given to the controller 30 when the user tilts or shakes the controller 30. The camera 31 is for photographing the face of a user who observes a pathological image using the viewer computer 500.

  A network 300 is connected to the network interface unit 28. A liquid crystal display, an EL (Electro Luminescence) display, a plasma display, or the like is applied to the output unit 26 for image display, and a speaker or the like is applied for audio output. For the storage unit 27, a magnetic disk represented by an HDD (Hard Disk Drive), a semiconductor memory, an optical disk, or the like is applied.

  The CPU 21 develops, in the RAM 23, a program corresponding to an instruction given from the operation input unit 24 among a plurality of programs stored in the ROM 22, the storage unit 27, and the like, and according to the expanded program, the output unit 26 and the storage unit 27 is appropriately controlled.

  CPU21 implement | achieves each function block mentioned later. The CPU 21 executes programs stored in the ROM 22, the storage unit 27, and the like, and controls each of the above members as necessary. Thereby, the viewer computer 500 can implement | achieve various functional blocks, and can operate each said member as the viewer computer 500. FIG.

[Configuration of Image Management Server 400]
Next, the hardware configuration of the image management server 400 will be described.
The hardware configuration of the image management server is basically the same as the hardware configuration of the viewer computer 500 except that the controller 30 and the camera 31 are not connected to the interface unit 25. Therefore, detailed description is omitted.

[Functional Block of Image Management Server 400]
Next, functional blocks of the image management server 400 will be described. First, the main function of the image management server 400 is to provide a pathological image according to a request from the viewer computer 500. Secondly, the display history acquired from the viewer computer 500 is stored and provided according to a request from the viewer computer 500.

  Thirdly, a comment (hereinafter referred to as an annotation) attached by a pathologist to a specific portion of a pathological image on the viewer is saved. FIG. 3 is a diagram showing functional blocks of the image management server 400.

  The image management server 400 includes functional blocks of an image storage unit 41, an image providing unit 42, a display history storage unit 43, and a display history management unit 44.

  The image storage unit 41 stores pathological images that are divided into tiles and compressed by JPEG. The stored pathological image is provided to the viewer computer 500 via the image providing unit 42 in accordance with a request from the viewer computer 500. Also, annotations attached to the pathological image by the user using the viewer on the viewer computer 500 are stored here.

  The image providing unit 42 acquires a pathological image corresponding to the image request transmitted from the viewer computer 500 via the network 300 from the image storage unit 41 and transmits the pathological image to the viewer computer 500 via the network 300.

  The display history storage unit 43 stores the display history of the viewer operated by the user on the viewer computer 500.

  The display history management unit 44 acquires the display history recorded on the viewer computer 500 and once collected via the network 300. Then, the display history management unit 44 stores the acquired display history in the display history storage unit 43. Further, the display history management unit 44 receives a display history request from the viewer computer 500, acquires a display history corresponding to the display history request from the display history storage unit 43, and transmits the display history to the viewer computer 500 via the network 300. .

  Since the image management server 400 and the viewer computer 500 constitute a client-server system, what functions are provided on the client side and which functions are provided on the server side is a design matter. Therefore, the execution location of each functional block is not limited to the above-described image management server 400, but may be configured to be executed on the viewer computer 500 on the client side.

[Function Blocks of Viewer Computer 500]
Next, functional blocks of the viewer computer 500 will be described. The main function of the viewer computer 500 is to first accept an operation instruction from a user who is a pathologist, obtain a corresponding pathological image from the image management server 400, and display it to the user. Secondly, the displayed range and the displayed time of the pathological image are represented as a trajectory on the pathological image to generate a trajectory image.

  Third, the display of an image corresponding to the viewer operation when the user performs image diagnosis is recorded, and the display history is sent to the image management server 400 for storage. Fourth, in accordance with a request from the user, the display history stored in the image management server 400 is acquired, and the display of the image corresponding to the operation performed by the user is reproduced based on the display history for the user. is there.

  FIG. 4 is a diagram illustrating functional blocks of the viewer computer 500.

  The viewer computer 500 includes functional blocks of an image acquisition unit 51 (acquisition unit), a display history control unit 52, and a trajectory image generation unit 53 (generation unit).

  The image acquisition unit 51 acquires, from the image management server 400 via the network 300, the pathological image corresponding to the instruction of the user who is a pathologist input from the operation input unit 24, and outputs the acquired pathological image to the output unit 26. Present to the user via

  The display history control unit 52 records a change in screen display according to a viewer operation when the user observes a pathological image in accordance with a user instruction. The recording is first saved using the RAM 23 or the storage unit 27 of the viewer computer 500. In response to the recording stop instruction, the recordings are collected, sent to the image management server 400 as a display history, and stored.

  The display history control unit 52 also acquires a display history corresponding to the instruction from the image management server 400 according to the user's instruction, and displays the viewer screen display recorded in the acquired display history via the output unit 26. And present it to the user.

  The display history control unit 52 further passes to the trajectory image generation unit 53 information indicating which part of the pathological image is displayed on the viewer screen and what the display time is.

  The trajectory image generation unit 53 acquires, from the display history control unit 52, the position information indicating which part of the pathological image is currently displayed and the time information indicating how long the path image is displayed, and the mask image. Decrease the pixel transparency. Details of how to lower the transparency will be described later.

[About Viewer Screen]
Next, a viewer screen used for the user to observe a pathological image on the viewer computer 500 will be described. FIG. 5 is a diagram illustrating an example of a viewer screen.

  The viewer window 60 includes a thumbnail map 61 indicating which part of the pathological image is magnified, an observation area 62 for observing the pathological image, and a display recording / reproducing GUI 63. The thumbnail map 61 includes a reduced image of the entire pathological image (hereinafter referred to as an entire pathological image) and a frame FR that equivalently shows the range of the image displayed in the viewer window 60 on the thumbnail map 61.

  The frame FR can be moved on the thumbnail map 61 by an arbitrary amount in an arbitrary direction according to an instruction from the user. In the thumbnail map 61, the operation of moving the frame can also be performed by a drag operation with a mouse or the like.

  The display recording / reproducing GUI 63 receives a recording start instruction or a recording stop instruction for a change in the display screen according to a viewer operation by the user, and transmits the received instruction to the display history control unit 52.

[Flow of recording and playback on the viewer screen display]
Next, the flow of recording / reproducing the history of screen display by the viewer operation will be described. FIG. 6 is a sequence diagram for explaining the flow of recording / reproducing the history of screen display by the viewer operation.

First, the flow of recording the display history will be described.
First, the user instructs the display history control unit 52 to start recording of the viewer display by clicking the record button of the display recording / reproducing GUI 63 (S1). Thereafter, the user selects a pathological image to be observed from the list of pathological images displayed on the viewer.

  After the display history control unit 52 receives an instruction to start recording, the display history control unit 52 periodically records changes in the screen display due to the change of the display position performed by the user with respect to the viewer and the change of the observation magnification. (S2).

  When the display range is changed or the observation magnification is changed by the user, the image acquisition unit 51 requests the image management server 400 for a corresponding tiled image and acquires it (S3).

  The image acquired from the image management server 400 is displayed on the screen via the image acquisition unit 51 (S4). At this time, the trajectory image generation unit 53 converts the range displayed on the observation area 62 of the pathological image into the entire pathological image in the thumbnail map 61 while changing the transparency of the mask image according to the displayed time and observation magnification. Record.

  The user continues to operate the viewer screen (S5), and the image acquisition unit 51 displays the pathological image on the viewer screen (S6). Meanwhile, the screen display state by the display history control unit 52 is continuously recorded as a display history. In addition, the trajectory (including location and time) of the movement of the observation region 62 by the trajectory image generation unit 53 is generated by being synthesized as a trajectory image with the entire pathological image in the thumbnail map 61.

  The user clicks the stop button of the display recording / reproducing GUI 63 to instruct the display history control unit 52 to stop recording of the viewer display (S7). At this time, the recorded display history is given a name. When a stop instruction is given, the display history control unit 52 transmits the display history temporarily stored locally to the image management server 400 (S8). The display history management unit 44 stores the received display history in the display history storage unit 43.

  The above is the flow of recording the display history. Next, the flow of reproducing the display history will be described.

  First, the user selects the name of the display history to be reproduced from the display history list and clicks the reproduction button of the display recording / reproducing GUI 63 to instruct the display history control unit 52 to reproduce the display history (S9). In the display history list, in addition to the display history name, a trajectory image corresponding to the display history is additionally displayed as a thumbnail.

  When playback of the display history is instructed, the display history control unit 52 requests the display history specified by the user from the display history management unit 44 of the image management server 400 and acquires it from the image management server 400 ( S10).

  When a display history reproduction instruction is given, the image acquisition unit 51 acquires an image to be displayed at the time of reproduction from the image storage unit 41 of the image management server 400 (S11).

  The display on the viewer screen is reproduced using the acquired display history and image (S12).

Finally, when the user clicks the stop button of the display recording / playback GUI 63 to instruct the display history control unit 52 to stop playback of the display history (S13), the display history control unit 52 stops playback of the display history. .
The above is the flow of reproducing the display history.

[Alpha value and alpha blend]
Next, details of generating a trajectory image will be described. First, alpha values and alpha blending used when a mask image is overlaid on the entire pathological image displayed in the thumbnail map 61 to express a display locus will be described.

  The alpha value is transparency information set for each pixel in digital image data handled by a computer. Alpha blending is to synthesize two images with a coefficient (alpha value). In the case of the present technology, the two images are an entire pathological image in the thumbnail map 61 and a mask image set in front of it.

  FIG. 7 is a diagram illustrating a trajectory image in which the entire pathological image and the display trajectory are combined. FIG. 8 is a diagram illustrating a state in which the entire pathological image A and the mask image B are different images and they are superimposed. FIG.

  As can be seen from these figures, the entire pathology image and the mask image are different images. The trajectory image generation unit 53 adjusts the alpha value representing the transparency of the mask image, and records the display trajectory on the mask image. Then, a trajectory image is generated by synthesizing the mask image with the adjusted alpha value and the entire pathological image by alpha blending.

  The alpha value takes an integer value ranging from 0 to 255, for example. When the alpha value of a certain pixel is 0, the pixel of the mask image is completely transparent, and the pixel of the entire pathological image behind is completely transparent. When the alpha value is about 128, the pixels of the mask image are translucent and colored (for example, green). At that time, only half of the color of the pixel of the entire pathological image behind is seen through. When the alpha value is 255, the pixels of the mask image are completely opaque and the color of the entire pathology image behind is completely invisible.

  In this technique, the transparency of the mask image is initially made completely transparent, and according to the time displayed in the observation area 62, the alpha value is increased and the transparency of the mask image is decreased to obtain the color of the mask image. The trajectory of the display is recorded by following. Conversely, the transparency of the mask image is initially set to about 70%, the alpha value is decreased according to the time displayed in the observation area 62, the transparency of the mask image is increased, and the color of the mask image is erased. It may be configured to record the display trajectory by moving.

  Since the mask image is an image different from the entire pathological image, the recording of the trajectory can be reset by returning all the alpha values of the mask image to 0.

[Method of adding alpha value (basic)]
In the present technology, when displaying at least a part of a pathological image in the observation region 62, display is performed while redrawing is repeated using a frame rate of, for example, 60 fps like a moving image. The same applies to the thumbnail map 61. Therefore, for example, when a certain range of the pathological image is continuously displayed in the observation area 62, it is conceivable to increase the alpha value by 1 every frame. In this way, after 60 frames (one second) have elapsed, the alpha value of the pixel of the mask image corresponding to the position displayed in the observation area becomes 60, which is opaque by about 23%.

  If the display time exceeds 4 seconds, the alpha value reaches 255, and the mask image becomes completely opaque. In this case, since the entire pathological image in the background cannot be seen, it is difficult for the user to grasp where the pathological image has been observed by comparing the shape of the entire pathological image with the display trajectory. Therefore, it is preferable to set an upper limit for the alpha value. For example, if the upper limit of the alpha value to be added is 180, the increase of the alpha value stops at a transparency of about 70%, so that the entire pathological image in the back is not invisible.

  In the above example, the configuration in which the alpha value is increased by 1 for each frame is adopted. However, for example, a configuration in which the alpha value is increased by 1 every 30 seconds may be used. In this way, since it takes 90 minutes for the alpha value to reach the upper limit of 180, even when recording a long-time observation, it is possible to record the display trajectory appropriately with a difference in transparency depending on the time. In any case, it is preferable to determine the increase rate of the alpha value based on the length of typical observation time.

[Method of adding alpha value (considering observation magnification)]
In the above description, when a certain range of the pathological image is continuously displayed in the observation area 62, the alpha value is unconditionally increased. However, the rate of increase of the alpha value is different from the value depending on the observation magnification for observing the pathological image. It is also possible to do. When the color of the trajectory in the trajectory image is darker (the higher the opacity of the mask image), the more the pathological image is observed, the higher the opacity is, the longer the observation time at one place is. Similarly, the higher the observation magnification, the more detailed observation is performed, so it is better to increase the rate of increase of the alpha value.

  For example, when the observation magnification is less than 2, the increase amount of the alpha value per unit time is set to 0, and the locus is not recorded. When the observation magnification is 2 times or more and less than 4 times, the increase amount of the alpha value per unit time is set to 1. When the observation magnification is 4 times or more, the increase amount of the alpha value per unit time is set to 2. By adopting such a configuration, it is possible to record the display locus in consideration of the observation magnification.

  FIG. 9 is a graph showing how the alpha value increases along this example. As shown in the upper graph of the figure, when the observation magnification is less than 2, the alpha value remains zero and does not increase over time. When the observation magnification is 2 times or more and less than 4 times, the alpha value gradually increases. When the observation magnification is 4 times or more, the observation magnification suddenly increases. However, when the upper limit is reached as described above, the observation magnification does not increase.

[Addition method of alpha value (considering observation time)]
In the above configuration, when the observation of a certain place is continued, the alpha value increases monotonously. However, observing a certain place for a long time is nothing but observing in detail so that it is desirable to increase the rate of increase of the alpha value.

  FIG. 10 is a graph showing how the amount of increase in the alpha value is changed when observing a certain place for a long time. For example, the alpha value is increased by n per unit time until the time t1 elapses after a certain point is displayed in the observation area 62 and observation is started. When the time t1 elapses, the increase amount of the alpha value is increased 1.1 times to 1.1n per unit time.

  When the same portion is continuously observed and time t2 elapses, the amount of increase in alpha value is increased by a factor of 1.2 to 1.2n per unit time. As described above, the value of n varies depending on the observation magnification. When the image to be displayed in the observation area 62 is moved, the increase amount of the alpha value also starts from n.

  By adopting such a configuration, when a certain part is observed for a long time, the locus can be emphasized and recorded.

[Recording method for rotating display]
Next, a recording method when the image is rotated when a part of the pathological image is displayed in the observation area 62 will be described. Basically, the alpha value of the pixel included in the rectangular area corresponding to the observation area 62 is only increased as in the case where the rotation is not performed. However, since the rotation is performed, the recording is performed on the mask image with the rectangle corresponding to the observation region 62 rotated. FIG. 11 is a diagram illustrating an example recorded on the mask image in a state where the rectangle R corresponding to the observation region 62 is rotated.

[Flow of trace image generation]
Next, a flow of processing in which the trajectory image generation unit 53 generates a trajectory image will be described. FIG. 12 is a flowchart for explaining the flow of processing for generating a trajectory image. As described above, since the update of the trajectory image is performed for each frame (for example, every 1/60 seconds for 60 fps), the processing of this flowchart is also performed at the timing for each frame.

  First, the trajectory image generation unit 53 determines an alpha value based on the current observation magnification (step ST1).

  Next, the trajectory image generation unit 53 determines whether or not a predetermined time has elapsed since the start of displaying the current image in the observation area 62, and increases the alpha value if the predetermined time has elapsed. (Step ST2).

  Next, the trajectory image generation unit 53 determines a rectangular range for changing the alpha value on the mask image from the range of the image displayed in the observation area 62 (step ST3).

  Next, the trajectory image generation unit 53 records a rectangle as a trajectory in the mask image (step ST4). Here, recording is performed by adding the increment of the alpha value determined in step ST1 or ST2 to the alpha value of the target pixel on the mask image. When recording is performed, a rectangle indicating the range of the observation region 62 is displayed on the entire pathological image on the thumbnail map 61 by the color of the mask image.

Here, the trajectory image generation unit 53 determines whether or not a request for resetting the display trajectory has been received from the operation input unit 24. If a reset request has been input (Y in step ST5), the trajectory image generation unit 53 is on the thumbnail map 61. All the trajectories are deleted (step ST6). Erasing is performed by returning the alpha values of all the pixels of the mask image to the initial values.
The above is the flow of processing for generating a trajectory image by the trajectory image generation unit 53.

[Actual example of a trajectory image]
Next, an actual example of a trajectory image generated on the thumbnail map 61 by the trajectory image generation unit 53 will be described.

  In the following, first, an example of what time distribution the user has used to observe the pathological image will be described, and what color shading will indicate the display of the trajectory corresponding to the observation .

  FIG. 13 is a diagram showing a process in which the user browses the specimen SPL taken while displaying it in the observation area 62. FIG. 14 is a diagram showing an example in which the display process is recorded on the entire pathological image in the thumbnail map 61 as a display locus.

  How the user displays the pathological image in the observation area 62 will be described with reference to FIG.

  First, the user displays the range of the upper part D1 of the specimen SPL as a partial image in the observation region 62 at an observation magnification of 1.25, and performs observation for 8 seconds. The center coordinates of the display range D1 are (x1, y1).

  Next, the user changes the display range of the partial image from D1 to D2, and observes for 20 seconds. The center coordinates of the partial image display range D2 are (x2, y2).

  Next, the user enlarges the observation magnification from 1.25 times to 20 times, displays the display range D3 of the partial image, and performs observation for 35 seconds. At this time, the center coordinates of the partial image remain unchanged (x2, y2).

  Next, the user moves the display range D3 of the partial image to the display range D4 and performs observation for 40 seconds. The center coordinates of the partial image are (x3, y3).

  Next, the user enlarges the observation magnification from 20 times to 40 times, displays the display range D5 of the partial image, and performs observation for 2 minutes. At this time, the center coordinates of the partial image remain unchanged (x3, y3).

  Hereinafter, it is assumed that the observation is performed in the same manner while changing the observation magnification and the observation time.

  Next, a recording example of the display trajectory of the above-described observation process will be described with reference to FIG.

  The trajectories T1, T2, T7, etc., where the observation magnification is low or the observation time is short, are recorded with lighter colors and higher transparency. On the other hand, trajectories shown most opaque by the darkest color are T5, T9, T10, and the like. In particular, the trajectory T5 and the like are indicated by the density obtained by cumulatively adding the color density of the trajectory T2, the color density of the trajectory T4, and the original color density of the trajectory T5. Will be recorded.

[Application example (display with threshold)]
Next, as an application example when recording a trajectory image, an example in which a trajectory image is displayed with a threshold value is shown.

  As described above, the alpha value of the mask image is a continuous value that gradually increases from 0. For this reason, when displayed on the trajectory image, the color of the mask image is displayed as a gradation, and the boundary line where the observer often observed is not clear.

  Therefore, a predetermined threshold value is set in advance, and if the alpha value is equal to or greater than this threshold value, the alpha value is increased to the maximum value to make the mask image completely opaque, and if less than the threshold value, the alpha value is decreased to the minimum value, Make the mask image completely transparent. In this way, by converting the mask image from an image having a continuous gradation into a binary image, it is possible to easily distinguish between an observed part and an unobserved part.

  When making the track opaque, in the above description, the alpha value is raised to the maximum value to make it completely opaque. However, the alpha value is set so that it becomes the upper limit with a value lower than the maximum value as the upper limit. May be.

  FIG. 15 is a diagram illustrating an example of a locus image combined with a mask image converted into a binary image. Since the trajectories T3, T4, T5, T6, T9, T10, etc. are expressed by opaque colors, it is easy to see where you have observed well.

[Application example (using thumbnails of trajectory images)]
Next, as an application example of the trajectory image, when the user selects a display history to be reproduced from the display history stored in the image management server 400, the trajectory image is displayed as a thumbnail of each display history displayed in the list. Can be considered.

  FIG. 16 is an example of a display history list screen that the display history control unit 52 presents to the user. Each row represents one display history, and a thumbnail of the trajectory image, a display history name, and a recording time are displayed in one row from the left.

  The point to be noted in this example is the position of the locus shown in the thumbnail. Without these thumbnails, the user must search for the display history that he / she wants to see from the display history name and recording time, which is difficult. On the other hand, if there is a thumbnail of the trajectory image, it is obvious at a glance which position in the pathological image is the display history when observed.

  In this way, in order to use a trajectory image that records which part of the pathological image is observed as a thumbnail when a list of display histories is presented, it is stored in association with display history attribute information. It is useful to leave.

[Another configuration of this technology]
In addition, this technique can also take the following structures.
(1) An acquisition unit that acquires a pathological image, a display unit that displays at least a part of the acquired pathological image as a partial display region, an input unit that receives an instruction to move the partial display region from a user, and the display For each pixel position of the displayed partial display area, a composite image given a value corresponding to the length of display time of the partial display area is generated at a constant time period and accumulated for the pathological image An information processing apparatus comprising: a generation unit configured to generate a trajectory image indicating a trajectory of movement of a range displayed as the partial display area.
(2) The information processing apparatus according to (1), wherein the generation unit corrects the value corresponding to the display time length of the range according to a height of an observation magnification, An information processing apparatus that generates images at a constant time period.
(3) The information processing apparatus according to (1) or (2), wherein the generation unit displays the displayed portion as the display time increases when a display range of the partial display region is constant. An information processing apparatus that increases a luminance value of a trajectory color and decreases a luminance value of the pathological image using a color representing the trajectory in the trajectory image as a trajectory color for all pixel positions in a display area.
(4) The information processing apparatus according to (3), wherein the generation unit is configured to display the locus color when the display range of the partial display area is constant and the display time exceeds a predetermined time. An information processing apparatus that increases the increase rate of the luminance value stepwise.
(5) The information processing apparatus according to (3) or (4), wherein the generation unit has the luminance value of the trajectory color equal to or greater than a predetermined threshold value for all pixel positions of the trajectory image. If so, an information processing apparatus that represents the color of the pixel of the trajectory image only by the trajectory color, and that represents the color of the pixel of the trajectory image only by the color of the pathological image if the luminance value is less than the threshold value.
(6) The information processing apparatus according to any one of (1) to (5), wherein the generation unit uses an alpha value as the value and performs alpha blending as a synthesis method for each pixel. Information processing device to be used.
(7) The information processing apparatus according to any one of (1) to (6), wherein the generation unit associates at least position information of the partial display area in the pathological image with a display time. An information processing apparatus for adding the trajectory image as a thumbnail to a display history periodically recorded.
(8) The acquisition unit acquires a pathological image, the display unit displays at least a part of the acquired pathological image as a partial display region, and the input unit receives an instruction to move the partial display region from the user. The generation unit generates a combined image given a value corresponding to the length of time of display of the partial display area for each pixel position of the displayed partial display area at a constant time period; An information processing method that cumulatively synthesizes the pathological image and generates the synthesis result as a trajectory image indicating a trajectory of movement of the range displayed as the partial display area.
(9) An acquisition unit that acquires a pathological image, a display unit that displays at least a part of the acquired pathological image as a partial display region, an input unit that receives an instruction to move the partial display region from the user, and the displayed For each pixel position of the partial display area, a synthesized image is generated at a constant time period and given a value corresponding to the display time length of the partial display area, and cumulatively applied to the pathological image. An information processing program for causing a computer to function as a generation unit that combines and generates the combined result as a trajectory image indicating a trajectory of movement of a range displayed as the partial display area.

[Additional notes]
In addition, the present technology is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the present invention.

10 ... Microscope 20 ... Computer for scanner 21 ... CPU
22 ... ROM
23 ... RAM
DESCRIPTION OF SYMBOLS 24 ... Operation input part 25 ... Interface part 26 ... Output part 27 ... Memory | storage part 28 ... Network interface part 29 ... Bus 30 ... Controller 31 ... Camera 41 ... Image storage part 42 ... Image provision part 43 ... Display history storage part 44 ... Display History management unit 51 ... Image acquisition unit 52 ... Display history control unit 53 ... Trajectory image generation unit 60 ... Viewer window 61 ... Thumbnail map 62 ... Observation area 63 ... Display recording / playback GUI
DESCRIPTION OF SYMBOLS 100 ... Scanner 300 ... Network 400 ... Image management server 500 ... Viewer computer

Claims (18)

A method for displaying medical images,
Displaying the medical image in a first region;
Displaying at least a part of the medical image as a partial display area in the second area;
When the partial display area is displayed at a predetermined observation magnification or higher, a mask is provided at a pixel position corresponding to the partial display area displayed in the second area of the medical image displayed in the first area. Display an image,
When the partial display area is displayed at a magnification lower than a predetermined observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is changed. The same display method as before the display. The display method according to claim 1,
The second area of the medical image displayed in the first area when the partial display area is displayed at a predetermined observation magnification or more and the display range of the medical image is constant for a predetermined time or more. A display method for displaying a mask image at a pixel position corresponding to the partial display area displayed on the display. The display method according to claim 1 or 2,
When the partial display area is displayed at a magnification less than a predetermined observation magnification, the medical image displayed in the first area is displayed in the second area regardless of the display time of the partial display area. A display method in which the color of the pixel corresponding to the partial display area is the same as that before the display. A display method according to any one of claims 1 to 3,
The display method, wherein the medical image is a pathological image of at least a part of a living tissue. A display method according to any one of claims 1 to 4, wherein
The medical image is a pathological image acquired from an image management server capable of storing a pathological image. A display method according to any one of claims 1 to 5,
The partial display area displayed in the second area superimposes a frame indicating which part of the medical image displayed in the first area is enlarged on the medical image. Method. The display method according to claim 6,
A display method in which the partial display area displayed in the second area is changed in conjunction with the movement of the frame in the first area based on an instruction from a user. The display method according to claim 6 or 7, wherein
A display method in which an observation magnification at which the partial display area is displayed in the second area is changed in conjunction with a change in the size of the frame in the first area based on an instruction from a user. The display method according to any one of claims 1 to 8,
When the partial display area is displayed at a predetermined observation magnification or higher, which part of the medical image is displayed as the partial display area, or its position information is displayed in the second area. Memorized time display method. A display method according to any one of claims 1 to 9,
The mask image is an image representing an observation magnification at which the partial display area is displayed. A display method according to any one of claims 1 to 10,
The mask image is displayed as a gradation according to an observation magnification at which the partial display area is displayed. The display method according to any one of claims 1 to 11,
A display method in which the mask image is cumulatively combined with the medical image, and the combined result is generated as a trajectory image indicating a movement trajectory of the range displayed as the partial display area. A method for displaying medical images,
Displaying the medical image in a first region;
Displaying at least a part of the medical image as a partial display area in the second area;
When the partial display area is displayed at a predetermined observation magnification or higher, a history is recorded at a pixel position corresponding to the partial display area displayed in the second area of the medical image displayed in the first area. Display an image,
When the partial display area is displayed at a magnification lower than a predetermined observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is changed. The same display method as before the display. The display method according to claim 13,
The history image is an image indicating which portion of the medical image is displayed as the partial display area, its positional information, or an observation magnification at which the partial display area is displayed. The display method according to claim 13 or 14,
A display method in which the position information and the observation magnification at which the partial display area is displayed are periodically recorded based on an instruction from a user. The display method according to claim 15, comprising:
A display method in which periodic recording of the position information and the observation magnification at which the partial display area is displayed is stopped based on an instruction from a user. A medical image is displayed in a first area of the screen of the display unit, at least a part of the medical image is displayed as a partial display area in the second area of the screen, and the partial display area is greater than or equal to a predetermined observation magnification. When displayed, a mask image is displayed at a pixel position corresponding to the partial display area displayed in the second area of the medical image displayed in the first area, and the partial display area is predetermined. When the display magnification is less than the observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is the same as that before the display. An information processing apparatus comprising an arithmetic processing circuit. On the computer,
Display the medical image in the first area of the display screen,
Displaying at least a portion of the medical image as a partial display area in the second area of the screen;
When the partial display area is displayed at a predetermined observation magnification or higher, a mask is provided at a pixel position corresponding to the partial display area displayed in the second area of the medical image displayed in the first area. Display an image,
When the partial display area is displayed at a magnification lower than a predetermined observation magnification, the color of the pixel corresponding to the partial display area displayed in the second area of the medical image displayed in the first area is changed. A computer-readable storage medium storing a program for executing the same operation as before the display.