KR20230097945A - A method and an apparatus for displaying pathology slide images - Google Patents

Abstract

A method of outputting a pathology slide image according to an aspect includes receiving a user input related to an output method of at least one region included in a pathology slide image; determining an area of a guide to be output to the pathology slide image and an area of the pathology slide image included in the guide based on the user input; and outputting a pathology slide image overlaid with the guide based on the determined area and region.

Description

A method and apparatus for outputting an image of a pathology slide. {A method and an apparatus for displaying pathology slide images}

A method and apparatus for outputting an image of a pathology slide.

The field of digital pathology is a field of obtaining histological information or predicting a prognosis of a patient using a whole slide image generated by scanning a pathological slide image.

In the field of conventional pathology, a pathologist places a glass slide on an optical microscope, adjusts the magnification and focus of the microscope, and reads the slide image while moving the glass slide up and down or left and right. Meanwhile, in the field of digital pathology, a pathologist outputs a pathology slide image converted into an image file on a monitor screen, adjusts magnification with a mouse or keyboard, and reads the slide image while moving the slide.

Accordingly, there is an increasing demand for a technology capable of resolving side effects that may occur due to a difference between an image reading method in the field of conventional pathology and an image reading method in the field of digital pathology.

An object of the present invention is to provide a method and apparatus for outputting an image of a pathology slide. In addition, it is to provide a computer-readable recording medium recording a program for executing the method on a computer. The technical problem to be solved is not limited to the technical problems described above, and other technical problems may exist.

A method of outputting a pathology slide image according to an aspect includes receiving a user input related to an output method of at least one region included in a pathology slide image; determining an area of a guide to be output to the pathology slide image and an area of the pathology slide image included in the guide based on the user input; and outputting a pathology slide image overlaid with the guide based on the determined area and region.

A computer-readable recording medium according to another aspect includes a recording medium recording a program for executing the above-described method on a computer.

A computing device according to another aspect includes at least one memory; and at least one processor, wherein the processor determines an area of a guide to be output to the pathology slide image based on a user input related to an output method of at least one region included in the pathology slide image and the pathology slide image. An area of the pathology slide image included in the guide is determined, and a display device is controlled to output a pathology slide image overlaid with the guide based on the determined area and area.

1 is a diagram for explaining an example of a system for outputting a pathology slide image according to an exemplary embodiment.
2 is a configuration diagram illustrating an example of a user terminal 10 according to an embodiment.
3 is a flowchart illustrating an example of a method of outputting a pathology slide image according to an exemplary embodiment.
4A and 4B are diagrams for explaining an example in which a pathology slide image overlaid with a guide is output, according to an exemplary embodiment.
5A and 5B are diagrams for explaining another example in which a pathology slide image overlaid with a guide is output, according to an exemplary embodiment.
6 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.
7A and 7B are diagrams for explaining another example of outputting a pathology slide image according to an exemplary embodiment.
8 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.
9A to 9C are diagrams for explaining an example in which an observed region and other regions are divided according to a predetermined criterion according to an exemplary embodiment.
10 is a diagram for explaining an example of outputting a thumbnail image according to an exemplary embodiment.
11 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.
12 is a diagram for explaining an example in which a processor operates according to an exemplary embodiment.
13 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.
14 is a diagram for explaining an example in which a notification signal is generated according to an exemplary embodiment.

The terms used in the embodiments have been selected as general terms that are currently widely used as much as possible, but they may vary according to the intention of a person skilled in the art or a precedent, the emergence of new technologies, and the like. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description. Therefore, terms used in the specification should be defined based on the meaning of the term and the overall content of the specification, not simply the name of the term.

When it is said that a certain part "includes" a certain component throughout the specification, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as “??unit” and “??module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. there is.

Also, terms including ordinal numbers such as “first” or “second” used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, embodiments may be implemented in many different forms and are not limited to the examples described herein.

1 is a diagram for explaining an example of a system for outputting a pathology slide image according to an exemplary embodiment.

Referring to FIG. 1 , a system 1 includes a user terminal 10 and a server 20 . For example, the user terminal 10 and the server 20 may be connected in a wired or wireless communication method to transmit and receive data (eg, image data, etc.) between them.

For convenience of description, FIG. 1 shows that the system 1 includes the user terminal 10 and the server 20, but is not limited thereto. For example, the system 1 may include other external devices (not shown), and the operation of the user terminal 10 and the server 20 to be described below may be performed by a single device (eg, the user terminal 10). Alternatively, it may be implemented by the server 20).

The user terminal 10 outputs the pathology slide image 40 , and the user 30 may manipulate the user terminal 10 in various ways to observe the pathology slide image 40 from various angles.

Here, the pathology slide image 40 may refer to a scanned image of a pathology slide that has been fixed and stained through a series of chemical treatments in order to observe a tissue removed from the human body under a microscope. As an example, the pathology slide image 40 may refer to a whole slide image including high-resolution images of the entire slide. As another example, the pathology slide image 40 may refer to a portion of such a high-resolution overall slide image.

Meanwhile, the pathology slide image 40 may refer to a patch region divided in patch units from the entire slide image. For example, a patch may have a certain area size. Alternatively, a patch may refer to a region including each of included objects within the entire slide.

Also, the pathology slide image 40 may refer to a digital image obtained by converting a pathology slide into an image file using a digital scanner, and may refer to a digital image of cells, tissues and/or structures in the human body. information may be included.

The user terminal 10 may store and output the pathology slide image 40 , receive a user input from the user 30 , overlay the guide on the pathology slide image 40 , and output the result. For example, the user terminal 10 may enlarge or reduce at least one region of the pathology slide image 40 and check the region observed by the user 30 in the pathology slide image 40 . In addition, the user terminal 10 may store the pathology slide image 40 updated by the user 30's manipulation. In addition, the user 30 may record and store information on findings obtained by observing the pathology slide image 40 in the user terminal 10 .

The user terminal 10 may transmit and receive various data as well as the pathology slide image 40 through communication with the server 20 and/or other external devices (not shown). In addition, the user terminal 10 may store the transmitted/received data itself or request the server 20 and/or other external device (not shown) to store the transmitted/received data.

The user terminal 10 may be a computing device including a display device and a device (eg, a keyboard, mouse, etc.) that receives a user input, and includes a memory and a processor. For example, the user terminal 10 may include a notebook PC, a desktop PC, a laptop, a tablet computer, a smart phone, and the like, but is not limited thereto.

The server 20 may be a device that communicates with an external device (not shown) including the user terminal 10 . As an example, the server 20 may be a device that stores various data including the pathology slide image 40 . As another example, the server 20 may be a computing device including a memory and a processor and having its own computing capability. When the server 20 is a computing device, the server 20 may perform at least some of the operations of the user terminal 10 to be described later with reference to FIGS. 1 to 14 . For example, the server 20 may be a cloud server, but is not limited thereto.

The user 30 observes the pathology slide image 40 output on the user terminal 10, manipulates the user terminal 10 on which the pathology slide image 40 is output, or observes the pathology slide image 40. It may be a subject capable of recording opinions. For example, the user 30 may be a pathologist, but is not limited thereto.

In the traditional field of pathology, a pathologist reads an image by touching an objective lens and a driving control knob with his hand while viewing a circular image through the lens of an optical microscope. Meanwhile, in the field of digital pathology, the user 30 observes the pathology slide image 40 output on the display device and reads the image through a computer and a mouse.

For pathologists who are accustomed to reading images with a conventional optical microscope, a method of adjusting the magnification of a slide image or searching for an area of interest (eg, a lesion area, etc.) due to an interface that is different depending on digital pathology. It is not easy to adapt to Accordingly, problems such as taking a long time to read the pathology slide image 40 or inaccurate reading results may occur.

For example, in the case of breast cancer, histological examination of breast cancer based on the number of mitosis of cancer cells confirmed in an area 10 times larger than the range that can be observed in a high power field (HPF) of an optical microscope. rating is determined. However, since it is difficult to estimate the range of HPF visible on a microscope on the pathology slide image 40 , it may not be easy for a pathologist to determine an accurate histological grade based on the pathology slide image 40 .

According to an embodiment, the user terminal 10 determines the area of the guide to be output to the pathology slide image 40 based on a user input and the area included in the guide (ie, the area included in the guide among the pathology slide images 40). decide Then, the user terminal 10 outputs the pathology slide image overlaid with the guide according to the determination result. Accordingly, the user terminal 10 can output an image that can be read in the same manner as reading an image with an optical microscope. Accordingly, even the user 30 who is accustomed to reading images with an existing optical microscope can easily read images according to the field of digital pathology.

In addition, the user terminal 10 according to an embodiment may distinguish and output a region observed by the user and other regions (ie, unobserved regions) in the pathology slide image 40 . Accordingly, the user 30 can intuitively check whether there is an unobserved area in the pathology slide image 40 .

Also, the user terminal 10 according to an embodiment determines whether the region of interest is included in the region observed by the user in the pathology slide image 40 . Accordingly, the user 30 can intuitively check whether or not the region of interest has been fully observed.

Hereinafter, a method for the user terminal 10 to output a pathology slide image will be described in detail with reference to FIGS. 2 to 14 .

Meanwhile, for convenience of explanation, throughout the specification, the user terminal 10 determines the area of the guide to be output to the pathology slide image 40 and the area included in the guide, and the pathology slide image 40 overlaid with the guide. ) has been described as outputting, but is not limited thereto. For example, at least some of the operations performed by the user terminal 10 may be performed by the server 20 .

In other words, at least some of the operations of the user terminal 10 described with reference to FIGS. 1 to 11 may be performed by the server 20 . For example, the server 20 may determine the area of the guide to be output on the pathology slide image 40 and the area included in the guide, and transmit the determination result to the user terminal 10 . Further, the user terminal 10 is provided with a pathology slide overlaid with a guide according to information transmitted from the server 20 (for example, a pathology slide image, information about the area of the guide, information about the area included in the guide, etc.) The image 40 can be output. However, the operation of the server 20 is not limited to the above.

2A is a configuration diagram illustrating an example of a user terminal 10 according to an embodiment.

Referring to FIG. 2A , the user terminal 100 includes a processor 110, a memory 120, an input/output interface 130, and a communication module 140. For convenience of description, only components related to the present invention are shown in FIG. 2A. Accordingly, in addition to the components shown in FIG. 2A , other general-purpose components may be further included in the user terminal 100 . In addition, the processor 110, the memory 120, the input/output interface 130, and the communication module 140 shown in FIG. 2A may be implemented as independent devices with ordinary knowledge in the technical field related to the present invention. be self-evident

The processor 110 may process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Here, the command may be provided from the memory 120 or an external device (eg, the server 20). Also, the processor 110 may overall control operations of other components included in the user terminal 100 .

In particular, the processor 110 determines the area of the guide to be output on the pathology slide image and the area of the pathology slide image included in the guide based on a user input related to an output method of at least one region included in the pathology slide image. can determine Also, the processor 110 may control the display device to output a pathology slide image overlaid with a guide based on the determined area and region.

In addition, the processor 110 may check the region observed by the user according to a predetermined criterion in the pathology slide image, and may control the display device to distinguish and output the observed region and other regions according to the check result.

Also, the processor 110 may set at least one region of interest in the pathology slide image, and check a region observed by the user according to a predetermined criterion in the pathology slide image. Also, the processor 110 may determine whether the region observed by the user includes the region of interest. Also, when a user input related to the end of observation of the pathology slide image is received, the processor 110 may generate a notification signal indicating whether to additionally observe the pathology slide image based on the determination result.

The processor 110 may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory in which programs executable by the microprocessor are stored. For example, the processor 110 may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, the processor 110 may include an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. For example, processor 110 may be a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors coupled with a digital signal processor (DSP) core, or any other such combination. It may also refer to a combination of processing devices, such as a combination of configurations.

Specific examples of the operation of the processor 110 according to an embodiment will be described with reference to FIGS. 3 to 14 .

Memory 120 may include any non-transitory computer readable recording medium. As an example, the memory 120 may include a permanent mass storage device such as random access memory (RAM), read only memory (ROM), a disk drive, a solid state drive (SSD), and flash memory. device) may be included. As another example, a non-perishable mass storage device such as a ROM, SSD, flash memory, or disk drive may be a separate permanent storage device from memory. In addition, the memory 120 may store an operating system (OS) and at least one program code (eg, a code for the processor 110 to perform an operation to be described later with reference to FIGS. 3 to 14 ).

These software components may be loaded from a computer-readable recording medium separate from the memory 120 . The recording medium readable by such a separate computer may be a recording medium that can be directly connected to the user terminal 100, and for example, a floppy drive, a disk, a tape, a DVD/CD-ROM drive, a memory card, etc. A readable recording medium may be included. Alternatively, the software components may be loaded into the memory 120 through the communication module 140 rather than a computer-readable recording medium. For example, at least one program is a computer program installed by files provided by developers or a file distribution system that distributes application installation files through the communication module 140 (eg, FIGS. 3 to 14 It may be loaded into the memory 120 based on a computer program for the processor 110 to perform an operation to be described later with reference to .

The input/output interface 130 may be a means for interface with a device (eg, keyboard, mouse, etc.) for input or output that may be connected to the user terminal 100 or included in the user terminal 100 . In FIG. 2 , the input/output interface 130 is shown as a separate component from the processor 110 , but is not limited thereto, and the input/output interface 130 may be included in the processor 110 .

The communication module 140 may provide a configuration or function for the server 20 and the user terminal 100 to communicate with each other through a network. Also, the communication module 140 may provide a configuration or function for the user terminal 100 to communicate with other external devices. For example, control signals, commands, data, etc. provided under the control of the processor 110 may be transmitted to the server 20 and/or an external device via the communication module 140 and a network.

Meanwhile, although not shown in FIG. 2A , the user terminal 100 may further include a display device. Alternatively, the user terminal 100 may be connected to an independent display device through a wired or wireless communication method to transmit and receive data between them.

2B is a configuration diagram illustrating an example of a server according to an embodiment.

Referring to FIG. 2B , the server 20 includes a processor 210 , a memory 220 and a communication module 230 . For convenience of description, only components related to the present invention are shown in FIG. 2B. Therefore, in addition to the components shown in FIG. 2B, other general-purpose components may be further included in the server 200. In addition, it is apparent to those skilled in the art that the processor 210, memory 220, and communication module 230 shown in FIG. 2B may be implemented as independent devices.

The processor 210 may determine the area of the guide to be output to the pathology slide image and the area of the pathology slide image included in the guide based on a user input related to an output method of at least one region included in the pathology slide image. In addition, the processor 210 may transmit related information to the user terminal 10 so that the pathology slide image on which the guide is overlaid is output on the user terminal 10 .

Also, the processor 210 may check a region observed by the user according to a predetermined criterion in the pathology slide image.

Also, the processor 210 may set at least one region of interest in the pathology slide image, and check a region observed by the user according to a predetermined criterion in the pathology slide image. Also, the processor 210 may determine whether the region observed by the user includes the region of interest. Also, when a user input related to the end of observation of the pathology slide image is received, the processor 210 may generate a notification signal indicating whether to additionally observe the pathology slide image based on the determination result.

In other words, the operation of the processor 110 described above with reference to FIG. 2A may be performed by the processor 210 . In this case, the user terminal 10 may output the information transmitted from the server 20 through the display device.

Meanwhile, since an implementation example of the processor 210 is the same as the implementation example of the processor 110 described above with reference to FIG. 2A , a detailed description thereof will be omitted.

The memory 220 may store various data such as pathology slide images and data generated according to the operation of the processor 210 . Also, an operating system (OS) and at least one program (eg, a program required for the processor 210 to operate) may be stored in the memory 220 .

Meanwhile, since an implementation example of the memory 220 is the same as the implementation example of the memory 220 described above with reference to FIG. 2A , a detailed description thereof will be omitted.

The communication module 230 may provide a configuration or function for the server 200 and the user terminal 100 to communicate with each other through a network. Also, the communication module 140 may provide a configuration or function for the server 200 to communicate with other external devices. For example, control signals, commands, data, etc. provided under the control of the processor 210 may be transmitted to the user terminal 100 and/or an external device via the communication module 230 and a network.

3 is a flowchart illustrating an example of a method of outputting a pathology slide image according to an exemplary embodiment.

Referring to FIG. 3 , the method of outputting a pathology slide image is composed of steps sequentially processed by the user terminal 10 or 100 or the processor 110 shown in FIGS. 1 and 2 . Therefore, even if the content is omitted below, the content described above regarding the user terminals 10 and 100 or the processor 110 shown in FIGS. 1 and 2 can be applied to the method of outputting the pathology slide image of FIG. 3. there is.

In step 310, the user terminal 10 or 100 receives a user input related to an output method of at least one region included in the pathology slide image.

For example, the pathology slide image 40 is output to the display device of the user terminal 10 or 100, and then the user terminal 10 or 100 may receive a user input. User input may be received through a separate image (image configured to receive user input) output on the display device, received through movement of a mouse wheel, or received through a shortcut key input preset by the user. Not limited to this.

The user input may be manipulation of a region in the pathology slide image 40 that the user 30 wishes to closely observe. For example, the user 30 may enlarge or reduce a partial area of the pathology slide image 40 through a user input.

In step 320, the processor 110 determines the area of the guide to be output to the pathology slide image 40 and the area of the pathology slide image 40 included in the guide based on the user input.

As an example, the processor 110 may enlarge or reduce the area of the guide based on a magnification specified according to a user input, while the area of the pathology slide image 40 included in the guide is fixed. As another example, the processor 110 may enlarge or reduce the area of the pathology slide image 40 included in the guide based on a magnification specified according to a user input, while the area of the guide is fixed.

Here, the guide may limit an area that the user 30 wishes to closely observe. When an image reading operation is performed with an optical microscope according to the prior art, the user 30 observes the image through an eyepiece. Accordingly, the guide may have a shape appearing on the pathology slide image 40 to be the same as an area observed by the user 30 through the eyepiece.

Meanwhile, although not shown in FIG. 3 , the processor 110 may calculate a scan magnification and a length per pixel of the pathology slide image 40 . Accordingly, the processor 110 may calculate the area of the guide to be displayed on the pathology slide image 40 and the area of the pathology slide image 40 included in the guide according to a user input.

In step 330, the processor 110 outputs a pathology slide image overlaid with a guide based on the determined area and region. For example, the processor 110 may control the display device to output a pathology slide image overlaid with a guide.

The processor 110 manipulates (eg, enlarges or reduces) the pathology slide image 40 according to the area of the guide determined in step 320 and the area of the pathology slide image 40 included in the guide, so that the image is displayed on the pathology slide. You can overlay guides.

Hereinafter, examples in which a pathology slide image overlaid with a guide is output will be described with reference to FIGS. 4A to 5B .

4A and 4B are diagrams for explaining an example in which a pathology slide image overlaid with a guide is output, according to an exemplary embodiment.

Referring to FIG. 4A , a guide 410 is overlaid on the pathology slide image. The guide 410 is a mark that distinguishes an area that the user 30 wants to closely observe from another area, and may be designated by the user 30 . For example, the guide 410 may have a circular shape, but is not limited thereto.

Examples of overlaying the guide 410 on the pathology slide image may vary. As an example, the guide 410 is initially displayed at a predetermined position of the pathology slide image (eg, the central position of the pathology slide image, the position of the region of interest, etc.), and then the guide 410 is guided according to the user's 30 manipulation. The position of 410 can be changed or fixed. As another example, the guide 410 may initially be displayed along with a mark (eg, a mouse cursor) on the pathology slide image, and then the position of the guide 410 may be fixed according to the manipulation of the user 30 . . However, the example displayed on the pathology slide image on the guide 410 is not limited to the above, and any method that can help the user 30 observe the pathology slide image may be applied without limitation.

Meanwhile, a magnification 431 may be displayed in an area adjacent to the guide 410 . Here, the magnification means an actual magnification that can be realized with an optical microscope. Accordingly, through the guide 410 and the magnification 431 displayed on the pathology slide image, the user 30 may have an experience similar to observing the image through an optical microscope.

According to the user input received while the pathology slide image is output on the display device, the processor 110 overlays the guide on the pathology slide image and outputs the overlay. For example, when a user input for raising or lowering the magnification is received while the pathology slide image shown in FIG. 4A is output on the display device, the processor 110 determines the area of the guide corresponding to the user input and the area of the guide included in the guide. determine the area Then, the guide is overlaid on the pathology slide image based on the determined area of the guide and the area included in the guide.

For example, the processor 110 may enlarge or reduce the area of the guide based on a magnification specified according to a user input, while the area included in the guide is fixed.

In the examples of FIGS. 4A and 4B , it is assumed that the user input increases the magnification from 40x to 100x. In this case, the processor 110 may determine the area of the guide 410 to be larger than the area of the guide 420 while fixing the same size of the pathology slide image output to the display device. That is, the size of the guide 420 corresponding to 100 magnification is smaller than the size of the guide 410 corresponding to 40 magnification. In other words, in a state where the size of the pathology slide image output on the display device is the same, the actual area of the region observed by the optical microscope at 40 magnification (ie, the area of the region within reference numeral 410) and the optical microscope at 100 magnification The processor 110 adjusts the size of the guide 420 so that the actual area of the observed area (ie, the area of the area within reference numeral 420) is the same.

The processor 110 overlays the guide 420 on the pathology slide image based on the area of the guide 420 and outputs it. Also, the processor 110 may display a magnification 432 corresponding to the user input on the output image (the image on which the guide 420 is overlaid). Here, the magnification 432 means the actual magnification of the optical microscope. That is, the magnification 432 means a magnification set in the optical microscope when the user 30 intends to observe the area within the guide 420 through the optical microscope.

Meanwhile, although not shown in FIGS. 4A and 4B , an image for setting magnifications 431 and 432 may be output to the display device along with the pathology slide image or before the pathology slide image is output. For example, the user 30 may designate the magnifications 431 and 432 through images, and the processor 110 outputs the guides 410 and 420 on the pathology slide image in response to a user input designating the magnifications. can do. Meanwhile, images for setting the magnifications 431 and 432 may be configured in various ways so that the user 30 can select a desired magnification.

5A and 5B are diagrams for explaining another example in which a pathology slide image overlaid with a guide is output, according to an exemplary embodiment.

In the examples of FIGS. 5A and 5B , it is assumed that the user input increases the magnification from 40x to 100x, as in the examples of FIGS. 4A and 4B .

When a user input for raising or lowering the magnification is received while the pathology slide image shown in FIG. 5A is output on the display device, the processor 110 determines the area of the guide corresponding to the user input and the area included in the guide. . Then, based on the determined area of the guide and the area included in the guide, the guide is overlaid on the pathology slide image and output.

For example, the processor 110 may enlarge or reduce an area included in the guide based on a magnification specified according to a user input, while the area of the guide is fixed.

When a user input for increasing the magnification from 40x to 100x is received, the processor 110 determines the actual area of the area included in the guide 520 while the areas of the guide 510 and the guide 520 are fixed to be the same on the screen. It may be determined that the actual area of the region included in the guide 510 is smaller than the area. That is, the actual area of the area included in the guide 520 corresponding to the 100 magnification is smaller than the actual area of the area included in the guide 510 corresponding to the 40 magnification. In other words, the size of the guide corresponding to an optical microscope with a magnification of 40 (ie, the area on the screen of the guide 510) and the size of the guide corresponding to an optical microscope with a magnification of 100 (ie, the area on the screen of the guide 520) The processor 110 adjusts the actual area of the area included in the guide 520 so that is the same.

Meanwhile, the processor 110 displays the magnifications 531 and 532 in areas adjacent to the guides 510 and 520, so that the user 30 can intuitively know the magnification of the currently displayed guide. Here, the magnifications 531 and 532 mean actual magnifications of an optical microscope. That is, the magnifications 531 and 532 refer to magnifications set in the optical microscope when the user 30 intends to observe the area within the guides 510 and 520 through the optical microscope.

Meanwhile, although not shown in FIGS. 5A and 5B , images for setting magnifications 531 and 532 may be output to the display device as described above with reference to FIGS. 4A and 4B .

According to the above description with reference to FIGS. 3 to 5B , the user 30 who is accustomed to reading images with an existing optical microscope can easily read images according to the field of digital pathology.

Meanwhile, an example of receiving a user input for increasing magnification has been described with reference to FIGS. 4A and 5B . The same method may also be applied when a user input for lowering magnification is received.

6 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.

Referring to FIG. 6 , steps 610 to 630 are the same as steps 310 to 330 of FIG. 3 . Therefore, detailed descriptions of steps 610 to 630 are omitted below.

In step 640, the user terminal 10 or 100 outputs an enlarged or reduced image of at least one portion of the pathology slide image 40 based on the additionally received user input. For example, the processor 110 may control the display device to output an image in which at least one portion of the pathology slide image 40 is enlarged or reduced.

Here, the additional user input may be an input for enlarging or reducing the pathology slide image 40 without adjusting magnification. For example, the user 30 zooms in or zooms out the pathology slide image 40 using a mouse wheel, or UI output on a display device (eg, buttons, scroll The pathology slide image 40 may be enlarged or reduced by additional user input in various ways, such as manipulating a bar or the like. At this time, the guide overlaid on the pathology slide image 40 may be enlarged or reduced together with the pathology slide image 40 or may be fixed regardless of the enlargement or reduction of the pathology slide image 40 . Hereinafter, an example of outputting an image based on an additional user input by the processor 110 will be described with reference to FIG. 7 .

7A and 7B are diagrams for explaining another example of outputting a pathology slide image according to an exemplary embodiment.

Referring to FIG. 7A , it is assumed that a pathology slide image displayed with a guide 710 corresponding to a magnification of 40 is output on the display device. Then, according to the received user input, the processor 110 may enlarge or reduce the pathology slide image.

For example, referring to FIG. 7B , the guide 720 shown in FIG. 7B means the same magnification as the guide 710 shown in FIG. 7A (magnification of 40). That is, according to a user input for enlarging the pathology slide image, a pathology slide image with a partial area enlarged is output as shown in FIG. 7B , and at the same time, the size of the guide 720 may be equally enlarged.

Meanwhile, an example of receiving a user input for enlarging a pathology slide image has been described with reference to FIGS. 7A and 7B . The same method can also be applied when a user input for reducing the pathology slide image is received.

8 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.

Referring to FIG. 8 , steps 810 to 830 are the same as steps 310 to 330 of FIG. 3 . Therefore, detailed descriptions of steps 810 to 830 are omitted below.

In step 840, the processor 110 identifies a region observed by the user in the pathology slide image 40 according to a predetermined criterion.

Here, the predetermined criterion includes that the magnification specified according to the user input is greater than or equal to the threshold magnification. That is, the processor 110 may determine which region is the region observed by the user at a magnification higher than a threshold in the pathology slide image 40 .

For example, the threshold magnification may be a magnification at a level at which the user 30 can check the morphology of cells based on the image. The threshold magnification may be preset or adjusted by the user 30 .

In step 850, the user terminal 10 or 100 distinguishes the observed region from the other regions according to the confirmation result and outputs them. For example, the processor 110 may control the display device to distinguish and output the observed region and other regions.

For example, a thumbnail image may be superimposed on a portion of the pathology slide image 40 and output, and the thumbnail image may be an image expressed so that the observed region and other regions are distinguished according to a predetermined criterion. can

Hereinafter, with reference to FIGS. 9A to 10 , an example in which an observed area and other areas are distinguished and output according to a predetermined criterion will be described.

9A to 9C are diagrams for explaining an example in which an observed region and other regions are divided according to a predetermined criterion according to an exemplary embodiment.

9A shows an initial pathology slide image (i.e., an image without observed regions according to a predetermined criterion).

As described above with reference to FIGS. 3 to 7B , the user 30 may enlarge or reduce at least a portion of the initial pathology slide image while observing it. In particular, the user 30 may observe the image while enlarging or reducing it according to a desired magnification.

The processor 110 may check whether the magnification of the region observed by the user 30 is greater than or equal to a threshold magnification. For example, upon receiving information on a magnification designated by the user 30 as a user input, the processor 110 may check whether the magnification according to the user input is greater than or equal to the threshold magnification through comparison with the threshold magnification. In addition, the processor 110 may control the display device to distinguish and output an area observed at a threshold magnification or higher and other areas.

Referring to FIG. 9B , an area 910 that the user 30 is currently observing is displayed. The processor 110 checks whether the magnification corresponding to the region 910 is greater than or equal to the threshold magnification. In this way, the processor 110 identifies the region 920 observed at or above the threshold magnification and other regions 930, and controls the display device to output an image in which the region 920 and region 930 are separated. can do.

Referring to FIGS. 9B and 9C , the area 920 may be distinguished from each other by being displayed with a brighter brightness value than the area 930 . However, the method for dividing the area 920 and the area 930 is not limited to the above.

10 is a diagram for explaining an example of outputting a thumbnail image according to an exemplary embodiment.

Referring to FIG. 10 , an example in which a thumbnail image 1000 is overlapped and output on a portion of a pathology slide image is shown. Here, the thumbnail image 1000 may be the image described above with reference to FIGS. 9A to 9C . However, the area where the thumbnail image 1000 is output and its size are not limited to the example shown in FIG. 10 .

11 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.

Although not shown in FIG. 11 , an example to be described later with reference to FIG. 11 may be combined with at least some of the examples described above with reference to FIGS. 3 to 10 in parallel or in time series. Therefore, hereinafter, a detailed description of the contents described above with reference to FIGS. 3 to 10 will be omitted.

In step 1110, the processor 110 sets at least one region of interest in the pathology slide image.

Here, the region of interest refers to a region that requires observation by the user 30 within the pathology slide image 40 . For example, the region of interest may be, but is not limited to, an important region to be observed in pathology interpretation of a disease. For example, when diagnosis of lung cancer is required, a region in which lung cancer cells are present within the pathology slide image 40 may be a region of interest, and when mitosis is evaluated in breast cancer, a similar region within the pathology slide image 40 may be used. Regions expected to be highly disrupted may be regions of interest.

The processor 110 analyzes the pathology slide image 40 and sets at least one region of interest. As an example, the processor 110 may detect a region of interest by analyzing the pathology slide image 40 using a predetermined image processing technique. As another example, the processor 110 may detect a region of interest in the pathology slide image 40 using a machine learning model. In this case, the machine learning model may be trained to detect a region of interest in the reference pathology slide images by using learning data including a plurality of reference pathology slide images and a plurality of reference label information.

As an example, the processor 110 may set at least one ROI by analyzing the entire pathology slide image 40 . As another example, the processor 110 may set at least one region of interest by analyzing a portion of the pathology slide image 40 . Specifically, the user 30 may designate a region of interest in the pathology slide image 40 in advance, and the processor 110 analyzes only the region designated by the user 30 to determine at least one region of interest. can be set

In step 1120, the processor 110 identifies a region observed by the user according to a predetermined criterion in the pathology slide image.

Here, the predetermined criterion includes that the magnification specified according to the user input is greater than or equal to the threshold magnification. That is, the processor 110 may determine which region is the region observed by the user at a magnification higher than a threshold in the pathology slide image 40 .

For example, the threshold magnification may be a magnification at a level at which the user 30 can check the morphology of cells based on the image. The threshold magnification may be preset or adjusted by the user 30 .

In operation 1130, the processor 110 determines whether the region observed by the user includes at least one region of interest.

In other words, the processor 110 may check whether the region of interest is included in the region observed by the user. Through step 1110, the processor 110 may check the region of interest within the pathology slide image 40 in advance. Accordingly, the processor 110 may check whether the region of interest is included in the region observed by the user 30 in the pathology slide image 40 .

Hereinafter, an example in which the processor 110 determines whether the region of interest is included in the region observed by the user will be described with reference to FIG. 12 .

12 is a diagram for explaining an example in which a processor operates according to an exemplary embodiment.

Referring to FIG. 12 , the processor 110 may distinguish a region 1210 observed by the user 30 from another region 1220 in the pathology slide image. Here, the area 1210 may be an area observed to meet a predetermined criterion as described above with reference to step 1120 . Also, a method for dividing the area 1210 and the area 1220 by the processor 110 is the same as described above with reference to FIG. 8 . That is, the area 1210 and the area 1220 may be determined by the area 1230 that the user 30 is currently observing.

The processor 110 determines whether the region 1210 includes the region of interest 1240 . For example, the region of interest 1240 may be detected in advance according to a preset criterion as described above with reference to step 1110 . The processor 110 may distinguish the region 1210 from the region 1220 based on the region 1230 and determine whether the region 1210 includes the region of interest 1240 .

Meanwhile, the processor 110 may provide a notification to the user 30 according to whether the region 1210 includes the region of interest 1240 , which will be described with reference to FIGS. 13 and 14 .

13 is a flowchart illustrating another example of a method of outputting a pathology slide image according to an exemplary embodiment.

Referring to FIG. 13 , steps 1310 and 1320 are the same as steps 1110 and 1120 of FIG. 11 . Therefore, detailed descriptions of steps 1310 and 1320 are omitted below.

In step 1330, the user terminal 10 or 100 receives a user input related to the end of observation of the pathology slide image.

The user input includes various attempts of the user 30 to end observation of the pathology slide image 40 currently output to the display device. For example, user input may include an attempt by the user 30 to replace the currently output pathology slide image 40 with another image or to terminate a program for outputting and observing the pathology slide image 40 . there is. Alternatively, the user input in step 1330 may include a case where no user input is received for a predetermined period of time. For example, when a user input is not received for a predetermined time after the pathology slide image 40 is output, the user input in step 1330 may be received.

In step 1340, the processor 110 determines whether at least one region of interest is included in the region observed by the user.

Here, the region observed by the user includes a region observed by the user according to a predetermined criterion within the pathology slide image 40, and the predetermined criterion includes a magnification designated according to a user input greater than or equal to a threshold magnification. . An example of determining whether the region of interest is included in the region observed by the user by the processor 110 is as described above with reference to steps 1120 and 1130 . If the region of interest is a plurality of regions, the processor 110 may determine whether all of the plurality of regions of interest are included in the region observed by the user.

Meanwhile, an example in which the processor 110 determines whether the region of interest is included in the region observed by the user is not limited to the above. For example, if the region observed by the user does not correspond to a predetermined ratio or more to the total region of interest, the processor 110 may determine that the region of interest is not included in the region observed by the user. When mitosis is evaluated in breast cancer, a region in which mitosis is expected to be high in the pathology slide image 40 may be set as a region of interest. In this case, if the region observed by the user does not include a predetermined percentage (eg, 50%) or more of the region of interest, the processor 110 may determine that the region observed by the user does not include the region of interest. can

If the region of interest is not included in the region observed by the user, step 1350 is performed. Otherwise, the process ends.

In step 1350, the processor 110 generates a notification signal about whether the pathology slide image 40 is additionally observed.

As described above with reference to steps 1330 and 1340, observation of the currently output pathology slide image 40 may be terminated in a state in which the region of interest is not observed. In this case, the processor 110 may help the user 30 determine whether to proceed with additional observation of the pathology slide image 40 by generating and outputting a notification signal.

Meanwhile, the processor 110 may determine whether to generate a notification signal according to whether a predetermined condition is satisfied regardless of whether all of the ROI is observed. As an example, when at least a part of the region of interest is included in a region previously observed by the user 30 in the pathology slide image 40, even if the region of interest exists in a region not observed by the user 30, the processor (110) may not generate a notification signal. As another example, regardless of whether the region of interest is included in the region previously observed by the user 30, if the region of interest exists in an area not observed by the user 30, the processor 110 generates a notification signal. can do.

For example, the notification signal may be an image output on a display device, sound output through a speaker, or vibration output through a vibration motor. However, examples of notification signals are not limited to those described above.

14 is a diagram for explaining an example in which a notification signal is generated according to an exemplary embodiment.

14 illustrates a case where the region of interest 1420 is not included in the region 1410 observed by the user. In this situation, when a user input 1430 related to the end of observation of the pathology slide image is received, the processor 110 generates and outputs a notification signal 1440 .

Meanwhile, an example of outputting the notification signal 1440 is not limited to that shown in FIG. 14 . Specifically, as long as it is intended to inform the user that at least a part of the region of interest 1420 has not been observed, the notification signal 1440 may be any type such as an image, sound, or vibration. Also, when the notification signal 1440 is an image, any image displayed with a message may be used.

If an additional observation attempt of the pathology slide image is received from the user 30 after the notification signal 1440 is output, the processor 110 may assist the user 30 in observing by outputting the ROI 1420 to the display device. there is.

According to the foregoing, even the user 30 who is accustomed to reading images with an existing optical microscope can easily read images according to the field of digital pathology. In addition, the user 30 can intuitively check whether there is an unobserved area in the pathology slide image 40 . In addition, the user 30 may intuitively check whether the region of interest included in the pathology slide image 40 has been completely observed.

On the other hand, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. In addition, the structure of data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (eg, ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (eg, CD-ROM, DVD, etc.) It can, but is not limited to this.

Those skilled in the art related to the present embodiment will be able to understand that it may be implemented in a modified form within a range that does not deviate from the essential characteristics of the above description. Therefore, the disclosed methods should be considered from an explanatory point of view rather than a limiting point of view, and the scope of rights is shown in the claims rather than the foregoing description, and should be construed to include all differences within the equivalent range.

1: system
10: user terminal
20: server
30: user
40: pathology slide image

Claims (19)

receiving a user input related to an output method of at least one region included in a pathology slide image;
determining an area of a guide to be output to the pathology slide image and an area of the pathology slide image included in the guide based on the user input;
and outputting a pathology slide image on which the guide is overlaid based on the determined area and region. According to claim 1,
The determining step is
A method of enlarging or reducing the area in a state in which the area is fixed, based on a magnification designated according to the user input. According to claim 1,
The determining step is
A method of enlarging or reducing the area in a state where the area is fixed based on a magnification specified according to the user input. According to claim 1,
The method further comprising outputting an enlarged or reduced image of at least a portion of the pathology slide image based on an additionally received user input. According to claim 1,
identifying a region observed by a user according to a predetermined criterion in the pathology slide image; and
The method further comprising: distinguishing and outputting the observed region and other regions according to the confirmation result. According to claim 5,
The predetermined criterion includes that a magnification specified according to the user input is equal to or greater than a threshold magnification. According to claim 5,
The outputting step is
A method of outputting a thumbnail image expressed so that the observed region and other regions are distinguished according to the predetermined criterion. According to claim 1,
setting at least one region of interest in the pathology slide image;
identifying a region observed by a user according to a predetermined criterion in the pathology slide image; and
The method further comprising determining whether a region observed by the user includes the at least one region of interest. According to claim 8,
receiving a user input related to termination of observation of the pathology slide image; and
The method further comprising generating a notification signal for additional observation of the pathology slide image based on the determination result. A computer-readable recording medium recording a program for executing the method of claim 1 on a computer. at least one memory; and
at least one processor;
the processor,
determining an area of a guide to be output to the pathology slide image and an area of the pathology slide image included in the guide based on a user input related to an output method of at least one region included in the pathology slide image; A computing device that controls a display device to output a pathology slide image overlaid with the guide based on the determined area and region. According to claim 11,
the processor,
A computing device that enlarges or reduces the area based on a magnification specified according to the user input in a state in which the area is fixed. According to claim 11,
the processor,
A computing device that enlarges or reduces the area based on a magnification specified according to the user input, while the area is fixed. According to claim 11,
the processor,
A computing device that controls the display device to output an image in which at least a portion of the pathology slide image is enlarged or reduced based on an additionally received user input. According to claim 11,
the processor,
A computing device for controlling the display device to identify a region observed by a user according to a predetermined criterion in the pathology slide image, and to distinguish the observed region from other regions and output them according to the check result. According to claim 15,
The predetermined criterion includes that a magnification specified according to the user input is greater than or equal to a threshold magnification. According to claim 15,
the processor,
A computing device that controls the display device to output a thumbnail image in which the observed region and other regions are distinguished according to the predetermined criterion. According to claim 11,
the processor,
At least one region of interest is set in the pathology slide image, a region observed by a user is identified in the pathology slide image according to a predetermined criterion, and the region observed by the user includes the at least one region of interest. A computing device that determines whether or not According to claim 18,
the processor,
A computing device configured to generate a notification signal for additional observation of the pathology slide image based on a result of the determination when a user input related to termination of observation of the pathology slide image is received.

Images