WO2011058482A2

WO2011058482A2 - Visualization of an object and an annotation

Info

Publication number: WO2011058482A2
Application number: PCT/IB2010/054999
Authority: WO
Inventors: Iwo Willem Oscar Serlie
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2009-11-12
Filing date: 2010-11-04
Publication date: 2011-05-19
Also published as: WO2011058482A3

Abstract

A system for visualizing an image is proposed. The system comprises an object detector (101) for detecting a main portion of the image, the main portion comprising an object of interest. The system comprises an annotation detector (102) for detecting an annotation portion of the image, the annotation portion comprising a graphical representation of an annotation, the annotation portion being separate from the main portion. The system comprises a visualization subsystem (103) for displaying the main portion together with the annotation portion. The visualization subsystem (103) comprises a zoom subsystem (104) for enlarging the main portion to substantially fill a display area with the main portion. The visualization subsystem (103) is arranged for displaying the annotation portion as an overlay of the main portion.

Description

Visualization of an object and an annotation

FIELD OF THE INVENTION

The invention relates to visualizing an image, in particular to visualizing an image and an annotation. The invention further relates to mammography image viewing. BACKGROUND OF THE INVENTION

Breast cancer is an important cause of death among women. Early detection through mass screening with mammography contributes to reducing mortality.

Mammography screening is a field in which a large number of images are reviewed in batches in a relatively small amount of time. Radiologists scan the mammograms for abnormalities, in particular micro-calcifications, in order to detect breast cancer in an early stage. Because a large portion of the population is scanned preventively, the number of images to be reviewed is very large. Consequently, the efficiency of review workstations is important. The majority of existing mammography review stations are arranged for filling an available viewing area with a complete mammography image for initial display. The image size is reduced such that the full image is fitted to an available display domain area. After initial display, the user needs to pan and zoom the images to inspect the breast in further detail. The images may contain annotations, for example lead letters which are held in the x- ray beam during x-ray acquisition.

JP-3149584-A discloses a breast image processing device having a memory that memorizes a left/right pair of digital breast images. The breast image processing device further memorizes left and right mark images that correspond with the left and right image of the above mentioned pair of breast images. The device comprises a display means for displaying a frame in a display domain with a prescribed size. The entire frame is split in two halves by a left-right division line. A control means selects, from the above mentioned pair of breast images that have been displayed within the above mentioned left and right display domains, one breast image, based on the positional co-ordinates of a pointer, and carries out image processing of the selected breast image. A display domain modifying means displays a frame line that consists of four rims around the display domain that corresponds with the selected breast image, and makes it possible to move one rim of the left and right rims in the left and right direction, using a pointer. This causes the size of the display domain to be modified. An image upward/downward moving means displays a horizontal reference line and makes it possible to move the selected breast image to be moved in upward/downward direction inside the display domain. The device has control means which read the header information that has individually been added to the processed breast images, and

discriminates the left and the right side and photographing direction, based on the header information that has been read. The device further comprises a left and right mark arranging means that, inside the above mentioned left and right display domains, arranges the corresponding left and right mark images in the original images, and re-arranges the corresponding marks inside the modified display domain.

SUMMARY OF THE INVENTION

It would be advantageous to have an improved visualization of an image. To better address this concern, a first aspect of the invention provides a system comprising: - an object detector for detecting a main portion of the image, the main portion comprising an object of interest;

an annotation detector for detecting an annotation portion of the image, the annotation portion comprising a graphical representation of an annotation, the annotation portion being separate from the main portion;

- a visualization subsystem for displaying the main portion together with the annotation portion.

A relatively high reliability may be attributed to the annotations which are incorporated in the image. Because the main portion is visualized together with the annotation portion, the reliability of the system is improved. Moreover, the user would not need to revert to the full image to verify the annotation portion, because the annotation portion is already visualized together with the main portion. This may save considerable time. Because only the main portion and the annotation portion need to be displayed, valuable display area is saved, since irrelevant portions of the screen, such as background, do not have to be displayed. This allows the main portion to be displayed in a larger format without the annotation having to be shifted off of the display area. Moreover, the system does not need to parse any metadata to determine an appropriate annotation.

The visualization subsystem may comprise a zoom subsystem for enlarging the main portion to substantially fill a display area with the main portion. By filling the display area with the main portion, the largest possible visualization of the main portion may be achieved. This way, further manual zooming during review may be unnecessary. The visualization subsystem may be arranged for displaying the enlarged main portion as an initial display of the image. In contrast, existing viewing stations start with displaying a display area with a complete image containing a lot of background. Because of this, the user has to zoom in manually to view the image at full resolution. This delays the mammography screening.

The visualization subsystem may be arranged for displaying the annotation portion as an overlay of the main portion. This makes the system easy to use, because healthcare professionals are used to look for the annotation in the image.

The visualization subsystem may be arranged for displaying the overlay on top of a background portion of the main portion, separate from the object of interest. The main portion may comprise more than just the object of interest. For example, the main portion may be a square or rectangular area, whereas the object of interest may have any curved shape. The remainder of the image may be considered background portion. This background portion of the main portion may be a highly suitable place for displaying the annotation portion, because the background portion may not contain any interesting information.

The visualization subsystem may be arranged for displaying the main portion and the annotation portion in separate display areas. This way, the main portion is not disturbed by the annotation portion.

The object of interest may comprise at least part of an anatomical object. The object detector may be arranged for detecting such an anatomical object. This may be a particular kind of anatomical object, such as a breast. This feature is useful for medical review stations.

The annotation may comprise a graphical representation of at least one character. This allows to detect annotations which comprise a character.

The character may comprise a lead character. Such lead characters may be kept in the x-ray beam during image acquisition. Because this lead letter has been put in the x-ray beam during the acquisition, it has also been verified at the time of the acquisition. It thus avoids errors which may be present in metadata.

The system may comprise a contrast and/or brightness setting subsystem for setting the contrast and/or brightness of the main portion independently of the annotation portion. Since the gray level distribution of the main portion may be substantially different from the gray level distribution of the annotation portion, different contrast and/or brightness settings may be used to display them optimally. The system may be arranged for processing a set of images for simultaneous and/or sequential display, wherein the zoom subsystem or the contrast/brightness setting subsystem, respectively, is arranged for applying the same zoom or contrast/brightness setting to the images of the set of images. This allows comparing the sizes and gray levels of the different images in the set of images. For example, a smallest zoom factor which allows full display of the main portion of each image may be used.

A medical image acquisition apparatus may comprise the system set forth. Such apparatus may comprise an image detector for acquiring the image of the object of interest together with at least one physical object representing the annotation. The physical object may comprise a lead character.

The system set forth may also be included in a medical workstation.

Another aspect of the invention provides a method of visualizing an image, comprising:

detecting a main portion of the image, the main portion comprising an object of interest;

detecting an annotation portion of the image, the annotation portion comprising a graphical representation of an annotation, the annotation portion being separate from the main portion;

displaying the main portion together with the annotation portion. Another aspect of the invention provides a computer program product comprising instructions for causing a processor system to perform the method set forth. Also, functionality of the subsystems described above may be implemented partly or completely in software.

It will be appreciated by those skilled in the art that two or more of the above- mentioned embodiments, implementations, and/or aspects of the invention may be combined in any way deemed useful.

Modifications and variations of the image acquisition apparatus, the workstation, the system, and/or the computer program product, which correspond to the described modifications and variations of the system, can be carried out by a person skilled in the art on the basis of the present description.

A person skilled in the art will appreciate that the method may be applied to multi-dimensional image data, e.g. to two-dimensional (2-D), three-dimensional (3-D) or four-dimensional (4-D) images, acquired by various acquisition modalities such as, but not limited to, standard X-ray Imaging, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound (US), Positron Emission Tomography (PET), Single Photon Emission Computed Tomography (SPECT), and Nuclear Medicine (NM).

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. In the drawings,

Fig. 1 shows a diagram of a system for visualizing an image;

Fig. 2 shows a diagram of a method of visualizing an image;

Fig. 3 shows a pair of CC breast images;

Fig. 4 shows a pair of MLO breast images;

Fig. 5 shows a pair of CC breast images with indication of main portion;

Fig. 6 shows a pair of MLO breast images with indication of main portion; Fig. 7 shows main portions and annotations of a pair of CC breast images; and Fig. 8 shows main portions and annotations of a pair of MLO breast images.

DETAILED DESCRIPTION OF EMBODIMENTS

Systems and methods for visualizing an image are disclosed herein. In the examples described below, the focus is on breast images. However, this is not a limitation. The systems and methods may be adapted to operate on other kinds of images with other kinds of objects. These objects may be anatomical, but other kinds of objects are also possible.

Fig. 1 illustrates a system for visualizing an image. The system comprises an object detector 101 for detecting a main portion of the image. To this end, the object detector detects an object of interest in the image. The main portion of the image is selected such that it comprises the object of interest. Image analysis techniques may be used to detect the object of interest. Prior knowledge may be used to define image properties of the object of interest, and these image properties may be detected using techniques known in the art per se. For example, gradient detection may be used to detect a boundary of an object. Also, an adaptive shape model may be fitted to the image. For example, the image may be a projection image such as an x-ray image. The main portion may be substantially smaller than the complete image. For example, the main portion may be a bounding box around the object of interest. The main portion may also be slightly larger than a bounding box, to include a margin of background around the object of interest. The shape, for example the aspect ratio, of the main portion may be configured to fit to a predetermined display area after appropriate

magnification.

The system may further comprise an annotation detector 102 for detecting an annotation portion of the image. Such an annotation portion may comprise a graphical representation of an annotation. For example, one or more characters may be blended into the image. The annotation portion may be separate from the main portion. However, this is not a limitation. The annotation portion may be detected, for example, using text recognition technology (optical character recognition). Other symbols different from characters may also be used, for example a circle, a cross or a square may be used as annotations. The annotation detector may accordingly be arranged to recognize such annotations. The annotation portion may have a predetermined shape, for example rectangular. The annotation portion may also include a margin around the annotation.

It is possible to combine the object detector 101 and the annotation detector 102. For example, it is possible to detect non-connected objects in an image. A method for doing so is disclosed in US 6,891,920 and US 5,572,565. For example, the largest non- connected object found may be considered to be the object of interest. Any further objects may be regarded as annotation objects.

The system may further comprise a visualization subsystem 103 for displaying the main portion together with the annotation portion. For example, the main portion and the annotation portion may be displayed on the same display screen. Moreover, the portions of the image which are not part of either the main portion or the annotation portion are not shown on the display. However, the system may have a selectable display mode to show also the portions of the image which are not part of either the main portion or the annotation portion. The visualization subsystem 103 may be arranged such that the first view of the image which is displayed when a particular image is selected for reviewing shows only the main portion and the annotation portion. It is possible to detect further image portions comprising objects of interest in the image, which may also be shown on the first view.

The visualization subsystem 103 may comprise a zoom subsystem 104 for enlarging the main portion to substantially fill a display area with the main portion. It is also possible to limit the zoom by a maximum zoom factor. For example, the largest zoom factor may be the actual size of the image (i.e., 100% zoom). The visualization subsystem may be arranged for initially showing the enlarged main portion, together with the annotation portion, when an image is selected for review. The visualization subsystem 103 may be arranged for displaying the annotation portion as an overlay of the main portion. In such a case, the annotation portion may be displayed on top of the main portion. For example, the overlay may be displayed on top of a background portion of the main portion, separate from the object of interest. For example the margin containing background, as described above, may be used for displaying the annotation portion. Alternatively, the visualization subsystem may be arranged for displaying the main portion and the annotation portion in separate display areas. For example, separate display devices may be used. In another example, two separate display area domains are reserved for, respectively, the main portion and the annotation portion. Usually the display area domain reserved for display of the main portion is larger than the display area domain reserved for the annotation portion, because the main portion may be enlarged for accurate review, and the object of interest may be larger than the annotation.

The object of interest may comprise at least part of an anatomical object, such as a breast. The annotation may comprise a graphical representation of at least one character, such as a lead character.

The system may further comprise a contrast and/or brightness setting subsystem 105 for setting the contrast and/or brightness of the main portion independently of the annotation portion. For example, automatic optimization of display parameters such as contrast and/or brightness may be performed for the main portion for optimal display of the object of interest. Separate parameter values may be established for the annotation portion.

The system may be arranged for processing a set of images for simultaneous and/or sequential display, wherein the zoom subsystem or the contrast/brightness setting subsystem, respectively, is arranged for applying the same zoom or contrast/brightness setting to the images of the set of images. For example, when reviewing a set of

mammograms of the same patient, the main portions of these mammograms may be subjected to the same zoom, contrast, and/or brightness settings. This also applies to other kinds of images. For example, techniques disclosed in US 7,127,684 may be used to synchronize the zoom of a set of images.

The system may be incorporated in a medical image acquisition apparatus. For example, an x-ray mammography scanner may include the review facility provided by the present disclosure. The medical image acquisition apparatus may comprise an image detector for acquiring the image of the object of interest together with at least one physical object representing the annotation. Such physical object may be put in the scanned area by a medical professional, for example. The system may also be included in a medical workstation, for example a mammography review workstation. However, other medical and non-medical applications are also possible.

Fig. 2 shows a flow chart of a method of visualizing an image. The method comprises a step 201 of detecting a main portion of the image, the main portion comprising an object of interest. The method further comprises a step 202 of detecting an annotation portion of the image, the annotation portion comprising a graphical representation of an annotation, the annotation portion being separate from the main portion. The method further comprises a step 203 of displaying the main portion together with the annotation portion. The method may be implemented as a computer program product comprising instructions for causing a processor system to perform the method. Such a computer program product may be stored on a computer readable medium.

Mammography images that contain imprinted information additional to the anatomical object, may be displayed using the techniques disclosed herein, to optimize viewing and/or improve the effectiveness of the use of the viewing area. Normally, the available viewing areas are filled with the complete mammography images for initial display: image fit to window. After initial display the user may pan and zoom images to view each area of the object.

Figs. 3 to 8 illustrate, schematically, the application of the techniques disclosed herein to mammography reviewing. However, the invention is not limited to mammography. Figs. 3 and 4 are sketches of original mammograms as acquired with a mammography x-ray scanner. Fig. 3 shows a pair of CC mammograms 21 and 22 showing breasts 1 and 2 of a woman. The mammograms 21 and 22 further comprise annotations RCC 5 and LCC 6, respectively. Fig. 4 shows a pair of MLO mammograms 23 and 24 showing the pair of breasts 7 and 8 from MLO direction. The mammograms 23 and 24 further comprise annotations RMLO 11 and LMLO 12, respectively. It is noted that a substantial portion of the images, for example image 21, is background 29. This background 29 may not be needed for clinical diagnoses, but does fill some of the display area which could otherwise be used to display the breast 1 larger. However, if the image 21 is zoomed to display the breast 1 larger, then the annotation RCC 5 will move off the display area and become invisible.

Figs. 5 and 6 show the same images 21, 22, 23, 24, as sketched in Figs. 3 and 4, with an indication of main image portions 3, 4, 9, 10. When one of the four images of Figs. 3 and 4, for example image 21, is fed to the system of Fig. 1, it may be processed by the object detector 101 and the annotation detector 102. It may also be possible to feed more than one image to the system for simultaneous processing. Alternatively, the images may be fed to the system sequentially. The object detector 101 may detect the breast 1 in the image 21, using techniques known in the art per se. Consequently, the main image portion 21 may be determined such that it includes the detected breast 1. In this example, the main image portion 21 is a rectangular box which includes the breast 1. The annotation detector 102 may detect the annotation RCC 5, for example using optical character recognition techniques, which are known in the art per se. From this, an annotation portion is determined, for example a rectangular image portion just large enough to include the annotation RCC 5. Likewise, the breasts 2, 7, and 8 may be detected in the other images to determine

corresponding main image portions 4, 9, and 10, respectively. Moreover, the annotation portions for annotations 6, 11, 12 may be determined.

Figs. 7 and 8 show display areas 25, 26, 27, 28 showing the main portions 3, 4, 9, 10 as detected in the images 21, 22, 23, 24 and as displayed by the visualization subsystem 103. Moreover, the annotation portions 15, 16, 19, 20 comprising annotations 5, 6, 11, 12 are displayed as an overlay on top of the main portions 3, 4, 9, 10. It is noted that the zoom subsystem 104 has enlarged the main portions 3, 4, 9, 10 such that they substantially fill the display areas 25, 26, 27, 28. Annotations 5, 6, 11, 12 may also be enlarged, independently of a zoom factor applied to the main portion. Zoom factors and other display parameters such as contrast and/or brightness have been determined so as to be identical for the set of main portions 3, 4, 9, 10 of images of the same patient.

It will be appreciated that the invention also applies to computer programs, particularly computer programs on or in a carrier, adapted to put the invention into practice. The program may be in the form of a source code, an object code, a code intermediate source and object code such as in a partially compiled form, or in any other form suitable for use in the implementation of the method according to the invention. It will also be appreciated that such a program may have many different architectural designs. For example, a program code implementing the functionality of the method or system according to the invention may be sub-divided into one or more sub-routines. Many different ways of distributing the functionality among these sub-routines will be apparent to the skilled person. The sub- routines may be stored together in one executable file to form a self-contained program. Such an executable file may comprise computer-executable instructions, for example, processor instructions and/or interpreter instructions (e.g. Java interpreter instructions). Alternatively, one or more or all of the sub-routines may be stored in at least one external library file and linked with a main program either statically or dynamically, e.g. at run-time. The main program contains at least one call to at least one of the sub-routines. The sub-routines may also comprise function calls to each other. An embodiment relating to a computer program product comprises computer-executable instructions corresponding to each processing step of at least one of the methods set forth herein. These instructions may be sub-divided into sub- routines and/or stored in one or more files that may be linked statically or dynamically.

Another embodiment relating to a computer program product comprises computer-executable instructions corresponding to each means of at least one of the systems and/or products set forth herein. These instructions may be sub-divided into sub-routines and/or stored in one or more files that may be linked statically or dynamically.

The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a storage medium, such as a ROM, for example, a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example, a floppy disc or a hard disk. Furthermore, the carrier may be a transmissible carrier such as an electric or optical signal, which may be conveyed via electric or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted to perform, or used in the performance of, the relevant method.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

CLAIMS:

1. A system for visualizing an image, comprising:

an object detector (101) for detecting a main portion of the image, the main portion comprising an object of interest;

an annotation detector (102) for detecting an annotation portion of the image, the annotation portion comprising a graphical representation of an annotation, the annotation portion being separate from the main portion;

a visualization subsystem (103) for displaying the main portion together with the annotation portion.

2. The system according to claim 1, wherein the visualization subsystem (103) comprises a zoom subsystem (104) for enlarging the main portion to substantially fill a display area with the main portion.

3. The system according to claim 1, wherein the visualization subsystem (103) is arranged for displaying the annotation portion as an overlay of the main portion.

4. The system according to claim 3, wherein the visualization subsystem (103) is arranged for displaying the overlay on top of a background portion of the main portion, separate from the object of interest.

5. The system according to claim 1, wherein the visualization subsystem (103) is arranged for displaying the main portion and the annotation portion in separate display areas.

6. The system according to claim 1, wherein the object of interest comprises at least part of an anatomical object.

7. The system according to claim 1, wherein the annotation comprises a graphical representation of at least one character.

8. The system according to claim 7, wherein the character comprises a lead character.

9. The system according to claim 1, further comprising a contrast and/or brightness setting subsystem (105) for setting the contrast and/or brightness of the main portion independently of the annotation portion.

10. The system according to claim 2 or 9, wherein the system is arranged for processing a set of images for simultaneous and/or sequential display, wherein the zoom subsystem or the contrast/brightness setting subsystem, respectively, is arranged for applying the same zoom or contrast/brightness setting to the images of the set of images.

11. A medical image acquisition apparatus, comprising the system for visualizing an image according to claim 1.

12. The medical image acquisition apparatus according to claim 11, further comprising an image detector for acquiring the image of the object of interest together with at least one physical object representing the annotation.

13. A medical workstation comprising the system according to claim 1.

14. A method of visualizing an image, comprising:

detecting (201) a main portion of the image, the main portion comprising an object of interest;

detecting (202) an annotation portion of the image, the annotation portion comprising a graphical representation of an annotation, the annotation portion being separate from the main portion;

displaying (203) the main portion together with the annotation portion.

15. A computer program product comprising instructions for causing a processor system to perform the method according to claim 14.