JP2008173447A - Enhancement of visual perception through dynamic cue - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a radiologist to take combination of any visual sensitivity into consideration so as to detect a pattern or a subject which conventionally has not been conspicuous. <P>SOLUTION: A method to convert static images into dynamic images is provided to assist detection, and is provided to identify pathology and abnormality of a patient based on the images. The system receives as input the static images, and generates as output the dynamic images including dynamic cues. The dynamic cue is generated by the action of a motion function and an observation function. Sequence of an image frame is produced to make the dynamic cues discernable on the images when examined. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a system and method for reading an image, particularly used to increase perception and thus allow an observer to detect previously inconspicuous patterns or objects. The present invention relates to a system and method for adding dynamic cues to still images, taking into account all possible visual sensitivity combinations.

  Today, images are a powerful tool in areas where human observation is used. Such areas include medical image sorting, industrial sorting, military research, astronomical observations, and the like. For example, in medical image sorting, lesion detection and correct interpretation is top priority for correct diagnosis, but unfortunately this is not always easy to achieve.

  Medical images often contain very small and barely detectable objects or patterns that can be paramount to diagnosis. Perceptual assist design for the detection of these types of objects is an area of research currently underway. Most studies focus on still image processing, and the end result is an image or map of possible locations of the lesion. In general, lesions can be detected by differences in color, brightness, structure, or surface texture between the lesion and its surroundings.

  The human vision system (HVS) is a very powerful tool for perception and image processing. Among them, its basic capabilities are figure segmentation, registration and still image processing. It is also very well adapted for efficient pattern recognition. Literature: Brian A. Wandell, “Foundation of Vision”, Sinauer Aassociates Inc., Sunderland, Massachusetts, May 1995. Literature: Bruno A. Olshausen, David J. Field "Sparse coding with an overcomplete basis set: A Strategy employed by v1?", Vision Research, 37: 3311-3225, 1997. This aspect of HVS is that image analysis is based primarily on the human ability to distinguish between normal and abnormal patterns.

  For HVS, even if not impossible, the detection of certain patterns or objects is very complex and there are various cases. This is often the case for medical images. There it is necessary to find an object that is almost the same color as that of the object or in front of the background of the surface feeling. A typical example of this type of image is a mammogram with minute calcifications. Very small calcifications are small deposits of calcium in breast tissue, indicating breast cancer if present in a malignant cluster. Unfortunately, mammograms tend to be low contrast, and minute calcifications are very small in size, making it rather difficult for laboratory radiologists to detect them and make a correct diagnosis. ing. Literature: Edward A. Sickles, “Breast calcifications: Mammographic evaluation”, Radiology, 160 (2): 289-293, August 1986.

  Improving the correct detection rate of breast cancer is paramount. In the United States, more than 212,000 new cases of breast cancer are diagnosed each year. From 1992 to 2002, the incidence increased by 0.4% annually. Recently, breast cancer is the second most common cause of cancer death in women, surpassing only lung cancer. In the United States, about 1 in 33 women die from breast cancer, totaling more than 40,000 annually. Literature: LAGRies, MPEisner, CLKosary, BFHankey, BAMillar, L.Clegg, A. Mariotto, EJFeuer and BKEdwards, editors, “SEER Cancer Statistics Review 1975-2002”, Bethesda, Maryland, 2004 November, American Cancer Society.

  If detected at an early stage, breast cancer is treatable and thus mortality is greatly reduced. This makes microcalcifications and early detection of tumors an extremely important task in oncology. The sooner the detection, the better the chances of treatment and survival. The work of Ries and others has shown that breast cancer mortality has declined despite the increasing incidence of breast cancer. In fact, breast cancer mortality dropped 2.4% annually from 1992 to 2002. This is largely the result of early detection and good treatment of malignant masses. If detection methods continue to improve, cancer mortality will continue to decline. The true sum of cancer cases allows even a slight improvement in detection algorithms to save hundreds or thousands of lives each year.

  It is an object of the present invention to assist in detection and to identify image objects and features. This method is adapted to biological HVS, which is well suited to the perception of moving objects, and well suited to explain the existence of motion sensitivity and flicker sensitivity, away from traditional amplitude sensitivity ing. Thus, artificial movements and / or the insertion of flicker (dynamic cues) into still images is to increase perception and, as a result, patterns or objects that the observer has not previously noticed allows to take into account all visual sensitivity combinations that should be used to be able to detect (objects).

  Yet another object of the present invention is to assist radiologists and general medical physicians in the perception of medical images. Artificial motion and / or the insertion of flicker (dynamic cues) into a still image is to increase perception and, as a result, patterns or objects that the radiologist has not previously noticed. All visual sensitivity combinations that should be used to be detectable can be taken into account.

  The present invention provides a method for transforming a still image into a movie to aid in detection and to identify image objects and features. In this movie, the observer sees movement or flicker that depends on the features of the still image. The system receives a still image as input and generates a movie that includes a dynamic cue as output. Two types of dynamic queues are used. Spatial motion (pixels move around in a certain manner) and temporal change in luminance at a fixed position (no change in location, only luminance). The definition of motion is described by a “motion function” that mathematically represents a type of motion that affects the pixel. The nature of the movement (how much it moves and what frequency) depends on the data through what is called the “observation function”.

  The method of the present invention is well suited for radiology because of its technical advantages, but also because it does not replace the human observer who must be responsible for the diagnosis. The method can also be useful in other areas where human observers are employed. Some of these areas are industrial sorting, military research, astronomical observations, etc.

  A more complete understanding of the present invention and the attendant advantages and features will be readily apparent from the following detailed description when considered in conjunction with the accompanying drawings.

  The present invention provides a method for transforming a still image into a movie to aid in detection and to identify image objects and features. In this movie, the observer sees movement or flicker that depends on the features of the still image. The system receives a still image as input and generates a movie that includes a dynamic cue as output.

 Two types of dynamic queues are used. Spatial motion (pixels move around in a certain manner) and temporal change in luminance at a fixed position (no change in location, only luminance). Spatial motion, as its name implies, can be understood as an object moving in space throughout the frame. In the present invention, the object to be moved is a pixel (image element). The concept of temporal motion applies to pixels that change their brightness over time rather than moving in space.

  The definition of motion is described by a “motion function” that mathematically represents the type of motion that affects the pixel. The nature of the movement (how much it moves and what frequency) depends on the data through what is called the “observation function”.

An exemplary method for inserting this artificial motion into an image is based on two functions. The motion function f _mov and the observation function f _obs . The motion function specifies the type of motion, while the observation function sets parameters for this motion.

[Motion function]
From a single grayscale still image I ₀ of m × n pixels, it is possible to generate a sequence of k = 0... p images of m × n pixels, of the k ^th th image. The luminance I of the pixel corresponding to the coordinates (i, j) is expressed as follows.

I _k (i, j) = f _mov (f _obs (I ₀ (i, j)), k) Equation (1)
I _k (i, j) is a discrete sequence of m × n pixels that depends on a motion function f _mov () that defines the motion of the pixels through the sequence of frames. This movement can be spatial or temporal, indicating that a pixel can change its spatial position or its brightness in time. FIG. 1 shows a possible configuration of spatial movement as a lateral vibration of all pixels with a constant amplitude and frequency depending on the initial luminance of each pixel. Thus, light pixels vibrate faster from one side to the other than darker pixels.

  Another way to insert spatial motion is a pulsating motion, as shown in FIG. For lateral vibration, the pulse frequency depends on the initial brightness of each pixel.

  For temporal motion, one example would be a change that draws a luminance sinusoid for all pixels whose frequency of change depends on the initial luminance of the pixel. Again, the initially lighter pixels will change faster than the originally darker pixels. An example is shown in FIG. 3, and the equation for a possible configuration is given by equation (2).

I _k (i, j) = f _DC (I ₀ (i, j)) + a · COS (2πf ₀ k · f _obs (I ₀ (i, j)) + φ) Equation (2)
In equation (2), I _k (i, j) represents the luminance of the pixel at the coordinate (i, j) of the k ^th th frame. f _DC () is an offset for all pixels. a is the amplitude of the sine wave. f ₀ is the fundamental frequency. f _obs () is an observation function, and φ is a phase shift. For each of these configurations, the rate of change of position (oscillating motion), magnitude (pulsating motion), or gray level (pulsating brightness) can be varied according to different functions. For example, Equation (2) constitutes a sine wave. The motion function may be any periodic or oscillating function and may include, for example:

[Observation function]
The observation function f _obs defines what the motion is based on exactly. The motion function defines what should be done at each pixel, while the observation function defines the pixel itself, ie, the initial luminance of the pixels used to generate the sequence. f _obs is the brightness, and if f _obs (I ₀ ) = I ₀ , the special case is that throughout the sequence the motion depends on the brightness of the pixels of the first image. When f _obs (I ₀ ) = (d / dx) I ₀ , the motion changes according to the first derivative of the first image. On the other hand, more complex configurations are also possible, as will be shown next.

[result]
In the case of a pulsating luminance motion function that varies in a sinusoidal manner and an identification matrix as an observation function, the equation for such a sequence is I _k (i, j) = αI ₀ ( i, j) + (1-α) COS (2πf ₀ k · I ₀ (i, j)). Here, the luminance of the pixel varies from 0 (black) to 1 (white). The first frame is α times the first image because k = 0. For k = 1... p, the frame calculates a sequence of matrices in which each pixel changes its luminance in a sinusoidal manner in proportion to the initial luminance.

  By viewing the entire array of images as a movie sequence, it is possible to detect an object with a phase shift generated between two pixels with different initial brightness. Even though these two adjacent pixels have a relatively small difference in the brightness of the original still image, the phase shift will become clearly visible in the movie. An example of this is shown in FIG. 4, using this proportional sinusoidal approach, two pixels with the smallest luminance difference show a 180 ° phase shift in a very short time. .

  FIG. 5 shows the same two pixels changing their luminance from white to black with a sinusoid at a frequency proportional to the initial luminance. Both pixels start with a slightly different gray intensity and are not perceptually visible. After a short time, the phase shift becomes clear so that one pixel represents a faster cycle than the other.

  In one aspect, it resembles an actively changing image contrast and as such is a tool often used in medical image processing. Here, the observer is presented with a sequence of images having a contrast that varies in a sinusoidal manner.

  To illustrate this effect on real images, a sequence of medical images is provided. The sequence of medical images is provided for illustrative purposes only and is not intended to limit the scope or authority of the present invention. The sequence of real medical images consists of 30 frames generated from the static digital mammography picture shown in FIG. 6 using a sinusoidal pulsating luminance motion function and an unsharp mask observation function. ing. FIG. 7 (A) -7 (I) shows an example of this sequence.

  The mammogram presented here includes a malignant cluster of microcalcifications in the slightly visible central right section of the first image shown in FIG. 7 (A). Nevertheless, in some subsequent frames, it becomes quite visible. In this particular example, this is especially true for the frame 15, as shown in FIG. For comparison purposes, the initial image and frame 15 are shown in FIGS. 8 and 9, respectively.

  Obviously, the malignant cluster is much more visible in FIG. 9, and that alone is very useful for diagnosis. However, in addition to improved visibility, the fact that the cluster rapidly circulates from a nearly invisible state to a very visible state gives the cluster a blinking appearance. It is said that something that blinks is very easy to draw attention without looking directly at. The explanation for this lies in the use of motion and flicker sensitivity. Because amplitude sensitivity and central vision are used to focus on a specific point, motion and flicker sensitivity pair with peripheral vision to give a wide field of view and mainly to detect moving or blinking objects Used. Once detected, central vision can be focused on these objects for detailed analysis. This approach is the basic idea behind dynamic queues.

[Statistical analysis]
In order to compare diagnoses with and without dynamic cues, a test was devised that offered the observer one set of static mammograms and the same set once, but supported by dynamic cues. In both cases, the observer diagnosed the presence of microcalcifications in the range from category 1 (not clearly present) to category 5 (clearly present) and at the possible location of each mammogram. Must be issued.

  The results of these tests were analyzed using the Receiver Operating Characteristics (ROC) curve technique. Literature: Charles E. Mets and others, “Statistical comparison of two roc-curve estimates obtained from partially paired datasets”. Medical Decision Making, 18 (1): 110-121, January-March 1998. In this method, a True Positive Fraction (TPF) is plotted through the entire category of diagnosis and plotted against a False Positive Fraction (FPT). A completely random diagnosis equivalent to tossing a coin is a diagonal line connecting the points (0,0) and (1,1) of the ROC plot that produces the lower area (AUC) of the 0.5 curve. I will. A complete diagnosis has an associated ROC curve consistent with the vertical line connecting point (0,0) and point (0,1) and the horizontal line continuing from point (0,1) to point (1,1). I will. The area under the curve is equal to 1. All other tests in the middle will have an AUC between approximately 0.5 and 1. The higher the AUC, the better the test.

200 micron pixels with a resolution between 640 x 640 pixels and 970 x 970 pixels per image and a gray level resolution of 8 bits per pixel, taken from the Mammography Image Analysis Association (MIAS) mammography database A set of n _y = 124 mammograms digitized at the edge was prepared. Literature: John Suckling and others, “The mammographic image analysis society digital mammogram database”, Exerpta Medica. No. 1069 International Congress, pages 375-378, 1994. In total, there are 21 cases with tiny lime deposit clusters and 103 normal cases. This database is typical of clinical cases, as varying degrees of visibility of microcalcifications are included in the set.

  The test was conducted with 5 radiologists and 5 non-radiologists. Scores for ROC curves and associated AUC curves fitted to the resulting binormal were calculated using the ROCKIT program proposed by Metz and others. Two different average calculation techniques were used for both the radiologist and non-radiologist data sets to calculate the representative average. The first technique is a simple standard average. Here, the vertical intercept and slope values fitted to the binormal are averaged, and the curve is calculated from the resulting values. This method tends to overestimate the results but is still within the extremes of the individual results.

A second and possibly even more representative technique is pool average. This average combines all diagnoses of the five participants in the group as if one was analyzing the case of 5 × n _y = 620 for each group of radiologists or non-radiologists And by calculating the resulting ROC curve. Literature: Vit Drga, “The theory of Group Operating Characteristic Analysis in Discrimination Tasks”, PhD paper, Victoria University, Wellington, New Zealand, 1999. This method usually tends to underestimate the results, but it is still well suited for comparison purposes because it underestimates both test categories.

The AUC score and curve parameters of the test results of the radiologist and non-radiologist are listed in Tables 1 and 2 along with their average values.

  As can be seen, diagnostics assisted by dynamic cues have significantly increased the ACU of both groups. Considering a complete diagnosis with an area of 1 and a completely random diagnosis with an area of 0.5, a non-radiologist has an AUC of (0.91-0.806) / (1-0.5) = 20.8%, while the radiologist can conclude that (0.942-0.9) / (1-0.5) = 8.4%.

In general, the individual ROC score for screening for microcalcifications in mammograms is between 0.75 and 0.95. Literature: Xiao-Hua Zhou, Donna K. McClish, and Nancy A. Obuchowski, "Statistical Metohds in Diagnosis Medicine, Wiley series Probability and statistics", Wiley, first edition, July 2002. The scores obtained from our tests are well within these typical ranges. In addition, the score shows very good results for classification using dynamic cues compared to their general score.

  The average ROC curves for both groups are shown in FIGS.

  Tables 1 and 2 show significant improvements in both expert and non-expert diagnosis. If completed as a standard visualization tool for radiologists and configured for clinical use, an 8.4% improvement in radiologists represents a huge potential for improved diagnosis. These tests also take into account that these medical doctors had sufficient experience with traditional static mammography analysis, but never seen dynamic cues in this type of application. Must. The total training provided prior to the test was a training set consisting of four images with and without microcalcifications that were examined using dynamic cues. Despite short training, the results produced something that was demonstrable, a clear indication of the potential of dynamic cues.

  The motion function and observation function, ie, the luminance motion function that pulsates in time and the mask observation function that is not sharp, are exemplary functions. Other configured motion functions may include the lateral oscillating function shown in FIG. 1 and the spatially pulsating function shown in FIG. Other observation functions can include a luminance matrix and a wavelet based transformation. Wavelet transforms based on the methods tested here have been adapted to various algorithms proposed by Wang and others. Literature: Ted C. Wang and Nicolaos B. Karayiannis, “Detection of microcalcifications in digital mammograms using wavelets”, IEEE Transactions on Medical Imaging, 17 (4): 498-509, August 1988.

  The strength of the dynamic cue method lies in the use of the highly evolved human brain functions already provided without throwing away traditional static two-dimensional brain functions, but using the context of image analysis. There is nothing. In short, it utilizes brain functions and all mammals are highly trained for movement.

  FIG. 12 is a block diagram of a computer system useful for configuring the scheme of the present invention. The computer system includes one or more processors, such as processor 100. The processor 100 is connected to a communication infrastructure (for example, a communication bus, a crossover bar, or a network). Various software embodiments are described in terms of this exemplary computer system. After reading this description, it will become apparent to a person skilled in the art how to configure the invention using other computer systems and / or computer architectures.

  The computer system can include a display interface 104 that transfers graphics, text, and other data from the communications infrastructure 102 (or from a frame buffer not shown) for display on the display device 106. The computer system also includes a main storage device 108, preferably a random access storage device (RAM), and an auxiliary storage device 110. The auxiliary storage device 110 includes, for example, a hard disk drive 112 and / or a removable storage drive 114 meaning a floppy disk drive, magnetic tape drive, optical disk drive, and the like. Removable storage drive 114 reads from and / or writes to removable storage device 116 in a manner well known to those skilled in the art. Removable storage device 116 means a floppy disk, magnetic tape, optical disk, etc., but is read and written by removable storage drive 114. As will be appreciated, the removable storage device 116 includes a computer-usable storage medium having computer software and / or data stored therein.

  In an alternative embodiment, auxiliary storage device 110 includes other similar means by which computer programs or other instructions can be loaded into the computer system. Such means include, for example, a removable storage device 118 and an interface 120. Examples of such means include program cartridges and cartridge interfaces (as found in video game devices), removable memory chips (such as EPROM or PROM) and associated sockets, and other removable storage devices 118 and software. And an interface 120 through which data can be transferred from the removable storage device 118 to the computer system.

  The computer system also includes a communication interface 122. Communication interface 122 allows software and data to be transferred between the computer system and external devices. Examples of the communication interface 122 include a modem, a network interface (such as an Ethernet card), a communication port, a PCMCIA slot and a card, and the like. Software and data transferred via the communication interface 122 are in the form of signals that can be received by the communication interface 122, for example, electronic, electromagnetic, optical, or other signals. These signals are provided to the communication interface 122 via a communication path (or channel) 124. This channel 124 carries signals and is configured using wires or cables, optical fibers, telephone lines, cellular phone links, RF links and / or other communication channels.

As used herein, the terms “computer program medium”, “computer usable medium” and “computer readable medium” are installed on the main storage device 108 and the auxiliary storage device 110, the removable storage drive 114 and the hard disk drive 112 Used to get a standard inquiry, such as a hard disk and a signal. These computer program products are means for providing software to a computer system. A computer readable medium allows a computer system to read data, instructions, messages or message packets, and other computer readable information from a computer readable medium. Computer readable media include, for example, non-volatile storage devices such as floppy, ROM, flash memory, disk drive memory, CD-ROM and other permanent storage devices. Computer-readable media are useful, for example, in carrying information such as data and computer instructions between computer systems. In addition, a computer readable medium may be a transitory state medium such as a network link and / or network interface that includes a wired or wireless network that enables the computer to read such computer readable information. A computer program (also called computer control logic) containing medium) computer readable information is stored in main memory 108 and / or auxiliary memory 122. The computer program is also received via the communication interface 122. Such computer programs, when executed, enable the computer system to execute the features of the invention described herein. In particular, the computer program, when executed, enables the processor 100 to execute features of the computer system. Accordingly, such a computer program means a control device of a computer system.

  All references cited herein are hereby expressly incorporated herein by reference in their entirety.

  It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Furthermore, it should be noted that all of the accompanying drawings are not to scale unless the description is contrary to the above. Various modifications and changes can be made in light of the above-described technology without departing from the technical scope and spirit of the present invention.

Schematic of a motion function that oscillates laterally. Schematic of pulsating motion function. Schematic of temporal motion function with pulsating brightness. FIG. 5 is a graph of phase transitions of two pixels having slightly different brightness. The figure which shows the light-intensity phase transition of two pixels which have a slightly different brightness | luminance. Exemplary mammographic image. FIG. 7 shows a sequence of frames of the mammogram picture of FIG. 6 generated using the method of the present invention. The enlarged image of the mammography photograph of FIG. FIG. 7F is an enlarged image of the generated frame. Pool average ROC curve for non-radiologist. Pool average ROC curve for radiologist. FIG. 2 is a schematic diagram of a computer system constituting the method of the present invention.

Claims (20)

Add static cues to static medical images to provide static medical images and generate dynamic medical images,
A method for reading a medical image, comprising: observing a dynamic cue in the dynamic medical image.   The method of claim 1, wherein adding a dynamic cue to a still image includes adding spatial motion to at least one pixel of the still image.   The method of claim 1, wherein adding a dynamic cue to a still image includes adding temporal motion to at least one pixel of the still image over a period of time. The process of adding a dynamic queue to a still image is
Define the motion function,
The method of claim 1 including a process of defining an observation function.   The method of claim 4, wherein the motion function defines a motion of at least one pixel of a still image.   The method according to claim 5, wherein the motion function is a sine wave function, a triangular wave function, a rectangular wave function, or a sawtooth wave function.   6. The method of claim 5, wherein the observation function defines at least one pixel on which a motion function acts.   The method of claim 7, wherein the observation function defines an initial luminance of at least one pixel.   The observation function is a function of an initial luminance of at least one pixel, a derivative of an initial luminance of at least one pixel, a canny filter of an initial luminance of at least one pixel, or at least 9. The method of claim 8, wherein the initial luminance of one pixel is an unsharp mask function or is a wavelet transform of the initial luminance of at least one pixel.   The invention according to claim 1, wherein the dynamic cue is used for identifying a pathology and an abnormality in a medical image. Provide still images,
Define the motion function,
Define the observation function,
Including the process of applying motion and observation functions to still images to generate dynamic images,
The motion function defines a motion of at least one pixel in a still image, and the observation function adds a dynamic cue to the still image defining at least one pixel on which the motion function acts.   The method according to claim 11, wherein the motion function is a periodic function or a vibration function.   The method of claim 11, wherein the observation function defines an initial luminance of at least one pixel.   The observation function is a function of an initial luminance of at least one pixel, a derivative of an initial luminance of at least one pixel, a canny filter of an initial luminance of at least one pixel, or at least 14. The method of claim 13, wherein the initial luminance of one pixel is an unsharp mask function, or is a wavelet transform of the initial luminance of at least one pixel.   The method of claim 11, wherein the motion function adds spatial motion to at least one pixel of a still image.   The method of claim 11, wherein the motion function adds temporal motion to at least one pixel of a still image over a period of time. A computer system including a processor, main storage, auxiliary storage, display, and input device;
A motion function stored in a computer system;
A stationary medical image reading system configured to receive a stationary medical image and store it in an auxiliary storage device.   18. The observation function is configured in a processor to identify at least one pixel of a stationary medical image, and the at least one pixel is identified as a function of an initial luminance of at least one pixel. System.   The system of claim 18, wherein the motion function is configured in a computer processor to add spatial or temporal motion to at least one pixel.   The observation function and the motion function are configured to generate a sequence of frames of a stationary medical image, and the display is configured to display the sequence of frames so that motion is added to the stationary medical image. The invention according to claim 19, wherein the movement is used for identifying a pathology and an abnormality in a still medical image.

Images