JP5550411B2 - Method for estimating patient change patterns - Google Patents

Description

  The present invention relates generally to estimating change patterns, and more particularly to estimating and visualizing change patterns from video acquired using ultrasonography.

  Patient respiration patterns can be used to optimize particle beam therapy. It is intended to estimate a pattern corresponding to “inspiration” and “expiration”. The estimation should be dynamic and real-time to control the energy of the particle beam, so that the maximum dose is delivered to any abnormal tissue while the dose to normal tissue is minimized.

  Instead of breathing information, the depicted target tissue change pattern can also be used to achieve adaptive gating of such treatment systems. For example, the current state, eg, an ultrasound image of the tumor, is compared to a safe radiation state, ie, an ultrasound image where the tumor is in the desired location. If these conditions match, radiation is applied. However, changes in these conditions indicate that the tumor may not be in the desired location and should therefore not be irradiated.

  Evaluation of the accuracy of estimation is also important, though not easy. In theory, if an accurate signal is available, statistical quantities such as Euclidean distance, cross-correlation, or phase correlation between the accurate signal and the estimated pattern can be determined. However, an accurate signal is not available during the procedure.

  As a result, the only way to assess change patterns is by visualization. Once the change pattern is estimated from the ultrasound video, the signal pattern must be visually compared with the video to measure the correlation. In the case of breathing, this visualization does not reveal whether the expected pattern of “inspiration / expiration” in the video matches the estimated pattern or whether the pattern deviates. Must not.

  That is, the radiotherapist needs effective visualization to detect any deviant changes in the breathing pattern during the procedure and to decide whether to continue or end therapy.

  It would also be desirable to provide an effective visualization for determining whether the phase and frequency of the estimated change pattern is consistent with the organ motion seen in the video. Given a 2D video and a change pattern signal, the goal is to emphasize the periodicity of the basic motion in the video and to effectively compare the correlation between the motion in the 2D video and the 1D signal. It is. However, long videos take time to watch.

  Video surveillance applications can visualize events by representing video as a static space-time volume. Traditional volume rendering techniques have visualized basic events after low-level image features, such as gradients or motion flows, have been extracted from the volume and then enhanced. For example, direct volume rendering with a transfer function that assigns high transparency to static areas can hide the background in the scene and can be extracted by applying flow visualization techniques such as glyph or streamline integration. Can visualize the movement flow. A common goal in these techniques is to visualize movement in the environment.

  When viewing an ultrasound video as a conventional movie, the user cannot accurately account for the periodicity, particularly the cycle duration and phase shift in the pattern, which is common in medical images. This is because the phase and frequency of respiration are dynamic. It is also difficult to investigate the correlation between a moving 2D pattern and a 1D signal over a long time interval. For example, video and signals may be positively correlated in one part of the video but negatively correlated in another part. The user cannot recall and detect such differences immediately when the phase of the correlation is continuously lagging.

  Conventional video representations use a 2D display screen as an X, Y coordinate space and change the image (frame) over time, while plots of 1D signal y = f (t) are often 2D The display screen is used as a T, Y coordinate space. This difference makes comparison between 2D spatial video and 1D temporal signal patterns less intuitive, especially when video frames are changing. If a video with thousands of frames is represented as a long volume, only a portion of the video can be displayed on the screen at any point in the Y, T coordinate space.

  It is therefore desirable to visualize 2D video and 1D change signals simultaneously.

  Patient change patterns can be used to optimize radiation therapy using particle beams. Embodiments of the present invention provide a method for estimating patient change patterns from ultrasound video. By using a graphics processing unit (GPU), the change pattern can be estimated and visualized in real time.

  The embodiments provide a visualization scheme for revealing the correlation between the estimated change pattern and the ultrasound video. The change pattern forms a signal, ie a 1D discrete time signal or a time series signal. Visualization comprises an online interface for detecting anomalies. The off-line method provides a static overview of the long video, whereby both the change pattern and the video correlation, frequency, and phase shift can be detected.

  However, the patient's exact pattern of change is not available during the procedure. Therefore, the patient's video is acquired using ultrasonography. The particle beam can then be controlled to maximize the dose delivered to the abnormal tissue while minimizing the dose to normal tissue.

Specifically, the method estimates changes in the patient's change pattern, particularly the breathing pattern.
Divide the ultrasound video into picture groups (GOP). The change model is initialized using the pixels from the first GOP. Based on the change model, a change pattern for the next GOP is estimated, and the change model is changed to match the change pattern. Repeat the estimation and update until the end condition is reached.

1 is a block diagram of a system and method for estimating a change pattern according to an embodiment of the invention. FIG. 3 is an image of a slice of a 3D ultrasound video volume in Y and T coordinate space, according to an embodiment of the invention. 4 is an anti-phase image of change band energies according to an embodiment of the present invention. FIG. 3 is a block diagram of a method for estimating a change pattern according to an embodiment of the present invention. 3 is an image of an online visualization interface according to an embodiment of the present invention. 4 is an image of an offline visualization interface according to an embodiment of the present invention. 4 is an image of an offline visualization interface according to an embodiment of the present invention. 4 is an image of an offline visualization interface according to an embodiment of the present invention. FIG. 3 is a block diagram of a visualization method according to an embodiment of the present invention.

  Embodiments of the present invention provide methods and systems for estimating patient change patterns. The change pattern can be used for particle beam control during radiation therapy and other applications where the interaction between change and treatment is an important factor. Embodiments also provide a method for visualizing a pattern and correlating the pattern to an ultrasound video. It should be understood that the term “patient” can refer generically to any living body that inhales and exhales. It should also be noted that periodic patterns of other organs such as the heart can be visualized.

System FIG. 1 illustrates a radiotherapy configuration using an embodiment of the present invention. The abnormal tissue 100 of the patient 101 is treated with the particle beam 102 generated by the teletherapy device 103 while the patient inhales and exhales (104). The ultrasonic sensor 110 creates time series data in the form of an ultrasonic video 111. The processor 120 performs the steps of the method 400 for estimating the change pattern. The results can be rendered on a display device for a user, eg, a radiation therapy technician. This result can also be used to control the particle beam.

Analysis Change patterns are detected in the video by analyzing the spectrum of pixel intensity over time interval t. The method can track the dynamics of the change by updating the pattern over time. A processor with a graphics processing unit (GPU) 121 is used to analyze the spectrum in real time.

  Visualization comprises an online (real-time) visualization interface and an offline visualization interface. The online interface allows the user to monitor radiation therapy by comparing change signals with video frames. The user aligns the 1D signal with the video frame to maximize the correlation between the basic motion in the video and the change signal. The user can quickly compare correlations within the same 2D coordinate system and can detect when the correlation is gone. Offline visualization provides an overview of change signals and ultrasound video.

  The video is projected into 2D space, and small multiples (small multiples) of patterns corresponding to the same stage in change are generated. The user can quickly identify different patterns and determine the periodicity and phase of the change signal.

Change Estimation In FIG. 2, the intuition supporting the method of the present invention is shown. FIG. 2 shows the rows cascaded in the middle of the first 512 video frames of the ultrasound video. By representing the horizontal dimension as a frame index, we can see the quasi-sinusoidal pattern along the horizontal time direction, because there are pixels whose intensity should be correlated to the changing signal over time. It suggests.

  The method is designed based on this intuition. The method selects a set of pixels whose intensity is similar to the intensity of the change signal over a finite number of time intervals.

Terminology A set of consecutive video frames is automatically divided into picture groups (GOP). The number of video frames in a GOP can be set to any constant or adaptively changing number. In order to utilize a technique based on Fast Fourier Transform (FFT), the number is preferably set to a power of 2, i.e. ²ⁿ .

  The GOP can overlap in time with at most all frames (except the oldest frame, ie the first frame in the GOP). Alternatively, GOPs do not overlap. The amount of overlap determines the computational complexity and the smoothness of the estimated signal. Larger overlap between GOPs results in smoother estimates, however, such overlap requires more computational resources than if there was no overlap at all.

  The frame of the GOP can be the entire input frame or a specific region of interest (ROI) in the input frame. The ROI can be tracked from frame to frame or can correspond to the same location within each input frame.

  The pixel intensity over time partially corresponds to the change pattern signal. Thus, the frequency and spectrum of the signal corresponds to the pixel intensity over time.

  The change patterns considered in the following description are due to respiratory motion. The frequency range of respiration is a respiration band, which is 10 cycles / minute to 18 cycles / minute (this range is a minimum of 5 cycles and a maximum of 30 cycles depending on the age, health condition, etc. of the person). can do). In the signal, the energy in the respiratory band is the respiratory band energy (RBE). As shown in FIG. 3, since the signals are classified into two sets having opposite phases, the amplitude of the average signal is small.

  Given the average spectrum of a set of signals of the same length, the frequency with the largest amplitude in the respiratory band is defined as the primary respiratory frequency of the set of signals. The phase of the main respiratory frequency in the average spectrum is defined as the main respiratory phase of the set of signals. Pattern types over time can include tissue harmonic motion and deformation, as observed in the video. In the following section, we will use “patterns” to refer to all activities in the video.

Method As shown in FIG. 4, the method 400 initializes 410 a modified model 411 (inspiration / expiratory movement, breathing) from the first GOP 112 of the video 111. A model is a collection of pixels in a GOP that is assumed to represent a basic (inspiration / expiration) periodic motion pattern. In other words, the model of the present invention is a subset of pixels and can change at each GOP in the image area.

  The method estimates a change (breathing) pattern 421 for the next GOP based on the model (420).

  The model is updated (430) to fit the patient's change (respiration) dynamics for this GOP.

  The method then repeats the estimation and update for the next GOP until an end condition is reached, eg, the end of the video.

Initialization Consider the breathing model in the following description. On the other hand, the method covers all periodic changes and is therefore called a change model.

  The ROI is selected as the entire frame. The ROI can also be selected by the user as a region in the image that depicts a desired periodic change, eg, a breathing pattern.

  The change model is initialized based on a set of pixels whose intensity in the first GOP contains a distinct periodic pattern at a frequency corresponding to respiration. The RBE of each pixel in the GOP is obtained. Pixels with an RBE greater than the threshold are selected as candidates for modeling respiration, where the number of candidate images and the threshold can be specified by the user. Instead of thresholding, a certain number of pixels with the highest RBE can be selected.

  The intensity of the candidate pixel includes a pattern at a frequency corresponding to respiration. Thus, the average signal can represent the patient's breathing.

  The average signal amplitude of all candidate pixels can be reduced because the phases of the candidate pixels can be reversed as shown in FIG. FIG. 3 shows the signal with the highest RBE from the first GOP. Since the phases of the two groups of signals are opposite to each other, their amplitudes cancel out during averaging, resulting in an unstable signal. This is because the ultrasound video is often noisy. Therefore, only candidate pixels with similar phases are selected to model respiration.

  One naive option groups pixels using a conventional clustering process, such as K-means clustering (here K = 2). However, the K-means process is iterative and time consuming. Therefore, a simpler process is used to cluster the pixels. First, after the average spectrum of candidate pixels is obtained, the frequency having the maximum amplitude in the respiration band in this average spectrum is obtained as the main respiration frequency of all candidate pixels.

  Next, a dominant phase is determined from the phases at the main respiratory frequency of all candidates. The process of the present invention uses the construction of a polarity histogram for the phase. The phase of the bin with the most counts is selected as the dominant phase. The pixel with the phase in this bin is then selected to model respiration.

Estimation For each of the following GOPs, the corresponding respiratory signal is estimated as the average intensity of the selected candidate pixels in the frame. The selected pixel is used until the last frame in the next GOP is processed.

Update When the last frame of the next GOP is reached, a new pixel set for updating the change model is selected from this GOP. The average signal of the newly selected pixel includes a distinct periodic pattern corresponding to respiration and forms a continuous transition from the estimated signal in the previous GOP to the estimated signal in the next GOP.

  In order to generate a continuous phase transition of the respiratory signal, the main respiratory frequency and the main respiratory phase of the previously selected pixel are determined in this next GOP. The pixels are then sorted according to their RBE in this GOP. The pixels are processed in order from the highest RBE to the lowest.

  For each pixel, if the difference between the phase of the main respiratory frequency and the main respiratory phase is less than the threshold, the pixel is selected. The selection ends after a certain number of pixels have been selected.

Implementation The main performance bottleneck of the method is the calculation of the spectrum per pixel. To accelerate the method, steps are performed in real time on the GPU.

Visualization The purpose of the visualization of the present invention is to reveal the correlation between tissue motion and deformation in the ultrasound video and the estimated respiratory signal. Online visualization can effectively compare the observable pattern in the video with the estimated respiratory signal. Offline visualization generates an overview to reveal the correlation between video and respiratory signals within a single image, as well as both the frequency and phase shift of the respiratory signal.

Online Visualization Interface A simple solution is to display video and respiratory signals simultaneously in different windows. However, switching the user focus between windows is inefficient. This is because video frames and breathing signals often use different 2D coordinate spaces and therefore it is difficult to establish comprehensive vision.

  The online visualization interface of the present invention has a plurality of display windows for presenting various information extracted from the ultrasound video. In one window, both video and respiratory signals are displayed simultaneously. Here, the respiration signal and the video frame are overlaid to maximize the correlation between the respiration signal in the video and the periodic motion.

  As shown in FIG. 5, in one of the displays, the user can specify an axis, for example, along the direction of the main distinct periodic motion in the video frame. This axis is called a signal axis 501. Subsequently, the amplitude of the respiratory signal is plotted along the signal axis so that the previous signal is shown in a smaller scale and a different color to indicate the direction of motion. Both ends of the signal axis can be linearly mapped to the signal value range. Each time interval t of the signal can be an axis that is linearly plotted according to its value.

  Each time interval is plotted as blotch 503. The radius of the blotch is inversely proportional to the distance between the time and the current time step. This display window allows multiple signal axes, eg, 501 and 502, to be defined to improve perception. As shown in FIG. 3, two signal axes (501 and 502) are displayed on the frame. The interface also displays the RBE for each pixel. In other words, the RBE and pixel intensity in the original ultrasound frame are mixed in the display window, for example by multiplication. This enhances the visual understanding of the most useful pixels used to estimate inspiratory and expiratory movements.

  In order to make the interface more effective, the breathing pattern is shown across multiple video frames within a single display image. For each video frame, the image intensity is sampled along a signal axis into a single column and this column is attached to the end of the image. By displaying images with corresponding signals, the user can visualize periodic motion and signals over several frames within a single view, which can be useful to detect deviating breaths. .

  Figures 6A-6C show three different results of this expanded visualization. The video and signal axes for the three figures are the same, but the time intervals are different.

  FIG. 6A presents the image after the user has aligned the signal axis with the motion. This indicates that the periodicity of the estimated signal is highly positively correlated with motion. This is unexpected because respiration estimation does not rely on the spatial distribution of patterns in the video frame.

  FIG. 6B shows the case where the pattern is aperiodic in the image and the estimated signal is unstable.

  FIG. 6C shows that the correlation between the signal and the pattern is a negative or inverted correlation, in contrast to the situation of FIG. 6A. The cases in both FIG. 6B and FIG. 6C are deviating and require user interaction during the procedure.

  FIG. 7 shows an overview of the visualization method of the present invention. An input video frame is displayed in the frame window 701. A region of interest (ROI) 702 specified by the user is extracted. The ROIs of a plurality of frames are combined with the spatiotemporal data as a video volume 703 and displayed in the display window 706.

  A respiration estimator method 400 estimates a respiration signal f (t) 704 from the ROI. The estimated signal and the selected pixel are displayed in the signal window 705.

  The ROI and estimated signal are then correlated and visualized in the online visualization interface 706. In the extended visualization interface 707, the signal and pixel intensity along the time-signal in the online visualization interface is displayed and the periodic pattern in the video and the signal are correlated and visualized.

Offline visualization Offline visualization renders static images as an overview of both the ultrasound video and the estimated respiratory signal. In addition to revealing the correlation between video and signal, the overview also reveals the frequency and phase of the respiratory signal.

  The entire video is first treated as a 3D volume and projected onto a 2D image that contains the same periodic pattern as in the video. Next, the 2D image is weighted according to the respiratory signal. This is equivalent to filtering the image pattern corresponding to the peak of the signal. Finally, the weighted image is wrapped around the display boundary, effectively displaying the entire video with the filtered pattern being less compressed and distorted.

  This projection aligns the x-axis of the image with the time dimension of the video. In other words, the number of columns of this projected image is proportional to the number of video frames, and the temporal trend in the video is displayed vertically from left to right.

  The y-axis projection can have different options that determine the contents of each column. Given a column, the content of the column can be obtained by cutting a slice parallel to the time axis through the video volume, or by using pixel intensity sampled over time along the signal axis. . The content of the column is obtained by using conventional volume rendering techniques such as Maximum Intensity Projection (MIP) or Direct Volume Rendering (DVR) to have the sensor 110 coordinate z-axis perpendicular to the time axis and the sensor x-axis to time. It can also be obtained by being parallel to the axis.

  For long video sequences, this rendering cannot be completed in a single pass. Thus, the video volume is rendered as a series of images and registered to form a long image. Use orthogonal projection during rendering to avoid distortion due to perspective projection.

  The projected image is weighted according to the respiratory signal. First, the signal value is linearly normalized to the range [0, 1]. Each row in the image is multiplied by the weighted value of the corresponding time step in the normalized signal. By this weighting, it is possible to emphasize an image column in which the corresponding signal is near the peak and exceeds the other image columns. Continuously enhanced image columns include the projection of patterns in the video.

  By visually comparing the similarity between these patterns, the user can easily observe whether the correlation between the signal and the motion in the video is consistent. The length of each pattern and the distance between successive patterns also reflect the frequency of the estimated signal.

  Since the number of columns is proportional to the number of time intervals, the image may be too long, reducing visualization efficiency. This is because each pattern can only cover a limited space on the display. To ensure that each pattern can cover enough space on the display screen, the long image is wrapped around the display boundary. This wrap makes effective use of 2D screen space and intuitively reveals the phase of the signal.

  If a clear diagonal appears in the image wrapped between the various time steps, the frequency is constant during that period.

  Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Accordingly, it is the object of the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the invention.

Claims (14)

A method for estimating a patient change pattern in an ultrasound video, the method being performed by a processor,
Dividing the patient video acquired by an ultrasonic sensor into picture groups ;
Selecting a pixel group from the first picture group as a change model;
Estimating a change pattern for a next picture group based on the change model;
Updating the change model to match the change pattern;
Repeating the estimation and the updating until an end condition is reached;
Including a method.   The method of claim 1, wherein the picture groups overlap in time.   The method of claim 1, wherein the change pattern indicates a breathing pattern of the patient.   The initialization is based on a set of pixels in the first picture group, wherein the intensity of the set includes a distinct periodic pattern at a frequency corresponding to respiration. the method of. The breathing frequency range is a breathing band, the energy in the breathing band is breathing band energy, and the method comprises:
Determining the breathing band energy for each pixel in the picture group, and selecting a pixel having a breathing band energy greater than a threshold and a similar phase with respect to initialization of the change model;
The method of claim 4, further comprising: The change pattern indicates a breathing pattern of the patient, the frequency range of the breathing pattern is a breathing band, the energy in the breathing band is breathing band energy, and the method includes:
Determining the respiratory band energy for each of the pixels in the picture group, and selecting the number of pixels having the largest respiratory band energy for initialization of the change model;
The method of claim 4, further comprising:   5. The method of claim 4, further comprising grouping the selected pixels in the initialized change model into similar phases. Constructing a bin polarity histogram for the phase; and selecting the phase with the largest count as the dominant phase.
The method of claim 4, further comprising:   The method of claim 4, wherein the pattern is estimated as an average intensity of the selected pixels.   The method of claim 1, further comprising controlling a particle beam according to the pattern.   The method of claim 1, further comprising visualizing the pattern as a signal for each picture group in real time.   The method of claim 11, wherein the visualization displays the video and the signal simultaneously in a single display window.   The method of claim 1, further comprising visualizing the entire video and the pattern as a signal offline.   The method of claim 13, further comprising weighting the frames of video according to the signal.

Images