JP6305250B2 - Image processing apparatus, treatment system, and image processing method - Google Patents

Description

  Embodiments described herein relate generally to an image processing apparatus, a treatment system, and an image processing method.

  Radiotherapy, heavy particle beam therapy, and the like are treatments that destroy a lesion by irradiating the lesion with radiation or a heavy particle beam. When performing radiotherapy or heavy particle beam therapy, it is difficult to obtain a therapeutic effect unless the lesion is accurately irradiated with radiation or heavy particle beam. Therefore, CT (Computed Tomography) imaging of a treatment target (for example, a subject) is performed, and the position of the lesion is grasped three-dimensionally. The doctor or engineer determines a treatment plan including an angle at which radiation or heavy particle beam is irradiated, irradiation intensity, and the like based on the grasped three-dimensional position of the lesion. When treatment is actually performed, positioning is performed so that the position of the subject at the time of treatment matches the position of the subject when the treatment plan is determined. Since the treatment is performed a plurality of times based on the treatment plan, positioning is performed each time. Since it is necessary to maintain the posture of the subject from the start of the positioning to the end of the treatment, the positioning is preferably performed in a short time. However, since accuracy is also required for positioning, it may be difficult to perform positioning in a short time.

JP 2011-212130 A

  The problem to be solved by the present invention is to provide an image processing device, a treatment system, and an image processing method capable of improving the positioning accuracy and reducing the time required for positioning.

  The image processing apparatus according to the embodiment includes an image acquisition unit, a reconstruction unit, a point acquisition unit, a point detection unit, a calculation unit, and a determination unit. The image acquisition unit acquires a plurality of first fluoroscopic images obtained by photographing the target from at least two directions different from each other. The reconstruction unit reconstructs a plurality of second fluoroscopic images obtained when the object is photographed from at least two different directions from the volume data of the object acquired in advance. The point acquisition unit acquires a plurality of set points in the first perspective image or the second perspective image. When the set point is a point in the first perspective image, the point detection unit detects a corresponding point corresponding to the set point in the second perspective image, and the set point is a point in the second perspective image The corresponding point corresponding to the set point is detected in the first perspective image. The calculation unit calculates a displacement amount of the target position when the second perspective image is generated with respect to the target position when the first perspective image is captured based on the set point and the corresponding point. The determination unit determines whether the displacement amount satisfies a predetermined condition. When the displacement amount does not satisfy the predetermined condition, the reconstruction unit reconstructs a plurality of second fluoroscopic images from the volume data moved based on the displacement amount, and the reconstruction unit reconstructs the point detection unit. Corresponding points are detected in the second fluoroscopic image and the first fluoroscopic image, and the calculation unit calculates the displacement amount again based on the newly detected corresponding point and the set point.

1 is a block diagram showing a configuration of a treatment system 10. FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus 100. FIG. It is a figure which shows the process at the time of producing | generating a reconstructed image. 4 is a diagram illustrating an arrangement of a first radiation source unit 420, a second radiation source unit 430, a first imaging unit 440, and a second imaging unit 450 in the imaging apparatus 400. FIG. It is a figure which shows the point corresponding between a fluoroscopic image and a reconstruction image. 5 is a flowchart illustrating a displacement amount calculation process performed by the image processing apparatus 100. It is a block diagram which shows the structure of 100 A of image processing apparatuses. It is a flowchart which shows the displacement amount calculation process which 100A of image processing apparatuses perform.

  Hereinafter, an image processing apparatus, a treatment system, and an image processing method according to embodiments will be described with reference to the drawings. Note that, in the following embodiments, the same reference numerals are assigned to the same operations, and duplicate descriptions are omitted as appropriate.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of a treatment system 10 in the first embodiment. The treatment system 10 includes an image processing device 100, an input / output device 200, and an imaging device 400. The treatment system 10 may further include a planning device 300, a treatment device 500, and a bed device 600. In the treatment system 10, a user such as an engineer or a doctor operates the input / output device 200, the imaging device 400, the treatment device 500, and the bed device 600 on the basis of the treatment plan set by using the planning device 300. Give treatment. The user operates the input / output device 200 based on the reconstructed image generated by the image processing device 100 and the fluoroscopic image captured by the imaging device 400.

  The planning device 300 is a device that determines a treatment plan for the subject A to be subjected to radiation therapy, proton beam therapy, particle beam therapy, or the like. The planning device 300 determines a treatment plan based on information such as an image obtained by photographing the internal form of the target A and an operation input by a user such as an engineer or a doctor. The image used in the planning device 300 is an image taken by an imaging device that can take a picture through the inside of the target A. Examples of such an imaging apparatus include an X-ray apparatus, a computer tomography (CT) apparatus, a magnetic resonance imaging (MRI) apparatus, a PET (Positron Emission Tomography) apparatus, and a SPECT (Single Photon Emission Computed). Tomography) devices. Further, the image used in the planning apparatus 300 may be either a two-dimensional image or a three-dimensional image. In the present embodiment, a case will be described in which an image based on volume data collected by an X-ray CT apparatus is used when determining a treatment plan.

  The planning apparatus 300 includes a database unit 310, a display unit 320, an operation unit 330, and a control unit 340. The database unit 310 stores data obtained by photographing the target A. The data stored in the database unit 310 may be voxel data itself obtained by photographing the target A, or logarithmic conversion, offset correction, sensitivity correction, beam, and the like for the data obtained by photographing. It may be voxel data composed of data subjected to correction processing such as hardening correction and scattered radiation correction. In addition to the voxel data, the database unit 310 may store a two-dimensional image reconstructed from the voxel data. In the present embodiment, a case where voxel data is stored as volume data in the database unit 310 will be described.

  The display unit 320 displays a reconstructed image based on control by the control unit 340. A reconstructed image is obtained by reconstructing voxel data stored in the database unit 310. In the present embodiment, a case where an image (DRR: Digitally Reconstructed Radiograph) obtained by simulating an image obtained when the object A is seen through from a predetermined direction is used as a reconstructed image will be described. The reconstructed image displayed by the display unit 320 preferably corresponds to the type of image captured by the imaging device 400. For example, when the imaging apparatus 400 is an X-ray apparatus, the reconstructed image used in the display unit 320 is preferably a DDR image that simulates an image captured by the X-ray apparatus.

  The operation unit 330 receives an operation input by the user and sends information corresponding to the received operation input to the control unit 340. The control unit 340 controls the operation of each unit provided in the planning device 300 based on the information received from the operation unit 330. The control unit 340 is an information processing apparatus having, for example, a central processing unit (CPU), and performs a control operation based on a specific program. The control unit 340 causes the database unit 310 to store information indicating the site of the target A to be treated based on the reconstructed image and information corresponding to the operation input.

  The imaging apparatus 400 is an X-ray apparatus or the like, and is an apparatus that performs imaging while seeing through the inside of the target A at the time of treatment. In the present embodiment, a case where the imaging apparatus 400 is an X-ray apparatus will be described. The imaging apparatus 400 includes a control unit 410, a first radiation source unit 420, a second radiation source unit 430, a first imaging unit 440, and a second imaging unit 450.

  The first imaging unit 440 generates a fluoroscopic image in which the target A is seen through based on the X-rays emitted from the first radiation source unit 420 and transmitted through the target A. The first imaging unit 440 includes a flat panel detector (FPD). The FPD receives X-rays transmitted through the object A and converts them into digital signals. The first imaging unit 440 generates a fluoroscopic image based on the digital signal obtained by the FPD.

  The second imaging unit 450 generates a fluoroscopic image in which the target A is seen through based on the X-rays emitted from the second radiation source unit 430 and transmitted through the target A. Similar to the first imaging unit 440, the second imaging unit 450 has an FPD. The second imaging unit 450 generates a fluoroscopic image based on the digital signal obtained by the FPD.

  Note that the direction in which the first imaging unit 440 sees through the object A is different from the direction in which the second imaging unit 450 sees through the object A. For example, the first radiation source unit 420, the second radiation source unit 430, the first imaging unit 440, and the FPD imaging surface of the first imaging unit 440 and the FPD imaging surface of the second imaging unit 450 are orthogonal to each other. The second imaging unit 450 is arranged. The first imaging unit 440 and the second imaging unit 450 may have an image intensifier (II) instead of the FPD.

  The control unit 410 controls each unit included in the imaging device 400. The control unit 410 is an information processing apparatus having a central processing unit (CPU), for example, and performs a control operation based on a specific program. The control unit 410 sends a pair of perspective images of the target A generated by the first imaging unit 440 and the second imaging unit 450 to the image processing apparatus 100.

  The image processing apparatus 100 acquires shooting parameters when shooting a fluoroscopic image of the target A from the imaging apparatus 400. In addition, the image processing apparatus 100 reads volume data used when the treatment plan is determined from the database unit 310 of the planning apparatus 300. The image processing apparatus 100 generates a reconstructed image from the same viewpoint as the viewpoint at which the imaging apparatus 400 captured the fluoroscopic image of the target A based on the imaging parameters.

  The image processing apparatus 100 acquires from the input / output device 200 a set of set points indicating an arbitrary part of the target A defined by the user in the fluoroscopic image or the reconstructed image. The set point is determined by the user operating the input / output device 200 based on the perspective image and the reconstructed image displayed on the input / output device 200. When the set point is set on the fluoroscopic image, the image processing apparatus 100 detects a corresponding point corresponding to the set point on the reconstructed image. Further, when a set point is determined on the reconstructed image, the image processing apparatus 100 detects a corresponding point corresponding to the set point on the fluoroscopic image.

  Here, the corresponding point is the center point of the partial image having the highest degree of similarity with the image in the rectangular area having a predetermined size with the set point as the center. When the set point is determined on the fluoroscopic image, the corresponding point on the reconstructed image is the center point of the partial image most similar to the partial image near the set point. When a set point is set on the reconstructed image, the corresponding point in the fluoroscopic image is the center point of the partial image that is most similar to the partial image near the set point.

  Based on the set point and the corresponding point, the image processing apparatus 100 calculates a deviation between the position of the target A when the fluoroscopic image is captured and the position of the target A when the reconstructed image is generated. The image processing apparatus 100 outputs positioning information based on the calculated deviation to the bed apparatus 600. In the following description, the shift between the position of the target A when the fluoroscopic image is captured and the position of the target A when the reconstructed image is generated is referred to as a displacement amount.

  The input / output device 200 includes a display unit 210 and an operation unit 220. The display unit 210 receives image data from the image processing apparatus 100. The display unit 210 displays the received image data. The image data includes, for example, a reconstructed image generated based on volume data, a fluoroscopic image taken by the imaging device 400, and the like.

  The operation unit 220 receives an operation input by a user and outputs information corresponding to the operation input to the image processing apparatus 100. The operation unit 220 receives an operation input for designating a position on the image displayed on the display unit 210, and supplies coordinate information indicating the position to the image processing apparatus 100. The operation unit 220 includes a pointing device such as a mouse and a touch panel, a keyboard, and the like. When the operation unit 220 includes a touch panel, the display unit 210 and the operation unit 220 may be configured as one device.

  The treatment apparatus 500 is an apparatus that performs radiation therapy, proton beam therapy, or particle beam therapy on the subject A. The treatment device 500 includes a control unit 510 and a plurality of radiation source units 520. The control unit 510 controls each unit included in the treatment apparatus 500. The control unit 510 is an information processing apparatus having a central processing unit (CPU), for example, and performs a control operation based on a specific program. Control unit 510 makes radiation source unit 520 movable after detecting that bed apparatus 600 has moved object A based on the amount of displacement. Each of the plurality of radiation source units 520 emits radiation, proton beams, or particle beams toward the target A based on user control when in an operable state. The plurality of radiation source units 520 are arranged so that the radiation, proton beam, or particle beam irradiated from each radiation source unit 520 intersects at one point (isocenter).

  The bed apparatus 600 includes a movable table on which the object A is placed. The bed apparatus 600 moves the movable table on which the subject A is placed based on the positioning information acquired from the image processing apparatus 100. Thereby, the site | part of the object A and the isocenter which were determined at the time of treatment planning are matched. In addition, what has shapes, such as a bed and a chair, is used for a movable stand.

  FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus 100 according to the first embodiment. As illustrated in FIG. 2, the image processing apparatus 100 includes a perspective image acquisition unit 101, a reconstructed image generation unit 102, a point acquisition unit 103, a point detection unit 104, a calculation unit 105, a determination unit 106, and an image data generation unit 107. Prepare. In the image processing apparatus 100 and the treatment system 10 in the first embodiment, the isocenter of the treatment apparatus 500 is set as the origin of the world coordinate system for positioning. Further, for the three-dimensional coordinates used in the image processing apparatus 100 and the treatment system 10, a world coordinate system having an isocenter as the origin is used.

  Here, an X-ray image and a reconstructed image as a fluoroscopic image to be subjected to image processing in the image processing apparatus 100 will be described.

The X-ray image is an image obtained by converting the magnitude of the energy of the X-ray when the X-ray irradiated from the X-ray source toward the subject passes through the subject and reaches the FPD into a pixel value. In the FPD, an X-ray detector is arranged on a two-dimensional plane, and energy detected by each X detector is converted into a pixel value. Since the energy of the X-ray when reaching the FPD is attenuated according to the tissue in the subject, the X-ray image is an image seen through the subject. The energy P _i of the X-ray that reaches the X-ray detector at the position iεR ² of each pixel in the X-ray image can be expressed by the following equation (1).

In Expression (1), P ₀ is the energy of X-rays when entering the subject. μ (l, p) is the linear attenuation coefficient of the object at position l. The source weak coefficient is a value that changes according to the energy P of X-rays passing through the substance. A value obtained by performing line integration of the source weak coefficient of the substance on the X-ray path from the radiation source to the X-ray detector arranged at the pixel position i reaches the X-ray detector. X-ray energy. The detection characteristics of the X-ray detector is designed to be linear with respect to those taking the logarithm of P _i, X-ray image by linearly converting a signal X-ray detector outputs the pixel value Is obtained. That is, the pixel value T _i of each pixel of the X-ray image can be expressed by the following equation (2). Note that log (P ₀ ) is a constant.

  As described above, each pixel of the X-ray image obtained by X-ray imaging is obtained by multiplying the product sum of the source A weak coefficients of the object A on the path where the X-rays irradiated from the source reach the X-ray detector of the FPD. It is converted into a pixel according to.

  A reconstructed image (DRR: Digitally Reconstructed Radiograph) is generated by perspective projection from an arbitrary direction, for example, when an object A represented by volume data is virtually arranged on the bed apparatus 600. FIG. 3 is a diagram illustrating processing when a reconstructed image is generated. The coordinates in the three-dimensional coordinate system with the isocenter as the origin are (x, y, z), and the two-dimensional coordinates in the reconstructed image are (u, v). The pixel value I (u, v) of the pixel at the coordinates (u, v) of the reconstructed image is calculated by the following equation (3).

  In Expression (3), V (x, y, z) is a value of volume data at the coordinates (x, y, z) of the target A virtually arranged on the bed apparatus 600. Equation (3) shows that the pixel value I (u, v) is obtained by integrating the value of the volume data on the light beam L. W (V) is a weighting factor applied to the value of the volume data. By controlling the weighting factor W (V), a reconstructed image in which the value of specific volume data is emphasized can be generated. The control of the weighting factor W (V) can enhance the tissue that is noticed when collating the reconstructed image and the X-ray image, or can enhance the visibility by enhancing the tissue that the user is interested in. .

  The data value V (x, y, z) is a value based on the source weak coefficient of the substance located at the position (x, y, z). Therefore, when the reconstructed image is generated using the sum of the source weak coefficients of the substances on the path of the light L, the X-ray image is also the sum of the source weak coefficients on the light as shown in Expression (2). Since the pixel value is determined, the reconstructed image and the X-ray image are similar.

  In order to generate a reconstructed image, it is necessary to determine the ray path L and the position of the volume data of the object A. When positioning the target A in the treatment system 10, a reconstructed image is generated based on the path through which X-rays in the imaging apparatus 400 when the fluoroscopic image of the target A is taken during treatment reaches the FPD. The light path L and the position of the object A are determined.

  FIG. 4 is a diagram illustrating an arrangement of the first radiation source unit 420, the second radiation source unit 430, the first imaging unit 440, and the second imaging unit 450 in the imaging apparatus 400 that captures fluoroscopic images from two different directions. X-rays emitted from the first radiation source unit 420 pass through the object A and reach the first imaging unit 440. The first imaging unit 440 generates a fluoroscopic image based on the X-ray energy that has passed through the target A. Similarly, the X-rays emitted from the second source unit 430 pass through the target A and reach the second imaging unit 450. The second imaging unit 450 generates a fluoroscopic image based on the energy of X-rays that have passed through the target A.

  The imaging apparatus 400 is calibrated for the imaging position, and includes a three-dimensional space coordinate system determined for the treatment system and a two-dimensional coordinate system on the imaging surface of the first imaging unit 440 and the second imaging unit 450. A perspective projection matrix for performing a coordinate revealer is obtained in advance. Thus, if the imaging apparatus 400 has been calibrated, the positional relationship between the first radiation source unit 420, the first imaging unit 440, the second radiation source unit 430, and the second imaging unit 450 is already known. It is possible to determine the light path L used when generating the reconstructed image. Specifically, the start point of the path L is the first source unit 420 and the end point is each pixel on the imaging surface of the first imaging unit 440. The starting point of the path L is the second source unit 430 and the end point is each pixel on the imaging surface of the second imaging unit 450.

  As described above, when the reconstructed image is generated using the same path L as the X-ray path when the fluoroscopic image is captured, the position of the volume data of the target A is that of the target A when the fluoroscopic image is captured. If it is the same as the position, the reconstructed image and the fluoroscopic image are most similar.

  In other words, when positioning is performed, the user moves the volume data so that the two fluoroscopic images photographed from two directions and the two reconstructed images are most similar to each other, and searches for a matching position. Find the amount. In order to move the volume data, it is necessary to define six parameters of rotation and translation around the XYZ axes. In order to determine these six parameters, as shown in FIG. 5, corresponding points are determined in each of the fluoroscopic image and the reconstructed image, and the displacement amount is calculated based on the corresponding points. FIG. 5 is a diagram illustrating an example of corresponding points between the fluoroscopic image and the reconstructed image. In FIG. 5, corresponding points in the fluoroscopic image and the reconstructed image are indicated by black dots.

  When a point is input on one of the fluoroscopic image and the reconstructed image, the image processing apparatus 100 detects a corresponding point in the other image and calculates a displacement amount. The image processing apparatus 100 generates a reconstructed image after moving the volume data of the target A based on the calculated displacement amount, detects the corresponding point again, and calculates the displacement amount. The image processing apparatus 100 repeats the detection of the corresponding points and the calculation of the displacement amount, thereby saving the user from inputting the corresponding points and improving the positioning accuracy.

  The fluoroscopic image acquisition unit 101 acquires a pair of fluoroscopic images of a first fluoroscopic image and a second fluoroscopic image of the target A that are captured by the imaging device 400. In addition, the fluoroscopic image acquisition unit 101 acquires the camera parameters of the first imaging unit 440 that has captured the first fluoroscopic image and the camera parameters of the second imaging unit 450 that has captured the second fluoroscopic image. The fluoroscopic image acquisition unit 101 supplies the acquired pair of fluoroscopic images to the point detection unit 104. The perspective image acquisition unit 101 supplies the acquired two camera parameters to the reconstructed image generation unit 102 and the point acquisition unit 103.

  The reconstructed image generation unit 102 reads volume data from the database unit 310 of the planning device 300. The reconstructed image generation unit 102 acquires two camera parameters from the perspective image acquisition unit 101. Also, the reconstructed image generation unit 102 acquires the displacement amount from the calculation unit 105. The reconstructed image generation unit 102 generates a reconstructed image for each camera parameter based on the volume data, the displacement amount, and the two camera parameters. The displacement amount is a value calculated by the calculation unit 105 based on a pair of reconstructed images and a pair of perspective images generated for each camera parameter. Further, the displacement amount is a value representing a deviation between the position of the object A represented by the volume data and the position of the object A when the fluoroscopic image is captured. When the reconstructed image generation unit 102 first generates a reconstructed image, the displacement amount is set to zero because the displacement amount is not calculated. The reconstructed image generation unit 102 supplies the generated pair of reconstructed images to the point detection unit 104.

  The point acquisition unit 103 acquires, from the input / output device 200, two-dimensional coordinates in an image of a point group including points corresponding to at least three parts of the target A defined by the user in the perspective image or the reconstructed image. In the following description, a point determined by the user is referred to as a set point. As the set point, for example, a point determined by the user that there is a feature in the fluoroscopic image or the reconstructed image, a unique point on the image, or the like is selected. In either case where the set point is determined in the fluoroscopic image or in the reconstructed image, a point corresponding to the site of the target A in each of the images from different directions is determined.

  Each set point included in the point group acquired by the point acquisition unit 103 is a point indicating the same position on the object A in a perspective image or a reconstructed image taken from two different directions. Therefore, the three-dimensional position of the object A can be calculated from the position of each set point as follows.

  A position of a set point on an image captured by the imaging apparatus 400 whose projection matrix P included in the camera parameter is known is assumed to be a vector x = (x, y, 1). If the position of the target at the set point in the three-dimensional space is a vector X = (X, Y, Z, 1), the following equation (4) is established.

In formula (4), λ is a constant representing the indefiniteness of the scale factor. Here, two different when the vector x ₁ and vector x ₂ is the same position on the object A on the captured image from the direction is specified, holds the simultaneous equations of the formula (5).

In Expression (5), P ₁ and P ₂ are projection matrices when imaged from two directions, and λ ₁ and λ ₂ are constants representing scale indefiniteness. By solving Equation (5) for the vector X, the three-dimensional coordinates of the set point can be obtained. The point acquisition unit 103 calculates the three-dimensional coordinates of each set point included in the acquired point group. The point acquisition unit 103 supplies coordinate information including the three-dimensional coordinates of the point group to the point detection unit 104 and the calculation unit 105.

  Note that the point acquisition unit 103 may acquire the three-dimensional coordinates of the point group in the target A set in the past treatment. In addition, the point acquisition unit 103 may acquire a point defined in the cross-sectional view of the target A generated based on the volume data as a set point, and may acquire three-dimensional coordinates based on the volume data. Even when the position of the point group is acquired as a three-dimensional coordinate, the point acquisition unit 103 can acquire a point on the image by converting it into a two-dimensional coordinate on the image based on Expression (5). Moreover, since the treatment of the subject A by the treatment system 10 is often performed a plurality of times, the point acquisition unit 103 may acquire a plurality of set points determined in the past treatment.

  The point detection unit 104 detects a corresponding point corresponding to each set point included in the point group acquired from the point acquisition unit 103. For each set point included in the point group, the point detection unit 104 extracts a partial image near the set point using a rectangular window having a predetermined size centered on the set point. When the set point is a point on the fluoroscopic image, the point detection unit 104 searches the reconstructed image for a partial image having the highest similarity with the extracted partial image. The point detector 104 sets the center of the detected partial image on the reconstructed image as a corresponding point corresponding to the set point. In addition, when the set point is a point on the reconstructed image, the point detection unit 104 searches the perspective image for a partial image having the highest similarity with the detected partial image. The point detection unit 104 sets the center of the detected partial image on the perspective image as a corresponding point corresponding to the set point.

  The point detection unit 104 uses normalized cross-correlation, for example, for calculating similarity. Note that other image feature amounts may be used as the similarity. For example, a vector type feature quantity may be used for a feature quantity based on a luminance gradient, a SIFT feature quantity, an LBP feature quantity, or a similarity. In the case of using a vector-type feature amount, the similarity is increased as the distance between the vectors is smaller.

  The point detection unit 104 calculates the three-dimensional coordinates of each detected corresponding point based on Expression (5). The point detection unit 104 supplies coordinate information including the three-dimensional coordinates of each detected corresponding point to the calculation unit 105.

  The calculation unit 105 calculates a displacement amount based on the coordinate information of the point group acquired by the point acquisition unit 103 and the coordinate information of each corresponding point calculated by the point detection unit 104. The calculation unit 105 generates the reconstructed image and the position of the target A when the fluoroscopic image is captured based on points corresponding to the four images of the fluoroscopic image and the reconstructed image captured from two different directions. The position of the target A at the time is calculated.

The calculation unit 105 sets the three-dimensional position of the volume data of the target A as X _i , the three-dimensional position Y _i of the target A when the fluoroscopic image is taken, and the target A changes from the three-dimensional position X _i to the three-dimensional position Y _i. The amount of displacement when moving to is calculated. At this time, if the object A is moving rigidly including rotation and translation in the three-dimensional space, the following expression (6) is satisfied.

The matrix R in Equation (6) is a 3 × 3 rotation matrix, and the vector t is a 3 × 1 translation vector. The matrix R and the vector t are displacement amounts calculated by the calculation unit 105. Displacement amount using simultaneous equations assuming that equation (6) holds for the three-dimensional positions X _i , Y _i (i = 1, 2,..., N) of all corresponding points obtained as set points and corresponding points. Is calculated.

If the center of gravity of the three-dimensional position X _i (i = 1, 2,..., N) is ￣X and the center of gravity of the three-dimensional position Y _i (i = 1, 2,..., N) is ￣Y, then the translation vector. t is expressed as the following equation (7). Therefore, if the rotation matrix R is obtained, the translation vector t is determined. “￣X” indicates a parameter with an overline on X for convenience. In addition, when an overline is added to another parameter, “パ ラ メ ー タ (overline)” is added in front of the parameter similarly.

Here, when X ′ _i = X _i −￣X and Y ′ _i = Y _i −￣Y, a rotation matrix R represented by the following equation (8) is calculated. The problem of obtaining such a matrix R is known as an Orthogonal Procrustes problem, and there are various solutions. The calculation unit 105 calculates the rotation matrix R using any of known solutions. Further, the calculation unit 105 calculates the translation vector t by substituting the calculated rotation matrix R into Expression (7). The calculation unit 105 supplies the displacement amount including the calculated rotation matrix R and translation vector t to the determination unit 106 and the reconstructed image generation unit 102.

  The determination unit 106 determines whether or not the displacement amount calculated by the calculation unit 105 satisfies a predetermined criterion. The determination unit 106 outputs positioning information including the displacement amount to the bed apparatus 600 when the displacement amount satisfies the reference. When the displacement amount does not satisfy the standard, the determination unit 106 performs control for causing the reconstructed image generation unit 102 to generate a reconstructed image, causing the point detection unit 104 to detect corresponding points, and causing the calculation unit 105 to calculate the displacement amount. Do.

  Here, the reference for the displacement amount will be described. The determination unit 106 determines that the displacement amount satisfies the standard when the similarity between the reconstructed image and the fluoroscopic image is equal to or greater than a predetermined threshold. The reconstructed image used for the determination is a reconstructed image generated by the reconstructed image generation unit 102 moving the volume data of the target A based on the displacement amount. For example, normalized cross-correlation is used to calculate the similarity between the reconstructed image and the fluoroscopic image. Instead of calculating normalized cross-correlation, the degree of similarity is calculated using the mutual information amount, the degree of similarity of the histogram of each pixel value of each image is calculated, and the pixel value of each image is normalized. Or a square error may be calculated. Further, the similarity may be calculated based on the feature amount based on the gradient of the pixel value in each image.

  In addition, when the similarity between the reconstructed image and the fluoroscopic image is calculated, the determination unit 106 replaces the calculation of the similarity of the entire image with a partial image having a certain size including the set point and the corresponding point. Local similarity may be calculated. That is, the determination unit 106 may use the local similarity of the partial images near each of the set point and the corresponding point as the similarity between the reconstructed image of the volume data moved based on the displacement amount and the fluoroscopic image. .

  Further, when the displacement amount is repeatedly calculated, the determination unit 106 determines that the similarity between the fluoroscopic image and the reconstructed image is lower than the similarity calculated at the previous determination. Then, the operation of repeatedly calculating the displacement amount is stopped, and positioning information including the previous displacement amount is output to the bed apparatus 600.

  As a reference for the displacement amount, the rotation angle around the XYZ axes and the magnitude of the translation vector t are calculated based on the rotation matrix R included in the displacement amount instead of the similarity between the reconstructed image and the fluoroscopic image. Also good. At this time, the determination unit 106 determines whether or not each rotation angle and size are equal to or less than a predetermined threshold value. The determination unit 106 outputs positioning information including a displacement amount to the bed apparatus 600 when each rotation angle and size is equal to or less than a threshold value. When any one of the rotation angles and the magnitudes is larger than the threshold, the determination unit 106 generates a reconstructed image in the reconstructed image generation unit 102, detects a corresponding point in the point detection unit 104, and shifts in the calculation unit 105. Control is performed to calculate.

  Further, the determination unit 106 may determine whether or not the calculation of the displacement amount has been performed a predetermined number of times, instead of using the similarity between the reconstructed image and the fluoroscopic image or the determination using the displacement amount. At this time, when the displacement amount is calculated a predetermined number of times, the determination unit 106 outputs positioning information including the finally calculated displacement amount to the bed apparatus 600. The determination unit 106 may determine the number of repetitions according to the tissue of the target A projected on the reconstructed image. When the determination unit 106 determines the number of repetitions, the determination unit 106 identifies the tissue of the target A to be projected based on the volume data value V (x, y, z) used when generating the reconstructed image. . The organization of the target A may be identified based on the value pattern of the volume data. The determination unit 106 may identify the tissue of the target A based on information included in DICOM (Digital Imaging and Communication in Medicine) data or information input by the user. For example, the determination unit 106 calculates the width of the region of the bone portion in the reconstructed image. When the ratio of the bone region in the reconstructed image is equal to or greater than a predetermined threshold, the determination unit 106 makes the number of repetitions smaller than the reference number. The determination unit 106 increases the number of repetitions more than the reference number when the ratio of the bone region in the reconstructed image is smaller than the threshold. This is because, when a fluoroscopic image is acquired by an X-ray apparatus, the more bone portions, the higher the contrast in the fluoroscopic image and the higher the detection accuracy of corresponding points. This is because the amount of displacement can be obtained. Based on this tendency, the determination unit 106 may decrease the number of repetitions as the contrast of at least one of the reconstructed image and the fluoroscopic image is higher.

  The image data generation unit 107 detects the perspective image acquired by the perspective image acquisition unit 101, the reconstructed image generated by the reconstructed image generation unit 102, the set point acquired by the point acquisition unit 103, and the point detection unit 104 The image data including the corresponding points is generated. At this time, the image data generation unit 107 may generate image data in which the set point and the corresponding point are superimposed on the fluoroscopic image or the reconstructed image. The image data generation unit 107 outputs the generated image data to the input / output device 200.

  FIG. 6 is a flowchart illustrating a displacement amount calculation process performed by the image processing apparatus 100. In the image processing apparatus 100, when the displacement amount calculation process is started, the fluoroscopic image acquisition unit 101 acquires a plurality of fluoroscopic images photographed from different directions and camera parameters corresponding to the fluoroscopic images from the imaging apparatus 400 (steps). S101).

The reconstructed image generating unit 102 generates a reconstructed image corresponding to each fluoroscopic image based on the volume data of the target A and the camera parameters acquired by the fluoroscopic image acquiring unit 101 (step S102).
The point acquisition unit 103 acquires a point group including at least three set points designated by the user (step S103).

The point detection unit 104 detects a point corresponding to each set point in the fluoroscopic image or the reconstructed image. The point detection unit 104 sets the detected point as a corresponding point (step S104).
Based on the three-dimensional coordinates of the set points and the three-dimensional coordinates of the corresponding points, the calculation unit 105 calculates a displacement amount of the position of the volume data with respect to the position when the fluoroscopic image is captured (step S105).

The reconstructed image generation unit 102 generates a reconstructed image corresponding to each fluoroscopic image based on the volume data moved using the displacement amount and the camera parameter (step S106).
The determination unit 106 calculates the similarity between the reconstructed image generated in step S105 and the fluoroscopic image (step S107). The determination unit 106 determines whether or not the calculated similarity is greater than or equal to a threshold value (step S108).

When the degree of similarity is not equal to or greater than the threshold (step S108: NO), the determination unit 106 returns the process to step S104, and repeatedly performs the processes after step S104.
When the similarity is equal to or greater than the threshold (step S108: YES), the determination unit 106 outputs positioning information including the displacement amount to the bed apparatus 600 (step S109), and ends the displacement amount calculation process.

  When the image processing apparatus 100 according to the first embodiment acquires a set point indicating a specific part of the target A in a plurality of fluoroscopic images, the image processing apparatus 100 acquires a corresponding point corresponding to the set point in the plurality of reconstructed images. The amount of displacement is calculated based on the corresponding points. In addition, when the image processing apparatus 100 acquires a set point indicating a specific part of the target A in the plurality of reconstructed images, the image processing apparatus 100 acquires a corresponding point corresponding to the set point in the plurality of fluoroscopic images, The amount of displacement is calculated based on As described above, the position of the volume data relative to the position when the fluoroscopic image is captured by using the image processing apparatus 100 without setting a point indicating a specific part of the target A in both the fluoroscopic image and the reconstructed image. The amount of displacement is obtained. Thereby, the user can save time and effort for determining a point indicating a specific part of the target A, and the time required for positioning in the treatment system 10 can be shortened. In addition, the image processing apparatus 100 may improve the accuracy of the displacement amount by repeatedly generating the reconstructed image and calculating the displacement amount until the similarity between the fluoroscopic image and the reconstructed image satisfies the criterion. And the positioning accuracy can be improved.

In addition, the image processing apparatus 100 can improve the accuracy of the calculated displacement amount by determining whether or not to repeatedly calculate the displacement amount based on the similarity between the fluoroscopic image and the reconstructed image. it can.
Further, the image processing apparatus 100 determines the detection accuracy of the corresponding points by determining whether or not to repeatedly calculate the displacement amount based on the similarity between the set points and the partial images near the corresponding points. In addition, it is possible to reduce the amount of calculation in repeated determination, and to shorten the time required for positioning.

  Further, in the image processing apparatus 100, when the determination unit 106 determines the number of repetitions according to the ratio of a specific tissue included in the reconstructed image, for example, the ratio of the bone, positioning is required according to the region of the target A to be imaged. Time can be shortened.

(Second Embodiment)
FIG. 7 is a block diagram illustrating a configuration of an image processing apparatus 100A according to the second embodiment. As illustrated in FIG. 2, the image processing apparatus 100A includes a perspective image acquisition unit 111, a reconstructed image generation unit 102, a point acquisition unit 113, a point detection unit 104, a calculation unit 105, a determination unit 106, and an image data generation unit 107. Prepare. The image processing apparatus 100A of the second embodiment is used in place of the image processing apparatus 100 in the treatment system 10 shown in FIG. In the image processing apparatus 100A, the same components as those included in the image processing apparatus 100 of the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  The fluoroscopic image acquisition unit 111 acquires a pair of fluoroscopic images of a first fluoroscopic image and a second fluoroscopic image of the target A that are captured by the imaging device 400. Further, the fluoroscopic image acquisition unit 111 acquires the camera parameters of the first imaging unit 440 that has captured the first fluoroscopic image and the camera parameters of the second imaging unit 450 that has captured the second fluoroscopic image. The perspective image acquisition unit 111 includes an actual size per pixel in the image of the target A indicated by the reconstructed image generated by the reconstructed image generation unit 102, and an actual size per pixel in the image of the target A indicated by the acquired fluoroscopic image. The size of the fluoroscopic image is changed so as to match. Further, the perspective image acquisition unit 111 performs a smoothing filter process for smoothing each pixel value on the perspective image obtained by changing the size. The fluoroscopic image acquisition unit 111 supplies a pair of fluoroscopic images subjected to size change and smoothing filter processing to the point detection unit 104. The perspective image acquisition unit 111 supplies the acquired two camera parameters to the reconstructed image generation unit 102 and the point acquisition unit 103.

  The point acquisition unit 113 acquires, from the input / output device 200, correction information for instructing correction of the position of the corresponding point detected by the point detection unit 104 in addition to the operation performed by the point acquisition unit 103 in the first embodiment. I do. The instruction to correct the corresponding point is acquired, for example, when the user confirms the perspective image and the reconstructed image on which the corresponding point is superimposed, and the operation unit 220 of the input / output device 200 receives the user's operation input. The point acquisition unit 113 supplies the acquired correction information to the calculation unit 105. When the correction information is supplied, the calculation unit 105 changes the position of the corresponding point on the fluoroscopic image or the reconstructed image based on the correction information, and based on the position of the corresponding point after the change and the position of the set point The displacement amount is calculated.

  FIG. 8 is a flowchart illustrating a displacement amount calculation process performed by the image processing apparatus 100A. In the image processing apparatus 100, when the displacement amount calculation process is started, the fluoroscopic image acquisition unit 111 acquires a plurality of fluoroscopic images photographed from different directions and camera parameters corresponding to the fluoroscopic images from the imaging apparatus 400 (step). S101). The fluoroscopic image acquisition unit 111 performs a size change and a smoothing process on the acquired fluoroscopic images so that the actual dimensions per pixel coincide with the reconstructed image (step S201).

The reconstructed image generating unit 102 generates a reconstructed image corresponding to each fluoroscopic image based on the volume data of the target A and the camera parameters acquired by the fluoroscopic image acquiring unit 101 (step S102).
The point acquisition unit 113 acquires a point group including at least three set points designated by the user (step S103).

The point detection unit 104 detects a point corresponding to each set point in the fluoroscopic image or the reconstructed image. The point detection unit 104 sets the detected point as a corresponding point (step S104).
The point acquisition unit 113 acquires correction information for the position of the corresponding point detected by the point detection unit 104 (step S205).

The calculation unit 105 corrects the position of the corresponding point detected by the point detection unit 104 based on the correction information (step S206).
Based on the three-dimensional coordinates of each set point and the corrected three-dimensional coordinates of each corresponding point, the calculation unit 105 calculates a displacement amount of the position of the volume data with respect to the position when the fluoroscopic image is captured (step S207). .

The reconstructed image generation unit 102 generates a reconstructed image corresponding to each fluoroscopic image based on the volume data moved using the displacement amount and the camera parameter (step S106).
The determination unit 106 calculates the similarity between the reconstructed image generated in step S105 and the fluoroscopic image (step S107). The determination unit 106 determines whether or not the calculated similarity is greater than or equal to a threshold value (step S108).

When the degree of similarity is not equal to or greater than the threshold (step S108: NO), the determination unit 106 returns the process to step S104, and repeatedly performs the processes after step S104.
When the similarity is equal to or greater than the threshold (step S108: YES), the determination unit 106 outputs positioning information including the displacement amount to the bed apparatus 600 (step S109), and ends the displacement amount calculation process.

  When the image processing apparatus 100A in the second embodiment acquires a set point indicating a specific part of the target A in a plurality of fluoroscopic images, the image processing apparatus 100A acquires corresponding points corresponding to the set point in the plurality of reconstructed images, and the fluoroscopic image The image data including the image in which the set point is superimposed on and the image in which the corresponding point is superimposed on the reconstructed image is output to the input / output device 200. When the user confirms the image data displayed on the input / output device 200 and there is a shift in the position of the corresponding point on the reconstructed image, the input / output device 200 provides correction information that instructs the user to correct the position of the corresponding point. Enter through. The image processing apparatus 100A corrects the position of the detected corresponding point based on the correction information, and calculates the amount of displacement based on the corresponding point whose position has been corrected and the set point. Thus, by providing the user with the opportunity to confirm and correct the detected corresponding points, it is possible to improve the accuracy of the corresponding points when there are many similar partial images in the reconstructed image. Thereby, the image processing apparatus 100A can improve the accuracy of the calculated displacement amount, and can improve the positioning accuracy. In addition, even when the image processing apparatus 100A acquires a set point indicating a specific part of the target A in the reconstructed image, the image processing apparatus 100A can improve the accuracy of the displacement amount and improve the positioning accuracy. it can.

  Further, the image processing apparatus 100A performs image conversion on the fluoroscopic image to match the actual size per pixel in the fluoroscopic image with the actual size per pixel in the reconstructed image. Accordingly, the similarity between the fluoroscopic image and the reconstructed image when the position of the target A when the fluoroscopic image is captured and the position of the volume data of the target A when the reconstructed image is generated is increased. Thus, it is possible to improve the accuracy of the determination as to whether or not to repeat. As a result, the accuracy of the amount of displacement calculated by the image processing apparatus 100A can be improved.

  Further, the image processing apparatus 100A performs smoothing on the fluoroscopic image obtained by changing the size. When the resolution of the reconstructed image is lower than the resolution of the fluoroscopic image, a plurality of pixels of the fluoroscopic image correspond to one pixel of the reconstructed image. In this case, one pixel value of the reconstructed image is compared with a plurality of pixel values of the fluoroscopic image, and unnecessary variation occurs in calculating the similarity between the reconstructed image and the fluoroscopic image. By smoothing a fluoroscopic image obtained by changing the size, the above-described variation can be suppressed, and the accuracy of determination using similarity can be improved.

In each of the above embodiments, the case where the set point is determined in either the fluoroscopic image or the reconstructed image has been described. However, a plurality of set points may be determined in both the fluoroscopic image and the reconstructed image. At that time, the point detection unit 104 detects corresponding corresponding points in the perspective image or the reconstructed image for each set point.
Further, when the number of set points acquired by the point acquisition unit is four or more, the image processing apparatus uses a robust estimation method RANSAC method or a minimum median method (LMedS) to capture a fluoroscopic image. The displacement amount may be calculated by estimating the posture of the target A and the posture of the volume data of the target A when the reconstructed image is generated.

  According to at least one embodiment described above, a point detection unit that detects corresponding points corresponding to set points determined by the user, the position of the target A when the fluoroscopic image is captured, and the position of the volume data of the target A A calculation unit that calculates the displacement amount based on the set point and the corresponding point, and determines whether or not the displacement amount satisfies a predetermined condition. If not, the corresponding point is detected and the displacement amount is calculated. By having the determination unit that repeatedly performs the above, it is possible to improve the positioning accuracy and reduce the time required for positioning.

  The image processing apparatus described in each embodiment can also be realized by using, for example, a general-purpose computer apparatus as basic hardware. That is, the fluoroscopic image acquisition unit, the reconstructed image generation unit, the point acquisition unit, the point detection unit, the calculation unit, the determination unit, and the image data generation unit are realized by causing a processor mounted on the computer device to execute a program. can do. At this time, the image processing apparatus may be realized by installing the above program in a computer device in advance, or may be stored in a storage medium such as a CD-ROM or distributed through the network. Thus, this program may be realized by appropriately installing it in a computer device.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 10 ... Treatment system, 100, 100A ... Image processing apparatus, 101, 111 ... Perspective image acquisition part, 102 ... Reconstructed image generation part, 103, 113 ... Point acquisition part, 104 ... Point detection part, 105 ... Calculation part, 106 Determining unit 107 Image data generating unit 200 Input / output device 210 Display unit 220 Operation unit 300 Planning unit 310 Database unit 320 Display unit 330 Operation unit 340 Control , 400 ... imaging device, 410 ... control unit, 420 ... first radiation source unit, 430 ... second radiation source unit, 440 ... first imaging unit, 450 ... second imaging unit, 500 ... treatment device, 510 ... control Part, 520 ... radiation source part, 600 ... bed apparatus

Claims (11)

An image acquisition unit that acquires a plurality of first fluoroscopic images obtained by capturing an object from at least two different directions;
A reconstruction unit which reconstructs a different multiple second perspective image derived from at least two directions when the photographs the object, previously acquired the target volume data,
An acquisition unit that acquires a set point that put the first fluoroscopic image or the second fluoroscopic image,
When the set point is a point in the first perspective image, a corresponding point corresponding to the set point is detected in the second perspective image, and the set point is a point in the second perspective image Detecting a corresponding point corresponding to the set point in the first fluoroscopic image;
Calculation for calculating a displacement amount of the position of the target when the second perspective image is generated with respect to the position of the target when the first perspective image is captured based on the set point and the corresponding point. And
A determination unit for determining whether or not the displacement amount satisfies a predetermined condition;
With
When the displacement amount does not satisfy a predetermined condition, the reconstruction unit reconstructs the plurality of second fluoroscopic images from the volume data moved based on the displacement amount, and the detection unit detecting the corresponding points in the second fluoroscopic image reconstruction unit is reconstructed again with the first perspective image, wherein the calculating unit image processing for leaving calculate the displacement amount based on the corresponding points newly detected apparatus. The determination unit
The similarity between the second fluoroscopic image reconstructed from the volume data moved based on the displacement amount and the first fluoroscopic image is calculated, and the calculated similarity is equal to or greater than a predetermined threshold. The image processing apparatus according to claim 1, wherein it is determined whether or not the displacement amount satisfies the condition. The determination unit
Calculating the similarity of the partial image including the set point and the corresponding point in the second perspective image and the first perspective image reconstructed from the volume data moved based on the displacement amount; The image processing apparatus according to claim 1, wherein it is determined whether or not the displacement amount satisfies the condition by determining whether or not the calculated similarity is equal to or greater than a predetermined threshold value. The determination unit
The image processing apparatus according to claim 1, wherein when the calculation unit calculates the displacement amount a predetermined number of times, the displacement amount is determined to satisfy the condition. The determination unit
The image processing apparatus according to claim 4, wherein the predetermined number of times is determined based on a contrast of at least one of the first perspective image or the second perspective image. The image acquisition unit
The image processing apparatus according to any one of claims 1 to 5, wherein a smoothing process is performed on the first fluoroscopic image. The image acquisition unit
7. The size of the first perspective image is changed so that the actual size per pixel in the first perspective image and the actual size per pixel in the second perspective image coincide with each other. The image processing apparatus according to any one of claims. 8. The generation unit according to claim 1, further comprising: a generation unit configured to generate image data in which the set point and the corresponding point are superimposed on the first perspective image and the second perspective image. Image processing apparatus. An imaging device for capturing a fluoroscopic image of the object;
The image processing apparatus according to any one of claims 1 to 8,
A treatment system comprising: an input / output device that displays the first fluoroscopic image and the second fluoroscopic image and receives a user input operation. A bed apparatus having a movable table for carrying the object;
A treatment device for irradiating the subject with a treatment line,
The treatment according to claim 9, wherein the bed apparatus moves the movable table based on a displacement amount calculated by the image processing apparatus so that the treatment line is irradiated to a site where the treatment of the target is performed. system. An image processing method performed by an image processing apparatus,
An image acquisition step in which the image acquisition unit acquires a plurality of first fluoroscopic images obtained by capturing an object from at least two different directions;
Reconfiguration unit, a first reconstruction step of reconstructing different plurality of second fluoroscopic images obtained from at least two directions when the photographs the object from previously acquired the target volume data to one another ,
Point acquisition unit, an acquisition step that acquires the set point that put the first fluoroscopic image or the second fluoroscopic image,
When the set point is a point in the first fluoroscopic image, the detection unit detects a corresponding point corresponding to the set point in the second fluoroscopic image, and the set point is in the second fluoroscopic image. If it is a point, the detecting unit detects a corresponding point corresponding to the set point in the first fluoroscopic image;
Based on the set point and the corresponding point , the calculation unit generates a displacement amount of the target position when the second perspective image is generated with respect to the target position when the first perspective image is captured. A calculating step for calculating
A determination unit for determining whether the displacement amount satisfies a predetermined condition; and
If the displacement amount does not satisfy the predetermined condition, the reconstruction unit, a second reconstruction step of reconstructing a plurality of the second perspective image from the volume data is moved based on the displacement amount When,
A redetection step in which the detection unit detects corresponding points in the second fluoroscopic image reconstructed in the second reconstructing step and the first fluoroscopic image;
The calculating unit, an image processing method and a re-calculating a displacement amount based on the newly detected corresponding points in the redetection step.

Images