JP2017127623A - Image processor, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate deformation conformed as an entire image.SOLUTION: An image processor acquires a first image and a second image obtained by photographing a subject under different conditions, acquires an area of interest in the first image, acquires a first deformation displacement field between the first image and the second image, generates an approximated displacement field by using an approximation conversion model less in freedom than the first deformation displacement field to approximate the first deformation displacement field, generates correspondence information between the first image and an image where the first image is displaced on the basis of the approximated displacement field with respect to the area of interest, and acquires a second deformation displacement field between the first image and the second image by using the correspondence information.SELECTED DRAWING: Figure 1

Description

  The present invention processes three-dimensional images captured by various imaging apparatuses (modalities) such as a nuclear magnetic resonance imaging apparatus (MRI), an X-ray computed tomography apparatus (X-ray CT), and an ultrasonic diagnostic imaging apparatus (US). The present invention relates to an image processing apparatus, an image processing method, and a program.

  In image diagnosis using a three-dimensional image (a three-dimensional tomographic image representing information inside a subject), a doctor compares images taken with a plurality of imaging devices (modalities), different positions, times, imaging parameters, and the like. Make a diagnosis. However, since the shape of the subject is different between images, it is difficult to identify and compare the lesioned part. Therefore, attempts have been made to perform deformation position alignment (deformation estimation) between a plurality of images. Thus, it is possible to generate a deformed image in which one image is deformed and the position and shape of the subject depicted in the image are substantially matched with the other image. It is also possible to calculate and present the position of the corresponding point on the other image of the point of interest on one image. As a result, the doctor can easily identify and compare a lesioned portion between a plurality of images. In fields other than medical treatment, the same operation may be performed for the purpose of inspecting the internal state of an object using a three-dimensional image.

  At this time, the image includes various organs and body tissues, and the hardness of the tissues varies depending on the type. For example, since the bone is very hard, even if the posture and shape of the subject change between images to be compared, the shape hardly deforms. Also, some lesions such as tumors are harder than the surrounding tissue depending on the type, and their shape is difficult to deform. Therefore, if such a hard tissue is handled in the same way as the surrounding soft tissue and deformation positioning (deformation estimation) is performed, it is estimated that the hard region that is not actually deformed is deformed, resulting in an incorrect position. In some cases, a matching result may be obtained.

  As a solution to this problem, in Patent Document 1, in the displacement field obtained by non-rigid body alignment, the region of interest is deformed by bringing only the displacement field of the hard region of interest that should be a rigid body close to rigid body transformation. A technique for avoiding such an erroneous estimation has been proposed. More specifically, in Patent Document 1, only the deformation of the region of interest is approximated to rigid transformation, a displacement field of the approximated rigid transformation is generated for the region of interest, and the original deformation alignment of the non-interest region is generated. Techniques for generating displacement fields and generating displacement fields that are spatially coupled are described. Here, the displacement field is a field that holds the transformation (displacement) between each position between two images. As described above, Patent Document 1 discloses a technique for performing deformation alignment (deformation estimation) between images using a conversion model having different properties depending on regions in the image. Here, the conversion model represents a model for converting the coordinates of each position on the image in alignment.

JP 2013-141603 A

K. Rohr, H .; S. Stiehl, R.M. Sprangel, T .; M.M. Buzz, J. et al. Weese, and M.M. H. Kuhn, “Landmark-Based Elastic Registration Using Applicating Thin-Plate Springs”, IEEE Transactions on Medical Imaging, Vol. 20, no. 6 June2001.

  However, in the technique of Patent Document 1, coordinate transformation is represented by a transformation model having different properties between a region of interest and a non-region of interest, and the entire deformation position is aligned by interpolating and joining them. For this reason, there is a problem that a consistent deformation may not be obtained particularly at the joint.

  The present invention has been made in view of the above problems, and an object of the present invention is to estimate a consistent deformation of the entire image.

  As a means for achieving the above object, an image processing apparatus of the present invention comprises the following arrangement. That is, the image processing apparatus includes a data acquisition unit that acquires a first image and a second image obtained by imaging a subject under different conditions, and an interest that acquires a region of interest in the first image. Area acquisition means, first deformation acquisition means for acquiring a first deformation displacement field between the first image and the second image, and the first deformation displacement field as the first deformation. Deformation approximating means for generating an approximate displacement field by approximation using an approximate transformation model having a lower degree of freedom than the displacement field, and for the region of interest, the first image and the first image are based on the approximate displacement field. A correspondence information generating means for generating correspondence information between the first and second images and a second deformation displacement field between the first image and the second image using the correspondence information. And second deformation acquisition means.

  According to the present invention, a region of interest and a non-region of interest can be described seamlessly with a single conversion model, so that a consistent deformation can be estimated for the entire image.

1 is a diagram illustrating a configuration of an image processing system according to a first embodiment. 5 is a flowchart showing an overall processing procedure in the first embodiment. The figure which shows the 1st image and 2nd image regarding the same subject. The figure explaining a mode that a 1st deformation | transformation is applied with respect to a 1st image and a representative point. The figure explaining a mode that the 1st deformation | transformation of a region of interest is approximated by rigid body transformation. The figure explaining a mode that an approximate virtual corresponding point is produced | generated on a region of interest. The figure explaining a mode that a 2nd deformation | transformation is applied with respect to a 1st image. The figure which shows the structure of the image processing apparatus in 2nd Embodiment. The flowchart which shows the whole process sequence in 2nd Embodiment. The figure which shows the structure of the image processing apparatus in 3rd Embodiment. The flowchart which shows the whole process sequence in 3rd Embodiment. The figure which shows an intermediate | middle representative point. The figure which shows the structure of the image processing apparatus in 4th Embodiment. The flowchart which shows the whole process sequence in 4th Embodiment. The figure which shows the structure of the image processing apparatus in 5th Embodiment. The flowchart which shows the whole process sequence in 5th Embodiment. The flowchart which shows the detailed procedure of the process which acquires the 1st deformation | transformation in 5th Embodiment.

  Hereinafter, preferred embodiments of an image processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings. However, the scope of the invention is not limited to the illustrated examples.

<First Embodiment>
The image processing apparatus according to the present embodiment performs deformation position alignment between the first image and the second image that are three-dimensional images. At this time, when a region that should be hard, such as a hard lesion or bone, is given as a region of interest on the first image, the image processing apparatus maintains consistency as the entire image while maintaining the region of interest as a rigid body. Perform the deformed alignment. In addition, the image processing apparatus is not limited to a rigid body, and can perform deformation alignment by applying deformation having a smaller degree of freedom than the whole to the region of interest. Here, the deformation alignment means that the deformation to be applied to the first image is estimated so that the shape of the specimen in the first image matches the shape of the specimen in the second image. Further, it means that a deformed image obtained by deforming the first image is generated.

  Specifically, the image processing apparatus according to the present embodiment first performs first deformation alignment between two images based on a predetermined evaluation function for evaluating the appropriateness of alignment. Next, the image processing apparatus approximates the local deformation of the region of interest in the first image obtained by the first deformation alignment by a transformation with a small degree of freedom such as a rigid body transformation. Then, the image processing apparatus obtains a point (displacement point) obtained by displacing the representative point from the first image to the second image using approximate conversion for each of the plurality of representative points on the region of interest. Corresponding corresponding points (a pair of corresponding points) are generated virtually between two images composed of representative points and displacement points. Finally, the image processing apparatus adds the generated plurality of virtual corresponding points to the alignment evaluation function as the alignment constraint between the two images, and again the second deformation position between the two images. Align. By setting hard regions such as hard lesions and bones as regions of interest (regions where deformation is to be suppressed), the shape of the region of interest is maintained while applying appropriate deformation based on the evaluation function to the entire image. Can be estimated. That is, it is possible to estimate a deformation that is consistent with the entire image.

  Hereinafter, the configuration and processing of this embodiment will be described with reference to FIGS. 1 to 7. FIG. 1 is a diagram illustrating a configuration of an image processing apparatus 100 according to the present embodiment. As shown in the figure, the image processing apparatus 100 is connected to a data server 110 and a display unit 120. Note that the image processing apparatus 100 may include a data server 110 and a display unit 120.

  The first and second images held by the data server 110 are three-dimensional tomographic images (volume data) obtained by imaging the subject in advance under different conditions (different modalities, imaging modes, date / time, posture, etc.). . The modalities for imaging 3D tomographic images are MRI (Magnetic Resonance Imaging) apparatus, X-ray CT (Computed Tomography) apparatus, 3D ultrasound imaging apparatus, photoacoustic tomography apparatus, PET (Positron Emission Tomography) / SPECT (Single Photon Emission Computed Tomography (OCT), OCT (Optical Coherence Tomography) apparatus, etc. may be used. The first and second images may be captured at the same time in different modalities and different imaging modes, for example. Moreover, the image which image | photographed the same patient for the follow-up observation with the same modality and the same posture on the different date may be sufficient. The first and second images are input to the image processing apparatus 100 via the data acquisition unit 1010.

  The display unit 120 is a liquid crystal monitor or the like, and displays various types of information such as a display image generated by the image processing apparatus 100. The display unit 120 is also provided with a GUI (Graphical User Interface) for obtaining an instruction from the user. This GUI also functions as an operation unit in the following description.

  The image processing apparatus 100 includes a data acquisition unit 1010, a region of interest acquisition unit 1020, a first deformation acquisition unit 1030, a deformation approximation unit 1040, a correspondence information generation unit 1050, a second deformation acquisition unit 1060, and a display control unit 1070. Is done.

  The data acquisition unit 1010 acquires the first and second images input to the image processing apparatus 100 from the data server 110. The region-of-interest acquisition unit 1020 acquires information on a region (hereinafter referred to as a region of interest) that is desired to suppress deformation on the first image. The first deformation acquisition unit 1030 performs a first deformation position alignment between the first image and the second image, and acquires a first deformation displacement field for deforming the first image. The deformation approximating unit 1040 generates an approximate displacement field by approximating the first deformation displacement field in the region of interest using an approximate transformation model having a smaller degree of freedom than the displacement field. The correspondence information generation unit 1050 generates information on virtual corresponding points (approximate virtual corresponding points) on the region of interest based on the approximate displacement field as virtual correspondence information (virtual correspondence information) between images. The second deformation acquisition unit 1060 performs second processing between the first image and the second image based on an evaluation function obtained by adding a constraint condition based on virtual correspondence information to an evaluation function for evaluating the degree of coincidence between images. To obtain a second deformation displacement field for deforming the first image. The display control unit 1070 performs control for causing the display unit 120 to display information such as the first image, the second image, and the tomographic image of the deformed image based on the alignment result.

  FIG. 2 is a flowchart illustrating an overall processing procedure performed by the image processing apparatus 100.

(S2000) (Data acquisition)
In step S2000, the data acquisition unit 1010 acquires the first image and the second image from the data server 110. Then, the data acquisition unit 1010 outputs the first and second images to the region of interest acquisition unit 1020, the first deformation acquisition unit 1030, the second deformation acquisition unit 1060, and the display control unit 1070.

  FIG. 3 is a diagram showing a first image 3000 and a second image 3030 relating to the same subject. In the example of FIG. 3, the first image and the second image are captured images of different types of modalities, and the subject is a breast in a human body. Each image represents, for example, three-dimensional volume data such as CT, MRI, and 3D ultrasound image. However, for the convenience of the two-dimensional paper surface, a cross-sectional image cut out parallel to the xy plane from the volume data. Shown in the format.

  In the first image 3000, a subject area 3010 and a lesion area 3020 are shown. Similarly, the second image 3030 shows a subject area 3040 and a lesion area 3050. The lesion area 3020 and the lesion area 3050 are anatomically corresponding areas. In the example of FIG. 3, the regions 3010 and 3040 of the subject that are breasts are greatly deformed between the first image and the second image, but the lesion regions 3020 and 3050 are anatomically hard lesions. Therefore, it represents a situation where the deformation is not so much.

(S2010) (Obtain region of interest)
In step S2010, the region-of-interest acquisition unit 1020 acquires information on the region of interest (information on a lesion or the like whose deformation is to be suppressed) on the image acquired from the data acquisition unit 1010. Then, the region of interest acquisition unit 1020 outputs the acquired information of the region of interest to the correspondence information generation unit 1050.

  Here, the region of interest may be a region that is difficult to deform between captured images, such as a region such as a hard lesion, a bone region, a chest wall, and a great pectoral muscle. Furthermore, when the first image and the second image are time-lapse images captured at different dates and times, the region of interest may be a lesion region that is a subject of follow-up observation. This is because, for the purpose of determining the therapeutic effect, etc., if it is desired to observe the progress of the lesion area after aligning the positions of the images over time, the size of the lesion due to changes over time, etc. This is because the change of cannot be observed. Therefore, even in the case of a lesion that is not actually a rigid body, it is easy to observe changes in size and the like by treating the lesion area as a rigid body (a region in which the shape is not changed) between images.

In the present embodiment, for example, the region-of-interest acquisition unit 1020 acquires a lesion region 3020 whose outline is clearly depicted on the first image 3000 in FIG. 3 as the region of interest. In the present embodiment, the region of interest acquisition unit 1020 acquires the information of the region of interest when the user manually operates the operation unit (not shown) to manually extract the contour shape of the lesion on the image. The acquired lesion region 3020 as the region of interest is represented by, for example, a mask image I _ROI that is binarized inside and outside the region of interest. The region of interest may be expressed in any format as long as the region of interest can be distinguished. Moreover, the region of interest does not necessarily have to be one place, and a plurality of regions of interest may be acquired.

  Note that the region-of-interest acquisition unit 1020 may perform the process of acquiring the region of interest by a commonly used image analysis technique. For example, the region-of-interest acquisition unit 1020 may extract a region by a region expansion method after the user gives coordinates in a lesion region that suppresses deformation as a seed point. The extraction method of the lesion area is not limited to this, and a known area division method such as SNAKES or LevelSet method may be used. Further, after the region-of-interest acquisition unit 1020 performs general image processing for detecting a plurality of lesion candidate regions on the first or second image, the user operates an operation unit (not shown) on the region for controlling deformation. The region of interest acquisition unit 1020 may acquire the region of interest by designating (selecting).

(S2020) (Acquisition of first deformation)
In step S2020, the first deformation acquisition unit 1030 executes the first deformation alignment between the first image and the second image acquired from the data acquisition unit 1010, and obtains the first deformation displacement field. get. Then, the first deformation acquisition unit 1030 outputs the acquired first deformation displacement field to the correspondence information generation unit 1050. Furthermore, the first deformation acquisition unit 1030 generates a first deformation image obtained by deforming the first image based on the first deformation displacement field, and outputs the first deformation image to the display control unit 1070.

  At this time, the first deformation acquisition unit 1030 defines an evaluation function (cost function) that evaluates the appropriateness of alignment by the deformation Φ, and estimates the deformation Φ that minimizes the evaluation function. The deformation Φ estimated at this time is referred to as a first deformation. The deformation Φ is expressed by a displacement field (deformation displacement field). Here, the form of the displacement field may be a displacement vector field composed of displacement vectors for displacing each position on the image, or an arbitrary position on the image may be represented even if not represented by a spatial field. It may be expressed in the form of conversion parameters of a predetermined conversion model that can be displaced.

  Here, the evaluation function defined in this step is referred to as a first evaluation function E1 (Φ). In this embodiment, the degree of coincidence between images due to deformation is applied as an index constituting the first evaluation function. For example, an error in the position of a feature point (corresponding point) corresponding to an anatomy between the first image and the second image is applied. In order to distinguish from virtual corresponding points described later, such corresponding points that actually correspond between images are referred to as actual corresponding points. In particular, as described above, actual corresponding points corresponding anatomically between images are referred to as landmark corresponding points. That is, the first deformation acquisition unit 1030 estimates a deformation that matches the positions of the landmark corresponding points as much as possible.

  In order to acquire corresponding points, the first deformation acquisition unit 1030 first performs known image feature point extraction processing, such as an interest operator, on the first image and the second image, respectively, for example. Extract anatomical feature points such as blood vessel bifurcation. Thereafter, the first deformation acquisition unit 1030 performs a process of generating a set of landmark corresponding points by associating the extracted feature points with one-to-one correspondence. This process is performed by setting a region of interest in the vicinity of a feature point for each feature point on each image and associating feature points that increase the image similarity of the region of interest between images.

  Note that the method for acquiring landmark corresponding points is not limited to the above method. For example, even if landmark corresponding points are acquired by corresponding points manually input by a user using an operation unit (not shown) with respect to the first image and the second image displayed on the display unit 120. Good. Further, the first evaluation function E1 (Φ) is not limited to the error of the actual corresponding point position as described above. For example, the image similarity between the deformed first image and the second image The degree may be used as an index. Further, as the evaluation function E1 (Φ), either one of the above-described similarity of the whole image or the error of the actual corresponding point position may be used together.

  The first deformation acquisition unit 1030 optimizes the conversion parameter of a predetermined conversion model expressing the deformation Φ so as to minimize the first evaluation function E1. As a result, the optimum deformation that is desirable as the alignment of the entire subject in the image is estimated. Here, a conversion parameter expressed by a predetermined conversion model is defined as p. In the present embodiment, FFD (Free Form Deformation) in which the deformation basis function is represented by a B-spline function is applied as the predetermined transformation model. The conversion parameter p in the FFD is expressed by a control amount of control points regularly arranged in the image. Note that the conversion parameter p estimated in this step is referred to as a first conversion parameter p1. Note that the conversion model to be applied is not limited to FFD. For example, it may be any known transformation model that can express deformation, such as a radial basis function such as TPS (Thin Plate Spline) or LDDMM (Large Deformation Diffeomorphic Metric Mapping).

  Thus, the first deformation acquisition unit 1030 generates the conversion parameter p1 as the first deformation displacement field. Then, the first deformation acquisition unit 1030 generates a first deformed image obtained by deforming the first image so that the shape matches the second image based on the generated first deformation displacement field. Further, if necessary, a deformed image (reverse deformed image) obtained by deforming the second image so as to match the shape of the first image is generated based on the first deformed displacement field.

(S2030) (Approximate first deformation in region of interest by approximate transformation)
In step S2030, the correspondence information generating unit 1050 and the deformation approximating unit 1040 generate an approximate displacement field approximated by a conversion model having a lower degree of freedom than the first deformation displacement field in the region of interest. Then, the correspondence information generation unit 1050 outputs the generated approximate displacement field to the second deformation acquisition unit 1060.

  Hereinafter, a conversion model for generating an approximate displacement field (that is, a conversion model having a smaller degree of freedom than the first deformation displacement field) is referred to as an approximate conversion model. For example, a rigid body transformation model can be applied as the approximate transformation model. Since the first deformation displacement field is a result of the non-rigid body deformation alignment, the rigid body transformation model can be said to be a model having a smaller degree of freedom than the first deformation displacement field. More specific processing will be described below.

  First, the correspondence information generation unit 1050 is a virtual image between images including a representative point and a displacement point obtained by displacing the representative point using a first deformation displacement field for each of a plurality of representative points on the region of interest. Corresponding points (deformed virtual corresponding points) are generated. Then, the correspondence information generation unit 1050 outputs the generated deformation virtual corresponding point to the deformation approximation unit 1040.

  A specific process for the process of the correspondence information generation unit 1050 will be described. First, the correspondence information generation unit 1050 sets a representative point on the region of interest. Here, the region where the representative point is set may be only the inner region of the region of interest, or may be a region within a predetermined range surrounding the region of interest. For example, the region where the representative point is set can be set to a region within 5 mm from the outline of the region of interest. Further, the correspondence information generation unit 1050 may arrange the representative points at equal intervals (0.25 mm or the like) in the region, or may arrange them at random. For example, the correspondence information generation unit 1050 sets a plurality of representative points at equal intervals on the region of interest. Next, the correspondence information generation unit 1050 calculates the position of the displacement point corresponding to each representative point by displacing the position of each representative point using the first deformation displacement field. The calculated displacement point is referred to as a deformation representative point. In the present embodiment, a representative point and a corresponding deformation representative point are defined as virtual corresponding points between images by the first deformation, and hereinafter, these are referred to as deformation virtual corresponding points.

  FIG. 4 is a diagram illustrating a state in which a deformation virtual corresponding point is generated based on the first deformation displacement field. FIG. 4 shows a first deformed image 4000 obtained by deforming the first image 3000 based on the first deformed displacement field. The first deformed image 4000 shows a deformed subject area 4010 and a deformed region of interest 4020. The representative point 4030 represents a representative point set on the lesion area 3020 as the region of interest, and the deformed representative point 4040 represents a representative point obtained by deforming the representative point 4030 based on the first deformation field. In the example of FIG. 4, a state in which the region of interest 4020 is crushed in the same manner as the region 4010 of the subject and is estimated to be translated and rotated by the first deformation alignment is shown. . At this time, similarly to the region of interest 4020, the distribution of the deformation representative points 4040 is also calculated as a distribution in which the representative points 4030 are crushed as a whole and further translational movement and rotation occur.

  Next, the deformation approximating unit 1040 generates an approximate displacement field in which the displacement of the deformation virtual corresponding point is approximated by an approximate conversion model based on the generated deformation virtual corresponding point. More specifically, the deformation approximating unit 1040 calculates a parameter (approximate conversion parameter) q of the approximate conversion model Φ ′ that minimizes the error sum e shown in Expression (1). When a rigid transformation model is used as the approximate transformation model, six parameters of position and orientation that define a 3 × 4 rigid transformation matrix T representing the movement and rotation of the object correspond to the approximate transformation parameter q. That is, as in Expression (2), the deformation approximation unit 1040 calculates the sum e of norms of the difference between the product of the rigid body transformation matrix and the representative point and the deformation representative point, and the rigid body transformation matrix that minimizes the sum e. T is calculated under the constraints of rigid transformation.

Here, Φ ′ (x | q) represents coordinates obtained by displacing the coordinate x by the approximate conversion model Φ ′ represented by the parameter q. Further, (x _D1i , x _D2i ) represents a set of coordinates of the i-th deformed virtual corresponding point on the first image and the second image. That _is, the coordinates of _{x D1i} is i-th representative _{point, x D2i} represent the coordinates of the i-th deformation representative points. N _D represents the number of deformation virtual corresponding points. Since the matrix T can be calculated by a known method using singular value decomposition or the like, description of the calculation method is omitted.

  In this way, the deformation approximating unit 1040 acquires the approximate displacement field of the first deformation displacement field in the region of interest. Finally, the deformation approximating unit 1040 outputs the acquired approximate conversion parameter q to the correspondence information generating unit 1050 as approximate displacement field information.

  FIG. 5 is a diagram for explaining a state in which the first deformation in the lesion area 3020 as the region of interest is approximated by rigid transformation. In FIG. 5, the coordinate axis 5000 represents a coordinate axis parallel to the xy axis of the first image 3000 with the center of gravity of the representative point (representative point 4030 in FIG. 4) on the lesion area 3020 as the region of interest as the origin. ing. A coordinate axis 5010 represents a coordinate axis after the coordinate axis 5000 is converted using the rigid body transformation matrix T. However, for the sake of simplicity of explanation, the axis parallel to the z axis is omitted for the convenience of the plane of the paper. The plane formed by the xy axes of the coordinate axes 5010 is not necessarily parallel to the plane formed by the xy axes of the first modified image 4000, but is shown in a simplified manner. As shown in the figure, the coordinate axis 5010 translates / rotates by removing only deformation components that crush the region from the transformation from the representative point 4030 to the deformation representative point 4040 (first deformation displacement field) in FIG. It can be seen that it matches the conversion from which the components were extracted.

(S2040) (Generation of approximate virtual correspondence information based on approximate transformation)
In step S2040, the correspondence information generation unit 1050 generates a plurality of virtual correspondence points (virtual correspondence information) based on the approximate displacement field on the region of interest as virtual correspondence information between images. In the present embodiment, this is hereinafter referred to as an approximate virtual corresponding point. Then, the generated approximate virtual corresponding point is output to the second deformation acquisition unit 1060.

  Specific processing will be described below. First, the correspondence information generation unit 1050 displaces the respective positions of the plurality of representative points set on the region of interest using the approximate displacement field Φ ′, as in the processing of step S2030, and sets the representative points as the representative points. The position of the corresponding displacement point is calculated. The calculated displacement point is referred to as an approximate representative point. Then, the representative point and the corresponding approximate representative point are defined as virtual corresponding points between the images based on the approximate conversion model, and are hereinafter referred to as approximate virtual corresponding points.

  FIG. 6 is a diagram illustrating a state in which approximate virtual corresponding points are generated based on the approximate displacement field. The approximate representative point 6000 represents an approximate representative point obtained by converting the representative point 4030 based on the approximate deformation field. As shown in the figure, it can be seen that the distribution of the approximate representative points 6000 is a distribution in which only the translation and rotation indicated by the conversion from the coordinate axis 5000 to the coordinate axis 5010 in FIG.

(S2050) (Acquisition of second deformation based on approximate virtual corresponding point)
In step S2050, the second deformation acquisition unit 1060 uses the approximate virtual corresponding point (virtual correspondence information) acquired in step S2040 as a constraint condition, and performs a second change between the first image and the second image. Perform deformation alignment. Then, the second deformation acquisition unit 1060 outputs the generated second deformation displacement field to the data server 110 as a result of deformation estimation between the first image and the second image. The output to the data server 110 can be performed using an output unit (not shown). Further, the second deformation acquisition unit 1060 generates a second deformation image obtained by deforming the first image based on the second deformation displacement field, and outputs the second deformation image to the display control unit 1070.

  At this time, as in step S2020, the second deformation acquisition unit 1060 estimates a deformation Φ that minimizes an evaluation function that evaluates the appropriateness of alignment by the deformation Φ. The deformation Φ estimated at this time is referred to as a second deformation. The evaluation function defined in this step is referred to as a second evaluation function E2 (Φ). In the present embodiment, an evaluation function obtained by adding an error term of the corresponding point position of the approximate virtual corresponding point to the first evaluation function E1 (Φ) used in step S2020 is applied as the second evaluation function.

Here, if the error term of the corresponding point position of the approximate virtual corresponding point due to the deformation Φ is defined as R (Φ), the second evaluation function E2 (Φ) is expressed by the following equation.
Here, Φ (x | p2) represents coordinates obtained by displacing the coordinate x by the second deformation Φ represented by a predetermined conversion model with the parameter p2. Also, (x _R1i , x _R2i ) represents a set of coordinates of the i-th approximate virtual corresponding point on the first image and the second image. That is, _xR1i represents the coordinates of the i-th representative point, and _xR2i represents the coordinates of the i-th approximate representative point. N _R represents the total number of approximate virtual corresponding points. That is, R (Φ) is a term for matching the representative point after displacement by the second deformation Φ as much as possible to the corresponding approximate representative point.

  The second deformation acquisition unit 1060 optimizes the conversion parameter p2 of a predetermined conversion model expressing the deformation Φ so as to minimize the second evaluation function E2 (Φ). As a result, the optimum deformation that is desirable as the alignment of the entire subject in the image is estimated. In the present embodiment, the same conversion model as that used in step S2020 is applied as the predetermined conversion model. That is, when FFD is applied in step S2020, FFD is also applied in this step. Here, when the conversion model is FFD and the second evaluation function E2 (Φ) is expressed only by the error term of the actual corresponding point (landmark corresponding point) position and the error term of the approximate virtual corresponding point position think of. In this case, the relationship between the FFD parameter and the positional error between corresponding points can be expressed by a linear equation. Therefore, the second deformation acquisition unit 1060 estimates the transformation parameter p2 that minimizes the position error between corresponding points by a known linear optimization method. Thereby, the conversion parameter p2 can be calculated at high speed. Similarly, when the conversion model is TPS, the second deformation acquisition unit 1060 can calculate the parameter p2 by a known linear optimization method. When the similarity between images and a nonlinear regularization term are considered as the evaluation function E2 (Φ), or when LDDMM is used as the conversion model, the second deformation acquisition unit 1060 uses a known nonlinear optimization. The parameter p2 is estimated by At this time, the second deformation acquisition unit 1060 can use the parameter p1 calculated by the first deformation acquisition unit 1030 in step S2020 as an initial value of the parameter p2. According to this, the calculation of the parameter p2 can be performed with fewer iterative processes.

  As described above, the second deformation acquisition unit 1060 generates the conversion parameter p2 as the second deformation displacement field. Then, the second deformation acquisition unit 1060 generates a second deformed image obtained by deforming the first image so that the shape matches the second image based on the generated second deformation displacement field. Further, if necessary, the second deformation acquisition unit 1060 converts a deformed image (reverse deformed image) obtained by deforming the second image so as to match the shape of the first image into the second deformation displacement field. Generate based on

  As described above, in the present embodiment, based on the first evaluation function E1 (Φ), appropriate alignment as the whole subject is realized, and based on the error term R (Φ) of the approximate virtual corresponding point position. Thus, it is possible to estimate a deformation that keeps the region of interest in a shape (for example, a rigid body) represented by the approximate conversion model as much as possible. Furthermore, since the deformation of the entire image is expressed by a single transformation model (FFD), it is possible to estimate a deformation that is consistent as a whole.

  In addition, the second deformation obtained in this step has the same conditions as the first deformation for both the index in the evaluation function for evaluating the region other than the region of interest and the conversion model. Therefore, the deformation of the region other than the region of interest can be made similar to the first deformation. As a result, the approximate transformation component around the region of interest in the second modification is also similar to the first modification. Accordingly, even if the approximate transformation component of the region of interest of the first deformation is directly applied to the deformation of the region of interest due to the constraint of the approximate virtual corresponding point, the approximate transformation component substantially matches the surroundings of the region of interest. For this reason, it is possible to acquire an alignment result with no sense of incongruity.

  FIG. 7 is a diagram illustrating a state in which a second deformed image is generated by deforming the first image based on the second deformed displacement field. The second deformed image 7000 shows a region 7010 of the subject after deformation and a region of interest 7020 after deformation. As shown in the figure, the deformed subject area 7010 has substantially the same shape as the deformed subject area 4010 in FIG. On the other hand, the region of interest 7020 after the deformation does not cause a crushed deformation like the region of interest 4020 after the deformation in FIG. 4, and the translation is performed while the shape of the lesion region 3020 as the region of interest is kept rigid. It shows that only the movement and rotation result in the same transformation as the region of interest 4020. This is because, in the second alignment, the effect of the constraint condition that constrains the transformation of the region of interest to the approximate transformation model (here, a rigid body) by the error term of the approximate virtual corresponding point position composed of the representative point 4030 and the approximate representative point 6000. Indicates that is being demonstrated.

(S2060) (Display of deformation image)
In step S2060, the display control unit 1070 performs control to display the acquired deformed image and the cross-sectional image of the second image on the display unit 120 in accordance with an operation on the operation unit (not illustrated) by the user. At this time, the display control unit 1070 uses the same cross section of the first modified image based on the first modified displacement field, the second modified image based on the second modified displacement field, and the second image as modified images. Can be configured to be displayed side by side. Further, the display control unit 1070 may perform control so that the first deformed image and the second deformed image are switched and displayed within one display screen area in accordance with a user operation. Note that the display control unit 1070 may not generate or display the first modified image. The above is the description of the processing of the image processing apparatus 100 in the present embodiment.

  According to the present embodiment, the image processing apparatus 100 uses a single transformation model to appropriately match the entire image and an index for keeping the deformation of the region of interest with a smaller degree of freedom (for example, a rigid body). Based on the above, the deformation is estimated. As a result, the image processing apparatus 100 appropriately transforms the region other than the region of interest in the subject and performs transformation (for example, rigid transformation) with a lower degree of freedom on the region of interest, and then performs the entire image. As a result, a consistent deformation can be estimated.

(Modification 1-1) (Approximate transformation model may not be rigid transformation)
In the first embodiment, rigid transformation is applied as the approximate transformation model that approximates the deformation of the region of interest in step S2030. However, the approximate transformation model has any degree of freedom that is smaller than the first transformation. It may be a model. For example, the deformation approximation unit 1040 may approximate the first deformation by affine transformation.

  Specifically, first, the correspondence information generation unit 1050 generates a deformed virtual corresponding point as in the first embodiment. Next, the deformation approximating unit 1040 generates an affine displacement field that approximates the displacement of the generated virtual deformation corresponding point by affine transformation. More specifically, the deformation approximating unit 1040 calculates a 3 × 4 matrix T that minimizes the error sum e shown in Expression (2) by constraining the affine transformation. In this equation, it is generally known that the relationship between the affine transformation matrix T and the norm can be expressed by a linear equation. Therefore, the matrix T can be calculated by a known method such as LU decomposition or QR decomposition. Since the description of this method is a known technique, the description is omitted. Thereby, for example, when the region of interest is not completely rigid, but the complex deformation is not performed, the deformation of the region of interest can be estimated in a form close to the actual one.

In addition, the deformation approximating unit 1040 may approximate the deformation of the region of interest by affine transformation to which a constraint condition for volume preservation is added instead of simple affine transformation. At this time, the constraint condition for volume preservation is expressed by the following equations, where s _x , s _y , and s _z are the scale factor components in the x, y, and z directions in the affine transformation matrix T, respectively.
That is, the deformation approximating unit 1040 solves the problem of calculating the parameter q in step S2030, and the parameter of the affine transformation matrix T that minimizes the evaluation function e of the equation (2) under the condition that the equation (5) is satisfied. Can be regarded as a problem to estimate. The matrix T can be calculated using a known nonlinear optimization method. Thereby, for example, when the region of interest does not undergo complex deformation and compression / expansion does not occur during the deformation process, the deformation of the region of interest can be estimated in a form close to reality.

(Modification 1-2) (The constraint condition may be to constrain the approximate virtual corresponding point to a planar shape)
In the first embodiment, in step S2050, a constraint condition in which the positional relationship between corresponding points constituting the approximate virtual corresponding point matches is applied to the second evaluation function E2 (Φ). However, the constraint condition added to the second evaluation function E2 (Φ) is not necessarily limited to this.

  For example, when the region of interest is a planar region such as the chest wall or pectoral muscle surface, the approximate virtual corresponding point is constrained on the surface of the region of interest (the position may not match if on the surface) The condition may be applied to the second evaluation function E2 (Φ). Thereby, an effect can be expected in the following cases.

  For example, it is known that when an external force is applied to the breast in a deformation that compresses the breast, the mammary gland and fat tissue in the breast slide and deform on the pectoralis major muscle surface behind it. . At this time, since the pectoralis muscle surface hardly deforms, it is assumed that the pectoral muscle surface is used as a region of interest and that the pectoral muscle surface is deformed so as to be a rigid body. At this time, in step S2050, when a constraint condition is applied to the second evaluation function E2 (Φ) so that the positions of the approximate virtual corresponding points themselves coincide, the tissue in the vicinity of the pectoralis major muscle is directly on the pectoral muscle surface. Because it is constrained by the position, it does not deform so as to slide on the pectoral muscle surface. Therefore, in this modification, a condition that the approximate virtual corresponding point is constrained only on the surface of the region of interest (the pectoralis major muscle surface) is added to the second evaluation function E2 (Φ). This allows a case where the approximate virtual corresponding point slides on the surface of the region of interest. That is, the representative point set on the region of interest including the outer surface of the region of interest is allowed to move along the surface of the region of interest even after being displaced by the approximate displacement field. As a result, it becomes possible to allow the tissue in the vicinity of the pectoralis major muscle to slide on the pectoral pectoral surface.

A specific method will be described below. First, it is assumed that the representative points constituting the approximate virtual corresponding points are evenly arranged on the region of interest which is a surface region. At this time, in step S2050, the constraint condition for constraining the approximate virtual corresponding point on the surface of the region of interest is expressed by the following expression as a function R (Φ) that constitutes the evaluation function E2 (Φ).
Here, in the expression (6), the same symbols as those in the expression (4) represent the same meaning as in the expression (4). Cov (x) represents a function that returns a covariance matrix related to a three-dimensional error at the position x. Hereinafter, covariance Cov (x) is abbreviated as Cov.

In the case of this modification, the covariance Cov is, for example, a covariance representing a Gaussian distribution having a narrow distribution in the normal direction in the vicinity of the i-th approximate representative point _xR2i and a very wide distribution in the direction of a plane orthogonal thereto. Can be a function that returns Here, the normal direction of the approximate representative point x _R2i vicinity, in approximate representative points are distributed in the planar, the plane spanned by the set of approximate representative points of the approximate representative point x _R2i near normal vector Means the direction. This normal vector can be calculated as a third principal component vector by _performing principal component analysis on the position of a set of approximate representative points in the vicinity of the approximate representative point _xR2i , which is a known technique, for example.

Note that the normal vector calculation method is not limited to this, and any known method may be employed. Here, in Equation (6), a three-dimensional vector representing an error between the representative point after displacement by the second deformation Φ and the corresponding approximate representative point is defined as e. According to equation (6), when the normal component in the input vector is large, the product of Cov ⁻¹ and the error vector e returns a large value, and when the other components are large, a very small value is returned. Therefore, according to the equation (6), R (Φ) is a component in the normal direction of the approximate representative point of the error e between the representative point after displacement by the second deformation Φ and the corresponding approximate representative point. Can only be penalized. That is, R (Φ) functions as a term for preventing the displaced representative point from deviating only from the corresponding approximate representative point in the normal direction.

  As a calculation method of the conversion parameter p2 that minimizes the evaluation function E2 (Φ) defined in this way, it can be calculated by a known nonlinear optimization method as in step S2050. It can also be calculated by a linear optimization method based on the above-described covariance Cov described in Non-Patent Document 1.

(Modification 1-3)
In the first embodiment, virtual corresponding points are applied as virtual correspondence information between images used in steps S2040 and S2050. However, the virtual correspondence information to be used is not necessarily a corresponding point.

For example, as the virtual correspondence information, virtual correspondence information between images of information regarding the shape of the region of interest may be used. Specifically, in step S2040, the correspondence information generation unit 1050 displaces the region of interest itself using the approximate deformation field Φ ′, and acquires the corresponding region of interest after the displacement (approximate region of interest). More specifically, the correspondence information generation unit 1050 obtains a mask image I _{A_ROI} of the approximate region of interest by displacing the mask image I _ROI representing the region of interest using the approximate deformation field Φ ′. Then, the correspondence information generation unit 1050 generates the region of interest and the approximate region of interest as a virtual correspondence region between images based on the approximate conversion model. Since the information on the virtual corresponding region holds information on the contour shape of the region of interest, this is referred to as an approximate virtual corresponding shape as virtual correspondence information between images in this modification.

Then, the correspondence information generation unit 1050 outputs the generated approximate virtual correspondence shape to the second deformation acquisition unit 1060. In step S2050, the second deformation acquisition unit 1060 uses the approximate virtual corresponding shape acquired in step S2040 as a constraint condition, and performs the second deformation alignment between the first image and the second image. Execute. At this time, in the second evaluation function E2 (Φ) that is minimized in this step, in this modification, instead of the error term R (Φ) of the corresponding point position of the approximate virtual corresponding point in the first embodiment, The error term R ′ (Φ) of the approximate virtual corresponding shape is applied. The error term R ′ (Φ) of the approximate virtual corresponding shape is expressed by the following equation.
Here, the function S (I ₁ , I ₂ ) represents an image similarity function between the image I ₁ and the image I ₂ . As a scale for calculating the image similarity, a commonly used method such as a commonly used SSD (Sum of Squared Difference), mutual information, cross-correlation coefficient, or the like is used. Thereby, the deformation Φ can be estimated so that the mask image I _{ROI of the} region of interest that is the approximate virtual corresponding shape matches the mask image I _{A_ROI} of the approximate region of interest as _much as possible. That means
A deformation that keeps the contour shape of the region of interest in the shape represented by the approximate conversion model can be estimated.

Further, the following information may be applied as the virtual correspondence information. First, the correspondence information generation unit 1050 generates a distance field from the contour position of the region of interest in the image from each of the mask image I _ROI of the region of interest and the mask image I _{A_ROI} of the approximate region of interest. Subsequently, the correspondence information generation unit 1050 generates a distance field image obtained by converting each distance value in each distance field into a luminance value and imaging it. _Here, the distance field image corresponding the distance field image corresponding to the _{I ROI I DIST,} the _{I A_ROI} defined as _{I A_DIST,} they, and virtual correspondence information between the images due to the approximate conversion model. In this case, by applying I _DIST and I _{A_DIST} instead of I _ROI and I _{A_ROI in} the equation (6), the luminance information can be made similar not only to the contour of the region of interest but also to the inside. Thereby, it is possible to estimate a deformation that keeps the shape of the region of interest I _ROI including the inside of the region of interest in the shape represented by the approximate conversion model.

(Modification 1-4)
In the first embodiment, the deformed virtual corresponding point generated in step S2030 and the approximate virtual corresponding point generated in step S2040 are generated from the same representative point on the first image. However, they do not have to be generated from the same representative point, and the deformed virtual corresponding point and the approximate virtual corresponding point may be generated from different representative points. For example, a deformed virtual corresponding point is generated from a representative point set on a region within a predetermined range (within 5 mm circumference) surrounding the region of interest, and an approximate virtual corresponding point is strictly set on the region of interest. You may take the method of producing | generating from a representative point. As a result, in the second modification, it is possible to extract an approximate transformation component that is more consistent with a region around the region of interest, and to give a constraint condition that strictly maintains the shape only for the region of interest.

(Modification 1-5) (E1 (Φ) may not be the same in the first deformation estimation and the second deformation estimation)
In the first embodiment, the second evaluation function E2 (Φ) in step S2050 uses the first evaluation function E1 (Φ) as shown in the equation (3). However, it is not always necessary to use the first evaluation function E1 (Φ) in step S2050. For example, in step S2050, an evaluation function that requires a higher calculation cost and performs high-precision evaluation may be used as E1 (Φ). For example, in the first deformation estimation, the landmark corresponding point position error can be used as E1 (Φ), and in the second deformation estimation, the image similarity of the entire image can be used as E1 (Φ). This is because the first deformation estimation is merely an intermediate process for obtaining an approximate deformation field, but the second deformation estimation is intended to obtain a registration result.

(Modification 1-6) (The conversion model in the second modification may not be the same as the first modification)
In the first embodiment, the same deformation model as the first deformation in step S2020 is used as the conversion model in the second deformation in step S2050. However, if the approximate transformation component around the region of interest has a result similar to the first transformation, the transformation model used in the second transformation does not necessarily need to match step S2020.

  For example, in the present embodiment, FFD is adopted as a conversion model in step S2020. However, even if FFD in which the control point interval is changed, which is one of the parameters constituting the FFD model, is adopted in step S2050. Good. Specifically, when the control points are arranged at intervals of 20 mm in step S2020, they may be arranged at intervals of 10 mm, which is half of the control points, in step S2050. As a result, the degree of freedom of deformation can be increased without changing the characteristic that smooth deformation can be obtained by the B-spline basis function. That is, more detailed deformation can be expressed. Further, in order to make the general deformation of the entire image similar between the first deformation and the second deformation, for example, a conversion model that increases the degree of freedom of deformation stepwise is applied in the second deformation. Specifically, it is expressed by “fine control points (10 mm intervals)” with high degree of freedom of deformation from FFD expressed by “rough control points (20 mm intervals)” with low degree of freedom of deformation similar to the first deformation. A multi-resolution FFD that is a known technique that is applied in the order of FFDs to be applied is applied as a conversion model.

  Thus, in the second deformation, the first deformation state is maintained as a rough deformation while achieving more detailed alignment than the first deformation, and thus the approximate transformation component around the region of interest. Is similar to the first variant. Also, if the approximate transformation components around the region of interest do not change significantly, a transformation model with different properties such as FFD and TPS, FFD and LDDMM, etc. can be applied as a combination of the first and second transformations. Good. Further, any conversion model or evaluation function may be used as long as the error term R (Φ) of the corresponding point position of the virtual corresponding point is used as a part of the evaluation function E2 (Φ) of the second deformation. .

<Second Embodiment>
In the first embodiment, the first deformation of the region of interest is approximated using a predetermined approximate conversion model. However, an approximate conversion model may be adaptively selected according to the nature of the region of interest. The image processing apparatus according to the present embodiment selects an appropriate approximate conversion model according to the type of a part (an organ, a lesion, or the like) corresponding to the region of interest, and uses the selected approximate conversion model for the first of the region of interest. Approximate deformation. Thereby, by using a predetermined approximate conversion model, an approximate conversion model having a property different from that of the deformation of the region of interest is applied, and it is possible to prevent the alignment accuracy from being lowered. Hereinafter, the difference between the image processing apparatus according to the present embodiment and the first embodiment will be described.

  FIG. 8 is a diagram illustrating a configuration of the image processing apparatus 800 according to the present embodiment, and the image processing apparatus 100 according to the first embodiment (FIG. 1) except that an approximate conversion model selection unit 8000 is newly added. It is the same. Further, since the functions of the region of interest acquisition unit 1020 and the deformation approximation unit 1040 are different from those of the first embodiment, their functions will be described below. Since other functions are the same as those of the first embodiment, the description of the same functions is omitted.

  The region-of-interest acquisition unit 1020 acquires a region of interest from the first image and classifies (identifies) the type of part corresponding to the region of interest. The approximate conversion model selection unit 8000 selects an approximate conversion model appropriate for the modified expression of the part according to the type of the classified (identified) part. The deformation approximation unit 1040 generates an approximate displacement field by approximating the first deformation displacement field in the region of interest using the selected approximate conversion model.

  FIG. 9 is a flowchart illustrating an overall processing procedure performed by the image processing apparatus 800. However, in this flowchart, the processes from step S9000, step S9020, and steps S9040 to S9060 are the same as the processes from step S2000, step S2020, and steps S2040 to S2060 in FIG. Only the differences from the flowchart of FIG. 2 will be described below.

(S9010) (Classification of part type corresponding to acquisition of region of interest)
In step S9010, the region-of-interest acquisition unit 1020 acquires information representing the region of interest for which deformation is desired to be suppressed on the image acquired from the data acquisition unit 1010, and classifies (identifies) the type of part corresponding to the region of interest. . Then, the acquired region of interest and its classification information are output to the approximate conversion model selection unit 8000.

  Specific processing of this step will be described. For example, consider a case where the region of interest is an organ. The region-of-interest acquisition unit 1020 extracts an organ region from the first image and classifies the corresponding organ type by a known image analysis process such as statistical atlas fitting for each organ type. Then, the region-of-interest acquisition unit 1020 acquires a region of interest based on the extracted and classified organ region. As the region of interest, at least one region of the extracted organ region is set. This may be set by an input such as designation of an area on the image by the user, or an area corresponding to an organ type (such as bone or liver) to which a predetermined deformation should be approximated is extracted. It may be set from the area. Note that, when there are a plurality of regions of interest, the region of interest acquisition unit 1020 executes these processes for each region of interest.

  Note that the region-of-interest acquisition unit 1020 may classify the organ region based on information described in the clinical information of the subject stored in the data server. For example, when the captured image is an abdominal contrast-enhanced CT image, the organ shown in the image is a region of the liver, spleen, or the like, and thus the region-of-interest acquisition unit 1020 can narrow down the types of organs. For example, the region-of-interest acquisition unit 1020 can classify the extracted region as a liver if the volume is large, or as a spleen if the volume is small.

  Further, when the region of interest is a lesioned part and the finding information is described as information accompanying the first image, the region-of-interest acquiring unit 1020 displays the classification information of the lesioned part according to the finding information. You may make it acquire. For example, if there is a description such as “hard cancer”, “benign tumor”, “DCIS”, or the like, the region-of-interest acquisition unit 1020 can set the information as site classification information. Moreover, when there is no description of them, the region-of-interest acquisition unit 1020 can set the classification information to “unknown”.

(S9015) (Selection of approximate conversion model)
In step S9015, the approximate conversion model selection unit 8000 selects the type of approximate conversion model according to the type and property of the organ corresponding to the region of interest. For example, the approximate transformation model selection unit 8000 can apply a rigid transformation model to a region classified as a bone and apply a volume-preserving affine transformation model to a region classified as a liver. Note that when there are a plurality of regions of interest, the approximate conversion model selection unit 8000 executes these processes for each region of interest. The approximate transformation model selection unit 8000 applies a rigid transformation model to “hard cancer”, a volume-preserving affine transformation model to “benign tumor”, and “no approximation” to “DCIS” and “unknown”. Can do. Note that the approximate conversion model selection unit 8000 can exclude the region of interest to which “no approximation” is applied from the target of the subsequent approximation processing. Then, the approximate conversion model selection unit 8000 outputs the type of the selected approximate conversion model to the correspondence information generation unit 1050.

(S9030) (Approximation of first deformation in region of interest by approximate transformation)
In step S9030, the deformation approximating unit 1040 calculates an approximate displacement field obtained by approximating the first deformation displacement field in the region of interest with the approximate conversion model acquired from the approximate conversion model selection unit 8000 having a smaller degree of freedom than the displacement field. Generate. Then, the deformation approximating unit 1040 outputs the generated approximate displacement field to the second correspondence information generating unit 1050.

  The processing in this step is different from step S2030 in the first embodiment only in the following points. That is, in step S2030 of the first embodiment, the deformation approximation unit 1040 uses a predetermined approximate conversion model. On the other hand, in this step, the deformation approximation unit 1040 uses the approximate conversion model acquired from the approximate conversion model selection unit 8000. The rest is the same as the processing in step S2030, and a detailed description thereof will be omitted. When there are a plurality of regions of interest, the deformation approximating unit 1040 executes these processes for each region of interest. As described above, the processing of the image processing apparatus 800 is performed.

  As described above, the image processing apparatus according to the present embodiment selects an appropriate approximate conversion model according to the type of organ corresponding to the region of interest, and performs the first deformation of the region of interest using the selected approximate conversion model. Approximate. Thereby, by using a predetermined approximate conversion model, an approximate conversion model having a property different from that of the deformation of the region of interest is applied, and it is possible to prevent the alignment accuracy from being lowered.

<Third Embodiment>
In the first embodiment, the virtual correspondence information obtained by approximating the first deformation of the region of interest by the approximate transformation with a small degree of freedom is used as it is as an alignment index for the second deformation alignment. . However, the virtual correspondence information obtained from the approximate conversion does not have to be directly used as the alignment index for the second deformation alignment. In the present embodiment, intermediate correspondence information located between the virtual correspondence information obtained from the first modification and the virtual correspondence information obtained from the approximate transformation is converted into the second modification alignment. Use as alignment index. Hereinafter, the difference between the image processing apparatus according to the present embodiment and the first embodiment will be described.

  FIG. 10 is a diagram illustrating a configuration of the image processing apparatus 1000 according to the present embodiment, and is the same as the image processing apparatus 100 according to the first embodiment except that a shape maintenance degree acquisition unit 10010 is newly added. . However, regarding the correspondence information generation unit 1050, the second deformation acquisition unit 1060, and the display control unit 1070, the functions are different from those of the first embodiment, and the functions will be described below. Since other functions are the same as those of the first embodiment, the description of the same functions is omitted.

  The shape maintenance degree acquisition unit 10010 acquires a shape maintenance degree value indicating how much the shape is maintained in the region of interest in the deformation alignment by an input from the operation unit (not shown) by the user. In other words, the shape maintenance degree is an index indicating the degree of change from the first deformation displacement field to the second deformation displacement field with respect to the region of interest. The correspondence information generation unit 1050 approximates the deformation virtual corresponding point based on the shape maintenance degree acquired from the shape maintenance degree acquisition unit 10010 in addition to the function of generating the deformation virtual correspondence point and the approximate virtual correspondence point as in the first embodiment. A virtual corresponding point located between the virtual corresponding points is generated. This is called an intermediate virtual corresponding point. The second deformation acquisition unit 1060 does not limit the deformation alignment processing performed based on the approximate virtual corresponding points in the first embodiment to any of the approximate virtual corresponding points or the intermediate virtual corresponding points. The replacement process is performed based on the virtual corresponding point. Since other functions are the same as those of the first embodiment, detailed description thereof is omitted. In addition to the same function as in the first embodiment, the display control unit 1070 acquires an instruction on whether to end the process from the operation unit (not shown) by the user, and when the instruction “end” is acquired, The process ends. When the instruction “do not end” is acquired, the state of waiting for input by the user is maintained.

  FIG. 11 is a flowchart illustrating an overall processing procedure performed by the image processing apparatus 1000. However, in this flowchart, the processing from step S11000 to S11020, S11050, S11060, and step S11100 is the same as the processing from step S2000 to S2020, S2030, S2040, and step S2060 in the flowchart of FIG. To do. In the present embodiment, the virtual correspondence information is assumed to be an approximate virtual correspondence point as described in the first embodiment. Hereinafter, differences from the flowchart of FIG. 2 will be described.

(S11025) (Display of first modified image)
In step S11025, the display control unit 1070 performs control for displaying the acquired cross-sectional images of the first deformed image and the second image on the display unit 120 in response to an operation on the operation unit (not illustrated) by the user. Do. At this time, the display control unit 1070 can control to display the same cross section of the first deformed image and the second image side by side on the display unit 120.

(S11030) (Acquisition of shape maintenance degree)
In step S11030, the shape maintenance degree acquisition unit 10010 acquires the shape maintenance degree input from the operation unit (not shown) by the user. Then, the acquired shape maintenance value is output to the correspondence information generation unit 1050 and the display control unit 1070.

In the following, the value of the shape retaining degree is expressed as lambda _R. Based on the shape maintenance degree value λ _R acquired in this step, the second deformation alignment in step S11090 is performed. The shape maintenance value λ _R satisfies the range of 0 ≦ λ _R ≦ 1. as lambda _R is closer to 0, approaches the intermediate virtual corresponding points to the position of the deformation virtual corresponding points, deformation of the region of interest close to the deformation unchanged the surroundings. That is, the region of interest does not change from the surroundings, and approaches a state that is not particularly restricted and is easily deformed. On the other hand, lambda higher _R is close to 1, close position of the intermediate virtual corresponding points to the position of the approximate virtual corresponding points, deformation of the region of interest is close to the approximate conversion. That is, the region of interest approaches a state that is unlikely to deform unlike the surroundings.

The shape maintenance degree acquisition unit 10010 can acquire λ _R by, for example, the user selecting from a plurality of predetermined values. For example, preset list of a plurality of lambda _R (e.g., {0,0.25,0.5,0.75,1.0}) is considered a case that is displayed on the display unit 120. In this case, the user through the operation unit (not shown), one of the values in the list (e.g., 0.5) is selected, the shape retaining degree acquisition unit 10010 acquires the selected value as a lambda _R .

In addition, the acquisition method of a shape maintenance degree is not restricted to this, What kind of method may be sufficient. For example, the user may input a value directly through an operation unit (not shown), and the shape maintenance degree acquisition unit 10010 may acquire the value. In this case, for example, if the user inputs a value of 1.0, the shape maintenance degree acquisition unit 10010 acquires λ _R = 1.0.

  Further, the user can observe the first deformed image and the second image displayed on the display unit 120 in step S11030, and input the value of the shape maintenance degree determined based on the result. For example, when the region of interest on the first deformation image displayed on the display unit 120 is somewhat close to the corresponding region on the second image, the user can convert the deformation of the region of interest into the transformation of the first deformation. Judging that the model can be expressed to some extent accurately, the shape maintenance degree can be set small. On the other hand, when the shape is far apart, the user can determine that the deformation of the region of interest is far from the conversion model of the first deformation, and can set the shape maintenance degree large.

As described above, the image processing apparatus 1000 according to the present embodiment can adjust the difficulty of deformation of the region of interest according to the degree of shape maintenance. For example, when rigid transformation is applied as the approximate transformation model, the image processing apparatus 1000 can adjust the approximate hardness of the region of interest according to the shape maintenance degree. Therefore, for example, the user may select the degree of qualitative hardness in a correspondence table created in advance between the “degree” of qualitative hardness and the value “λ _R ” of the shape maintenance degree. The shape maintenance degree acquisition unit 10010 may acquire the selected “degree” and λ _R corresponding thereto. More specifically, for example, “soft” = 0.0, “slightly soft” = 0.25, “intermediate” = 0.25, “slightly hard” = 0.75, “hard” = 1.0 Assume that a correspondence table for performing such mapping is stored in the image processing apparatus 1000 in advance. Then, the display control unit 1070 displays a list of qualitative hardness levels on the display unit 120 (for example, {“soft”, “slightly soft”, “intermediate”, “slightly hard”, “hard”}). indicate. When the user selects any degree (for example, “hard”) through an operation unit (not shown), the shape maintenance degree acquisition unit 10010 refers to the correspondence table and refers to a value corresponding to the degree (for example, , it is possible to obtain a 1.0) as lambda _R.

In addition, the acquisition method of shape maintenance degree (lambda) _R is not limited to the method by the manual input by a user. For example, the shape retaining degree acquisition unit 10010, a predetermined value stored in the storage unit (not shown) in advance the image processing apparatus 800 may acquire the lambda _R. The shape retaining degree acquisition unit 10010 may be acquired lambda _R based on the subject of clinical information stored in the data server 110.

For example, when the clinical information such as an interpretation report includes a finding or diagnosis name indicating a meaning such as “hard lesion”, “hard cancer”, or “calcification”, the shape maintenance degree acquisition unit 10010 Set _R = 1.0. Similarly, in the case where a finding or diagnostic name indicating a meaning such as “the lesion is soft as the surrounding tissue” or “non-invasive cancer” is described, the shape maintenance degree acquisition unit 10010 has λ _R = 0. Set to .0. In addition, an analysis unit (not shown) determines the qualitative hardness of the region of interest by analyzing the image feature of the region corresponding to the region of interest in the first image, and acquires the shape maintenance degree based on the determination. The unit 10010 may acquire λ _R by the mapping described above. Further, the region-of-interest acquisition unit 1020 extracts an organ region from the first image in the same manner as in step S9010 of FIG. 9 described in the second embodiment, and the shape maintenance degree acquisition unit 10010 receives each organ. Alternatively, a predetermined shape maintenance value may be set in each region. For example, the shape maintenance degree acquisition unit 10010 performs processing such as setting λ _R = 1.0 for a region classified as bone and λ _R = 0.75 for a region classified as liver or lung. Can do.

(S11040) (Determination whether shape maintenance degree is 0)
In step S11040, the shape maintenance degree acquisition unit 10010 determines whether or not the shape maintenance degree value λ _R is zero. If λ _R ≠ 0, the process advances to step S11050. On the other hand, if λ _R = 0, the process proceeds to step S11100. As a result, when the shape maintenance degree λ _R is other than 0, the image processing apparatus 1000 uses the processing from steps S11050 to S11090 to perform the second processing for maintaining the shape of the region of interest based on the shape maintenance degree value λ _R. A deformed image can be acquired as a final alignment result. On the other hand, lambda _R is not necessary to maintain the shape of the region of interest in the case of 0, the image processing apparatus 1000 can obtain the first modification image generated in step S11020 as it final alignment results. Thereby, the process which produces | generates a 2nd image unnecessarily can be reduced and the whole process flow can be made efficient.

(S11070) (Determination of whether or not the shape maintenance degree is 1)
In step S11070, the shape maintenance degree acquisition unit 10010 determines whether or not the shape maintenance degree value λ _R is 1. If λ _R ≠ 1, the process proceeds to step S11080. On the other hand, if λ _R = 1, the process proceeds to step S11090. Thus, otherwise 1 lambda _R, the image processing apparatus 1000 generates an intermediate virtual corresponding points by the processing in step S11080, it is aligned by applying it to the error term of the virtual corresponding point positions in step S11090 The second deformed image can be acquired as the final alignment result. Meanwhile, lambda when _R is 1, the image processing unit 1000, a second modified image of the approximate virtual corresponding point generated in step S11060, and aligned directly applied to the error term of the virtual corresponding point positions in step S11090 Can be obtained as the final alignment result. Accordingly, it is possible to reduce the process of generating intermediate virtual corresponding points unnecessarily, and to improve the efficiency of the entire process flow.

(S11080) (Generate intermediate virtual corresponding point based on shape maintenance degree)
In step S11080, the correspondence information generation unit 1050 is positioned between the deformation virtual corresponding point and the approximate virtual corresponding point on the region of interest D _ROI based on the shape maintenance degree value λ _R acquired from the shape maintenance degree acquisition unit 10010. An intermediate virtual corresponding point is generated. Then, the correspondence information generation unit 1050 outputs the generated intermediate virtual corresponding point to the second deformation acquisition unit 1060.

The position of the intermediate virtual corresponding point approaches the deformed virtual corresponding point when λ _R is close to 0, and approaches the approximate virtual corresponding point when λ _R is close to 1. The approximate virtual corresponding point constrains the deformation of the region of interest to follow the approximate transformation model. On the other hand, the intermediate virtual corresponding point has a feature that constrains the deformation of the region of interest to a state between a state that does not change from the surrounding deformation and a state that follows the approximate transformation model. A specific method for generating the intermediate virtual corresponding point will be described below.

  First, the correspondence information generation unit 1050 constitutes the deformed virtual corresponding point and the approximate virtual corresponding point, respectively, and the corresponding representative point obtained by converting the representative point on the first image, that is, the deformed representative point and the approximate representative point. A representative point located between the points is generated. This is referred to as an intermediate representative point. Next, the correspondence information generation unit 1050 generates a virtual corresponding point between images as an intermediate virtual corresponding point from the representative point and the corresponding intermediate representative point.

In the present embodiment, the following method is adopted as a specific method for generating the intermediate representative point. The i-th point at the deformation representative point and the approximate representative point is defined as P _Di and P _Ri , respectively. The coordinates of the points P _Di and P _Ri are _respectively represented by x _D2i and x _R2i as described above. i satisfies 1 ≦ i ≦ N _D = N _R. At this time, the i-th point P _Mi of the intermediate representative point is located on the line segment P _{Di PR Ri} connecting P _Di and _PRi , and the distance from the point P _{Di is equal} to the line segment P _{Di PR Ri} Is calculated as a position where the ratio becomes λ _R with respect to the distance of That is, _assuming that the coordinates of the point P _Mi are x _M2i , x _M2i is expressed by the following equation.
Thereby, a point located between the deformation representative point and the approximate representative point can be acquired as the intermediate representative point.

  FIG. 12 is a diagram showing the intermediate representative point 12000. As shown in the figure, the intermediate representative point 12000 is between a deformed representative point 4040 in which the distribution of the original representative point is slanted and a rigid representative point (approximate representative point) 6000 in the distribution that is not collapsed. It can be seen that this is a distribution.

(S11090) (Acquire second deformation based on acquired virtual corresponding point)
In step S11090, the second deformation acquisition unit 1060 aligns the first image acquired from the data acquisition unit 1010 with the second image based on the acquired virtual corresponding point. Whereas the virtual corresponding points used in step S2050 of the first embodiment are limited to approximate virtual corresponding points, the present embodiment uses a virtual corresponding point of either the approximate virtual corresponding point or the intermediate virtual corresponding point. Is different. That is, when λ _R = 1, the approximate virtual corresponding point generated at step S11060 is used, and when λ _R ≠ 1, the approximate virtual corresponding point generated at step S11080 is used. However, the deformation alignment processing based on the acquired virtual corresponding points is the same as that in step S2050, and thus detailed description thereof is omitted.

(S11110) (End determination)
In step S <b> 11110, the display control unit 1070 obtains an instruction on whether to end the process from an operation unit (not shown) by the user. If the instruction “end” is acquired, the process ends. If the instruction “do not end” is acquired, the process proceeds to step S11030, and the state of waiting for input by the user is maintained. As a result, when the user observes the first deformed image or the second deformed image as the alignment result in step S11100 and determines that the alignment is completed, the command “ends” through the operation unit (not shown). The image processing apparatus 1000 can end the processing. On the other hand, as a result of observing the image of the alignment result, if the user wants to change the shape maintenance degree and observe the result again, by giving a command “do not end” through an operation unit (not shown) The processing apparatus 1000 can again generate and display the alignment result by giving a new shape maintenance degree.

  As described above, the processing of the image processing apparatus 800 is performed. With this processing flow, for example, in step S11020 and step S11090, when conversion parameters can be calculated by linear optimization, each alignment is possible at high speed. Therefore, when the user interactively inputs the shape maintenance level to the image processing apparatus 1000, the second modified image corresponding to the input can be displayed. More specifically, it is assumed that an operation unit (not shown) is a slider that can adjust the shape maintenance degree from 0 to 1. At this time, when the user moves the slider, the image processing apparatus 1000 generates and displays a deformed image based on the shape maintenance degree of the slider position by giving the image processing apparatus 1000 a command “do not end”. it can. As a result, for example, when the user gradually moves the slider position from 0 to 1, the image processing apparatus 1000 displays the second deformed image whose shape is gradually maintained in association with it in almost real time. Can be displayed on the unit 120. Therefore, the user can easily determine the second deformed image based on an appropriate shape maintenance degree while observing the second image and the second deformed image displayed on the display unit 120. Note that the interface of the operation unit (not shown) for the user to interactively input the shape maintenance level is not limited to the slider, and may be, for example, a value directly input.

  According to the present embodiment, the image processing apparatus 1000 uses an intermediate virtual corresponding point located between the deformation virtual corresponding point and the approximate virtual corresponding point of the region of interest as an alignment index between images. Thereby, with respect to the deformation of the region of interest, an alignment result is obtained so that a deformation state between the first deformation and the approximate conversion is obtained. Therefore, the entire subject can be properly aligned even if the image includes a lesion that is harder than the surrounding but deformed.

(Modification 3-1) (Intermediate virtual corresponding point is generated by nonlinear interpolation)
In the third embodiment, in step S11080, the intermediate representative point constituting the intermediate virtual corresponding point is a point on the line segment connecting the deformation representative point and the approximate representative point. In other words, the coordinates of the transformation representative point and the approximate representative point are linearly interpolated. However, the method of generating the intermediate representative point is not limited to this, and may be coordinates obtained by nonlinearly interpolating between the coordinates of the deformed representative point and the approximate representative point. That is, the change from the first deformation displacement field to the second deformation displacement field with respect to the region of interest may be linear or non-linear.

For example, the following method may be employed. First, the correspondence information generation unit 1050 extracts only those set on the surface of the region of interest from each of the deformation representative point and the approximate body surface point of the region of interest. These are referred to as a deformed surface representative point and an approximate surface representative point, respectively. Next, the correspondence information generation unit 1050 linearly interpolates an intermediate surface representative point located between the deformed surface representative point and the approximate surface representative point based on the shape maintenance degree value λ _R as in step S11080. Generate. This is referred to as an intermediate surface representative point. Then, the correspondence information generation unit 1050 uses the positions of the approximate surface representative point and the intermediate surface representative point as boundary conditions, and estimates the deformation inside the region of interest in the direction from the approximate surface representative point toward the intermediate surface representative point by physical deformation simulation. To do.

  Here, the simulation is performed based on the approximate surface representative point because the state in which the region of interest follows the approximate conversion model is a state in which no external force is virtually generated. It is assumed that the region of interest corresponding to the deformed surface representative point is in a state where the external force is most generated, and the region of interest corresponding to the intermediate surface representative point is in a state where the intermediate external force is generated. A predetermined value is set in advance as the value of the Young's modulus / Poisson's ratio of the region of interest necessary for the simulation. For example, the image processing apparatus 1000 prepares a table of Young's modulus / Poisson ratio values corresponding to each type of region of interest in advance, and the correspondence information generation unit 1050 sets the type of region of interest of the case as clinical data or the like. To obtain and set the corresponding Young's modulus / Poisson's ratio value from the table. It is assumed that the information on the type of region of interest is acquired from the data server 110 by the correspondence information generation unit 1050 through the data acquisition unit 1010 and the region of interest acquisition unit 1020. Then, the correspondence information generation unit 1050 can acquire a representative point inside the region of interest corresponding to the intermediate surface representative point by converting the approximate representative point using the displacement field obtained by the simulation. This is referred to as an intermediate internal representative point. Then, when the intermediate surface representative point and the intermediate internal representative point are combined, the correspondence information generating unit 1050 obtains an intermediate converted representative point for the entire region of interest, and acquires this as the intermediate representative point.

  As a result, the surface of the region of interest becomes a coordinate obtained by linearly interpolating the coordinates of the approximate representative point and the deformation representative point. On the other hand, the internal region can be acquired as coordinates obtained by interpolating the coordinates of the approximate representative point and the deformation representative point in a nonlinear manner (in a more physically consistent position). Note that, for example, by changing the Young's modulus inside the region of interest depending on the region, different nonlinear interpolation can be performed depending on the region. For example, by increasing the Young's modulus closer to the center of gravity of the region of interest and decreasing the Young's modulus closer to the surface of the region of interest, the region near the center of gravity of the region of interest becomes harder and the amount of displacement decreases. On the other hand, the area near the surface of the region of interest becomes soft and the displacement becomes large. As a result, it is possible to acquire a deformation that is close to reality with respect to a lesion having a hard central portion and a soft peripheral portion.

(Modification 3-2) (A plurality of shape maintenance degrees are acquired at a time to generate a plurality of second images)
In 3rd Embodiment, in step S11030, the shape maintenance degree acquisition part 10010 acquired only one shape maintenance degree. Then, only one deformed image corresponding to the value is generated through steps S11040 to S11090. Further, if necessary, the process of reacquiring the shape maintenance degree and acquiring the corresponding deformed image has been repeated. However, the shape maintenance degree acquisition unit 10010 may acquire a plurality of shape maintenance degree values at a time and generate a plurality of deformed images corresponding to the values.

For example, in step S11030 after steps S11000 to S11025, the shape maintenance degree acquisition unit 10010 acquires a plurality of predetermined shape maintenance degree values λ _R. For example, the shape retaining degree acquisition unit 10010 acquires all values of {0,0.25,0.5,0.75,1.0} as lambda _R. At this time, the image processing apparatus 1000 does not perform the processes in steps S11040 and S11070. Next, the deformation approximating unit 1040 and the correspondence information generating unit 1050 perform the processes of steps S11050 and S11060 to generate approximate virtual corresponding points. Next, in step S11080, the correspondence information generation unit 1050 generates intermediate virtual corresponding points corresponding to all of the shape maintenance degrees where λ _R ≠ 0 and λ _R ≠ 1. In the above example, the correspondence information generation unit 1050 generates three types of intermediate virtual corresponding points corresponding to {0.25, 0.5, 0.75}, respectively. In step S11090, the second deformation acquisition unit 1060 corresponds to the second modified image corresponding to each of the approximate virtual corresponding points generated in step S11060 and all the intermediate virtual corresponding points generated in step S11080. Is generated. In the above example, three types of intermediate virtual corresponding points (corresponding to λ _R = 0.25, 0.5, 0.75) and approximate virtual corresponding points (corresponding to λ _R = 1) are supported. A second deformed image is generated.

In step S11100, the display control unit 1070 displays all the generated modified images. In the above example, the first modified image (corresponding to λ _R = 0) and the second modified image (corresponding to λ _R = 0.25, 0.5, 0.75, 1.0) A total of five types of deformed images are displayed. As a display method, for example, all the deformed images may be displayed side by side, or may be switched and displayed on the same display area in accordance with an input from a user operation unit (not shown).

  According to this modification, the image processing apparatus 1000 can process the processes from steps S11000 to S11090 in advance offline without user input. In this case, the image processing apparatus 1000 stores all the generated deformed images in a storage unit (not shown). In step S11100, the deformed image is read from the storage unit and displayed. In the third embodiment, in order to display the results of different shape maintenance degrees, it is necessary to perform the processing from step S11040 to S11090 each time, giving a waiting time to the user. However, the process of this modification only reads and displays a plurality of modified images generated in advance in step S11100. Therefore, the user can observe the deformed image without waiting time, and can work efficiently.

<Fourth Embodiment>
In the first embodiment, the error of the position corresponding to the landmark is employed as an example of the evaluation function for alignment. However, the evaluation function may include an information error that is not acquired as a point corresponding to one-to-one in advance. The image processing apparatus according to the present embodiment obtains the degree of coincidence of surface shapes between images with respect to a certain conversion parameter based on information on the surface shape of the subject extracted from each image, and uses this as an error term of the evaluation function. Combined. As a result, conversion parameters that match the surface shapes can be calculated. Hereinafter, the difference between the image processing apparatus according to the present embodiment and the first embodiment will be described.

  FIG. 13 is a diagram illustrating a configuration of the image processing apparatus 1300 according to the present embodiment. Except for the point that a surface shape acquisition unit 13010 is newly added, the image processing apparatus 100 according to the first embodiment (FIG. 1) and FIG. It is the same. Further, since the functions of the first deformation acquisition unit 1030 and the second deformation acquisition unit 1060 are different from those of the first embodiment, the functions will be described below. Since other functions are the same as those of the first embodiment, the description of the same functions is omitted.

  The surface shape acquisition unit 13010 acquires the surface shape of the subject from the first image and the second image, respectively. The functions of the first deformation acquisition unit 1030 and the second deformation acquisition unit 1060 are substantially the same as those of the first embodiment, but only the point of performing the deformation alignment based on the acquired surface shapes is only This is different from the first embodiment.

  FIG. 14 is a flowchart illustrating an overall processing procedure performed by the image processing apparatus 1300. However, in this flowchart, the processes of steps S14000, S14010, S14030, S14040, and S14060 are the same as the processes of steps S2000, S2010, S2030, S2040, and S2060 in FIG. Only the differences from the flowchart of FIG. 2 will be described below.

(S14015) (Acquisition of surface shape)
In step S14015, the surface shape acquisition unit 13010 acquires information representing the surface shape of the subject in each of the first image and the second image acquired from the data acquisition unit 1010. Here, the surface of the subject is, for example, the body surface or the pectoral muscle surface when the subject is a breast. When the subject is an organ such as the liver or kidney, the surface is the surface of the organ. Here, the surface shape can be automatically acquired by performing image processing such as a surface enhancement filter and edge detection on the image, for example. Alternatively, the user may observe the image, and the surface shape may be acquired based on an input operation or the like by the user. The surface shapes of the acquired first image and second image are referred to as a first surface shape and a second surface shape, respectively. In this embodiment, the surface shape is constituted by a point group. However, the surface shape does not necessarily need to be a point group, and any form can be used as long as the shape can be expressed. For example, the surface shape may be a function approximated by a mathematical expression that can express a curved surface such as a polynomial (hereinafter referred to as a curved surface function). Then, the acquired first surface shape and second surface shape are output to the first deformation acquisition unit 1030 and the second deformation acquisition unit 1060.

(S14020) (Acquisition of first deformation based on surface shape)
In step S14020, the first deformation acquisition unit 1030 performs first deformation alignment between the acquired first image and the second image based on the acquired first surface shape and second surface shape. To obtain the first deformation displacement field. Then, the first deformation acquisition unit 1030 outputs the acquired first deformation displacement field to the correspondence information generation unit 1050. Furthermore, the first deformation acquisition unit 1030 generates a first deformation image obtained by deforming the first image based on the first deformation displacement field, and outputs the first deformation image to the display control unit 1070.

  At this time, the first deformation acquisition unit 1030 defines an evaluation function E1 (Φ) that evaluates the appropriateness of the alignment by the deformation Φ, and minimizes the evaluation function, as in step S2020 of the first embodiment. The deformation Φ is estimated. The difference in processing between this step and step S2020 is only the method of calculating the evaluation function E1 (Φ), and the other processing is the same as that of step S2020, and thus the description thereof is omitted. In step S2020, in the evaluation function E1 (Φ), the error of the actual corresponding point position is represented by the error of the landmark corresponding point position, whereas in this step, the error of the actual corresponding point position is the landmark correspondence. It differs in that it is expressed by a function including the error of the surface shape between images in addition to the error of the point position. Here, the surface shape error is expressed as an error in the position of a corresponding point (surface corresponding point) in which point groups representing surface shapes between images are associated with each other.

  A specific calculation method will be described below. The surface-corresponding point position error term is generated by generating a surface-corresponding point corresponding to one-to-one between the first surface shape and the second surface shape represented by the point group, respectively. Calculated as the error of the surface corresponding point. However, the first surface shape and the second surface shape are not associated with each other from the beginning. Therefore, alignment is performed between the curved surface having the surface shape represented by the point group constituting the first surface shape and the curved surface having the surface shape represented by the point group constituting the second surface shape. It is necessary to associate the groups. This process can be executed using, for example, the Iterative Closest Point (ICP) method using a known method. If the original surface shape differs greatly between the images, first, a deformation displacement field using only the error of the landmark corresponding point is calculated by the same processing as in step S2020, and this deformation displacement field is calculated as the first surface. By applying to the shape, the first surface shape after the deformation that once brought the shape close to the second surface shape is acquired. And you may make it match | combine by the ICP method etc. between the point group showing the 1st surface shape after a deformation | transformation, and the point group showing the 2nd surface shape. Thereby, even when the original surface shape differs greatly between images, a point group can be appropriately matched between the first surface shape and the second surface shape. The method for associating the surface shape is not limited to this. For example, one surface shape is obtained as a point group, the other surface shape is obtained as a curved surface function, and the nearest point on the other curved surface function corresponding to one point group is obtained between these point groups and the curved surface function. While searching, the point cloud may be associated by the ICP method or the like. Thereby, the point group showing one surface shape is matched with the point on the surface shape expressed in the other continuous form. Therefore, when the surface shape of both images is discretized as a point group, there is no point on the other surface shape that exactly corresponds to the point on one surface shape, and an error occurs. There is no problem.

Next, the error term Surf (Φ) of the surface corresponding point is expressed by the following equation.
Here, in equation (9), Φ (x | p1) represents a coordinate obtained by displacing the coordinate x by a first deformation Φ represented by a predetermined conversion model with a parameter p1. _Further , (x _S1i , x _S2i ) represents a set of coordinates of the i-th surface corresponding point on the first image and the second image. NS represents the total number of surface corresponding points. Cov (x) represents a function that returns a covariance matrix related to a three-dimensional error at the position x, as in Modification 1-2 of the first embodiment. The equation (9) is an equation in which the approximate virtual corresponding point (x _R1i , x _R2i ) in the equation (6) is _replaced with the surface corresponding point (x _S1i , x _S2i ) and the conversion parameter p2 is replaced with the conversion parameter p1. is there. Therefore, when parameters are replaced in this manner in Modification 1-2, equation (9) can be interpreted as follows. That is, Surf (Φ) is used to prevent the surface corresponding point of the first surface shape after displacement by the conversion parameter p1 from deviating only in the normal direction from the surface corresponding point of the corresponding second surface shape. Functions as a term. That is, the surface corresponding point set on the surface shape is allowed to move along the surface shape surface when calculating the first displacement field. Note that the error of the evaluation function E1 (Φ), that is, the actual corresponding point can be calculated by adding the error term Surf (Φ) of the surface corresponding point to the error term of the landmark corresponding point as described above. The error of the actual corresponding point may be composed only of the error term Surf (Φ) of the surface corresponding point.

  As described above, since the evaluation function E1 (Φ) is represented only by the error term of the corresponding point position, as described in step S2050 of the first embodiment, the error (error) between the conversion parameter and the corresponding point is calculated. The relationship can be expressed by a linear equation. Therefore, the conversion parameter p1 that minimizes the error between corresponding points can be estimated by the linear optimization method.

(S14050) (Acquisition of second deformation based on surface shape)
In step S14050, the second deformation acquisition unit 1060 acquires the second deformation displacement field using the approximate virtual corresponding point (virtual correspondence information) acquired in step S2040 as a constraint condition. At this time, in the calculation of the second evaluation function E2 (Φ), the first point in consideration of the degree of coincidence between the first surface shape and the second surface shape acquired in step S14015 is the same as the processing in step S14020. This is different from the embodiment. Here, in the calculation of the second evaluation function E2 (Φ), the error term R (Φ) of the approximate virtual corresponding point added to the first evaluation function E1 (Φ) is also an error term of the corresponding point position. Therefore, since the final alignment evaluation function E2 (Φ) is also expressed only by the error term of the corresponding point position, the relationship between the conversion parameter and the error between the corresponding points can be expressed by a linear equation. Therefore, the conversion parameter p2 that minimizes the error between corresponding points can be estimated by the linear optimization method.

  Thus, the processing of the image processing apparatus 1300 is performed. According to this embodiment, it is possible to perform alignment so that the surface shape of the subject matches. At this time, even if the evaluation function includes the degree of coincidence (error) of the surface shapes between images that have not been previously acquired as corresponding points, the conversion parameter can be calculated by linear optimization, and the alignment result is acquired at high speed. be able to. Furthermore, according to the present embodiment, in the alignment, the surface corresponding points set on the surface shape of the subject are allowed to move along the surface shape surface. Thereby, the following effects are obtained. In this embodiment, the surface shapes are associated with each other by a method such as the ICP method, but anatomically on the surface shape surface of each image due to instability of the algorithm or insufficient accuracy. Strictly correct positions are not always associated. For this reason, an association error occurs to some extent. Accordingly, the distance between corresponding points existing in the vicinity on the surface shape may greatly differ between images due to an error in the correspondence of the position on the surface shape surface. As a result, if the position is simply aligned using an evaluation function that matches the positions of the corresponding points as much as possible, the surface shape locally expands and contracts unnaturally. On the other hand, by aligning with an evaluation function that allows movement on the surface shape surface, an error is allowed along the surface shape surface based on the properties of the transformation model that expresses the deformation. Alignment is performed. Thus, by applying a transformation model such as FFD or TPS that is known to express smooth deformation as in the present embodiment, the surface shape of the subject does not become locally unnaturally deformed. Such an alignment result can be obtained.

(Modification 4-1)
In the above embodiment, as in the first to third embodiments, the second deformation for controlling the deformation of the region of interest is acquired. However, even when the deformation of the region of interest is not controlled (it is not necessary to control), the effect of the above processing using the processing for matching the surface shape of the subject for alignment is not impaired. In other words, the first modification may be used as a modification result without executing the processes of steps S14010 and S14030 to S14050.

<Fifth Embodiment>
In the modification 4-1 of the fourth embodiment described above, the case where the degree of coincidence of the surface shapes between images is used as the error term of the evaluation function for alignment based on the surface shape of the subject has been described. However, these embodiments are only examples in the practice of the present invention. In the present embodiment, an example of a form different from Modification 4-1 will be described.

  FIG. 15 is a diagram illustrating a configuration of an image processing apparatus 1500 according to the present embodiment. The image processing apparatus 1500 according to the present embodiment includes the same constituent elements as some of the constituent elements of the image processing apparatus 1300 described in the fourth embodiment. Components having similar functions are denoted by the same reference numerals, and detailed description thereof is omitted.

  FIG. 16 is a flowchart showing an overall processing procedure performed by the image processing apparatus 1500 according to this embodiment.

(S15000) Data Acquisition In step S15000, the data acquisition unit 1010 acquires a first image and a second image from the data server 110. Since this process is the same as step S14000 in the fourth embodiment, a description thereof will be omitted.

(S15015) Acquisition of Surface Shape In step S15015, the surface shape acquisition unit 13010 acquires information representing the surface shape of the subject in each of the first image and the second image acquired from the data acquisition unit 1010. Since this process is the same as step S14015 in the fourth embodiment, a description thereof will be omitted.

(S15020) Acquisition of First Deformation Based on Surface Shape In step S15020, the first deformation acquisition unit 1030 acquires the first image and the first acquired based on the acquired first surface shape and the second surface shape. A first deformation alignment between the two images is performed to obtain a first deformation displacement field. Furthermore, the first deformation acquisition unit 1030 generates a first deformation image obtained by deforming the first image based on the first deformation displacement field, and outputs the first deformation image to the display control unit 1070.

  FIG. 17 is a flowchart for explaining in more detail the processing executed by the first deformation acquisition unit 1030 in this processing step.

(S15022) Correspondence Relationship Acquisition In step S15022, the first deformation acquisition unit 1030 executes an association process between point groups constituting the first surface shape and the second surface shape. As a specific processing method, the ICP method described as an example in step S14020 in the fourth embodiment can be used. By this processing, surface corresponding points (x _S1i , x _S2i ), 1 ≦ i ≦ N _S are acquired. Here, N _S is the total number of surface corresponding points.

(S15024) Normal Direction Acquisition In step S15024, the first deformation acquisition unit 1030 executes a normal direction calculation process related to the surface shape at the position for each of the surface corresponding points acquired in step S15022. Here, x _S1i and x _S2i constituting the surface corresponding points (x _S1i , x _S2i ) are points constituting the first surface shape and the second surface shape, respectively. In this process step, a vector representing a first surface vector representing the normal direction of the shape and (first normal vector) n _1i, the normal direction of the second surface shape at the position x _S2i at position x _S1i ( The second normal vector) n _2i is calculated and acquired. A method for calculating the normal vector from the surface shape can be calculated by the method described in Modification 1-2 (a method obtained by performing principal component analysis on the position of a point group near the position). Alternatively, the normal vector may be calculated by calculating a surface shape distance field and calculating the normal vector based on the spatial gradient of the distance field. The normal vector n _{1i at} the position x _S1i of the first surface shape and the normal vector n _2i at the position x _S2i of the second surface shape obtained by the above method are obtained. Note that n _1i and n _2i are three-dimensional vectors each having a norm normalized to 1.0.

(S15026) Deformation Calculation In step S15026, the first deformation acquisition unit 1030 acquires the first deformation displacement field based on the processing results of steps S15022 and S15024.

At this time, the first deformation acquisition unit 1030 defines an evaluation function E1 (Φ) that evaluates the appropriateness of alignment by the deformation Φ, and minimizes the evaluation function, as in step S14020 of the fourth embodiment. The deformation Φ is estimated.
In the equation (10), Φ (x | p1) represents coordinates obtained by displacing the coordinate x by a first deformation Φ expressed by a predetermined conversion model with a parameter p1. In the present embodiment, Φ (x | p1) means conversion from the coordinate system of the first image to the coordinate system of the second image. Further, Φ ⁻¹ (x | p1) is an inverse function of Φ (x | p1), and means conversion from the coordinate system of the second image to the coordinate system of the first image. Cov _1i and Cov _2i are covariance matrices related to a three-dimensional error at the position of the i-th surface corresponding point. Cov _1i is calculated based on the normal vector n _1i at the position x _S1i of the first surface shape. Cov _2i is calculated based on the normal vector n _2i at the position x _S2i of the second surface shape.

(S15028) Deformed Image Generation In step S15028, the first deformation acquisition unit 1030 generates a first deformed image obtained by deforming the first image based on the first deformation displacement field acquired in step S15026. The data is output to the display control unit 1070.

  From step S15022 to step S15028 described above, the process of step S15020 in this embodiment is executed.

(S15060) Display of Deformed Image In step S15060, the display control unit 1087 displays the acquired deformed image and the cross-sectional image of the second image to the display unit 120 in response to an operation on the operation unit (not shown) by the user. Control the display. Since this process is the same as step S14060 in the fourth embodiment, a detailed description thereof will be omitted.

  As described above, the processing of the image processing apparatus 1500 in the sixth embodiment is performed. According to the present embodiment, the first deformation displacement field can be acquired based on both the normal vector related to the first surface shape and the normal vector related to the second surface shape. Thereby, there exists an effect which can be made to operate | move stably with respect to the difference of a 1st surface shape, and a 2nd surface shape.

(Modification 5-1): Method for calculating normal direction (also using luminance gradient of image)
In the above embodiment, the case where the first normal vector is calculated based on the first surface shape has been described as an example. However, the implementation of the present invention is not limited to this. For example, the first normal vector n _1i may be calculated based on the luminance gradient in the vicinity of the surface corresponding point position x _S1i of the first image. Similarly, the second normal vector n _2i may be calculated based on the luminance gradient in the vicinity of the surface corresponding point position x _S2i of the second image. Even when it is difficult to calculate the normal vector with high accuracy from the information on the surface shape, such as when the distribution of the point cloud representing the surface shape is spatially sparse, the subsequent processing can be executed.

  The normal vector may be calculated based on both the surface shape and the luminance gradient of the image. For example, an intermediate vector between the normal vector calculated based on the surface shape and the normal vector calculated based on the luminance gradient of the image may be calculated, and the subsequent processing may be executed using the vector. . Further, based on the roughness density of the surface shape, switching between calculating the normal vector from the surface shape or calculating the normal vector from the luminance gradient of the image may be performed. For example, if the spatial density of the point cloud representing the surface shape is greater than a predetermined threshold, the normal vector is calculated based on the surface shape, otherwise the normal vector is calculated based on the brightness gradient of the image. You can do that. Further, the above switching may be performed depending on the magnitude of the luminance gradient of the image. According to these methods, there is an effect that the normal vector can be calculated more stably with respect to the point cloud data representing the surface shape and the luminance distribution of the image.

(Modification 5-2): Case where two normal directions are integrated and used In the above-described embodiment, the first normal vector and the second normal vector are calculated, and the equation (10) is calculated based on them. The case where the evaluation function described in 1 is used has been described as an example. However, the implementation of the present invention is not limited to this. For example, a normal vector obtained by integrating these may be calculated based on both the first normal vector and the second normal vector. And you may make it acquire 1st deformation | transformation transformation | conversion using the evaluation function described in Formula (9) demonstrated in 4th Embodiment based on the integrated normal vector. Specifically, the first normal vector defined in the coordinate system of the first image acquired in step S15015 is converted into the normal vector of the second image coordinate system based on the coordinate transformation Φ (x | p1). Convert to. More specifically, the normal vector is calculated by n _1i ′ = Φ (x _1i + n _1i | p1) −Φ (x _1i | p1). Then, an intermediate vector n _mi is calculated from n _1i ′ and n _2i . Then, the covariance matrix Cov (x) is calculated by the method described in step S14020 of the fourth embodiment, and the first deformation displacement field is obtained using the evaluation function described in the equation (9) of the same embodiment. You can do that. According to the method described above, the present invention can be implemented by a simpler calculation than the method described in the sixth embodiment using Expression (10).

Further, when calculating an intermediate vector between the first normal vector n _1i ′ converted into the coordinate system of the second image and the second normal vector n _2i , these vectors are weighted differently. May be. For example, this weight can be set based on the type of the imaging device (modality) of the first image and the second image. For example, when the first image is captured with a modality that can calculate the surface shape of the subject with higher reliability than the second image, the weight for the first normal vector can be increased. . According to this, since the normal vector can be calculated with emphasis on more reliable information on the surface shape, there is an effect that the first deformation displacement field can be acquired with high accuracy. The weighting is not limited to the type of modality, and for example, the reliability of the first surface shape and the second surface shape may be acquired, and the weight may be set based on the reliability. This reliability may be obtained, for example, by an input operation by a user, or in addition to this, the spatial density of point groups representing the first and second body surface shapes, and the first and second images. You may make it acquire based on a noise level etc. In this case, the same weight may be set for all the body surface corresponding points, or the weight may be individually set for each of the body surface corresponding points. The set weight may be a continuous value from 0 to 1, or may be a binary weight such as 0 or 1.

(Modification 5-3): Evaluation values are calculated by integrating evaluation values in respective normal directions In the sixth embodiment, calculation is performed based on the first normal vector as shown in Expression (10). An example has been described in which the overall evaluation value is calculated by adding the evaluation value to be calculated and the evaluation value calculated based on the second normal vector. However, the implementation of the present invention is not limited to this. For example, the evaluation function may be configured as shown in Expression (11).
Here, the function min (a, b) is a minimum value selection function, and the evaluation value (the smaller value) of the evaluation value that is the first argument and the evaluation value that is the second argument of the function min. ) Is adopted as the overall evaluation. According to this, there is an effect that a more appropriate deformation displacement field can be acquired when the directions of the first normal vector and the second normal vector are greatly different.

<Sixth Embodiment>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 110 Data server 120 Display part 1010 Data acquisition part 1020 Region of interest acquisition part 1030 1st deformation | transformation acquisition part 1040 Deformation approximation part 1050 Correspondence information generation part 1060 2nd deformation | transformation acquisition part 1070 Display control part

Claims (25)

Data acquisition means for acquiring a first image and a second image obtained by imaging the subject under different conditions;
A region of interest acquisition means for acquiring a region of interest in the first image;
First deformation acquisition means for acquiring a first deformation displacement field between the first image and the second image;
Deformation approximating means for generating an approximate displacement field by approximating the first deformation displacement field in the region of interest using an approximate transformation model having a lower degree of freedom than the first deformation displacement field;
Correspondence information generating means for generating correspondence information between the first image and an image in which the first image is displaced based on the approximate displacement field with respect to the region of interest;
Second deformation acquisition means for acquiring a second deformation displacement field between the first image and the second image using the correspondence information;
An image processing apparatus comprising: The first deformation acquisition means generates a first deformation image obtained by displacing the first image based on the first deformation displacement field,
The image processing according to claim 1, wherein the second deformation acquisition unit generates a second deformed image obtained by displacing the first image based on the second deformation displacement field. apparatus.   The correspondence information is acquired based on a plurality of representative points set on the region of interest and a plurality of representative points obtained by displacing each of the representative points using the approximate displacement field. The image processing apparatus according to claim 2, wherein the image processing apparatus is a corresponding point.   The image processing apparatus according to claim 3, wherein a positional relationship between the set representative points and the displaced representative points coincides.   The image processing apparatus according to claim 3, wherein the plurality of displaced representative points are allowed to move along a surface of the region of interest.   The correspondence information is a virtual shape acquired based on a region set on the region of interest and a region obtained by displacing the set region using the approximate displacement field. The image processing apparatus according to claim 2.   The approximate transformation model selection means for selecting an approximate transformation model used for generating the approximate displacement field according to the property of the region of interest is further provided. The image processing apparatus described.   When there are a plurality of the regions of interest, the approximate conversion model selection unit selects an approximate conversion model used for generating the approximate displacement field according to the property of each region of interest. The image processing apparatus according to claim 7. A shape maintenance degree acquisition means for acquiring a shape maintenance degree indicating a degree of change from the first deformation displacement field to the second deformation displacement field with respect to the region of interest;
The correspondence information generating means generates correspondence information between the first image and an image in which the first image is displaced based on the approximate displacement field and the shape maintenance degree. The image processing apparatus according to any one of claims 1 to 8.   The image processing apparatus according to claim 9, wherein a change from the first deformation displacement field to the second deformation displacement field with respect to the region of interest is linear.   The image processing apparatus according to claim 9, wherein a change from the first deformation displacement field to the second deformation displacement field with respect to the region of interest is nonlinear. The first deformation displacement field is calculated based on an error evaluation of a position of an actual corresponding point that actually corresponds between the first image and the second image,
The image processing apparatus according to claim 3, wherein the second deformation displacement field is calculated based on an error of a position of the actual corresponding point and an evaluation of an error of a position of the virtual corresponding point. Surface shape acquisition means for acquiring a first surface shape of the subject depicted in the first image and obtaining a second surface shape of the subject depicted in the second image is further provided. ,
The image processing apparatus according to claim 12, wherein the actual corresponding point is a surface corresponding point that corresponds between the first surface shape and the second surface shape.   The error of the position of the actual corresponding point is evaluated by greatly evaluating the position error in the normal direction of the curved surface of either the first surface shape or the second surface shape, and the direction along the curved surface. The image processing apparatus according to claim 13, wherein the position error is evaluated to be small.   The image processing apparatus according to claim 1, wherein the approximate conversion model is a rigid body conversion model.   The image processing apparatus according to claim 1, wherein the acquisition unit acquires the region of interest based on an operation by a user.   17. The image processing according to claim 1, further comprising display control means for performing control for displaying the first modified image and the second modified image on a display unit. apparatus. Data acquisition means for acquiring a first image and a second image obtained by imaging the subject under different conditions;
Surface shape obtaining means for obtaining a first surface shape of the subject depicted in the first image and obtaining a second surface shape of the subject depicted in the second image;
Deformation acquisition means for acquiring a deformation displacement field between the first image and the second image based on evaluation of an error between the first surface shape and the second surface shape. ,
In the evaluation of the error, the position error in the normal direction of one of the first surface shape and the second surface shape is greatly evaluated, and the position error in the direction along the curved surface is small. An image processing apparatus characterized by being evaluated. First shape acquisition means for acquiring a first surface shape of the subject in the first deformation state;
Second shape acquisition means for acquiring a second surface shape of the subject in a second deformation state;
Normal direction acquisition means for acquiring a first normal direction on the surface related to the first surface shape;
A correspondence acquisition means for acquiring a correspondence of a plurality of positions between the first and second surface shapes;
Calculation for calculating coordinate transformation between the first deformation state and the second deformation state related to the subject, using a distance calculated based on the normal direction of the correspondence relationship of the plurality of positions as an index. Means,
An image processing apparatus comprising: The first shape acquisition means acquires a first surface shape based on a first image obtained by imaging the subject in the first deformation state,
The second shape acquisition means acquires a second surface shape based on a second image obtained by imaging the subject in the second deformation state,
Generating means for generating a deformed image obtained by deforming the first image or the second image based on the coordinate transformation calculated by the calculating means;
The image processing apparatus according to claim 19, further comprising: The normal direction acquisition means further acquires a second normal direction related to the second surface shape,
The image processing apparatus according to claim 19, wherein the calculation unit calculates the coordinate transformation based further on the second normal direction calculated by the normal direction acquisition unit.   The said calculation means acquires the reliability regarding each of the said 1st normal line direction and the said 2nd normal line direction, and calculates the said coordinate transformation further based on the said reliability. The image processing apparatus according to 21. A first shape acquisition step of acquiring a first surface shape of a subject in a first deformation state;
A second shape acquisition step of acquiring a second surface shape of the subject in a second deformation state;
A normal direction acquisition step of acquiring a first normal direction on the surface with respect to the first surface shape;
A correspondence acquisition step of acquiring a correspondence of a plurality of positions between the first and second surface shapes;
Calculation for calculating coordinate transformation between the first deformation state and the second deformation state related to the subject, using a distance calculated based on the normal direction of the correspondence relationship of the plurality of positions as an index. An image processing method comprising the steps. A data acquisition step of acquiring a first image and a second image obtained by imaging the subject under different conditions;
A region of interest acquisition step of acquiring a region of interest in the first image;
A first deformation acquisition step of acquiring a first deformation displacement field between the first image and the second image;
A deformation approximation step of generating an approximate displacement field by approximating the first deformation displacement field in the region of interest using an approximate transformation model having a lower degree of freedom than the first deformation displacement field;
A correspondence information generating step for generating correspondence information between the first image and an image in which the first image is displaced based on the approximate displacement field with respect to the region of interest;
A second deformation acquisition step of acquiring a second deformation displacement field between the first image and the second image using the correspondence information;
An image processing method comprising:   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 22.