JP2018192306A - Information processing apparatus - Google Patents

Abstract

To allow a user to efficiently search for a corresponding region on a two-dimensional image which corresponds to a region of interest in a three-dimensional image.SOLUTION: An information processing apparatus is capable of projecting a given region (e.g., a region of interest or a lesion of interest) in a three-dimensional image (e.g., an MRI image or an X-ray CT image) onto a plane including a two-dimensional image (e.g., ultrasonic image) of an object, and causes display means to display an error range caused by projection (a range where a corresponding region of the region of interest in the two-dimensional image may exist, also referred to as a search range) including the projected region in such a manner that the error range is overlaid on the two-dimensional image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus for efficiently searching for a region corresponding to a region of interest between images having different photographing conditions such as modality, photographing position, photographing date and time.

  In the medical field, doctors display medical images (three-dimensional image data consisting of tomographic image groups representing three-dimensional information inside the subject) on a monitor, and interpret the displayed images. Diagnose the lesion. Examples of medical image acquisition devices (hereinafter referred to as modalities) that capture medical images include ultrasonic diagnostic imaging devices, magnetic resonance imaging devices (hereinafter referred to as MRI devices), X-ray computed tomography devices (hereinafter referred to as X-ray CT). And the like).

  It is difficult to correctly diagnose the state of a lesioned part only by observing medical images taken with these individual modalities. Therefore, an attempt has been made to correctly diagnose the state of a lesion by comparing each medical image captured with a plurality of modalities and a lesion in each medical image captured at different dates and times. .

  In order to use a plurality of types of medical images for diagnosis, it is important to identify (correlate) a lesioned portion or the like in each medical image. Since it is difficult to automate identification by image processing due to differences in modalities, deformation of the subject, etc., it is common for an operator such as a doctor to manually perform the identification work while viewing the image. While looking at the image of the target lesion of interest noted in one of the medical images (hereinafter referred to as the reference image), the worker uses the similarities such as the shape of the lesion and the appearance of the surrounding area as a clue, The corresponding lesion part corresponding to the lesion part is searched and identified from the other medical image (hereinafter, target image). Here, if the device that presents the medical image has a function of estimating and presenting the position of the corresponding lesion part in the coordinate system of the target image from the position of the target lesion part in the coordinate system of the reference image, the operator Can search for a corresponding lesion using the estimated position as a clue.

  Therefore, by measuring the position and orientation of the ultrasonic probe, the relationship of the coordinate system between the ultrasonic tomographic image that is the target image and the reference image is obtained, and the coordinate system of the ultrasonic tomographic image (ultrasonic coordinate system) Attempts have been made to guide the operation of the probe by estimating the position of the corresponding lesion in. For example, for the current ultrasonic tomographic image, the distance and direction to the center of the target (target lesion area) set in the reference image (cross-sectional image of a three-dimensional image such as an X-ray CT apparatus or MRI apparatus) is calculated. Patent Document 1 discloses displaying a three-dimensional arrow image and a numerical value based on the distance and direction. Thereby, since the operator can visually grasp the distance from the current ultrasonic tomographic image to the target, it becomes easy to grasp the correspondence (positional relationship) between the reference image and the ultrasonic tomographic image.

  In addition, when an image tracking point (target lesion) selected from a past ultrasonic tomographic image (volume or slice) is given, a square of a size or color based on the distance and direction from the current ultrasonic tomographic image is displayed. Patent Document 2 discloses that the current ultrasonic tomographic image is displayed as an in-plane indicator. Thus, when counting the number of nodules in the thyroid gland, the number of metastases in the liver, etc., even if the angle and position of the probe are changed, the currently visualized structure is newly identified, or It can be determined whether it has already been identified and counted.

JP 2008-246264 A JP 2008-212680 A

By the way, the measurement accuracy of the position and orientation of the ultrasonic probe is not perfect, and the shape of the subject at the time of reference image capturing and ultrasonic tomographic image capturing does not necessarily match. Therefore, an error is included in the estimated position of the corresponding lesion in the coordinate system of the ultrasonic tomographic image, and the position is shifted from the actual position of the corresponding lesion.
However, the positional deviation is not considered in the instructions based on the distance and direction disclosed in Patent Documents 1 and 2. For this reason, even if the above instruction is taken into account, the user may not be able to identify (cannot find) the actual corresponding lesion depending on the degree of positional deviation. In this case, the user eventually searches for the actual corresponding lesion from the entire ultrasonic tomographic image, and the search efficiency is poor.

An information processing apparatus according to the present invention includes:
3D image acquisition means for acquiring a 3D image of the subject;
Display control means for displaying an error range by the projection including a projection area obtained by projecting a predetermined area of the three-dimensional image on a plane including the two-dimensional image of the subject on the two-dimensional image and displaying on the display means; ,
It is characterized by having.

  According to the present invention, a predetermined region (for example, a region of interest or the like) of a three-dimensional image (for example, an MRI image or an X-ray CT image) on a plane including a two-dimensional image (for example, an ultrasonic image) of a subject. The target lesion area) can be projected, and an error range by the projection including the projected projection area (a range where a corresponding area on the two-dimensional image corresponding to the attention area can exist, also referred to as a search range). It can be displayed on the display means by being superimposed on the two-dimensional image. As a result, the user knows the search range for searching for the actual corresponding area on the two-dimensional image, and can thus efficiently search for and identify the actual corresponding area.

It is a flowchart which shows the process sequence of the information processing apparatus which concerns on 1st Embodiment. It is a figure which shows the apparatus structure of the information processing apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the process sequence of the information processing apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the process sequence of the image composition in the search mode of the information processing apparatus which concerns on 2nd Embodiment. It is a figure for demonstrating the display method which displays the tomographic image and presence range of the information processing apparatus which concerns on 1st and 2nd embodiment. It is a figure which shows the example of a 1st area | region and a 2nd area | region. It is a figure which shows the apparatus structure of the information processing apparatus which concerns on 3rd Embodiment. It is a figure which shows the apparatus structure of the information processing apparatus which concerns on 4th Embodiment. It is a flowchart which shows the process sequence of the information processing apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the process sequence of the information processing apparatus which concerns on 4th Embodiment. It is a figure which shows the apparatus structure of the information processing apparatus which concerns on 5th Embodiment. It is a flowchart which shows the process sequence of the information processing apparatus which concerns on 5th Embodiment. It is a figure which shows the positional information on the ultrasonic tomographic image in a three-dimensional existence range. It is a figure which shows matching with the color table of relative position information. It is a figure which shows presence range display information. It is a figure which shows the ultrasonic tomographic image on which the existence range display information was superimposed and displayed. It is a figure which shows the basic composition of the computer which can implement | achieve each part of the information processing apparatus which concerns on this embodiment with software.

  Hereinafter, preferred embodiments of an information processing system (information processing apparatus) according to the present invention will be described in detail with reference to the accompanying drawings. However, the scope of the invention is not limited to the illustrated examples.

(First embodiment: error)
The information processing apparatus according to the present embodiment has a predetermined region (for example, an MRI image or an X-ray CT image) on a plane including a two-dimensional image (for example, an ultrasonic image) of a subject. , A region of interest and a lesion part of interest) can be projected, and an error range by the projection including the projected projection region (a range where a corresponding region on the two-dimensional image corresponding to the region of interest can exist, a search range) Can be superimposed on the two-dimensional image and displayed on the display means. As a result, the user knows the search range for searching for the actual corresponding area on the two-dimensional image, and can thus efficiently search for and identify the actual corresponding area.

  The present embodiment is a technique based on the above projection. That is, when the projection is performed, if there is no error due to the projection, a region corresponding to a predetermined region is continuously displayed on the two-dimensional image of the subject. However, an area corresponding to a predetermined area may not be displayed on the two-dimensional image of the subject due to, for example, an error due to coordinate conversion described later. Therefore, by displaying this error range on the two-dimensional image, the user can easily search for a region corresponding to the predetermined region.

  Here, the position of the predetermined region may be coordinate-transformed (associated or aligned) from the 3D coordinate system of the 3D image to the 3D coordinate system of the 2D image (for example, the reference coordinate system of the position and orientation sensor). preferable. Thus, the error range due to the projection can be calculated based on the error due to coordinate conversion. Further, after setting a region (second region) larger than the predetermined region (first region) and including the predetermined region in the three-dimensional image, the position of this region (second region) is By converting the coordinates, the error range due to the projection can be calculated. At this time, the size of the second region is preferably set based on an error in coordinate conversion. For example, as shown in FIG. 6A, when the first area 701 is defined as a point having no size, a sphere having the first area 701 as the center and a coordinate conversion error as a radius. Can be the second region 702. In addition, as shown in FIG. 6B, when the first region 703 is defined as a closed region having a size, the error in coordinate transformation is the distance to the nearest point of the first region 703. Such a closed curved surface may be obtained, and the inside thereof may be used as the second region 704.

  Next, the information processing system according to the present embodiment will be described with reference to FIG. FIG. 1 is a flowchart showing a processing procedure of the information processing apparatus according to the present embodiment.

  First, in S <b> 1, the error acquisition unit 123 inputs error factor information to the information processing apparatus 100. For example, the distance between the ultrasonic probe and the position and orientation sensor is input as error factor information. In general, as the distance increases, an error due to the projection (for example, coordinate conversion) increases.

  Next, in S2, the tomographic image acquisition unit 110 acquires a tomographic image (for example, an ultrasonic tomographic image).

  In S3, the error acquisition unit 123 acquires an error (error due to the projection). For example, when the error factor information acquired in S1 is the distance between the ultrasound probe and the position / orientation sensor, S is stored in accordance with a previously stored table (hereinafter also referred to as information for calculating an error). The error can be obtained using the distance.

  Further, in S4, a range (an error range due to the above error, a search range) in which the region of interest (target lesion portion) can exist is acquired. Specifically, an error range by the projection including a projection area obtained by projecting a predetermined area of the three-dimensional image on a plane including the ultrasonic tomographic image is acquired. At this time, for example, a circle whose center is the projection area and whose radius is the distance obtained as an error is drawn, and an area where the circle overlaps the ultrasonic tomographic image is acquired.

  In S5, the overlapping area is displayed on the display unit so as to overlap the ultrasonic tomographic image. For example, an image as shown in FIG. 5A is displayed on the display means. At this time, it is preferable to display a range where the corresponding lesion can exist in a circle, put a translucent mask on the outside of the circle, and display the inner ultrasonic tomographic image as usual. Thereby, a search area for searching for an actual attention area is clarified. In addition, since the user can easily search the actual corresponding area, the search can be performed efficiently. Further, the range in which the corresponding area can exist may be drawn with only a circle line. In addition, when a color is added to the inside of the range where the corresponding region can exist, it is preferable to add a color that is transparent enough to be searched.

  In FIG. 5A, the user can designate the position of a predetermined location as the position of the corresponding region within the error range due to the projection on the ultrasonic tomographic image. Thereby, for example, a mark (for example, a circle or a cross) indicating that it is a corresponding region (a region on the two-dimensional image corresponding to the predetermined region) is superimposed on a predetermined portion on the two-dimensional image on the display means. Can be displayed. For this reason, since the position of the corresponding region on the two-dimensional image is easily understood, the diagnostic efficiency is improved. Further, it is preferable to correct an error due to projection in the two-dimensional image based on the difference between the designated predetermined location and the projection area. Thereby, the mark can be continuously displayed at a location corresponding to the predetermined location on the two-dimensional image acquired again.

  In FIG. 5A, when the user does not designate the position of a predetermined location, ultrasonic tomographic images are repeatedly acquired. At this time, the size of the error range due to the projection changes according to the position and orientation of the ultrasonic tomographic image. At this time, when the predetermined area (first area) and the projection area (area on the ultrasonic tomographic image corresponding to the first area) are at the same position (the scan plane of the ultrasonic tomographic image is When passing through a predetermined region), the size of the error range due to the projection is maximized. As shown in FIG. 5B, the error range 804 having the maximum size may be always displayed on the ultrasonic image 503 so that it can be distinguished from the actual error range 502.

(Second Embodiment: Search Mode and Linkage Mode)
The information processing system according to the present embodiment displays an area in which a region of interest (for example, a lesion of interest) in 3D image data may exist in an ultrasonic tomographic image captured in real time. By doing so, the operator (doctor or engineer) can easily draw the corresponding region (corresponding lesion) corresponding to the region of interest in the three-dimensional image data on the ultrasonic tomographic image. In the present embodiment, a case will be described in which a tomographic image group representing three-dimensional information inside a subject is handled as three-dimensional image data. The information processing system according to this embodiment will be described below.

  FIG. 2 shows a configuration of the information processing system according to the present embodiment. As shown in the figure, the information processing apparatus 100 according to the present embodiment includes a tomographic image acquisition unit (also referred to as a two-dimensional image acquisition unit) 110, a position and orientation acquisition unit 112, and a three-dimensional image data acquisition unit (three-dimensional image acquisition). 120, a region of interest acquisition unit 122, an error acquisition unit 123, a cross-sectional image generation unit (also referred to as a cross-sectional image acquisition unit) 130, an existence range calculation unit 135, an image composition unit 140, and a display control unit 150. Is done. It is connected to a data server 190 that holds 3D image data and error factor information described later. The information processing apparatus 100 is also connected to an ultrasonic image diagnostic apparatus as the second medical image acquisition apparatus 180 that captures an ultrasonic tomographic image of a subject.

(Input of 3D image data)
The three-dimensional image data held by the data server 190 is a reference tomographic image group obtained by imaging a subject in advance by an MRI apparatus or an X-ray CT apparatus as the first medical image acquisition apparatus 170. In the following, a case where an MRI apparatus is used as the first medical image collection apparatus 170 is described as an example, and a human breast is described as an object to be imaged.

  The position and orientation of each tomographic image that constitutes the reference tomographic image group captured by the MRI apparatus as three-dimensional image data is represented by the position and orientation in the MRI apparatus coordinate system. Here, the MRI apparatus coordinate system represents a coordinate system in which one point in the space with the MRI apparatus as a reference is defined as the origin. The 3D image data expressed in the MRI apparatus coordinate system is acquired by the 3D image data acquisition unit 120 and input to the information processing apparatus 100. Note that the three-dimensional image data acquisition unit 120 generates three-dimensional volume data in which luminance values are stored in three-dimensional voxels from the reference tomographic image group, and holds this. Then, in accordance with a request from the cross-sectional image generation unit 130, the held three-dimensional volume data is output to the cross-sectional image generation unit 130.

  Furthermore, the data server 190 holds the position of a lesion (designated lesion) that is designated in advance as a region of interest in the 3D image data. For example, the operator can sequentially display a reference tomographic image group on an image viewer (not shown), select a tomographic image in which the target lesion is shown, and select the target lesion with a mouse (not shown). It can be specified by clicking. The position of the attention lesion part held by the data server 190 is acquired by the attention area acquisition unit 122 and input to the information processing apparatus 100. In addition, the attention area acquisition unit 122 outputs the position of the attention lesion part that is held to the error acquisition unit 123, the existence range calculation unit 135, and the image composition unit 140. In the following description, it is assumed that the position of the lesion of interest is also expressed in the MRI apparatus coordinate system in the same manner as the three-dimensional image data.

(Acquire error estimate)
In addition, the data server 190 holds information for calculating an error in the position of the lesion in the 3D image data. Here, the information for calculating the error is a projection including how much the position of the corresponding region (corresponding lesion) corresponding to the target lesion in the three-dimensional image data on the ultrasonic tomographic image includes. This is information for calculating whether or not to be performed. In other words, it is information for calculating an alignment error (existing range of the corresponding lesion) between the three-dimensional image data and the ultrasonic tomographic image. In the present embodiment, the position / orientation acquisition unit 112 calculates the position and orientation of the ultrasonic tomographic image in the MRI apparatus coordinate system, so that the alignment between the three-dimensional image data and the ultrasonic tomographic image is performed. In the following description, information for calculating the error estimated value is referred to as error factor information (details will be described later). The error factor information held by the data server 190 is input to the information processing apparatus 100 via the error acquisition unit 123. In addition, the error acquisition unit 123 calculates an error estimation value for alignment between the three-dimensional image data and the ultrasonic tomographic image (existing range of the corresponding lesion) based on the error factor information. Then, the calculated error estimation value is output to the existence range calculation unit 135. At this time, the existence range calculation unit 135 calculates a circle obtained by cutting out a sphere having an error in radius around the position of the lesion of interest with an ultrasonic cross section, and serves as information indicating the existence range of the corresponding lesion. The image is output to the image composition unit 140.

(Acquisition of tomographic images)
The ultrasonic diagnostic imaging apparatus as the second medical image acquisition apparatus 180 captures an ultrasonic tomographic image of a subject in real time. The ultrasonic tomographic image is acquired by the tomographic image acquisition unit 110 and is sequentially input to the information processing apparatus 100. In addition, the tomographic image acquisition unit 110 converts the acquired ultrasonic tomographic image into digital data as necessary, and stores it in association with the position and orientation acquired from the position and orientation acquisition unit 112. Then, in accordance with a request from the image composition unit 140, the held ultrasonic tomographic image is output to the image composition unit 140.

  Usually, the operator holds an ultrasonic probe as an imaging unit (not shown) of the ultrasonic diagnostic imaging apparatus and picks up an image of the subject while moving it freely. An ultrasonic tomographic image can be acquired by detecting an ultrasonic wave with an ultrasonic probe. At this time, it is not clear which position and orientation in the space the ultrasonic tomographic image is based on the subject. Therefore, in the present embodiment, a position / orientation sensor (not shown) is attached to the ultrasonic image diagnostic apparatus, and the position / orientation of the ultrasonic probe is measured. The position / orientation sensor is configured by, for example, FASTRAK of Polhemus, USA. The position / orientation sensor may be configured in any way as long as the position / orientation of the ultrasonic probe can be measured.

  The position and orientation of the ultrasound probe obtained as described above are acquired by the position and orientation acquisition unit 112 and input to the information processing apparatus 100. Here, the position and orientation of the ultrasonic probe are represented by a position and orientation in the reference coordinate system, for example. Here, the reference coordinate system represents a coordinate system in which one point in the space with the subject as a reference is defined as the origin. The position and orientation acquisition unit 112 acquires the position and orientation of the ultrasonic probe in the reference coordinate system, and calculates the position and orientation of the ultrasonic tomographic image in the MRI apparatus coordinate system based on the acquired position and orientation. Then, in accordance with a request from the cross-sectional image generation unit 130, the calculated position and orientation are output to the cross-sectional image generation unit 130. Note that this calculation process is based on a known coordinate transformation based on the relative positional relationship between the ultrasonic probe and the ultrasonic tomographic image and the relative positional relationship between the reference coordinate system and the MRI apparatus coordinate system. It is executed by calculation. In the present embodiment, the relative positional relationship information (hereinafter referred to as calibration data) is derived in advance by a known calibration method and is held as a known value in the memory in the position / orientation acquisition unit 112. To do.

(Cross section image generation)
The cross-sectional image generation unit 130 uses the 3D volume data output from the 3D image data acquisition unit 120, the position and orientation of the ultrasonic tomographic image output from the position and orientation acquisition unit 112, and the output of the attention area acquisition unit 122. Enter the position of a certain lesion. Based on these data, a cross-sectional image (second two-dimensional cross-sectional image) is generated from the three-dimensional volume data and output to the image composition unit 140. The cross-sectional image generation unit 130 performs different processes according to two operation modes described later. At this time, the image composition unit 140 acquires information indicating the existence range (second region) of the corresponding lesion part from the existence range calculation unit 135 and superimposes it on the ultrasonic tomographic image acquired from the tomographic image acquisition unit 110. And draw. Further, a composite image obtained by combining the image and the cross-sectional image acquired from the cross-sectional image generation unit 130 is output to the display control unit 150 or the outside. Note that an image obtained by superimposing the existence range of the corresponding lesion on the ultrasonic tomographic image and an image obtained by superimposing a predetermined region (first region) on the cross-sectional image may be displayed side by side on the display unit. In addition, the display control unit 150 acquires a composite image that is an output of the image composition unit 140 and displays the composite image on the display unit 160.

  2 (tomographic image acquisition unit 110, position and orientation acquisition unit 112, three-dimensional image data acquisition unit 120, attention area acquisition unit 122, error acquisition unit 123, cross-sectional image generation unit 130, existence range calculation unit 135, the image composition unit 140, and the display control unit 150) may be realized as independent devices. Alternatively, it may be implemented as software that implements its function by installing it on one or a plurality of computers and executing it by the CPU of the computer. In the present embodiment, each unit is realized by software and installed in the same computer.

(Basic computer configuration)
FIG. 17 illustrates a tomographic image acquisition unit 110, a position / orientation acquisition unit 112, a 3D image data acquisition unit 120, a region of interest acquisition unit 122, an error acquisition unit 123, a cross-sectional image generation unit 130, an existence range calculation unit 135, and an image synthesis unit. 140 is a diagram illustrating a basic configuration of a computer for realizing the functions of the display control unit 150 by executing software. FIG.

The CPU 1001 controls the entire computer using programs and data stored in the RAM 1002 and the ROM 1003. In addition, the tomographic image acquisition unit 110, the position / orientation acquisition unit 112, the 3D image data acquisition unit 120, the attention area acquisition unit 122, the error acquisition unit 123, the cross-sectional image generation unit 130, the existence range calculation unit 135, the image composition unit 140, The execution of software in each of the display control units 150 is controlled to realize the function of each unit.
The RAM 1002 includes an area for temporarily storing programs and data loaded from the external storage device 1007 and the storage medium drive 1008, and a work area required for the CPU 1001 to perform various processes.

  The ROM 1003 generally stores computer programs and setting data. A keyboard 1004 and a mouse 1005 are input devices, and an operator can input various instructions to the CPU 1001 using these devices.

  The display unit 1006 includes a CRT, a liquid crystal display, or the like, and the display unit 160 corresponds to this. The display unit 1006 can display a message to be displayed for image processing, a GUI, and the like in addition to the composite image generated by the image composition unit 140.

  The external storage device 1007 is a device that functions as a large-capacity information storage device such as a hard disk drive, and stores an OS (operating system), a program executed by the CPU 1001, and the like. In the description of the present embodiment, information that is described as being known is stored here, and loaded into the RAM 1002 as necessary.

  The storage medium drive 1008 reads a program or data stored in a storage medium such as a CD-ROM or DVD-ROM in accordance with an instruction from the CPU 1001 and outputs it to the RAM 1002 or the external storage device 1007.

  The I / F 1009 includes an analog video port or a digital input / output port such as IEEE 1394, an Ethernet (registered trademark) port for outputting information such as a composite image to the outside, and the like. The data input by each is taken into the RAM 1002 via the I / F 1009. Some of the functions of the tomographic image acquisition unit 110, the position / orientation acquisition unit 112, the 3D image data acquisition unit 120, the attention area acquisition unit 122, and the error acquisition unit 123 are realized by the I / F 1009.

  The above-described components are connected to each other by a bus 1010.

(Search mode and linked mode)
The above-described information processing system has two modes as operation modes: a search mode (a mode for searching for a corresponding lesion site) and a linked mode (a mode for observing and confirming the associated lesion site). Given the position and orientation of the ultrasonic tomographic image and the position of the lesion of interest, a cross-sectional image corresponding to the mode is generated (acquired) from the three-dimensional image data of MRI based on these pieces of information. The cross-sectional image generated in the search mode is a cross-sectional image parallel to the plane including the above-described ultrasonic tomographic image and a cross-sectional image passing through the lesion of interest. By generating such a cross-sectional image, the inclination (posture) of two cross-sectional images (an ultrasonic tomographic image and an MRI cross-sectional image) with respect to the subject can be obtained regardless of the posture of the ultrasonic probe. You can always display them aligned. As a result, the operator only needs to match the position where the ultrasonic probe is pressed with the MRI data. Since it is possible to save the trouble of aligning the tilts, it is possible to easily perform alignment by the operator. Switching between the search mode and the interlocking mode is performed by specifying the corresponding lesion in the ultrasonic tomographic image and pointing out its position. In the interlock mode, an image obtained by cutting out the same cross section as the ultrasonic tomographic image from the three-dimensional image data of MRI is displayed as an MRI cross-sectional image. In this mode, it is possible to observe the spread of the lesion while simultaneously viewing the MRI cross-sectional image around the lesion of interest linked to the position and orientation of the ultrasound probe and the ultrasound tomographic image.

  The ultrasonic tomographic image and the MRI cross-sectional image are displayed on the display unit 160, respectively. In the search mode, the operator can perform alignment by comparing whether or not the contents shown in the respective images match while changing the position where the ultrasonic probe is pressed. In the interlocking mode, it is possible to observe the periphery of the lesion while viewing each image that has been aligned.

  FIG. 3 is a flowchart illustrating an overall processing procedure performed by the information processing apparatus 100. In the present embodiment, the flowchart is realized by the CPU 1001 executing a program that realizes the function of each unit. It is assumed that the program code according to the flowchart is already loaded from, for example, the external storage device 1007 to the RAM 1002 in the previous stage of performing the following processing.

(S3000: Data input)
In step S3000, the information processing apparatus 100 acquires a reference tomographic image group as 3D image data from the data server 190 as processing of the 3D image data acquisition unit 120. Then, from the reference tomographic image group, each pixel of each tomographic image is three-dimensionally arranged and interpolated to generate three-dimensional volume data in which luminance values are stored in three-dimensional voxels. Further, as the processing of the attention area acquisition unit 122, the position of the attention lesion part is acquired from the data server 190.

(S3005: Input of error factor information)
In step S3005, the information processing apparatus 100 acquires, from the data server 190, various error factor information (described later) used for calculation of the error estimated value as processing of the error acquisition unit 123.

(S3010: Acquisition of tomographic image)
In step S <b> 3010, the information processing apparatus 100 acquires an ultrasonic tomographic image from the second medical image acquisition apparatus 180 as processing of the tomographic image acquisition unit 110. Further, as the processing of the position / orientation acquisition unit 112, the position / orientation of the ultrasonic probe when the ultrasonic tomographic image is captured is acquired from the second medical image acquisition device 180. Then, the position and orientation of the ultrasonic tomographic image in the MRI apparatus coordinate system are calculated from the position and orientation of the ultrasonic probe in the reference coordinate system using the calibration data stored in advance as known values. Further, in the case where the alignment correction parameter is held, the calculated value of the position of the ultrasonic tomographic image is corrected by the correction parameter, thereby accurately aligning the target lesion portion and the corresponding lesion portion. Note that the estimation of the relative relationship between the lesion of interest (region of interest) and the tomographic image is realized by acquiring the position of the lesion of interest in step S3000 and calculating the position and orientation of the ultrasonic tomographic image in this step.

(S3015: Acquisition of estimated error value)
In step S3015, the information processing apparatus 100 calculates an error estimated value based on various data used for calculation of the error acquired in step S3005 as processing of the error acquisition unit 123.

  In the present embodiment, the relationship between the MRI apparatus coordinate system and the reference coordinate system is expressed by rigid body transformation. However, the position and orientation of the subject with respect to the MRI apparatus coordinate system when the MRI image is captured and the position and orientation of the subject with respect to the reference coordinate system when performing ultrasonic imaging are not necessarily in a rigid body transformation relationship. Absent. Therefore, an error may be mixed when the relationship between the coordinate systems is expressed by rigid transformation. Further, even if the difference in position and orientation of the subject in the two coordinate systems is exactly related to rigid transformation, it is practically difficult to obtain the rigid transformation correctly, and errors are still mixed. There is a case. In addition, the error includes an error (position and orientation measurement error) mixed in the position and orientation measurement value acquired by the position and orientation acquisition unit 112.

  The process of calculating the error estimation value can be executed based on, for example, the characteristics of a position and orientation sensor that measures the position and orientation of the ultrasonic probe of the second medical image acquisition apparatus 180. For example, an error reference value can be determined in advance for each measurement method of the position and orientation sensor, and a value can be selected according to the measurement method of the sensor to be used. For example, since the measurement accuracy of a magnetic sensor is generally lower than that of an optical sensor, information indicating that the optical sensor is used is acquired from the data server 190 as error factor information, and the magnetic sensor is used based on the information. The estimated error value can be calculated as a small value compared to the case where Further, the error estimation value is not limited to the difference in the measurement method of the position / orientation sensor, and can be calculated based on the relationship between the spatial position and orientation with respect to the measurement reference of the position / orientation sensor. For example, when the position / orientation sensor is composed of a magnetic sensor, an estimated error value is defined as a function of the distance between the magnetism generation device that is a measurement reference and the ultrasonic probe, and the distance is large. In this case, a large value can be calculated as the error. Even when the position / orientation sensor is constituted by an optical sensor, the error can be calculated based on the distance or orientation of the ultrasonic probe with respect to the measurement reference.

  In addition, the error estimation value can be calculated based on a part or the like in the subject where the lesion of interest exists. For example, when the lesion of interest is present in a soft tissue such as the breast of the subject, it is assumed that deformation occurs in the relevant part of the subject between the time when the MRI image is captured and the time when the ultrasound image is captured. Therefore, when the target lesion part exists in such a part, the error estimated value can be calculated large. Similarly, a large error estimate can be calculated in the heart and a region in the vicinity of the heart where the position variation due to the heartbeat is large, or in the lung and the region near the lung where the position variation due to respiration is large. Specifically, information indicating a part (organ name or position in the organ) in the subject where the target lesion is present and data (table) describing correspondence between the part and the magnitude of the error are error factor information. And an error estimation value can be calculated based on these pieces of information.

  Further, information on the position of the part used as an index when the MRI apparatus coordinate system and the reference coordinate system are calibrated is acquired as error factor information, and based on the positional relationship between these parts and the target lesion part. An error estimated value can be calculated. For example, if the index used for the calibration is the xiphoid process of the subject, an error estimation value is calculated as a function of the distance between the position of the xiphoid process in the MRI image and the lesion of interest. it can. Similarly, when the position of the nipple is used as an index, an error estimated value can be calculated as a function of the distance between the nipple and the lesion of interest in the MRI image. In addition, an error estimation value may be acquired by performing multiple error estimations using a plurality of the methods exemplified above.

(S3020: Determination of whether or not in search mode)
In step S3020, the information processing apparatus 100 determines whether the current operation mode is the search mode or the interlock mode. If the operation mode is the search mode, the process proceeds to step S3030. If the operation mode is the interlock mode, the process proceeds to step S3070. Note that the operation mode in the initial state of the present embodiment is the search mode.

(S3030: Generation and display of image in search mode)
In step S3030, the information processing apparatus 100 generates and displays an image in the search mode. Details of the processing in this step will be described later in detail using the flowchart shown in FIG.

  Next, when the operator designates the position of the corresponding lesion on the ultrasonic tomographic image by the following processing (steps S3040 and S3050), the information processing apparatus 100 determines the position of the target lesion and the corresponding lesion. Correction of deviation from the actual position is performed.

(S3040: designation of position of corresponding lesion on tomographic image)
In step S3040, the information processing apparatus 100 determines whether or not the position of the corresponding lesion on the ultrasonic tomographic image has been designated as the processing of the position / orientation acquisition unit 112. For example, the position of the corresponding lesion part is designated by the operator clicking the position of the corresponding lesion part on the ultrasonic tomographic image displayed on the display unit 160 with the mouse 1005. When the position of the corresponding lesion is designated, the position of the corresponding lesion in the MRI apparatus coordinate system is calculated based on the position of the corresponding lesion on the ultrasonic tomographic image and the position and orientation of the ultrasonic tomographic image. . Then, the process proceeds to step S3050. On the other hand, if the position is not designated, the process proceeds to step S3100. For example, if the corresponding lesion is not displayed on the ultrasonic tomographic image, the user acquires the ultrasonic tomographic image again in S3010 without designating the position.

(S3050: Calculation of correction value)
In step S3050, the information processing apparatus 100 performs an offset (correction value) between the position of the corresponding lesion part acquired in step S3040 and the position of the target lesion part acquired in step S3000 as processing of the position / orientation acquisition unit 112. Is calculated. This value is stored in the memory as a correction parameter for alignment. If the position / orientation acquisition unit 112 holds the correction parameter, the position / orientation acquisition unit 112 converts the ultrasonic tomographic image newly acquired in S3010 to the MRI apparatus coordinate system, and converts the ultrasonic tomographic image in the converted coordinate system. The position is corrected (subtracted) by the correction parameter (offset). Thereby, it is possible to correct the influence of the measurement error of the position and orientation sensor and the deformation of the subject.

(S3060: Switching from search mode to interlocking mode)
In step S3060, the information processing apparatus 100 switches the operation mode of the system from the search mode to the interlock mode, and advances the process to step S3100. In this mode, the corrected ultrasonic tomographic image can be continuously acquired without going through the step of specifying the position from the ultrasonic tomographic image. For example, when the user needs to know the size of a lesion to be excised by surgery, the user may want to observe the extent of the lesion. In this case, the periphery of the corresponding lesion can be observed with the corrected ultrasonic tomographic image, which is convenient.

(S3070: Generation and display of image in linked mode)
In step S3070, the information processing apparatus 100 generates and displays an image in the interlock mode. Specifically, as the processing of the cross-sectional image generation unit 130, based on the position and orientation of the ultrasonic tomographic image obtained in step S3010, a cross-sectional image obtained by cutting out the same cross section as the tomographic image from the three-dimensional volume data is generated. Then, as the processing of the image composition unit 140, the ultrasonic tomographic image obtained in step S3010 and the cross-sectional image obtained in step S6030 are synthesized. For example, an image in which these images are arranged side by side is generated. At this time, when the position of the lesion of interest is included in the plane of the cross-sectional image, a rectangular mark or the like representing the lesion of interest is superimposed and drawn at a corresponding position in the ultrasonic tomographic image and the slice image. . Further, as a process of the display control unit 150, the synthesized image is displayed on the display unit 160.

(S3080: Determination of whether or not to switch from the interlock mode to the search mode)
In step S3080, the information processing apparatus 100 determines whether to switch the system operation mode from the interlock mode to the search mode. For example, the switching instruction input by the operator pressing a predetermined key (operation mode switching key) on the keyboard 1004 is acquired. If it is determined that the operation mode is to be switched, the process proceeds to step S3090. If it is determined that the operation mode is not to be switched, the process proceeds to S3100. For example, when an image similar to the image of the corresponding lesion is displayed, there is a possibility that the user designates a region different from the actual corresponding lesion in S3040. In such a case, the user observes the periphery of the corresponding lesioned part by using the ultrasonic tomographic image acquired again in the interlock mode. If the user notices the actual corresponding lesioned part, the user switches to the search mode in S3080, The position of the corresponding lesion can be reacquired.

(S3090: Switch from linked mode to search mode)
In step S3090, the information processing apparatus 100 switches the operation mode of the system from the interlock mode to the search mode, and advances the process to step S3100.

(S3100: Determination of whether to end the entire process)
In step S3100, the information processing apparatus 100 determines whether to end the entire process. For example, an end instruction input by an operator pressing a predetermined key (end key) on the keyboard 1004 is acquired. When it is determined that the processing is to be ended, the entire processing of the information processing apparatus 100 is ended. On the other hand, if it is not determined that the process is to be terminated, the process returns to step S3010, and the processes after step S3010 are executed again on the newly acquired ultrasonic tomographic image.

  As described above, the process of the information processing apparatus 100 is executed.

(Generation and display of images in search mode)
Next, details of the display image generation processing in the search mode performed by the information processing apparatus 100 in step S3030 will be described with reference to the flowchart of FIG.

(S4000: Acquisition of cross-sectional image)
In step S4000, the information processing apparatus 100 performs processing by the cross-sectional image generation unit 130 on the three-dimensional volume data and the position of the lesion of interest obtained in step S3000 and the position and orientation of the ultrasonic tomographic image obtained in step S3010. Based on this, a cross-sectional image in the search mode is generated.

  First, the cross-sectional image generation unit 130 calculates a cross-section (plane) based on the position of the target lesion and the posture of the ultrasonic tomographic image. Specifically, first, the position and orientation of the cross-sectional coordinate system (coordinate system representing the position and orientation of the cross section) are initialized. Next, the cross section is rotated so that the posture of the cross section matches the posture of the ultrasonic tomographic image. Then, the cross section is moved in parallel so that the target lesion is located on the cross section. The cross section calculated as described above includes the lesion of interest in the plane (that is, the plane representing the cross section passes through the lesion of interest) and has the same posture as the ultrasound tomographic image (ultrasound tomographic image). (In parallel). Finally, a range for generating a cross-sectional image on the cross-section is calculated. For example, the positions of the four corner points of the ultrasonic tomographic image are calculated based on the position and orientation of the ultrasonic tomographic image, and the generated cross-sectional image is composed of four points consisting of perpendicular feet drawn from the respective points to the cross section. Determine the range. As a result, even if the posture of the ultrasonic tomographic image is moved, the posture of the cross-sectional image also moves in conjunction with each other, so that the user can easily observe.

  Finally, the cross-sectional image generation unit 130 generates an image corresponding to the cross section obtained above by cutting it out from the three-dimensional volume data. Note that a method for cutting out and generating a specified cross-sectional image from the three-dimensional volume data is well known, and a detailed description thereof will be omitted.

(S4010: Acquisition of the presence range of the target lesion)
In step S4010, the information processing apparatus 100 calculates the existence range of the corresponding lesion portion on the ultrasonic tomographic image obtained in step S3010 as the processing of the existence range calculation unit 135. In the present embodiment, the existence range of the corresponding lesion portion in the three-dimensional space is defined as a sphere centered on the position of the target lesion obtained in step S3000 and the error estimated value obtained in step S3015 is a radius. In addition, the existence range of the corresponding lesion part on the ultrasonic tomographic image is defined as a circle that is a region (cross section of the sphere) where the sphere representing the existence range of the corresponding lesion part in the three-dimensional space and the tomographic image intersect. The Therefore, the image composition unit 140 calculates the center position and radius of this circle on the ultrasonic tomographic image. In addition, since the calculation method of the intersection area | region of the sphere and plane defined in three-dimensional space is a well-known thing, the description is abbreviate | omitted. If the sphere representing the existence range and the ultrasonic tomographic image do not intersect, information that “there is no existence range on the cross section” is stored.

(S4020: Draw the existence range of the lesion of interest on the tomographic image)
In step S4020, the information processing apparatus 100 superimposes and draws information representing the existence range of the corresponding lesion on the ultrasonic tomographic image calculated in step S4010 on the ultrasonic image as processing of the image synthesis unit 140. To do. At this time, it is preferable to display a range where the corresponding lesion can exist in a circle, put a translucent mask on the outside of the circle, and display the inner ultrasonic tomographic image as usual. As a result, a search area for searching for an actual corresponding lesion is clarified. In addition, since the user can easily search for an actual corresponding lesion, it can be searched efficiently. Further, the range in which the corresponding lesioned part may exist may be drawn using only a circle line. In addition, when coloring inside the range where the corresponding lesion can exist, it is preferable to add a color that is transparent enough to be searched. As a result of the processing in this step, an ultrasonic tomographic image 503 in which the existence range 502 of the corresponding lesion 501 is superimposed on the tomographic image as shown in FIG. 5A is generated. If it is determined in step S4010 that “there is no existing area on the cross section”, the process in this step is not executed.

(S4030: Compositing a tomographic image and a cross-sectional image)
In step S4030, the information processing apparatus 100 performs processing by the image composition unit 140 as a cross-sectional image obtained in step S4000 and an ultrasonic tomographic image obtained in step S4020 (more precisely, the existence range of the corresponding lesion part is a tomographic image. An image obtained by synthesizing the image superimposed on the image is generated. For example, an image in which these images are arranged side by side is generated. Then, as a process of the display control unit 150, the synthesized image is displayed on the display unit 160. Further, if necessary, this is output to the outside via the I / F 1009 and further stored in the RAM 1002 as a state usable by other applications.

  As described above, a composite image of the cross-sectional image including the lesion of interest in the same posture as the ultrasound tomographic image and the ultrasound tomographic image in which the existence range of the corresponding lesion is drawn is presented to the operator.

  As described above, according to the information processing apparatus according to the present embodiment, the existence range of the corresponding lesion part in consideration of the alignment error is displayed on the ultrasonic tomographic image as a guide when the operator searches for the corresponding lesion part. Presented. As a result, the operator's unnecessary work of searching a wider range than necessary can be reduced, and the search work load can be reduced. In addition, since the search range can be limited, it is possible to reduce the risk of an operator performing an incorrect association.

  In the above-described embodiment, when the existence range of the corresponding lesion is superimposed and presented on the ultrasonic tomographic image, a cross-sectional image including the lesion of interest in the same posture as the ultrasonic tomographic image is obtained as an MRI cross-sectional image. Was presenting. However, the presented MRI cross-sectional image may be in other forms. For example, as in the interlocking mode of the present embodiment, a cross-sectional image that is interlocked with both the position and orientation of the ultrasonic tomographic image (the same cross section as the ultrasonic tomographic image is cut out) may be displayed. Further, the MRI cross-sectional image designated by the doctor may be displayed as a still image (without linking the posture). For example, an MRI cross-sectional image when the doctor points out the lesion of interest may be displayed. Further, the MRI cross-sectional image is not necessarily displayed.

  Moreover, when displaying the cross-sectional image interlock | cooperated with both the position and attitude | position of an ultrasonic tomographic image, the number of attention lesion parts may be plural. In this case, since the existence range of the corresponding lesion part in the three-dimensional space is defined as a sphere with respect to each target lesion part, an intersection region between each sphere and the ultrasonic tomographic image is obtained, and the corresponding lesion on the tomographic image is obtained. It may be displayed as a part existing range.

(Third embodiment)
The information processing system according to the present embodiment captures, in real time, a region where a corresponding region (corresponding lesion) corresponding to a region of interest (for example, a target lesion) in 3D image data (reference image) can exist. Is displayed in the ultrasonic tomographic image (target image). By doing so, the operator (physician or engineer) can easily search and identify the corresponding region on the ultrasonic tomographic image. The information processing system according to this embodiment will be described below.

  FIG. 7 shows a configuration of the information processing system according to the present embodiment. As shown in the figure, an information processing apparatus 900 according to this embodiment includes a tomographic image acquisition unit (also referred to as a two-dimensional image acquisition unit) 910, a position and orientation acquisition unit 912, and a three-dimensional image data acquisition unit (three-dimensional image acquisition). 920, a region of interest acquisition unit 922, an error acquisition unit 923, a cross-sectional image generation unit (also referred to as a cross-sectional image acquisition unit) 930, an existence range calculation unit 935, an image composition unit 940, and a display control unit 950. Is done. It is connected to a data server 990 that holds 3D image data and error factor information described later. The information processing apparatus 900 is also connected to an ultrasonic image diagnostic apparatus as a second medical image collection apparatus 980 that captures an ultrasonic tomographic image of a subject.

(Acquisition of 3D image data)
The three-dimensional image data held by the data server 990 is an image obtained by imaging a subject in advance with an MRI apparatus or an X-ray CT apparatus as the first medical image acquisition apparatus 970. Hereinafter, a case where an MRI apparatus is used as the first medical image collection apparatus 970 will be described as an example. In this embodiment, it is assumed that the 3D image data is expressed as 3D volume data in which luminance values are stored in 3D voxels. The coordinates of each voxel are expressed in the MRI apparatus coordinate system. The 3D image data held by the data server 990 is acquired by the 3D image data acquisition unit 920 and input to the information processing apparatus 900.

(Obtain attention area)
Furthermore, the data server 990 holds information (details will be described later) representing a region of interest in the 3D image data. Information representing the attention area held by the data server 990 is acquired by the attention area acquisition unit 922 and input to the information processing apparatus 900. In the following description, it is assumed that the information indicating the region of interest is also expressed in the MRI apparatus coordinate system, like the three-dimensional image data.

(Acquisition of tomographic images)
The ultrasonic diagnostic imaging apparatus as the second medical image acquisition apparatus 980 captures an ultrasonic tomographic image of a subject in real time. The ultrasonic tomographic image is acquired by the tomographic image acquisition unit 910 and is sequentially input to the information processing apparatus 900. Further, the position and orientation of the ultrasonic probe are measured by a position and orientation sensor (not shown), acquired by the position and orientation acquisition unit 912, and input to the information processing apparatus 900. Here, the position and orientation of the ultrasonic probe are represented by the position and orientation in a reference coordinate system with the subject as a reference, for example. Further, the position / orientation acquisition unit 912 acquires the position / orientation of the ultrasonic probe in the reference coordinate system, and calculates the position / orientation of the ultrasonic tomographic image in the MRI apparatus coordinate system based on the acquired position / orientation.

(Calculation of existence range)
The existence range calculation unit 935 generates a region (corresponding to the attention region) based on the information indicating the attention region acquired by the attention region acquisition unit 922 and the position and orientation of the ultrasonic tomographic image acquired by the position and orientation acquisition unit 912. Region) is estimated on the ultrasonic coordinate system. Here, the ultrasonic coordinate system is a three-dimensional coordinate system based on the ultrasonic tomographic image. For example, the x-axis and the y-axis are set on the plane of the tomographic image with one point on the tomographic image as the origin. It can be defined as a coordinate system in which the z axis is set in a direction perpendicular to the plane. Then, based on the estimated corresponding area and an error estimated value acquired by an error acquisition unit 923 described later, an existence range (second area) of the corresponding area on the ultrasonic tomographic image is calculated.

(Acquire error estimate)
In addition to the information described above, the data server 990 holds information (error factor information; details will be described later) for calculating an estimated error value of the corresponding region. Here, the error factor information is, in other words, information for calculating the existence range of the corresponding region on the ultrasonic tomographic image. The error factor information held by the data server 990 is input to the information processing apparatus 900 via the error acquisition unit 923. The error acquisition unit 923 calculates an error estimation value for the corresponding region based on the acquired error factor information. Then, the calculated error estimation value is output to the existence range calculation unit 935.

(Cross section image generation)
The cross-sectional image generation unit 930 receives the 3D volume data that is the output of the 3D image data acquisition unit 920 and the position and orientation of the ultrasonic tomographic image that is the output of the position and orientation acquisition unit 912. Based on these data, a cross-sectional image corresponding to the ultrasonic tomographic image is generated from the three-dimensional volume data and output to the image composition unit 940. The image composition unit 940 obtains information indicating the existence range (second region) of the corresponding lesion part from the existence range calculation unit 935 and superimposes it on the ultrasonic tomographic image obtained from the tomographic image acquisition unit 910 for drawing. To do. Further, a composite image obtained by combining the image and the cross-sectional image acquired from the cross-sectional image generation unit 930 (for example, arranged side by side) is generated and output to the display control unit 950 or the outside. The display control unit 950 acquires the composite image that is the output of the image composition unit 940 and displays it on the display unit 960.

  7 (tomographic image acquisition unit 910, position and orientation acquisition unit 912, three-dimensional image data acquisition unit 920, attention area acquisition unit 922, error acquisition unit 923, cross-sectional image generation unit 930, existence range calculation unit 935, the image composition unit 940, and the display control unit 950) may be realized as independent devices. Alternatively, it may be implemented as software that implements its function by installing it on one or a plurality of computers and executing it by the CPU of the computer. In the present embodiment, each unit is realized by software and installed in the same computer.

(Basic computer configuration)
FIG. 17 illustrates a tomographic image acquisition unit 910, a position / orientation acquisition unit 912, a three-dimensional image data acquisition unit 920, an attention area acquisition unit 922, an error acquisition unit 923, a cross-sectional image generation unit 930, an existence range calculation unit 935, and an image synthesis unit. 940 is a diagram illustrating a basic configuration of a computer for realizing the functions of the display control unit 950 by executing software. FIG.

  The CPU 1001 controls the entire computer using programs and data stored in the RAM 1002 and the ROM 1003. In addition, the tomographic image acquisition unit 910, the position / orientation acquisition unit 912, the 3D image data acquisition unit 920, the attention area acquisition unit 922, the error acquisition unit 923, the cross-sectional image generation unit 930, the existence range calculation unit 935, the image composition unit 940, The execution of software in each of the display control units 950 is controlled to realize the function of each unit.

  The RAM 1002 includes an area for temporarily storing programs and data loaded from the external storage device 1007 and the storage medium drive 1008, and a work area required for the CPU 1001 to perform various processes.

  The ROM 1003 generally stores computer programs and setting data. A keyboard 1004 and a mouse 1005 are input devices, and an operator can input various instructions to the CPU 1001 using these devices.

  The display unit 1006 includes a CRT, a liquid crystal display, or the like, and the display unit 960 corresponds to this. The display unit 1006 can display a message to be displayed for image processing, a GUI, and the like in addition to the composite image generated by the image composition unit 940.

  The external storage device 1007 is a device that functions as a large-capacity information storage device such as a hard disk drive, and stores an OS (operating system), a program executed by the CPU 1001, and the like. In the description of the present embodiment, information that is described as being known is stored here, and loaded into the RAM 1002 as necessary.

  The storage medium drive 1008 reads a program or data stored in a storage medium such as a CD-ROM or DVD-ROM in accordance with an instruction from the CPU 1001 and outputs it to the RAM 1002 or the external storage device 1007.

  The I / F 1009 includes an analog video port or a digital input / output port such as IEEE 1394, an Ethernet (registered trademark) port for outputting information such as a composite image to the outside, and the like. The data input by each is taken into the RAM 1002 via the I / F 1009. Some of the functions of the tomographic image acquisition unit 910, position and orientation acquisition unit 912, 3D image data acquisition unit 920, attention area acquisition unit 922, and error acquisition unit 923 are realized by the I / F 1009.

  The above-described components are connected to each other by a bus 1010.

  FIG. 9 is a flowchart illustrating an overall processing procedure performed by the information processing apparatus 900. In the present embodiment, the flowchart is realized by the CPU 1001 executing a program that realizes the function of each unit. It is assumed that the program code according to the flowchart is already loaded from, for example, the external storage device 1007 to the RAM 1002 in the previous stage of performing the following processing.

(S11000: Data input)
In step S <b> 11000, the information processing apparatus 900 acquires 3D image data from the data server 990 as processing of the 3D image data acquisition unit 920. Further, as processing of the attention area acquisition unit 922, information representing the attention area is acquired from the data server 990. Here, the information indicating the attention area is, for example, the position of the attention lesion (the position of the center of gravity of the area) or the coordinates of the point group located at the area boundary of the attention lesion.

(S11010: Input of error factor information)
In step S <b> 11010, the information processing apparatus 900 acquires, from the data server 990, various error factor information (described later) used for calculation of an error estimated value as processing of the error acquisition unit 923.

(S11020: Acquisition of tomographic image)
In step S11020, the information processing apparatus 900 acquires an ultrasonic tomographic image from the second medical image acquisition apparatus 980 as processing of the tomographic image acquisition unit 910. Further, as the processing of the position / orientation acquisition unit 912, the position / orientation of the ultrasonic probe when the ultrasonic tomographic image is captured is acquired from the second medical image acquisition device 980. Then, using the calibration data stored in advance as known values, the position and orientation of the ultrasonic tomographic image in the MRI apparatus coordinate system are calculated from the position and orientation of the ultrasonic probe in the reference coordinate system.

(S11030: Acquisition of error estimated value)
In step S11030, the information processing apparatus 900 calculates an error estimated value based on the various error factor information (various data used for error calculation) acquired in step S11010 as processing of the error acquisition unit 923.

  In the present embodiment, the relationship between the MRI apparatus coordinate system and the reference coordinate system is expressed by rigid body transformation. However, the position and orientation of the subject with respect to the MRI apparatus coordinate system when the MRI image is captured and the position and orientation of the subject with respect to the reference coordinate system when performing ultrasonic imaging are not necessarily in a rigid body transformation relationship. Absent. Therefore, an error may be mixed when the relationship between the coordinate systems is expressed by rigid transformation. Further, even if the difference in position and orientation of the subject in the two coordinate systems is exactly related to rigid transformation, it is practically difficult to obtain the rigid transformation correctly, and errors are still mixed. There is a case. The error also includes an error (position and orientation measurement error) mixed in the position and orientation measurement value acquired by the position and orientation acquisition unit 912.

  The process of calculating the error estimation value can be executed based on, for example, the characteristics of a position / orientation sensor that measures the position / orientation of the ultrasonic probe of the second medical image acquisition device 980. For example, an error reference value can be determined in advance for each measurement method of the position and orientation sensor, and a value can be selected according to the measurement method of the sensor to be used. For example, since the measurement accuracy of a magnetic sensor is generally lower than that of an optical sensor, information indicating that the optical sensor is used is acquired from the data server 990 as error factor information, and the magnetic sensor is used based on the information. The estimated error value can be calculated as a small value compared to the case where Further, the error estimation value is not limited to the difference in the measurement method of the position / orientation sensor, and can be calculated based on the relationship between the spatial position and orientation with respect to the measurement reference of the position / orientation sensor. For example, when the position / orientation sensor is composed of a magnetic sensor, an estimated error value is defined as a function of the distance between the magnetism generation device that is a measurement reference and the ultrasonic probe, and the distance is large. In this case, a large value can be calculated as the error estimated value. When the position / orientation sensor is constituted by an optical sensor, the positions of a plurality of indices (markers) arranged on the ultrasonic probe are measured by the optical sensor, and the ultrasonic probe is detected based on these positions. The position and orientation of the tentacle are calculated. Here, when the distribution of a plurality of indices as viewed from the optical sensor that is the measurement reference is biased, the position and orientation error becomes large. Therefore, an estimated value of error is defined as a function of the distance between the optical sensor and the ultrasonic probe, and when the value is large, a large value can be calculated as the estimated error value. Further, when an error calculation estimated value is defined as a function of an angle formed by a vector from the optical sensor toward the ultrasonic probe and a normal direction of a surface on which a plurality of indices are arranged, and the value is large, A large value can be calculated as the error estimation value.

  In addition, the error estimation value can be calculated based on a part or the like in the subject where the lesion of interest exists. For example, when the lesion of interest is present in a soft tissue such as the breast of the subject, it is assumed that deformation occurs in the relevant part of the subject between the time when the MRI image is captured and the time when the ultrasound image is captured. Therefore, when the target lesion part exists in such a part, the error estimated value can be calculated large. Similarly, a large error estimate can be calculated in the heart and a region in the vicinity of the heart where the position variation due to the heartbeat is large, or in the lung and the region near the lung where the position variation due to respiration is large. Specifically, information indicating a part (organ name or position in the organ) in the subject where the target lesion is present and data (table) describing correspondence between the part and the magnitude of the error are error factor information. And an error estimation value can be calculated based on these pieces of information.

  The error estimation value may be calculated as a different value for each axial direction. For example, when the lesion of interest is present in a soft tissue such as the breast of the subject, the operation direction of the ultrasonic probe is likely to be deformed by pushing, so the error estimation value in the operation direction is orthogonal to the operation direction. It can be calculated larger (compared to two directions). The operation direction of the ultrasonic probe can be calculated by a known method based on the current position and the past (for example, 100 milliseconds before) of the ultrasonic probe. In this case, the error estimation value is expressed as three orthogonal vectors representing the direction and the magnitude.

  Further, information on the position of the part used as an index when the MRI apparatus coordinate system and the reference coordinate system are calibrated is acquired as error factor information, and based on the positional relationship between these parts and the target lesion part. An error estimated value can be calculated. For example, if the index used for the calibration is the xiphoid process of the subject, an error estimation value is calculated as a function of the distance between the position of the xiphoid process in the MRI image and the lesion of interest. it can. Similarly, when the position of the nipple is used as a calibration index, an error estimated value can be calculated as a function of the distance between the nipple and the lesion of interest in the MRI image. In addition, an error estimation value may be acquired by performing multiple error estimations using a plurality of the methods exemplified above.

(S11040: Acquisition of cross-sectional image)
In step S11040, the information processing apparatus 900 generates a cross-sectional image of a reference image corresponding to the ultrasonic tomographic image obtained in step S11020 as processing of the cross-sectional image generation unit 930. Specifically, based on the position and orientation of the ultrasonic tomographic image obtained in step S11020, a cross-sectional image is generated by cutting out the same cross section as the ultrasonic tomographic image from the three-dimensional volume data obtained in step S11000.

(S11050: Acquisition of existence range)
In step S11050, the information processing apparatus 900 calculates the existence range of the corresponding region on the ultrasonic tomographic image obtained in step S11020 as the processing of the existence range calculation unit 935.

  Specifically, the existence range calculation unit 935 first estimates a corresponding area for the attention area in the ultrasonic coordinate system. For example, when the position of the target lesion is given as information representing the attention area, the position of the corresponding lesion on the ultrasonic coordinate system is estimated as information representing the corresponding area. In addition, when the coordinates of the point group located at the region boundary of the target lesion area are given as information representing the attention area, the coordinates of the point group located at the boundary area of the corresponding lesion section are used as information representing the corresponding area. Is estimated on the ultrasonic coordinate system. These estimations can be performed based on the position and orientation of the ultrasonic tomographic image acquired by the position and orientation acquisition unit 912.

  Next, the existence range calculation unit 135 calculates the existence range of the corresponding region on the ultrasonic tomographic image based on the corresponding region estimated above and the error estimated value acquired in step S11030.

  When the information indicating the corresponding region is the position of the corresponding lesion and the error estimation value does not depend on the axial direction, the three-dimensional existence range of the corresponding lesion in the ultrasonic coordinate system is estimated. It is defined as a sphere centered at the location of the corresponding lesion and the error estimate is a radius. In addition, the existence range of the corresponding lesion on the ultrasonic tomographic image is defined as a circle that is an area (cross section of the sphere) where the sphere and the tomographic image intersect. Therefore, the existence range calculation unit 935 calculates the center position and radius of this circle on the ultrasonic tomographic image as the existence range of the corresponding lesion part. In addition, since the calculation method of the intersection area | region of the sphere and plane defined in three-dimensional space is a well-known thing, the description is abbreviate | omitted. When the sphere and the tomographic image do not intersect, information that “there is no existing range on the cross section” is stored.

  In addition, when the information representing the corresponding region is the position of the corresponding lesion and an error estimation value is given for each axial direction, the three-dimensional existence range of the corresponding lesion in the ultrasonic coordinate system is estimated. Centering on the position of the corresponding lesion, the error estimation value in each axial direction is defined as an ellipsoid having a radius in each axial direction. Therefore, the existence range calculation unit 935 calculates the intersection region (cross section of the ellipsoid) between the ellipsoid and the tomographic image as the existence range of the corresponding lesion on the ultrasonic tomographic image.

  On the other hand, when the information representing the corresponding region is the coordinates of the point group located at the region boundary of the corresponding lesion, a sphere or ellipsoid having the radius of the error estimation value is obtained for each point group as described above. The region defined as the union can be defined as the three-dimensional existence range of the corresponding region. Therefore, the existence range calculation unit 935 calculates the intersection area between the area and the ultrasonic tomographic image as the existence range of the corresponding area on the tomographic image.

(S11060: Draw the existence range on the tomographic image)
In step S11060, the information processing apparatus 900 superimposes and draws information representing the existence range of the corresponding lesion on the ultrasonic tomographic image calculated in step S11050 on the ultrasonic image as processing of the image synthesis unit 940. To do. At this time, it is preferable to display the range where the corresponding lesion can exist as a closed curve, put a translucent mask outside the closed curve, and display the inner ultrasonic tomographic image as usual. As a result, a search area for searching for an actual corresponding lesion is clarified. In addition, since the user can easily search for an actual corresponding lesion, it can be searched efficiently. In addition, the range in which the corresponding lesioned part may exist may be drawn using only a closed curve line. In addition, when coloring inside the range where the corresponding lesion can exist, it is preferable to add a color that is transparent enough to be searched. As a result of the processing in this step, an ultrasonic tomographic image 503 in which the existence range 502 of the corresponding lesion 501 is superimposed on the tomographic image as shown in FIG. 5A is generated. If it is determined in step S11050 that “there is no area on the cross section”, the processing in this step is not executed.

  Note that it is determined whether or not the tomographic image intersects with the corresponding region itself obtained in step S11050. If the tomographic image intersects, the intersecting region on the tomographic image is superimposed and displayed on the tomographic image. You may do it.

(S11070: Combining a tomographic image and a cross-sectional image)
In step S11070, the information processing apparatus 900 generates, as the processing of the image synthesis unit 940, an image in which the cross-sectional image obtained in step S11040 and the ultrasonic lesion image obtained in step S11060 are superimposed with the corresponding lesion part existing range. . For example, an image in which these images are arranged side by side is generated. Then, as a process of the display control unit 950, the synthesized image is displayed on the display unit 960. Further, if necessary, this is output to the outside via the I / F 1009 and further stored in the RAM 1002 as a state usable by other applications.

(S11080: Determination of whether to end the entire process)
In step S11080, the information processing apparatus 900 determines whether to end the entire process. For example, an end instruction input by an operator pressing a predetermined key (end key) on the keyboard 1004 is acquired. When it is determined that the processing is to be ended, the entire processing of the information processing apparatus 900 is ended. On the other hand, if it is not determined to end, the process returns to step S11010, and the processes subsequent to step S11010 are executed again on the ultrasonic tomographic image that is newly imaged.

  As described above, the processing of the information processing apparatus 900 is executed.

  As described above, according to the information processing apparatus according to the present embodiment, the presence range of the corresponding lesion portion in consideration of the position estimation error is displayed on the ultrasonic tomographic image as a guide when the operator searches for the corresponding lesion portion. Presented. As a result, the operator's unnecessary work of searching a wider range than necessary can be reduced, and the search work load can be reduced. In addition, since the search range can be limited, it is possible to reduce the risk of an operator performing an incorrect association.

  In the above-described embodiment, the cross-section image cut out from the three-dimensional image data that is the reference image of the cross section that is the same as the ultrasonic tomographic image is presented side by side with the ultrasonic tomographic image, but this cross-sectional image is not necessarily displayed. You don't have to. In this case, acquisition of 3D image data that is a reference image and generation processing of a cross-sectional image may not be performed.

(Fourth embodiment: including deformation estimation)
In the information processing system according to the third embodiment, it is assumed that the shape of the subject at the time of photographing the three-dimensional image data and the shape of the subject at the time of ultrasonic photographing are not changed (is a rigid body). Then, by measuring the position and orientation of the ultrasonic probe with respect to the subject, the corresponding region (and its existence range) of the region of interest in the ultrasonic coordinate system has been estimated. On the other hand, in this embodiment, when estimating the deformation from the shape of the subject at the time of 3D image data imaging to the shape of the object at the time of ultrasonic imaging and obtaining the corresponding region, the ambiguity of the deformation estimation A case where the existence range of the corresponding region is estimated by considering the above will be described. Hereinafter, only the difference between the information processing system according to the present embodiment and the third embodiment will be described.

  FIG. 8 shows a configuration of the information processing system according to the present embodiment. The same parts as those in FIG. 7 are given the same numbers and symbols, and the description thereof is omitted. As illustrated in FIG. 8, the information processing apparatus 1000 according to this embodiment is connected to a shape measuring apparatus 1085.

  The range sensor as the shape measuring device 1085 measures the surface shape of the subject at the time of ultrasonic imaging to obtain surface shape data. The shape measuring device 1085 may be configured in any way as long as it can measure the shape of the target object, and may be a stereo image measuring device, for example.

  The shape acquisition unit 1027 acquires the surface shape data of the subject input to the information processing apparatus 1000 and outputs it to the deformation estimation unit 1028.

  The deformation estimation unit 1028 estimates the deformation state of the subject based on the surface shape data acquired by the shape acquisition unit 1027. Then, a variation range (details will be described later) of the deformation parameter is calculated and output to the existence range calculation unit 1035. Also, a deformed three-dimensional image is generated by deforming the three-dimensional image data into the shape of the subject at the time of ultrasonic imaging, and this is output to the cross-sectional image generation unit 1030.

  The existence range calculation unit 1035 is based on the information representing the attention region acquired by the attention region acquisition unit 122 and the variation range of the deformation parameter estimated by the deformation estimation unit 1028, and the existence range of the corresponding region on the ultrasonic tomographic image. Is calculated.

  The cross-sectional image generation unit 1030 corresponds to the ultrasonic tomographic image based on the deformed three-dimensional image that is the output of the deformation estimation unit 1028 and the position and orientation of the ultrasonic tomographic image that is the output of the position and orientation acquisition unit 112. An image is generated from the deformed three-dimensional image and output to the image composition unit 140.

  FIG. 10 is a flowchart illustrating an overall processing procedure performed by the information processing apparatus 1000 according to the present embodiment.

(S12000: Data input)
In step S12000, the information processing apparatus 1000 performs processing similar to that in step S11000 in the third embodiment, and acquires 3D image data and information representing a region of interest. Further, as the processing of the shape acquisition unit 1027, the surface shape data of the subject is acquired from the shape measuring device 1085.

(S12005: Deformation estimation)
In step S12005, the information processing apparatus 1000 estimates the deformation state of the subject based on the surface shape data acquired in step S12000 as processing of the deformation estimation unit 1028. For example, using a deformation estimation method described in Japanese Patent Application Laid-Open No. 2011-092263, a deformation model of a subject is generated from three-dimensional image data, and the deformation parameter is estimated by applying this to the shape data. Further, based on the estimated deformation parameter, a deformed three-dimensional image is generated by deforming the three-dimensional image data into the shape of the subject at the time of ultrasonic imaging.

(S12010: Calculation of variation range of deformation parameter)
In step S12010, the information processing apparatus 1000 calculates a variation range of the deformation parameter as a process of the deformation estimation unit 1028. Here, the variation range of the deformation parameter is a parameter such that when the deformation parameter value is varied in the vicinity of the estimated value, the evaluation value of the degree of coincidence between the shape data and the deformation shape falls within a certain range. It is a range. This fluctuation range represents the ambiguity of the estimated value of the deformation parameter.

(S12015: Calculation of existence range in three-dimensional space)
In step S12015, the information processing apparatus 1000 calculates the existence range of the corresponding region in the three-dimensional space based on the solution ambiguity in the deformation estimation as the processing of the existence range calculation unit 1035. Specifically, the deformation parameter is varied within the variation range obtained in step S12010, and the displacement of the corresponding region by each deformation parameter is calculated. Then, a region including all the regions after the displacement is set as a corresponding region existing range in the three-dimensional space. For example, the smallest ellipsoid that includes all the areas after displacement is derived, and this is used as the existence range of the corresponding area.

(S12020: Acquisition of tomographic image)
In step S12020, the information processing apparatus 1000 performs processing similar to that in step S11020 in the third embodiment, and acquires an ultrasonic tomographic image and its position and orientation.

(S12040: Acquisition of cross-sectional image)
In step S12040, the information processing apparatus 1000 generates a cross-sectional image of a deformed three-dimensional image corresponding to the ultrasonic tomographic image obtained in step S12020 as processing of the cross-sectional image generation unit 1030.

(S12050: Acquisition of existence range on cross-sectional image)
In step S12050, the information processing apparatus 1000 calculates the existence range of the corresponding region on the ultrasonic tomographic image as the processing of the existence range calculation unit 1035. Specifically, the existence range of the corresponding area on the two-dimensional plane is calculated by cutting out the existence area of the corresponding area in the three-dimensional space calculated in step S12015 with an ultrasonic cross section.

  Note that the processing in steps S12060, S12070, and S12080 is the same as steps S11060, S11070, and S11080 in the third embodiment, and thus description thereof is omitted.

  As described above, the information processing apparatus according to the present embodiment determines the corresponding region by estimating the deformation from the shape of the subject at the time of photographing the three-dimensional image data to the shape of the subject at the time of ultrasonic photographing. The existence range of the corresponding area is calculated by taking into account the ambiguity of the deformation estimation. By doing so, even when the lesion of interest exists in a soft tissue such as the breast of the subject, the user can more accurately know the search range of the actual corresponding area on the two-dimensional image. It is possible to search efficiently.

(Variation 1 of the fourth embodiment: calculated from a group of previous deformation simulations)
In the present embodiment, as an example of a method for calculating the existence range of the corresponding region by considering the ambiguity of the deformation estimation, a method based on the parameter fluctuation range around the solution at the time of deformation parameter estimation has been described. The implementation of is not limited to this.

  For example, the existence range of the corresponding region may be calculated based on variations in the attention region displaced by a plurality of deformation simulations. For example, after displacing the center of gravity of the lesion of interest by various deformation simulations, a polyhedron (for example, a convex hull at all positions) or a closed surface (for example, an ellipsoid) containing all the positions after the displacement is calculated. Thus, it can be set as the existence range of the corresponding lesion.

(Modification 2 of the fourth embodiment: using an error estimation value related to deformation)
Also, when considering deformation as in the fourth embodiment, an estimated error value may be acquired and the existence range of the corresponding region may be calculated based on the error estimation value, as in the third embodiment.

  For example, the existence range of the corresponding region may be calculated based on the deformation estimation error distribution in the past cases. For example, consider a case in which a statistical value of error is held in the data server for each size of the breast and each site where a lesion exists. At this time, information on the size of the breast and the location of the lesion relating to the subject can be acquired from the image, and a corresponding statistical value can be acquired based on the acquired information, and the existing range can be calculated using this as an error estimation value.

  Further, the error estimation value may be calculated based on a residual when the deformation estimation unit estimates the deformation parameter (an amount of deviation between the shape of the subject at the time of ultrasonic imaging and the shape of the deformed three-dimensional image). For example, when the residual is large, it is possible to calculate a large error estimated value.

  Further, the error estimation value may be switched according to the deformation estimation method to be used. For example, the error estimation value can be calculated to be small when a highly accurate deformation estimation method is used, and the error estimation value can be calculated to be small when a simple deformation estimation method is used.

  Further, the difference between the surface position of the ultrasonic probe and the body surface position in the cross-sectional image may be calculated, and the error estimation value may be calculated based on the difference. For example, when the difference is large, it is possible to calculate a large error estimated value.

  Further, the error estimated value may be calculated based on the reliability of image analysis such as body surface detection performed as pre-processing for deformation estimation. For example, when the reliability of image analysis is low, the error estimation value can be calculated to be large.

  In this modification, when the error estimated value is calculated as a single value, the corresponding lesion part existing range is the center of the target lesion position and the error estimated value is the radius, as in the third embodiment. It may be defined as a certain sphere.

(Fifth embodiment)
According to the above-described embodiment, among the three-dimensional existence range including a portion (corresponding region, corresponding lesion portion) corresponding to a predetermined portion (attention region, attention lesion portion) in the three-dimensional image (reference image) data, The range (two-dimensional region) in which the corresponding lesion portion can exist in the ultrasonic tomographic image can be calculated by considering the error of position estimation. Here, the three-dimensional existence range is a three-dimensional region in which a corresponding lesion can exist, and is expressed in the same space (coordinate system) as the ultrasonic tomographic image (target image). The existence range is presented on the ultrasonic tomographic image as a guide when the user searches for the corresponding lesion. As a result, the user knows the range (search range) in which the corresponding lesion should be searched, so that the user can efficiently search for and identify the corresponding lesion.

  However, in the above-described embodiment, since only one section of the three-dimensional existence range of the corresponding lesion is displayed, it is not possible to easily grasp which part of the three-dimensional existence range is cut out. It was. For example, when the three-dimensional existence range is given in a spherical shape, the two-dimensional existence range on the tomographic image is always circular even if the ultrasonic tomographic image intersects any part of the three-dimensional existence range. Is displayed. Therefore, it is not easy for a user who does not know the size of the sphere to know whether the current intersection position is near the center of the sphere or near the end of the sphere.

  Therefore, the image processing system according to the present embodiment has information indicating which position is cut out as an ultrasonic tomographic image in the three-dimensional existence range (for example, color information associated with position information of the tomographic image). ) Is added to the information (display form) indicating the two-dimensional area (intersection area) and displayed. At this time, the display form changing unit (an example of the display information generating unit 937) changes the display form to indicate the two-dimensional area associated with the position information according to the position information of the tomographic image. Thereby, the user can easily grasp the whole image of the three-dimensional existence range (three-dimensional region), and can efficiently search and identify the corresponding lesion part. Note that it is desirable to have a designation unit (not shown) for manually or automatically design the attention area of the three-dimensional image. In addition, it is desirable to have a determination unit (an example of a display information generation unit 937) that determines a presence range in which a corresponding region corresponding to a region of interest exists as a predetermined region.

  First, FIG. 11 shows a configuration of an information processing system according to the present embodiment. The information processing apparatus 900 according to this embodiment includes a tomographic image acquisition unit 910, a position / orientation acquisition unit 912, a three-dimensional image data acquisition unit 920 (also referred to as a three-dimensional image acquisition unit), an attention area acquisition unit 922, an error acquisition unit 923, A cross-sectional image generation unit 930, a three-dimensional existence range calculation unit (also referred to as a three-dimensional region acquisition unit) 935, a two-dimensional existence range acquisition unit (also referred to as a two-dimensional region acquisition unit or an intersection region acquisition unit) 934, and a position information calculation unit 936 The image composition unit 940 and the display control unit 950 are configured. It is connected to a data server 990 that holds 3D image data and error factor information described later. The information processing apparatus 900 is also connected to an ultrasonic image diagnostic apparatus as a second medical image collection apparatus 980 that captures an ultrasonic tomographic image of a subject.

(Acquisition of 3D image data)
The three-dimensional image data held by the data server 990 is an image obtained by imaging a subject in advance with an MRI apparatus or an X-ray CT apparatus as the first medical image acquisition apparatus 970. Hereinafter, a case where an MRI apparatus is used as the first medical image collection apparatus 970 will be described as an example. In this embodiment, it is assumed that the three-dimensional image data is expressed as three-dimensional volume data in which luminance values are stored in three-dimensional voxels. The coordinates of each voxel are expressed in the MRI apparatus coordinate system. The 3D image data held by the data server 990 is acquired by the 3D image data acquisition unit 920 and input to the information processing apparatus 900.

(Obtain attention area)
Furthermore, the data server 990 holds information (details will be described later) representing a region of interest in the three-dimensional image data. Information representing the attention area held by the data server 990 is acquired by the attention area acquisition unit 922 and input to the information processing apparatus 900. In the following description, it is assumed that the information indicating the region of interest is also expressed in the MRI apparatus coordinate system, like the three-dimensional image data.

(Acquisition of tomographic images)
The ultrasonic diagnostic imaging apparatus as the second medical image acquisition apparatus 980 captures an ultrasonic tomographic image of a subject in real time. The ultrasonic tomographic image is acquired by the tomographic image acquisition unit 910 and is sequentially input to the information processing apparatus 900. Further, the position and orientation of the ultrasonic probe are measured by a position and orientation sensor (not shown), acquired by the position and orientation acquisition unit 912, and input to the information processing apparatus 900. Here, the position and orientation of the ultrasonic probe are represented by the position and orientation in a reference coordinate system with the subject as a reference, for example. Further, the position / orientation acquisition unit 912 acquires the position / orientation of the ultrasonic probe in the reference coordinate system, and calculates the position / orientation of the ultrasonic tomographic image in the MRI apparatus coordinate system based on the acquired position / orientation.

(Calculation of 3D existence range)
The three-dimensional existence range calculation unit 935 is an area corresponding to the attention area based on the information indicating the attention area acquired by the attention area acquisition unit 922 and the position and orientation of the ultrasonic tomographic image acquired by the position and orientation acquisition unit 912. (Corresponding region) is estimated on the ultrasonic coordinate system. Here, the ultrasonic coordinate system is a three-dimensional coordinate system based on the ultrasonic tomographic image. For example, the origin is one point on the tomographic image, and the x axis and the y axis are on the plane of the tomographic image. It can be defined as a coordinate system in which the z axis is set in a direction perpendicular to the plane. Then, based on the estimated corresponding area and an error estimated value acquired by an error acquisition unit 923 described later, a three-dimensional existence range (three-dimensional area in the present embodiment) of the corresponding area in the ultrasonic coordinate system is calculated. .

(Acquisition of 2D existence range)
The two-dimensional existence range acquisition unit 934 obtains an intersection area between the ultrasonic tomographic image and the three-dimensional existence range based on the three-dimensional existence range calculated by the three-dimensional existence range calculation unit 935, and calculates this on the ultrasonic tomographic image. Is output to the display information generation unit 937 as a two-dimensional existence range of the corresponding region.

(Calculation of location information)
The position information calculation unit 936 is based on information representing the three-dimensional existence range of the corresponding region in the ultrasonic coordinate system acquired from the three-dimensional existence range calculation unit 935 and information on the position and orientation of the ultrasonic tomographic image. The relative position information of the ultrasonic tomographic image with respect to the existence range is calculated. Then, the calculated relative position information is output to the display information generation unit 937.

(Generate display information of existence range)
The display information generation unit 937 performs relative processing based on the information on the two-dimensional existence range acquired from the two-dimensional existence range acquisition unit 934 and the relative position information on the ultrasonic tomographic image acquired from the position information calculation unit 936. Display information of a two-dimensional existence range to which various position information is added.

(Acquire error estimate)
In addition to the information described above, the data server 990 holds information (error factor information; details will be described later) for calculating an estimated error value of the corresponding region. Here, the error factor information is, in other words, information for calculating the three-dimensional existence range of the corresponding region on the ultrasonic tomographic image. The error factor information held by the data server 990 is input to the information processing apparatus 900 via the error acquisition unit 923. The error acquisition unit 923 calculates an error estimation value for the corresponding region based on the acquired error factor information. Then, the calculated error estimation value is output to the three-dimensional existence range calculation unit 935.

(Cross section image generation)
The cross-sectional image generation unit 930 receives the 3D volume data that is the output of the 3D image data acquisition unit 920 and the position and orientation of the ultrasonic tomographic image that is the output of the position and orientation acquisition unit 912. Based on these data, a cross-sectional image corresponding to the ultrasonic tomographic image is generated from the three-dimensional volume data and output to the image composition unit 940. The image composition unit 940 obtains the display information of the two-dimensional existence range from the display information generation unit 936, and superimposes and draws it on the ultrasonic tomographic image obtained from the tomographic image acquisition unit 910. Further, a composite image obtained by combining the image and the cross-sectional image acquired from the cross-sectional image generation unit 930 (for example, arranged side by side) is generated and output to the display control unit 950 or the outside. The display control unit 950 acquires the composite image that is the output of the image composition unit 940 and displays it on the display unit 960.

  11 (tomographic image acquisition unit 910, position / orientation acquisition unit 912, 3D image data acquisition unit 920, attention area acquisition unit 922, error acquisition unit 923, cross-sectional image generation unit 930, 3D existence range) At least a part of the calculation unit 935, the two-dimensional existence range acquisition unit 934, the position information calculation unit 936, the display information generation unit 937, the image synthesis unit 940, and the display control unit 950) may be realized as an independent device. Alternatively, it may be implemented as software that implements its function by being installed in one or a plurality of computers and executed by the CPU of the computer. In the present embodiment, each unit is realized by software and installed in the same computer.

(Basic computer configuration)
FIG. 17 illustrates a tomographic image acquisition unit 910, a position / orientation acquisition unit 912, a 3D image data acquisition unit 920, an attention area acquisition unit 922, an error acquisition unit 923, a cross-sectional image generation unit 930, a 3D existence range calculation unit 935, A basic configuration of a computer for realizing the functions of the dimension existence range acquisition unit 934, the position information calculation unit 936, the display information generation unit 937, the image composition unit 940, and the display control unit 950 by executing software is shown. FIG.

  The CPU 1001 controls the entire computer using programs and data stored in the RAM 1002 and the ROM 1003. In addition, the tomographic image acquisition unit 910, the position / orientation acquisition unit 912, the three-dimensional image data acquisition unit 920, the attention area acquisition unit 922, the error acquisition unit 923, the cross-sectional image generation unit 930, the three-dimensional existence range calculation unit 935, and the two-dimensional existence The execution of software in each of the range acquisition unit 934, the position information calculation unit 936, the display information generation unit 937, the image composition unit 940, and the display control unit 950 is controlled to realize the function of each unit.

  The RAM 1002 includes an area for temporarily storing programs and data loaded from the external storage device 1007 and the storage medium drive 1008, and a work area required for the CPU 1001 to perform various processes.

  The ROM 1003 generally stores computer programs and setting data. A keyboard 1004 and a mouse 1005 are input devices, and a user can input various instructions to the CPU 1001 using these devices.

  The display unit 1006 includes a CRT, a liquid crystal display, or the like, and the display unit 960 corresponds to this. The display unit 1006 can display a message to be displayed for image processing, a GUI, and the like in addition to the composite image generated by the image composition unit 940.

  The external storage device 1007 is a device that functions as a large-capacity information storage device such as a hard disk drive, and stores an OS (operating system), a program executed by the CPU 1001, and the like. In the description of the present embodiment, information that is described as being known is stored here, and loaded into the RAM 1002 as necessary.

  The storage medium drive 1008 reads a program or data stored in a storage medium such as a CD-ROM or DVD-ROM in accordance with an instruction from the CPU 1001 and outputs it to the RAM 1002 or the external storage device 1007.

The I / F 1009 includes an analog video port or a digital input / output port such as IEEE 1394, an Ethernet (registered trademark) port for outputting information such as a composite image to the outside, and the like. The data input by each is taken into the RAM 1002 via the I / F 1009. Some of the functions of the tomographic image acquisition unit 910, the position and orientation acquisition unit 912, the three-dimensional image data acquisition unit 920, the attention area acquisition unit 922, and the error acquisition unit 923 are realized by the I / F 1009.
The above-described components are connected to each other by a bus 1010.

  FIG. 12 is a flowchart illustrating an overall processing procedure performed by the information processing apparatus 900. In the present embodiment, the flowchart is realized by the CPU 1001 executing a program that realizes the function of each unit. It is assumed that the program code according to the flowchart is already loaded from, for example, the external storage device 1007 to the RAM 1002 in the previous stage of performing the following processing.

(S11000: Data input)
In step S <b> 11000, the information processing apparatus 900 acquires 3D image data from the data server 990 as a process of the 3D image data acquisition unit 920. Further, as processing of the attention area acquisition unit 922, information representing the attention area is acquired from the data server 990. Here, the information indicating the attention area is, for example, the position of the attention lesion (the position of the center of gravity of the area) or the coordinates of the point group located at the area boundary of the attention lesion.

(S11010: Input of error factor information)
In step S <b> 11010, the information processing apparatus 900 acquires various error factor information used for calculation of an error estimated value from the data server 990 as processing of the error acquisition unit 923. Here, the error factor information is information for calculating the existence region of the corresponding region on the ultrasonic tomographic image. For example, information indicating the type (for example, sensor A, sensor B, etc.) of the position and orientation sensor that measures the position and orientation of the ultrasound probe is acquired from the data server 990 as error factor information.

(S11020: Acquisition of tomographic image)
In step S11020, the information processing apparatus 900 acquires an ultrasonic tomographic image from the second medical image acquisition apparatus 980 as processing of the tomographic image acquisition unit 910. Further, as the processing of the position / orientation acquisition unit 912, the position / orientation of the ultrasonic probe when the ultrasonic tomographic image is captured is acquired from the second medical image acquisition device 980. Then, using the calibration data stored in advance as known values, the position and orientation of the ultrasonic tomographic image in the MRI apparatus coordinate system are calculated from the position and orientation of the ultrasonic probe in the reference coordinate system.

(S11030: Acquisition of error estimated value)
In step S11030, the information processing apparatus 900 calculates an error estimation value based on the various error factor information (various data used for error calculation) acquired in step S11010 as a process of the error acquisition unit 923, and performs three-dimensional The data is output to the existence area calculation unit 935.

  The process of calculating the error estimation value can be executed based on, for example, characteristics of a position / orientation sensor that measures the position / orientation of the ultrasonic probe. An error reference value is determined in advance for each type of position and orientation sensor, and a value can be selected according to the type of sensor to be used. For example, when the error factor information input in step S11010 is information that the sensor A that is an optical sensor is used, the error is estimated as a smaller value than when the sensor B that is a magnetic sensor is used. The value can be calculated. The process for estimating the error may be another process.

(S11040: Acquisition of cross-sectional image)
In step S11040, the information processing apparatus 900 generates a cross-sectional image of a reference image corresponding to the ultrasonic tomographic image obtained in step S11020 as processing of the cross-sectional image generation unit 930. Specifically, based on the position and orientation of the ultrasonic tomographic image obtained in step S11020, a cross-sectional image is generated by cutting out the same cross section as the ultrasonic tomographic image from the three-dimensional volume data obtained in step S11000.

(S11050: Acquisition of 3D existence range)
In step S11050, the information processing apparatus 900 calculates the three-dimensional existence range (three-dimensional area) of the corresponding area in the ultrasonic coordinate system of the ultrasonic tomographic image obtained in step S11020 as the processing of the three-dimensional existence range calculation unit 935. To do.

  Specifically, the three-dimensional existence range calculation unit 935 first estimates a corresponding area for the attention area in the ultrasonic coordinate system. For example, when the position of the target lesion is given as information representing the region of interest, the position of the corresponding lesion in the ultrasonic coordinate system is estimated as information representing the corresponding region.

  Next, the three-dimensional existence range calculation unit 935 calculates the three-dimensional existence range of the corresponding area in the ultrasonic coordinate system based on the corresponding area estimated above and the error estimated value acquired in step S11030. For example, when the information representing the corresponding region is the position of the corresponding lesion, and the error estimation value does not depend on the axial direction, the three-dimensional existence range of the corresponding lesion in the ultrasonic coordinate system is estimated. It is calculated as a sphere centered at the position of the corresponding lesion and the error estimate is a radius.

(S11051: Acquisition of two-dimensional existence range)
In step S11051, the information processing apparatus 900 performs processing of the two-dimensional existence range acquisition unit 934 based on the three-dimensional existence range calculated by the three-dimensional existence range calculation unit 935 based on the ultrasonic tomographic image and the three-dimensional existence range. The intersection region (two-dimensional existence range) is obtained and output to the display information generation unit 937. When the three-dimensional existence range is a sphere, the two-dimensional existence range is defined as a circle that is an area (a cross section of the sphere) where the sphere and the tomographic image intersect. Therefore, the two-dimensional existence range acquisition unit 934 calculates the center position and radius of this circle on the ultrasonic tomographic image as the two-dimensional existence range. In addition, since the calculation method of the intersection area | region of the sphere and plane defined in three-dimensional space is a well-known thing, the description is abbreviate | omitted. When the sphere and the tomographic image do not intersect, information that “there is no existing range on the cross section” is stored.

(S11052: Calculate position information of tomographic image)
In step S11052, the information processing apparatus 900 acquires the position information of the three-dimensional existence range and the ultrasonic tomographic image from the three-dimensional existence range calculation unit 935 as the processing of the position information calculation unit 936, and acquires ultrasonic waves in the three-dimensional existence range. The relative position information of the tomographic image is calculated. However, if it is determined in step S11051 that “there is no existing area on the cross section”, the processing in this step is not executed. The specific calculation method is shown below.

  FIG. 13 is a diagram illustrating position information of an ultrasonic tomographic image in a three-dimensional existence range. Reference numeral 301 denotes a three-dimensional existence range, 302 denotes an ultrasonic tomographic image, and 303 denotes a two-dimensional existence range obtained by cutting out the three-dimensional existence range 301 by the ultrasonic cross section 302 calculated in step S11050.

  At this time, the information processing apparatus 900 sets a plane parallel to the ultrasonic tomographic image 302 (hereinafter referred to as a parallel plane) in a three-dimensional space, and sets the set parallel plane as an axis orthogonal to the ultrasonic tomographic image 302. A position where the parallel plane starts to intersect with the three-dimensional existence range 301 and a position where the intersection ends are calculated when translated along. Here, a position where the intersection starts is defined as a start position (an example of a predetermined position), and a position where the intersection ends is defined as an end position (an example of the predetermined position). Reference numeral 304 denotes a start position, and 305 denotes an end position.

  Next, the information processing apparatus 900 plots the corresponding position of the ultrasonic tomographic image 302, the corresponding position of the start position, and the corresponding position of the end position on an axis orthogonal to the ultrasonic tomographic image 302. It is defined as a tomographic image corresponding position, a start corresponding position, and an end corresponding position. Ax is an orthogonal axis orthogonal to the ultrasonic tomographic image 302, Ps is a tomographic image corresponding position, P0 is a start corresponding position, and P1 is an end corresponding position. Then, the relative position (positional relationship) of the tomographic image corresponding position Ps between the start corresponding position P0 and the end corresponding position P1 is calculated as relative position information. In this embodiment, for example, when normalized so that the coordinate of P0 on the axis Ax is 0 and the coordinate of P1 is 1, the coordinate of Ps between them (eg, 0.3) is used as the relative position information. To do.

  As described above, the start position 304 and the end position 305 of the three-dimensional existence range 301 are obtained along the axis orthogonal to the ultrasonic tomographic image 302, and the relative position information of the ultrasonic tomographic image 302 between them is calculated. . This is because the current ultrasonic tomographic image 302 with respect to the position where the user begins to intersect with the three-dimensional existence range and the position where the user finishes intersecting when the ultrasonic probe is translated while the posture of the ultrasonic tomographic image 302 being displayed is fixed. It means that the relative position information is obtained. As a result, when the user translates the ultrasonic probe while the posture of the ultrasonic tomographic image 302 is fixed, to which direction and how much to translate, which side of the three-dimensional existence range can be reached Can be easily grasped.

  However, the calculation method of the start position and the end position is not limited to the method of translating the parallel plane along the axis orthogonal to the ultrasonic tomographic image. For example, the parallel plane may be translated along an arbitrary axis, and the position where the three-dimensional existence range 301 starts to intersect and the position where the intersection ends can be calculated.

  Then, the calculated relative position information is transmitted to the display information generation unit 937.

(S11054: Generate display information of existence range to which position information is added)
In step S <b> 11054, the information processing apparatus 900 performs processing of the display information generation unit 937 as shape information representing the outline of the two-dimensional existence range acquired from the two-dimensional existence range acquisition unit 934 and the super information acquired from the position information calculation unit 936. Display information in which the relative position information is added to the two-dimensional existence range is generated based on the relative position information of the acoustic tomographic image. However, if it is determined in step S11051 that “there is no existing area on the cross section”, the process of this step is not executed. A specific generation method is shown below.

  First, the information processing apparatus 900 obtains color information corresponding to the position of the ultrasonic tomographic image in the three-dimensional existence range by associating the relative position information of the ultrasonic tomographic image with the color table.

  FIG. 14A is a diagram illustrating association of relative position information with a color table. Ax, Ps, P1, and P2 represent the same as in FIG. At this time, a color table is set between the positions P0 and P1 along the axis Ax so that the color changes (changes) according to the position. In the present embodiment, for example, a color table in which the intensity of a specific color continuously changes is set such that the closer to P0, the lighter the color, and the closer to P1, the darker the color. In step S10052, color information on the color table corresponding to the relative position information of the ultrasonic tomographic image normalized between P0 and P1 is acquired. T1 represents a color table, and C1 represents color information corresponding to relative position information. For example, when the color table T1 represents a color table in which the intensity of blue changes (light blue → dark blue in the direction from P0 to P1), and the relative position information of Ps is 0.3, Ps is set to P0. Since the color information C1 is mapped to a position on a close color table, the color information C1 is represented in a slightly light blue color. By referring to this color information, it is intuitively understood that the tomographic image corresponding position Ps is relatively close to the position P0.

  However, the color table setting method is not limited to the above-described method. For example, a color table may be used in which the intensity of a specific color is not continuous but changes stepwise (for example, 10 steps). Alternatively, instead of a single color, a plurality of typical colors are arranged at a predetermined interval between P0 and P1, and the color continuously changes between adjacent colors (eg, in the direction from P0 to P1). Black → Blue → Green → Yellow → Red → White) A color table may be used. Of course, a color table in which P0 and P1 are divided into a plurality of stages and different colors are assigned to the respective stages may be used.

  Next, the information processing apparatus 900 generates display information associated with the shape information of the two-dimensional existence range based on the color information corresponding to the relative position information of the acquired ultrasonic tomographic image. This is called presence range display information. In this embodiment, display information obtained by adding the acquired color information to the contour shape of the two-dimensional existence range is obtained.

  FIG. 15 is a diagram showing the existence range display information. FIG. 15A shows display information in which color information is added to the contour shape (contour line) of the existence range. In FIG. 15A, S1 represents the contour shape (contour line) of the existence range to which color information is added. For example, when the relative position information is 0.3 and is mapped to slightly light blue as described above, S1 has a contour shape (contour line) colored slightly light blue.

  However, the generation method of the existence range display information is not limited to the above method. For example, instead of adding the relative position information of the ultrasonic tomographic image as color information to the shape information of the two-dimensional existence range, a curve indicating the shape may be represented by a dotted line and represented by the density of the dotted line. In this case, instead of the color table, a table in which the density of the dotted line changes according to the position between the positions P0 and P1 is set in advance (eg, a rough dotted line → a dense dotted line in the direction from P0 to P1). The position Ps needs to be mapped on the table. FIG. 15B shows display information obtained by adding dotted line density information to the contour shape of the existence range. In FIG. 15B, S2 represents the shape of the existence range to which the dotted line density information is added. For example, when the relative position information is 0.3 as described above and is mapped to a slightly rough dotted line, S2 has a contour shape drawn with a slightly rough dotted line.

  Further, the target to which the relative position information of the ultrasonic tomographic image is given is not limited to the contour shape in the two-dimensional existence range. For example, the inner area of the two-dimensional existence range may be colored. FIG. 15C shows display information in which color information is added to the internal region of the existence range. In FIG. 15C, R1 represents an internal area of the existence range to which color information is added. For example, when the relative position information is mapped to slightly light blue as described above, R1 is an area colored with light blue. Or, conversely, the outer region of the two-dimensional existence range may be colored. FIG. 15D shows display information in which color information is added to the external area of the existence range. In FIG. 15D, R2 represents an external area of the existence range to which color information is added. When the same color mapping as described above is performed, R2 is an area colored with slightly light blue. Then, the generated existence range display information is transmitted to the image composition unit 940.

(S11060: Draw the existence range on the tomographic image)
In step S11060, the information processing apparatus 900, as the processing of the image composition unit 940, is obtained by adding the relative position information of the ultrasonic tomographic image to the two-dimensional existence range of the corresponding lesion part acquired from the display information generation unit 937. The range display information is drawn by being superimposed on the ultrasonic image. However, if it is determined in step S11051 that “there is no existing area on the cross section”, the process of this step is not executed.

  FIG. 16 is a diagram illustrating an ultrasonic tomographic image on which existence range display information is superimposed and displayed. FIG. 16A shows an ultrasonic tomographic image in which display information obtained by adding relative position information as color information to a contour shape of a two-dimensional existence range is superimposed and displayed. In FIG. 16A, reference numeral 601 denotes an ultrasonic tomographic image, 602 denotes a corresponding lesion, and S1 denotes existing range display information similar to that in FIG. At this time, by superimposing the existence range display information S1 on the ultrasonic tomographic image 601, a search region for searching for an actual corresponding lesion is clarified. In addition, since the user can easily search for an actual corresponding lesion, it can be searched efficiently.

  The user can intuitively grasp the relative position information of the ultrasonic tomographic image 601 in the three-dimensional existence range by referring to the color information added to the contour shape of the existence range. When the color information is expressed in slightly light blue as shown in FIG. 15A, the ultrasonic tomographic image 601 can be easily located near the start position 304 in FIG. 13 with respect to the three-dimensional existence range. I understand.

  Here, the existence range display information drawn on the ultrasonic tomographic image is not limited to the display information shown in FIG. For example, display information expressed by the density of a curve that draws the contour shape of the two-dimensional existence range as relative position information may be drawn on the ultrasonic tomographic image. FIG. 16B shows an ultrasonic tomographic image on which a two-dimensional existence range in which relative position information is expressed by the density of the contour shape is superimposed. In FIG. 16B, 601 and 602 represent the same as in FIG. 16A, and S2 represents the same as in FIG. 15B.

  From FIG. 16B, the user can intuitively grasp the relative position information of the ultrasonic tomographic image 601 in the three-dimensional existence range by referring to the density information of the dotted line added to the outline shape of the existence range. In addition, when coloring the contour shape, it may be difficult to visually recognize the contour shape as the color becomes lighter. Even if changes, the dotted line can be easily recognized. When the dotted lines are expressed with a slightly coarse density as shown in FIG. 15B, the ultrasonic tomographic image 601 can be easily located near the start position 304 in FIG. 13 with respect to the three-dimensional existence range. I understand.

  Further, display information in which color information representing relative position information is colored in an internal region of a two-dimensional existence range may be drawn on an ultrasonic tomographic image. FIG. 16C shows an ultrasonic tomographic image on which a two-dimensional existence range in which relative position information is drawn in an internal region is superimposed. In FIG. 16C, 601 and 602 are the same as those in FIG. 16A, and R1 is the same as that in FIG. 15C.

  From FIG. 16C, the user can intuitively grasp the relative position information of the ultrasonic tomographic image 601 by referring to the color information colored in the internal region of the existence range. Further, by coloring the inside of the region, it becomes easier to visually recognize the color information as compared with the case of coloring the contour shape. At this time, it is preferable that the inner region is colored so transparent that the user can search for the corresponding lesion 602.

  In contrast to the case of FIG. 16C, display information in which color information indicating relative position information is colored in an external region of the two-dimensional existence range may be drawn on the ultrasonic tomographic image. FIG. 16D shows an ultrasonic tomographic image on which a two-dimensional existence range in which relative position information is drawn in an external region is superimposed. In FIG. 16D, 601 and 602 represent the same as in FIG. 16A, and R2 represents the same as in FIG.

  From FIG. 16D, the user can intuitively grasp the relative position information of the ultrasonic tomographic image 601 by referring to the color information colored in the outer region of the existence range. Further, in this case, since the inside of the region is not colored, the inside of the region can be observed more easily than in the case where the inside of the region is colored, and the corresponding lesion portion can be easily searched. At this time, it is preferable that the outer region is colored so transparent that the state of the ultrasonic tomographic image outside the two-dimensional existence range can be observed.

  Note that it is determined whether or not the tomographic image intersects with the corresponding region itself obtained in step S11050. If the tomographic image intersects, the intersecting region on the tomographic image is superimposed and displayed on the tomographic image. You can do it.

(S11070: Combining a tomographic image and a sectional image)
In step S11070, the information processing apparatus 900 synthesizes the cross-sectional image obtained in step S11040 and the image obtained by superimposing the existence range of the corresponding lesion on the ultrasonic tomographic image obtained in step S11060 as the processing of the image synthesis unit 940. . For example, an image in which these images are arranged side by side is generated. Then, as a process of the display control unit 950, the synthesized image is displayed on the display unit 960. Further, if necessary, this is output to the outside via the I / F 1009 and further stored in the RAM 1002 as a state usable by other applications.

(S11080: Determination of whether to end the entire process)
In step S11080, the information processing apparatus 900 determines whether to end the entire process. For example, an end instruction input by the user pressing a predetermined key (end key) on the keyboard 1004 is acquired. When it is determined that the processing is to be ended, the entire processing of the information processing apparatus 900 is ended. On the other hand, if it is not determined that the process is to be terminated, the process returns to step S11020, and the processes after step S11020 are executed again on the ultrasonic tomographic image that is newly imaged. As described above, the processing of the information processing apparatus 900 is executed.

  As described above, the information processing apparatus according to the present embodiment displays the two-dimensional existence range to which the relative position information of the displayed ultrasonic tomographic image is added on the ultrasonic tomographic image, so that the user can It is possible to easily grasp the current position of the ultrasonic tomographic image with respect to the three-dimensional existence range while paying attention to the two-dimensional existence range on the acoustic tomographic image. Therefore, the user can easily grasp how much the existence range to be searched is left behind, and can efficiently search and identify the corresponding lesion part.

(Sixth embodiment)
In the information processing system according to the fifth embodiment, the mapping of the position information of the ultrasonic tomographic image to the color table is performed based on the start position and end position of the three-dimensional existence range. On the other hand, in the present embodiment, in addition to the start position and end position, mapping to a color table is performed based on position information representative of a three-dimensional existence range (also referred to as a representative position, which is an example of a predetermined position). . The configuration of the information processing system according to the present embodiment is the same as that of FIG. 11, but only part of the processing of the display information generation unit 937 is different from that of the fifth embodiment. The processing flow of the information processing system according to the present embodiment is the same as that in FIG. 12, but only part of the processing in step S11054 is different from the fifth embodiment. Hereinafter, the information processing system according to the present embodiment will be described only with respect to differences from the fifth embodiment.

  In step S <b> 11054, the information processing apparatus 1000 processes the display information generation unit 937 as shape information representing the outline of the two-dimensional existence range acquired from the three-dimensional existence range calculation unit 935 and the super information acquired from the position information calculation unit 936. Display information in which relative position information is added to the two-dimensional existence range is generated based on the position information of the acoustic tomographic image and the position information representing the three-dimensional existence range. A specific generation method is shown below. In the present embodiment, a case will be described in which the position corresponding to the center of gravity of the three-dimensional existence range is used as position information representing the three-dimensional existence range.

  First, the display information generation unit 937 sets a color table reflecting the position of the center of gravity of the three-dimensional existence range, and associates the relative position information of the ultrasonic tomographic image with the color table.

  FIG. 14B is a diagram illustrating the association of relative position information with the color table reflecting the position corresponding to the center of gravity. Ax, P0, P1, and Ps represent the same as in FIG. Pg represents the position corresponding to the center of gravity described above. At this time, when setting the color table between the positions P0 and P1 along the axis Ax, the color table in which the color continuously changes according to the position between P0 and Pg and between Pg and P1. Set. Specifically, a color table is set so that the color intensity becomes maximum at the position of Pg, and the color intensity decreases as it approaches P0 and P1 from Pg. In step S10052, color information on the color table corresponding to the relative position information of the tomographic image corresponding position Ps normalized between P0 and P1 is acquired.

  T2 represents a color table reflecting the position of the center of gravity corresponding position Pg, and C2 represents color information corresponding to the relative position information. For example, when the color table T2 represents a color table in which the intensity of blue changes, and the relative position information of Ps is 0.3, Ps is mapped to a position on the color table close to the position 0.5 of Pg. Therefore, the color information C2 is represented in a slightly dark blue color. By referring to this color information, it can be intuitively understood that the ultrasonic tomographic image is relatively close to the position of the center of gravity of the three-dimensional existence range.

  Alternatively, different typical colors may be set at the positions P0, Pg, and P1, and a color table in which the colors continuously change according to the position may be set. For example, red is set for P0, green is set for Pg, and blue is set for P1, and a color table that changes from red to green between P0 and Pg and from green to blue is set between Pg and P1. Thereby, it is intuitively understood which position of the ultrasonic tomographic image is close to P0, Pg, and P1. In addition, since the color scheme is different between P0 and Pg, and between Pg and P1, it is easy to determine which side of the ultrasonic tomographic image is P0 or P1 with respect to the barycentric position of the three-dimensional existence range. . When the relative position information of Ps is 0.3, it is represented by a color close to green between red and green. Of course, a color table in which P0, Pg, and P1 are divided into a plurality of stages and different colors are assigned to the respective stages may be used.

  Next, as in the fifth embodiment, the information processing apparatus 1000 is associated with the shape information of the two-dimensional existence range based on the color information corresponding to the relative position information of the acquired ultrasonic tomographic image. Generate range display information.

  Note that the position information representing the three-dimensional existence range is not limited to the position of the center of gravity. For example, when a plane parallel to the ultrasonic tomographic image 302 is translated along the orthogonal axis Ax, the area of the region where the three-dimensional existence range 301 and the ultrasonic tomographic image 302 intersect (two-dimensional existence range 303) is as follows. The position of the plane when the maximum is obtained and the position plotted on the orthogonal axis Ax may be used as position information representing the three-dimensional existence range.

  Thereby, it is possible to read how close the position of the ultrasonic tomographic image in the three-dimensional existence range is to the center of gravity position of the three-dimensional existence range, and the position can be grasped more intuitively.

(Modification)
In the fifth embodiment and the sixth embodiment, the example in which the three-dimensional existence range in which the corresponding lesion portion can exist is a three-dimensional region has been described. However, the embodiment is not limited to this. For example, a region representing a site of interest such as a lesion or organ in an X-ray CT image or MRI image may be a three-dimensional region. This replaces the ultrasonic tomographic image acquired in step S11020 in the above embodiment with the slice image currently displayed, and further converts the three-dimensional existence range acquired in step S11050 to the three-dimensional region of the target region. This can be realized by replacement.

  As a result, when a doctor interprets an X-ray CT image or an MRI image, the position of the attention site in the slice image currently displayed is located in the three-dimensional region of the attention site in the orthogonal direction of the slice image. Intuitive grasp. Therefore, it is easy to know how much of the three-dimensional region of the target region to be interpreted is left behind, so that it is possible to interpret efficiently.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 100 Information processing apparatus 110 Tomographic image acquisition part 112 Position and orientation acquisition part 120 Three-dimensional image data acquisition part 122 Attention area acquisition part 123 Error acquisition part 130 Cross-sectional image generation part 135 Existence range calculation part 140 Image composition part 150 Display control part

Claims (18)

Two-dimensional image acquisition means for acquiring a two-dimensional image of the subject;
Display control means for displaying on the display means an error range by the projection including a projection area obtained by projecting a predetermined area of the three-dimensional image of the subject on a plane including the two-dimensional image; ,
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the error range by the projection is a range in which a region of the two-dimensional image corresponding to the predetermined region can exist. Coordinate conversion means for converting the position of the predetermined region from the three-dimensional coordinate system of the three-dimensional image to the three-dimensional coordinate system of the two-dimensional image;
Calculating means for calculating an error range due to the projection based on an error due to the coordinate transformation;
The information processing apparatus according to claim 1, further comprising: Setting means for setting a region larger than the predetermined region and including the predetermined region in the three-dimensional image;
The information processing apparatus according to claim 3, wherein the coordinate conversion unit converts the position of the region from a three-dimensional coordinate system of the three-dimensional image to a three-dimensional coordinate system of the two-dimensional image. Designating means for designating a predetermined location from within the error range by the projection;
Correction means for correcting an error due to the projection in the two-dimensional image based on a difference between the predetermined portion and the projection region;
The information processing apparatus according to claim 1, further comprising:   The display control unit causes the display unit to display a mark indicating that the region of the two-dimensional image corresponds to the predetermined region at the predetermined position on the two-dimensional image. The information processing apparatus according to claim 5.   When the predetermined portion is displayed on the two-dimensional image without displaying the error range due to the projection on the two-dimensional image of the subject acquired again after being corrected by the correcting means. The information processing apparatus according to claim 6, further comprising a switching unit that switches to a mode in which the mark is displayed on the display unit while being superimposed on the two-dimensional image.   Cross-sectional image acquisition means for acquiring a cross-sectional image of the three-dimensional image that passes through the predetermined region and corresponds to the posture based on the position of the predetermined region in the three-dimensional image and the posture of the two-dimensional image. The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. An ultrasonic probe that detects ultrasonic waves from the subject to acquire the two-dimensional image, and a position and orientation acquisition unit that acquires the position and orientation of the ultrasonic probe;
The information processing apparatus according to claim 8, wherein an attitude of the two-dimensional image is acquired based on a position and an attitude of the ultrasonic probe.   The display control means causes the display means to display an image in which an error range due to the projection is overlaid on the two-dimensional image and an image in which the predetermined region is overlaid on the cross-sectional image. The information processing apparatus according to 8 or 9. A designation means for designating a predetermined portion of the three-dimensional image of the subject;
Two-dimensional image acquisition means for acquiring a two-dimensional image of the subject;
Display control means for displaying on the display means a display form indicating a two-dimensional area in the two-dimensional image among three-dimensional areas including a place corresponding to the predetermined place according to the position information of the two-dimensional image;
An information processing apparatus comprising:   The information processing apparatus according to claim 11, wherein the display control unit causes the display unit to display the two-dimensional area in a display form indicating color information associated with the position information.   The information processing apparatus according to claim 11, wherein the display control unit causes the display unit to display an outline of the two-dimensional area associated with the position information. According to the position information of the tomographic image, the display form changing means for changing to the display form associated with the position information,
The information processing apparatus according to claim 11, wherein the display control unit causes the display unit to display the two-dimensional area in a display form associated with the position information.   15. The calculation apparatus according to claim 11, further comprising a calculation unit configured to calculate position information of the tomographic image based on a positional relationship between a predetermined position in the two-dimensional region and the tomographic image. Information processing device.   The calculation means specifies a start position, an end position, and a representative position of the two-dimensional area as the predetermined position, and based on a positional relationship between the start position, the end position, the representative position, and the tomographic image. The information processing apparatus according to claim 15, wherein the position information is calculated. A location corresponding to the predetermined location is represented by a coordinate system of the tomographic image,
The display control unit causes the display unit to display a display form indicating the two-dimensional region on the tomographic image in accordance with position information of the tomographic image in a coordinate system of the tomographic image. Item 17. The information processing apparatus according to any one of Items 11 to 16. Two-dimensional image acquisition means for acquiring a two-dimensional image of the subject;
Display control means for causing a display means to display a display form indicating a two-dimensional area in the two-dimensional image among three-dimensional areas including a position corresponding to a predetermined position of the three-dimensional image of the subject;
An information processing apparatus comprising:

Images