WO2017072916A1

WO2017072916A1 - Image processing device and image processing method, and computer program

Info

Publication number: WO2017072916A1
Application number: PCT/JP2015/080576
Authority: WO
Inventors: 宏美武居; 達也織茂
Original assignee: パイオニア株式会社
Priority date: 2015-10-29
Filing date: 2015-10-29
Publication date: 2017-05-04

Abstract

Through the present invention, using a two-dimensional image and a three-dimensional image obtained by capturing images of the same object, a steric distribution in the three-dimensional image of an object included in the two-dimensional image is specified. An image processing device (15) is provided with an acquiring means for acquiring a two-dimensional image and a three-dimensional image generated by capturing images of the same object, a first specifying means (1514) for specifying a first image portion representing the object from within the two-dimensional image, and a second specifying means (1516) for specifying a third image component in the three-dimensional image having a feature of the second image portion in the three-dimensional image corresponding to the first image portion.

Description

Image processing apparatus, image processing method, and computer program

The present invention relates to a technical field of an image processing apparatus and an image processing method for performing image processing on a two-dimensional image and a three-dimensional image obtained by imaging the same object, and a computer program.

There is known a surgery support apparatus that superimposes and displays an observation image of a human body photographed by a video camera or the like during surgery on a three-dimensional image of the human body photographed by an MRI apparatus or the like before surgery (for example, Patent Document 1). .

JP 2010-274044 A

In recent years, a technique has been proposed in which a spectral image captured by a hyperspectral camera or the like is analyzed to identify an image portion showing a tumor (hereinafter referred to as “tumor image portion”) in the spectral image. The inventors of the present application display the image part corresponding to the tumor in a three-dimensional image of the human body based on the identified tumor image part in a manner that can be distinguished from other image parts. We are considering the development of technology that supports the recognition of

Therefore, a technique for superimposing and displaying the identified tumor image portion on the image portion corresponding to the tumor in the three-dimensional image by using the technique described in Patent Document 1 is assumed. However, since the spectral image is a two-dimensional image (that is, a planar image), the tumor image portion is also a two-dimensional image. Therefore, from the spectral image, only the outer surface of the tumor (that is, the portion of the tumor that can be observed from the outside) is identified. That is, it is difficult to specify how the tumor is three-dimensionally distributed from the spectral image. Therefore, by simply using the surgery support apparatus described in Patent Document 1 and displaying the tumor image portion superimposed on the three-dimensional image, only a three-dimensional image representing the outer surface of the tumor is displayed. In other words, even if a tumor image portion that merely represents the outer surface of a tumor is simply superimposed and displayed on a three-dimensional image that can three-dimensionally show the outer surface and the inside of the human body, This causes a technical problem that it is impossible to display a three-dimensional image that clearly shows the distribution.

Note that the above-described technical problem is not limited to a case where a tumor image is superimposed and displayed on a three-dimensional image of a human body, but can also occur in a case where an arbitrary two-dimensional image is superimposed and displayed on an arbitrary three-dimensional image.

The present invention has been made in view of, for example, the above-described problems, and a two-dimensional image obtained by photographing the same object and a three-dimensional image are used to convert the object included in the two-dimensional image into a three-dimensional image. It is an object of the present invention to provide an image processing apparatus, an image processing method, and a computer program that can specify how the image is three-dimensionally distributed in an image.

An image processing apparatus for solving the above problem represents a two-dimensional image and a three-dimensional image generated by photographing the same object, and represents the object among the two-dimensional image. A first specifying means for specifying a first image portion; a second for specifying a third image portion in the three-dimensional image having characteristics of the second image portion in the three-dimensional image corresponding to the first image portion; Specifying means.

An image processing method for solving the above problem represents a two-dimensional image and a three-dimensional image generated by capturing the same target object, and represents the target object among the two-dimensional images. A first specifying step of specifying a first image portion; and a second specifying of a third image portion in the three-dimensional image having characteristics of the second image portion in the three-dimensional image corresponding to the first image portion. A specific process.

A computer program for solving the above problem causes a computer to execute the above-described image processing method.

It is a block diagram which shows the structure of a surgery assistance system. It is a block diagram which shows the structure of an image processing apparatus. It is a flowchart which shows the flow of operation | movement of a surgery assistance system. It is a schematic diagram which shows an MRI image. It is a top view which shows the spectral image by which the tumor image part is specified. It is a schematic diagram which shows the concept of the operation | movement which pinpoints an outer surface image part. It is a schematic diagram which shows the concept of the operation | movement which pinpoints an internal image part.

Hereinafter, embodiments of an image processing apparatus, an image processing method, and a computer program will be described in order as modes for carrying out the invention.

(Embodiment of Image Processing Device)
<1>
The image processing apparatus according to the present embodiment includes a acquiring unit that acquires a two-dimensional image and a three-dimensional image generated by photographing the same object, and a first image representing the object among the two-dimensional images. First specifying means for specifying a portion; and second specifying means for specifying a third image portion in the three-dimensional image having features of the second image portion in the three-dimensional image corresponding to the first image portion; Is provided.

According to the image processing apparatus of the present embodiment, first, the first specifying unit specifies the first image portion representing the object in the two-dimensional image. Since the first image part is at least a part of the two-dimensional image, the first image part represents at least a part of the object planarly (in other words, two-dimensionally). That is, the first image portion represents at least a part of the outer surface of the object.

After that, the second specifying means has a third image portion having the same feature as the feature of the second image portion in the three-dimensional image corresponding to the first image portion in the three-dimensional image based on the first image portion. Is identified. Since the first image part represents at least a part of the outer surface of the object, the second image part corresponding to the first image part is also visible from the outer surface of the object (that is, the object is visible from the outside). Part). Furthermore, since the second image portion is at least a part of the three-dimensional image, the second image portion represents at least a part of the outer surface of the object in a three-dimensional manner (in other words, three-dimensionally).

Here, since the characteristics of the second image portion and the features of the third image portion are the same, it can be estimated that the third image portion also represents at least a part of the object, like the second image portion. It is. Furthermore, the third image portion has the same characteristics as the features of the second image portion, instead of being directly identified from the first image portion, which only represents at least part of the outer surface of the object. The image portion is specified by searching from the three-dimensional image. For this reason, it can be estimated that the third image portion represents at least the inside of the object (that is, the portion that cannot be visually recognized from the outside of at least a part of the object). Furthermore, since the third image portion is at least a part of the three-dimensional image, the third image portion three-dimensionally represents the inside of the object.

As described above, the image processing apparatus according to the present embodiment specifies not only the second image part that stereoscopically represents at least a part of the outer surface of the object, but also the third image part that stereoscopically represents the inside of the object. can do. For this reason, the image processing apparatus of this embodiment can specify how the objects included in the two-dimensional image are three-dimensionally distributed in the three-dimensional image. As a result, any display device supports the user's recognition of the object by displaying the second and third image parts corresponding to the object in the three-dimensional image in a manner distinguishable from other image parts. can do.

In the above description, the third image portion represents the inside of the object. However, the third image portion may include a second image portion representing the outer surface of the object. In other words, an image portion obtained by combining an image portion representing the inside of the object and an image portion representing the outer surface of the object may be used as the third image portion. This is because the outer shell of the third image portion substantially corresponds to the second image portion.
<2>
In another aspect of the image processing apparatus of the present embodiment, the second image portion represents at least a part of the outer surface of the object in the three-dimensional image, and the third image portion is a portion of the three-dimensional image. Of these, at least the inside of the object is represented.

According to this aspect, the second specifying means has at least the inside of the object having the same feature as the feature of the second image portion that represents at least a part of the outer surface of the object specified by the second specifying means. Three image portions can be specified.

<3>
In another aspect of the image processing apparatus of the present embodiment, the image processing apparatus further includes a third specifying unit that specifies the second image portion, and the two-dimensional image is generated by shooting a shooting target including the target object, The third specifying means is (i) present on a path of light reaching each pixel of an image sensor included in an imaging device that generates the two-dimensional image by photographing the photographing target among the three-dimensional images. Identifying a fourth image portion corresponding to an outer surface of the imaging target, and (ii) based on a first parameter indicating a state of an optical system included in the imaging device and a second parameter indicating an installation environment of the imaging device, The fourth image portion is associated with the two-dimensional image, and (iii) the second image portion is specified based on a result of the association between the fourth image portion and the two-dimensional image.

According to this aspect, the third specifying means can preferably specify the second image portion. In particular, as will be described later, the third specifying means does not specify the position in the two-dimensional image of the marker that may be placed on the shooting target in order to specify the position of the shooting target or the object, The second image portion can be specified.

<4>
In another object of the image processing apparatus of the present embodiment, the display device is controlled to display another three-dimensional image representing the second image portion and the third image portion in a manner distinguishable from other image portions. And a control means.

According to this aspect, under the control of the control means, the display device can display the second and third image portions corresponding to the object in the three-dimensional image in a manner distinguishable from the other image portions. .

<5>
In another aspect of the image processing apparatus of the present embodiment, the object includes at least one of a predetermined part of the subject, an affected area of the subject, and a tumor part of the subject.

According to this aspect, the image processing apparatus has the same characteristics as the characteristics of the second image portion representing the outer surface of the predetermined part, affected part, or tumor part of the subject, and the inside of the predetermined part, affected part, or tumor part of the subject. A third image portion representing can be specified.

<6>
In another aspect of the image processing apparatus of the present embodiment, the three-dimensional image is an MRI (Magnetic Resonance Imaging) image that three-dimensionally shows the outer surface and the inside of the subject, and the two-dimensional image is the subject FIG.

According to this aspect, the image processing apparatus includes not only the second image portion representing the outer surface of the subject including the outer surface of the object based on the MRI image and the spectroscopic image, but also the inside of the subject including the inside of the object. It is also possible to specify the third image portion representing

(Embodiment of Image Processing Method)
<7>
The image processing method according to the present embodiment includes an acquisition step of acquiring a two-dimensional image and a three-dimensional image generated by imaging the same object, and a first image representing the object among the two-dimensional images. A first specifying step of specifying a portion; and a third image portion in the three-dimensional image having features of the second image portion in the two-dimensional image corresponding to the first image portion of the three-dimensional image. A second specifying step of specifying.

According to the image processing method of the present embodiment, various effects enjoyed by the above-described image processing apparatus of the present embodiment can be suitably enjoyed.

Incidentally, in response to various aspects adopted by the image processing apparatus of the present embodiment, the image processing method of the present embodiment may adopt various aspects.

(Embodiment of computer program)
<8>
The computer program of this embodiment causes a computer to execute the image processing method of this embodiment described above.

According to the computer program of this embodiment, various effects enjoyed by the above-described image processing apparatus of this embodiment can be suitably enjoyed.

Incidentally, in response to various aspects adopted by the image processing apparatus of this embodiment, the computer program of this embodiment may adopt various aspects. The computer program may be recorded on a recording medium.

Such an operation and other advantages of the present embodiment will be clarified from examples described below.

As described above, the image processing apparatus according to this embodiment includes an acquisition unit, a first specifying unit, and a second specifying unit. The image processing method of the present embodiment includes an acquisition process, a first specifying process, and a second specifying process. The computer program of this embodiment causes a computer to execute the image processing method of this embodiment described above. Therefore, it is specified how the objects included in the two-dimensional image are three-dimensionally distributed in the three-dimensional image using the two-dimensional image and the three-dimensional image obtained by photographing the same object. The

Hereinafter, embodiments of an image processing apparatus, an image processing method, and a computer program will be described with reference to the drawings. In the following description, an example in which the image processing apparatus, the image processing method, and the computer program are applied to a surgery support system for supporting a surgeon performing surgery on a patient will be described. In this case, the image processing apparatus, the image processing method, and the surgery support system to which the computer program is applied include an MRI image obtained by imaging a patient with an MRI (Magnetic Resonance Imaging) apparatus (that is, an external surface and an internal surface of the patient). A three-dimensional image that represents the structure of the patient three-dimensionally) and a tumor image portion obtained by analyzing the spectral image captured by the hyperspectral camera (that is, a two-dimensional image that planarly represents the outer surface of the patient's tumor) Thus, it is specified how the tumor represented by the tumor image portion is three-dimensionally distributed in the MRI image. However, the image processing apparatus, the image processing method, and the computer program use a two-dimensional image and a three-dimensional image obtained by photographing the same object, and the object included in the two-dimensional image is a three-dimensional image. The present invention may be applied to any device that specifies how the three-dimensional distribution is distributed within the network.

(1) Configuration of Surgery Support System 1 First, the configuration of the surgery support system 1 of the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing the configuration of the surgery support system 1 of the present embodiment.

As shown in FIG. 1, the surgery support system 1 includes an MRI apparatus 11, a hyperspectral camera 12, a pointer 13, a position measuring device 14, an image processing device 15, and a display device 16.

The MRI apparatus 11 generates an MRI image that is a three-dimensional image that three-dimensionally represents the external surface and internal structure of the patient by imaging the external surface and internal structure of the patient. An MRI image is an aggregate of patient tomographic images (in other words, two-dimensional slice images). However, the MRI image may be any image as long as the outer surface and the internal structure of the patient can be three-dimensionally represented. For example, the MRI image may be a three-dimensional model image or three-dimensional volume data instead of an aggregate of tomographic images. The MRI image generated by the MRI apparatus 11 is input to the image processing apparatus 15.

In this embodiment, it is assumed that the patient is an operation target. In this case, a part of the patient's body becomes the surgical site. Furthermore, in this embodiment, the surgical site is a part of the patient's head. In addition, in this embodiment, the operation is performed on a tumor in the patient's head. In this case, a tumor is present at the surgical site. The patient and the surgical site are specific examples of “imaging target”, and the tumor is a specific example of “target”. However, the surgery support system 1 of the present embodiment is used when performing an arbitrary operation (for example, an operation performed on an arbitrary part of a patient) different from an operation performed on a tumor of the patient's head. May be. Furthermore, the patient may be a human or any living body other than a human (for example, an animal).

At least three markers m1 (four markers m1 in the example shown in FIG. 1) are installed around the surgical site. The marker m1 is a marker (for example, Fiducial Marker) that can be imaged by the MRI apparatus 11.

The hyperspectral camera 12 shoots a patient to generate a spectral image of the patient (that is, a collection of images for each light spectrum (wavelength)). The spectral image generated by the hyperspectral camera 12 is input to the image processing device 15.

The hyperspectral camera 12 is provided with at least three markers m2 (three markers m2 in the example shown in FIG. 1). The marker m2 is used for the position measurement device 14 to measure the position of the hyperspectral camera 12 (particularly, the three-dimensional position and orientation in the real space coordinate system that defines the three dimensions in the real space where the patient actually exists). Used.

The pointer 13 is a surgical tool that points to the marker m1 installed around the surgical site. The surgeon operates the pointer 13 to point the marker m1 at the tip of the pointer 13. The pointer 13 is provided with at least three markers m3 (three markers m3 in the example shown in FIG. 1). The marker m3 is used for the position measuring device 14 to measure the position of the pointer 13 (particularly, the three-dimensional position and orientation in the real space coordinate system).

The position measuring device 14 measures the positions of the marker m2 and the marker m3 (particularly, a three-dimensional position in the real space coordinate system). For example, the position measuring device 14 includes a stereo camera (two-lens camera) and an LED (Light Emitting Diode). The LED emits measurement light (for example, infrared rays) toward each of the marker m2 and the marker m3. The stereo camera detects the measurement light reflected by each of the marker m2 and the marker m3. The detection result of the measurement light by the position measurement device 14 (that is, the measurement result of the measurement light) is input to the image processing device 15. The image processing device 15 specifies the position of the hyperspectral camera 12 and the position of the pointer 13 (specifically, the position of the marker m1 as will be described in detail later) based on the measurement result of the position measuring device 14.

The image processing device 15 performs image processing on the MRI image and the spectral image. The image processing apparatus 15 includes a CPU (Central Processing Unit) 151 and a memory 152. The memory 152 stores a computer program for causing the image processing apparatus 15 to perform image processing. When the computer program is executed by the CPU 151, a logical processing block for performing image processing is formed inside the CPU 151. However, the computer program may not be recorded in the memory 152. In this case, the CPU 151 may execute a computer program downloaded via a network.

Specifically, as illustrated in FIG. 2, the image processing apparatus 15 includes an MRI image acquisition unit 1511 that is a specific example of “acquisition unit”, a marker, and a logical processing block formed in the CPU 151. A position specifying unit 1512; a spectroscopic image acquiring unit 1513 which is a specific example of “acquiring means”; a tumor specifying unit 1514 which is a specific example of “first specifying means”; a marker position specifying unit 1515; A specific example of “two specifying means” and “third specifying means”, an association processing unit 1516, a three-dimensional model generating unit 1517 that is a specific example of “control means”, and a specific example of “control means” And a three-dimensional viewer processing unit 1518.

The MRI image acquisition unit 1511 acquires the MRI image generated by the MRI apparatus 11.

The marker position specifying unit 1512 analyzes the MRI image acquired by the MRI image acquisition unit 1511 to thereby detect the position of the marker m1 included in the MRI image (particularly in the MRI coordinate system that defines the three-dimensional position in the MRI image). 3D position).

The spectral image acquisition unit 1513 acquires the spectral image generated by the hyperspectral camera 12.

The tumor identification unit 1514 identifies the position of the tumor (particularly, the two-dimensional position in the spectral coordinate system that defines the two-dimensional position in the spectral image) by analyzing the spectral image acquired by the spectral image acquisition unit 1513. . That is, the tumor specifying unit 1514 specifies a tumor image portion that is an image portion representing a tumor in the spectral image. The tumor image portion is a specific example of “first image portion”.

The marker position specifying unit 1515 specifies the position of the marker m2 based on the measurement result of the position measuring device 14. In addition, the marker position specifying unit 1515 specifies the position of the marker m1 (particularly, the three-dimensional position in the real space coordinate system) based on the position of the marker m3 placed on the pointer 13 pointing to the marker m1.

The association processing unit 1516 mainly performs three association processes described below.

Specifically, the association processing unit 1516 performs, as the first association processing, the position of the marker m1 in the MRI coordinate system specified by the marker position specifying unit 1512 and the real space coordinates specified by the marker position specifying unit 1515. A process of associating the position of the marker m1 with the system is performed.

The association processing unit 1516 performs a process of associating the MRI image and the spectral image (particularly, the tumor image portion) as the second association process. More specifically, the association processing unit 1516 identifies an image portion corresponding to the tumor image portion (that is, an image portion corresponding to the outer surface of the tumor) in the MRI image based on the tumor image portion. Note that a specific example of the process of specifying the image portion corresponding to the tumor image portion in the MRI image will be described in detail later, and thus the description thereof is omitted here.

Here, since the spectral image is a two-dimensional image, the tumor image portion is also a two-dimensional image. Therefore, the tumor image portion is an image portion that two-dimensionally represents the outer surface of the tumor (in other words, the outer shell) as a plane. In addition, the outer surface of a tumor means the part visible from the outside among tumors. Therefore, the image portion corresponding to the tumor image portion in the MRI image is also an image portion representing the outer surface of the tumor. However, since the MRI image is a three-dimensional image, the image portion corresponding to the tumor image portion in the MRI image is an image portion that three-dimensionally (in other words, three-dimensionally) with the outer surface of the tumor as a curved surface. . In the following description, an image portion corresponding to the tumor image portion in the MRI image is referred to as an “outer surface image portion”. The outer surface image portion is a specific example of “second image portion”. The outer surface image portion represents at least a part of the outer surface of the patient's tumor (typically, the outer surface of the tumor portion included in the imaging region of the hyperspectral camera 12 of the patient's tumor).

As the third association process, the association processing unit 1516 identifies an image portion corresponding to the inside of the tumor (that is, an image portion representing a structure inside the tumor) based on the outer surface image portion. Process. In addition, the inside of a tumor means the part which cannot be visually recognized from the outside among tumors. Note that a specific example of the process of specifying the image portion corresponding to the inside of the tumor in the MRI image will be described later in detail, and the description thereof is omitted here.

The image part corresponding to the inside of the tumor in the MRI image is an image part that represents the inside of the tumor three-dimensionally (in other words, three-dimensionally) as a three-dimensional structure. In the following description, an image portion corresponding to the inside of the tumor in the MRI image is referred to as an “internal image portion”. The internal image portion is a specific example of “third image portion”.

The three-dimensional model generation unit 1517 determines whether the tumor is in the other part based on the processing result by the association processing unit 1516 (that is, the identification result of the outer surface image portion and the inner image portion) and the MRI image acquired by the MRI image acquisition unit 1511. A three-dimensional model of a patient expressed in a manner distinguishable from the above is generated.

The 3D viewer processing unit 1518 generates an observation image that is observed when the surgeon observes the 3D model generated by the 3D model generation unit 1517 from a desired viewpoint.

1 again, the display device 16 displays the observation image generated by the three-dimensional viewer processing unit 1518. As a result, the surgeon can recognize how the tumor is three-dimensionally distributed inside the patient. The display device 16 may display other arbitrary information (for example, information related to surgery support).

(2) Operation of Surgery Support System 1 Next, the operation of the surgery support system 1 (mainly the operation of the image processing device 15) will be described with reference to FIG. FIG. 3 is a flowchart showing an operation flow of the surgery support system 1.

As shown in FIG. 3, first, the MRI image acquisition unit 1511 acquires an MRI image (step S11). Specifically, the marker m1 is first installed around the patient's surgical site. Thereafter, the patient lies on the bed provided in the MRI apparatus 11. Thereafter, the MRI apparatus 11 sequentially takes tomographic images (see FIG. 4) of the patient. That is, the MRI apparatus 11 generates an MRI image. As a result, the MRI image acquisition unit 1511 acquires an MRI image.

Thereafter, the patient is transferred from the bed provided in the MRI apparatus 11 to the operating table. In this case, the patient is transported so that the position where the marker m1 is installed does not change (that is, the position where the marker m1 is installed is fixed). However, the bed provided in the MRI apparatus 11 may be used as an operating table. The position where the marker m1 is installed does not change (is fixed) while the series of operations shown in FIG. 3 is performed.

The marker m1 may be directly driven into the bone of the patient's head before the MRI image is acquired so that the position where the marker m1 is installed does not change. In this case, the marker m1 may be driven into the skull that has opened the patient lying on the bed, turned over the scalp, and appeared.

Thereafter, the marker position specifying unit 1512 specifies the position of the marker m1 included in the MRI image by analyzing the MRI image acquired in step S11 (step S12). That is, the marker position specifying unit 1512 specifies the three-dimensional position of the marker m1 in the MRI coordinate system. For example, the marker position specifying unit 1512 specifies an image portion corresponding to the marker m1 from the MRI image using a matching process or the like. Thereafter, the marker position specifying unit 1512 specifies the position of the image portion corresponding to the marker m1.

Before or after the operation of step S12 or in parallel, the marker position specifying unit 1515 specifies the position of the marker m1 based on the measurement result of the position measuring device 14 (step S13). That is, the marker position specifying unit 1515 specifies the three-dimensional position of the marker m1 in the real space coordinate system. Specifically, the surgeon operates the pointer 13 to point the marker m1 at the tip of the pointer 13. Under the situation where the marker m1 is pointed by the tip of the pointer 13, the position measurement device 14 detects the measurement light reflected by the marker m3. The marker position specifying unit 1515 can specify the position of the marker m3 based on the measurement result of the position measuring device 14. When the position of the marker m3 is specified, the position of the pointer 13 (however, the “position” in this case includes the direction of the pointer 13) is also specified. Here, under the situation where the tip of the pointer 13 points to the marker m1, the position of the pointer 13 substantially corresponds to the position of the marker m1. Furthermore, the shape of the pointer 13 (the positional relationship between the positions and the tips of at least three markers m3) is known. Therefore, the marker position specifying unit 1515 can specify the position of the marker m1 in the real space coordinate system by specifying the position of the marker m3 (that is, specifying the position of the pointer 13). The above operation is performed for all the markers m1.

Thereafter, the association processing unit 1516 associates the position of the marker m1 in the MRI coordinate system identified in step S12 with the position of the marker m1 in the real space coordinate system identified in step S13 (step S14). As a result, the association processing unit 1516 calculates a transformation matrix for converting the position in the MRI coordinate system to a position in the real space coordinate system or the position in the real space coordinate system to a position in the MRI coordinate system. be able to. That is, the association processing unit 1516 can specify a position in the real space coordinate system corresponding to an arbitrary position in the MRI coordinate system. Similarly, the association processing unit 1516 can specify a position in the MRI coordinate system corresponding to an arbitrary position in the real space coordinate system.

Specifically, the association processing unit 1516 performs the position (Xm1, Ym1, Zm1) of the first marker m1 in the MRI coordinate system and the position (Xa1) of the same first marker m1 in the real space coordinate system. , Ya1, Za1). As a result of this association, the association processing unit 1516 generates an equation of (Xm1, Ym1, Zm1) = conversion matrix × (Xa1, Ya1, Za1). The association processing unit 1516 performs this operation for all the markers m1. As a result, if the required number of equations for deriving the transformation matrix is generated, the transformation matrix is identified. As a result, the association processing unit 1516 can associate the MRI image with the real space. That is, the association processing unit 1516 can associate the patient in the MRI image with the patient located in the real space.

Thereafter, the spectral image acquisition unit 1513 acquires a spectral image (step S21). Specifically, the hyperspectral camera 12 images the patient with the patient positioned on the operating table. That is, the hyperspectral camera 12 generates a spectral image. As a result, the spectral image acquisition unit 1513 acquires a spectral image.

Thereafter, the tumor identification unit 1514 identifies the position of the tumor by analyzing the spectral image acquired in step S21 (step S22). That is, as shown in FIG. 5, the tumor specifying unit 1514 specifies the two-dimensional position of the tumor in the spectral coordinate system that defines the two-dimensional position in the spectral image. That is, the tumor specifying unit 1514 specifies a tumor image portion that is an image portion representing a tumor in the spectral image (step S22). In addition, arbitrary operation | movement may be used as an operation | movement which pinpoints the position (tumor image part) of a tumor from a spectral image. For example, a known operation may be used as an operation for specifying the position of the tumor from the spectral image.

Thereafter, the image processing apparatus 15 associates the tumor image portion specified from the spectral image with the MRI image, thereby corresponding to the outer surface image portion corresponding to the outer surface of the tumor in the MRI image and the inside of the tumor in the MRI image. An operation for specifying the internal image portion is performed (step S31 to step S34).

Specifically, first, the marker position specifying unit 1515 specifies the three-dimensional position of the hyperspectral camera 12 in the real space coordinate system based on the measurement result of the position measuring device 14 (step S31). Specifically, the position measuring device 14 detects the measurement light reflected by the marker m2 installed in the hyperspectral camera 12. The marker position specifying unit 1515 can specify the position of the marker m <b> 2 based on the measurement result of the position measuring device 14. When the position of the marker m2 is specified, the position of the hyperspectral camera 12 (however, the position in this case includes the orientation of the hyperspectral camera 12) is specified. In the following description, for convenience of description, the position of the hyperspectral camera 12 specified in step S31 is specified by Equation 1. The matrix of 3 rows × 3 rows composed of rc11, rc12, rc13, rc21, rc22, rc23, rc31, rc32, and rc33 in Equation 1 is the rotation amount of the hyperspectral camera 12 in the real space coordinate system (ie, yaw). The rotation amount in the direction (in other words, the tilt amount, the same applies hereinafter), the rotation amount in the roll direction, and the rotation amount in the pitch direction) are shown. Tcx in Equation 1 represents the translation amount of the hyperspectral camera 12 along the X axis from the origin of the real space coordinate system. Tcy in Equation 1 indicates the translation amount of the hyperspectral camera 12 along the Y axis from the origin of the real space coordinate system. Tcz in Formula 1 indicates the translation amount of the hyperspectral camera 12 along the Z axis from the origin of the real space coordinate system.

Thereafter, the association processing unit 1516 identifies an outer surface image portion corresponding to the tumor image portion in the MRI image (step S32). That is, the association processing unit 151 displays an outer surface image portion that three-dimensionally shows the outer surface of the tumor as a curved surface (or uneven surface) in the MRI image that is a three-dimensional image based on the tumor image portion that is a two-dimensional image. Identify. Here, the association processing unit 1516 arbitrarily selects an outer surface image portion corresponding to the tumor image portion of the MRI image (that is, three-dimensional image) based on the tumor image portion (that is, two-dimensional image). May be used. For example, the association processing unit 1516 corresponds to the two-dimensional image of the three-dimensional image based on the two-dimensional image as the operation of specifying the outer surface image portion corresponding to the tumor image portion of the MRI image based on the tumor image portion. A known operation for specifying an image portion to be performed may be used.

Hereinafter, an example of an operation for specifying the outer surface image portion corresponding to the tumor image portion in the MRI image based on the tumor image portion will be described.

The association processing unit 1516 identifies an internal parameter indicating the state of the optical system (for example, a lens group) of the hyperspectral camera 12. As an example of the internal parameter, the focal length fcx of the optical system with respect to the X axis constituting the spectral coordinate system (that is, the focal length fcx corresponding to the photographing magnification of the subject along the X axis), or the Y axis constituting the spectral coordinate system Is the focal length fcy of the optical system (that is, the focal length fcy corresponding to the photographing magnification of the subject along the Y axis). Another example of the internal parameter is a deviation amount of the center of the imaging device of the hyperspectral camera 12 (that is, the center of the spectral image) with respect to the center of the optical axis of the optical system. This deviation amount includes a deviation amount ccx along the X axis constituting the spectral coordinate system and a deviation amount ccy along the Y axis constituting the spectral coordinate system. Note that the association processing unit 1516 can specify an internal parameter using the calibration result of the hyperspectral camera 12 based on the checker pattern imaging result.

The association processing unit 1516 further specifies an external parameter indicating the installation state of the hyperspectral camera 12 in the real space coordinate system. In this embodiment, the external parameter is the position of the hyperspectral camera 12 in the real space coordinate system shown in Equation 1.

As shown in FIG. 6, the association processing unit 1516 can specify the light path to each pixel constituting the imaging device of the hyperspectral camera 12 based on the internal parameter and the external parameter in the real space coordinate system. it can. Furthermore, since the position in the real space coordinate system can be converted to the position in the MRI coordinate system, the association processing unit 1516 can specify the light path to each pixel in the MRI coordinate system. As a result, the association processing unit 1516 can specify a portion located on the light path reaching each pixel in the outer surface of the patient represented by the MRI image. For example, in the MRI image, the association processing unit 1516 firstly reaches the position of the outer surface of the patient in the MRI coordinate system (X1 (X1 ( 3), Y1 (3), Z1 (3)) can be specified.

Here, it can be seen from FIG. 6 that the portion of the outer surface of the patient that is located on the light path leading to each pixel is the portion of the outer surface of the patient that is captured by the hyperspectral camera 12. Similarly, the portion of the outer surface of the patient that is located on the light path to each pixel is a portion that is also imaged by the MRI apparatus 11. Then, the image portion corresponding to the outer surface of the patient located on the light path reaching each pixel in the MRI image corresponds to the spectroscopic image. That is, the image portion located on the light path in the MRI image corresponds to the spectral image.

Therefore, the association processing unit 1516 includes an image portion located on the light path in the MRI image (that is, an image portion where the position (X1 (3), Y1 (3), Z1 (3)) is specified). Is associated with the spectral image. Specifically, the positions (X1 (3), Y1 (3), Z1 (3)) in the MRI coordinate system are the positions (X1 (3) ′, Y1 (3) ′, Z1 in the real space coordinate system. As described above, (3) ′) can be converted. Furthermore, each pixel that is a light emission destination has a one-to-one correspondence with specific coordinates in a spectral coordinate system that specifies a two-dimensional position in the spectral image. Then, the association processing unit 1516 determines the position (X1 (3) ′, Y1 (3) ′, Z1 (3) ′) in the real space coordinate system and the spectral coordinate system based on the internal parameter and the external parameter. The positions (X1 (2), Y1 (2)) can be associated with each other. That is, the association processing unit 1516 determines the position (X1 (3), Y1 (3), Z1 (3)) in the MRI coordinate system and the position (X1 (2), Y1 (2)) in the spectral coordinate system. Can be associated. In this embodiment, the association processing unit 1516 uses Equation 2 to calculate the position (X1 (3), Y1 (3), Z1 (3)) in the MRI coordinate system and the position (X1 ( 2) and Y1 (2)). Note that s1 represents a predetermined coefficient or a predetermined conversion matrix.

As a result, the association processing unit 1516 can associate the MRI image (particularly, the image portion of the MRI image corresponding to the outer surface of the patient located on the light path reaching each pixel) with the spectral image. That is, the association processing unit 1516 can specify the image portion of the spectral image corresponding to a certain image portion constituting the MRI image. Similarly, the association processing unit 1516 can specify the image portion of the MRI image corresponding to a certain image portion constituting the spectral image. Then, since the position of the tumor image portion in the spectral image has already been specified, the association processing unit 1516 corresponds to the tumor image portion in the MRI image when the MRI image and the spectral image are associated with each other. The outer surface image portion to be identified can be specified.

As described above, the association processing unit 1516 can associate the MRI image and the spectral image without specifying the position of the marker m1 included in the spectral image in the spectral coordinate system. The reason for not specifying the position of the marker m1 included in the spectral image in the spectral coordinate system is that the operation performed by the association processing unit 1516 in order to associate the MRI image with the spectral image is performed by the patient in the MRI image. This is because it is assumed that the outer surface image portion corresponding to the outer surface corresponds to a spectral image generated by photographing the outer surface of the patient. In other words, the operation performed by the association processing unit 1516 for associating the MRI image and the spectroscopic image is two-dimensionally representing the outer surface of the patient as a plane, and the same outer surface of the same patient having the same MRI image as a curved surface. This is because it is assumed to be expressed three-dimensionally. In other words, the association processing unit 1516 uses the fact that each of the spectral image (further, the tumor image portion) and the outer surface image portion represents the same outer surface of the same patient, so that the outer surface of the patient is a two-dimensional plane. Based on the tumor image portion expressed in (3), an outer surface image portion that three-dimensionally represents the same outer surface of the same patient as a curved surface is specified.

Thereafter, the association processing unit 1516 identifies the feature of the outer surface image portion identified in step S32 (step S33). The “feature” specified in step S33 may be any feature as long as the feature is unique to the image or related to the image. Examples of such features include brightness, hue, saturation, brightness, and the like.

Since the feature specified in step S33 is a feature of the outer surface image portion showing the outer surface of the tumor, it is also estimated that it is a feature of the image portion corresponding to the tumor in the MRI image. Therefore, the association processing unit 1516 identifies an image portion having the same feature as the feature identified in step S33 in the MRI image as an internal image portion representing the inside of the tumor (step S34). As a result, as shown in FIG. 7, on the MRI image, not only the outer surface image portion representing the outer surface of the tumor but also the inner image portion representing the inside of the tumor is specified.

Thereafter, the three-dimensional model generation unit 1517 generates a three-dimensional model of the patient that represents the tumor in a manner that can be distinguished from other parts (step S41). That is, the three-dimensional model generation unit 1517 generates a three-dimensional model of the patient that represents the outer surface image portion and the inner image portion in a manner that can be distinguished from other image portions (step S41). For example, the three-dimensional model generation unit 1517 may generate the three-dimensional model by modifying the MRI image so that the MRI image represents the outer surface image portion and the inner image portion in a manner that can be distinguished from other image portions. Good. For example, the three-dimensional model generation unit 1517 may generate a three-dimensional model by newly generating an MRI image so as to represent the outer surface image portion and the inner image portion in a manner distinguishable from other image portions. . In addition, as an example of the display method of the tumor in a mode that can be distinguished from other parts, for example, a display method that highlights and displays the tumor can be given.

Thereafter, the three-dimensional viewer processing unit 1518 generates an observation image that is observed when the surgeon observes the three-dimensional model generated in step S41 from a desired viewpoint (step S42). The three-dimensional viewer processing unit 1518 outputs the generated observation image to the display device 16.

Thereafter, the display device 16 displays the observation image generated in step S42 (step S43). As a result, the surgeon can recognize how the tumor is three-dimensionally distributed inside the patient by visually recognizing the observation image.

Thereafter, the MRI image acquisition unit 1511 determines whether or not it is time to acquire the MRI image again (step S51). For example, when a predetermined time has elapsed since the last MRI image was acquired, the MRI image acquisition unit 1511 may determine that the timing for acquiring the MRI image again has arrived. For example, when the user requests acquisition of the MRI image again, the MRI image acquisition unit 1511 may determine that the timing for acquiring the MRI image again has arrived.

As a result of the determination in step S51, when it is determined that the timing for acquiring the MRI image again has come (step S51: Yes), the MRI image acquisition unit 1511 acquires the MRI image again (step S1). Thereafter, the operations after step S2 are performed again.

On the other hand, as a result of the determination in step S51, when it is determined that the timing for acquiring the MRI image again has not arrived (step S51: No), the 3D model generation unit 1517 generates the 3 generated in step S42. An additional model corresponding to an additional image is added to the dimension model (step S52). As an additional image, an image of a surgical instrument (for example, a knife) handled by the surgeon is raised.

In order to add the image of the surgical tool to the three-dimensional model as an additional image, the image processing device 15 performs the operation described below. First, as a premise, similar to the hyperspectral camera 12 and the pointer 13, a marker is placed on the surgical instrument. The position measurement device 14 detects measurement light reflected by a marker installed on the surgical instrument. The marker position detection unit 1515 specifies the position of the surgical instrument in the real space coordinate system based on the measurement result of the position measurement device 14. The association processing unit 1516 converts the position of the surgical tool in the real space coordinate system into the position of the surgical tool in the MRI coordinate system. The position of the surgical instrument in the MRI coordinate system corresponds to the position where the surgical instrument is to be placed in the three-dimensional model. Therefore, the three-dimensional model generation unit 1517 gives an additional model corresponding to the surgical tool to the three-dimensional model of the patient, and the positional relationship between the three-dimensional model of the patient and the additional model corresponding to the surgical tool is actually It can be added so as to match the positional relationship between the patient and the actual surgical instrument.

Thereafter, the image processing unit 15 determines whether or not the operation is completed (step S53). As a result of the determination in step S53, when it is determined that the operation has not been completed (step S53: No), the operations after step S51 are repeated. On the other hand, as a result of the determination in step S53, when it is determined that the operation is completed (step S53: Yes), the operation support system 1 ends the operation illustrated in FIG.

As described above, the image processing apparatus 15 according to the present embodiment can specify not only the outer surface image portion that three-dimensionally represents the outer surface of the tumor but also the inner image portion that represents the inside of the tumor three-dimensionally. . That is, the image processing apparatus 15 according to the present embodiment can specify how the tumor identified from the spectral image that is a two-dimensional image is three-dimensionally distributed in the MRI image that is a three-dimensional image. . As a result, the display device 16 can display a three-dimensional model of the patient that represents the tumor in a manner that can be distinguished from other parts. As a result, the surgeon can appropriately recognize the tumor.

In addition, the structure and operation | movement of the surgery assistance system 1 mentioned above are an example to the last. Therefore, at least a part of the configuration and operation of the surgery support system 1 may be modified as appropriate. Hereinafter, at least some modifications of the configuration and operation of the surgery support system 1 will be described.

The surgery support system 1 may include an arbitrary device that can generate an image representing a three-dimensional representation of the outer surface and internal structure of the patient by imaging the patient in addition to or instead of the MRI apparatus 11. Good. As an example of such an arbitrary apparatus, there is a CT (Computed Tomography) apparatus and a PET (Positron Emission Tomography) apparatus.

During surgery, the shape of the surgical site and the shape of the tumor may change. In this case, the image processing apparatus 15 specifies an outer surface image portion representing the outer surface of the tumor and an inner image portion representing the inside of the tumor, and then at least one of the spectral image (or the outer surface image portion and the inner image portion). ) May be subjected to a non-rigid registration process. As a result, the image processing apparatus 15 can generate a three-dimensional model that represents the tumor whose shape has changed in a manner that can be distinguished from other parts.

The inner image portion may include the outer surface image portion as a part thereof. This is because the outer shell of the inner image portion substantially corresponds to the outer surface image portion. For this reason, the internal image portion may represent the outer surface of the tumor in addition to the inside of the tumor.

Further, the present invention can be appropriately changed without departing from the gist or concept of the invention that can be read from the claims and the entire specification, and an image processing apparatus, an image processing method, and a computer that involve such a change The program is also included in the technical idea of the present invention.

DESCRIPTION OF SYMBOLS 1 Surgery support system 11 MRI apparatus 12 Hyper spectrum camera 13 Pointer 14 Position measuring apparatus 15 Image processing apparatus 151 CPU
1511 MRI image acquisition unit 1512 Marker position specifying unit 1513 Spectral image acquiring unit 1514 Tumor specifying unit 1515 Marker position specifying unit 1516 Association processing unit 1517 3D model generation unit 1518 3D viewer processing unit 152 Memory 16 Display device

Claims

An acquisition means for acquiring a two-dimensional image and a three-dimensional image generated by photographing the same object;
A first specifying means for specifying a first image portion representing the object in the two-dimensional image;
An image processing apparatus comprising: a second specifying unit that specifies a third image portion in the three-dimensional image having a feature of the second image portion in the three-dimensional image corresponding to the first image portion. .
The second image portion represents at least a part of the outer surface of the object in the three-dimensional image,
The image processing apparatus according to claim 1, wherein the third image portion represents at least the inside of the object in the three-dimensional image.
A third specifying unit for specifying the second image portion;
The two-dimensional image is generated by shooting a shooting target including the target object,
The third specifying means is (i) present on a path of light reaching each pixel of an image sensor included in an imaging device that generates the two-dimensional image by photographing the photographing target among the three-dimensional images. Identifying a fourth image portion corresponding to an outer surface of the imaging target, and (ii) based on a first parameter indicating a state of an optical system included in the imaging device and a second parameter indicating an installation environment of the imaging device, The fourth image portion is associated with the two-dimensional image, and (iii) the second image portion is specified based on a result of the association between the fourth image portion and the two-dimensional image. The image processing apparatus according to claim 1.
The apparatus further comprises control means for controlling the display device to display another three-dimensional image representing the second image portion and the third image portion in a manner distinguishable from other image portions. The image processing apparatus according to any one of 1 to 3.
The image according to any one of claims 1 to 4, wherein the object includes at least one of a predetermined part of a subject, an affected area of the subject, and a tumor part of the subject. Processing equipment.
The three-dimensional image is an MRI (Magnetic Resonance Imaging) image that stereoscopically represents the outer surface and the inside of the subject.
The image processing apparatus according to claim 5, wherein the two-dimensional image is a spectral image that planarly represents an outer surface of the subject.
An acquisition step of acquiring a two-dimensional image and a three-dimensional image generated by imaging the same object;
A first specifying step of specifying a first image portion representing the object in the two-dimensional image;
And a second specifying step of specifying a third image portion in the three-dimensional image having a feature of the second image portion in the three-dimensional image corresponding to the first image portion. .
A computer program for causing a computer to execute the image processing method according to claim 7.