JP5307357B2 - Ultrasonic diagnostic equipment - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a guide image correctly indicating an actual anatomical location and orientation of an ultrasonic tomographic image, even if a region of interest itself is moved or deformed when observing the ultrasonic tomographic image of the region of interest such as an internal organ, an organ and tissue. <P>SOLUTION: A reference image storage section 55 stores reference image data of normal pancreas and reference image data after rotating or moving the pancreas, and a control circuit 63 determines which one out of the reference image data is used. An interpolation circuit 56 rereads the reference image data by a command from the control circuit 63, and voxel spaces of an interpolation memory and a composite memory are filled with the reference image data read corresponding to a combination of keys. The three-dimensional human body image data and the guide image are thus changed over instantaneously by the key operation by an operator so as to display the guide image correctly indicating the actual anatomical location and orientation of the ultrasonic tomographic image. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to an ultrasound diagnostic apparatus that acquires ultrasound tomographic images by transmitting and receiving ultrasound into a living body.

  2. Description of the Related Art Conventionally, an ultrasonic diagnostic apparatus that acquires an ultrasonic tomographic image by inserting an ultrasonic probe into a body cavity such as a digestive tract, a bile pancreatic duct, or a blood vessel is well known.

  For example, in Patent Document 1 (Japanese Patent Laid-Open No. 2006-149481) and Patent Document 2 (International Application WO2006 / 057296), the position and orientation of an ultrasonic tomographic image are detected, and this position and orientation are used. An ultrasonic diagnostic apparatus that displays a guide image of an anatomical position and orientation of an ultrasonic tomographic image based on reference image data such as 3D-CT data held in advance is disclosed. This ultrasonic diagnostic apparatus employs an ultrasonic endoscope as an example of an ultrasonic probe.

  Patent Document 3 (Japanese Patent Application Laid-Open No. 2002-263101) discloses an ultrasonic probe that transmits / receives ultrasonic waves to / from tissue inside a living body and outputs a reception signal thereby, and based on the reception signal. An ultrasonic image forming means for forming an ultrasonic image of the tissue, an image database storing a plurality of illustration images schematically representing the inside of the living body, and an image for selecting an illustration image corresponding to the tissue from the image database An ultrasonic diagnostic apparatus including a selection unit and a display unit that displays the selected illustration image together with the ultrasonic image is disclosed.

As described in paragraph [0045], the image data in Patent Document 3 is classified by subject and type, and coordinate information is associated with each image data. For example, when “abdomen” is specified as the subject and “whole” is specified as the type, image data having the file name specified by them is searched, and the image data is output to the display processing unit. The
JP 2006-149481 A International application WO2006 / 057296 JP 2002-263101 A

  By the way, when approaching a region of interest such as a living organ, organ, or tissue or observing the entire region of interest without omission, the ultrasonic probe is moved or deformed while observing a real-time ultrasonic tomogram. It is customary to change the direction.

  However, the movement, deformation, and change in orientation of the ultrasonic probe may cause movement and deformation of the observation site itself such as an organ. The techniques disclosed in Patent Documents 1 to 3 deal with such a problem. It is difficult.

  That is, in the ultrasonic diagnostic apparatus disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2006-149481) and Patent Document 2 (International Application WO2006 / 057296), an organ in the reference image data and an ultrasonic probe are used. There is a possibility that the position and shape differ from the organ being scanned, and the guide image cannot show the actual anatomical position and orientation of the ultrasonic tomographic image.

  In addition, as described above, the ultrasonic diagnostic apparatus disclosed in Patent Document 3 (Japanese Patent Application Laid-Open No. 2002-263101) has a “subject” such as “abdomen” and a “type” such as “whole”. It is possible to select the expression form for but not the organ or organ or tissue's own state.

  Therefore, the movement, deformation, and change in orientation of the ultrasonic probe described above cause movement and deformation of the observation site, for example, the organ itself, so that the organ in the reference image data and the organ that is being scanned by the ultrasonic probe are between. However, the problem that the position and shape are different and the guide image cannot show the actual anatomical position and orientation of the ultrasonic tomographic image remains unsolved.

  The present invention has been made in view of the above circumstances. When observing an ultrasonic tomographic image of a region of interest such as an organ, an organ, or a tissue, the ultrasonic tomography may be detected even if the region of interest itself moves or deforms. An object of the present invention is to provide an ultrasonic diagnostic apparatus capable of displaying a guide image that correctly indicates the actual anatomical position and orientation of an image.

In order to achieve the above object, an ultrasonic diagnostic apparatus according to an aspect of the present invention is an elongated ultrasonic wave that is inserted into a body cavity of a living body and obtains an ultrasonic signal for creating an ultrasonic tomographic image by scanning the ultrasonic wave. An ultrasonic probe, an ultrasonic tomographic image creation unit that creates an ultrasonic tomographic image based on the ultrasonic signal, a detection unit that detects a position and / or orientation of the ultrasonic tomographic image, and an in vivo body of the ultrasonic probe And a plurality of states related to the region of interest when the region of interest is in a first state, which is a normal state, with an organ, organ or tissue as a region of interest. A first plurality of image data groups corresponding to each and a second region corresponding to each of a plurality of states related to the region of interest when the region of interest is in a second state where the organ is deformed or moved Multiple image data And a reference data holding unit for holding reference data constituted by, and a guide for anatomical position and / or orientation of the ultrasonic tomogram based on the reference data using the position and / or orientation A guide image creation unit that creates an image, a display unit that displays the guide image, and scanning information in the living body of the ultrasound probe selected by the ultrasound probe information selection unit. A state selection unit that determines whether a region of interest is in the first state or the second state and selects which state of the region of interest is one of the plurality of states; The guide image creation unit converts the image data corresponding to the state of the region of interest selected by the state selection unit into the first plurality of image data groups or the second image data group. Obtained from a plurality of image data groups, to create the guide image based on the acquired image data.

  According to the present invention, when observing an ultrasonic tomographic image of a region of interest such as an organ, an organ, or a tissue, the region of interest itself moves or deforms due to movement, deformation, orientation change, or the like of the ultrasonic probe. Even so, it is possible to display a guide image that correctly indicates the actual anatomical position and orientation of the ultrasonic tomographic image.

  Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment]
1 to 23 relate to a first embodiment of the present invention, FIG. 1 is a block diagram showing a configuration of an ultrasonic diagnostic apparatus, and FIG. 2 is an explanatory diagram schematically showing a body surface detection coil in a use example. 3 is a side view showing the body cavity contact probe, FIG. 4 is a block diagram showing the configuration of the image processing apparatus, FIG. 5 is an explanatory view showing reference image data stored in the reference image storage unit, and FIG. 6 is a voxel space. FIG. 7 is an explanatory diagram showing a keyboard key arrangement, FIG. 8 is an explanatory diagram showing an orthogonal base in which an origin is set on a transmission antenna to represent position / orientation data, and FIG. FIG. 10 is an explanatory diagram showing how image feature data is created by the image index creation circuit, and FIG. 11 is an insertion created by the insertion shape creation circuit. An explanation of how shape data is created FIG. 12, FIG. 12 is an explanatory diagram showing three-dimensional human body image data, FIG. 13 is an explanatory diagram showing how image index data and insertion shape data are embedded in the voxel space in the synthesis memory by the synthesis circuit, and FIG. FIG. 15 is an explanatory diagram showing three-dimensional guide image data when observed from the same direction as the ultrasonic tomographic image, and FIG. 16 is a display device. FIG. 17 is a flowchart showing the entire processing content, FIG. 18 is a body surface feature point and body cavity feature point designation on the reference image in FIG. FIG. 19 is a flowchart showing the specific processing content of the correction value calculation processing in FIG. 17, FIG. 20 is an explanatory diagram of the processing in FIG. 19, and FIG. 17 is a flowchart showing specific processing contents of ultrasonic tomographic image / three-dimensional guide image creation / display processing in FIG. 17, FIG. 22 is an explanatory diagram showing rotation / movement of three-dimensional human body image data, and FIG. 23 is a combination of keys and reference. It is explanatory drawing which shows a response | compatibility with image data.

  First, the configuration of the ultrasonic diagnostic apparatus in the present embodiment will be described. As shown in FIG. 1, the ultrasonic diagnostic apparatus 1 of the present embodiment includes an electronic radial scanning type ultrasonic endoscope 2 as an ultrasonic probe, an optical observation apparatus 3, and an ultrasonic endoscope 2. An ultrasonic observation device 4 as an ultrasonic tomographic image creation unit for creating an ultrasonic tomographic image based on the ultrasonic signal of the position, and a position / orientation calculation device 5 as a detection unit for detecting the position and / or orientation of the ultrasonic tomographic image. A transmission antenna 6, a body surface detection coil 7, an intra-body cavity contact probe 8, an A / D unit 9, an image processing device 11, a mouse 12, a keyboard 13, and a display device as a display unit 14 and these are connected by a signal line.

  Outside the ultrasonic diagnostic apparatus 1, an X-ray three-dimensional helical CT apparatus (X-ray 3 dimentional computer tomography system) 15, a three-dimensional MRI apparatus (3 dimentional magnetic resonance imaging system) 16, optical communication using them, There is a high-speed network 17 such as ADSL. The X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16 are connected to the image processing apparatus 11 of the ultrasonic diagnostic apparatus 1 via the network 17.

  The ultrasonic endoscope 2 includes a rigid portion 21 made of a hard material such as stainless steel at the tip and a rear end side of the rigid portion 21 so as to be used by being inserted into a body cavity such as the esophagus, stomach, or duodenum. It consists of a long flexible portion 22 made of a flexible material and an operation portion 23 made of a hard material on the rear end side of the flexible portion 22. The rigid portion 21 and the flexible portion 22 form an insertion portion that is inserted into the body cavity.

  The rigid portion 21 is provided with an optical observation window 24 formed of a cover glass, and an objective lens 25 that connects an optical image inside the optical observation window 24 and an image pickup device disposed at the image formation position thereof. For example, a CCD (charge coupled device) 26 is provided. An illumination light irradiation window (illumination window) (not shown) that irradiates illumination light into the body cavity is provided adjacent to the optical observation window 24.

  The CCD 26 is connected to the optical observation device 3 by a signal line 27. An illumination light irradiation window (not shown) is configured to irradiate illumination light and illuminate the body cavity. The image of the body cavity surface is formed on the CCD 26 from the optical observation window 24 via the objective lens 25, and the CCD signal from the CCD 26 is output to the optical observation device 3 via the signal line 27.

  The rigid portion 21 is provided with an ultrasonic transducer group in which, for example, a cylindrical tip portion is finely cut into a strip shape, and arranged in an annular array around the insertion axis. A sound wave transducer array 29 is formed. Each ultrasonic transducer 29 a constituting the ultrasonic transducer array 29 is connected to the ultrasonic observation apparatus 4 via the operation unit 23 via the signal line 30. The center of the ring of the ultrasonic transducer array 29 is the turning center of the ultrasonic beam by radial scanning described later.

Here, orthonormal bases (unit vectors in each direction) V, V ₃ and V ₁₂ fixed to the rigid portion 21 are defined as shown in FIG. V is a vector parallel to the longitudinal direction (insertion axis direction) of the rigid portion 21, and is a normal direction vector of the ultrasonic tomographic image, as will be described later. V ₃ orthogonal to the vector V is a 3 o'clock direction vector, and V ₁₂ is a 12 o'clock direction vector.

In the rigid portion 21, an image position / orientation detection coil 31 as an image position / orientation detection element for the ultrasonic transducer array 29 is fixed in the very vicinity of the center of the ring of the ultrasonic transducer array 29. . The image position / orientation detection coil 31 is formed integrally with a coil wound in two axial directions so that the two directions (axis) of the vectors V and V ₃ are directed, and detects both directions of the vectors V and V _3. It is set to be possible.

  In order to detect the insertion shape of the flexible portion 22 constituting the insertion portion of the ultrasonic endoscope 2 in the flexible portion 22, a plurality of insertion shape detection coils 32 are arranged along the insertion axis at regular intervals, for example. Is provided. As shown in FIG. 1, the insertion shape detection coil 32 is a coil wound in one axial direction, and is fixed inside the flexible portion 22 so that the winding axis direction coincides with the insertion axis direction of the flexible portion 22. ing.

  The position and orientation of the rigid portion 21 can be detected from the position of the image position / orientation detection coil 31. Further, in many cases, a bendable bending portion is provided near the distal end of the flexible portion 22, and a plurality of insertion shape detection coils 32 are provided only in the vicinity of the bending portion, and the insertion portion of the ultrasonic endoscope 2 is provided. Alternatively, the insertion shape of the tip side portion may be detected.

  In the present embodiment, the insertion shape is detected using a magnetic field by employing a plurality of insertion shape detection coils 32. Thereby, it is possible to prevent the operator and the patient (subject) from being exposed to radiation for detecting the insertion shape.

  The position / orientation calculation device 5 detects the position and orientation of the image position / orientation detection coil 31, the positions of the plurality of insertion shape detection coils 32, and the like, and includes the transmission antenna 6, the plurality of A / D units 9 a, The A / D unit 9 having 9b and 9c is connected to the image processing apparatus 11 through signal lines. Among these, the position / orientation calculation device 5 and the image processing device 11 are connected by, for example, an RS-232C standard cable 33.

  The transmission antenna 6 is composed of a plurality of transmission coils (not shown) having different winding axis orientations, and these transmission coils are housed integrally in, for example, a rectangular parallelepiped housing. The plurality of transmission coils are each connected to the position / orientation calculation device 5.

  Each of the A / D units 9a, 9b, and 9c includes an amplifier (not shown) that amplifies an input analog signal and an analog / digital conversion circuit (not shown) that samples the amplified signal and converts it into digital data. . The A / D unit 9a is connected to the image position / orientation detection coil 31 and each of the plurality of insertion shape detection coils 32 via signal lines 34 individually. The A / D unit 9 b is connected to the long body cavity contact probe 8 through a signal line 35. The A / D unit 9 c is individually connected to each of the plurality of body surface detection coils 7 by a signal line 36.

  As shown in FIG. 2, the body surface detection coil 7 is composed of four coils wound in one axial direction, each of which is a tape, a belt, a band, etc. Specifically, it is detachably fixed to a characteristic point on the abdominal body surface (hereinafter simply referred to as a body surface feature point) and used for position detection using the magnetic field of the body surface feature point. In normal upper endoscopy, the subject 37 takes a so-called left-side-down position lying on the bed 38 with the left side down, and the endoscope is inserted through the mouth. I draw with a posture.

  In the present embodiment, the body surface feature points are skeleton features (xiphoid process), left upper front iliac spine (left anterior superior iliac spine) on the left side of the pelvis (pelvis) 4 points: right anterior superior iliac spine on the right side of the pelvis and spinous process of vertebral body on the spine between the left and right upper iliac spines I will explain to you. These four points can be identified by the operator through palpation. Further, these four points are not in the same plane, but form an oblique coordinate system (un-orthogonal reference frame) having three vectors from the sword-like projection as an origin to other feature points as basic vectors. This oblique coordinate system is indicated by a bold line in FIG.

  FIG. 3 shows the body cavity contact probe 8. The body cavity contact probe 8 has an outer cylinder 41 made of a flexible material. A body cavity detection coil 42 is fixedly provided at the front end of the outer cylinder 41, and a connector 43 is provided at the rear end of the outer cylinder 41.

  As shown in FIG. 3, the body cavity detection coil 42 is a coil wound in one axial direction and is fixed to the distal end of the body cavity contact probe 8. The body cavity detection coil 42 is fixed so that the winding axis direction thereof coincides with the insertion axis direction of the body cavity contact probe 8. The body cavity detection coil 42 is used to detect the position of a region of interest in the body cavity with which the tip of the body cavity contact probe 8 is in contact.

  As shown in FIG. 1, the ultrasonic endoscope 2 is a treatment instrument insertion port for inserting forceps or the like as a first opening from the operation unit 23 through the flexible unit 22 to the rigid unit 21 (first opening). In the following, for the sake of simplicity, a tubular treatment instrument channel 46 is provided, which is provided with a forceps opening 44, and the rigid portion 21 is provided with a protruding opening 45 as a second opening.

  The treatment instrument channel 46 is configured such that the body cavity contact probe 8 can be inserted through the forceps opening 44 and protrude from the protrusion 45. The opening direction of the protrusion 45 is directed so that the body cavity contact probe 8 falls within the optical field of view of the optical observation window 24 when the body cavity contact probe 8 protrudes from the protrusion 45.

  As shown in FIG. 4, the image processing apparatus 11 includes a matching circuit 51, an image index creation circuit 52, an insertion shape creation circuit 53, a communication circuit 54, and a reference image as a reference data holding unit that holds reference data. The storage unit 55, the interpolation circuit 56, the three-dimensional human body image creation circuit 57, the synthesis circuit 58, the rotation conversion circuit 59, and 3 as a guide image creation unit that creates three-dimensional guide images in two different gaze directions. A dimension guide image creation circuit 60 (hereinafter referred to as a 3D guide image creation circuit A and a 3D guide image creation circuit B), a mixing circuit 61, a display circuit 62, and a control circuit 63 are provided. The communication circuit 54 includes a large-capacity and high-speed communication modem, and is connected to the X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16 via the network 17.

  The matching circuit 51 receives the position / orientation data output from the position / orientation calculation device 5 and maps the position / orientation data calculated on the orthogonal coordinate axes 0-xyz according to a predetermined conversion formula, as will be described later. New position / orientation data in the orthogonal coordinate axes O′-x′y′z ′ is calculated. The matching circuit 51 outputs the new position / orientation data as position / orientation mapping data to an image index creating circuit 52 for creating image index data and an insertion shape creating circuit 53 for creating insertion shape data. To do.

  The reference image storage unit 55 includes a hard disk drive that can store a large amount of data. The reference image storage unit 55 stores a plurality of reference image data as anatomical image information. As shown in FIG. 5, the reference image data is tomographic image data of the subject 37 obtained from the X-ray three-dimensional helical CT apparatus 15 or the three-dimensional MRI apparatus 16 via the network 17.

  In the present embodiment, for convenience of description, reference image data is obtained from a specific subject out of a plurality of subjects, and is perpendicular to the body axis (axis extending from the head to the foot), and 0. The data is a tomographic image of a square having a pitch of 5 mm to several mm and a side of several tens of centimeters. The reference image data in the reference image storage unit 55 in FIG. 5 are numbered from 1 to N for convenience of explanation. Here, as shown in FIG. 5, the orthogonal coordinate axes O′-x′y′z ′ fixed for a plurality of reference image data and their orthonormal bases (unit vectors in each axis direction) i ′, j ′, k 'Is defined on the reference image data by defining the origin O' at the bottom left of the first reference image data.

  As shown in FIG. 4, the interpolation circuit 56 and the synthesis circuit 58 each have a built-in volume memory VM. For convenience of explanation, hereinafter, the volume memory VM provided in the interpolation circuit 56 is referred to as an interpolation memory 56a, and the volume memory VM provided in the synthesis circuit 58 is referred to as a synthesis memory 58a.

  The volume memory VM is configured to store a large amount of data. A voxel space is allocated to a part of the storage area of the volume memory VM. As shown in FIG. 6, the voxel space is composed of memory cells (hereinafter referred to as voxels) having addresses corresponding to the orthogonal coordinate axes O'-x'y'z '.

  A three-dimensional human body image creation circuit 57 that creates a three-dimensional human body image shown in FIG. 4 and a rotation conversion circuit 59 that performs rotation conversion perform image processing such as extraction of voxels and pixels by luminance, rotation conversion, similarity conversion, and parallel movement. It incorporates a high-speed processor (not shown) that performs at high speed.

  The display circuit 62 has a switch 62a for switching the input. The switch 62a has an input terminal α, an input terminal β, an input terminal γ, and one output terminal. The input terminal α is connected to the reference image storage unit 55. The input terminal β is connected to an output terminal (not shown) of the optical observation device 3. The input terminal γ is connected to the mixing circuit 61. The output terminal is connected to a display device 14 that displays an optical image, an ultrasonic tomographic image, a three-dimensional guide image, and the like.

  The control circuit 63 is connected to each unit and each circuit by a signal line (not shown) so that a command can be output to each unit and each circuit in the image processing apparatus 11. The control circuit 63 is directly connected to the ultrasound observation apparatus 4, the mouse 12, and the keyboard 13 through control lines.

  As shown in FIG. 7, the keyboard 13 has a body cavity feature point designation key 65, a scanning control key 66, a display switching key 13α, a display switching key 13β, a display switching key 13γ, and a stomach as an ultrasonic endoscope scanning region key. A duodenal bulb key 18a, a duodenal descending leg key 18b, and a PUSH key 19a and a PULL key 19b as ultrasonic endoscope scanning information keys are provided.

  When the display switching key 13α, 13β, or 13γ is pressed, the control circuit 63 outputs a command to the display circuit 62 to switch the switch 62a to the input terminal α, β, or γ. The switch 62a switches to the input terminal α when the display switching key 13α is pressed, switches to the input terminal β when the display switching key 13β is pressed, and switches to the input terminal γ when the display switching key 13γ is pressed.

Next, the function of the ultrasound diagnostic apparatus 1 of the present embodiment having the above configuration will be described with reference to FIGS. Each arrow line in FIG. 1 and FIG. 4 indicates the following signal / data flow.
(A) First: signal / data flow related to optical image indicated by dotted line (b) Second: signal / data flow related to ultrasonic tomographic image indicated by broken line (c) Third: related to position indicated by solid line Signal / data and data flow created by processing them (d) 4: Reference image data indicated by alternate long and short dash lines and data flow created by processing them (e) Fifth: indicated by bold lines Signal / data flow related to the final display screen obtained by synthesizing ultrasonic tomographic image data (described later) and three-dimensional guide image data (described later). Sixth: related to other control indicated by the flow curve Signal and data flow

  Hereinafter, description will be made sequentially along the flow of signals and data shown in FIGS.

(A) Signal / Data Flow Related to Optical Image An illumination light irradiation window (not shown) of the rigid portion 21 irradiates illumination light toward the optical visual field range side. The CCD 26 images an object in the optical field of view, photoelectrically converts it, and outputs a CCD signal to the optical observation device 3. The optical observation device 3 creates image data in the optical visual field range based on the input CCD signal, and uses this data as optical image data to the input terminal β of the switch 62a of the display circuit 62 in the image processing device 11. Output.

(B) Signal / Data Flow Related to Ultrasonic Tomographic Image When the operator presses a scan control key 66 as a condition input unit for inputting conditions for selecting scan information, the control circuit 63 performs a radial scan described later. A scanning control signal for commanding ON / OFF control is output to the ultrasonic observation apparatus 4. Upon receiving this scanning control signal, the ultrasound observation apparatus 4 selects a part and a plurality of ultrasound transducers 29a from among the ultrasound transducers 29a constituting the ultrasound transducer array 29 to excite the pulse voltage. Send a signal. The partial and plural ultrasonic transducers 29a receive the excitation signal and convert it into an ultrasonic wave that is a dense wave of the medium.

  At this time, the ultrasonic observation apparatus 4 delays each excitation signal so that the time at which each excitation signal arrives at each ultrasonic transducer 29a is different. The value (delay amount) of the delay is adjusted so that one ultrasonic beam is formed when the ultrasonic waves excited by the ultrasonic transducers 29 a are superimposed in the subject 37.

  The ultrasonic beam is irradiated to the outside of the ultrasonic endoscope 2, and the reflected wave from the inside of the subject 37 follows the path opposite to that of the ultrasonic beam and returns to each ultrasonic transducer 29a. Each ultrasonic transducer 29a converts the reflected wave into an electrical echo signal and transmits it to the ultrasonic observation apparatus 4 through a path opposite to the excitation signal.

  The ultrasonic observation apparatus 4 includes a center of the ring of the ultrasonic transducer array 29 so that the ultrasonic beam swirls in a plane perpendicular to the rigid portion 21 and the flexible portion 22 (hereinafter referred to as a radial scanning plane). The plurality of ultrasonic transducers 29a involved in the formation of the ultrasonic beam are selected again, and the excitation signal is transmitted again. In this way, the transmission angle of the ultrasonic beam changes. By repeating this repeatedly, so-called radial scanning is realized.

  At this time, the ultrasonic observation device 4 uses the echo signal converted from the reflected wave by the ultrasonic transducer 29 a to 1 perpendicular to the insertion axis of the rigid portion 21 for one radial scan of the ultrasonic transducer array 29. One piece of digitized ultrasonic tomographic image data is created and output to the mixing circuit 61 of the image processing apparatus 11. At this time, the ultrasonic observation apparatus 4 creates ultrasonic tomographic image data by processing it into a square.

As described above, in the present embodiment, since the ultrasonic observation apparatus 4 reselects the plurality of ultrasonic transducers 29a involved in the formation of the ultrasonic beam and transmits the excitation signal again, For example, the 12 o'clock direction is determined by which ultrasonic transducer 29a selects and transmits the excitation signal as the 12 o'clock direction. Thus, the normal direction vector V, 3 o'clock direction vector V ₃ , and 12 o'clock direction vector V _{12 of the} ultrasonic tomographic image are defined. Furthermore, the ultrasonic observation apparatus 4 creates ultrasonic tomographic image data in a direction observed from a direction −V opposite to the normal vector V.

  Radial scanning by the ultrasonic transducer array 29, generation of ultrasonic tomographic image data by the ultrasonic observation apparatus 4, and output to the mixing circuit 61 are performed in real time.

(C) Flow of position-related signals and data and data created by processing the position / orientation calculation device 5 excites a transmission coil (not shown) of the transmission antenna 6. The transmission antenna 6 applies an alternating magnetic field in the space.

Two coils whose winding axes wound in the directions of vectors V and V ₃ constituting the image position / orientation detection coil 31 are orthogonal to each other, a plurality of insertion shape detection coils 32, and a body cavity detection coil 42 and the body surface detection coil 7 respectively detect an alternating magnetic field, convert the alternating magnetic field into respective position electric signals, and output the signals to the A / D units 9a, 9b, 9c.

  The A / D units 9 a, 9 b, 9 c amplify the position electrical signal with an amplifier, sample it with an analog-digital conversion circuit, convert it into digital data, and output the digital data to the position orientation calculation device 5.

Next, the position / orientation calculation apparatus 5 calculates the position of the image position / orientation detection coil 31 and the direction of the winding axis perpendicular thereto, that is, the vectors V and V ₃ based on the digital data from the A / D unit 9a. calculate. Then, the position orientation calculation device 5, by calculating the outer product V × V ₃ of the winding axis direction of the vector V and V ₃ of orthogonal, to calculate the 12 o'clock direction of the vector V ₁₂ is the remaining orthogonal direction . In this way, the position / orientation calculation apparatus 5 calculates three orthogonal directions, that is, vectors V, V ₃ and V ₁₂ .

  Next, the position / orientation calculation device 5 uses the digital data from the A / D units 9a to 9c to determine the position of each of the plurality of insertion shape detection coils 32 and each of the body surface detection coils 7. The position and the position of the body cavity detection coil 42 are calculated. Then, the position / orientation calculation apparatus 5 includes the position and orientation of the image position / orientation detection coil 31, the positions of the plurality of insertion shape detection coils 32, and the positions of the four body surface detection coils 7. And the position of the body cavity detection coil 42 are output to the matching circuit 51 of the image processing apparatus 11 as position / orientation data.

Details of the position / orientation data will be described below.
In the present embodiment, the origin O is defined on the transmission antenna 6 as shown in FIG. 8, and the orthogonal coordinate axis O-xyz and its normal orthogonal basis (each axis are set on the actual space where the operator examines the subject 37. Define directional unit vectors i, j, and k. The position of the image position / orientation detection coil 31 is O ″. Since the image position / orientation detection coil 31 is fixed very close to the center of the ring of the ultrasonic transducer array 29, the position O ″ is radial. It coincides with the center of the scan and the center of the ultrasonic tomogram.

Here, the position / orientation data is defined as follows.
Each direction component of the position vector OO "of the position O" of the image position / orientation detection coil 31 on the orthogonal coordinate axis O-xyz:
(x0, y0, z0)
Each angle component of Euler angles (described later) indicating the orientation of the image position orientation detection coil 31 with respect to the orthogonal coordinate axis O-xyz:
(ψ, θ, φ)
Each direction component of each position vector of the plurality of insertion shape detection coils 32 on the orthogonal coordinate axis O-xyz:
(xi, yi, zi) (i is a natural number from 1 to the total number of insertion shape detection coils 32)
Each direction component of each position vector of the four body surface detection coils 7 on the orthogonal coordinate axis O-xyz:
(xa, ya, za), (xb, yb, zb), (xc, yc, zc), (xd, yd, zd)
Each direction component of the position vector of the body cavity detection coil 42 on the orthogonal coordinate axis O-xyz:
(xp, yp, zp)

Here, the Euler angle is obtained by adding rotation around the z axis, rotation around the y axis, and rotation around the z axis again in this order to the orthogonal coordinate axis O-xyz in FIG. The angle is such that the directions of the axes coincide.
I = V ₃ after rotation, j = V ₁₂ after rotation, k = V after rotation
Ψ is the first rotation angle around the z axis, θ is the rotation angle around the y axis, and φ is the rotation angle around the z axis again.

  H in FIG. 8 is an intersection of a perpendicular line from the position O ″ to the xy plane and the xy plane. Each angular component (ψ, θ, φ) of this Euler angle is the orientation of the image position orientation detecting coil 31. That is, this corresponds to the orientation of ultrasonic tomographic image data.

  The matching circuit 51 converts the position / orientation expressed on the orthogonal coordinate axis O-xyz from the following first, second, third, and fourth data groups to the orthogonal coordinate axis O′-x′y′z ′. The conversion formula that maps to the position / orientation in the voxel space expressed above is calculated. This calculation method will be described later. Further, the position / orientation data described in the following first and second changes depending on the body movement of the subject 37. A conversion formula is also newly created as the body movement of the subject 37 changes. New creation of this conversion formula will also be described later.

  The first data group includes a body surface detection coil 7 attached to each of the xiphoid process, upper left anterior iliac spine, upper right anterior iliac spine, and lumbar spine process of the subject 37 in the position / orientation data. Are the directional components (xa, ya, za), (xb, yb, zb), (xc, yc, zc), (xd, yd, zd) of the position vector on the orthogonal coordinate axis O-xyz. FIG. 9 shows the body surface detection coil 7 attached thereto.

  The second data group is each direction component (xp, yp, zp) of the position vector of the body cavity detection coil 42 on the orthogonal coordinate axis O-xyz in the position / orientation data. In FIG. 9, the body cavity contact probe 8 in which the body cavity detection coil 42 is fixed and housed is shown by a thick dotted line.

  The third data group includes the xiphoid process, the upper left anterior iliac spine, the upper right anterior iliac spine, and the lumbar spine spinous processes on any of the reference image data Nos. 1 to N, respectively. (Xa ', ya', za '), (xb', yb ', zb'), (xc ', yc',) on the orthogonal coordinate axis O'-x'y'z 'of the pixel closest to the body surface zc ′) and (xd ′, yd ′, zd ′). These pixels are designated in advance on any of the reference image data Nos. 1 to N by the operator. This designation method will be described later.

  FIG. 9 shows these pixels with black circles ● and white circles ○. (xa ', ya', za '), (xb', yb ', zb'), (xc ', yc', zc '), (xd', yd ', zd') are as shown in FIG. The body surface feature point coordinates are read from the reference image storage unit 55 to the matching circuit 51.

  The fourth data group includes coordinates (xp ", yp", xp ", yp", pixel coordinates on the orthogonal coordinate axis O'-x'y'z 'corresponding to the duodenal papilla on any of the reference image data Nos. 1 to N. zp "). These pixels are designated in advance by the operator on any of the reference image data Nos. 1 to N. This designation method will be described later.

  9, this pixel is indicated by P ″. The coordinates (xp ″, yp ″, zp ″) of the fourth pixel are matched from the reference image storage unit 55 as body cavity feature point coordinates as shown in FIG. Read to circuit 51.

  Next, the matching circuit 51 maps the position / orientation data calculated on the orthogonal coordinate axis O-xyz in accordance with the above conversion formula, and calculates new position / orientation data on the orthogonal coordinate axis O′-x′y′z ′. To do. Then, the matching circuit 51 outputs the new position / orientation data to the image index creation circuit 52 and the insertion shape creation circuit 53 as position / orientation mapping data.

  The image index creation circuit 52 includes each direction component (x0, y0, z0) of the position vector OO "of the position O" of the image position / orientation detection coil 31 on the orthogonal coordinate axis O-xyz, and the image position relative to the orthogonal coordinate axis O-xyz. Image index data is generated from position / orientation mapping data having a total of six degrees of freedom with each of the Euler angle components (ψ, θ, φ) indicating the orientation of the orientation detection coil 31, and is output to the synthesis circuit 58.

  This is shown in FIG. That is, image index data is created from the upper position / orientation mapping data in FIG. 10 as shown in the lower part of FIG. This image index data includes, for example, a parallelogram ultrasonic tomogram marker Mu, a blue tip direction marker Md (indicated as blue in FIG. 10), and a yellow green arrow-shaped 6 o'clock direction marker Mt (in FIG. 10). Image data on the Cartesian coordinate axis O'-x'y'z '.

  The insertion shape creation circuit 53 includes each direction component (x0, y0, z0) of the position vector OO "of the position O" of the image position / orientation detection coil 31 and a plurality of insertion shape detection coils on the orthogonal coordinate axis O-xyz. Insertion shape data is created (by interpolation and marker creation processing) from the position / orientation mapping data of each of the 32 position vectors and each direction component (xi, yi, zi), and output to the synthesis circuit 58.

  This is shown in FIG. The insertion shape data includes a string-like insertion shape marker Ms obtained by interpolating the positions of the image position / orientation detection coil 31 and the plurality of insertion shape detection coils 32 in order, and a coil position marker Mc indicating each coil position. Is the image data on the orthogonal coordinate axis O′-x′y′z ′.

(D) Flow of reference image data and data created by processing the reference image The operator presses a predetermined key on the keyboard 13 or selects a menu on the screen with the mouse 12 to acquire the reference image data. Instruct. At the same time, the surgeon also instructs the supplier. In response to this instruction, the control circuit 63 instructs the communication circuit 54 to take in reference image data and obtain it.

  For example, when the acquisition destination is the X-ray three-dimensional helical CT apparatus 15, the communication circuit 54 takes in a plurality of two-dimensional CT images from the network 17 as reference image data and stores them in the reference image storage unit 55. . When imaging with the X-ray three-dimensional helical CT apparatus 15, an X-ray contrast medium is injected from the blood vessel of the subject 37 before imaging, and blood vessels such as the aorta and superior mesenteric vein (Vessels in a broad sense) and organs containing many blood vessels are displayed on the two-dimensional CT image with high or medium luminance so that a difference in luminance from the surrounding tissues is likely to occur.

  For example, when the acquisition destination is the three-dimensional MRI apparatus 16, the communication circuit 54 captures a plurality of two-dimensional MRI images from the network 17 as reference image data and stores them in the reference image storage unit 55. When imaging with the three-dimensional MRI apparatus 16, an MRI contrast agent with high nuclear magnetic resonance sensitivity is injected from the blood vessel of the subject 37 before imaging, and blood vessels such as the aorta and superior mesenteric vein, Many organs are displayed on a two-dimensional MRI image with high or medium luminance so that a difference in luminance from the surrounding tissue is likely to occur.

  Hereinafter, when the operator selects the X-ray three-dimensional helical CT apparatus 15 as the acquisition destination, and when the operator selects the three-dimensional MRI apparatus 16, the operation is the same, so the X-ray three-dimensional helical CT apparatus as the acquisition destination. The operation will be described only when 15 is selected and the communication circuit 54 captures a plurality of two-dimensional CT images as reference image data.

  FIG. 5 shows an example of reference image data stored in the reference image storage unit 55. Due to the action of the X-ray contrast medium, blood vessels such as the aorta and superior mesenteric vein are contrasted with high brightness, organs with many peripheral blood vessels such as pancreas are contrasted with medium brightness, and duodenum is contrasted with low brightness. Has been.

  The interpolation circuit 56 reads all the reference image data from No. 1 to No. N from the reference image storage unit 55. Next, the interpolation circuit 56 fills the read reference image data in the voxel space of the interpolation memory 56a. Specifically, the luminance of each pixel of the reference image data is output to a voxel having an address corresponding to the pixel. Next, the interpolation circuit 56 performs interpolation based on the luminance value of the adjacent reference image data, and fills the vacant voxel with the data. In this way, all the voxels in the voxel space are filled with data based on the reference image data (hereinafter referred to as voxel data).

  The three-dimensional human body image creation circuit 57 extracts voxels (mainly blood vessels) with high luminance values and voxels with medium luminance values (mainly organs including many peripheral blood vessels such as pancreas) from the interpolation circuit 56 for each luminance value range. Separately and color. Next, the three-dimensional human body image creation circuit 57 fills the extracted voxels in the voxel space of the synthesis memory 58a of the synthesis circuit 58 as three-dimensional human body image data. At this time, the three-dimensional human body image creation circuit 57 fills the extracted voxel addresses in the interpolation memory 56a so that the addresses in the voxel space in the synthesis memory 58a are the same.

  FIG. 12 shows an example of 3D human body image data. In the example shown in FIG. 12, the three-dimensional human body image data is obtained by extracting the aorta and superior mesenteric vein which are high-luminance blood vessels, and the pancreas which is a medium-luminance organ. It is colored in green, and is shown as three-dimensional data observed from the ventral side with the head side of the subject 37 on the right side and the foot side on the left side.

  The synthesis circuit 58 embeds the image index data and the insertion shape data in the voxel space in the synthesis memory 58a. This is shown in FIG. In FIG. 13, for convenience of explanation, the three-dimensional human body image data existing in the voxel space is omitted (when the three-dimensional human body image data is not omitted, it is shown in FIG. 14). In this way, the synthesis circuit 58 embeds the three-dimensional human body image data, the image index data, and the insertion shape data in the same synthesis memory in the same voxel space. Hereinafter, it is synthesized as synthesized three-dimensional data.

  The three-dimensional guide image creation circuit A performs rendering processing such as hidden surface removal and shading on the synthesized three-dimensional data, and creates image data (hereinafter, three-dimensional guide image data) that can be output to the screen. The default direction of the three-dimensional guide image data is the direction from the ventral side of the human body. Therefore, the three-dimensional guide image creation circuit A creates three-dimensional guide image data observed in the direction from the ventral side of the subject 37.

  Note that the default orientation of the 3D guide image data is the orientation from the abdomen side of the human body, but 3D guide image data of the orientation from the back side may be created. Further, three-dimensional guide image data from other directions may be created.

  Next, the three-dimensional guide image creation circuit A outputs the three-dimensional guide image data observed from the subject's ventral side to the mixing circuit 61. The three-dimensional guide image data is shown in FIG. The right side of FIG. 14 is the subject head side, and the left side is the subject foot side. In the three-dimensional guide image data of FIG. 14, the ultrasonic tomographic image marker Mu in the image index data is made translucent, the 6 o'clock direction marker Mt, the tip direction marker Md, and the insertion shape data of the image index data. The insertion shape marker Ms and the coil position marker Mc are seen through. For other organs, the ultrasonic tomographic image marker Mu is made opaque so that the portion on the back side of the ultrasonic tomographic image marker Mu cannot be seen. In FIG. 14, each marker that is behind the ultrasonic tomographic image marker Mu and overlaps the ultrasonic tomographic image marker Mu is indicated by a broken line.

  The rotation conversion circuit 59 reads the combined three-dimensional data and performs a rotation process on the combined three-dimensional data in accordance with the rotation instruction signal from the control circuit 63.

  The three-dimensional guide image creation circuit B performs rendering processing such as hidden surface removal and shading on the combined three-dimensional data subjected to the rotation processing, and creates three-dimensional guide image data that can be output to the screen. In the present embodiment, as an example, an input from the operator's mouse 12 or keyboard 13 causes the rotation instruction signal from the control circuit 63 to be in the −V direction, that is, an ultrasonic tomogram based on the position / orientation mapping data. It is assumed that the instruction contents for observing the three-dimensional guide image data with the normal line of the marker Mu as the observation line of sight are provided. The rotation conversion circuit 59 is set so that the combined three-dimensional data faces the screen so as to coincide with the screen normal line of the display device 14 and the 6 o'clock direction marker Mt is directed downward in the screen of the display device 14. Create a guide image.

  Further, as shown in FIG. 15, the three-dimensional guide image creation circuit B makes the ultrasonic tomographic image marker Mu in the image index data translucent, the 6 o'clock direction marker Mt of the image index data, and the tip direction marker Md. In addition to the insertion shape marker Ms and the coil position marker Mc in the insertion shape data, the three-dimensional guide image data is created so that the back side portion of the ultrasonic tomographic image marker Mu among other organs can be seen through. .

  Specifically, the three-dimensional guide image creation circuit B has a dark portion on the ultrasonic tomographic image marker Mu, hides a portion in front of the ultrasonic tomographic image marker Mu, and displays the ultrasonic tomographic image marker Mu. The portion on the back side is reduced in luminance to create three-dimensional guide image data and output to the mixing circuit 61. In the case of the pancreas, the portion on the ultrasonic tomographic image marker Mu is created in dark green, and the portion on the back side is created in light green. In the case of a blood vessel, three-dimensional guide image data is created with the portion on the ultrasonic tomographic image marker Mu being dark red and the back portion being light red.

  The three-dimensional guide image creation circuit B outputs the three-dimensional guide image data created in this way to the mixing circuit 61.

(E) Signal / data flow related to the final display screen obtained by synthesizing the ultrasonic tomographic image data and the three-dimensional guide image data The mixing circuit 61 in FIG. 4 includes the ultrasonic tomographic image data from the ultrasonic observation apparatus 4. The three-dimensional guide image data obtained by observing the subject 37 from the three-dimensional guide image creation circuit A from the ventral side and the subject 37 from the three-dimensional guide image creation circuit B were observed in the same direction as the ultrasonic tomographic image. The mixed data for display is created by arranging the three-dimensional guide image data.

  The display circuit 62 converts this mixed data into an analog video signal and outputs it to the display device 14. Based on the analog video signal, the display device 14 generates an ultrasonic tomographic image, a three-dimensional guide image obtained by observing the subject 37 from the ventral side, and a three-dimensional guide image observed in the same direction as the ultrasonic tomographic image. Display side by side.

  As shown in FIG. 16, the display device 14 displays each organ expressed on the three-dimensional guide image by color-coding by organ according to the luminance value on the reference image data. In the display example of FIG. 16, the pancreas is displayed in green, the aorta, and the superior mesenteric vein are displayed in red. In FIG. 16, each marker that is behind the ultrasonic tomographic image marker Mu and overlaps the ultrasonic tomographic image marker Mu is indicated by a broken line. Here, the three-dimensional guide image observed from the ventral side is actually a wide range of guide images, and the three-dimensional guide image observed in the same direction as the ultrasonic tomographic image is a detailed guide image.

(F) Signal / Data Flow Related to Control The matching circuit 51, the image index creation circuit 52, the insertion shape creation circuit 53, the communication circuit 54, and the reference image storage unit 55 in the image processing apparatus 11 of FIG. The interpolation circuit 56, the three-dimensional human body image creation circuit 57, the synthesis circuit 58, the rotation conversion circuit 59, the three-dimensional guide image creation circuit A, the three-dimensional guide image creation circuit B, the mixing circuit 61, and the display The circuit 62 is controlled by a command from the control circuit 63. Details of the control will be described later.

  Hereinafter, the entire operation of the image processing device 11, the keyboard 13, the mouse 12, and the display device 14 according to the present embodiment will be described in accordance with the usage pattern of the surgeon. FIG. 17 is a flowchart of the entire process, and the processes in steps S1 to S4 are executed in this order.

  The first step S1 is a body surface feature point and body cavity feature point designation process on the reference image data. That is, in this step S1, processing for designating body surface feature points and body cavity feature points is performed on the reference image data.

  In the next step S <b> 2, the surgeon fixes the body surface detection coil 7 to the subject 37. The surgeon places the subject 37 in a body position with its left side thin, so-called left-sided position. The surgeon palpates the subject 37 and places the body at a position on the body surface closest to the four body surface feature points, the xiphoid process, the upper left anterior iliac spine, the upper right anterior iliac spine, and the lumbar spine process. The table detection coil 7 is fixed.

  The next step S3 is a correction value calculation process. In this step S3, the image processing apparatus 11 acquires the position / orientation data of the feature points in the body cavity, and the position / orientation data expressed on the orthogonal coordinate axis O-xyz is used as the orthogonal coordinate axis O′-x′y′z ′. A conversion equation that maps to the position / orientation mapping data in the voxel space expressed above is calculated, and further, a correction value of the conversion equation is calculated from the body cavity feature point coordinates.

  In the next step S4, ultrasonic tomographic image / three-dimensional guide image creation / display processing is performed. This step S4 is processing for creating and displaying an ultrasonic tomographic image and a three-dimensional guide image.

  Next, the process of step S1 in the flowchart of FIG. 17, that is, the body surface feature point and body cavity feature point designation process on the reference image data will be specifically described. FIG. 18 shows details of processing for designating body surface feature points and body cavity feature points on the reference image data in step S1 of FIG.

  In the first step S1-1, the operator presses the display switching key 13α. The control circuit 63 issues a command to the display circuit 62. The switch 62a of the display circuit 62 is switched to the input terminal α according to a command.

  In the next step S1-2, the operator uses the mouse 12 and the keyboard 13 to specify any one of the reference image data items 1 to N.

  In the next step S1-3, the control circuit 63 reads the designated reference image data from any one of the reference image data Nos. 1 to N stored in the reference image storage unit 55 in the display circuit 62. Let The display circuit 62 converts the reference image data from the reference image storage unit 55 into an analog video signal, and outputs the reference image data to the display device 14. The display device 14 displays reference image data.

  In the next step S1-4, the operator uses the mouse 12 and the keyboard 13 to designate body surface feature points on the reference image data. Specifically, it is as follows.

  The surgeon shows in the displayed reference image data any of the four body surface feature points of the subject 37, the xiphoid process, the upper left anterior iliac spine, the upper right anterior iliac spine, and the lumbar spine spinous process. Like that. If none is shown, the process returns to step S1-2, and the operator respecifies other reference image data. In step S1-3, display of different reference image data is repeated until the reflected reference image data is displayed. .

  The surgeon uses the mouse 12 and the keyboard 13 to display the xiphoid process which is four points on the body surface of the subject 37 on the displayed reference image data, the left upper anterior iliac spine, the upper right anterior iliac spine, the lumbar vertebrae A pixel corresponding to a point on the body surface closest to the spinous process is designated. The designated points are indicated by black circles ● and white circles ○ in FIG. 9.

  In the present embodiment, for the sake of explanation, the xiphoid process ○ is shown on the reference image data of n1 (1 ≦ n1 ≦ N), and the upper left anterior iliac spine, the upper right anterior iliac spine, the lumbar vertebral body The description will be made on the assumption that the spinous process ● is reflected on the reference image data of No. 2 (1 ≦ n2 ≦ N). In FIG. 9, for convenience of explanation, sword-shaped protrusions are indicated by circles at positions corresponding to the sword-shaped protrusions on the reference image data No.

  In the next step S1-5, the operator designates the body cavity feature point P "using the mouse 12 and the keyboard 13. In this embodiment, the duodenal papilla (duodenum of the common bile duct) is designated as the body cavity feature point P". A description will be given by taking as an example an opening to a duodenal papilla. Specifically, it is as follows.

  The surgeon uses the mouse 12 and the keyboard 13 to designate any reference image data from 1 to N. The control circuit 63 reads designated reference image data out of any one of reference image data 1 to N stored in the reference image storage unit 55 via a signal line (not shown) in the display circuit 62. Let The display circuit 62 outputs the read reference image data to the display device 14. The display device 14 displays this reference image data. If the duodenal papilla, which is a feature point in the body cavity of the subject 37, is not reflected in the displayed reference image data, the surgeon respecifies the other reference image data and differs until the reflected reference image data is displayed. Repeat display of reference image data.

  The operator uses the mouse 12 and the keyboard 13 to designate a pixel corresponding to the duodenal papilla that is a point in the body cavity of the subject 37 on the displayed reference image data. The designated point is indicated by P "in FIG. 9. In this embodiment, for convenience of explanation, the duodenal papilla P" is described as being reflected on the reference image data of the n2th (1 ≦ n2 ≦ N). .

  In the next step S1-6, the control circuit 63 determines each pixel corresponding to each body surface feature point designated in step S1-4 and a pixel corresponding to the body cavity feature point P ″ designated in step S1-5. And the coordinates on the orthogonal coordinate axes O′-x′y′z ′ stretched in the voxel space from the address on the reference image data are calculated and output to the matching circuit 51.

  The calculated values of the coordinates on the orthogonal coordinate axes O′-x′y′z ′ of the pixels corresponding to the body surface feature points designated in step S1-4 are (xa ′, ya ′, za ′), (xb ', yb', zb '), (xc', yc ', zc'), (xd ', yd', zd '). The calculated value of each coordinate on the pixel orthogonal coordinate axis O'-x'y'z 'corresponding to the feature point in the body cavity designated in step S1-5 is set to (xp ", yp", zp "). 51 stores the coordinates.

  After step S1-6 is completed, the process proceeds to step S2 in FIG. Then, after the process of step S2, the process proceeds to the correction value calculation process of step S3 of FIG. Details of the correction value calculation processing in step S3 are shown in FIG.

  As described above, in this step S3, the position / orientation data of the body cavity feature point is acquired, and the position / orientation data expressed on the orthogonal coordinate axis O-xyz is obtained on the orthogonal coordinate axis O'-x'y'z '. Is a process of calculating a conversion formula for mapping to position / orientation mapping data in the voxel space expressed by the above, and further calculating a correction value of the conversion formula from the position / orientation data of the feature points in the body cavity.

  When the correction value calculation process in step S3 in FIG. 17 is started, the operator presses the display switching key 13β in the first step S3-1 in FIG. In response to this instruction, the control circuit 63 issues a command to the display circuit 62. The switch 62a of the display circuit 62 is switched to the input terminal β according to a command.

  Next, in step S 3-2, the display circuit 62 converts the optical image data from the optical observation device 3 into an analog video signal, and outputs the optical image to the display device 14. The display device 14 displays an optical image.

  In the next step S3-3, the operator inserts the rigid portion 21 and the flexible portion 22 of the ultrasonic endoscope 2 into the body cavity into the subject 37.

  In the next step S3-4, the surgeon searches the feature point in the body cavity by moving the rigid portion 21 while observing the optical image. After the feature point in the body cavity is found, the operator moves the rigid portion 21 to the vicinity of the feature point in the body cavity.

  In the next step S3-5, the operator inserts the body cavity contact probe 8 from the forceps port 44 and projects from the projecting port 45 while observing the optical image. Then, the operator brings the tip of the body cavity contact probe 8 into contact with the body cavity feature point under the optical image field of view. This is shown in FIG. In FIG. 20, an optical image is displayed on the display screen. The optical image displays the duodenal papilla P and the body cavity contact probe 8 as examples of body cavity feature points.

  In the next step S3-6, the operator presses the body cavity feature point designation key 65. In the next step S3-7, the control circuit 63 issues a command to the matching circuit 51. The matching circuit 51 takes in the position / orientation data from the position / orientation calculation device 5 according to the command and stores it.

  In this position / orientation data, as described above, each directional component of each position vector of the four body surface detection coils 7 on the orthogonal coordinate axis O-xyz, that is, in this case, four body surface feature points. Coordinates on Cartesian coordinate axis O-xyz: (xa, ya, za), (xb, yb, zb), (xc, yc, zc), (xd, yd, zd), detection in body cavity on Cartesian coordinate axis O-xyz Each direction component of the position vector of the coil for use 42, that is, in this case, two types of data of coordinates (Xp, yp, zp) on the orthogonal coordinate axis O-xyz of the feature point in the body cavity is included.

  In the next step S3-8, the matching circuit 51 creates a first conversion expression that expresses the first mapping from the coordinates of the body surface feature points. Specifically, it is as follows.

First, the matching circuit 51 stores the following contents.
First, each coordinate on the orthogonal coordinate axis O'-x'y'z 'in the voxel space of each pixel corresponding to each body surface feature point specified in step S1: (xa', ya ', za' ), (Xb ', yb', zb '), (xc', yc ', zc'), (xd ', yd', zd ')
Secondly, the coordinates of the pixel corresponding to the feature point in the body cavity designated in step S1 on the orthogonal coordinate axis O'-x'y'z 'in the voxel space: (xp ", yp", zp ")
Thirdly, the coordinates of the body surface feature points captured in step S3-7 on the orthogonal coordinate axis O-xyz: (xa, ya, za), (xb, yb, zb), (xc, yc, zc) , (Xd, yd, zd)
Fourth, the coordinates of the body cavity feature points taken in step S3-7 on the orthogonal coordinate axis O-xyz: (xp, yp, zp)

  The matching circuit 51 includes the third coordinates (xa, ya, za), (xb, yb, zb), (xc, yc, zc), (xd, yd, zd), and the first coordinates. From the coordinates (xa ', ya', za '), (xb', yb ', zb'), (xc ', yc', zc '), (xd', yd ', zd'), the orthogonal coordinate axis O A first transformation expression is generated that represents a first mapping of an arbitrary point on -xyz to a point on the orthogonal coordinate axis O'-x'y'z 'in the voxel space. The first mapping and the first conversion formula are defined as follows.

  As shown in FIG. 9, three vectors from the xiphoid process to another point using the xiphoid process, the upper left anterior iliac spine, the upper right anterior iliac spine, and the lumbar spine spine process, which are body surface feature points Two oblique coordinate systems having a fundamental vector as a virtual vector are virtually (in the data space obtained by interpolating the reference image data in FIG. 9) on the object 37 and in the voxel space (in FIG. 9). Setting).

  The first mapping is “coordinates of an arbitrary point on the orthogonal coordinate axis O-xyz expressed in the oblique coordinate system on the subject 37” and “this arbitrary coordinate on the orthogonal coordinate axis O′-x′y′z ′”. This is a mapping from the subject 37 to the voxel space such that the coordinates after the point mapping are the same as the coordinates expressed in the oblique coordinate system in the voxel space. Also, the first conversion formula is to convert “coordinates of the arbitrary point on the orthogonal coordinate axis O-xyz” into “coordinates of the point after the first mapping in the voxel space on the orthogonal coordinate axis O′-x′y′z ′”. It is an expression to do.

  For example, as shown in FIG. 9, the position of the image position / orientation detection coil 31, that is, the point after mapping by the first mapping of the center of radial scanning and the center O ″ of the ultrasonic tomographic image is Q ′. Let the coordinates of Q ′ on the orthogonal coordinate axis O′-x′y′z ′ be (x0 ′, y0 ′, z0 ′). Using the first conversion formula, the coordinates of the point O ″ on the orthogonal coordinate axis O-xyz ( x0, y0, z0) are converted into coordinates (x0 ', y0', z0 ') of the point Q' on the orthogonal coordinate axes O'-x'y'z '.

  In the next step S3-9, as shown in FIG. 9, the matching circuit 51 maps the body cavity feature point P to the point P ′ in the voxel space by the first conversion formula. The coordinates of the body cavity feature point P on the orthogonal coordinate axis O-xyz are (xp, yp, zp). The coordinates of the point P ′ after the first mapping on the orthogonal coordinate axis O′-x′y′z ′ are defined as (xp ′, yp ′, zp ′).

In the next step S3-10, the matching circuit 51 determines the coordinates (xp ′, yp ′, zp ′) of the point P ′ on the orthogonal coordinate axis O′-x′y′z ′ in the voxel space and the step S1. From the coordinates (xp ", yp", zp ") on the orthogonal coordinate axis O'-x'y'z 'in the voxel space of the point P" corresponding to the specified body cavity feature point, a vector as follows P'P "is calculated.
P'P "= (xp", yp ", zp")-(xp ', yp', zp ') = (xp "-xp', yp" -yp ', zp "-zp')

  In the next step S3-11, the matching circuit 51 stores the vector P′P ″. The vector P′P ″ is used to correct the first conversion formula and create the second conversion formula in the process described later. Acts as a correction value. After step S3-11 ends, the process proceeds to next step S4.

  Next, the ultrasonic tomographic image / three-dimensional guide image creation / display process in step S4 will be described with reference to FIG. FIG. 21 shows details of processing for creating and displaying an actual ultrasonic tomographic image / three-dimensional guide image of the subject 37 in step S4.

  When the processing of step S4 is started, the operator presses the display switching key 13γ in the first step S4-1. The control circuit 63 issues a command to the display circuit 62. The switch 62a of the display circuit 62 is switched to the input terminal γ by this command.

  In the next step S4-2, the operator presses the scan control key 66. In the next step S4-3, the control circuit 63 outputs a scanning control signal to the ultrasonic observation apparatus 4. Then, the ultrasonic transducer array 29 starts radial scanning.

  In the next step S4-4, the control circuit 63 issues a command to the mixing circuit 61. The mixing circuit 61 sequentially captures the ultrasonic tomographic image data input according to the radial scan from the ultrasonic observation apparatus 4 in accordance with this command.

In the next step S4-5, the control circuit 63 issues a command to the matching circuit 51. The matching circuit 51 takes in the position / orientation data from the position / orientation calculation device 5 according to the command and stores it. This capture is instantaneous. Therefore, the matching circuit 51 captures position / orientation data including the following data at the moment when the mixing circuit 61 captures the ultrasonic tomographic image data in step S4-4.
Each position component of the position vector OO "of the position of the image position / orientation detection coil 31 on the orthogonal coordinate axis O-xyz, that is, the center of radial scanning and the center O" of the ultrasonic tomographic image: (x0, y0, z0)
Each angular component of Euler angle indicating the orientation of the image position orientation detecting coil 31 with respect to the orthogonal coordinate axis O-xyz, that is, the orientation of the ultrasonic tomographic image: (ψ, θ, φ)
Each direction component of each position vector of the plurality of insertion shape detection coils 32 on the orthogonal coordinate axis O-xyz: (xi, yi, zi) (i is a natural number from 1 to the total number of insertion shape detection coils 32)
Each direction component of each position vector of the four body surface detection coils 7 on the orthogonal coordinate axis O-xyz: (xa, ya, za), (xb, yb, zb), (xc, yc, zc), ( xd, yd, zd)

  In the next step S4-6, the matching circuit 51 determines each direction of the position vector of each of the four body surface detection coils 7 on the orthogonal coordinate axis O-xyz out of the position / orientation data acquired in step S4-5. Using the components (xa, ya, za), (xb, yb, zb), (xc, yc, zc), (xd, yd, zd), the first conversion formula stored in step S3 is updated.

Next, the matching circuit 51 newly creates a second conversion expression expressing the second mapping by combining the updated first conversion expression with the parallel movement by the vector P′P ″ stored in step S3. The concept of the mapping of 2 is as follows.
2nd map = 1st map + translation by vector P'P "

  The parallel movement by the vector P'P "has the following correction effect. The vector P'P" acts as a correction value. The first mapping is “coordinates of the arbitrary point on the orthogonal coordinate axis O-xyz expressed in the oblique coordinate system on the subject 37” and “the coordinates of the arbitrary point on the orthogonal coordinate axis O′-x′y′z ′”. The mapping from the subject 37 to the voxel space was made so that the “coordinates expressed by the oblique coordinate system in the voxel space of the points after mapping” were the same.

  Ideally, it is desirable that the mapping point P ′ by the first mapping of the in-vivo feature point P into the voxel space and the point P ″ corresponding to the in-body cavity feature point specified in step S1 match. However, it is actually difficult to match exactly.

  This is because “the spatial positional relationship between an arbitrary point on the orthogonal coordinate axis O-xyz and the oblique coordinate system on the subject 37” and “the orthogonal coordinate axis O′-x corresponding to the arbitrary point anatomically. The point in 'y'z' and the spatial positional relationship between the oblique coordinate system in the voxel space are not completely coincident due to various factors. In the present embodiment, the first mapping and the first transformation formula are obtained from the coordinates of the body surface feature points having features on the skeleton. The duodenal papilla P, which is a feature point in the body cavity, is obtained on the skeleton. This is because the body surface feature points are not always in the same positional relationship.

  This is mainly because the X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16 are normally imaged in the supine position and are different in posture from the time of the ultrasonic endoscope 2 examination in the left lateral position. The various organs in the subject 37 can be displaced according to gravity.

  For this reason, the mapping of the feature point P in the body cavity corresponds to the feature point in the body cavity in the voxel space by combining the first mapping with the parallel movement by the vector P′P ”as the correction value. Matches the point P ". Further, other points of the subject 37, for example, the center O "of the ultrasonic tomographic image, are more accurately anatomically matched by the second mapping.

  In the next step S4-7, the matching circuit 51 includes each directional component of the position vector OO "of the center O" of the ultrasonic tomographic image on the orthogonal coordinate axis O-xyz among the position / orientation data acquired in step S4-5. (x0, y0, z0), each angular component (ψ, θ, φ) of Euler angles indicating the orientation of the image position orientation detection coil 31 with respect to the orthogonal coordinate axis O-xyz, and a plurality of angular components on the orthogonal coordinate axis O-xyz. A second transformation formula that newly creates each directional component (xi, yi, zi) of each position vector of the insertion shape detection coil 32 (i is a natural number from 1 to the total number of insertion shape detection coils 32). To convert to position / orientation mapping data.

As shown in FIG. 9, in the first conversion formula, the center O ″ of the ultrasonic tomographic image is mapped to the point Q ′ on the voxel space, but by using the second conversion formula newly created in this step, As shown in FIG. 9, the center O "of the ultrasonic tomogram is mapped to a point Q" on the voxel space. A vector Q'Q "indicating the difference between Q 'and Q" is parallel to the second map. Since it matches the amount of correction by movement, it is the same as the vector P'P ". That is, the following expression is established.
Q'Q "＝ P'P"

  The next step S4-8 is a process for creating three-dimensional guide image data. That is, the image index creation circuit 52 creates image index data. The insertion shape creation circuit 53 creates insertion shape data. The synthesizing circuit 58 synthesizes the three-dimensional human body image data, the image index data, and the insertion shape data, and creates synthesized three-dimensional data. The rotation conversion circuit 59 performs rotation processing on the combined three-dimensional data. The three-dimensional guide image creation circuit A and the three-dimensional guide image creation circuit B each create three-dimensional guide image data. Each of the above processes is as described above.

  In the next step S4-9, the mixing circuit 61 generates the mixed data for display by arranging the ultrasonic tomographic image data and the three-dimensional guide image data. The display circuit 62 converts this mixed data into an analog video signal. The display device 14 arranges the ultrasonic tomographic image, the three-dimensional guide image obtained by observing the subject 37 from the ventral side, and the three-dimensional guide image observed in the same direction as the ultrasonic tomographic image based on the analog video signal. Is displayed. Each of the above processes is as described above.

  In the next step S4-10, the control circuit 63 confirms whether or not the operator presses the scan control key 66 again during steps S4-4 to S4-9. If the surgeon has pressed the scan control key 66 again, the control circuit 63 ends the above processing, and outputs a scan control signal for commanding the radial scan control OFF to the ultrasound observation apparatus 4. To do. Thereby, the ultrasonic transducer array 29 ends the radial scanning. If the surgeon has not pressed the scan control key 66 again, the process jumps to step S4-4.

  In this manner, by repeating the processing described in steps S4-4 to S4-9, the ultrasonic transducer array 29 performs one radial scan, and the ultrasonic observation apparatus 4 converts the ultrasonic tomographic image data. Each time the tomographic image data is created and input from the ultrasonic observation device 4 to the mixing circuit 61, two new three-dimensional guide images are created and displayed on the display screen of the display device 14 together with the new ultrasonic tomographic image. It is displayed while being updated in real time.

  That is, as shown in FIG. 16, the ultrasonic tomographic image marker Mu on the image index data is linked with the movement of the radial scanning plane accompanying the manual operation of the flexible portion 22 and the rigid portion 21 of the operator. The tip direction marker Md, the 6 o'clock direction marker Mt, the insertion shape marker Ms on the insertion shape data, and the coil position marker Mc move or deform on the three-dimensional human body image data.

  The above is the operation until the three-dimensional guide image is displayed. In the present embodiment, the following actions are further added before the above actions.

  It is originally desirable that the reference image data captured by the X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16 is captured from the same subject as the subject that scans the ultrasound with the ultrasound endoscope 2. The reason for this is to eliminate the anatomical individual difference between the guide image and the ultrasonic tomographic image simultaneously displayed on the display device 14 and improve the degree of coincidence.

  However, even if images are taken from the same subject, it is difficult to obtain a good match for the following reasons. Rather, reference image data of a subject different from the subject from which an ultrasonic tomographic image created under a certain condition is used is used. May have better anatomical agreement.

  G_A in FIG. 22 is three-dimensional human body image data based on normal reference image data, and pancreas and blood vessels are extracted. When an inspection using the ultrasonic endoscope 2 is performed on an actual human body, there are mainly four methods described below.

(First inspection method: duodenum descending leg PULL scan)
This is a method of observing the head of the pancreas (the side close to the aorta in the pancreas in FIG. 22) while pulling the ultrasonic endoscope 2 to the mouth side in the descending leg of the duodenum. Has been.

(Second inspection method: duodenum descending leg PUSH scan)
This is a method of observing the head of the pancreas while pushing the ultrasonic endoscope 2 toward the anus in the duodenal descending leg, and is in the opposite direction to the duodenal descending leg PULL scanning.

(Third examination method: stomach / duodenal bulb PULL scan)
The body part of the pancreas (near the middle of the pancreas in FIG. 22) and the tail (the lower part of the pancreas in FIG. 22 are narrowed while pulling the ultrasonic endoscope 2 from the duodenal bulb into the stomach toward the mouth. 22), which is indicated by a thick broken line arrow in FIG. The duodenal bulb and stomach are on the near side (vertical upper side on the paper) of the pancreas.

(Fourth examination method: stomach / duodenal bulb PUSH scan)
This is a method of observing the body and tail of the pancreas while pushing the ultrasonic endoscope 2 toward the anus from the stomach toward the duodenal bulb, and is in the opposite direction to the duodenal bulb PULL scanning. The duodenal bulb and stomach are on the near side (vertical upper side on the paper) of the pancreas.

  Of these, when the first duodenal descending leg PULL scan is performed, the head side of the pancreas is moved together as indicated by the block arrow shown in the image data G_A in FIG. 22 as the ultrasonic endoscope 2 is pulled to the mouth side. A phenomenon occurs in which it is pulled and rotated and moved. When imaging a subject with the X-ray three-dimensional helical CT apparatus 15 or the three-dimensional MRI apparatus 16, such a phenomenon is not assumed. If reference image data obtained by a normal imaging method is used, ultrasound is used. It is difficult to obtain a good anatomical match between the tomographic image and the guide image. Such a phenomenon does not occur in the second duodenal descending leg PUSH scan, the third stomach / duodenal bulb PULL scan, and the fourth stomach / duodenum bulb PUSH scan.

  Therefore, the reference image storage unit 55 stores in advance a plurality of pieces of image data corresponding to each of a plurality of states of an organ or an organ or tissue created by a specific subject as reference data. In the present embodiment, image data of the pancreas in a normal state and data obtained by rotating and moving the pancreas head are stored in advance in the reference image storage unit 55 as partial model image data. There are several ways to create data after rotating and moving.

(First data creation method)
In addition to normal imaging with the X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16, the ultrasound endoscope 2 is inserted, and the X-ray is again taken while pulling the head of the pancreas with the duodenal descending leg PULL scan. Imaging is performed by the three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16. In this way, normal pancreas reference image data and reference image data after the pancreas is rotated or moved can be obtained.

(Second data creation method)
After normal imaging with an X-ray 3D helical CT apparatus 15 or 3D MRI apparatus 16 on a specific subject, imaging with the ultrasound endoscope 2 is performed on this subject, and the guide image and the ultrasound tomographic image match Thus, new reference image data is created by rotating or moving the pancreas in the reference image data. In this way, normal pancreas reference image data and reference image data after the pancreas is rotated or moved can be obtained.

  In the second data creation method, creation of the three-dimensional human body image data G_B is performed according to the following procedure.

  First, the surgeon inputs the pancreas movement direction, movement distance, and rotation angle via the keyboard 13 and mouse 12. The three-dimensional human body image creation circuit 57 creates the three-dimensional human body image data by rotating and moving the pancreas from the original reference image data based on the input movement direction, movement distance, and rotation angle.

  Next, the synthesizing circuit 58 synthesizes the three-dimensional human body image data, the image index data, and the insertion shape data of the pancreas that has been rotated and moved to create synthesized three-dimensional data. The combined three-dimensional data is displayed on the display device 14 via the rotation conversion circuit 59, the three-dimensional guide image creation circuit A, the three-dimensional guide image creation circuit B, and the mixing circuit 61.

  The surgeon compares the anatomical coincidence between the ultrasonic tomogram on the display screen of the display device 14 and the two three-dimensional guide images. Then, the operator again moves the pancreas through the keyboard 13 and the mouse 12 and moves and moves and rotates so that the ultrasonic tomographic image and the two three-dimensional guide images coincide well anatomically. Enter the angle. Then, the above operation is repeated.

  Even if the reference image data created by either of the first and second methods is used, there is no change in the operation of the present embodiment. Therefore, in the following, the operation of the present embodiment is premised on the second method. Will be explained.

  In the present embodiment, the reference image storage unit 55 stores normal pancreas reference image data and reference image data after the pancreas is rotated or moved. The three-dimensional human body image data based on the normal pancreas reference image data is shown in G_A of FIG. 22, and the three-dimensional human body image data based on the reference image data after the pancreas is rotated and moved is shown in G_B of FIG.

  Selection of normal pancreas reference image data and reference image data after the pancreas is rotated or moved is performed by the keyboard 13, the mouse 12, and the control circuit 63 as a state selection unit. In the present embodiment, the operator presses one of the PUSH key 19a and the PULL key 19b, which are ultrasonic endoscope scanning information keys on the keyboard 13. If one of these keys 19a and 19b is ON, the other is OFF. In addition, the surgeon presses either the stomach / duodenum bulb key 18a or the duodenum descending leg key 18b, which is an ultrasonic scanning region key of the keyboard 13. If one of these keys 18a and 18b is ON, the other is OFF.

  The control circuit 63 determines which of the three-dimensional human body image data G_A and the three-dimensional human body image data G_B in FIG. 22 is used from the combination of ON and OFF of these keys 18a, 18b, 19a, and 19b. The judgment of the control circuit 63 is based on the table of FIG. That is, when the duodenum lowering leg key 18b and the PUSH key 19a are ON, the control circuit 63 is connected to the stomach / duodenum bulb key 18a and the PULL key 19b. Is selected as the reference image data, and when the duodenal descending leg key 18b and the PULL key 19b are ON, the image data G_B is selected as the reference image data.

  The interpolation circuit 56 rereads the reference image data again according to a command from the control circuit 63. In this way, the voxel space of the interpolation memory and the synthesis memory is filled with the reference image data read corresponding to the key combinations shown in the table of FIG. 23, and the 3D human body image data and the synthesis 3D data. Three-dimensional guide image data is replaced.

  Control circuit 63, interpolation circuit 56, 3D human body image creation circuit 57, synthesis circuit 58, rotation conversion circuit 59, 3D guide image creation circuit A, 3D guide image creation circuit B, and mixing circuit Since 61 operates in real time, the guide image is instantaneously switched in response to a key operation by the operator.

  In this way, the operator can perform the PULL scan of the ultrasonic endoscope 2 regardless of whether the leg is a descending leg of the duodenum, the stomach / duodenal bulb, or the PUSH scan of the ultrasonic endoscope 2. Even in this case, an anatomically good match can be obtained between the ultrasonic tomogram and the guide image.

  According to the present embodiment described above, the reference image storage unit 55 stores normal pancreas reference image data and reference image data after the pancreas is rotated or moved, and the control circuit 63 displays the reference image data. 22 is used to determine which one of the three-dimensional human body image data G_A or the three-dimensional human body image data G_B is to be used, and the interpolation circuit 56 rereads the reference image data in response to a command from the control circuit 63, and the interpolation memory and the synthesis memory The voxel space is filled with the reference image data read out corresponding to the key combinations shown in the table of FIG. 23, and the 3D human body image data and the guide image are instantaneously generated by the operator's key operation. It is configured to be switched. Thereby, a guide image that correctly indicates the actual anatomical position and orientation of the ultrasonic tomographic image can be displayed. In particular, in the duodenal descending leg PULL scan, the pancreatic head rotates and moves, so this effect is remarkable.

  In this embodiment, the image index creating circuit 52 creates image index data in which the ultrasonic tomographic image marker Mu is combined with the blue tip direction marker Md and the yellow-green arrow-shaped 6 o'clock direction marker Mt. The synthesizing circuit 58 synthesizes the three-dimensional human body image data, the image index data, and the insertion shape data in the same voxel space, and the mixing circuit 61 combines the ultrasonic tomographic image data from the ultrasonic observation apparatus 4 The display circuit 62 converts the mixed data into an analog video signal, and the display device 14 converts the ultrasonic tomogram and the three-dimensional guide on the basis of the analog video signal. It has a configuration and operation that displays images side by side.

  Therefore, according to the present embodiment, the positional relationship between the ultrasonic tomogram and the region of interest such as the pancreas can be guided, and the radial scanning plane of the ultrasonic endoscope with respect to the body cavity wall such as the digestive tract It is possible to guide the orientation and shape of the flexible portion and the hard portion. Therefore, the surgeon can visually grasp these relationships, and can easily perform diagnosis, treatment, etc. on the region of interest.

  In the present embodiment, the matching circuit 51 repeats the processes described in steps S4-4 to S4-9, and the mixing circuit 61 captures position / orientation data at the moment when the ultrasonic tomographic image data is captured. A new second transformation expression that expresses the second mapping is created by combining the first transformation expression and the translation by the vector P′P ″, and the position of the center O ″ of the ultrasonic tomographic image on the orthogonal coordinate axis O-xyz. Each direction component (x0, y0, z0) of the vector OO ", each angle component (ψ, θ, φ) of Euler angles indicating the orientation of the image position orientation detection coil 31 with respect to the orthogonal coordinate axis O-xyz, and the orthogonal coordinate axis Each directional component (xi, yi, zi) (i is a natural number from 1 to the total number of insertion shape detection coils 32) of each position vector of the plurality of insertion shape detection coils 32 in O-xyz・ Has a configuration and operation that repeats the process of converting to orientation map data There.

  Therefore, according to the present embodiment, even if the body position of the subject 37 changes during the examination with the ultrasonic endoscope 2, the ultrasonic tomogram, the flexible part 22, the rigid part 21, and the three-dimensional The ultrasonic tomographic image marker Mu, the tip direction marker Md, the 6 o'clock direction marker Mt, and the insertion shape marker Ms on the guide image have an effect of more accurately anatomically matching each other.

  The X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16 are normally imaged in the supine position, and the body position is different from that in the case of ultrasonic endoscopy in the left supine position. Therefore, the matching circuit 51 has a configuration and an operation for creating a second transformation expression that expresses the second mapping by combining the first mapping with the parallel movement by the vector P′P ”as the correction value. Therefore, in the present embodiment, the various organs in the subject 37 are subjected to ultrasonic endoscopy in the left lateral position as compared with the X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16. Even if it is displaced according to gravity at times, a more accurate anatomical coincidence is achieved with the second mapping at a point of the subject 37, for example, the center O "of the ultrasonic tomographic image. Therefore, the three-dimensional guide image can more accurately guide the ultrasonic tomographic image.

  Further, according to the present embodiment, the three-dimensional guide image creation circuit A has the three-dimensional guide image data observed in the direction from the subject's ventral side, the right side being the subject's head side, and the left side being the subject's foot side. It is configured and operated to create. The subject 37 is usually examined in the position of the left lateral position in the ultrasonic endoscopy, and the 3D guide image is also displayed in the left lateral position. Therefore, it is easy to compare the subject and the 3D guide image. The surgeon can easily understand the three-dimensional guide image, and can improve or appropriately support the operability during diagnosis and treatment by the surgeon.

  Further, according to the present embodiment, since the 3D guide image creating circuit A and the 3D guide image creating circuit B create a 3D guide image in which the line of sight is set in different directions, the ultrasonic tomographic image and the pancreas The position relationship with the region of interest such as the gastrointestinal tract can be guided from a plurality of directions, and the ultrasonic tomogram, the flexible portion 22 of the ultrasonic endoscope 2 and the rigid portion with respect to the body cavity wall such as the digestive tract It is possible to guide the orientation and shape of the head 21 from a plurality of directions, which is easy for the operator to understand.

  Further, according to the present embodiment, the three-dimensional guide image creation circuit B uses the normal line of the ultrasonic tomographic image marker Mu as the observation line of sight, that is, the screen normal line of the display device 14 based on the position / orientation mapping data. This three-dimensional guide image is configured and operated so as to create a three-dimensional guide image that faces the screen so that they coincide with each other and that the 6 o'clock direction marker Mt is set to face downward on the screen of the display device 14. And the ultrasonic tomographic images displayed in real time side by side on the screen of the display device 14 coincide with each other. Therefore, it is easy for the operator to compare the two, and to easily interpret the ultrasonic tomogram anatomically.

  In particular, the three-dimensional guide image observed from the ventral side is effectively a wide range of guide images, and the three-dimensional guide image observed in the same direction as the ultrasonic tomographic image is a detailed guide image. The rough anatomical position of the image can be grasped, and the ultrasonic scanning plane can be fine-tuned while obtaining the detailed anatomical interpretation of the ultrasonic tomographic image in the latter, so that the examination is efficient.

  Further, according to the present embodiment, the three-dimensional guide image creation circuit B has the distal end side of the flexible portion 22, that is, the display device, out of two regions divided by the ultrasonic tomographic image marker Mu in the image index data. The front side of the screen 14 is not displayed, and the three-dimensional guide image data in which the luminance of the portion on the ultrasonic tomographic image marker Mu and the portion on the back side is changed is configured and operated. Therefore, the organ on the near side does not obstruct the operator's observation of the three-dimensional guide image, and this three-dimensional guide image is compared with the ultrasonic tomographic image displayed in real time on the screen of the display device 14. It is easier to do and anatomical interpretation of ultrasonic tomograms.

<Modification>
In the above-described embodiment, the ultrasonic endoscope 2 including the treatment instrument channel 46 and the body cavity contact probe 8 inserted through the treatment instrument channel 46 are provided. However, the configuration is not limited to this. Absent.

  If the objective lens 25 is focused on the feature point in the body cavity via the optical observation window 24 and the rigid part 21 itself can be accurately brought into contact with the feature point in the body cavity without using the contact probe 8 in the body cavity, the rigid part 21 is obtained. The image position / orientation detection coil 31 fixed to the body cavity may be substituted for the body cavity detection coil 42 of the body cavity contact probe 8. At this time, the image position / orientation detection coil 31 acts not only as an image position / orientation detection element but also as a body cavity detection element.

  In the present embodiment, the electronic radial scanning ultrasonic endoscope 2 is used as an ultrasonic probe. However, like the ultrasonic diagnostic apparatus disclosed in Japanese Patent Application Laid-Open No. 2004-113629, It can be a mechanical scanning ultrasonic endoscope, an electronic convex scanning ultrasonic endoscope with a group of ultrasonic transducers on one of the insertion axes, or a capsule-type ultrasonic sonde. It is not limited to. An ultrasonic probe without the optical observation window 24 may be used.

  In the present embodiment, the ultrasonic transducers are cut into strips in the rigid portion 21 of the ultrasonic endoscope 2 and arranged as an annular array around the insertion axis. 29 may be provided all around 360 ° or may be missing. For example, the ultrasonic transducer array 29 may be formed in a portion extending over 270 ° or 180 °.

  In this embodiment, the transmitting antenna 6 and the receiving coil are used as position detecting means and the position and orientation are detected and detected by the magnetic field. However, transmission and reception may be reversed. In the case of detecting the position and orientation using a magnetic field, the position (orientation) detection means can be formed with a simple configuration, and the cost and size can be reduced. However, the position and orientation are limited to those using a magnetic field. Instead, the position and orientation may be detected by acceleration or other means. Further, in the present embodiment, the origin O is set to a specific position on the transmission antenna 6, but it may be configured to be set to another place where the positional relationship with the transmission antenna 6 does not change.

  Further, in the present embodiment, the image position / orientation detection coil 31 is fixed to the rigid portion 21, but it may not be completely inside the rigid portion 21 as long as the position is fixed to the rigid portion 21. . Further, in the present embodiment, each organ on the three-dimensional guide image data is configured to be color-coded for each organ. However, the present invention is not limited to the color-coded mode, but other modes such as luminance, brightness, and saturation. For example, the luminance value may be changed for each organ.

  In this embodiment, the reference image data is configured and operated so that a plurality of two-dimensional CT images and two-dimensional MRI images captured by the X-ray three-dimensional helical CT apparatus 15 and the three-dimensional MRI apparatus 16 are used. However, three-dimensional image data acquired in advance using another modality such as PET (Positoron Emission Tomography) may be used. Further, three-dimensional image data acquired in advance by a so-called extracorporeal ultrasonic diagnostic apparatus that irradiates ultrasonic waves from outside the body may be used.

  Further, in the present embodiment, a body surface detection coil 7 composed of four coils wound around one axis is provided, and a plurality of body surface feature points are provided on the subject body surface by tape, belt, band, etc. The position / orientation data of the body surface feature points are obtained at the same time. However, instead of using one coil, for example, the detection coil 42 in the body cavity, for the examination by the ultrasonic endoscope 2. Prior to placing the subject 37 in the left-side position, the position and orientation data of the body surface feature points are sequentially obtained by sequentially bringing the tip of the body cavity contact probe 8 into contact with a plurality of body surface feature points. Anyway.

  Further, in the present embodiment, the position / orientation calculation device calculates the position of the body surface detection coil 7 as position / orientation data, but may calculate the direction of the winding axis instead of the position. Further, both the position and the direction of the winding axis may be calculated. The number of body surface detection coils 7 can be reduced by increasing the degree of freedom that the position / orientation calculation device 5 calculates for one body surface detection coil 7, and the body surface detection coil 7 is attached to the subject 37. It is possible to reduce the burden on the operator and the subject 37 during fixation or during the ultrasonic endoscopy.

  In this embodiment, the body surface feature points are the xiphoid process of the abdominal body surface, the upper left anterior iliac spine, the upper right anterior iliac spine, and the lumbar spine spinous process, and the body cavity feature point is the duodenal papilla. However, the present invention is not limited to this example, but may be a feature point in the chest body surface, a chest body cavity, or another example. In general, the accuracy of the orientation of the ultrasonic tomographic image marker Mu is better when the body surface feature point is a point related to the skeleton.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. FIGS. 24 to 26 relate to the second embodiment of the present invention, FIG. 24 is an explanatory diagram showing composition and deformation of 3D human body image data, and FIG. 25 is a correspondence between key combinations and 3D human body image data. FIG. 26 is a block diagram showing the configuration of the image processing apparatus.

  The second embodiment is different from the first embodiment in the configuration of the composition circuit 58 of the image processing apparatus 11. As shown in FIG. 26, the synthesis circuit 58_2 of the image processing apparatus 11_2 of this embodiment adds another synthesis memory (volume memory) 58b to the synthesis circuit 58 of the first embodiment. ing.

  The operation of the second embodiment differs from that of the first embodiment in the operation of the reference image storage unit 55 and the three-dimensional human body image creation circuit 57 due to the addition of a volume memory.

  In the first embodiment, the reference image storage unit 55 stores normal pancreas reference image data and reference image data after the pancreas is rotated or moved, and the control circuit 63 is a three-dimensional human body shown in FIG. It acted to determine which of the image data G_A and the three-dimensional human body image data G_B is used.

  On the other hand, in the second embodiment, the three-dimensional human body image creation circuit 57 generates the three-dimensional human body image data G_A based on the normal pancreas reference image data shown in FIG. 22 and the synthesis memories 58a and 58b of the synthesis circuit 58_2. One of them is memorized.

  Further, as shown in FIG. 24, the three-dimensional human body image creation circuit 57 combines the three-dimensional human body image data G_A with the three-dimensional human body image data G_B based on the reference image data after the pancreas is rotated or moved. The three-dimensional human body image data G_C is generated by deformation. The three-dimensional human body image data G_C is data newly created from the three-dimensional human body image data G_A and the three-dimensional human body image data G_B, and includes the deformed virtual pancreas.

  In FIG. 24, for convenience of explanation, the position is aligned at the tail end point of the pancreas as an example. The reason for the tail end point is that even if the tail end point is the duodenal descending leg PULL scan, it is the stable point that does not rotate or move in the pancreas.

  A method of generating the three-dimensional human body image data G_C by synthesizing and transforming the above-described three-dimensional human body image data G_A and the three-dimensional human body image data G_B is as follows.

  The surgeon inputs the pancreas movement direction, movement distance, and rotation angle via the keyboard 13 and mouse 12. The three-dimensional human body image creation circuit 57 creates the three-dimensional human body image data by rotating and moving the pancreas from the original reference image data based on the input movement direction, movement distance, and rotation angle.

  Next, the synthesizing circuit 58_2 synthesizes the three-dimensional human body image data, image index data, and insertion shape data of the pancreas that has been rotated and moved to create synthesized three-dimensional data. The combined three-dimensional data is displayed on the display device 14 via the rotation conversion circuit 59, the three-dimensional guide image creation circuit A, the three-dimensional guide image creation circuit B, and the mixing circuit 61.

  The surgeon compares the anatomical coincidence between the ultrasonic tomogram on the display screen of the display device 14 and the two three-dimensional guide images. Again, the surgeon then moves the pancreas through the keyboard 13 and the mouse 12 and the distance and rotation of the pancreas so that the ultrasound tomographic image and the two three-dimensional guide images agree well anatomically. Enter the angle.

  Then, the above-described operation is repeated to create a three-dimensional guide image. The steps so far are the same as those in the first embodiment.

  Further, the surgeon superimposes the three-dimensional guide image via the keyboard 13 and the mouse 12 as shown in FIG. 24 and displays them on the display screen of the display device 14. Then, the display screen is traced via the keyboard 13 and the mouse 12 so that the pancreas has an appropriate shape.

  The three-dimensional human body image creation circuit 57 creates the three-dimensional human body image data G_C again based on the traced information and stores it in the other one of the synthesis memories 58a and 58b of the synthesis circuit 58_2.

  Which one of the three-dimensional human body image data G_A in FIG. 22 and the three-dimensional human body image data G_C in FIG. 24 is used depends on whether the ultrasonic endoscope scanning part key or the ultrasonic endoscope scanning information key is ON or OFF. The control circuit 63 determines from the combination state.

  The determination of the control circuit 63 is based on the table of FIG. That is, when the duodenum lowering leg key 18b and the PUSH key 19a are ON, the control circuit 63 is connected to the stomach / duodenum bulb key 18a and the PULL key 19b. When ON is ON, image data G_A is selected as reference image data, and when the duodenal descending leg key 18b and PULL key 19b are ON, 3D human body image data G_A and 3D human body image data G_B are combined and transformed. The image data G_C created in this way is selected as reference image data.

  The synthesizing circuit 58_2 switches between the three-dimensional human body image data G_A and the three-dimensional human body image data G_C as the three-dimensional human body image data to be combined with the image index data and the insertion shape data in accordance with a command from the control circuit 63. .

  In this manner, the 3D human body image data, the synthesized 3D data, and the 3D guide image data are replaced. Which of the control circuit 63, the three-dimensional human body image creation circuit 57, the synthesis circuit 58_2, the rotation conversion circuit 59, the three-dimensional guide image creation circuit A, the three-dimensional guide image creation circuit B, and the mixing circuit 61? Since it also operates in real time, the guide image can be switched instantaneously in response to a key operation by the operator.

  In this way, the operator can perform the PULL scan of the ultrasonic endoscope 2 regardless of whether the leg is a descending leg of the duodenum, the stomach / duodenal bulb, or the PUSH scan of the ultrasonic endoscope 2. Even in this case, an anatomically good match can be obtained between the ultrasonic tomogram and the guide image. Other operations are the same as those in the first embodiment.

  In the second embodiment, the three-dimensional human body image creation circuit 57 generates three-dimensional human body image data G_A based on the normal pancreas reference image data shown in FIG. 22 in the two synthesis memories 58a and 58b of the synthesis circuit 58_2. One side is memorize | stored, 3D human body image data G_C and 3D human body image data G_B based on the reference image data after the pancreas is rotated and moved are synthesized and transformed to create 3D human body image data G_C. The three-dimensional human body image data G_C is stored in the other of the synthesis memories 58a and 58b of the synthesis circuit 58_2.

  Then, the control circuit 63 determines which one of the 3D human body image data G_A in FIG. 22 and the 3D human body image data G_C in FIG. Judging from the combination of ON and OFF of the information key, the synthesizing circuit 58_2 converts the three-dimensional human body image data to be synthesized with the image index data and the insertion shape data into the three-dimensional human body image data G_A according to a command from the control circuit 63. Switching to 3D human body image data G_C, the 3D human body image data, the synthesized 3D data, and the 3D guide image data are replaced, and the guide image is switched instantaneously in response to a key operation by the operator.

  Therefore, in the second embodiment, there is no need to read the reference image data and create 3D human body image data again as in the first embodiment, and the actual anatomical image of the ultrasonic tomographic image is eliminated. A guide image that correctly indicates the position and orientation can be obtained and displayed at higher speed. Further, since the pancreas deformed by the duodenal descending leg PULL scan mainly occurs in the head of the pancreas and not in the tail, in the second embodiment, a guide closer to the actual pancreas deformed than the first embodiment. An image can be displayed. Other effects are the same as those of the first embodiment.

The block diagram which shows the structure of the ultrasonic diagnosing device concerning the 1st Embodiment of this invention. As above, an explanatory diagram schematically showing a body surface detection coil in a usage example, Side view showing the body cavity contact probe As above, a block diagram showing the configuration of the image processing apparatus As above, an explanatory diagram showing reference image data stored in the reference image storage unit Same as above, explanatory diagram showing voxel space Same as above, explanatory diagram showing the keyboard layout Same as above, explanatory diagram showing an orthogonal base with the origin set on the transmit antenna to represent position and orientation data The same as above, an explanatory view showing a state in which the feature point in the body cavity on the subject side is mapped to the voxel space As above, an explanatory diagram showing how image index data is created by the image index creating circuit As above, an explanatory diagram showing how the insertion shape data created by the insertion shape creation circuit is created Explanatory drawing showing 3D human body image data As above, an explanatory diagram showing a state in which the image index data and the insertion shape data are embedded in the voxel space in the synthesis memory by the synthesis circuit. As above, an explanatory diagram showing three-dimensional guide image data when observed from the ventral side of the subject As above, an explanatory diagram showing three-dimensional guide image data when observed from the same direction as the ultrasonic tomographic image The figure which shows a three-dimensional guide image and an ultrasonic tomogram displayed on a display apparatus as above. Same as above, flowchart showing overall processing Flowchart showing specific processing contents of body surface feature point and body cavity feature point designation processing on reference image in FIG. Same as above, a flowchart showing specific processing contents of the correction value calculation processing in FIG. Same as above, explanatory diagram of processing in FIG. The flowchart showing the specific processing contents of the ultrasonic tomographic image / three-dimensional guide image creation / display processing in FIG. Explanatory drawing showing rotation / movement of 3D human body image data As above, an explanatory diagram showing correspondence between key combinations and reference image data Explanatory drawing which shows composition and modification of 3D human body image data according to the second embodiment of the present invention. Same as above, explanatory diagram showing correspondence between key combinations and 3D human body image data As above, a block diagram showing the configuration of the image processing apparatus

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Ultrasonic diagnostic apparatus 2 Ultrasound endoscope 3 Optical observation apparatus 4 Ultrasonic observation apparatus 5 Position orientation calculation apparatus 11 Image processing apparatus 12 Mouse 13 Keyboard 18a Duodenal bulb part key 18b Duodenum descending leg key 19a PUSH key 19b PULL key 29 Ultrasonic transducer array 29a Ultrasonic transducer 55 Reference image storage unit 57 3D human body image creation circuit 60 3D image creation circuit 63 Control circuit 66 Scan control key

Claims (9)

An elongated ultrasonic probe that is inserted into a body cavity of a living body and obtains an ultrasonic signal for creating an ultrasonic tomographic image by scanning ultrasonic waves;
An ultrasonic tomographic image creating unit for creating an ultrasonic tomographic image based on the ultrasonic signal;
A detection unit for detecting the position and / or orientation of the ultrasonic tomogram;
An ultrasonic probe information selection unit for selecting scanning information of the ultrasonic probe in the living body;
A plurality of first image data groups corresponding to each of a plurality of states related to the region of interest when the region of interest is in a first state, which is a normal state, with the organ, organ or tissue as a region of interest; Reference data constituted by a plurality of second image data groups corresponding to each of a plurality of states related to the region of interest when the region of interest is in a second state where the organ is deformed or moved. A reference data holding unit to hold,
A guide image creation unit that creates a guide image of the anatomical position and / or orientation of the ultrasonic tomogram based on the reference data using the position and / or orientation;
A display unit for displaying the guide image;
Based on the in-vivo scanning information of the ultrasound probe selected by the ultrasound probe information selection unit, it is determined whether the region of interest affected by the ultrasound probe is in the first state or the second state. And a state selection unit that selects which state of the plurality of states the region of interest has,
Comprising
The guide image creation unit acquires image data corresponding to the state of the region of interest selected by the state selection unit from the first plurality of image data groups or the second plurality of image data groups, and Create the guide image based on the acquired image data
An ultrasonic diagnostic apparatus. The ultrasonic diagnostic apparatus according to claim 1, wherein the scanning information is information related to any one of a position, a moving direction, and a moving locus of the ultrasonic probe in a living body . A condition input unit for inputting a condition for selecting information related to any of the position, movement direction, or movement locus of the ultrasonic probe in the living body;
The ultrasonic probe information selection unit selects information related to any one of a position, a moving direction, and a moving locus of the ultrasonic probe in a living body according to the condition input from the condition input unit. Item 3. The ultrasonic diagnostic apparatus according to Item 2 . The position of the ultrasonic probe in the living body is any one of the esophagus, stomach, duodenum, duodenal bulb, duodenal descending leg or duodenal transverse leg, or any specific part of these. The ultrasonic diagnostic apparatus according to claim 2 . The ultrasonic diagnostic apparatus according to claim 2, wherein the moving direction is a direction in which the ultrasonic probe is pulled out or pushed in the body cavity of the living body . The ultrasonic diagnostic apparatus according to claim 1, wherein the reference data includes a plurality of partial model image data including the region of interest . Specific partial model image data of the plurality of partial model image data includes a plurality of image data corresponding to each of a plurality of states of the region of interest,
The guide image creation unit uses image data corresponding to the state selected by the state selection unit as the specific partial model image data, and a reference composed of the plurality of partial model image data including the image data The ultrasonic diagnostic apparatus according to claim 6, wherein the guide image is created based on data . Equipped with a slender ultrasonic probe that is inserted into the body cavity of a living body and obtains an ultrasonic signal for creating an ultrasonic tomogram by scanning ultrasonic waves,
The plurality of states are states before and after deformation or movement or rotation of the organ caused by the ultrasonic probe,
The ultrasonic diagnostic apparatus according to claim 7, wherein the specific partial model image data is a plurality of image data corresponding to states before and after deformation, movement, or rotation of the organ . Equipped with a slender ultrasonic probe that is inserted into the body cavity of a living body and obtains an ultrasonic signal for creating an ultrasonic tomogram by scanning ultrasonic waves,
The plurality of states are states before and after deformation of the organ caused by the ultrasonic probe,
The specific partial model image data is a plurality of image data corresponding to respective states before and after the deformation of the organ;
8. The image data corresponding to the post-deformation state of the organ is image data obtained by combining the pre-rotation state image data and the post-rotation state image data. The ultrasonic diagnostic apparatus as described .