WO2017195540A1

WO2017195540A1 - Ultrasound imaging device, image processing device and method therefor

Info

Publication number: WO2017195540A1
Application number: PCT/JP2017/015573
Authority: WO
Inventors: 子盛黎; 荒井　修
Original assignee: 株式会社日立製作所
Priority date: 2016-05-12
Filing date: 2017-04-18
Publication date: 2017-11-16
Also published as: JP2017202125A; JP6689666B2; CN108697410A; CN108697410B

Abstract

The present invention displays, automatically and in real time, information of an anatomical feature part of a subject, in an intraoperative ultrasound image, and accurately guides an operation. An ultrasound imaging device comprises: an image generation unit that transmits ultrasonic waves to a subject, generates an ultrasonic 2D image from reception signals of an ultrasonic probe which receives ultrasonic waves from a subject, and generates an ultrasonic 3D image by sending and receiving ultrasonic waves multiple times; and an image processing device that processes the ultrasonic 2D image and the ultrasonic 3D image. The image processing device includes: an ultrasonic 3D image feature part position estimation and identification unit 22 that estimates and identifies a feature part of a subject from an ultrasonic 3D image acquired by an ultrasonic 3D image acquisition unit 21; an ultrasonic 2D-3D image position aligning unit 25 that aligns the positions of the ultrasonic 2D image and the ultrasonic 3D image; and an image display unit 26 that uses obtained feature part position information and the position alignment results to display, on a real-time ultrasonic 2D image, information on the feature part position, name and distance relationship with the ultrasonic 2D image.

Description

Ultrasonic imaging apparatus, image processing apparatus, and method thereof

The present invention relates to an ultrasonic imaging apparatus, and more particularly to an imaging technique for simultaneously displaying an imaged ultrasonic image and a predetermined characteristic part in a subject.

Since the ultrasonic imaging apparatus irradiates the subject with ultrasonic waves and images the structure inside the subject using the reflected signal, the patient can be observed non-invasively and in real time. In addition, the positional relationship of the scan plane is calculated by attaching a position sensor to the ultrasonic probe, and it corresponds to the image of the ultrasonic scan plane from the three-dimensional diagnostic volume (3D image) data captured from the medical image diagnostic apparatus. An image diagnostic system that constructs and displays a two-dimensional cross-sectional image is also becoming popular. The diagnostic 3D image data is generally image data taken from another medical image pickup device such as an X-ray CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device in addition to ultrasound.

In Patent Document 1, a two-dimensional cross-sectional image corresponding to an ultrasonic two-dimensional (2D) image that is an image of an ultrasonic scan plane is constructed from diagnostic three-dimensional (3D) image data captured from a medical image diagnostic apparatus. At this time, the cross-sectional direction of the two-dimensional cross-sectional image is set according to the purpose of diagnosis and the type of the ultrasonic probe. Based on the obtained cross-sectional direction and the position information of the position sensor attached to the ultrasonic probe, the ultrasonic scan plane image and the diagnostic 3D image are aligned, and the diagnostic 3D image is two-dimensionally converted. Build and display cross-sectional images. Non-Patent Document 1 discloses an ultrasonic 3D image stitching method and the like.

JP 2014-239731 A

In recent years, it has been desired to confirm an area to be operated such as a tumor during an operation on a subject with an intraoperative ultrasound image that can be imaged non-invasively and in real time. Furthermore, in order to accurately guide the operation, it is desired to display information such as the position, name, and distance relationship of the tumor in the subject and the anatomical features in real time on the intraoperative ultrasound image. Yes. In addition, it is desirable to avoid as much as possible that the hand of a user such as a doctor during surgery touches a switch or a mouse of a manual input device in order to align the intraoperative ultrasound image and the subject. In addition, it is desirable to perform alignment in as short a time as possible in order to reduce the burden on the subject in the open state.

However, in the technique of Patent Document 1, a user sets a point on a mark that can be observed on both the ultrasonic scan plane image and the two-dimensional cross-sectional image constructed from the diagnostic 3D image, and matches the positions. Need to be aligned. Such a complicated user operation and the burden on the subject in an open state are major issues. Moreover, in the technique of Patent Document 1, there is a problem that information such as the position and name of an anatomical feature portion cannot be displayed in real time.

An object of the present invention is to provide an ultrasonic imaging apparatus, an image processing apparatus, and a method thereof that are capable of automatically and in real time displaying information on characteristic parts in an intraoperative ultrasonic image and accurately guiding a surgery. It is in.

In order to achieve the above object, according to the present invention, there is provided an ultrasonic imaging apparatus that transmits an ultrasonic wave to a subject and receives an ultrasonic wave from the subject, and an ultrasonic probe. An image generation unit that generates an ultrasonic 2D image from a received signal of a child and generates an ultrasonic 3D image by transmitting and receiving ultrasonic waves a plurality of times, and an image processing device that receives and processes the ultrasonic 2D image and the ultrasonic 3D image The image processing apparatus estimates and identifies the characteristic part of the subject from the ultrasonic 3D image, aligns the ultrasonic 2D image and the ultrasonic 3D image, and places the characteristic part on the ultrasonic 2D image. An ultrasonic imaging apparatus for displaying the information is provided.

In order to achieve the above object, according to the present invention, an image processing apparatus includes a feature part position estimation / identification unit that estimates and identifies a feature part of a subject from an ultrasonic 3D image of the subject, Image processing including an image alignment unit that performs alignment between the ultrasonic 2D image and the ultrasonic 3D image of the subject and calculates the position of the 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image Providing the device.

Furthermore, in order to achieve the above object, according to the present invention, there is provided an image processing method in an image processing apparatus, wherein the image processing apparatus estimates and identifies a characteristic part of a subject from an ultrasonic 3D image of the subject. Provided is an image processing method for aligning an ultrasonic 2D image and an ultrasonic 3D image of a subject and calculating a position of a 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image.

According to the present invention, the characteristic part of the subject is estimated and identified from the ultrasonic 3D image, the ultrasonic 2D image and the ultrasonic 3D image are aligned, and the characteristic part information is displayed on the ultrasonic 2D image. Therefore, it is possible to accurately guide the operation.

1 is a block diagram showing an example of the overall configuration of an ultrasonic imaging apparatus according to Embodiment 1. FIG. FIG. 1 is a block diagram illustrating a hardware configuration example of an ultrasonic imaging apparatus according to a first embodiment. 1 is a functional block diagram of an image processing apparatus of an ultrasonic imaging apparatus according to Embodiment 1. FIG. FIG. 3 is a flowchart showing a process flow of the ultrasonic imaging apparatus according to the first embodiment. FIG. 3 is an explanatory diagram illustrating an example of a characteristic part according to the first embodiment. Explanatory drawing which shows the other example of the characteristic part based on Example 1. FIG. Explanatory drawing which shows the other example of the characteristic part based on Example 1. FIG. FIG. 3 is a diagram illustrating an example of ultrasonic feature region information according to the first embodiment. FIG. 3 is a flowchart showing a position estimation / identification process of a characteristic part from volume data according to the first embodiment. FIG. 3 is a flowchart illustrating an alignment process between an ultrasonic 2D image and an ultrasonic 3D image according to the first embodiment. FIG. 3 is a diagram illustrating an initial position of an alignment process between an ultrasonic 2D image and an ultrasonic 3D image according to the first embodiment. FIG. 6 is a functional block diagram of an image processing apparatus of an ultrasonic imaging apparatus according to a second embodiment. The figure which shows an example of the display screen of a display, and a button selection means based on each Example. FIG. 9 is a block diagram illustrating an example of the overall configuration of an ultrasonic imaging apparatus according to a third embodiment. FIG. 9 is a block diagram illustrating a hardware configuration example of an ultrasonic imaging apparatus according to a third embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted. In this specification, the feature part information means the position and name of the feature part, and also the distance relation information, and the distance relation means the projection distance from the feature part to the ultrasonic 2D image. .

The first embodiment is an ultrasonic imaging apparatus that transmits an ultrasonic wave to a subject and receives an ultrasonic wave from the subject, and an ultrasonic wave 2D from a reception signal of the ultrasonic probe. The image processing apparatus includes an image generation unit that generates an image and generates an ultrasonic 3D image by transmitting and receiving ultrasonic waves a plurality of times, and an image processing apparatus that receives and processes the ultrasonic 2D image and the ultrasonic 3D image. The ultrasonic imaging that estimates and identifies the characteristic part of the subject from the ultrasonic 3D image, aligns the ultrasonic 2D image and the ultrasonic 3D image, and displays the characteristic part information on the ultrasonic 2D image It is the Example of an apparatus. In addition, the image processing apparatus is a feature part position estimation / identification unit that estimates and identifies a feature part of a subject from an ultrasound 3D image of the subject, an ultrasound 2D image of the subject, and an ultrasound 3D. It is an Example of an image processing apparatus provided with the image position alignment part which performs position alignment with an image and calculates the position of 2D cross-sectional image of the ultrasonic 3D image corresponding to an ultrasonic 2D image, and its method.

In the ultrasonic imaging apparatus according to the present embodiment, position estimation and name identification are performed on a predetermined anatomical feature portion from an ultrasonic 3D image obtained by imaging a subject. In addition, an ultrasonic 2D image captured in real time during surgery, that is, a 2D image of an ultrasonic scan surface and an ultrasonic 3D image are aligned, a geometric transformation matrix for alignment is calculated, and an estimated feature part And the distance relationship between the position of the captured ultrasonic 2D image. By displaying the distance / relationship between the name / position of the obtained characteristic part and the ultrasonic 2D image on the ultrasonic 2D image as characteristic part information, it is possible to guide the operation in real time.

Here, the distance relationship which is one of the information on the characteristic part in the present embodiment means the projection distance from the characteristic part in the subject to the ultrasonic 2D image. Preferably, the 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image from the three-dimensional position of the characteristic part of the subject estimated from the ultrasonic 3D image, that is, the coordinates of the point in the 3D space. , That is, the projection distance to the position of the surface in the three-dimensional space, and the calculated projection distance is set as a distance relationship with the ultrasonic 2D image of the characteristic part.

<Configuration and operation>
Hereinafter, a specific configuration example of the ultrasonic imaging apparatus according to the first embodiment will be described in detail. As shown in FIG. 1, the ultrasonic imaging apparatus of the present embodiment includes an ultrasonic probe 7, an image generation unit 107, and an image processing device 108, and further includes a transmission unit 102, a transmission / reception switching unit 101, The receiving unit 105, a user interface (UI) 121, and a control unit 106 are configured. An image obtained by the ultrasonic imaging apparatus 100 is displayed on the display 16. The display 16 may be included in the user interface (UI) 121. The configuration example of the ultrasonic imaging apparatus shown in FIG. 1 is commonly used in other embodiments.

The transmission unit 102 generates a transmission signal under the control of the control unit 106 and passes it to each of a plurality of ultrasonic elements constituting the ultrasonic probe 7 called an ultrasonic probe. Thereby, each of the plurality of ultrasonic elements of the ultrasonic probe 7 transmits an ultrasonic wave toward the subject 120. The ultrasonic waves reflected by the subject 120 reach the plural ultrasonic elements of the ultrasonic probe 7 again and are received and converted into electric signals. The signal received by the ultrasonic element is delayed by a predetermined delay amount corresponding to the position of the reception focal point and phased and added by the receiving unit 105. This is repeated for each of a plurality of reception focal points. The phasing addition signal is transferred from the reception unit 105 to the image generation unit 107. The transmission / reception switching unit 101 selectively connects the transmission unit 102 or the reception unit 105 to the ultrasound probe 7.

The image generation unit 107 performs processing such as arranging the phasing addition signal received from the reception unit 105 at a position corresponding to the reception focus, and generates an ultrasonic 2D image. While the user rolls the ultrasonic probe 7, the image generation unit 107 can generate a plurality of ultrasonic 2D images and synthesize the ultrasonic 3D images.

The image processing apparatus 108 receives the ultrasonic 3D image from the image generation unit 107, and performs name identification and position estimation of a predetermined anatomical feature part. Further, the image processing apparatus 108 receives the ultrasonic 2D image generated in real time, aligns the ultrasonic 2D image and the ultrasonic 3D image, and obtains the name and position of the obtained characteristic part and the ultrasonic 2D. The distance relationship with the image is displayed on the ultrasonic 2D image generated in real time.

Hereinafter, specific configurations and operations of the image processing apparatus 108 and the user interface (UI) 121 will be described in detail.
FIG. 2 is a block diagram illustrating a hardware configuration example of the image processing apparatus 108 and the user interface 121. The hardware configuration example shown in FIG. 2 is commonly used in other embodiments as well as the configuration of the ultrasonic imaging apparatus in FIG.

The image processing apparatus 108 includes a CPU (processor) 1, a ROM (nonvolatile memory: a read-only storage medium) 2, a RAM (volatile memory: a storage medium capable of reading and writing data) 3, a storage device 4, and display control. Part 15 is provided. The user interface 121 includes a medium input unit 11, an input control unit 13 and an input device 14, and a display 16. The image generation unit 107, the image processing apparatus 108, and the user interface 121 are connected to each other via the bus 5.

At least one of the ROM 2 and the RAM 3 of the image processing apparatus 108 stores in advance a program and data for the arithmetic processing of the CPU 1 necessary for realizing the operation of the image processing apparatus 108. Various processes of the image processing apparatus 108 are realized by the CPU 1 executing a program stored in advance in at least one of the ROM 2 and the RAM 3. The program executed by the CPU 1 may be stored in a storage medium 12 such as an optical disk, for example, and the medium input unit 11 (for example, an optical disk drive) may read the program and store it in the RAM 3. Alternatively, the program may be stored in the storage device 4 and the program may be loaded from the storage device 4 into the RAM 3. Further, the program may be stored in the ROM 2 in advance.

The storage device 4 may include a nonvolatile semiconductor storage medium such as a flash memory, for example. An external storage device connected via a network or the like may be used.

The input device 14 is a device that receives user operations, and includes, for example, a keyboard, a trackball, an operation panel, a foot switch, and the like. The input control unit 13 receives an operation input input by a user. The operation input received by the input control unit 13 is processed by the CPU 1. For example, the display control unit 15 performs control to display the image data obtained by the processing of the CPU 1 on the display 16. The display 16 displays an image under the control of the display control unit 15.

FIG. 3 is a functional block diagram showing one function of the image processing apparatus 108 of the present embodiment. As illustrated in FIG. 3, the image processing apparatus 108 includes an ultrasonic 3D image acquisition unit 21, an ultrasonic 3D image feature site position estimation / identification unit 22, and an ultrasonic 2D image acquisition unit 24. Further, the image processing apparatus 108 includes ultrasonic feature part information 23 indicating information on the name and position of the feature part, an ultrasonic 2D-3D image registration unit 25, and an image display unit 26.

The operation processing of the image processing apparatus 108 shown in FIG. 3 will be described using the flowchart shown in FIG. First, in step S201, the ultrasonic probe 7 is applied and a display prompting the user to perform scanning while turning is displayed on the display 16. When the user scans the ultrasound probe 7 in the area of the organ according to the display, the transmission unit 102, the reception unit 105, and the image generation unit 107 continuously generate an ultrasonic 2D image. The image generation unit 107 synthesizes an ultrasonic 3D image from continuously generated ultrasonic 2D images. The ultrasonic 3D image acquisition unit 21 receives the synthesized ultrasonic 3D image.

In step S202, the characteristic part position estimation / identification unit 22 of the ultrasonic 3D image estimates the position of a predetermined anatomical characteristic part from the ultrasonic 3D image using a known machine learning technique, and estimates According to the result, the name of each characteristic part is identified. Here, the characteristic site is a medically defined organ or site within the organ such as the portal vein umbilicus of the liver, the inflow of the inferior vena cava, the gallbladder, and the bifurcation points of the liver portal vein or vein. is there.

FIG. 5A, FIG. 5B, and FIG. 5C are explanatory diagrams showing the three-dimensional position of the portal vein umbilical portion of the liver, the inflow portion of the inferior vena cava, and the features of the gallbladder as features in the ultrasound 3D image. is there. The cube 50 shown in FIG. 5A, FIG. 5B, and FIG. 5C shows the surrounding local area | region of the position of each said anatomical feature part. Details of the position estimation and name identification of the characteristic part in the characteristic part position estimation / identification unit 22 of the ultrasonic 3D image will be described later.

FIG. 6 shows an example of the characteristic part name estimated and identified from the ultrasonic 3D image and the three-dimensional position information as the ultrasonic characteristic part information 23. These ultrasonic characteristic part information 23 can be stored as a table in the RAM 3 or the storage device 4.

In step S203, the ultrasonic 2D image acquisition unit 24 receives the 2D ultrasonic image acquired in real time from the image generation unit 107.

In step S204, the ultrasonic 2D-3D image registration unit 25 receives the ultrasonic 3D image and the ultrasonic 2D image from the ultrasonic 3D image acquisition unit 21 and the ultrasonic 2D image acquisition unit 24, respectively, and aligns the two. A registration transformation matrix for performing the above is calculated. Details of the alignment transformation matrix calculation will be described later.

In step S205, the image display unit 26 receives the ultrasonic 3D image, the ultrasonic 2D image, the ultrasonic characteristic part information 23, the alignment conversion matrix, and further the position of the 2D cross-sectional image. The image display unit 26 uses these data to calculate the projection distance from the coordinates of the point on the 3D space of the characteristic part of the subject to the position of the 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image. The calculated projection distance is set as a distance relationship with the ultrasonic 2D image of the characteristic part. Then, the image display unit 26 displays the ultrasonic 2D image on the screen of the display 16 as shown in an example in FIG. Then, the position of the identified feature part,

names

17A and 18A, and the distance relationship between the feature part and the ultrasonic 2D image are displayed on the screen. At that time, the image display unit 26 uses the alignment conversion matrix to project from the position of the characteristic part in the three-dimensional coordinate system of the ultrasonic 3D image to the currently displayed ultrasonic 2D image, and the projected location The markers 17B and 18B indicated by the location of the characteristic part and the X mark are displayed. That is, the image display unit 26 displays in real time on the ultrasonic 2D image on the display 16 the name of the characteristic part estimated from the ultrasonic 3D image and the positional relationship between the characteristic part and the ultrasonic 2D image acquired in real time. Therefore, accurate surgical navigation for the user can be realized.

As described above, the image display unit 26 determines the position of the 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image from the coordinates of the point on the 3D space of the characteristic part of the subject estimated from the ultrasonic 3D image. Is calculated as the distance relationship between the characteristic part and the ultrasonic 2D image acquired in real time. Then, the sizes of the markers 17B and 18B are displayed in proportion to the projection distance that is the distance relationship between the calculated feature portion and the ultrasonic 2D image. That is, in the image display unit 26, when displaying the marker indicating the characteristic part on the ultrasonic 2D image, the size of the markers 17B and 18B is displayed in proportion to the calculated projection distance, whereby the positional relationship between the two is displayed. Can be grasped at a glance, so that the usability of the user is further improved.

The image display unit 26 can turn on and off the display of the

feature part names

17A and 18A and the markers 17B and 18B by user selection of the check box 28, and only when the user needs it.

Characteristic part names

17A and 18A and markers 17B and 18B can be displayed. Note that the touch panel operation button 19 in FIG. 11A and FIG. 11B are described in the second embodiment.

Further, at the time of the display in step S205, the image display unit 26 changes one color of the real-time ultrasonic 2D image and the 2D cross-sectional image at a position corresponding to the ultrasonic 2D image of the ultrasonic 3D image. An image in which these two are superimposed transparently can be generated and displayed on the display 16. Furthermore, the image display unit 26 displays the

feature part names

17A and 18A and the markers 17B and 18B on the 2D image generated by superimposing them. Even in this case, the image display unit 26 can display the size of the markers 17B and 18B in proportion to the calculated projection distance from the characteristic part to the ultrasonic 2D image.

Subsequently, the processing of the characteristic part position estimation / identification unit 22 of the ultrasonic 3D image of the present embodiment will be described in detail using the flowchart shown in FIG. As described above, since the image processing apparatus 108 is realized by the program execution of the CPU 1, each process of FIG. 7 is also realized by the program process of the CPU 1.

First, in step S401, the characteristic part position estimation / identification unit 22 of the ultrasonic 3D image receives the ultrasonic 3D image from the image generation unit 107. In step S402, the characteristic part position estimation / identification unit 22 of the ultrasonic 3D image performs position estimation and name identification of the characteristic part candidate. In order to improve the processing speed, the feature part position estimation / identification unit 22 of the ultrasound 3D image reduces the size of the ultrasound 3D image and searches for feature part candidates with a coarse resolution using machine learning. As a method of position estimation and name identification of a characteristic part, for example, a Hough Forest method that is a known machine learning method can be used.

Next, in step S403, the characteristic part position estimation / identification unit 22 of the ultrasonic 3D image obtains a local 3D image that is a local region around the searched characteristic part candidate from the normal size ultrasonic 3D image. . In step S404, the feature part position estimation / identification unit 22 of the ultrasound 3D image searches and identifies the feature part in detail in the local region surrounding the feature part candidate. Here, the above Hough Forest method can be used. When a more accurate position estimation / identification result is desired, a known deep learning method 3D CNN (convolutional neural network) method can be used.

In step S405, the characteristic part position estimation / identification unit 22 excludes the characteristic part as an erroneous identification part when the characteristic part identification score obtained in the search in step S404 is equal to or less than a predetermined threshold. In step S <b> 406, the characteristic part position estimation / identification unit 22 of the ultrasonic 3D image outputs the position / name information of the identified characteristic part as the ultrasonic characteristic part information 23.

Subsequently, the processing of the ultrasonic 2D-3D image alignment unit 25 of the present embodiment will be described in detail using the flowchart shown in FIG. The processing of the ultrasonic 2D-3D image alignment unit 25 is also realized by executing a program by the CPU 1.

In step S301, the ultrasonic 2D-3D image registration unit 25 receives the ultrasonic 3D image from the ultrasonic 3D image acquisition unit 21 and the ultrasonic 2D image from the ultrasonic 2D image acquisition unit 24, and converts the ultrasonic 3D image. The corresponding three-dimensional position of the ultrasonic 2D image is roughly estimated from the inside. That is, the initial position of the corresponding ultrasonic 2D image is estimated.

FIG. 9 shows 15 examples of three-dimensional initial position candidates used in the ultrasonic 2D-3D image alignment unit 25. Each position 91 is shown in each pattern. When estimating the initial position, the ultrasonic 2D image is input to the 15-pattern discriminator using machine learning, and the position 91 of the pattern that can obtain the highest discrimination score is obtained from the ultrasonic 2D image acquisition unit 24. The initial position of the ultrasonic 2D image. The ultrasonic 2D-3D image registration unit 25 selects a 3D position candidate of a 2D cross-sectional image corresponding to the ultrasonic 2D image from the ultrasonic 3D image and uses it as an initial position. The ultrasonic 2D image and the ultrasonic wave Position alignment with the 3D image is performed, and a 2D cross-sectional image of the ultrasound 3D image corresponding to the ultrasound 2D image is calculated.

識別 Classifiers for each pattern are created by learning. A large number of ultrasonic 3D images are collected as learning data, 2D cross-sectional images at each position 91 in FIG. 9 are extracted from each ultrasonic 3D image, and created as learning data of a pattern corresponding to the position. Further, in order to increase the quantity and diversity of learning data, a 2D cross-sectional image is extracted from each position 91 in FIG. 9 by performing a random and small range of translation and rotation angle, and used as learning data at that position. . Thereby, for example, learning data of the three-dimensional initial positions of the 15 patterns in FIG. 9 can be created, and a discriminator for each pattern can be created by using machine learning. As a machine learning method, for example, a well-known Adaboost method or a deep learning method can be used.

In step S302, the ultrasonic 2D-3D image registration unit 25 estimates the translation and rotation angle, which is geometric conversion information for ultrasonic 2D-3D image alignment, from the estimated initial position of the ultrasonic 2D image. To do. In step S303, the ultrasonic 2D-3D image alignment unit 25 constructs a 2D cross-sectional image corresponding to the ultrasonic 2D image from the ultrasonic 3D image using the obtained parallel movement and rotation angle.

In step S304, the ultrasonic 2D-3D image alignment unit 25 performs an image similarity evaluation function between the 2D cross-sectional image obtained from the ultrasonic 3D image and the ultrasonic 2D image acquired by the ultrasonic 2D image acquisition unit 24. Perform the operation. A known mutual information amount can be used as the image similarity.

In step S305, the ultrasonic 2D-3D image registration unit 25 performs translation and rotation so that the image similarity between the 2D cross-sectional image obtained from the ultrasonic 3D image and the ultrasonic 2D image is maximized or maximized. Convergence calculation is performed to obtain the angle.

In step S306, if the image similarity is not converged, the translation and the rotation angle are updated in order to obtain a higher similarity. Then, Steps S303 to S305 are performed again using the updated parallel movement and rotation angle.

On the other hand, if the similarity is converged in step S305, the ultrasonic 2D-3D image registration unit 25 outputs the translation and rotation angle information obtained in step S307, the position of the 2D cross-sectional image, and the like. Thus, the processing of the ultrasonic 2D-3D image alignment unit 25 in FIG. 3 can be completed.

In addition, with respect to a plurality of ultrasonic 2D images acquired continuously in real time, the initial position of the alignment of the subsequent ultrasonic 2D image after completion of the alignment of the first ultrasonic 2D image is the previous time. The translation and rotation angle information, which is the alignment result of the acquired ultrasonic 2D image, can be used. That is, if the processes in steps S303 to S307 are performed using the parallel movement and rotation angle of the previous ultrasonic 2D image, real-time ultrasonic 2D-3D image alignment is possible.

As described above, according to the ultrasonic imaging apparatus of the present embodiment, the relationship between the name and position of the characteristic part in the patient and the projection distance from the characteristic part to the ultrasonic 2D image is represented in the ultrasonic 2D image. Real-time display and automatic and accurate surgical navigation can be realized.

In the configuration of the first embodiment, the image processing apparatus 108 is provided inside the ultrasonic imaging apparatus 100. However, the image processing apparatus 108 illustrated in FIGS. 1 and 2 is different from the ultrasonic imaging apparatus 100. It is also possible to use the device. In that case, the image processing apparatus 108 and the ultrasonic imaging apparatus 100 are connected via a signal line or a network. For example, the image processing apparatus 108 is mounted on a general computer or a processing apparatus such as a workstation, and is connected to the ultrasonic imaging apparatus 100 via a network.

In this case, the image processing apparatus 108 receives the ultrasonic 3D image for identifying the characteristic part and the ultrasonic 2D image for alignment from the ultrasonic imaging apparatus that is the client terminal via the network, respectively. The position estimation / identification process and the image alignment process of FIG. 8 are performed. Then, the name and position information of the identified characteristic part and the ultrasonic 2D-3D image alignment result are transmitted to the client terminal. As a result, it is not necessary to mount the image processing apparatus 108 that requires a relatively large calculation amount on the ultrasonic imaging apparatus 100 that is a client terminal. Therefore, the ultrasonic imaging apparatus 100 can perform alignment processing by using the calculation capability of the image processing apparatus 108 connected via a network. Therefore, the ultrasonic imaging apparatus 100 is a small and simple ultrasonic imaging apparatus. It is possible to provide a device capable of displaying the name and distance relationship of a characteristic part on an ultrasonic 2D image in real time.

As described above, according to the present embodiment, position estimation and name identification are performed on a predetermined anatomical feature part from an ultrasonic 3D image obtained by imaging a subject, and the part and name are being operated on. It is possible to display an ultrasonic 2D image including a distance relationship with the ultrasonic 2D image captured in real time to realize automatic and accurate surgical navigation.

In the first embodiment, position estimation and name identification are performed on a predetermined anatomical feature portion from an ultrasound 3D image, and the ultrasound 2D image and the ultrasound 3D image captured in real time during surgery are aligned. The distance relationship between the position / name of the identified characteristic part and the ultrasonic 2D image is displayed on the ultrasonic 2D image. In the second embodiment, in addition to the configuration of the first embodiment, the addition and correction of feature parts or the correction of the geometric transformation calculation of the alignment can be performed based on a user instruction. This is an example. That is, an ultrasonic imaging apparatus, the image processing apparatus stitches together an ultrasonic 3D image and an ultrasonic 3D image obtained by imaging with an ultrasonic probe other than the ultrasonic 3D image. A second ultrasonic 3D image is generated, the ultrasonic 2D image and the second ultrasonic 3D image are aligned, and the position of the 2D cross-sectional image of the second ultrasonic 3D image corresponding to the ultrasonic 2D image is calculated. FIG. 4 is an example of an ultrasonic imaging apparatus that displays the position, name, and distance relationship of a characteristic part on an ultrasonic 2D image using the position of the characteristic part and the position of a 2D cross-sectional image of the second ultrasonic 3D image. .

In addition, the image processing apparatus includes a correction unit that joins an ultrasonic 3D image and an ultrasonic 3D image other than the ultrasonic 3D image to generate a second ultrasonic 3D image, and estimates a characteristic part position. The identification unit estimates and identifies the characteristic part of the subject from the second ultrasonic 3D image, and the image alignment unit performs alignment between the ultrasonic 2D image and the second ultrasonic 3D image, and the ultrasonic 2D It is an Example of the image processing apparatus which calculates the position of 2D cross-sectional image of the 2nd ultrasonic 3D image corresponding to an image, and its image processing method. In the description of the second embodiment, the same components and processes as those of the first embodiment are denoted by the same reference numerals and the description thereof is omitted.

<Configuration and operation>
FIG. 10 is a functional block diagram illustrating functions of the image processing apparatus 108 according to the second embodiment, and FIG. 11 is a diagram illustrating an example of a display screen and a button selection unit according to the second embodiment. As illustrated in FIG. 10, the image processing apparatus 108 according to the present exemplary embodiment includes an ultrasonic 3D image acquisition unit 21, an ultrasonic 3D image characteristic part position estimation / identification unit 22, and an ultrasonic 2D image acquisition unit 24. Including. In addition to the ultrasonic feature part information 23 indicating the name / position information of the feature part, the ultrasonic 2D-3D image registration unit 25, and the image display unit 26, the image processing apparatus 108 further includes feature part identification and And an alignment result correction unit 27. Also in the present embodiment, the image display unit 26 displays the ultrasonic 2D image on the screen of the display 16 as shown in FIG. 11 (a) as an example. Based on the distance relationship with the 2D image, etc., the positions of the identified feature parts,

names

17A and 18A, and the markers 17B and 18B indicating the distance relation between the feature part and the ultrasonic 2D image are displayed on the screen.

In the present embodiment, as shown in FIG. 11A, for example, the position and name of the identified feature part and the distance relationship between the feature part and the ultrasonic 2D image are displayed on the display 16. The feature part identification and alignment result correcting unit 27 displays the touch panel operation button 19 as a display for inquiring whether the user determines that the feature part identification and alignment is successful. That is, touch panel operation buttons 19 such as volume addition, characteristic part manual correction, alignment initial position correction, and alignment detail correction are displayed on the display 16, and the user's judgment is made via the button selection means of the input device 14 such as a mouse. Accept. When the user inputs through the input device 14 that the feature part identification and the alignment are successful, the alignment process is completed. As the button selection means, instead of the input device 14 and the touch panel operation button 19, a foot switch 20A connected by a trackball or a USB cable 20B for use during an operation as shown in FIG. Can be used.

On the other hand, when the user determines that the feature part identification and the alignment are unsuccessful and inputs them, the feature part identification and alignment result correction unit 27 of this embodiment performs the correction process of the feature part identification and the alignment. Execute.

The feature part identification and alignment result correction unit 27 displays a display asking whether or not the user determines to additionally acquire an ultrasonic volume on the display 16, and the user is input via the input device 14 or the touch panel operation button 19. Accept the judgment. When the user determines that the feature part information is insufficient and additionally acquires one or more ultrasound 3D images from the ultrasound probe in addition to the ultrasound 3D image described above, the input is performed via the input device 14 or the touch panel operation button 19. In such a case, the ultrasonic imaging apparatus 100 additionally acquires an ultrasonic 3D image. Then, the feature part identification and alignment result correcting unit 27 performs a stitching, ie, stitching, with the additionally acquired ultrasonic 3D image and the original ultrasonic 3D image, that is, one synthesized ultrasonic 3D image, that is, A second ultrasonic 3D image is generated. As a stitching method, for example, the method described in Non-Patent Document 1 can be used.

The feature part identification and alignment result correcting unit 27 uses the second ultrasonic 3D image generated by the stitching process to identify the characteristic part in FIG. 7 and the ultrasonic 2D-3D in FIG. Perform image alignment processing. Since the characteristic part identification process and the alignment process are the same as those in the first embodiment, description thereof will be omitted.

On the other hand, when the user determines and inputs that correction is performed manually without additionally acquiring an ultrasonic 3D image, the ultrasonic characteristic part information 23 shown as an example in FIG. 6, that is, the position of the characteristic part of the ultrasonic 3D image The user manually corrects the names using the input device 14 or the like. The feature part identification and alignment result correction unit 27 receives the position / name information of the corrected 3D image feature part, and outputs it to the image display unit 26. The image display unit 26 executes the image display process in step 205 of FIG.

In addition, when the user determines and inputs the initial alignment correction via the input device 14 or the touch panel operation button 19, the feature part identification and alignment result correction unit 27 is displayed on the display 16 as shown in FIG. The alignment initial position pattern is displayed and the user's selection is accepted. The user can also manually fine-correct the selected initial alignment position pattern. Then, the user acquires an ultrasonic 2D image from the corrected alignment initial position. The feature part identification and alignment result correction unit 27 receives the corrected alignment initial position pattern and the ultrasonic 2D image, and executes the processes of steps S302 to S307 in FIG.

Furthermore, when the user determines / inputs the detailed alignment correction via the input device 14 or the touch panel operation button 19, the feature region identification and alignment result correction unit 27 receives from the ultrasonic 2D image acquisition unit 24. The ultrasonic 2D image and the 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image are superimposed on the display 16 and displayed. The user corrects the alignment with the ultrasonic 2D image while manually adjusting the position and rotation angle of the 2D cross-sectional image. The feature part identification and alignment result correction unit 27 receives the corrected alignment result and outputs it to the image display unit 26.

Then, the image display unit 26 executes the image display process in step 205 of FIG. 4 described above. Also in the present embodiment, the image display unit 26 can display the

names

17A and 18A of the characteristic parts and the markers 17B and 18B when the user needs, by user selection of the check box 28.

As described above, according to the present embodiment, it is possible to configure an ultrasonic imaging apparatus capable of executing addition or correction of feature parts and recalculation of coordinate conversion information for alignment based on a user instruction. .

In the third embodiment, the ultrasonic probe includes a position sensor, and the image generation unit generates an ultrasonic 3D image from the ultrasonic 2D image and the position information of the ultrasonic probe obtained from the position sensor. It is an Example of an ultrasonic imaging device. In the first embodiment, position estimation and name identification are performed on a predetermined anatomical feature part from an ultrasonic 3D image generated by transmitting and receiving ultrasonic waves a plurality of times, and an ultrasonic 2D image captured in real time during surgery Alignment with the ultrasonic 3D image was performed, and the distance relationship between the position / name of the identified characteristic part and the ultrasonic 2D image was displayed on the ultrasonic 2D image. In the third embodiment, a position sensor is attached to an ultrasonic probe, an ultrasonic image is generated from a reception signal of the ultrasonic probe, and position information of the ultrasonic probe obtained from the ultrasonic image and the position sensor is used. From the ultrasonic 3D image, position estimation and name identification are performed on a predetermined anatomical feature part.

Furthermore, in the third embodiment, when an ultrasonic 2D image is captured in real time during surgery, the positional information from the position sensor of the ultrasonic probe is used to align the ultrasonic 2D image and the ultrasonic 3D image. Then, the position, name, and distance relationship between the identified characteristic part and the ultrasonic 2D image are displayed on the ultrasonic 2D image. In the description of the present embodiment, the same components and processes as those of the first embodiment are denoted by the same reference numerals and description thereof is omitted.

<Configuration and operation>
FIG. 12 illustrates a configuration example of the ultrasonic imaging apparatus according to the third embodiment. FIG. 13 is a block diagram illustrating a hardware configuration example of the image processing apparatus 108 and the user interface 121 according to the third embodiment. 12 and 13, a position detection unit 6 and a position sensor 8 are further added to the configuration of the first embodiment. The position detection unit 6 detects the position of the ultrasonic probe 7 from the output of the position sensor 8. For example, a magnetic sensor unit can be used as the position detection unit 6. The position detection unit 6 can detect the coordinates from the position serving as the reference point, that is, the position information of the ultrasonic probe, by forming a magnetic field space and the magnetic sensor as the position sensor 8 detecting the magnetic field. .

As in the first embodiment, FIG. 3 is a functional block diagram illustrating a functional example of the image processing apparatus 108 according to the third embodiment. Further, the operation processing of the image processing apparatus 108 in the third embodiment shown in FIG. 3 is as shown in the flowchart of FIG.

The image generation unit 107 according to the third embodiment generates an ultrasonic image from the reception signal of the ultrasonic probe 7, and generates ultrasonic 3D from the ultrasonic image and the position information of the ultrasonic probe obtained from the position sensor 8. Generate an image. In step S204 of FIG. 4 in the third embodiment, the ultrasonic 2D-3D image alignment unit 25 detects the position information of the ultrasonic probe added to the ultrasonic 3D image and the ultrasonic 2D imaged in real time during the operation. An alignment transformation matrix for performing alignment is calculated using the position information of the ultrasound probe added to the image. Other processes in the third embodiment are the same as those in the first embodiment.

As described above, in the third embodiment, the position sensor is attached to the ultrasonic probe, the ultrasonic image is generated from the reception signal of the ultrasonic probe, and the ultrasonic probe obtained from the ultrasonic image and the position sensor is used. An ultrasonic 3D image is generated from the position information of the tentacles, and position estimation and name identification are performed on a predetermined anatomical feature portion from the ultrasonic 3D image. Further, when an ultrasonic 2D image is captured in real time during surgery, the position information of the identified characteristic part is registered by aligning the ultrasonic 2D image and the ultrasonic 3D image using the position information of the ultrasonic probe. Thus, an ultrasonic imaging apparatus capable of displaying the name and information on the distance relationship with the ultrasonic 2D image on the ultrasonic 2D image can be configured.

In the present invention described in detail above, the position of a predetermined anatomical characteristic part is estimated and the name is identified from an ultrasound 3D image, and an ultrasound scan that is an ultrasound 3D image and an ultrasound 2D image captured in real time. Alignment with the two-dimensional image of the surface, projecting the position of the feature part on the ultrasound 2D image, calculating the distance relationship between the feature part and the ultrasound 2D image, and positioning the feature part on the ultrasound 2D image The ultrasonic imaging apparatus which displays the name and the information on the distance relationship with the ultrasonic 2D image can be provided.

In addition, this invention is not limited to the above-mentioned Example, Various modifications are included. For example, the above-described embodiments have been described in detail for better understanding of the present invention, and are not necessarily limited to those having all the configurations described. As described above, the present invention is not limited to the ultrasonic imaging apparatus, but can be realized as an image processing apparatus connected to the ultrasonic imaging apparatus via a network and an image processing method thereof. . Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Further, the above-described configuration, function, image processing apparatus, and the like have been described as an example of creating a program that realizes part or all of them. It may be realized by wear.

1 CPU
2 ROM
3 RAM
4 Storage Device 5 Bus 6 Position Detection Unit 7 Ultrasound Probe 8 Position Sensor 10 Image Imaging Device 11 Medium Input Unit 12 Storage Medium 13 Input Control Unit 14 Input Device 15 Display Control Unit 16

Display

17A, 18A Name 17B, 18B Marker 19 Touch Panel Operation Button

20A Foot Switch

20B USB Cable 21 Ultrasonic 3D Image Acquisition Unit 22 Ultrasonic 3D Image Feature Part Location Estimation / Identification Unit 23 Ultrasonic Feature Part Information 24 Ultrasonic 2D Image Acquisition Unit 25 Ultrasonic 2D-3D Image Alignment unit 26 Image display unit 27 Feature part identification and alignment result correction unit 28 Check box 50 Cube 91 Position 100 Ultrasound imaging device 101 Transmission / reception switching unit 102 Transmission unit 105 Reception unit 106 Control unit 107 Image generation unit 108 Image processing Device 120 User 121 User In Face (UI)

Claims

An ultrasonic imaging device comprising:
An ultrasonic probe that transmits ultrasonic waves to the subject and receives ultrasonic waves from the subject; and
An image generation unit that generates an ultrasonic two-dimensional (2D) image from a reception signal of the ultrasonic probe, and generates an ultrasonic three-dimensional (3D) image by transmitting and receiving ultrasonic waves a plurality of times;
The ultrasonic 2D image, and an image processing device that receives and processes the ultrasonic 3D image,
The image processing apparatus includes:
Estimating and identifying the characteristic part of the subject from the ultrasound 3D image;
Align the ultrasonic 2D image and the ultrasonic 3D image,
Displaying the information of the characteristic part on the ultrasonic 2D image;
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 1,
The information on the characteristic part is a position relation, a name of the characteristic part, and a distance relationship with the ultrasonic 2D image.
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 2,
The image processing apparatus includes:
A feature part position estimation / identification unit that estimates and identifies the position and name of the feature part from the ultrasonic 3D image,
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 3,
The image processing apparatus includes:
An image alignment unit for aligning the ultrasonic 2D image and the ultrasonic 3D image;
The image alignment unit calculates a position of a 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image;
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 4,
An image display unit for displaying the ultrasonic 2D image;
The image display unit
Using the position of the characteristic part and the position of the 2D cross-sectional image, the distance relationship between the ultrasonic 2D image is calculated, and the distance relation between the position of the characteristic part, the name, and the ultrasonic 2D image Display on ultrasound 2D image,
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 5,
The image display unit
Displaying an image in which the ultrasonic 2D image and the 2D cross-sectional image are superimposed;
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 4,
The image alignment unit
A candidate of a 3D position of a 2D cross-sectional image corresponding to the ultrasound 2D image is selected from the ultrasound 3D image and used as an initial position, and the ultrasound 2D image and the ultrasound 3D image are aligned, Calculating a 2D cross-sectional image of the ultrasound 3D image corresponding to the ultrasound 2D image;
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 4,
The image processing apparatus includes:
Connecting the ultrasonic 3D image and the ultrasonic 3D image obtained by imaging with the ultrasonic probe other than the ultrasonic 3D image to generate a second ultrasonic 3D image;
Aligning the ultrasonic 2D image and the second ultrasonic 3D image, calculating a position of a 2D cross-sectional image of the second ultrasonic 3D image corresponding to the ultrasonic 2D image;
Using the position of the characteristic part and the position of the 2D cross-sectional image of the second ultrasonic 3D image, the position, name, and distance relationship of the characteristic part are displayed on the ultrasonic 2D image.
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 4,
The image processing apparatus includes:
A correction unit for correcting the position and name of the characteristic part estimated and identified by the characteristic part position estimation / identification unit;
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 9,
The correction unit corrects the position of the 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image calculated by the image alignment unit;
An ultrasonic imaging apparatus.
The ultrasonic imaging apparatus according to claim 1,
The ultrasonic probe includes a position sensor,
The image generation unit generates the ultrasonic 3D image from the ultrasonic 2D image and position information of the ultrasonic probe obtained from the position sensor.
An ultrasonic imaging apparatus.
An image processing apparatus,
A feature part position estimation / identification unit that estimates and identifies a feature part of the subject from an ultrasound 3D image of the subject;
An image alignment unit that performs alignment between the ultrasonic 2D image of the subject and the ultrasonic 3D image, and calculates a position of a 2D cross-sectional image of the ultrasonic 3D image corresponding to the ultrasonic 2D image; Comprising
An image processing apparatus.
The image processing apparatus according to claim 12,
A correction unit for connecting the ultrasonic 3D image and an ultrasonic 3D image other than the ultrasonic 3D image to generate a second ultrasonic 3D image;
The characteristic part position estimation / identification unit estimates and identifies the characteristic part of the subject from the second ultrasonic 3D image,
The image alignment unit aligns the ultrasonic 2D image and the second ultrasonic 3D image, and calculates a position of a 2D cross-sectional image of the second ultrasonic 3D image corresponding to the ultrasonic 2D image. To
An image processing apparatus.
An image processing method in an image processing apparatus,
The image processing apparatus includes:
Estimating and identifying a characteristic part of the subject from an ultrasound 3D image of the subject,
Aligning the ultrasound 2D image of the subject and the ultrasound 3D image, and calculating the position of the 2D cross-sectional image of the ultrasound 3D image corresponding to the ultrasound 2D image;
An image processing method.
The image processing method according to claim 14, comprising:
The image processing apparatus includes:
Connecting the ultrasonic 3D image and an ultrasonic 3D image other than the ultrasonic 3D image to generate a second ultrasonic 3D image;
Estimating and identifying the characteristic part of the subject from the second ultrasonic 3D image;
The ultrasonic 2D image and the second ultrasonic 3D image are aligned, and the position of the 2D cross-sectional image of the second ultrasonic 3D image corresponding to the ultrasonic 2D image is calculated.
An image processing method.