WO2023176741A1

WO2023176741A1 - Image processing device, image processing system, image display method, and image processing program

Info

Publication number: WO2023176741A1
Application number: PCT/JP2023/009449
Authority: WO
Inventors: 泰一坂本; 俊祐吉澤; クレモンジャケ; ステフェンチェン; トマエン; 亮介佐賀
Original assignee: テルモ株式会社
Priority date: 2022-03-16
Filing date: 2023-03-10
Publication date: 2023-09-21
Also published as: JP2023136332A

Abstract

This image processing device, which, on the basis of the positional relationship between a viewing point set in a virtual three-dimensional space and a biological tissue object arranged in the three-dimensional space, carries out rendering of the object and displays the object on a screen as a three-dimensional image of the biological tissue, is provided with a control unit which: in response to a position specification operation for specifying two positions on the screen, specifies a first corresponding point which corresponds to one of the two specified positions on a plane corresponding to the screen in the three-dimensional space and a second corresponding point which corresponds to the other; calculates the distance between a first intersection, that is, an intersection where an extension of a straight line connecting the viewing point and the first corresponding point in the three-dimensional space intersects with the object, and a second intersection, that is, an intersection where an extension of a straight line connecting the viewing point and the second corresponding point in the three-dimensional space intersects with the object; and outputs the obtained calculation result.

Description

Image processing device, image processing system, image display method, and image processing program

The present disclosure relates to an image processing device, an image processing system, an image display method, and an image processing program.

Patent Documents 1 to 3 describe techniques for generating three-dimensional images of heart chambers or blood vessels using a US imaging system. "US" is an abbreviation for ultrasound.

Patent Document 4 discloses that when two points on a screen on which a three-dimensional image is displayed are specified, the two-dimensional coordinates and density of the two points are determined, and the density difference is added to the distance between the two-dimensional coordinates. A method for calculating the distance between the two points is disclosed.

US Patent Application Publication No. 2010/0215238 US Patent No. 6,385,332 US Patent No. 6,251,072 Japanese Unexamined Patent Publication No. 63-063433

Treatment using IVUS is widely performed for the intracardiac, cardiovascular, and lower limb arterial regions. "IVUS" is an abbreviation for intravascular ultrasound. IVUS is a device or method that provides two-dimensional images in a plane perpendicular to the longitudinal axis of a catheter.

IVUS is often used for procedures that use a catheter separate from the IVUS catheter, such as ablation. For example, in the transatrial septal approach for atrial fibrillation ablation, a septal puncture needle inserted into the right atrium punctures the fossa ovalis, creating a path from the right atrium to the left atrium. law is used. During puncture, it is desirable to carefully confirm the puncture position because there is a risk of complications such as perforation or cardiac tamponade. In this respect, an IVUS catheter that can obtain 360-degree information is excellent in confirming the puncture position within the same plane. However, when using IVUS, images are acquired intermittently along the IVUS catheter axis, making it difficult to image the three-dimensional structure. As a result, confirmation of the puncture position in the axial direction may become insufficient. Therefore, it is conceivable to automatically generate a three-dimensional image representing the structure of the living tissue from the two-dimensional image of IVUS and display the generated three-dimensional image to the surgeon. If the generated 3D image is displayed as is, the surgeon can only see the outer wall of the tissue, so it is possible to cut out part of the structure of the living tissue in the 3D image so that the inner cavity can be seen. . When two points on the screen are specified, the distance between the two points is automatically calculated and the calculated distance is sent to the surgeon so that any distance in the three-dimensional image, such as the length of the fossa ovalis, can be measured. It is also conceivable to display it directed towards the user.

There is a need to be able to arbitrarily change the way a three-dimensional image is displayed by changing the viewpoint or the position of the light source with respect to the three-dimensional object. However, depending on the display method, the displayed density does not necessarily correspond to the depth distance, so as disclosed in Patent Document 4, the distance between two specified points is calculated by taking into account the difference in density. This method cannot be applied. If such a method were to be used, the accuracy of distance calculation would be insufficient.

An object of the present disclosure is to improve the accuracy of distance calculation between two points specified on a screen on which a three-dimensional image is displayed.

An image processing device according to an aspect of the present disclosure renders a living tissue object based on a positional relationship between a viewpoint set in a virtual three-dimensional space and an object of biological tissue arranged in the three-dimensional space. an image processing device that displays the object on a screen as a three-dimensional image of the living tissue, the image processing device displaying the object on the screen as a three-dimensional image of the living tissue, the image processing device A first corresponding point corresponding to one of the two specified positions and a second corresponding point corresponding to the other on the plane corresponding to the above are identified, and the viewpoint and the first corresponding point are identified in the three-dimensional space. a first intersection point that is an intersection between the object and an extension of a straight line connecting the viewpoint, and a second intersection point that is an intersection between the object and an extension of the straight line that connects the viewpoint and the second corresponding point in the three-dimensional space. A control unit is provided that calculates the distance between the intersection points and outputs the obtained calculation result.

In one embodiment, the control unit outputs a numerical value representing the distance on the screen as the calculation result.

In one embodiment, the position specifying operation includes, as a first operation, an operation of pressing a push button of an input device, and the control unit is configured to control the location on the screen when the first operation is performed on the plane. A point corresponding to the position of the pointer is specified as the first corresponding point.

In one embodiment, the first operation is an operation of pressing the push button while pressing a predetermined first key.

In one embodiment, the position specifying operation includes, as a second operation, an operation of releasing the push button, which is performed following the first operation and a drag operation of moving the pointer while holding the push button; The control unit specifies, as the second corresponding point, a point on the plane that corresponds to the position of the pointer when the second operation is performed.

In one embodiment, the second operation is an operation of releasing the push button while pressing a predetermined second key.

In one embodiment, the control unit may control a first corresponding position on the screen corresponding to an intersection between the plane and a straight line connecting the viewpoint and the first intersection in the three-dimensional space; A mark is displayed at each second corresponding position corresponding to an intersection between the plane and a straight line connecting the viewpoint and the second intersection in space.

In one embodiment, the control unit specifies a corresponding range on the plane that corresponds to the specified range in response to a range specification operation that specifies a range on the screen, and specifies the corresponding range on the plane that corresponds to the specified range, and A mark displayed at a position corresponding to an intersection point between the first intersection point and the second intersection point that exists in a three-dimensional area extending from the viewpoint in a conical shape through the outer edge of the corresponding range in the three-dimensional space. change the appearance of

In one embodiment, the control unit receives an operation to delete marks whose appearance has been changed all at once.

An image processing system as one aspect of the present disclosure includes the image processing device and a display that displays the screen.

An image display method as an aspect of the present disclosure renders an object based on a positional relationship between a viewpoint set in a virtual three-dimensional space and an object of biological tissue arranged in the three-dimensional space. an image display method for displaying the object on a screen as a three-dimensional image of the living tissue, the method comprising: A first corresponding point corresponding to one of the two specified positions and a second corresponding point corresponding to the other on the plane corresponding to the above are identified, and the viewpoint and the first corresponding point are identified in the three-dimensional space. a first intersection point that is an intersection between the object and an extension of a straight line connecting the viewpoint, and a second intersection point that is an intersection between the object and an extension of the straight line that connects the viewpoint and the second corresponding point in the three-dimensional space. The distance between the intersection points is calculated and the calculated result is output.

An image processing program as an aspect of the present disclosure renders an object based on a positional relationship between a viewpoint set in a virtual three-dimensional space and an object of biological tissue arranged in the three-dimensional space. and displays the object on the screen as a three-dimensional image of the living tissue, in response to a position designation operation that specifies two positions on the screen, a plane corresponding to the screen in the three-dimensional space. The above process of identifying a first corresponding point corresponding to one of the two specified positions and a second corresponding point corresponding to the other, and connecting the viewpoint and the first corresponding point in the three-dimensional space. a first point of intersection that is an intersection between an extension of a straight line and the object, and a second point of intersection that is an intersection between an extension of a straight line connecting the viewpoint and the second corresponding point in the three-dimensional space and the object; A process of calculating the distance between the two and a process of outputting the obtained calculation result are executed.

According to the present disclosure, the accuracy of distance calculation between two points specified on a screen on which a three-dimensional image is displayed is improved.

FIG. 1 is a perspective view of an image processing system according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of distance calculation performed by the image processing system according to the embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of region calculation performed by the image processing system according to the embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a screen displayed on a display by an image processing system according to an embodiment of the present disclosure. 1 is a diagram showing an example of a two-dimensional image displayed on a display by an image processing system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a cutting area formed by an image processing system according to an embodiment of the present disclosure. FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present disclosure. FIG. 2 is a perspective view of a probe and a drive unit according to an embodiment of the present disclosure. 1 is a flowchart showing the operation of an image processing system according to an embodiment of the present disclosure. 1 is a flowchart showing the operation of an image processing system according to an embodiment of the present disclosure. 1 is a flowchart showing the operation of an image processing system according to an embodiment of the present disclosure. 1 is a flowchart showing the operation of an image processing system according to an embodiment of the present disclosure.

Hereinafter, one embodiment of the present disclosure will be described with reference to the drawings.

In each figure, the same or corresponding parts are given the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

An overview of this embodiment will be described with reference to FIGS. 1 to 15.

The image processing device 11 according to the present embodiment is a computer that displays three-dimensional data 52 representing a biological tissue 60 as a three-dimensional image 53 on a display 16. As shown in FIGS. 2 to 5, the image processing device 11 performs image processing based on the positional relationship between the viewpoint V0 set in the virtual three-dimensional space and the object 54 of the biological tissue 60 arranged in the three-dimensional space. , renders the object 54, and displays the object 54 on the screen 80 as a three-dimensional image 53 of the biological tissue 60. Viewpoint V0 corresponds to the position of virtual camera 71 arranged in three-dimensional space.

As shown in FIGS. 2 to 5, the image processing device 11 specifies a first corresponding point Q1 and a second corresponding point Q2 on the plane 55 corresponding to the screen 80 in the three-dimensional space in response to the position specifying operation. do. The position designation operation is an operation to designate two positions on the screen 80. The first corresponding point Q1 is a point on the plane 55 that corresponds to one of the two positions designated by the position designation operation. The second corresponding point Q2 is a point on the plane 55 that corresponds to the other of the two positions designated by the position designation operation. The image processing device 11 calculates the distance ||R2−R1|| between the first intersection R1 and the second intersection R2 in the three-dimensional space. The first intersection R1 is the intersection of the object 54 and an extension of the straight line connecting the viewpoint V0 and the first corresponding point Q1 in the three-dimensional space. The second intersection R2 is the intersection of the object 54 and an extension of the straight line connecting the viewpoint V0 and the second corresponding point Q2 in the three-dimensional space. The image processing device 11 outputs the obtained calculation results. Specifically, the image processing device 11 outputs a numerical value representing the distance on the screen 80 as the calculation result. Alternatively, the image processing device 11 may output the calculation result in another format such as audio. FIG. 4 shows an example in which the numerical value "10 mm" is displayed on the screen 80 as the calculation result.

According to this embodiment, the accuracy of distance calculation between two points specified on the screen 80 on which the three-dimensional image 53 is displayed is improved. For example, even if the position of the viewpoint V0 with respect to the object 54 is changed and the way the three-dimensional image 53 is displayed changes, the distance between two specified points will change depending on the position of the viewpoint V0, not the display density. Therefore, accurate distance calculation is possible. Moreover, as shown in FIG. 5, the distance between two designated points is not the distance between the coordinates on the plane 55 corresponding to the screen 80 ||Q2-Q1||, but from the viewpoint V0 on the object 54. Since the distance between the coordinates is calculated according to ||R2−R1||, which is reached by a straight line extending through the coordinates, accurate distance calculation is possible.

The position designation operation includes an operation of pressing a push button on the input device as a first operation. In this embodiment, the first operation is an operation of pressing a button on the mouse 15 as a push button of the input device, but it may also be an operation of pressing a specific key of the keyboard 14 as a push button of the input device. The image processing device 11 identifies a point on the plane 55 that corresponds to the position of the pointer 86 on the screen 80 when the first operation was performed as the first corresponding point Q1. FIG. 2 shows an example in which the pointer 86 is located at a position corresponding to the upper end of the fossa ovalis 66 represented in the three-dimensional image 53 when the first operation is performed. Although the pointer 86 has an arrow shape in this example, it may have another shape such as a cross shape. For example, when the user switches the operation mode from another mode to the position specification operation mode, the shape of the pointer 86 may be changed so that the user can easily recognize the operation mode. Then, when the user switches the operation mode from the position specifying operation mode to another mode, the shape of the pointer 86 may be changed again.

In this embodiment, the image processing device 11 displays the mark 87 at the first corresponding position on the screen 80. The first corresponding position is a position on the screen 80 that corresponds to the intersection of the plane 55 and a straight line connecting the viewpoint V0 and the first intersection R1 in the three-dimensional space. After the first operation is performed, the first corresponding position remains the same as the position specified in the first operation until the viewpoint V0 is moved. However, when the viewpoint V0 is moved, the first corresponding position also changes to the first corresponding position. Moves from a specified position with one operation. For example, by storing the information of the mark 87 in association with the voxel corresponding to the first intersection point R1, even if the position of the viewpoint V0 changes, the mark 87 can be displayed for the same voxel based on the stored information. I can do it. FIG. 3 shows an example in which the mark 87 is displayed at a position corresponding to the upper end of the fossa ovalis 66 as the first corresponding position.

From the viewpoint of preventing erroneous operations, the first operation may be an operation of pressing a push button while pressing a predetermined first key. The first key is, for example, the Ctrl key or the Shift key on the keyboard 14.

The position specifying operation includes, as a second operation, an operation of releasing the push button, which is performed following the first operation and a drag operation of moving the pointer 86 while holding down the push button. The image processing device 11 identifies a point on the plane 55 that corresponds to the position of the pointer 86 when the second operation is performed as a second corresponding point Q2. FIG. 3 shows an example in which the pointer 86 is at a position corresponding to the lower end of the fossa ovalis 66 when the second operation is performed.

In this embodiment, the image processing device 11 displays the mark 88 at the second corresponding position on the screen 80. The second corresponding position is a position on the screen 80 that corresponds to the intersection of the plane 55 and a straight line connecting the viewpoint V0 and the second intersection R2 in the three-dimensional space. After the second operation is performed, the second corresponding position remains the same as the position specified by the second operation until the viewpoint V0 is moved. However, when the viewpoint V0 is moved, the second corresponding position also changes to the second corresponding position. Move from the specified position with 2 operations. For example, by storing the information of the mark 88 in association with the voxel corresponding to the second intersection point R2, even if the position of the viewpoint V0 changes, the mark 88 can be displayed for the same voxel based on the stored information. I can do it. FIG. 4 shows an example in which the mark 88 is displayed at a position corresponding to the lower end of the fossa ovalis 66 as the second corresponding position.

From the viewpoint of preventing erroneous operations, the second operation may be an operation of releasing a push button while pressing a predetermined second key. The second key is, for example, the Ctrl key or the Shift key on the keyboard 14. The second key may be the same key as the first key, or may be a different key from the first key.

According to the examples in FIGS. 2 to 4, the user moves the pointer 86 to a desired position using the mouse 15, presses a button on the mouse 15 to specify the starting point position, and moves the pointer 86 to another position while holding down the button on the mouse 15. The end point position can be specified by moving the mouse 15 to a desired position and releasing the button on the mouse 15. As shown in FIG. 5, the image processing device 11 specifies the three-dimensional coordinates (xq1, yq1, dq) corresponding to the specified starting point position as the coordinates of the first corresponding point Q1. The image processing device 11 specifies the three-dimensional coordinates (xq2, yq2, dq) corresponding to the designated end point position as the coordinates of the second corresponding point Q2. The image processing device 11 calculates three-dimensional coordinates (xr1, yr1) where a straight line passing through the coordinates (xv, yv, dv) of the viewpoint V0 and the coordinates (xq1, yq1, dq) of the first corresponding point Q1 reaches the object 54. , dr1) as the coordinates of the first intersection R1. The image processing device 11 calculates three-dimensional coordinates (xr2, yr2) where a straight line passing through the coordinates (xv, yv, dv) of the viewpoint V0 and the coordinates (xq2, yq2, dq) of the second corresponding point Q2 reaches the object 54. , dr2) as the coordinates of the second intersection R2. The image processing device 11 calculates the Euclidean distance √((xr2−xr1) ² +(yr2− yr1) ² + (dr2-dr1) ² ) is calculated. The image processing device 11 outputs a numerical value representing the calculated Euclidean distance on the screen 80. Therefore, the user can easily and accurately measure any distance in the three-dimensional image 53, such as the length of the fossa ovalis 66.

The image processing device 11 further specifies a corresponding range 56 on the plane 55 corresponding to the screen 80 in the three-dimensional space in response to the range specification operation, as shown in FIGS. 6 to 9. The range specification operation is an operation for specifying a range 89 on the screen 80. The corresponding range 56 is a range on the plane 55 that corresponds to the designated range 89. The image processing device 11 selects a mark displayed on the screen 80 at a position corresponding to an intersection existing in the three-dimensional area 57 as shown in FIG. 9 between the first intersection R1 and the second intersection R2. Change appearance, such as color or shape. The three-dimensional area 57 is an area that extends in a conical shape from the viewpoint V0 through the outer edge of the corresponding range 56 in the three-dimensional space. FIG. 9 shows an example in which both the first intersection point R1 and the second intersection point R2 exist within the three-dimensional area 57. FIG. 8 shows an example in which the colors of

marks

87 and 88 displayed on the screen 80 at positions corresponding to the first intersection point R1 and the second intersection point R2 are changed.

The image processing device 11 accepts batch operations on marks existing within the specified range 89, such as an operation to collectively delete marks whose appearance has been changed. Therefore, the user can perform efficient operations such as selecting arbitrary marks on the screen 80 and erasing them all at once.

The range specification operation includes an operation of pressing a push button on the input device as a third operation. In this embodiment, the third operation is an operation of pressing a button on the mouse 15 as a push button of the input device, but it may also be an operation of pressing a specific key of the keyboard 14 as a push button of the input device. The image processing device 11 identifies a point on the plane 55 that corresponds to the position of the pointer 86 on the screen 80 when the third operation was performed. FIG. 6 shows an example in which the pointer 86 is at a position corresponding to a point away from the fossa ovalis 66 to the upper left when the third operation is performed. Although the pointer 86 has an arrow shape in this example, it may have another shape such as a cross shape. For example, when the user switches the operation mode from another mode to the range specification operation mode, the shape of the pointer 86 may be changed so that the user can easily recognize the operation mode. Then, when the user switches the operation mode from the range specification operation mode to another mode, the shape of the pointer 86 may be changed again.

From the viewpoint of preventing erroneous operations, the third operation may be an operation of pressing a push button while pressing a predetermined third key. The third key is, for example, the Ctrl key or the Shift key on the keyboard 14. The third key may be the same key as the first key, or may be a different key from the first key. The third key may be the same key as the second key, or may be a different key from the second key.

The range specifying operation includes, as a fourth operation, an operation of releasing the push button, which is performed following the third operation and the drag operation of moving the pointer 86 while holding down the push button. The image processing device 11 identifies a point on the plane 55 that corresponds to the position of the pointer 86 when the fourth operation was performed. FIG. 7 shows an example in which the pointer 86 is located at a position corresponding to a point farther to the lower right from the fossa ovalis 66 when the fourth operation is performed.

From the viewpoint of preventing erroneous operations, the fourth operation may be an operation of releasing a push button while pressing a predetermined fourth key. The fourth key is, for example, the Ctrl key or the Shift key on the keyboard 14. The fourth key may be the same key as the first key, or may be a different key from the first key. The fourth key may be the same key as the second key, or may be a different key from the second key. The fourth key may be the same key as the third key, or may be a different key from the third key.

In the present embodiment, the image processing device 11 determines the point on the plane 55 corresponding to the position of the pointer 86 on the screen 80 when the third operation is performed, and the point 86 when the fourth operation is performed. A rectangular range whose diagonal vertices are the point corresponding to the position of is specified as the corresponding range 56. FIG. 8 shows, as a range 89, a rectangular range having diagonal vertices at a position corresponding to a point distant to the upper left from the fossa ovalis 66 and a position corresponding to a point corresponding to a point distant to the lower right from the fossa ovalis 66. An example where is specified is shown.

As shown in FIGS. 10 to 12, a circular range may be specified as the range 89 instead of a rectangular range. That is, the image processing device 11 uses a point on the plane 55 that corresponds to the position of the pointer 86 on the screen 80 when the third operation was performed as a center point, and a point on the plane 55 that corresponds to the position of the pointer 86 on the screen 80 when the fourth operation was performed. A circular range whose circumferential points are points corresponding to the positions of may be specified as the corresponding range 56. FIG. 10 shows an example in which the pointer 86 is located at a position corresponding to the center point of the fossa ovalis 66 when the third operation is performed. FIG. 11 shows an example in which the pointer 86 is at a position corresponding to a point downwardly away from the fossa ovalis 66 when the fourth operation is performed. In FIG. 12, a circular range is designated as a range 89, with a center point at a position corresponding to the center point of the fossa ovalis 66 and a circumferential point at a position corresponding to a point downwardly away from the fossa ovalis 66. An example is shown below. Similar to FIG. 8, FIG. 12 shows an example in which the colors of the

marks

87 and 88 displayed on the screen 80 at positions corresponding to the first intersection point R1 and the second intersection point R2 are changed. .

According to the example of FIGS. 6 to 8, the user moves the pointer 86 to a desired position with the mouse 15, presses the button of the mouse 15 to specify one vertex position, and while holding down the button of the mouse 15, moves the pointer 86 to a desired position. can be moved to another desired position and the rectangular range can be designated as range 89 by releasing the mouse 15 button. Alternatively, according to the example of FIGS. 10 to 12, the user moves the pointer 86 to a desired position with the mouse 15, presses the button of the mouse 15 to specify the center point position, and while holding down the button of the mouse 15, moves the pointer 86 to a desired position. 86 to another desired position and release the mouse 15 button to designate the circular area as area 89. The image processing device 11 specifies a two-dimensional range corresponding to the designated range 89 as the corresponding range 56, as shown in FIG. The image processing device 11 specifies a cone-shaped area extending through the coordinates (xv, yv, dv) of the viewpoint V0 and the outer edge of the corresponding range 56 as a three-dimensional area 57. The image processing device 11 determines that the coordinates (xr1, yr1, dr1) of the first intersection R1 and the coordinates (xr2, yr2, dr2) of the second intersection R2 are within the three-dimensional area 57. The image processing device 11 changes the colors of the

marks

87 and 88 displayed on the screen 80 at positions corresponding to the first intersection R1 and the second intersection R2, respectively. Therefore, the user can easily select the

marks

87 and 88.

As shown in FIG. 14, the image processing device 11 forms a cutting region 62 in the three-dimensional data 52 that exposes the inner cavity 63 of the biological tissue 60 in the three-dimensional image 53. The image processing device 11 adjusts the viewpoint V0 when displaying the three-dimensional image 53 on the display 16 according to the position of the cutting area 62. When the image processing device 11 receives a user operation requesting rotation of the viewpoint V0, the image processing device 11 passes through a reference point located in the lumen 63 on a reference plane that extends horizontally in the three-dimensional image 53 and includes the viewpoint V0, and rotates the reference plane. The position of the cutting area 62 is changed from the first position, which is the position when the user's operation was performed, to the second position rotated around the rotation axis extending in the direction perpendicular to . The horizontal direction refers to the XY directions shown in FIG. The direction perpendicular to the reference plane is the Z direction shown in FIG. The reference point may be any point located within the lumen 63 on the reference plane, such as the center point of the IVUS catheter, but in this embodiment, it is the center of gravity of the lumen 63 on the reference plane. In the example of FIG. 14, when the cross section C1 of the living tissue 60 overlaps the reference plane, the corresponding center of gravity B1 becomes the reference point. The same applies to the cross sections C2, C3, and C4 of the living tissue 60, and the corresponding centers of gravity B2, B3, and B4. The image processing device 11 rotates the viewpoint V0 around the rotation axis according to the second position.

According to this embodiment, the usefulness for confirming the position within the living tissue 60 is improved. For example, when a procedure such as ablation using IVUS is performed, a user such as a doctor performing the procedure while operating a catheter or a clinical engineer operating the IVUS system while looking at the display 16 By performing a specific user operation, the viewpoint V0 can be rotated around the rotation axis.

As shown in FIGS. 2 to 4, 6 to 8, and 10 to 12, the image processing device 11 identifies a cross section 64 of the living tissue 60 and a region 65 corresponding to the cutting region 62 in the cross section 64. The two-dimensional image 58 and the three-dimensional image 53 are displayed on the display 16. In the two-dimensional image 58, the position of the camera 71 with respect to the cross section 64 is displayed.

According to this embodiment, it is possible to show how a part of the structure of the living tissue 60 is cut out. Therefore, the user can understand from the two-dimensional image 58 what kind of structure the portion of the biological tissue 60 that is cut out and not displayed in the three-dimensional image 53 has. For example, if the user is a surgeon, it becomes easier to perform surgery on the inside of the living tissue 60.

The biological tissue 60 includes, for example, blood vessels or organs such as the heart. The biological tissue 60 is not limited to anatomically a single organ or a part thereof, but also includes a tissue having a lumen spanning multiple organs. An example of such a tissue is, specifically, a part of the vascular tissue that extends from the upper part of the inferior vena cava, passes through the right atrium, and reaches the lower part of the superior vena cava.

In FIGS. 2 to 4, 6 to 8, and 10 to 12, an operation panel 81, a two-dimensional image 58, a three-dimensional image 53, and a pointer 86 are displayed on a screen 80.

The operation panel 81 is a GUI component for setting the cutting area 62. “GUI” is an abbreviation for graphical user interface. The operation panel 81 includes a check box 82 for selecting whether to activate the setting of the cutting area 62, a slider 83 for setting the base angle, a slider 84 for setting the opening angle, and a center of gravity. A check box 85 is provided for selecting whether or not to use the .

The base angle is the rotation angle of one of the two straight lines L1 and L2 extending from one point M in the cross-sectional image representing the cross-section 64 of the biological tissue 60. Therefore, setting the base angle corresponds to setting the direction of the straight line L1. The opening angle is the angle between the two straight lines L1 and L2. Therefore, setting the opening angle corresponds to setting the angle formed by the two straight lines L1 and L2. Point M is the center of gravity of cross section 64. Point M may be set at a point other than the center of gravity on the cross section 64 if it is selected not to use the center of gravity.

The two-dimensional image 58 is an image obtained by processing a cross-sectional image. In the two-dimensional image 58, the color of an area 65 corresponding to the cutting area 62 is changed to clearly indicate which part of the cross section 64 has been cut.

In this embodiment, the viewpoint V0 when displaying the three-dimensional image 53 on the screen 80 is adjusted according to the position of the cutting region 62.

In this embodiment, the cutting area 62 can be determined using the two-dimensional image 58. Specifically, as shown in FIG. 13, by adjusting the base angle or opening angle and setting the position or size of the area 65 separated by two straight lines L1 and L2 in the two-dimensional image 58. , the position or size of the cutting area 62 can be set. For example, if the base angle is changed so that the straight line L1 is rotated approximately 90 degrees counterclockwise, a region 65a that is moved in accordance with the change in the base angle is obtained in the two-dimensional image 58a. Then, the position of the cutting area 62 is adjusted according to the position of the area 65a. Alternatively, if the opening angle is changed so that the angle between the two straight lines L1 and L2 becomes larger, a region 65b that is enlarged according to the change in the opening angle is obtained in the two-dimensional image 58b. Then, the size of the cutting area 62 is adjusted according to the size of the area 65b. By adjusting both the base angle and the opening angle and setting both the position and size of the area 65 in the two-dimensional image 58, it is also possible to set both the position and size of the cutting area 62. The position of the camera 71 may be adjusted as appropriate depending on the position or size of the cutting area 62.

In this embodiment, the image corresponding to the current position of the sensor, that is, the latest image, is always displayed as the two-dimensional image 58.

As a modified example of this embodiment, the base angle may be set by dragging the straight line L1 instead of being set by operating the slider 83, or by inputting a numerical value. Good too. Similarly, instead of being set by operating the slider 84, the opening angle may be set by dragging the straight line L2 or by inputting a numerical value.

In the three-dimensional image 53, the cutting area 62 determined using the two-dimensional image 58 is hidden or transparent.

In FIG. 14, the X direction and the Y direction perpendicular to the X direction each correspond to the lateral direction of the lumen 63 of the living tissue 60. The Z direction orthogonal to the X direction and the Y direction corresponds to the longitudinal direction of the lumen 63 of the living tissue 60.

In the example of FIG. 14, it is assumed that the check box 85 on the operation panel 81 is checked, that is, the use of the center of gravity is selected. The image processing device 11 uses the three-dimensional data 52 to calculate the positions of the centers of gravity B1, B2, B3, and B4 of the cross sections C1, C2, C3, and C4 of the living tissue 60, respectively. The image processing device 11 sets two planes that intersect at a line Lb passing through the positions of the centers of gravity B1, B2, B3, and B4 and include two straight lines L1 and L2, respectively, as cutting planes P1 and P2. For example, if point M shown in FIGS. 2 to 4, 6 to 8, and 10 to 12 is point B3, straight line L1 is the intersection of cross section C3 and cut plane P1, and straight line L2 is This is the intersection line between the cross section C3 and the cut surface P2. The image processing device 11 forms a region in the three-dimensional image 53 between the cutting planes P1 and P2 and exposing the inner cavity 63 of the living tissue 60 as a cutting region 62 in the three-dimensional data 52 .

In the case of a three-dimensional model of a bent biological tissue 60 as shown in FIG. 14, if the three-dimensional model is cut along one plane to display the lumen 63, the inside of the biological tissue 60 may not be displayed correctly in some cases. In this embodiment, as shown in FIG. 14, by continuing to capture the center of gravity of each cross section of the living tissue 60, it is possible to cut the three-dimensional model so that the inside of the living tissue 60 can be reliably displayed. .

In FIG. 14, for convenience, four cross sections C1, C2, C3, and C4 are shown as multiple cross sections in the transverse direction of the internal cavity 63 of the living tissue 60, but the number of cross sections for which the center of gravity position is calculated is four. However, the number is preferably the same as the number of cross-sectional images obtained by IVUS.

As an example different from that shown in FIG. 14, assume that the check box 85 on the operation panel 81 is not checked, that is, it is selected not to use the center of gravity. In such an example, the image processing device 11 intersects at an arbitrary line passing through point M, such as a straight line extending in the Z direction through point M, and includes two straight lines L1 and L2, respectively. The planes are set as cutting planes P1 and P2.

With reference to FIG. 1, the configuration of an image processing system 10 according to the present embodiment will be described.

The image processing system 10 includes an image processing device 11, a cable 12, a drive unit 13, a keyboard 14, a mouse 15, and a display 16.

Although the image processing device 11 is a dedicated computer specialized for image diagnosis in this embodiment, it may be a general-purpose computer such as a PC. "PC" is an abbreviation for personal computer.

The cable 12 is used to connect the image processing device 11 and the drive unit 13.

The drive unit 13 is a device that is used by being connected to the probe 20 shown in FIG. 16 and drives the probe 20. Drive unit 13 is also called MDU. "MDU" is an abbreviation for motor drive unit. The probe 20 is applied to IVUS. The probe 20 is also called an IVUS catheter or an imaging catheter.

The keyboard 14, mouse 15, and display 16 are connected to the image processing device 11 via any cable or wirelessly. The display 16 is, for example, an LCD, an organic EL display, or an HMD. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescence. "HMD" is an abbreviation for head-mounted display.

The image processing system 10 further includes a connection terminal 17 and a cart unit 18 as options.

The connection terminal 17 is used to connect the image processing device 11 and external equipment. The connection terminal 17 is, for example, a USB terminal. "USB" is an abbreviation for Universal Serial Bus. The external device is, for example, a recording medium such as a magnetic disk drive, a magneto-optical disk drive, or an optical disk drive.

The cart unit 18 is a cart with casters for movement. An image processing device 11, a cable 12, and a drive unit 13 are installed in the cart body of the cart unit 18. A keyboard 14, a mouse 15, and a display 16 are installed on the top table of the cart unit 18.

With reference to FIG. 16, the configurations of the probe 20 and drive unit 13 according to this embodiment will be described.

The probe 20 includes a drive shaft 21, a hub 22, a sheath 23, an outer tube 24, an ultrasonic transducer 25, and a relay connector 26.

The drive shaft 21 passes through a sheath 23 inserted into the body cavity of a living body and an outer tube 24 connected to the proximal end of the sheath 23, and extends to the inside of the hub 22 provided at the proximal end of the probe 20. The drive shaft 21 has an ultrasonic transducer 25 at its tip that transmits and receives signals, and is rotatably provided within the sheath 23 and the outer tube 24 . Relay connector 26 connects sheath 23 and outer tube 24.

The hub 22, the drive shaft 21, and the ultrasonic transducer 25 are connected to each other so that they each move forward and backward in the axial direction. Therefore, for example, when the hub 22 is pushed toward the distal end, the drive shaft 21 and the ultrasonic transducer 25 move inside the sheath 23 toward the distal end. For example, when the hub 22 is pulled toward the proximal end, the drive shaft 21 and the ultrasonic transducer 25 move inside the sheath 23 toward the proximal end, as shown by the arrows.

The drive unit 13 includes a scanner unit 31, a slide unit 32, and a bottom cover 33.

The scanner unit 31 is also called a pullback unit. Scanner unit 31 is connected to image processing device 11 via cable 12 . The scanner unit 31 includes a probe connection section 34 that connects to the probe 20 and a scanner motor 35 that is a drive source that rotates the drive shaft 21 .

The probe connecting portion 34 is detachably connected to the probe 20 via the insertion port 36 of the hub 22 provided at the base end of the probe 20. Inside the hub 22, the base end of the drive shaft 21 is rotatably supported, and the rotational force of the scanner motor 35 is transmitted to the drive shaft 21. Further, signals are transmitted and received between the drive shaft 21 and the image processing device 11 via the cable 12. The image processing device 11 generates a tomographic image of a living body lumen and performs image processing based on signals transmitted from the drive shaft 21 .

The slide unit 32 carries the scanner unit 31 so that it can move forward and backward, and is mechanically and electrically connected to the scanner unit 31. The slide unit 32 includes a probe clamp section 37, a slide motor 38, and a switch group 39.

The probe clamp section 37 is disposed coaxially with the probe connection section 34 on the distal side thereof, and supports the probe 20 connected to the probe connection section 34 .

The slide motor 38 is a drive source that generates axial driving force. The scanner unit 31 is moved forward and backward by the drive of the slide motor 38, and the drive shaft 21 is accordingly moved forward and backward in the axial direction. The slide motor 38 is, for example, a servo motor.

The switch group 39 includes, for example, a forward switch and a pullback switch that are pressed when moving the scanner unit 31 forward or backward, and a scan switch that is pressed when starting and ending image depiction. The switch group 39 is not limited to this example, and various switches may be included in the switch group 39 as necessary.

When the forward switch is pressed, the slide motor 38 rotates forward and the scanner unit 31 moves forward. On the other hand, when the pullback switch is pressed, the slide motor 38 rotates in the reverse direction, and the scanner unit 31 moves backward.

When the scan switch is pressed, image depiction is started, and the scanner motor 35 is driven, and the slide motor 38 is driven to move the scanner unit 31 backward. A user such as a surgeon connects the probe 20 to the scanner unit 31 in advance so that the drive shaft 21 rotates and moves toward the proximal end side in the axial direction when image depiction starts. The scanner motor 35 and the slide motor 38 are stopped when the scan switch is pressed again, and image rendering is completed.

The bottom cover 33 covers the bottom surface of the slide unit 32 and the entire circumference of the side surface on the bottom side, and is movable toward and away from the bottom surface of the slide unit 32.

The configuration of the image processing device 11 will be described with reference to FIG. 15.

The image processing device 11 includes a control section 41, a storage section 42, a communication section 43, an input section 44, and an output section 45.

The control unit 41 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. “GPU” is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. “ASIC” is an abbreviation for application specific integrated circuit. The control unit 41 executes processing related to the operation of the image processing device 11 while controlling each part of the image processing system 10 including the image processing device 11.

The storage unit 42 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, RAM or ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, SRAM or DRAM. “SRAM” is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 42 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 42 stores data used for the operation of the image processing device 11, such as tomographic data 51, and data obtained by the operation of the image processing device 11, such as 3D data 52 and 3D images 53. .

The communication unit 43 includes at least one communication interface. The communication interface is, for example, a wired LAN interface, a wireless LAN interface, or an image diagnosis interface that receives and A/D converts IVUS signals. "LAN" is an abbreviation for local area network. "A/D" is an abbreviation for analog to digital. The communication unit 43 receives data used for the operation of the image processing device 11 and transmits data obtained by the operation of the image processing device 11. In this embodiment, the drive unit 13 is connected to an image diagnosis interface included in the communication section 43.

The input unit 44 includes at least one input interface. The input interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 44 accepts user operations such as inputting data used for the operation of the image processing device 11 . In this embodiment, the keyboard 14 and mouse 15 are connected to a USB interface included in the input unit 44 or an interface compatible with near field communication. If the touch screen is provided integrally with the display 16, the display 16 may be connected to a USB interface or an HDMI (registered trademark) interface included in the input section 44.

The output unit 45 includes at least one output interface. The output interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). The output unit 45 outputs data obtained by the operation of the image processing device 11. In this embodiment, the display 16 is connected to a USB interface or an HDMI (registered trademark) interface included in the output unit 45.

The functions of the image processing device 11 are realized by executing the image processing program according to the present embodiment by a processor serving as the control unit 41. That is, the functions of the image processing device 11 are realized by software. The image processing program causes the computer to function as the image processing apparatus 11 by causing the computer to execute the operations of the image processing apparatus 11 . That is, the computer functions as the image processing device 11 by executing the operations of the image processing device 11 according to the image processing program.

The program may be stored on a non-transitory computer-readable medium. The non-transitory computer-readable medium is, for example, a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. Distribution of the program is performed, for example, by selling, transferring, or lending a portable medium such as an SD card, DVD, or CD-ROM that stores the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

For example, a computer temporarily stores a program stored on a portable medium or a program transferred from a server in its main storage device. Then, the computer uses a processor to read a program stored in the main memory, and causes the processor to execute processing according to the read program. A computer may read a program directly from a portable medium and execute processing according to the program. The computer may sequentially execute processing according to the received program each time the program is transferred to the computer from the server. Processing may be performed using a so-called ASP type service that implements functions only by issuing execution instructions and obtaining results without transferring programs from the server to the computer. “ASP” is an abbreviation for application service provider. The program includes information that is used for processing by an electronic computer and is equivalent to a program. For example, data that is not a direct command to a computer but has the property of regulating computer processing falls under "something similar to a program."

A part or all of the functions of the image processing device 11 may be realized by a programmable circuit or a dedicated circuit as the control unit 41. That is, some or all of the functions of the image processing device 11 may be realized by hardware.

The operation of the image processing system 10 according to this embodiment will be described with reference to FIGS. 17 and 18. The operation of the image processing system 10 corresponds to the image display method according to this embodiment.

Before starting the flow in FIG. 17, the probe 20 is primed by the user. Thereafter, the probe 20 is fitted into the probe connection part 34 and probe clamp part 37 of the drive unit 13, and is connected and fixed to the drive unit 13. The probe 20 is then inserted to a target site within the living tissue 60 such as a blood vessel or heart.

In step S101, the scan switch included in the switch group 39 is pressed, and the pullback switch included in the switch group 39 is further pressed, thereby performing a so-called pullback operation. The probe 20 transmits ultrasonic waves inside the living tissue 60 by using the ultrasonic transducer 25 that retreats in the axial direction by a pullback operation. The ultrasonic transducer 25 transmits ultrasonic waves in a radial manner while moving inside the living tissue 60 . The ultrasonic transducer 25 receives reflected waves of the transmitted ultrasonic waves. The probe 20 inputs the signal of the reflected wave received by the ultrasound transducer 25 to the image processing device 11 . The control unit 41 of the image processing device 11 acquires tomographic data 51 including a plurality of cross-sectional images by processing the input signals and sequentially generating cross-sectional images of the biological tissue 60.

Specifically, the probe 20 rotates the ultrasonic transducer 25 in the circumferential direction and moves it in the axial direction inside the living tissue 60, and the ultrasonic transducer 25 causes a plurality of waves outward from the center of rotation. Send ultrasound waves in the direction. The probe 20 uses the ultrasonic transducer 25 to receive reflected waves from reflective objects existing in multiple directions inside the living tissue 60 . The probe 20 transmits the received reflected wave signal to the image processing device 11 via the drive unit 13 and the cable 12. The communication unit 43 of the image processing device 11 receives the signal transmitted from the probe 20. The communication unit 43 performs A/D conversion on the received signal. The communication section 43 inputs the A/D converted signal to the control section 41 . The control unit 41 processes the input signal and calculates the intensity value distribution of reflected waves from a reflecting object existing in the ultrasonic wave transmission direction of the ultrasonic transducer 25 . The control unit 41 acquires tomographic data 51, which is a data set of cross-sectional images, by sequentially generating two-dimensional images having a luminance value distribution corresponding to the calculated intensity value distribution as cross-sectional images of the biological tissue 60. The control unit 41 causes the storage unit 42 to store the acquired tomographic data 51.

In this embodiment, the reflected wave signal received by the ultrasound transducer 25 corresponds to the raw data of the tomographic data 51, and the cross-sectional image that the image processing device 11 generates by processing the reflected wave signal corresponds to the tomographic data 51. This corresponds to 51 processed data.

As a modification of the present embodiment, the control unit 41 of the image processing device 11 may store the signal input from the probe 20 as it is in the storage unit 42 as the tomographic data 51. Alternatively, the control unit 41 may cause the storage unit 42 to store data indicating the intensity value distribution of reflected waves calculated by processing the signal input from the probe 20 as the tomographic data 51. That is, the tomographic data 51 is not limited to a data set of cross-sectional images of the living tissue 60, but may be any data that represents the cross-section of the living tissue 60 at each movement position of the ultrasound transducer 25 in some format.

As a modification of this embodiment, instead of the ultrasonic transducer 25 that transmits ultrasonic waves in multiple directions while rotating in the circumferential direction, an ultrasonic vibrator that transmits ultrasonic waves in multiple directions without rotating is used. It's okay.

As a modification of this embodiment, the tomographic data 51 may be acquired using OFDI or OCT instead of being acquired using IVUS. "OFDI" is an abbreviation for optical frequency domain imaging. "OCT" is an abbreviation for optical coherence tomography. When OFDI or OCT is used, an ultrasound sensor that acquires tomographic data 51 by transmitting ultrasound in the lumen 63 of the living tissue 60 is used as a sensor that acquires the tomographic data 51 while moving through the lumen 63 of the living tissue 60. Instead of the vibrator 25, a sensor is used that emits light in the lumen 63 of the living tissue 60 to acquire the tomographic data 51.

As a modification of this embodiment, instead of the image processing device 11 generating a dataset of cross-sectional images of the living tissue 60, another device generates a similar dataset, and the image processing device 11 generates the dataset. It may also be acquired from the other device. That is, instead of the control unit 41 of the image processing device 11 processing the IVUS signal to generate a cross-sectional image of the biological tissue 60, another device processes the IVUS signal to generate a cross-sectional image of the biological tissue 60. The generated cross-sectional image may be input to the image processing device 11.

In step S102, the control unit 41 of the image processing device 11 generates three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 acquired in step S101. That is, the control unit 41 generates three-dimensional data 52 based on the tomographic data 51 acquired by the sensor. Here, if there is already generated 3D data 52, instead of regenerating all 3D data 52 from scratch, it is possible to update only the data at the location to which the updated tomographic data 51 corresponds. preferable. In that case, the amount of data processing when generating the three-dimensional data 52 can be reduced, and the real-time performance of the three-dimensional image 53 in the subsequent step S103 can be improved.

Specifically, the control unit 41 of the image processing device 11 stacks and three-dimensionalizes the cross-sectional images of the biological tissue 60 included in the tomographic data 51 stored in the storage unit 42, thereby creating three-dimensional images of the biological tissue 60. Generate dimensional data 52. As the three-dimensionalization method, any one of various processing methods such as rendering methods such as surface rendering or volume rendering, and accompanying texture mapping including environment mapping, and bump mapping may be used. The control unit 41 causes the storage unit 42 to store the generated three-dimensional data 52.

When a long medical device such as a catheter other than the IVUS catheter is inserted into the living tissue 60, the tomographic data 51 includes data on the long medical device in the same way as the data of the living tissue 60. It is included. Therefore, in step S102, the three-dimensional data 52 generated by the control unit 41 also includes data on the elongated medical instrument, similar to the data on the living tissue 60.

The control unit 41 of the image processing device 11 classifies the pixel group of the cross-sectional image included in the cross-sectional data 51 acquired in step S101 into two or more classes. These two or more classes include at least a "tissue" class to which the biological tissue 60 belongs, a "medical device" class to which long medical devices belong, a "blood cell" class, an indwelling stent, etc. may further include a class of "indwellings", or a class of "lesions", such as lime or plaque. Although any method may be used as the classification method, in this embodiment, a method of classifying pixel groups of a cross-sectional image using a trained model is used. The trained model is trained to detect regions corresponding to each class from sample IVUS cross-sectional images by performing machine learning in advance.

In step S103, the control unit 41 of the image processing device 11 displays the three-dimensional data 52 generated in step S102 on the display 16 as a three-dimensional image 53. At this point, the control unit 41 may set the angle at which the three-dimensional image 53 is displayed to an arbitrary angle. The control unit 41 causes the display 16 to display the latest cross-sectional image included in the tomographic data 51 acquired in step S101 together with the three-dimensional image 53.

Specifically, the control unit 41 of the image processing device 11 generates a three-dimensional image 53 from the three-dimensional data 52 stored in the storage unit 42. The three-dimensional image 53 includes a group of three-dimensional objects such as a three-dimensional object representing the biological tissue 60 and a three-dimensional object representing a long medical instrument. That is, the control unit 41 generates a three-dimensional object of the biological tissue 60 from the data of the biological tissue 60 stored in the storage unit 42, and generates a three-dimensional object of the biological tissue 60 from the data of the long medical instrument stored in the storage unit 42. A three-dimensional object of a medical device is generated. The control unit 41 displays the latest cross-sectional image of the biological tissue 60 included in the tomographic data 51 stored in the storage unit 42 and the generated three-dimensional image 53 on the display 16 via the output unit 45. to be displayed.

In step S104, if the user performs a change operation to set the angle at which the three-dimensional image 53 is displayed, the process in step S105 is executed. If there is no change operation by the user, the process of step S106 is executed.

In step S105, the control unit 41 of the image processing device 11 receives, via the input unit 44, an operation to set the angle at which the three-dimensional image 53 is displayed. The control unit 41 adjusts the angle at which the three-dimensional image 53 is displayed to the set angle. Then, in step S103, the control unit 41 causes the display 16 to display the three-dimensional image 53 at the angle set in step S105.

Specifically, the control unit 41 of the image processing device 11 allows the user to control the three-dimensional image 53 displayed on the display 16 by using the keyboard 14, the mouse 15, or the touch screen provided integrally with the display 16. A rotation operation is accepted via the input unit 44. The control unit 41 interactively adjusts the angle at which the three-dimensional image 53 is displayed on the display 16 in accordance with a user's operation. Alternatively, the control unit 41 may control the input unit 44 to allow the user to input a numerical value of the angle at which the three-dimensional image 53 is to be displayed using the keyboard 14, the mouse 15, or a touch screen provided integrally with the display 16. Accepted through. The control unit 41 adjusts the angle at which the three-dimensional image 53 is displayed on the display 16 in accordance with the input numerical value.

In step S106, if the tomographic data 51 is updated, the processes of step S107 and step S108 are executed. If the tomographic data 51 is not updated, in step S104, the presence or absence of the user's change operation is checked again.

In step S107, similarly to the process in step S101, the control unit 41 of the image processing device 11 processes the signal input from the probe 20 to newly generate a cross-sectional image of the biological tissue 60, thereby generating at least one cross-sectional image of the biological tissue 60. Tomographic data 51 including a new cross-sectional image is acquired.

In step S108, the control unit 41 of the image processing device 11 updates the three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 acquired in step S107. That is, the control unit 41 updates the three-dimensional data 52 based on the tomographic data 51 acquired by the sensor. Then, in step S103, the control unit 41 causes the display 16 to display the three-dimensional data 52 updated in step S108 as a three-dimensional image 53. The control unit 41 causes the display 16 to display the latest cross-sectional image included in the tomographic data 51 acquired in step S107 together with the three-dimensional image 53. In step S108, it is preferable to update only the data corresponding to the updated tomographic data 51. In that case, the amount of data processing when generating the three-dimensional data 52 can be reduced, and the real-time performance of the three-dimensional image 53 can be improved in step S108.

In step S111, if the user performs a setting operation to set the cutting area 62, the process in step S112 is executed.

In step S112, the control unit 41 of the image processing device 11 receives an operation to set the cutting area 62 via the input unit 44.

Specifically, the control unit 41 of the image processing device 11 performs an operation to set an area 65 corresponding to the cutting area 62 on the cross-sectional image displayed on the display 16 in step S103 via the input unit 44. accept. In this embodiment, the control unit 41 receives an operation for setting two straight lines L1 and L2 extending from one point M in the cross-sectional image as an operation for setting a region 65 corresponding to the cutting region 62.

More specifically, the control unit 41 of the image processing device 11 controls the base angle and the opening angle on the operation panel 81 as shown in FIGS. 2 to 4, 6 to 8, and 10 to 12. An operation in which the user specifies using the keyboard 14 , mouse 15 , or touch screen provided integrally with the display 16 is accepted via the input unit 44 . That is, the control unit 41 specifies the direction of one of the two straight lines L1, L2 and the angle formed by the two straight lines L1, L2 as an operation for setting the two straight lines L1, L2. accepts operations to do. Here, it is assumed that the check box 85 on the operation panel 81 is checked, that is, the use of the center of gravity is selected.

As a modified example of the present embodiment, the control unit 41 of the image processing device 11 allows the user to move two straight lines L1 and L2 on the cross-sectional image displayed on the display 16 using the keyboard 14, the mouse 15, or the display 16. An operation for drawing using an integrally provided touch screen may be accepted via the input unit 44. That is, the control unit 41 may receive an operation to draw the two straight lines L1 and L2 on the cross-sectional image as an operation to set the two straight lines L1 and L2.

In step S113, the control unit 41 of the image processing device 11 uses the latest three-dimensional data 52 stored in the storage unit 42 to calculate the position of the center of gravity of a plurality of cross-sections in the transverse direction of the internal cavity 63 of the living tissue 60. . The latest three-dimensional data 52 is the three-dimensional data 52 generated in step S102 if the process of step S108 has not been executed, and the latest three-dimensional data 52 is the three-dimensional data 52 generated in step S102 if the process of step S108 has been executed. This refers to the three-dimensional data 52 that has been created. Here, if there is already generated three-dimensional data 52, it is preferable to update only the data at the location to which the updated tomographic data 51 corresponds, rather than re-generating all three-dimensional data 52 from scratch. . In that case, the amount of data processing when generating the three-dimensional data 52 can be reduced, and the real-time performance of the three-dimensional image 53 in the subsequent step S117 can be improved. Specifically, the process in step S113 can be executed using a procedure similar to that disclosed in International Publication No. 2021/200294.

In step S114, the control unit 41 of the image processing device 11 performs smoothing on the calculation result of the center of gravity position in step S113. Specifically, the process in step S114 can be executed using a procedure similar to that disclosed in International Publication No. 2021/200294.

In step S115, the control unit 41 of the image processing device 11 sets two planes intersecting by a line Lb passing through the center of gravity position calculated in step S113 as cutting planes P1 and P2, as shown in FIG. . In this embodiment, the control unit 41 performs smoothing on the calculation result of the center of gravity position in step S114 and then sets the cutting planes P1 and P2, but the process in step S114 may be omitted.

Specifically, the control unit 41 of the image processing device 11 sets the curve of the center of gravity position obtained as a result of the smoothing in step S114 as the line Lb. The control unit 41 sets two planes that intersect at the set line Lb and include the two straight lines L1 and L2 set in step S112, respectively, as cutting planes P1 and P2. The control unit 41 determines the three-dimensional coordinates intersecting the cut planes P1 and P2 of the biological tissue 60 in the latest three-dimensional data 52 stored in the storage unit 42, and the internal cavity 63 of the biological tissue 60 in the three-dimensional image 53. Specify as the three-dimensional coordinates of the edge of the opening to be exposed. The control unit 41 causes the storage unit 42 to store the specified three-dimensional coordinates.

In step S116, the control unit 41 of the image processing device 11 forms a region in the three-dimensional data 52, which is sandwiched between the cutting planes P1 and P2 in the three-dimensional image 53 and exposes the inner cavity 63 of the biological tissue 60, as a cutting region 62. do.

Specifically, the control unit 41 of the image processing device 11 converts the portion specified by the three-dimensional coordinates stored in the storage unit 42 into a three-dimensional image in the latest three-dimensional data 52 stored in the storage unit 42. 53 is set to be hidden or transparent when displayed on the display 16. That is, the control unit 41 forms the cutting area 62 in accordance with the area 65 set in step S112.

In step S117, the control unit 41 of the image processing device 11 causes the display 16 to display the three-dimensional data 52 in which the cutting area 62 was formed in step S116 as the three-dimensional image 53. The control unit 41 displays a cross section 64 represented by the cross-sectional image displayed on the display 16 in step S103, which is indicated by the tomographic data 51 newly acquired by the sensor, and a region 65 corresponding to the cutting region 62 in the cross section 64. A two-dimensional image 58 representing the image is displayed on the display 16 together with the three-dimensional image 53.

Specifically, the control unit 41 of the image processing device 11 processes the latest cross-sectional image of the biological tissue 60 included in the tomographic data 51 stored in the storage unit 42, and converts the images from FIG. 4. Generate a two-dimensional image 58 as shown in FIGS. 6 to 8 and 10 to 12. The control unit 41 is shown in FIGS. 2 to 4, 6 to 8, and 10 to 12, in which the portion specified by the three-dimensional coordinates stored in the storage unit 42 is hidden or transparent. A three-dimensional image 53 is generated. The control unit 41 displays the generated two-dimensional image 58 and three-dimensional image 53 on the display 16 via the output unit 45.

In this embodiment, the control unit 41 of the image processing device 11 generates a two-dimensional image 58 corresponding to the cutting area 62, as shown in FIGS. 2 to 4, 6 to 8, and 10 to 12. An image is generated in which the color of the region 65 is different from that of the remaining regions. For example, it is conceivable to change the white part in a general IVUS image to red in the region 65.

In step S118, if there is an operation to set the cutting area 62 as a user change operation, the process of step S119 is executed. If there is no change operation by the user, the process of step S120 is executed.

In step S119, the control unit 41 of the image processing device 11 receives an operation to set the cutting area 62 via the input unit 44, similar to the process in step S112. Then, the processing from step S115 onwards is executed.

In step S120, if the tomographic data 51 is updated, the processes of step S121 and step S122 are executed. If the tomographic data 51 is not updated, in step S118, the presence or absence of the user's change operation is checked again.

In step S121, the control unit 41 of the image processing device 11 processes the signal input from the probe 20 to newly generate a cross-sectional image of the living tissue 60, similarly to the process in step S101 or step S107. Tomographic data 51 including at least one new cross-sectional image is acquired.

In step S122, the control unit 41 of the image processing device 11 updates the three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 acquired in step S121. After that, the processing from step S113 onwards is executed. In step S122, it is preferable to update only the data corresponding to the updated tomographic data 51. In that case, the amount of data processing when generating the three-dimensional data 52 can be reduced, and the real-time performance of the data processing after step S113 can be improved.

The operation of the image processing system 10 according to this embodiment will be further described with reference to FIG. 19.

The flow of FIG. 19 is such that in step S103 or step S117, the control unit 41 of the image processing device 11 selects the viewpoint V0 set in the virtual three-dimensional space and the object of the biological tissue 60 arranged in the three-dimensional space. This is performed when the object 54 is rendered based on the positional relationship with the biological tissue 60 and the object 54 is already displayed on the screen 80 as the three-dimensional image 53 of the biological tissue 60 .

In step S201, when a position specifying operation is performed, the control unit 41 of the image processing device 11 receives the position specifying operation via the input unit 44. The position designation operation is an operation to designate two positions on the screen 80. Specifically, the position specifying operation includes an operation of pressing a button on the mouse 15 as the first operation, and is performed successively after the first operation and a drag operation of moving the pointer 86 while holding down the button of the mouse 15. , an operation of releasing a button on the mouse 15 is included as the second operation. The first operation may be an operation of pressing a button on the mouse 15 while holding down a first key such as the Ctrl key or the Shift key on the keyboard 14. The second operation may be an operation of releasing a button on the mouse 15 while pressing a second key such as the Ctrl key or the Shift key on the keyboard 14.

In step S202, the control unit 41 of the image processing device 11 selects a first corresponding point Q1 and a second corresponding point Q1 on the plane 55 corresponding to the screen 80 in the three-dimensional space in response to the position specifying operation performed in step S201. Identify point Q2. As shown in FIG. 5, the first corresponding point Q1 is a point on the plane 55 that corresponds to one of the two positions designated by the position designation operation. The second corresponding point Q2 is a point on the plane 55 that corresponds to the other of the two positions designated by the position designation operation. The control unit 41 calculates the distance ||R2−R1|| between the first intersection point R1 and the second intersection point R2 in the three-dimensional space. As shown in FIG. 5, the first intersection R1 is the intersection of the object 54 and an extension of the straight line connecting the viewpoint V0 and the first corresponding point Q1 in the three-dimensional space. The second intersection R2 is the intersection of the object 54 and an extension of the straight line connecting the viewpoint V0 and the second corresponding point Q2 in the three-dimensional space. Specifically, the control unit 41 specifies the three-dimensional coordinates (xq1, yq1, dq) corresponding to the position specified by the first operation as the coordinates of the first corresponding point Q1. The control unit 41 specifies the three-dimensional coordinates (xq2, yq2, dq) corresponding to the position specified by the second operation as the coordinates of the second corresponding point Q2. The control unit 41 determines the three-dimensional coordinates (xr1, yr1, dr1) is specified as the coordinates of the first intersection point R1. The control unit 41 determines the three-dimensional coordinates (xr2, yr2, dr2) is specified as the coordinates of the second intersection R2. The control unit 41 calculates the Euclidean distance √((xr2-xr1) ² +(yr2-yr1) between the coordinates (xr1, yr1, dr1) of the first intersection R1 and the coordinates (xr2, yr2, dr2) of the second intersection R2. ) ² + (dr2-dr1) ² ) is calculated.

In step S203, the control unit 41 of the image processing device 11 outputs the calculation result obtained in step S202. Specifically, the control unit 41 outputs the numerical value representing the Euclidean distance calculated in step S202 on the screen 80, as shown in FIG.

In step S204, the control unit 41 of the image processing device 11 displays marks 87 and 88 at the first corresponding position and the second corresponding position on the screen 80, respectively. The first corresponding position is a position on the screen 80 that corresponds to the intersection of the plane 55 and a straight line connecting the viewpoint V0 and the first intersection R1 in the three-dimensional space. The second corresponding position is a position on the screen 80 that corresponds to the intersection of the plane 55 and a straight line connecting the viewpoint V0 and the second intersection R2 in the three-dimensional space. Specifically, as shown in FIG. 4, the control unit 41 displays marks 87 and 88 at the positions designated by the first operation and the second operation, respectively. For example, if the position of the viewpoint V0 is changed as a result of a subsequent operation accepted in step S105, step S112, or step S119, the control unit 41 also changes the positions of the

marks

87 and 88.

The flow in FIG. 19 may be repeated any number of times. For example, when N is an integer greater than or equal to 2, 2N marks may be displayed on the screen 80 as a result of the position designation operation being performed N or more times.

The operation of the image processing system 10 according to this embodiment will be further described with reference to FIG. 20.

The flow in FIG. 20 is executed after the flow in FIG. 19 is executed at least once.

In step S211, when a range specification operation is performed, the control unit 41 of the image processing device 11 receives the range specification operation via the input unit 44. The range specification operation is an operation for specifying a range 89 on the screen 80. Specifically, the range specifying operation includes an operation of pressing a button on the mouse 15 as a third operation, and is performed following the third operation and a drag operation of moving the pointer 86 while holding down the button of the mouse 15. , an operation of releasing a button on the mouse 15 is included as the fourth operation. The third operation may be an operation of pressing a button on the mouse 15 while holding down a third key such as the Ctrl key or the Shift key on the keyboard 14. The fourth operation may be an operation of releasing a button on the mouse 15 while pressing a fourth key such as the Ctrl key or the Shift key on the keyboard 14.

In step S212, the control unit 41 of the image processing device 11 specifies the corresponding range 56 on the plane 55 corresponding to the screen 80 in the three-dimensional space in response to the range specifying operation performed in step S211. The corresponding range 56 is a range on the plane 55 that corresponds to the range 89 designated by the range designation operation. The control unit 41 changes the appearance of the mark displayed on the screen 80 at a position corresponding to an intersection existing in the three-dimensional area 57 as shown in FIG. 9 among the marks displayed in step S204. . The three-dimensional area 57 is an area that extends in a conical shape from the viewpoint V0 through the outer edge of the corresponding range 56 in the three-dimensional space. Specifically, the control unit 41 controls the control unit 41 to set two areas on the plane 55 corresponding to a regular range such as a rectangular range or a circular range extending from the position specified in the third operation to the position specified in the fourth operation. The dimension range is specified as the corresponding range 56. The control unit 41 specifies, as a three-dimensional region 57, a region extending in a conical shape through the coordinates (xv, yv, dv) of the viewpoint V0 and the outer edge of the corresponding range 56. The control unit 41 changes the color of the mark displayed on the screen 80 at a position corresponding to an intersection located within the three-dimensional area 57 among the intersections identified in step S202. For example, if the first intersection point R1 and the second intersection point R2 exist within the three-dimensional area 57, the control unit 41 controls the first intersection point R1 and the second intersection point R2, as shown in FIG. 8 or FIG. The colors of the

marks

87 and 88 linked to each are changed. Alternatively, if one or more of the 2N intersections exists within the three-dimensional area 57, the control unit 41 changes the color of one or more marks associated with the one or more intersections. May be changed.

The present disclosure is not limited to the embodiments described above. For example, two or more blocks depicted in the block diagram may be combined, or one block may be divided. Instead of performing two or more steps described in a flowchart chronologically as described, they may also be performed in parallel or in a different order, depending on the processing power of the device performing each step or as necessary. good. Other changes are possible without departing from the spirit of the present disclosure.

10 Image processing system 11 Image processing device 12 Cable 13 Drive unit 14 Keyboard 15 Mouse 16 Display 17 Connection terminal 18 Cart unit 20 Probe 21 Drive shaft 22 Hub 23 Sheath 24 Outer tube 25 Ultrasonic transducer 26 Relay connector 31 Scanner unit 32 Slide Unit 33 Bottom cover 34 Probe connection section 35 Scanner motor 36 Inlet 37 Probe clamp section 38 Slide motor 39 Switch group 41 Control section 42 Storage section 43 Communication section 44 Input section 45 Output section 51 Fault data 52 3D data 53 3D Image 54 Object 55 Plane 56 Corresponding range 57 Three-

dimensional area

58, 58a, 58b Two-dimensional image 60 Biological tissue 61 Inner surface 62 Cutting area 63 Lumen 64

Cross section

65, 65a, 65b Area 66 Oval fossa 71 Camera 80 Screen 81 Operation Panel 82 Checkbox 83 Slider 84 Slider 85 Checkbox 86

Pointer

87, 88 Mark 89 Range

Claims

Based on the positional relationship between a viewpoint set in a virtual three-dimensional space and an object of the biological tissue placed in the three-dimensional space, rendering of the object is performed, and the object is converted into a three-dimensional object of the biological tissue. An image processing device that displays an image on a screen,
In response to a position specification operation that specifies two positions on the screen, a first corresponding point corresponding to one of the two specified positions on a plane corresponding to the screen in the three-dimensional space, and a first corresponding point corresponding to the other of the two specified positions. A corresponding second corresponding point is identified, and a first intersection point is an intersection between the object and an extension of a straight line connecting the viewpoint and the first corresponding point in the three-dimensional space, and An image processing device including a control unit that calculates a distance between a second intersection point, which is an intersection point between an extension of a straight line connecting the viewpoint and the second corresponding point and the object, and outputs the obtained calculation result.
The image processing device according to claim 1, wherein the control unit outputs a numerical value representing the distance on the screen as the calculation result.
The position specifying operation includes an operation of pressing a push button of an input device as a first operation,
3. The control unit specifies, as the first corresponding point, a point on the plane that corresponds to the position of the pointer on the screen when the first operation is performed. Image processing device.
The image processing apparatus according to claim 3, wherein the first operation is an operation of pressing the push button while pressing a predetermined first key.
The position specifying operation includes, as a second operation, an operation of releasing the push button, which is performed following the first operation and a drag operation of moving the pointer while holding down the push button,
The image processing device according to claim 3 or 4, wherein the control unit specifies a point on the plane corresponding to the position of the pointer when the second operation is performed as the second corresponding point. .
The image processing apparatus according to claim 5, wherein the second operation is an operation of releasing the push button while pressing a predetermined second key.
The control unit is configured to control a first corresponding position on the screen corresponding to an intersection between the plane and a straight line connecting the viewpoint and the first intersection in the three-dimensional space, and The image processing device according to any one of claims 1 to 6, wherein a mark is displayed at each of the second corresponding positions corresponding to the intersection of the plane and a straight line connecting the plane and the second intersection.
In response to a range specification operation that specifies a range on the screen, the control unit specifies a corresponding range on the plane corresponding to the specified range, and connects the first intersection point and the first intersection on the screen. Among the two intersection points, the appearance of a mark displayed at a position corresponding to an intersection point that exists in a three-dimensional area extending conically from the viewpoint through the outer edge of the corresponding range in the three-dimensional space is changed. The image processing device according to claim 7.
The image processing device according to claim 8, wherein the control unit accepts an operation to collectively delete marks whose appearance has been changed.
An image processing device according to any one of claims 1 to 9,
An image processing system comprising: a display that displays the screen.
Based on the positional relationship between a viewpoint set in a virtual three-dimensional space and an object of the biological tissue placed in the three-dimensional space, rendering of the object is performed, and the object is converted into a three-dimensional object of the biological tissue. An image display method for displaying an image on a screen,
In response to a position specification operation that specifies two positions on the screen, a first corresponding point corresponding to one of the two specified positions on a plane corresponding to the screen in the three-dimensional space, and a first corresponding point corresponding to the other of the two specified positions. Identify the corresponding second corresponding point,
A first intersection point, which is an intersection between the object and an extension of a straight line connecting the viewpoint and the first corresponding point in the three-dimensional space, and a first intersection point between the viewpoint and the second corresponding point in the three-dimensional space. Calculate the distance between the second intersection point, which is the intersection of the connecting straight line extension line and the object,
An image display method that outputs the obtained calculation results.
Based on the positional relationship between a viewpoint set in a virtual three-dimensional space and an object of the biological tissue placed in the three-dimensional space, rendering of the object is performed, and the object is converted into a three-dimensional object of the biological tissue. to the computer that displays it on the screen as an image,
In response to a position specification operation that specifies two positions on the screen, a first corresponding point corresponding to one of the two specified positions on a plane corresponding to the screen in the three-dimensional space, and a first corresponding point corresponding to the other of the two specified positions. a process of identifying a corresponding second corresponding point;
A first intersection point, which is an intersection between the object and an extension of a straight line connecting the viewpoint and the first corresponding point in the three-dimensional space, and a first intersection point between the viewpoint and the second corresponding point in the three-dimensional space. a process of calculating a distance between a second intersection point that is an intersection point between an extension line of the connecting straight line and the object;
An image processing program that executes processing to output the obtained calculation results.