JP2023036499A - Image processing apparatus, image processing method, and recording medium - Google Patents

Abstract

To provide a graphical user interface (GUI) which allows an inner structure of an object including a living body to be easily recognized.SOLUTION: An image processing device comprises: an input reception unit which is configured to receive change of at least one of the first depth and the second depth according to the operation content of a pointing device when the operation position of the pointing device is on a profile image; an inner structure image output unit which is configured to cause a display device to re-display an inner structure image by reconstructing the inner structure image obtained by changing a chromatic parameter within a depth range after the change received by the input reception unit in the same manner as the change mode of the chromatic parameter before the change from the first depth to the second depth after the change on the basis of volume data showing a three-dimensional inner structure of the object; and a profile image output unit which is configured to cause the display device to re-display the profile image by reconstructing the profile image in accordance with the first depth and the second depth after the change received by the input reception unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device, an image processing method, and a recording medium.

2. Description of the Related Art An image generation apparatus has been developed that reconstructs an image based on signal data of acoustic waves obtained by measuring a predetermined test site of a subject.

For example, in the medical field, research is actively progressing on an image generating apparatus that irradiates a subject such as a living body with light from a light source (such as a laser) and visualizes information inside the subject. Photoacoustic tomography (PAT: photoacoustic tomography, sometimes referred to as photoacoustic tomography) is one of such optical visualization techniques. In an imaging device using photoacoustic tomography, the irradiated light propagates inside the subject and absorbs the energy of the diffused light. ) are detected at a plurality of locations around the subject. Then, the obtained signal is mathematically analyzed and processed to visualize the optical characteristic value inside the object, especially the information related to the absorption coefficient distribution. In recent years, non-clinical research on imaging blood vessels in small animals using this photoacoustic tomography system and clinical research on applying this principle to image diagnosis for breast cancer and preoperative planning in the field of plastic surgery have been actively conducted. is underway.

Patent Literature 1 discloses a technique that enables estimation of an interface of a subject using image data. Further, Patent Literature 2 discloses a technique for improving the accuracy of classification of superficial blood vessels and body hair included in an internal structure image reconstructed from photoacoustic signals.

WO2018/110558 JP 2019-25251 A

SUMMARY OF THE INVENTION It is an object of the present invention to provide a graphical user interface (GUI) that facilitates recognition of the internal structure of objects, including living organisms.

According to one aspect of the invention,
From the first depth to the second depth within a depth range from a first depth to a second depth from a predetermined reference plane, based on volume data indicating the three-dimensional internal structure of an object. an internal structure image output unit that constructs an internal structure image in which the color parameter is continuously changed and displays it on a display device;
a profile image output unit that causes the display device to display a profile image showing the correspondence relationship between the depth from the reference plane and the color parameter;
an input reception unit that receives an operation of the pointing device,
The input accepting unit accepts a change of at least one of the first depth and the second depth according to an operation content of the pointing device when an operation position of the pointing device is on the profile image. configured as
The internal structure image output unit converts the color parameter within the changed depth range accepted by the input accepting unit to the color parameter before change from the first depth after change to the second depth. configured to reconstruct the internal structure image changed in the same manner as the change mode of and re-display it on the display device,
The profile image output unit is configured to reconstruct the profile image according to the changed first depth and the second depth accepted by the input acceptance unit and cause the display device to re-display the profile image. An image processing apparatus is provided.

ADVANTAGE OF THE INVENTION According to this invention, the graphical user interface (GUI) which makes it easy to recognize the internal structure of the object containing a living body can be provided.

FIG. 1 is a schematic configuration diagram of an image processing apparatus 10 according to the first embodiment of the invention. FIG. 2 is a diagram showing an example of an internal structure image 30 and a profile image 40 displayed on the display device 20 according to the first embodiment of the invention. FIG. 3 is a schematic configuration diagram of an image processing apparatus 10 according to Modification 1 of the first embodiment of the present invention. FIG. 4 shows an example of an internal structure image 30, a profile image 40, a first slider image 60, and a second slider image 70 displayed on the display device 20 according to Modification 1 of the first embodiment of the present invention. It is a diagram. FIG. 5A is a diagram showing an example of the internal structure image 30 and the profile image 40 before changing the depth range according to the operational example of the graphical interface of the present invention. FIG. 5B is a diagram showing an example of the internal structure image 30 and the profile image 40 after changing the depth range according to the operational example of the graphical interface of the present invention. FIG. 5C is a diagram showing another example of the internal structure image 30 and the profile image 40 after changing the depth range according to the operational example of the graphical interface of the present invention. FIG. 5D is a diagram showing another example of the internal structure image 30 and the profile image 40 after changing the depth range according to the operational case of the graphical interface of the present invention. FIG. 6 is a flow chart showing an outline of an image processing method by the image processing apparatus 10 according to the first embodiment of the invention. FIG. 7 is a schematic configuration diagram of an image processing apparatus 10 according to Modification 2 of the first embodiment of the present invention.

<First embodiment of the present invention>
A first embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to these exemplifications, but is indicated by the scope of the claims, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

(1) Configuration of Image Processing Apparatus 10 The image processing apparatus 10 according to the present embodiment, for example, is configured to allow a user to easily recognize the internal structure of a living tissue of a subject (subject). is configured to process volume data indicative of a typical internal structure. FIG. 1 is a schematic configuration diagram of an image processing apparatus 10 according to this embodiment. As shown in FIG. 1, the image processing apparatus 10 includes, for example, an internal structure image output section 11, a profile image output section 12, and an input reception section 13. As shown in FIG. 1, the internal structure image output unit 11 and the profile image output unit 12 include, for example, a data storage unit (SSD/HDD) 101, a general-purpose computing unit (CPU) 102, and an image computing unit (GPU). 103 , image processing storage unit (VRAM) 104 , general-purpose processing storage unit (RAM) 105 , and communication unit 106 . In this embodiment, volume data is obtained by image reconstruction of signal data of photoacoustic waves generated by light irradiation of a living body. A user means a person who operates the image processing apparatus 10 .

The SSD/HDD 101 is configured to store data such as programs related to photoacoustic wave measurement, for example. The CPU 102 is configured to control each section of the image processing apparatus 10 by executing a predetermined program stored in the SSD/HDD 101, for example. The GPU 103 is configured, for example, to cooperate with the CPU 102 to execute programs related to image processing. The VRAM 104 is configured to temporarily hold information, programs, and the like processed by the GPU 103, for example. The RAM 105 has, for example, a volume data memory (area) 107, an internal structure image memory (area) 108, a profile image memory (area) 109, etc., and temporarily holds information and programs processed by the CPU 102. is configured as The communication unit 106 is configured, for example, to acquire information necessary for image processing from the network file server 110 . The network file server 110 is configured, for example, to store imaging data captured by the photoacoustic wave imaging device 111 .

The internal structure image output unit 11 generates an image at a depth from a first depth to a second depth from a predetermined reference plane based on volume data representing the three-dimensional internal structure of an object (biological tissue in this embodiment). Within the range, an internal structure image 30 in which the color parameter is continuously changed from the first depth to the second depth is constructed and displayed on the display device 20 . Further, the internal structure image output unit 11 sets the color parameter of the internal structure image 30 constructed based on the volume data within the range below the first depth to be the same as the color parameter at the first depth, and the color parameter at the second depth. The color parameters of the internal structure image 30 constructed based on the volume data within the range of 0.5 are configured to form the internal structure image 30 in the same manner as the color parameters at the second depth, and are configured to be displayed on the display device 20. It is

A predetermined reference plane means a plane of a predetermined reference height or a continuous plane that defines the interface of the subject. The internal structure image output unit 11 switches the predetermined reference plane to a plane of a predetermined reference height or a continuous plane that defines the interface of the subject, reconstructs the internal structure image 30, and displays it on the display device 20. It is preferably configured to be displayable. In addition, when obtaining a continuous plane that defines the interface of the object, a known method such as cross simulation described in Patent Document 1 can be used, for example.

The first depth means, for example, the lower limit (shallowest position) of the depth range, and the second depth means, for example, the upper limit (deepest position) of the depth range.

Volume data is three-dimensional data as a collection of voxels, and a voxel is a unit area having physical characteristic information such as density, mass, and oxygen saturation of a unit spatial area of a subject as predetermined physical parameters. means that That is, each voxel is composed of coordinate values and physical parameters, and each voxel in the volume data corresponds to the unit space area of the subject. Also, the internal structure image 30 means a two-dimensional image consisting of a set of pixels for display.

A color parameter is a parameter including at least one attribute of hue, saturation, and lightness. The color parameter of each pixel in the internal structure image 30 changes continuously from red (light) to blue (deep), for example, according to the depth from the reference plane of each voxel corresponding to each pixel. . The internal structure image output unit 11 can switch which of the attributes (hue, saturation, and brightness) included in the hue parameter is to be changed continuously according to the depth from the reference surface. is preferably configured to

The internal structure image 30 is an image for two-dimensional display constructed based on volume data representing the internal structure of a living tissue. In this embodiment, the case where the internal structure image 30 is a photoacoustic image configured based on signal data of photoacoustic waves will be described.

The display device 20 is, for example, connected to the image processing device 10 and configured to display an internal structure image 30 and the like by executing a predetermined program. The display device 20 is a two-dimensional display device and may be capable of stereoscopic viewing. Specifically, the display device 20 is a liquid crystal display, an organic EL (OLED) display, a head-mounted display, a direct-view stereoscopic display, or the like. Note that the display device 20 may be a part of the image processing device 10 .

The profile image output unit 12 is configured to cause the display device 20 to display the profile image 40 indicating the correspondence relationship between the depth from the reference plane and the color parameters together with the internal structure image 30 .

FIG. 2 is a diagram showing an example of an internal structure image 30 and a profile image 40 displayed on the display device 20 according to this embodiment. As shown in FIG. 2, the profile image 40 has a color bar 41 and an index 42 in this embodiment. The color bar 41 is, for example, a vertically long bar-shaped image, and the color parameters change from the top end to the bottom end in the same manner as the internal structure image 30 . In the color bar 41, for example, the lower end corresponds to the first depth and the upper end corresponds to the second depth. The index 42 is, for example, a group of numerical values adjacent to the color bar 41, and indicates the correspondence between the color parameters of the color bar 41 and the depth from the reference plane. For example, in the numerical value group of the index 42, the lowest numerical value indicates the first depth, and the highest numerical value indicates the second depth. By checking the internal structure image 30 and the profile image 40 displayed on the display device 20, the user can confirm that the region in which the user is interested in the internal structure image 30 (hereinafter also referred to as the region of interest) is the reference It is possible to recognize how deep it is positioned from the surface. Note that, in the image processing apparatus 10 of the present embodiment, there are two ways to set the reference plane having a depth of zero. One is how to set the skin surface position of the subject to be zero. The skin surface position can be obtained, for example, by cross simulation processing described in Patent Document 1. Another is, as also shown in Patent Document 1, when a plurality of sensors are arranged in a bowl shape so as to surround the subject, the curvature center position of the sensor is set to z = zero with respect to the central axis (z axis) of the bowl. It is the method of setting to do. When this criterion is used, the height of the shallowest detection surface is approximately -(minus) 10 mm, and the photoacoustic wave sensor of this embodiment has the highest resolution in the range from -10 mm to +10 mm.

The input reception unit 13 is configured to receive an operation of the pointing device 14 . In this specification, the pointing device 14 means, for example, an input device for operating the pointer 50 displayed on the display device 20, and specifically includes a mouse, a trackball, a touch panel (by user's touch input). a touch sensor that detects position designation, dragging, and the like).

(2) Graphical Interface Provided by Image Processing Apparatus 10 The image processing apparatus 10 according to the present embodiment includes a graphical user interface (GUI) that facilitates recognition of the internal structure of living tissue. For example, the user operates the pointer 50 displayed on the display device 20 using the pointing device 14, and the image processing device 10 is configured to perform various processes according to the operation position and operation content of the pointing device 14. It is A part of the functions of the GUI included in the image processing apparatus 10 according to the present embodiment will be described below. In this specification, the operation content of the pointing device 14 means various operations such as clicking, dragging, wheel rotation, etc., and the operation position of the pointing device 14 is the pointer at which the above operation is started. means 50 positions.

(2-1) Image movement function According to the GUI provided in the image processing apparatus 10 according to the present embodiment, the internal structure image 30 and the profile image 40 displayed on the display device 20 can be moved to any arbitrary position on the display device 20. It is possible to move to each position.

When the operation position of the pointing device 14 accepted by the input accepting unit 13 is on the internal structure image 30, the internal structure image output unit 11 outputs the internal structure according to the operation content of the pointing device 14 accepted by the input accepting unit 13. It is configured to display the structural image 30 in a movable manner. Specifically, the internal structure image output unit 11 is configured, for example, to move the internal structure image 30 in the direction in which the pointer 50 moves when a drag operation is performed on the internal structure image 30. Just do it.

When the operation position of the pointing device 14 accepted by the input accepting unit 13 is on the profile image 40, the profile image output unit 12 outputs the profile image 40 according to the operation content of the pointing device 14 accepted by the input accepting unit 13. is configured to be displayed in a movable manner. Specifically, the profile image output unit 12 may be configured to move the profile image 40 in the direction in which the pointer 50 moves when a drag operation is performed on the profile image 40, for example. Since the user can move the profile image 40 to any position as necessary, for example, it becomes easier to recognize how deep the region of interest is positioned from the reference plane. Note that, in the present embodiment, that the operation position of the pointing device 14 is on the profile image 40 means, for example, that the operation of the pointing device 14 is started while the pointer 50 is on the color bar 41 (or on the index 42). means to

(2-2) Image Enlarging/Reducing Function According to the GUI included in the image processing device 10 according to the present embodiment, it is possible to enlarge or reduce the internal structure image 30 displayed on the display device 20 .

When the operation position of the pointing device 14 accepted by the input accepting unit 13 is on the internal structure image 30, the internal structure image output unit 11 outputs the internal structure according to the operation content of the pointing device 14 accepted by the input accepting unit 13. The structure image 30 is configured to be displayed in a mode in which at least one of enlargement and reduction is possible. Specifically, for example, the internal structure image output unit 11 enlarges the internal structure image 30 when the wheel operation is performed in one direction (for example, upward direction) on the internal structure image 30, and enlarges the internal structure image 30 in the other direction. It is sufficient that the internal structure image 30 is reduced when the wheel is operated in a direction (for example, downward).

(2-3) Depth Range Change Function According to the GUI provided in the image processing apparatus 10 according to the present embodiment, at least one of the first depth and the second depth is changed and displayed on the display device 20. It is possible to redisplay the internal structure image 30 and the profile image 40, respectively.

When the operation position of the pointing device 14 is on the profile image 40, the input reception unit 13 receives a change of at least one of the first depth and the second depth according to the operation content of the pointing device 14. It is configured.

The internal structure image output unit 11 converts the color parameters of each pixel within the changed depth range accepted by the input accepting unit 13 from the first depth after change to the second depth, to the color parameters before change. It is configured to reconstruct the internal structure image 30 that has been changed in the same manner as the change mode and to display it again on the display device 20 . In this specification, the “change mode” of the color parameter indicates which of the attributes included in the hue parameter is continuously changing. Also, changing the color parameter in the same manner as the color parameter before change means that the color parameter before change changes from red to blue from the first depth to the second depth, for example. , it means that the changed color parameter does not change from red to blue.

The profile image output unit 12 is configured to reconstruct the profile image 40 according to the changed first depth and second depth accepted by the input acceptance unit 13 and cause the display device 20 to re-display it. there is As described above, since the color parameter change mode does not change before and after changing the depth range, when reconstructing the profile image 40, the image of the color bar 41 can be left as it is, and only the index 42 needs to be changed. good. By a simple operation of operating the pointing device 14, the user can adjust the information in the depth direction of the region of interest so that it can be visually recognized more easily.

(2-4) First Depth (or Second Depth) Change Function The depth range change function described in (2-3) can be further classified. For example, according to the GUI included in the image processing apparatus 10 according to the present embodiment, only the first depth (or the second depth) is changed, and the internal structure image 30 and the profile image 40 displayed on the display device 20 and can be displayed again.

When the operation position of the pointing device 14 is on the first region of the profile image 40, the input reception unit 13 selects either one of the first depth and the second depth according to the operation content of the pointing device 14. Configured to accept changes. Here, the first region of the profile image 40 means, for example, the lower portion of the color bar 41 (for example, 1/4 length from the lower end). Specifically, for example, when the wheel is operated in one direction (for example, upward) at the bottom of the color bar 41, the input reception unit 13 increases only the first depth, and increases the depth in the other direction. It only needs to be configured to accept a change that decreases only the first depth when the wheel is operated (for example, downward).

When the operation position of the pointing device 14 is on the second region of the profile image 40, the input reception unit 13 selects one of the first depth and the second depth according to the operation content of the pointing device 14. It is configured to accept changes from the other that are different from the Here, the second region of the profile image 40 means, for example, the upper portion of the color bar 41 (for example, 1/4 length from the upper end). Specifically, for example, when the wheel is operated in one direction (for example, upward) above the color bar 41, the input reception unit 13 increases only the second depth, and increases the depth in the other direction. It only needs to be configured to accept a change that decreases only the second depth when the wheel is operated (for example, downward).

(2-5) Depth range translation function According to the GUI included in the image processing apparatus 10 according to the present embodiment, the first depth and the second depth are changed so as to translate the depth range. , the internal structure image 30 and the profile image 40 displayed on the display device 20 can be redisplayed.

When the operation position of the pointing device 14 is on the third area of the profile image 40, the input reception unit 13 determines the depth range without changing the size of the depth range according to the operation content of the pointing device 14. is configured to accept changes that move the to at least one of the deeper and shallower directions. Here, the third region of the profile image 40 means, for example, the central portion of the color bar 41 (for example, the vertical 1/4 length from the center). Specifically, for example, when the wheel is operated in one direction (for example, upward) at the center of the color bar 41, the input reception unit 13 sets the first depth and the second depth to the same value. It is only necessary to accept a change that increases the depth by a value and decreases the first depth and the second depth by the same value when the wheel is operated in the other direction (for example, downward).

(2-6) Depth range enlargement/reduction function According to the GUI provided in the image processing apparatus 10 according to the present embodiment, the first depth and the second depth are changed so as to enlarge or reduce the depth range. Then, the internal structure image 30 and the profile image 40 displayed on the display device 20 can be displayed again. When there is no change in each color parameter at the first depth and the second depth, expanding the depth range reduces the change in color parameter per unit depth, enabling inspection in a wide depth range. . On the other hand, similarly, when each color parameter does not change, if the depth range is reduced, the change in color parameter per unit depth increases, and the difference in depth can be read in detail in a narrow depth range. .

When the operation position of the pointing device 14 is on the profile image 40, the input reception unit 13 enlarges or reduces the depth range without changing the center of the depth range according to the operation content of the pointing device 14. configured to accept changes that enable at least one of them. Specifically, for example, the input reception unit 13 increases the first depth when a wheel operation is performed in one direction (for example, upward) while performing a right-click operation on the profile image 40. and decrease the depth range by decreasing the second depth by the same amount as the increase in the first depth, and when the wheel is operated in the other direction (e.g. downward), the first depth It is only necessary to accept a change that decreases the depth and increases the second depth by the same amount as the decrease in the first depth to expand the depth range.

(3) Effects obtained by this embodiment According to this embodiment, one or more of the following effects can be obtained.

(a) According to the GUI included in the image processing apparatus 10 according to the present embodiment, the internal structure image 30 and the profile image 40 displayed on the display device 20 are moved to arbitrary positions on the display device 20. Is possible. With this function, the user can move the profile image 40 to an arbitrary position as necessary, so that, for example, it becomes easier to recognize how deep the region of interest is positioned from the reference plane. .

(b) According to the GUI included in the image processing device 10 according to this embodiment, it is possible to enlarge or reduce the internal structure image 30 displayed on the display device 20 . With this function, the user can enlarge or reduce the internal structure image 30 as necessary, so that, for example, it becomes easier to recognize how deep the region of interest is positioned from the reference plane.

(c) According to the GUI included in the image processing apparatus 10 according to the present embodiment, at least one of the first depth and the second depth is changed, and the internal structure image 30 and the profile image displayed on the display device 20 40 and , can be displayed again. With this function, the user can adjust the information in the depth direction of the region of interest so that it can be visually recognized more easily by a simple operation of operating the pointing device 14 .

(d) According to the GUI included in the image processing apparatus 10 according to the present embodiment, only the first depth (or the second depth) is changed, and the internal structure image 30 and the profile image displayed on the display device 20 40 and , can be displayed again. With this function, the user can adjust the information in the depth direction of the region of interest so that it can be visually recognized more easily by a simple operation of operating the pointing device 14 .

(e) According to the GUI included in the image processing apparatus 10 according to the present embodiment, the first depth and the second depth are changed so as to translate the depth range, and the internal image displayed on the display device 20 is changed. It is possible to redisplay the structural image 30 and the profile image 40, respectively. With this function, the user can adjust the information in the depth direction of the region of interest so that it can be visually recognized more easily by a simple operation of operating the pointing device 14 .

(f) According to the GUI included in the image processing apparatus 10 according to the present embodiment, the first depth and the second depth are changed so as to expand or contract the depth range, and are displayed on the display device 20. It is possible to redisplay the internal structure image 30 and the profile image 40, respectively. With this function, the user can adjust the information in the depth direction of the region of interest so that it can be visually recognized more easily by a simple operation of operating the pointing device 14 .

(g) In the present embodiment, volume data representing the internal structure of living tissue is obtained by image reconstruction of signal data of photoacoustic waves generated by light irradiation of the living body. In photoacoustic tomography, for example, only blood vessels can be visualized, and the internal structure image 30 can be obtained by simultaneously displaying front and back vessels, so the present invention can be applied particularly effectively.

(4-1) Modification 1 of the first embodiment
The above-described embodiment can be modified as in the following modified examples, if necessary. Hereinafter, only elements different from the above-described embodiment will be described, and elements that are substantially the same as those described in the above-described embodiment will be given the same reference numerals, and descriptions thereof will be omitted.

FIG. 3 is a schematic configuration diagram of an image processing apparatus 10 according to this modification. As shown in FIG. 3, the image processing apparatus 10 according to this modification may further include a slider image output unit 15, for example. The slider image output section 15 may be part of a computer having the CPU 102 and the like, similar to the internal structure image output section 11 and the profile image output section 12 . For example, the slider image output unit 15 outputs a depth range in which the color parameter continuously changes for the entire depth range (also referred to as an imaging range) possessed by three-dimensional volume data obtained by imaging. A first slider image 60 indicating the position and size of (from the first depth to the second depth) and an internal structure image displayed on the display device 20 for the entire depth range of the volume data. A second slider image 70 indicating the depth position and size of the depth range of the display range (depth direction) of 30 is displayed on the display device 20 together with the internal structure image 30 and the profile image 40. there is

FIG. 4 is a diagram showing an example of an internal structure image 30, a profile image 40, a first slider image 60, and a second slider image 70 displayed on the display device 20 according to this modification. As shown in FIG. 4, the first slider image 60 is, for example, a horizontally long bar-shaped image and has a first slider bar 61 within the image. The second slider image 70 is, for example, a horizontally long bar-shaped image, and has a second slider bar 71 within the image. In the first slider image 60 and the second slider image 70, one end (for example, the left end) indicates the lower limit (shallowest position) of the depth range of the volume data, and the other end (for example, the right end) indicates the volume data. The upper limit (deepest position) of the depth range possessed is shown. One end (eg, left end) of the first slider bar 61 corresponds to the first depth, and the other end (eg, right end) corresponds to the second depth. In the second slider bar 71, one end (eg, left end) corresponds to the lower limit of the display range of the internal structure image 30, and the other end (eg, right end) corresponds to the upper limit of the display range of the internal structure image 30. are doing.

The slider image output unit 15 is configured to reconstruct the first slider image 60 according to the changed first depth and second depth received by the input receiving unit 13 and cause the display device 20 to re-display the first slider image 60 . It is Specifically, the slider image output unit 15 may be configured to change the position and size of the first slider bar 61 according to the changed first depth and second depth. By checking the first slider image 60 as necessary, the user can determine where the depth range in which the color parameter is continuously changing is located in the entire depth range of the volume data. It becomes easier to recognize whether

The first slider image 60, like the profile image 40, can be used as part of a GUI having a depth range changing function. In this case, when the operation position of the pointing device 14 is on the first slider image 60, the input reception unit 13 selects at least one of the first depth and the second depth according to the operation content of the pointing device 14. It only needs to be configured to accept changes. However, from the viewpoint that the depth range can be changed while visually confirming the correspondence between the depth from the reference surface and the color parameter, the profile Manipulating the pointing device 14 over the image 40 is preferred.

The second slider image 70 can be used as part of a GUI that has a function of changing the display range of the internal structure image 30 . In this case, when the operation position of the pointing device 14 is on the second slider image 70, the input receiving unit 13 determines at least the lower limit and the upper limit of the display range of the internal structure image 30 according to the operation content of the pointing device 14. It suffices if it is configured to accept one change. In addition, the internal structure image output unit 11 is configured to reconstruct the internal structure image 30 according to the changed display range of the internal structure image 30 received by the input receiving unit 13 and cause the display device 20 to display it again. It is good if it is. Further, the slider image output unit 15 is configured to reconstruct the second slider image 70 according to the changed display range of the internal structure image 30 accepted by the input acceptance unit 13 and cause the display device 20 to display the second slider image 70 again. It is good if it is. Specifically, the slider image output unit 15 may be configured to change the position and size of the second slider bar 71 according to the changed display range of the internal structure image 30 . By changing the display range of the internal structure image 30 as necessary, the user can, for example, hide the voxels of the unnecessary region displayed in front to make it easier to recognize the information of the region of interest. can.

(4-2) Modification 2 of the first embodiment
FIG. 7 is a schematic configuration diagram of an image processing apparatus 10 according to this modification. As shown in FIG. 7, the image processing apparatus 10 of this modified example includes, for example, an image processor 100, a display device 20, an input reception unit 13, and a pointing device . The image processor 100 includes, for example, an internal structure image output unit 11, a profile image output unit 12, a data storage unit (SSD/HDD) 101, and a communication unit . The internal structure image output unit 11 is composed of, for example, an image calculation unit (GPU) 103, an image processing storage unit (VRAM) 104, and a volume data area 107 and an internal structure image area 108 in the VRAM 104. . The profile image output unit 12 includes, for example, a general-purpose computing unit (CPU) 102 , a general-purpose processing storage unit (RAM) 105 , a VRAM 104 , and a profile image area 109 within the VRAM 104 . Volume data captured by the photoacoustic imaging device 111 is stored in the network file server 110 . When the user wants to observe the volume data, the volume data is stored in the volume data area 107 in the VRAM 104 via the communication section 106, the data storage section 101 and the CPU 102. FIG. A process for converting volume data into an internal structure image and a user's instruction via the input reception unit 13 are sent to the GPU 103 via the CPU 102, and the internal structure image is processed according to the instruction. 108. The information of the profile image 40 is written directly into the profile image area 109 in the VRAM 104 by the CPU 102 . The contents of internal structure image area 108 and profile image area 109 are displayed on display device 20 .

The image processor 100 comprises components described below. The data storage unit 101 is configured to store data such as a program related to photoacoustic wave measurement, for example. The CPU 102 is configured to control each section of the image processing apparatus 10 by executing a predetermined program stored in the data storage section 101, for example. The GPU 103 is configured, for example, to cooperate with the CPU 102 to execute programs related to image processing. The VRAM 104 is a memory for displaying image information processed by the GPU 103 and CPU 102 on the display device 20, for example. The VRAM 104 is also used as a working memory for image processing in the GPU 103 . A RAM 105 is a working memory for the CPU 102, and is configured to temporarily hold information, programs, and the like processed by the CPU 102, for example. The communication unit 106 is configured, for example, to acquire information necessary for image processing from the network file server 110 and store it in the data storage unit 101 . The network file server 110 is configured, for example, to store imaging data captured by the photoacoustic wave imaging device 111 .

(5) Operation example of graphical interface Next, an operation example of the graphical interface according to the present invention will be described. The following operation examples are examples of the present invention, and the present invention is not limited by these operation examples.

FIG. 5A is a diagram showing an example of the internal structure image 30 and the profile image 40 before changing the depth range according to this operation example. In this operation example, the internal structure image 30 is a photoacoustic image showing the internal blood vessels of the subject's leg. The reference plane is an example of a continuous plane defined as a reference position near the skin surface of the subject by the cross simulation described above.

As shown in FIG. 5A, before changing the depth range, the first depth was 0 mm and the second depth was 4 mm. By confirming the color parameters of the internal structure image 30 and the profile image 40 in association with each other, the user can, for example, determine which blood vessel exists in the region of interest within the depth range from the first depth to the second depth. , and how deep it is positioned from the reference plane.

For example, if the region of interest is in a deeper range (for example, about 5 mm), the user operates the pointing device 14 to move the pointer 50 to the center of the color bar 41 and operate the wheel upward. good. By the above operation, the first depth and the second depth can be increased by the same value, and the depth range can be changed (translated). FIG. 5B shows the internal structure image 30 and the profile image 40 after changing the depth range by the above operation. As shown in FIG. 5B, after changing the depth range, the first depth was 2 mm and the second depth was 6 mm. By confirming the re-displayed color parameters of the internal structure image 30 and the profile image 40 in correspondence, the user can, for example, see how far the blood vessels existing in the region of interest in the deeper range are from the reference plane. It can be recognized that it is located at depth.

However, in the example shown in FIG. 5B, since the depth range has been translated, for example, for a blood vessel existing in a depth range of 0 mm or more and 2 mm or less from the reference plane, the depth of each voxel constituting the blood vessel region is shown. The change in the color parameter in the internal structure image 30 is lost and is constant (same value as the first depth after change). Therefore, it is difficult for the user to recognize how deep the blood vessels existing in this range are positioned from the reference plane. For example, when the user wants to maintain the color parameter change in a shallow range while making the region of interest a deeper range, the user operates the pointing device 14 to move the pointer 50 to the upper part of the color bar 41 and move the wheel upward. operation can be performed. By the above operation, it is possible to change the depth range by increasing only the second depth. FIG. 5C shows the internal structure image 30 and the profile image 40 after changing the depth range by the above operation. As shown in FIG. 5C, after changing the depth range, the first depth was 0 mm and the second depth was 6 mm. By confirming the redisplayed color parameters of the internal structure image 30 and the profile image 40 in association with each other, the user can see, for example, the blood vessels existing in the region of interest in a deeper range and the depth before the change. It is possible to recognize how deep the blood vessels existing in the range are positioned from the reference plane.

Further, for example, when the region of interest has a narrower depth range, the user operates the pointing device 14 to move the pointer 50 onto the color bar, right-click, and rotate the wheel upward. Do it. By the above operation, the depth range can be changed (reduced) by increasing the first depth and decreasing the second depth by the same amount as the increase in the first depth. FIG. 5D shows the internal structure image 30 and the profile image 40 after changing the depth range by the above operation. As shown in FIG. 5D, after changing the depth range, the first depth was 0.8 mm and the second depth was 2.4 mm. By confirming the re-displayed color parameters of the internal structure image 30 and the profile image 40 in correspondence, the user can, for example, see how the blood vessels existing in the region of interest in a narrower depth range are located from the reference plane. It is possible to recognize in more detail whether it is located at a certain depth.

As described above, it has been confirmed that the user can adjust the information of the biological tissue in the depth direction so that the information can be visually recognized more easily by using the GUI included in the image processing apparatus 10 .

(6) Image Processing Method by Image Processing Apparatus 10 Next, an image processing method by the image processing apparatus 10 will be described. FIG. 6 is a flow chart showing an outline of an image processing method by the image processing apparatus 10 according to the first embodiment of the invention. As shown in FIG. 6, the image processing method of this embodiment includes, for example, a volume data acquisition step S100, a depth information and color parameter acquisition step S110, an internal structure image display step S120, and a profile image display step S130. , an input determination step S140, an operation position determination step S150, and a processing step S160.

In the volume data acquisition step S100, for example, the photoacoustic wave imaging device 111 is activated, photoacoustic wave imaging is performed, and volume data indicating the three-dimensional internal structure of an object (eg, biological tissue) is obtained.

In the depth information and color parameter acquisition step S110, for example, information on a first depth and a second depth from a predetermined reference plane, and volume data within a depth range from the first depth to the second depth are obtained. The color parameters of the originally constructed internal structure image 30 are obtained.

In the internal structure image display step S120, for example, based on the volume data and the color parameter, the internal structure image 30 is constructed by continuously changing the color parameter from the first depth to the second depth, and the display device 20 to display.

In the profile image display step S130, for example, the display device 20 is caused to display the profile image 40 showing the correspondence between the depth from the reference plane and the color parameters.

In the input determination step S140, for example, it is determined whether there is an input from the pointing device 14 or not. When there is an input from the pointing device 14, an operation position determination step S150 is performed. If there is no input from the pointing device 14, the input determination step S140 may be performed again.

In the operation position determination step S150, for example, it is determined whether the operation position of the pointing device 14 is on the profile image 40 or not. If the operation position of the pointing device 14 is on the profile image 40, for example, the depth information and color parameter acquisition step S110 may be performed again. If the operation position of the pointing device 14 is not on the profile image 40, processing step S160 may be performed.

In processing step S160, other processing is performed according to the operation content of the pointing device 14. FIG.

Through the above steps, image processing can be performed to facilitate recognition of the internal structure of the living tissue. The present invention can also be applied to a program that causes a computer to execute the above-described steps (procedures), and a computer-readable recording medium that records the program.

<Other embodiments of the present invention>
The embodiments of the present invention have been specifically described above. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

For example, in the above-described embodiment, the volume data representing the internal structure of the living tissue includes signal data of photoacoustic waves generated by light irradiation of the living body, and the internal structure image 30 is based on the signal data of the photoacoustic waves. Although the case of the composed photoacoustic image has been described, the internal structure image 30 is not limited to the photoacoustic image. The internal structure image 30 may be, for example, an ultrasound image constructed based on data acquired by an ultrasound diagnostic imaging method, or an MRI blood vessel constructed based on data acquired by an MRI angiographic diagnostic method. It may be a contrast-enhanced image. Further, in the above-described embodiment, the case of processing volume data representing the three-dimensional internal structure of living tissue has been described. etc. can also be applied.

<Preferred embodiment of the present invention>
Preferred embodiments of the present invention are described below.

(Appendix 1)
From the first depth to the second depth within a depth range from a first depth to a second depth from a predetermined reference plane, based on volume data indicating the three-dimensional internal structure of an object. an internal structure image output unit that constructs an internal structure image in which the color parameter is continuously changed and displays it on a display device;
a profile image output unit that causes the display device to display a profile image showing the correspondence relationship between the depth from the reference plane and the color parameter;
an input reception unit that receives an operation of the pointing device,
The input accepting unit accepts a change of at least one of the first depth and the second depth according to an operation content of the pointing device when an operation position of the pointing device is on the profile image. configured as
The internal structure image output unit converts the color parameter within the changed depth range accepted by the input accepting unit to the color parameter before change from the first depth after change to the second depth. configured to reconstruct the internal structure image changed in the same manner as the change mode of and re-display it on the display device,
The profile image output unit is configured to reconstruct the profile image according to the changed first depth and the second depth accepted by the input acceptance unit and cause the display device to re-display the profile image. image processing device.

(Appendix 2)
When the operation position of the pointing device accepted by the input accepting unit is on the internal structure image, the internal structure image output unit outputs the internal structure according to the operation content of the pointing device accepted by the input accepting unit. configured to movably display the structural image;
The image according to appendix 1, wherein the profile image output unit is configured to display the profile image in a movable manner according to the operation content when the operation position is on the profile image. processing equipment.

(Appendix 3)
When the operation position of the pointing device accepted by the input accepting unit is on the internal structure image, the internal structure image output unit outputs the internal structure according to the operation content of the pointing device accepted by the input accepting unit. 3. The image processing device according to appendix 1 or appendix 2, wherein the structural image is displayed in a manner in which at least one of enlargement and reduction is possible.

(Appendix 4)
The input reception unit selects one of the first depth and the second depth according to the operation content of the pointing device when the operation position of the pointing device is on the first region of the profile image. 3. The image processing device according to any one of appendices 1 to 3, configured to accept one change.

(Appendix 5)
The input reception unit, when the operation position of the pointing device is on the second area of the profile image, selects the one of the first depth and the second depth according to the operation content of the pointing device. 5. The image processing device of appendix 4, configured to accept a change of the other that is different from the .

(Appendix 6)
The input reception unit, when the operation position of the pointing device is on the third region of the profile image, according to the operation content of the pointing device, the depth range without changing the size of the depth range. 6. The image processing device according to any one of appendices 1 to 5, wherein the image processing apparatus is configured to receive a change that moves the depth range in at least one of a deep direction and a shallow direction.

(Appendix 7)
When the operation position of the pointing device is on the profile image, the input reception unit expands or expands the depth range without changing the center of the depth range according to the operation content of the pointing device. 7. The image processing device according to any one of appendices 1 to 6, wherein the image processing apparatus is configured to accept a modification that enables at least one of the reduction.

(Appendix 8)
8. The image processing apparatus according to any one of appendices 1 to 7, wherein the volume data is obtained by image reconstruction of signal data of photoacoustic waves generated by irradiating a living body with light.

(Appendix 9)
obtaining volume data representing the three-dimensional internal structure of the object;
A color parameter of an internal structure image is obtained based on information on a first depth and a second depth from a predetermined reference plane and the volume data within the depth range from the first depth to the second depth. process and
A step of constructing the internal structure image in which the color parameter is continuously changed from the first depth to the second depth based on the volume data and the color parameter, and displaying the image on a display device;
a step of displaying, on the display device, a profile image showing a correspondence relationship between the depth from the reference plane and the color parameter;
determining pointing device input;
determining an operating position of the pointing device;
An image processing method comprising:

(Appendix 10)
a procedure for obtaining volume data representing the three-dimensional internal structure of an object;
A color parameter of an internal structure image is obtained based on information on a first depth and a second depth from a predetermined reference plane and the volume data within the depth range from the first depth to the second depth. a procedure;
a step of constructing the internal structure image in which the color parameter is continuously changed from the first depth to the second depth based on the volume data and the color parameter, and displaying the image on a display device; ,
a step of displaying on the display device a profile image showing the correspondence relationship between the depth from the reference plane and the color parameter;
a procedure for determining pointing device input;
a procedure for determining an operation position of the pointing device;
A computer-readable recording medium that records a program that causes a computer to execute

(Appendix 11)
By the internal structure image output unit, based on the volume data indicating the three-dimensional internal structure of the object, within the depth range of the first depth or more and the second depth or less from the predetermined reference plane, the first depth A step of constructing an internal structure image in which the color parameter is continuously changed from to the second depth and displaying it on a display device;
causing a profile image output unit to display, on the display device, a profile image showing a correspondence relationship between the depth from the reference plane and the color parameter;
a step of receiving an operation of the pointing device by the input receiving unit;
The input reception unit receives a change of at least one of the first depth and the second depth according to the operation content of the pointing device when the operation position of the pointing device is on the profile image. and
By the internal structure image output unit, the color parameter within the depth range after change received by the input receiving unit is changed from the first depth after change to the second depth, and the color parameter before change a step of reconstructing the internal structure image changed in the same manner as the change mode of and redisplaying it on the display device;
a step of reconstructing the profile image by the profile image output unit according to the changed first depth and the second depth received by the input receiving unit and causing the display device to display the profile image again;
An image processing method further comprising:

(Appendix 12)
to the computer,
From the first depth to the second depth within a depth range from a first depth to a second depth from a predetermined reference plane, based on volume data indicating the three-dimensional internal structure of an object. an internal structure image output unit that constructs an internal structure image in which the color parameter is continuously changed and displays it on a display device;
a profile image output unit that causes the display device to display a profile image showing the correspondence relationship between the depth from the reference plane and the color parameter;
and an input reception unit that receives the operation of the pointing device,
The input receiving unit receives a change of at least one of the first depth and the second depth in accordance with an operation content of the pointing device when an operation position of the pointing device is on the profile image. Configure as
The internal structure image output unit converts the color parameter within the depth range after change accepted by the input acceptance unit to the color parameter before change from the first depth after change to the second depth configured to reconstruct the internal structure image changed in the same manner as in the change mode of and re-display it on the display device;
The profile image output unit is configured to reconstruct the profile image according to the changed first depth and the second depth received by the input receiving unit, and to display the profile image again on the display device. A computer-readable recording medium that records a program.

10 image processing device 11 internal structure image output unit 12 profile image output unit 13 input reception unit 14 pointing device 15 slider image output unit 20 display device 30 internal structure image 40 profile image 41 color bar 42 index 50 pointer 60 first slider image 61 First slider bar 70 Second slider image 71 Second slider bar 100 Image processor 101 Data storage unit (SSD/HDD)
102 general-purpose arithmetic unit (CPU)
103 Image processing unit (GPU)
104 image processing storage unit (VRAM)
105 storage unit for general-purpose processing (RAM)
106 Communication unit 107 Volume data memory (area)
108 internal structure image memory (area)
109 profile image memory (area)
110 network file server 111 photoacoustic wave imaging device S100 volume data acquisition step S110 depth information and color parameter acquisition step S120 internal structure image display step S130 profile image display step S140 input determination step S150 operation position determination step S160 processing steps

Claims (10)

From the first depth to the second depth within a depth range from a first depth to a second depth from a predetermined reference plane, based on volume data indicating the three-dimensional internal structure of an object. an internal structure image output unit that constructs an internal structure image in which the color parameter is continuously changed and displays it on a display device;
a profile image output unit that causes the display device to display a profile image showing the correspondence relationship between the depth from the reference plane and the color parameter;
an input reception unit that receives an operation of the pointing device,
The input accepting unit accepts a change of at least one of the first depth and the second depth according to an operation content of the pointing device when an operation position of the pointing device is on the profile image. configured as
The internal structure image output unit converts the color parameter within the changed depth range accepted by the input accepting unit to the color parameter before change from the first depth after change to the second depth. configured to reconstruct the internal structure image changed in the same manner as the change mode of and re-display it on the display device,
The profile image output unit is configured to reconstruct the profile image according to the changed first depth and the second depth accepted by the input acceptance unit and cause the display device to re-display the profile image. image processing device. When the operation position of the pointing device accepted by the input accepting unit is on the internal structure image, the internal structure image output unit outputs the internal structure according to the operation content of the pointing device accepted by the input accepting unit. configured to movably display the structural image;
2. The profile image output unit according to claim 1, wherein when the operation position is on the profile image, the profile image is displayed in a movable manner according to the operation content. Image processing device. When the operation position of the pointing device accepted by the input accepting unit is on the internal structure image, the internal structure image output unit outputs the internal structure according to the operation content of the pointing device accepted by the input accepting unit. 3. The image processing apparatus according to claim 1, wherein the structural image is displayed in a mode in which at least one of enlargement and reduction is possible. The input reception unit selects one of the first depth and the second depth according to the operation content of the pointing device when the operation position of the pointing device is on the first region of the profile image. 4. An image processing apparatus as claimed in any one of claims 1 to 3, configured to accept one change. The input reception unit, when the operation position of the pointing device is on the second area of the profile image, selects the one of the first depth and the second depth according to the operation content of the pointing device. 5. An image processing apparatus according to claim 4, configured to accept another change different from the other. The input reception unit, when the operation position of the pointing device is on the third region of the profile image, according to the operation content of the pointing device, the depth range without changing the size of the depth range. 6. The image processing apparatus according to any one of claims 1 to 5, configured to receive a change that moves the depth range in at least one of a deep direction and a shallow direction. When the operation position of the pointing device is on the profile image, the input reception unit expands or expands the depth range without changing the center of the depth range according to the operation content of the pointing device. 7. An image processing apparatus as claimed in any one of claims 1 to 6, configured to accept modifications that allow at least one of reduction. 8. The image processing apparatus according to any one of claims 1 to 7, wherein the volume data is obtained by image reconstruction of signal data of photoacoustic waves generated by irradiating a living body with light. obtaining volume data representing the three-dimensional internal structure of the object;
A color parameter of an internal structure image is obtained based on information on a first depth and a second depth from a predetermined reference plane and the volume data within the depth range from the first depth to the second depth. process and
A step of constructing the internal structure image in which the color parameter is continuously changed from the first depth to the second depth based on the volume data and the color parameter, and displaying the image on a display device;
a step of displaying, on the display device, a profile image showing a correspondence relationship between the depth from the reference plane and the color parameter;
determining pointing device input;
determining an operating position of the pointing device;
An image processing method comprising: a procedure for obtaining volume data representing the three-dimensional internal structure of an object;
A color parameter of an internal structure image is obtained based on information on a first depth and a second depth from a predetermined reference plane and the volume data within the depth range from the first depth to the second depth. a procedure;
a step of constructing the internal structure image in which the color parameter is continuously changed from the first depth to the second depth based on the volume data and the color parameter, and displaying the image on a display device; ,
a step of displaying on the display device a profile image showing the correspondence relationship between the depth from the reference plane and the color parameter;
a procedure for determining pointing device input;
a procedure for determining an operation position of the pointing device;
A computer-readable recording medium that records a program that causes a computer to execute

Images