JP2020028668A - Image processing system, image processing method and program - Google Patents

Abstract

To provide an image processing system used for a system capable of improving convenience related to observation of a structure of a contrast object by photoacoustic imaging.SOLUTION: An image processing system in an embodiment, which is an image processing system for processing image data generated based on a photoacoustic wave generated from inside an analyte by light irradiation to the analyte, has display control means for displaying, on a display device, image data, and an input interface for receiving input to an area of interest which is a part of an area of a lymphatic vessel in the analyte in the image data, and storage control means for storing the image data, and information inputted through the input interface, correlatively in a storage device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to image processing on an image generated by photoacoustic imaging.

  In the examination of blood vessels, lymph vessels, and the like, photoacoustic imaging using a contrast agent (also referred to as “optical ultrasonic imaging”) is known. Patent Literature 1 discloses a photoacoustic image generation device that evaluates a contrast agent used for imaging lymph nodes, lymph vessels, and the like, and emits light having a wavelength that generates a photoacoustic wave when the contrast agent is absorbed. Is described.

International Publication No. WO 2017/002337

  However, in the photoacoustic imaging described in Patent Literature 1, a structure to be imaged inside the subject (for example, running of blood vessels, lymph vessels, and the like) is merely depicted, and a user may observe the structure. Inconvenience could have occurred.

  Therefore, an object of the present invention is to provide an image processing apparatus used for a system that improves the convenience of observing a structure to be contrasted by photoacoustic imaging.

One embodiment of the present invention provides
An image processing apparatus that processes image data generated based on a photoacoustic wave generated from within the subject by irradiating the subject with light,
Display control means for displaying a display device with the image data and an input interface that receives an input for a region of interest that is a part of the region of the lymph vessel in the subject in the image data,
Storage control means for storing the image data in the storage device in association with information input via the input interface,
An image processing apparatus comprising:

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus used for the system which improves the convenience regarding the observation of the structure of a contrast target by photoacoustic imaging can be provided.

1 is a block diagram of a system according to an embodiment of the present invention. FIG. 1 is a block diagram showing a specific example of an image processing apparatus according to an embodiment of the present invention and its peripheral configuration. Detailed block diagram of a photoacoustic apparatus according to an embodiment of the present invention Schematic diagram of a probe according to one embodiment of the present invention Flow chart of an image processing method according to an embodiment of the present invention Flow chart of the image processing method according to the first embodiment Contour graph of the calculated value of equation (1) corresponding to the contrast agent when the combination of wavelengths is changed Line graph showing the calculated value of equation (1) corresponding to the contrast agent when the concentration of ICG is changed Graph showing the molar absorption coefficient spectrum of oxyhemoglobin and deoxyhemoglobin FIG. 2 is a diagram illustrating a GUI according to an embodiment of the present invention. Diagram illustrating a spectral image of a subject Diagram illustrating classification by lymphatic vessel condition Diagram exemplifying classification of lymphatic vessels according to number, area ratio, and volume ratio Diagram illustrating classification of lymphatic vessels according to distance from vein Flow chart of the image processing method according to the second embodiment FIG. 9 illustrates a GUI according to the second embodiment. FIG. 14 is a diagram illustrating a display example of a classification result of lymphatic vessels according to the second embodiment. The figure explaining the process of extracting the area corresponding to the contrast agent The figure explaining the process of extracting the area corresponding to the contrast agent

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. However, dimensions, materials, shapes, relative arrangements, and the like of the components described below should be appropriately changed depending on the configuration of the apparatus to which the invention is applied and various conditions. Therefore, it is not intended to limit the scope of the present invention to the following description.

  The photoacoustic image obtained by the system according to the present invention reflects the absorption amount and the absorption rate of light energy. The photoacoustic image is an image representing a spatial distribution of at least one object information such as a generated sound pressure (initial sound pressure) of a photoacoustic wave, a light absorption energy density, and a light absorption coefficient. The photoacoustic image may be an image representing a two-dimensional spatial distribution or an image (volume data) representing a three-dimensional spatial distribution. The system according to the present embodiment generates a photoacoustic image by photographing a subject into which a contrast agent has been introduced. In order to grasp the three-dimensional structure of the contrast target, the photoacoustic image may be an image representing a two-dimensional spatial distribution or a three-dimensional spatial distribution in a depth direction from the subject surface.

Further, the system according to the present invention can generate a spectral image of the subject using a plurality of photoacoustic images corresponding to a plurality of wavelengths. The spectral image of the present invention is based on a photoacoustic wave generated by irradiating a subject with light of a plurality of different wavelengths, and is an image generated using a photoacoustic signal corresponding to each of the plurality of wavelengths. is there.
Note that the spectral image may be an image generated using the photoacoustic signals corresponding to each of the plurality of wavelengths and indicating the concentration of the specific substance in the subject. When the light absorption coefficient spectrum of the contrast agent used and the light absorption coefficient spectrum of the specific substance are different, the image value of the contrast agent in the spectral image and the image value of the specific substance in the spectral image are different. Therefore, the region of the contrast agent and the region of the specific substance can be distinguished according to the image value of the spectral image. In addition, as the specific substance, a substance constituting the subject, such as hemoglobin, glucose, collagen, melanin, fat and water, may be mentioned. Also in this case, it is necessary to select a contrast agent having a light absorption spectrum different from the light absorption coefficient spectrum of the specific substance. Further, the spectral image may be calculated by a different calculation method according to the type of the specific substance.

ここで、Ｉ^λ _１（ｒ）は第１波長λ_１の光照射により発生した光音響波に基づいた計測値であり、Ｉ^λ _２（ｒ）は第２波長λ_２の光照射により発生した光音響波に基づいた計測値である。ε_Ｈｂ ^λ _１は第１波長λ_１に対応するデオキシヘモグロビンのモラー吸収係数［ｍｍ^−１ｍｏｌ^−１］であり、ε_Ｈｂ ^λ _２は第２波長λ_２に対応するデオキシヘモグロビンのモラー吸収係数［ｍｍ^−１ｍｏｌ^−１］である。ε_ＨｂＯ ^λ _１は第１波長λ_１に対応するオキシヘモグロビンのモラー吸収係数［ｍｍ^−１ｍｏｌ^−１］であり、ε_ＨｂＯ ^λ _２は第２波長λ_２に対応するオキシヘモグロビンのモラー吸収係数［ｍｍ^−１ｍｏｌ^−１］である。ｒは位置である。なお、計測値Ｉ^λ _１（ｒ）、Ｉ^λ _２（ｒ）としては、吸収係数μ_ａ ^λ _１（ｒ）、μ_ａ ^λ _２（ｒ）を用いてもよいし、初期音圧Ｐ_０ ^λ _１（ｒ）、Ｐ_０ ^λ _２（ｒ）を用いてもよい。 In the embodiment described below, an image calculated using the oxygen saturation calculation formula (1) will be described as a spectral image. The present inventors have calculated the optical saturation of blood hemoglobin based on the photoacoustic signal corresponding to each of the plurality of wavelengths (or an index having a correlation with the oxygen saturation). When the measured value I (r) of a photoacoustic signal obtained with a contrast agent whose wavelength dependence of the absorption coefficient is different from that of oxyhemoglobin and deoxyhemoglobin is substituted, the numerical range in which the oxygen saturation of hemoglobin can be taken From the calculated value Is (r). Therefore, if a spectral image having the calculated value Is (r) as an image value is generated, the hemoglobin region (blood vessel region) and the contrast agent existing region (for example, the contrast agent is introduced into the lymphatic vessels) inside the subject. In this case, it is easy to separate (discriminate) the lymphatic region) from the image.

Here, I ^λ ₁ (r) is a measurement value based on a photoacoustic wave generated by light irradiation of the first wavelength λ ₁ , and I ^λ ₂ (r) is generated by light irradiation of the second wavelength λ ₂ This is a measurement value based on a photoacoustic wave. ε _Hb ^λ ₁ is the molar absorption coefficient of deoxyhemoglobin corresponding to the first wavelength λ ₁ [mm ⁻¹ mol ⁻¹ ], and ε _Hb ^λ ₂ is the molar absorption coefficient of deoxy hemoglobin corresponding to the second wavelength λ ₂ [ mm ^-1 mol ^-1 ]. ε _HbO ^λ ₁ is the molar absorption coefficient of oxyhemoglobin [mm ⁻¹ mol ⁻¹ ] corresponding to the first wavelength λ ₁ , and ε _HbO ^λ ₂ is the molar absorption coefficient of oxyhemoglobin corresponding to the second wavelength λ ₂ [ mm ^-1 mol ^-1 ]. r is a position. As the measured values I ^λ ₁ (r) and I ^λ ₂ (r), the absorption coefficients μ _a ^λ ₁ (r) and μ _a ^λ ₂ (r) may be used, or the initial sound pressure P ₀ ^λ ₁ (r) and P ₀ ^λ ₂ (r) may be used.

  When a measurement value based on a photoacoustic wave generated from a region where a hemoglobin exists (a blood vessel region) is substituted into Expression (1), the oxygen saturation of hemoglobin (or a correlation with the oxygen saturation) is obtained as a calculated value Is (r). Index) is obtained. On the other hand, when a measurement value based on an acoustic wave generated from a region where the contrast agent is present (for example, a lymphatic region) in the subject into which the contrast agent is introduced is substituted into Expression (1), a pseudo value is calculated as a calculated value Is (r). A concentration distribution of the contrast agent is obtained. Note that even when calculating the concentration distribution of the contrast agent, the numerical value of the molar absorption coefficient of hemoglobin may be used as it is in Expression (1). The spectral image Is (r) obtained in this manner is in a state where both the hemoglobin existing region (blood vessel) and the contrast agent existing region (for example, lymphatic vessel) inside the subject are separable from each other (can be distinguished). The image is rendered.

  In the present embodiment, the image value of the spectral image is calculated using Expression (1) for calculating the oxygen saturation. However, when an index other than the oxygen saturation is calculated as the image value of the spectral image, the expression A calculation method other than (1) may be used. As the index and the method for calculating the index, known ones can be used, and a detailed description thereof will be omitted here.

Further, the system according to the present invention was based on the photoacoustic wave generated by the first light irradiation of the photoacoustic image and the second wavelength lambda _2, based on the photoacoustic wave generated by light irradiation of the first wavelength lambda ₁ An image indicating the ratio of the two photoacoustic images may be used as the spectral image. That is, the ratio of the second photoacoustic image based on the photoacoustic wave generated by light irradiation of the first photoacoustic image and the second wavelength lambda _2, based on the photoacoustic wave generated by light irradiation of the first wavelength lambda ₁ The image based on this may be a spectral image. Note that an image generated according to the modified expression of Expression (1) can also be expressed by the ratio between the first photoacoustic image and the second photoacoustic image. Image (spectral image).

In order to grasp the three-dimensional structure of the contrast target, the spectral image may be an image representing a two-dimensional spatial distribution or a three-dimensional spatial distribution in the depth direction from the subject surface.
Hereinafter, a system configuration and an image processing method of the present embodiment will be described.

  A system according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of a system according to the present embodiment. The system according to the present embodiment includes a photoacoustic device 1100, a storage device 1200, an image processing device 1300, a display device 1400, and an input device 1500. Transmission and reception of data between the devices may be performed by wire or wirelessly.

  The photoacoustic device 1100 generates a photoacoustic image by capturing an image of the subject into which the contrast agent has been introduced, and outputs the photoacoustic image to the storage device 1200. The photoacoustic device 1100 is a device that generates information of characteristic values corresponding to each of a plurality of positions in a subject using a reception signal obtained by receiving a photoacoustic wave generated by light irradiation. That is, the photoacoustic apparatus 1100 is an apparatus that generates a spatial distribution of characteristic value information derived from a photoacoustic wave as medical image data (photoacoustic image).

  The storage device 1200 may be a storage medium such as a ROM (Read Only Memory), a magnetic disk, or a flash memory. Further, the storage device 1200 may be a storage server via a network such as a PACS (Picture Archiving and Communication System).

The image processing device 1300 is a device that processes information such as a photoacoustic image and incidental information of the photoacoustic image stored in the storage device 1200.
A unit having an arithmetic function of the image processing apparatus 1300 can be configured by an arithmetic circuit such as a CPU, a processor such as a GPU (Graphics Processing Unit), or an FPGA (Field Programmable Gate Array) chip. These units may be configured not only from a single processor or arithmetic circuit, but also from a plurality of processors or arithmetic circuits.

  A unit having a storage function of the image processing apparatus 1300 can be configured by a non-temporary storage medium such as a ROM (Read Only Memory), a magnetic disk, or a flash memory. In addition, the unit having the storage function may be a volatile medium such as a RAM (Random Access Memory). The storage medium on which the program is stored is a non-temporary storage medium. Note that the unit having the storage function is not limited to a single storage medium, and may be configured from a plurality of storage media.

  A unit having a control function of the image processing apparatus 1300 is configured by an arithmetic element such as a CPU. A unit having a control function controls the operation of each component of the system. The unit having the control function may control each component of the system in response to an instruction signal from various operations such as the start of measurement from the input unit. Further, the unit having the control function may read out the program code stored in the computer 150 and control the operation of each component of the system.

  The display device 1400 is a display such as a liquid crystal display or an organic EL (Electro Luminescence). The display device 1400 may display an image or a GUI for operating the device.

  As the input device 1500, an operation console that can be operated by a user and includes a mouse, a keyboard, and the like can be employed. Further, the display device 1400 may be configured with a touch panel, and the display device 1400 may be used as the input device 1500.

FIG. 2 shows a specific configuration example of the image processing apparatus 1300 according to the present embodiment. The image processing apparatus 1300 according to the present embodiment includes a CPU 1310, a GPU 1320, a RAM 1330, a ROM 1340, and an external storage device 1350. In addition, a liquid crystal display 1410 as a display device 1400, a mouse 1510 as an input device 1500, and a keyboard 1520 are connected to the image processing device 1300. Further, the image processing apparatus 1300 includes a PACS (Picture Archiving and Communication).
The system is connected to an image server 1210 as a storage device 1200 such as a System. Thus, the image data can be stored on the image server 1210 or the image data on the image server 1210 can be displayed on the liquid crystal display 1410.
Next, a configuration example of an apparatus included in the system according to the present embodiment will be described. FIG. 3 is a schematic block diagram of devices included in the system according to the present embodiment.

  The photoacoustic apparatus 1100 according to the present embodiment includes a drive unit 130, a signal collection unit 140, a computer 150, a probe 180, and an introduction unit 190. The probe 180 has a light irradiation unit 110 and a reception unit 120. FIG. 4 is a schematic diagram of the probe 180 according to the present embodiment. The measurement target is the subject 100 into which the contrast agent has been introduced by the introduction unit 190. The drive unit 130 drives the light irradiation unit 110 and the reception unit 120 to perform mechanical scanning. The light irradiation unit 110 irradiates the subject 100 with light, and an acoustic wave is generated in the subject 100. An acoustic wave generated by the photoacoustic effect due to light is also called a photoacoustic wave. The receiving unit 120 outputs an electric signal (photoacoustic signal) as an analog signal by receiving the photoacoustic wave.

  The signal collecting unit 140 converts the analog signal output from the receiving unit 120 into a digital signal, and outputs the digital signal to the computer 150. The computer 150 stores the digital signal output from the signal collection unit 140 as signal data derived from a photoacoustic wave. The signal data is an example of received signal data.

  The computer 150 generates a photoacoustic image by performing signal processing on the stored digital signal. In addition, the computer 150 outputs the photoacoustic image to the display unit 160 after performing image processing on the obtained photoacoustic image. The display unit 160 displays an image based on the photoacoustic image. The display image is stored in a memory in the computer 150 or a storage device 1200 such as a data management system connected to the modality via a network based on a storage instruction from the user or the computer 150.

The computer 150 also performs drive control of components included in the photoacoustic device. The display unit 160 may display a GUI or the like in addition to the image generated by the computer 150. The input unit 170 is configured to allow a user to input information. The user can use the input unit 170 to perform operations such as start and end of measurement, and an instruction to save a created image.
Hereinafter, details of each configuration of the photoacoustic apparatus 1100 according to the present embodiment will be described.

(Light irradiation unit 110)
The light irradiation unit 110 includes a light source 111 that emits light, and an optical system 112 that guides light emitted from the light source 111 to the subject 100. Note that the light includes pulse light such as a so-called rectangular wave and a triangular wave.

The pulse width of the light emitted from the light source 111 is preferably 100 ns or less in consideration of the thermal confinement condition and the stress confinement condition. Further, the wavelength of the light may be in the range of about 400 nm to 1600 nm. When imaging a blood vessel with high resolution, a wavelength (400 nm or more and 700 nm or less) at which absorption in the blood vessel is large may be used. When imaging a deep part of a living body, light having a wavelength (700 nm or more and 1100 nm or less) that typically absorbs little in a background tissue (water or fat) of the living body may be used.

  As the light source 111, a laser or a light emitting diode can be used. When measuring using light of a plurality of wavelengths, a light source whose wavelength can be changed may be used. When irradiating the subject with a plurality of wavelengths, it is also possible to prepare a plurality of light sources that generate light having different wavelengths from each other, and irradiate each of the light sources alternately. When a plurality of light sources are used, they are collectively expressed as a light source. Various lasers such as a solid-state laser, a gas laser, a dye laser, and a semiconductor laser can be used as the laser. For example, a pulsed laser such as an Nd: YAG laser or an alexandrite laser may be used as a light source. Alternatively, a Ti: sa laser using an Nd: YAG laser beam as an excitation light or an OPO (Optical Parametric Oscillators) laser may be used as a light source. Further, a flash lamp or a light emitting diode may be used as the light source 111. Further, a microwave source may be used as the light source 111.

Optical elements such as lenses, mirrors, and optical fibers can be used for the optical system 112. When a breast or the like is used as the subject 100, the light emitting unit of the optical system may be configured with a diffusion plate or the like that diffuses light in order to irradiate the pulsed light with a wider beam diameter. On the other hand, in the photoacoustic microscope, in order to increase the resolution, the light emitting portion of the optical system 112 may be configured by a lens or the like, and the beam may be focused and irradiated.
The light irradiating unit 110 may directly irradiate the subject 100 with light from the light source 111 without including the optical system 112.

(Receiver 120)
The receiving unit 120 includes a transducer 121 that outputs an electric signal by receiving an acoustic wave, and a support 122 that supports the transducer 121. Further, the transducer 121 may be a transmitting unit that transmits an acoustic wave. The transducer as the receiving means and the transducer as the transmitting means may be a single (common) transducer or may have different configurations.

  As a member constituting the transducer 121, a piezoelectric ceramic material represented by PZT (lead zirconate titanate), a polymer piezoelectric film material represented by PVDF (polyvinylidene fluoride), or the like can be used. Further, an element other than the piezoelectric element may be used. For example, a transducer using a capacitive micro-machined Ultrasonic Transducers (CMUT) or the like can be used. Note that any transducer may be employed as long as an electrical signal can be output by receiving an acoustic wave. The signal obtained by the transducer is a time-resolved signal. That is, the amplitude of the signal obtained by the transducer represents a value based on the sound pressure received by the transducer at each time (for example, a value proportional to the sound pressure).

  The frequency component constituting the photoacoustic wave is typically 100 KHz to 100 MHz, and a transducer capable of detecting these frequencies may be employed as the transducer 121.

The support 122 may be made of a metal material having high mechanical strength. In order to cause a large amount of irradiation light to enter the subject, the surface of the support 122 on the subject 100 side may be subjected to mirror finishing or light scattering. In the present embodiment, the support 122 has a hemispherical shell shape, and is configured to be able to support the plurality of transducers 121 on the hemispherical shell. In this case, the directional axes of the transducers 121 disposed on the support body 122 gather near the center of curvature of the hemisphere. Then, when an image is formed using the signals output from the plurality of transducers 121, the image quality near the center of curvature becomes high. The support 122 may have any configuration as long as it can support the transducer 121. The support 122 may arrange a plurality of transducers in a plane or a curved surface such as a 1D array, a 1.5D array, a 1.75D array, and a 2D array. The plurality of transducers 121 correspond to a plurality of receiving units.

  Further, the support 122 may function as a container for storing the acoustic matching material. That is, the support 122 may be a container for disposing the acoustic matching material between the transducer 121 and the subject 100.

  Further, receiving section 120 may include an amplifier that amplifies a time-series analog signal output from transducer 121. Further, the receiving unit 120 may include an A / D converter that converts a time-series analog signal output from the transducer 121 into a time-series digital signal. That is, the receiving unit 120 may include a signal collecting unit 140 described later.

  The space between the receiving unit 120 and the subject 100 is filled with a medium through which a photoacoustic wave can propagate. For this medium, a material that can transmit an acoustic wave, has matching acoustic characteristics at the interface with the subject 100 and the transducer 121, and has the highest possible transmittance of the photoacoustic wave is used. For example, water, an ultrasonic gel, or the like can be used as the medium.

  FIG. 4 shows a side view of the probe 180. The probe 180 according to the present embodiment has a receiving unit 120 in which a plurality of transducers 121 are three-dimensionally arranged on a hemispherical support body 122 having an opening. In addition, a light emitting portion of the optical system 112 is disposed at the bottom of the support 122.

In the present embodiment, as shown in FIG. 4, the shape of the subject 100 is held by contacting the holding unit 200.
The space between the receiving unit 120 and the holding unit 200 is filled with a medium through which a photoacoustic wave can propagate. For this medium, a material that can transmit a photoacoustic wave, matches acoustic characteristics at the interface with the subject 100 and the transducer 121, and has a transmittance of the photoacoustic wave as high as possible is used. For example, water, an ultrasonic gel, or the like can be used as the medium.

  The holding unit 200 as a holding unit is used to hold the shape of the subject 100 during measurement. By holding the subject 100 by the holding unit 200, the movement of the subject 100 can be suppressed and the position of the subject 100 can be kept in the holding unit 200. A resin material such as polycarbonate, polyethylene, or polyethylene terephthalate can be used as the material of the holding section 200.

  The holding unit 200 is attached to the attachment unit 201. The attachment unit 201 may be configured so that a plurality of types of holding units 200 can be exchanged according to the size of the subject. For example, the mounting portion 201 may be configured to be exchangeable with a different holding portion such as a radius of curvature or a center of curvature.

(Drive unit 130)
The driving unit 130 is a unit that changes the relative position between the subject 100 and the receiving unit 120. The driving unit 130 includes a motor such as a stepping motor that generates a driving force, a driving mechanism that transmits the driving force, and a position sensor that detects position information of the receiving unit 120. As the driving mechanism, a lead screw mechanism, a link mechanism, a gear mechanism, a hydraulic mechanism, or the like can be used. As the position sensor, a potentiometer using an encoder, a variable resistor, a linear scale, a magnetic sensor, an infrared sensor, an ultrasonic sensor, or the like can be used.
The driving unit 130 is not limited to changing the relative position between the subject 100 and the receiving unit 120 in the XY directions (two-dimensional), and may change the relative position to one-dimensional or three-dimensional.

  The drive unit 130 may fix the receiving unit 120 and move the subject 100 as long as the relative position between the subject 100 and the receiving unit 120 can be changed. When the subject 100 is moved, a configuration in which the subject 100 is moved by moving a holding unit that holds the subject 100 can be considered. Further, both the subject 100 and the receiving unit 120 may be moved.

  The drive unit 130 may move the relative position continuously, or may move the relative position by step and repeat. The drive unit 130 may be an electric stage that moves along a programmed trajectory, or may be a manual stage.

In the present embodiment, the driving unit 130 scans by simultaneously driving the light irradiation unit 110 and the reception unit 120. However, the drive unit 130 drives only the light irradiation unit 110 or drives only the reception unit 120. You may.
When the probe 180 is a hand-held type provided with a grip, the photoacoustic device 1100 may not include the driving unit 130.

(Signal collection unit 140)
The signal collection unit 140 includes an amplifier that amplifies an electric signal that is an analog signal output from the transducer 121, and an A / D converter that converts an analog signal output from the amplifier into a digital signal. The digital signal output from the signal collection unit 140 is stored in the computer 150. The signal collection unit 140 is also called a Data Acquisition System (DAS). In the present specification, the electric signal is a concept including both an analog signal and a digital signal. Note that a light detection sensor such as a photodiode may detect light emission from the light irradiation unit 110, and the signal collection unit 140 may start the above process in synchronization with the detection result in response to a trigger.

(Computer 150)
The computer 150 as the information processing device is configured by the same hardware as the image processing device 1300. That is, the unit having the arithmetic function of the computer 150 can be configured by an arithmetic circuit such as a processor such as a CPU or a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) chip. These units may be configured not only from a single processor or arithmetic circuit, but also from a plurality of processors or arithmetic circuits.

  The unit that performs the storage function of the computer 150 may be a volatile medium such as a RAM (Random Access Memory). The storage medium on which the program is stored is a non-temporary storage medium. It should be noted that the unit having the storage function of the computer 150 may not only be constituted by one storage medium, but also constituted by a plurality of storage media.

  A unit having a control function of the computer 150 is configured by an arithmetic element such as a CPU. A unit having a control function of the computer 150 controls the operation of each component of the photoacoustic apparatus. A unit having a control function of the computer 150 may control each component of the photoacoustic apparatus by receiving an instruction signal from the input unit 170 through various operations such as a start of measurement. Further, the unit having the control function of the computer 150 reads out the program code stored in the unit having the storage function, and controls the operation of each component of the photoacoustic apparatus. That is, the computer 150 can function as a control device of the system according to the present embodiment.

  Note that the computer 150 and the image processing apparatus 1300 may be configured by the same hardware. One piece of hardware may perform the functions of both the computer 150 and the image processing device 1300. That is, the computer 150 may perform the function of the image processing apparatus 1300. Further, the image processing device 1300 may have the function of the computer 150 as the information processing device.

(Display unit 160)
The display unit 160 is a display such as a liquid crystal display and an organic EL (Electro Luminescence). The display unit 160 may display an image or a GUI for operating the apparatus.

  Note that the display unit 160 and the display device 1400 may be the same display. That is, one display may have the functions of both the display unit 160 and the display device 1400.

(Input unit 170)
As the input unit 170, an operation console that can be operated by a user and includes a mouse and a keyboard can be employed. Further, the display unit 160 may be configured by a touch panel, and the display unit 160 may be used as the input unit 170.

  Note that the input unit 170 and the input device 1500 may be the same device. That is, one device may perform both functions of the input unit 170 and the input device 1500.

(Introduction section 190)
The introduction unit 190 is configured to be able to introduce a contrast agent from outside the subject 100 into the inside of the subject 100. For example, the introducer 190 can include a container for the contrast agent and a needle for piercing the subject. However, the present invention is not limited to this, and various types can be applied to the introduction unit 190 as long as the contrast agent can be introduced into the subject 100. In this case, the introduction unit 190 may be, for example, a known injection system, an injector, or the like. The contrast agent may be introduced into the subject 100 by controlling the operation of the introduction unit 190 by the computer 150 as a control device. Further, the contrast agent may be introduced into the subject 100 by operating the introduction unit 190 by the user.

(Subject 100)
The subject 100 does not constitute a system, but will be described below. The system according to the present embodiment can be used for the purpose of diagnosing malignant tumors and vascular diseases of humans and animals, monitoring the progress of chemotherapy and the like. Therefore, the subject 100 is assumed to be a body to be diagnosed, specifically, a living body, specifically, a breast or each organ of a human body or an animal, a vascular network, a head, a neck, an abdomen, a limb including a finger or a toe. You. For example, if the human body is a measurement target, oxyhemoglobin or deoxyhemoglobin, a blood vessel containing many of them, a new blood vessel formed near a tumor, or the like may be the target of the light absorber. In addition, plaque of the carotid artery wall or the like may be a target of the light absorber. In addition, melanin, collagen, lipids, and the like contained in the skin and the like may be targeted for the light absorber. Furthermore, the contrast agent introduced into the subject 100 can be a light absorber. As a contrast agent used for photoacoustic imaging, a dye such as indocyanine green (ICG) or methylene blue (MB), a fine gold particle, a mixture thereof, or a substance which is integrated or chemically modified and externally introduced is used. May be. Further, a phantom imitating a living body may be used as the subject 100.

Each configuration of the photoacoustic device may be configured as a separate device, or may be configured as one integrated device. Further, at least a part of the configuration of the photoacoustic apparatus may be configured as one integrated apparatus.

  Each device constituting the system according to the present embodiment may be constituted by separate hardware, or all devices may be constituted by one piece of hardware. The function of the system according to the present embodiment may be configured by any hardware.

  Next, an image generation method according to the present embodiment will be described with reference to the flowchart shown in FIG.

(S400: Step of determining wavelength of irradiation light)
The computer 150 as the wavelength determining means determines the wavelength of the irradiation light based on the information on the contrast agent. In the present embodiment, a combination of wavelengths is determined so that a region corresponding to a contrast agent in a spectral image is easily identified. Note that the computer 150 can acquire information on the contrast agent, which is input by a user such as a doctor using the input unit 170, for example. Further, the computer 150 may store information on a plurality of contrast agents in advance, and acquire information on the contrast agent set by default from the information.

  FIG. 10 shows an example of a GUI displayed on the display unit 160. In the item 2500 of the GUI, examination order information such as a patient ID, an examination ID, and an imaging date and time is displayed. The item 2500 may have a display function of displaying inspection order information acquired from an external device such as a HIS or RIS, or an input function of allowing a user to input inspection order information using the input unit 170. The GUI item 2600 displays information on the contrast agent such as the type of the contrast agent and the concentration of the contrast agent. The item 2600 may have a display function of displaying information on a contrast agent acquired from an external device such as an HIS or RIS, or an input function that allows a user to input information on a contrast agent using the input unit 170. Good. In the item 2600, information on the contrast agent such as the type and concentration of the contrast agent may be input from a plurality of options by a method such as pull-down. Note that the GUI shown in FIG. 10 may be displayed on the display device 1400.

  Note that, when the image processing apparatus 1300 does not receive an instruction to input information on the contrast agent from the user, the information on the contrast agent set by default may be acquired from the information on the plurality of contrast agents. In the case of this embodiment, a case will be described in which ICG is set as the type of the contrast agent and 1.0 mg / mL is set as the concentration of the contrast agent by default. In the present embodiment, the type and density of the contrast agent set by default are displayed in the item 2600 of the GUI, but the information on the contrast agent may not be set by default. In this case, the information about the contrast agent may not be displayed on the GUI item 2600 on the initial screen.

Here, a change in an image value corresponding to a contrast agent in a spectral image when a combination of wavelengths is changed will be described. FIG. 7 shows a simulation result of an image value (oxygen saturation value) corresponding to a contrast agent in a spectral image in each combination of two wavelengths. The vertical axis and the horizontal axis in FIG. 7 represent the first wavelength and the second wavelength, respectively. FIG. 7 shows contour lines of image values corresponding to the contrast agent in the spectral image. FIGS. 7A to 7D show contrast agents in spectral images when the concentration of ICG is 5.04 μg / mL, 50.4 μg / mL, 0.5 mg / mL, and 1.0 mg / mL, respectively. Shows the image value corresponding to. As shown in FIG. 7, the image value corresponding to the contrast agent in the spectral image may be 60% to 100% depending on the combination of the wavelengths to be selected. As described above, if such a combination of wavelengths is selected, it becomes difficult to distinguish a blood vessel region and a contrast agent region in a spectral image. Therefore, in the combination of wavelengths shown in FIG. 7, it is preferable to select a combination of wavelengths such that the image value corresponding to the contrast agent in the spectral image is smaller than 60% or larger than 100%. Further, in the combination of wavelengths shown in FIG. 7, it is preferable to select a combination of wavelengths such that the image value corresponding to the contrast agent in the spectral image has a negative value.

  For example, consider a case where 797 nm is selected as the first wavelength and 835 nm is selected as the second wavelength. FIG. 8 shows the relationship between the concentration of ICG and the image value (oxygen saturation value) corresponding to the contrast agent in the spectral image when 797 nm is selected as the first wavelength and 835 nm is selected as the second wavelength. It is a graph. According to FIG. 8, when 797 nm is selected as the first wavelength and 835 nm is selected as the second wavelength, the contrast in the spectral image is increased regardless of the concentration of 5.04 μg / mL to 1.0 mg / mL. The image value corresponding to the agent is a negative value. Therefore, according to the spectral image generated by such a combination of wavelengths, since the oxygen saturation value of the blood vessel does not take a negative value in principle, the blood vessel region and the contrast agent region are clearly distinguished. be able to.

  Hereinafter, a region where a blood vessel exists is referred to as a blood vessel region, and a region where a contrast agent exists is also referred to as a contrast agent region. The blood vessel region is a region corresponding to an artery or a vein, and the contrast agent region is a region corresponding to a lymph vessel.

  Although the wavelength is determined based on the information on the contrast agent, the absorption coefficient of hemoglobin may be considered in determining the wavelength. FIG. 9 shows the spectrum of the molar absorption coefficient of oxyhemoglobin (dashed line) and the molar absorption coefficient of deoxyhemoglobin (solid line). In the wavelength range shown in FIG. 9, the magnitude relationship between the molar absorption coefficient of oxyhemoglobin and the molar absorption coefficient of deoxyhemoglobin is reversed at the boundary of 797 nm. That is, it can be said that it is easy to grasp the vein at a wavelength shorter than 797 nm, and it is easy to grasp the artery at a wavelength longer than 797 nm. In the treatment of lymphedema, lymphatic venule anastomosis for creating a bypass between lymphatic vessels and veins is used. For this preoperative examination, it is conceivable to use photoacoustic imaging to image both the veins and the lymph vessels in which the contrast agent has accumulated. In this case, by setting at least one of the plurality of wavelengths to a wavelength smaller than 797 nm, a vein can be more clearly imaged. Further, it is advantageous for imaging a vein that at least one of the plurality of wavelengths is set to a wavelength at which the molar absorption coefficient of deoxyhemoglobin is larger than the molar absorption coefficient of oxyhemoglobin. In addition, when a spectral image is generated from a photoacoustic image corresponding to two wavelengths, the vein is imaged by setting the wavelength at which the molar absorption coefficient of deoxyhemoglobin is larger than the molar absorption coefficient of oxyhemoglobin at any of the two wavelengths. This is advantageous. By selecting these wavelengths, in the preoperative examination of the lymphatic venule anastomosis, it is possible to accurately image both the lymphatic vessels and the veins into which the contrast agent has been introduced.

  By the way, if any of the plurality of wavelengths is a wavelength at which the absorption coefficient of the contrast agent is larger than that of the blood, the oxygen saturation accuracy of the blood decreases due to artifacts derived from the contrast agent. Therefore, in order to reduce artifacts derived from the contrast agent, at least one of the plurality of wavelengths may be a wavelength at which the absorption coefficient of the contrast agent is smaller than the absorption coefficient of blood.

  Here, the case where the spectral image is generated according to Equation (1) has been described, but the spectral image in which the image value corresponding to the contrast agent in the spectral image changes depending on the condition of the contrast agent and the wavelength of the irradiation light. Can also be applied when generating

(S500: Light Irradiation Step)
The light irradiation unit 110 sets the wavelength determined in S400 to the light source 111. The light source 111 emits light having the wavelength determined in S400. Light generated from the light source 111 is applied to the subject 100 as pulse light via the optical system 112. Then, the pulse light is absorbed inside the subject 100, and a photoacoustic wave is generated by the photoacoustic effect. At this time, the introduced contrast agent also absorbs the pulse light and generates a photoacoustic wave. The light irradiation unit 110 may transmit a synchronization signal to the signal collection unit 140 together with the transmission of the pulse light. The light irradiating unit 110 similarly irradiates each of a plurality of wavelengths with light.

  The user may use the input unit 170 to specify control parameters such as the irradiation conditions of the light irradiation unit 110 (such as the repetition frequency and wavelength of irradiation light) and the position of the probe 180. The computer 150 may set a control parameter determined based on a user's instruction. Further, the computer 150 may move the probe 180 to a specified position by controlling the driving unit 130 based on the specified control parameter. When imaging at a plurality of positions is designated, the drive unit 130 first moves the probe 180 to the first designated position. Note that the drive unit 130 may move the probe 180 to a position programmed in advance when a measurement start instruction is issued.

(S600: receiving a photoacoustic wave)
When receiving the synchronization signal transmitted from light irradiating section 110, signal collecting section 140 starts the signal collecting operation. That is, the signal collecting unit 140 generates an amplified digital electric signal by amplifying and AD converting the analog electric signal derived from the photoacoustic wave output from the receiving unit 120, and outputs the amplified digital electric signal to the computer 150. . The computer 150 stores the signal transmitted from the signal collecting unit 140. When imaging at a plurality of scanning positions is designated, the processes of S500 and S600 are repeatedly executed at the designated scanning positions, and irradiation of pulse light and generation of digital signals derived from acoustic waves are repeated. Note that the computer 150 may acquire and store the position information of the receiving unit 120 at the time of light emission based on the output from the position sensor of the drive unit 130 with the light emission as a trigger.

  Note that, in the present embodiment, an example in which each of a plurality of wavelengths of light is radiated in a time-division manner has been described. . For example, when encoding is performed by light irradiation, there may be a timing at which light of a plurality of wavelengths is irradiated almost simultaneously.

(S700: Step of Generating Photoacoustic Image)
The computer 150 as a photoacoustic image acquisition unit generates a photoacoustic image based on the stored signal data. The computer 150 outputs the generated photoacoustic image to the storage device 1200 and stores it. In the present embodiment, one volume data is generated by image reconstruction using a photoacoustic signal obtained by one light irradiation on the subject. Further, by performing light irradiation a plurality of times and performing image reconstruction for each light irradiation, time-series three-dimensional volume data is obtained.

  Reconstruction algorithms for converting signal data into a two-dimensional or three-dimensional spatial distribution include analytic reconstruction methods such as backprojection in the time domain and backprojection in the Fourier domain, and model-based methods (repetitive computations). Law) can be adopted. For example, as a back projection method in the time domain, Universal back-projection (UBP), Filtered back-projection (FBP), phasing addition (Delay-and-Sum), and the like can be given.

In the present embodiment, one three-dimensional photoacoustic image (volume data) is generated by image reconstruction using a photoacoustic signal obtained by a single light irradiation on the subject. Furthermore, by performing light irradiation a plurality of times and performing image reconstruction for each light irradiation, time-series three-dimensional image data (time-series volume data) is obtained. The three-dimensional image data obtained by reconstructing an image for each of the plurality of light irradiations is collectively referred to as three-dimensional image data corresponding to the plurality of light irradiations. Note that, since light irradiation is performed a plurality of times in a time series, three-dimensional image data corresponding to the light irradiations a plurality of times constitutes time-series three-dimensional image data.

  The computer 150 generates initial sound pressure distribution information (generated sound pressures at a plurality of positions) as a photoacoustic image by performing a reconstruction process on the signal data. Further, the computer 150 calculates the optical fluence distribution of the light radiated on the subject 100 inside the subject 100, and divides the initial sound pressure distribution by the light fluence distribution to obtain the absorption coefficient distribution information by photoacoustic. It may be obtained as an image. A known method can be applied to the calculation method of the light fluence distribution. In addition, the computer 150 can generate a photoacoustic image corresponding to each of the light of a plurality of wavelengths. Specifically, the computer 150 can generate a first photoacoustic image corresponding to the first wavelength by performing a reconstruction process on signal data obtained by irradiating light of the first wavelength. Further, the computer 150 can generate a second photoacoustic image corresponding to the second wavelength by performing a reconstruction process on the signal data obtained by irradiating the second wavelength light. As described above, the computer 150 can generate a plurality of photoacoustic images corresponding to lights of a plurality of wavelengths.

  In the present embodiment, the computer 150 acquires absorption coefficient distribution information corresponding to each of light of a plurality of wavelengths as a photoacoustic image. The absorption coefficient distribution information corresponding to the first wavelength is defined as a first photoacoustic image, and the absorption coefficient distribution information corresponding to the second wavelength is defined as a second photoacoustic image.

  In the present embodiment, an example has been described in which the system includes the photoacoustic apparatus 1100 that generates a photoacoustic image. However, the present invention is also applicable to a system that does not include the photoacoustic apparatus 1100. The present invention can be applied to any system as long as the image processing apparatus 1300 as a photoacoustic image acquisition unit can acquire a photoacoustic image. For example, the present invention can be applied to a system that does not include the photoacoustic device 1100 but includes the storage device 1200 and the image processing device 1300. In this case, the image processing device 1300 as the photoacoustic image acquisition unit can acquire the photoacoustic image by reading out the specified photoacoustic image from the photoacoustic image group stored in the storage device 1200 in advance. it can.

(S800: Step of generating a spectral image)
The computer 150 as a spectral image acquisition unit generates a spectral image based on a plurality of photoacoustic images corresponding to a plurality of wavelengths. The computer 150 outputs the spectral image to the storage device 1200 and causes the storage device 1200 to store the spectral image. As described above, the computer 150 may generate, as a spectral image, an image indicating information corresponding to the concentration of a substance constituting the subject, such as glucose concentration, collagen concentration, melanin concentration, and volume fraction of fat and water. Good. Further, the computer 150 may generate, as a spectral image, an image representing a ratio between the first photoacoustic image corresponding to the first wavelength and the second photoacoustic image corresponding to the second wavelength. In the present embodiment, an example will be described in which the computer 150 generates an oxygen saturation image as a spectral image according to Expression (1) using the first photoacoustic image and the second photoacoustic image.

  Note that the image processing apparatus 1300 as the spectral image acquiring unit may acquire a spectral image by reading out a designated spectral image from a spectral image group stored in the storage device 1200 in advance. Further, the image processing apparatus 1300 as a photoacoustic image acquisition unit includes at least one of a plurality of photoacoustic images used to generate the read spectral image from a group of photoacoustic images stored in the storage device 1200 in advance. May be read to obtain a photoacoustic image.

A plurality of times of light irradiation, subsequent acoustic wave reception and image reconstruction are performed, thereby generating time-series three-dimensional image data corresponding to the plurality of times of light irradiation. Photoacoustic image data and spectral image data can be used as the three-dimensional image data. Here, the photoacoustic image data refers to image data indicating a distribution of an absorption coefficient or the like, and the spectral image data is converted to photoacoustic image data corresponding to each wavelength when light of a plurality of wavelengths is irradiated on the subject. Indicates image data indicating the density or the like generated based on the image data.

(S1100: Displaying a spectral image)
The image processing apparatus 1300 serving as a display control unit displays a spectral image on the display device 1400 based on the information on the contrast agent so that the region corresponding to the contrast agent and the other region can be identified. In addition, as a rendering method, a maximum intensity projection method (MIP: Maximum)
Any method can be employed, such as intensity projection, volume rendering, and surface rendering. Here, setting conditions such as a display area and a line-of-sight direction when rendering a three-dimensional image in two dimensions can be arbitrarily specified according to the observation target.

  Here, a case will be described in which 797 nm and 835 nm are set in S400, and a spectral image is generated according to Expression (1) in S800. As shown in FIG. 8, when these two wavelengths are selected, the image value corresponding to the contrast agent in the spectral image generated according to Equation (1) is a negative value, regardless of the ICG concentration. .

  The oxygen saturation in blood vessels (arteries and veins) in a living body generally falls within a range of 60% to 100% in percent. Therefore, the wavelength (two wavelengths) of the light irradiating the subject is such that the oxygen saturation value (calculated value of the formula (1)) corresponding to the contrast agent in the spectral image is smaller than 60%, or smaller than 100%. Preferably, the wavelength is such that the wavelength becomes larger. This makes it easy to distinguish between an image corresponding to an artery and a vein and an image corresponding to a contrast agent in a spectral image. For example, when ICG is used as a contrast agent, two wavelengths of 700 nm or more and less than 820 nm and two wavelengths of 820 nm or more and 1020 nm or less are selected, and a spectral image is generated by Expression (1), thereby obtaining a region of the contrast agent. And a blood vessel region can be distinguished well.

  As shown in FIG. 10, the image processing apparatus 1300 causes a GUI to display a color bar 2400 as a color scale indicating the relationship between the image value of the spectral image and the display color. The image processing apparatus 1300 determines a numerical range of image values to be assigned to the color scale based on information on the contrast agent (for example, information indicating that the type of the contrast agent is ICG) and information indicating the wavelength of irradiation light. You may decide. For example, the image processing apparatus 1300 may determine a numerical range including a negative image value corresponding to the arterial oxygen saturation, the venous oxygen saturation, and the contrast agent. The image processing apparatus 1300 may determine a numerical range of -100% to 100%, and set a color bar 2400 in which -100% to 100% is assigned to a color gradation that changes from blue to red. With such a display method, in addition to the identification of the artery and vein, it is also possible to identify the area corresponding to the negative contrast agent. In addition, the image processing apparatus 1300 may cause the indicator 2410 indicating the numerical value range of the image value corresponding to the contrast agent to be displayed based on the information regarding the contrast agent and the information indicating the wavelength of the irradiation light. Here, in the color bar 2400, a negative value area is indicated by an indicator 2410 as a numerical value range of an image value corresponding to ICG. By displaying the color scale so that the display color corresponding to the contrast agent can be identified in this way, the region corresponding to the contrast agent in the spectral image can be easily identified.

  The image processing device 1300 as the region determining means may determine a region corresponding to the contrast agent in the spectral image based on information on the contrast agent and information indicating the wavelength of the irradiation light. For example, the image processing apparatus 1300 may determine a region having a negative image value in the spectral image as a region corresponding to the contrast agent. Then, the image processing device 1300 may display the spectral image on the display device 1400 so that the region corresponding to the contrast agent and the other region can be identified. The image processing apparatus 1300 displays an indicator (for example, a frame) indicating a region corresponding to the contrast agent, causing the display color of the region corresponding to the contrast agent to be different from that of the other region, blinking the region corresponding to the contrast agent, and the like. An identification display such as display may be employed.

  Note that by instructing the item 2730 corresponding to the display of the ICG displayed on the GUI shown in FIG. 10, it may be possible to switch to a display mode in which an image value corresponding to the ICG is selectively displayed. For example, when the user selects the item 2730 corresponding to the display of the ICG, the image processing apparatus 1300 selects a voxel having a negative image value from the spectral image and selectively renders the selected voxel, The ICG area may be selectively displayed. Similarly, the user may select an item 2710 corresponding to an artery display or an item 2720 corresponding to a vein display. Based on a user's instruction, the image processing apparatus 1300 selectively selects an image value corresponding to an artery (for example, 90% or more and 100% or less) or an image value corresponding to a vein (for example, 60% or more and less than 90%). The display mode may be switched to the display mode. The numerical value range of the image value corresponding to the artery or the image value corresponding to the vein may be changeable based on a user's instruction.

  Note that at least one of hue, lightness, and saturation is assigned to the image value of the spectral image, and an image in which the remaining parameters of hue, lightness, and saturation are assigned to the image value of the photoacoustic image is displayed as a spectral image. You may. For example, an image in which hue and saturation are assigned to image values of a spectral image and brightness is assigned to image values of a photoacoustic image may be displayed as a spectral image. At this time, when the image value of the photoacoustic image corresponding to the contrast agent is larger or smaller than the image value of the photoacoustic image corresponding to the blood vessel, if lightness is assigned to the image value of the photoacoustic image, the blood vessel and the contrast agent It may be difficult to see both of them. Therefore, the conversion table from the image value of the photoacoustic image to the brightness may be changed according to the image value of the spectral image. For example, when the image value of the spectral image is included in the numerical value range of the image value corresponding to the contrast agent, the brightness corresponding to the image value of the photoacoustic image may be smaller than that corresponding to the blood vessel. That is, when the contrast agent region and the blood vessel region are compared, if the image value of the photoacoustic image is the same, the brightness of the contrast agent region may be smaller than that of the blood vessel region. Here, the conversion table is a table indicating the brightness corresponding to each of the plurality of image values. When the image value of the spectral image is included in the numerical value range of the image value corresponding to the contrast agent, the brightness corresponding to the image value of the photoacoustic image may be larger than that corresponding to the blood vessel. That is, when the contrast agent region is compared with the blood vessel region, if the image value of the photoacoustic image is the same, the brightness of the contrast agent region may be greater than that of the blood vessel region. Further, the numerical value range of the image value of the photoacoustic image that does not convert the image value of the photoacoustic image into the brightness may differ depending on the image value of the spectral image.

  The conversion table may be changed to an appropriate one according to the type and concentration of the contrast agent and the wavelength of the irradiation light. Therefore, the image processing apparatus 1300 may determine the conversion table from the image value of the photoacoustic image to the brightness based on the information regarding the contrast agent and the information indicating the wavelength of the irradiation light. If it is estimated that the image value of the photoacoustic image corresponding to the contrast agent is larger than that corresponding to the blood vessel, the image processing apparatus 1300 sets the brightness corresponding to the image value of the photoacoustic image corresponding to the contrast agent to the blood vessel. It may be smaller than the corresponding one. Conversely, if the image value of the photoacoustic image corresponding to the contrast agent is estimated to be smaller than that corresponding to the blood vessel, the image processing apparatus 1300 may determine the brightness corresponding to the image value of the photoacoustic image corresponding to the contrast agent. May be larger than that corresponding to a blood vessel.

  The GUI shown in FIG. 10 includes an absorption coefficient image (first photoacoustic image) 2100 corresponding to a wavelength of 797 nm, an absorption coefficient image (second photoacoustic image) 2200 corresponding to a wavelength of 835 nm, and an oxygen saturation image (spectral image) 2300. Is displayed. The GUI may display which wavelength is generated by each image. In the present embodiment, both the photoacoustic image and the spectral image are displayed, but only the spectral image may be displayed. The image processing device 1300 may switch between displaying a photoacoustic image and displaying a spectral image based on a user's instruction.

  The display unit 160 may be capable of displaying a moving image. For example, the image processing apparatus 1300 generates at least one of the first photoacoustic image 2100, the second photoacoustic image 2200, and the spectral image 2300 in time series, and generates moving image data based on the generated time-series image. It may be configured to generate and output to the display unit 160. In addition, in view of the relatively small number of lymph flows, it is also preferable to display a still image or a time-compressed moving image in order to reduce the user's judgment time. In addition, in the moving image display, it is possible to repeatedly display a state in which lymph flows. The speed of the moving image may be a predetermined speed specified in advance or a predetermined speed specified by the user.

  It is also preferable that the display unit 160 capable of displaying a moving image has a variable frame rate of the moving image. In order to make the frame rate variable, a window for the user to manually input the frame rate, a slide bar for changing the frame rate, and the like may be added to the GUI of FIG. Here, since the lymph fluid flows intermittently in the lymphatic vessels, only part of the acquired time-series volume data that can be used to confirm the lymph flow is used. Therefore, if real-time display is performed when checking the flow of lymph, efficiency may decrease. Therefore, by making the frame rate of the moving image displayed on the display unit 160 variable, the fast-moving display of the displayed moving image becomes possible, so that the user can confirm the state of the fluid in the lymphatic vessel in a short time. Become.

  The display unit 160 may be capable of repeatedly displaying a moving image within a predetermined time range. At this time, it is also preferable to add a GUI such as a window or a slide bar for enabling the user to specify a range in which repeated display is performed, to FIG. This makes it easier for the user to grasp, for example, how the fluid flows in the lymphatic vessels.

  The state in which the fluid flows in the lymph vessels is displayed on the display unit 160 as flow information in the area of the lymph vessels. The display method of the flow information in the region of the lymphatic vessel is not limited to the above. For example, the image processing apparatus 1300 as a display control unit associates the flow information in the lymphatic vessel area with the lymphatic vessel area, and performs at least one of a luminance display, a color display, a graph display, and a numerical display, The information may be displayed on the same screen of the display device 1400. The image processing device 1300 as a display control unit may highlight at least one lymphatic vessel region.

(S1200: Step of displaying classification result of lymphatic vessel)
In step S1200, the image processing apparatus 1300 as the state estimating unit analyzes the image data, automatically extracts a region of a lymph vessel, and classifies the lymph vessel. The image processing device 1300 as a display control unit causes the display device 1400 to display the classification result of the lymphatic vessels.

  The image processing apparatus 1300 as the state estimating unit extracts an area of a lymph vessel in the subject by performing image analysis of the spectral image generated in S800. In the spectral image, for example, the lymph vessels and the veins in the subject can be distinguished from the calculated value of Expression (1), so that the image processing apparatus 1300 extracts the region of the lymph vessels in the subject. Can be.

The image processing device 1300 as the state estimating means classifies the extracted lymphatic vessels by analyzing the spectral image. The image processing apparatus 1300 may, for example, divide the lymphatic vessel into a plurality of divided regions, and determine and classify each divided region by determining a state such as Shooting Star, shrinkage, stay, stop, DBF (Dermal backflow), and the like. Shooting Star is a healthy state in which lymph flows like a meteor. Contraction is a condition in which the width of a specific portion of a lymphatic vessel changes and pumps out lymph (fluid). Retention is a condition in which there is a period of time when lymph flow is not seen. Stagnation is a condition in which little lymph flows.

  DBF is a state in which lymph flows back toward the skin. DBF also includes conditions of interstitial leakage and lymphatic dilatation. Interstitial leakage is a condition in which lymph flows back and leaks into the interstitium. Lymphatic dilatation is a condition in which refluxing lymph remains in the dilated capillary lymphatics and pre-collecting lymphatics.

  The image processing apparatus 1300 is not limited to the state of the lymphatic vessels, and may be configured based on the number of lymphatic vessels per unit area, the ratio of lymphatic vessels per unit area, or the ratio of lymphatic vessels per unit volume. May be classified. The number of lymphatic vessels per unit area, the ratio of lymphatic vessels per unit area, and the ratio of lymphatic vessels per unit volume are hereinafter also referred to as the number of lymphatic vessels, area ratio, and volume ratio. In addition, the image processing apparatus 1300 may classify the lymphatic vessels based on the distance between the lymphatic vessels and the veins or the depth from the skin of the subject.

  The region of the lymphatic vessel may be classified automatically as described above, or may be classified manually. When the classification is performed manually, the image processing apparatus 1300 as the specifying unit can specify a part of the lymphatic vessel region and classify the specified region according to a user instruction.

  The image processing device 1300 as a display control unit causes the display device 1400 to display the classification result of the lymphatic vessels. The image processing apparatus 1300 may display, for example, the region of the lymphatic vessel with a hue corresponding to the state of each divided region. In addition, the image processing apparatus 1300 may display the number of lymph vessels, the area ratio, or the volume ratio for each unit area of the subject so that the user can confirm it. The image processing device 1300 may display the distance between the lymph vessels and the veins and the depth of the lymph vessels and the veins from the skin.

  The image processing apparatus 1300 serving as a storage control unit stores the classification result of the lymphatic vessels in the storage device 1200 in association with the analyzed image data and the patient information. When displaying the image data or the patient information on the display device 1400, the image processing device 1300 can acquire the corresponding lymphatic vessel classification result from the storage device 1200 and display it together with the image data.

  As described above, at least one of the image processing apparatus 1300 and the computer 150 as the information processing apparatus includes a spectral image acquiring unit, an area determining unit, a photoacoustic image acquiring unit, a state estimating unit, a specifying unit, a display controlling unit, It functions as a device having at least one of the storage control means. In addition, each means may be comprised by mutually different hardware, and may be comprised by one hardware. Further, a plurality of units may be configured by one piece of hardware.

  In the present embodiment, by selecting a wavelength at which the image value corresponding to the contrast agent has a negative value, the blood vessel and the contrast agent can be identified.However, the image value corresponding to the contrast agent is a blood vessel and the contrast agent. Can be any value as long as the image value corresponding to the contrast agent can be identified. For example, the image processing described in this step can be applied to a case where the image value of the spectral image (oxygen saturation image) corresponding to the contrast agent becomes smaller than 60% or larger than 100%. .

(Example 1)
In the first embodiment, the image processing apparatus 1300 analyzes the image data generated based on the received signal data of the photoacoustic wave generated from the inside of the subject by irradiating the subject with light, thereby automatically forming the lymphatic vessels. Classify and estimate lymphatic condition. The image processing device 1300 causes the display device 1400 to display the classification result. Hereinafter, the image processing method according to the first embodiment will be described with reference to the flowchart illustrated in FIG.

(S1211: Step of extracting lymphatic region)
The image processing device 1300 as a state estimating unit extracts a region of a lymph vessel from image data. The image data for extracting the region of the lymphatic vessel may be, for example, a spectral image generated using a plurality of photoacoustic images corresponding to a plurality of wavelengths. FIG. 11 is a diagram illustrating a spectral image of the subject. The method for acquiring the spectral image shown in FIG. 11 will be described later. In the spectral image illustrated in FIG. 11, both the lymphatic vessel A1 and the vein A2 into which the contrast agent has been introduced are imaged. The lymphatic vessels A1 and the veins A2 can be distinguished and visually recognized by assigning at least one of hue, lightness, and saturation corresponding to each image value. Therefore, the image processing apparatus 1300 can extract a region of a lymph vessel by image analysis. Note that the image data for extracting the lymphatic vessel region may be a photoacoustic image derived from a single wavelength. Even in a photoacoustic image derived from a single wavelength, imaging of lymph vessels is possible, and the image processing apparatus 1300 can extract a region of lymph vessels by image analysis. An example of a method for extracting lymphatic vessels using a photoacoustic image derived from a single wavelength will be described. Of the images including the image data group corresponding to each of the multiple times of light irradiation, the region where the change in the image value in the photoacoustic image within a predetermined period is large reflects the intermittent lymph fluid flow described above. Therefore, it is possible that the region is a region of a lymphatic vessel. In addition, in the photoacoustic image as a three-dimensional image, the reference value of the image value derived from hemoglobin and a contrast agent corresponding to the depth and the thickness of the structure may be stored in the computer 150 in advance, and the lymphatic vessel or the blood vessel may be used. Can be identified.

(S1212: Step of Classifying Lymphatic Vessels)
Lymph vessels are classified based on various indicators such as the state of lymph flow and the distance from veins. By confirming these classification results, the user can specify a lymph vessel to be anastomosed in an anastomosis operation for connecting a lymph vessel and a vein. A method for classifying lymphatic vessels is exemplified below.

[Lymphatic vessel classification method 1]
A method of classifying lymph vessels using the state of the lymph vessels as an index will be described with reference to FIG. Here, an example is shown in which the state of the lymphatic vessel is determined based on a temporal change in the luminance value. FIG. 12 shows a lymphatic vessel A1 and a vein A2. The image processing apparatus 1300 divides the lymphatic vessel A1 into a predetermined length, and extracts divided areas A101, A102, and A103. The divided areas A101, A102, and A103 are approximated by, for example, a Hessian matrix, a gradient vector, or a Hough transform, and the major axis direction and the minor axis direction are determined, respectively.

  For example, in each divided region, a divided region in which a portion having a higher luminance value moves in the long-axis direction with time can be determined to be in the state of Shooting Star. In addition, it is possible to determine that a divided region in which a portion having a higher luminance value becomes narrower or wider in the short axis direction is in a contracted state. A divided region having a time zone in which the luminance value does not change can be determined to be in a stagnant state. The divided area where the luminance value does not change can be determined to be in a stationary state.

  When the divided area is in the state of DBF, whether it is interstitial leakage or lymphatic vessel dilation can be determined by, for example, the spatial frequency of the image. If the spatial frequency of the image is lower than the threshold value, it can be determined that the state is interstitial leakage, and if it is higher than the threshold value, it is determined that the state is lymphatic dilatation.

  In this way, the lymphatic vessels can be classified using the condition as an index. The user can determine the degree of health of the lymph vessels based on the state of the lymph vessels, select the lymph vessels to be anastomosed, and determine the anastomosis position.

In this example, the time change of the luminance value in the image is used, but the state of the lymphatic vessels is determined based on information corresponding to the image values such as the hue, lightness, and saturation other than the luminance value. Is also good. That is, in this example, it can be said that the state of each divided area is determined based on the time change of the image value in each divided area.

[Lymphatic vessel classification method 2]
A method of classifying lymph vessels using the number of lymph vessels per unit area, the area ratio, and the volume ratio as indices will be described with reference to FIG. FIG. 13 shows three lymph vessels A1a, A1b, and A1c. Each square block shown in FIG. 13 indicates a region corresponding to a unit area. The image processing apparatus 1300 analyzes the image data to calculate the number of lymph vessels present and the area ratio of the lymph vessels to the unit area for each unit area (for example, 2 cm ² ) of the subject. When the image data is an image representing a three-dimensional spatial distribution, the image processing apparatus 1300 can calculate the volume ratio of the lymphatic vessels (in the unit volume) to the unit area.

  In the example of FIG. 13, each block is color-coded according to the number of lymph vessels present. That is, the block B1 having two lymph vessels, the block B2 having one lymph vessel, and the block B3 having no lymph vessel are shown in different colors. The image processing apparatus 1300 is not limited to the number of lymph vessels existing per unit area, and displays each block in different colors according to the area ratio of lymph vessels per unit area or the volume ratio of lymph vessels per unit volume. Is also good.

  As described above, lymph vessels can be classified using the number of existing per unit area, the area ratio, and the volume ratio as indices. The user can select an anastomosis target lymph vessel and determine an anastomosis position in consideration of the number of lymph vessels, the area ratio, and the volume ratio.

[Lymphatic vessel classification method 3]
A method of classifying lymph vessels using the distance between the lymph vessels and veins as an index will be described with reference to FIG. The image processing apparatus 1300 extracts the lymph vessels A1 and veins A2 displayed in the image data and calculates the distance between them. The image processing apparatus 1300 can display the distance between the lymphatic vessel A1 and the vein A2, as shown in FIG. The distance between the lymphatic vessel A1 and the vein A2 may be a distance in an image representing a two-dimensional spatial distribution or a distance in an image representing a three-dimensional spatial distribution. The position at which the distance between the lymphatic vessel A1 and the vein A2 is displayed may be specified by the user. Further, the distance between the lymph vessel A1 and the vein A2 may be displayed at predetermined intervals along the lymph vessel A1. In this case, the image processing apparatus 1300 may not display the distance at a position where the distance between the lymphatic vessel A1 and the vein A2 exceeds a predetermined threshold.

  Furthermore, the image processing apparatus 1300 may highlight the positions where the lymphatic vessels A1 and the veins A2 intersect in plan view (A111 and A112 in FIG. 14). Further, a position where the distance between the lymph vessel A1 and the vein A2 calculated in the three-dimensional image data is short may be highlighted. Further, the lymphatic vessels A1 and the veins A2 may be displayed by assigning luminance according to the depth from the skin. In this way, the lymphatic vessels can be classified using the distance from the vein and the depth from the skin as indices. The user can select the lymph vessel to be anastomosed and determine the anastomosis position based on the distance between the lymph vessel and the vein or the depth from the skin. The user can select each index described above according to the position of the region of interest. The image processing device 1300 can cause the display device 1400 to display the region of the lymphatic vessel classified by the selected index. Further, a position where the distance between the lymph vessel and the vein calculated in the three-dimensional image data is short may be highlighted.

(S1213: Step of Displaying Classification Result)
When the image processing apparatus 1300 as the display control unit classifies the lymphatic vessels as an index (lymphatic vessel classification method 1), the divided regions of the lymphatic vessels can be displayed in hues corresponding to the states. When the image processing apparatus 1300 classifies the number of lymph vessels per unit area, the area ratio, and the volume ratio as indices (lymphatic vessel classification method 2), the image processing apparatus 1300 divides each block indicating the unit area by the number of lymph vessels, the area The color may be displayed in a hue corresponding to the value of the ratio or the volume ratio. When the image processing apparatus 1300 classifies the distance between the lymph vessel and the vein as an index (lymphatic vessel classification method 3), the image processing apparatus 1300 may display the distance between the lymph vessel and the vein. The image processing device 1300 may display the distance by assigning at least one of a luminance value, a hue, a lightness, and a saturation according to the depth from the skin, in addition to indicating the distance between the lymph vessel and the vein. At this time, it is preferable that the index assigned to the information on the depth from the skin be an index that can be distinguished from other information from the viewpoint of visibility. For example, when assigning a hue to the information indicating the state of the lymphatic vessels, an index other than the hue is assigned to the information on the depth from the skin. That is, at least one of the luminance value, hue, brightness, and saturation according to the state of the lymphatic vessel is assigned to the image value of the lymphatic vessel area, and the luminance value and the hue excluding those assigned to the state of the lymphatic vessel are assigned. , Brightness, and saturation are assigned to information about the depth from the subject's skin.

  In addition, the image processing apparatus 1300 evaluates indexes such as the state of the lymphatic vessels, the number of existing per unit area, the distance from the vein, and the depth from the skin, and highlights the lymphatic vessels and veins suitable for anastomosis. It may be. A lymphatic vessel suitable for anastomosis is preferably a lymphatic vessel in which lymph flows and is healthy (for example, in the state of Shoting Star), has a shorter distance from a vein, and has a smaller depth from the skin. . The image processing apparatus 1300 identifies lymphatic vessels satisfying predetermined conditions as lymphatic vessels suitable for anastomosis, based on indicators such as the state of lymphatic vessels, the number per unit area, distance from veins, and depth from skin. can do. The image processing device 1300 may further evaluate whether or not the lymphatic vessel is suitable for anastomosis based on the state in the upstream and downstream regions of the lymphatic vessel. By highlighting lymph vessels suitable for anastomosis, the user can select lymph vessels more suitable for anastomosis.

(S1214: Step of saving data)
The image processing device 1300 as the storage control unit may store the classification result of the lymphatic vessels in S1212 in the storage device 1200 in association with the analyzed image data and patient information. In this case, the image processing device 1300 can display the classification result of the lymph vessels stored in the storage device 1200 on the display device 1400 together with the image data. The image processing apparatus 1300 can display the classification result of the lymphatic vessels in a mode (for example, FIGS. 13 and 14) according to the index selected by the user. The user can repeatedly check the classification result of the lymphatic vessels associated with the patient information. The patient information may include information on physiotherapy performed on the patient in addition to the above-described patient ID. This makes it easy for the user to grasp changes in the state of the lymphatic vessels due to physiotherapy. Further, an interface that allows the user to select which mode to employ may be added to the GUI shown in FIGS. 10 and 16.

(Method of acquiring spectral images)
The method for acquiring the spectral image shown in FIG. 11 (first acquisition method) will be described below.
Since the region in which the contrast agent introduced into the body is present can be depicted by the spectral image, the lymphatic vessels into which the contrast agent has been introduced can be depicted. However, there is a case where the position of the lymphatic vessel cannot be correctly indicated by only one image. This is because the lymph flow is not as constant as blood.

Blood circulates constantly due to the pulsation of the heart, but there is no common organ that acts as a pump in the lymph vessels, and the smooth muscle inside the lymph vessel walls that make up the lymph vessels contracts. The transport of lymph fluid is performed. In addition to the contraction of the smooth muscle of the lymph vessel wall, which occurs once every few tens of seconds to several minutes, the lymph fluid also contains muscle contractions that occur with the movement of humans, pressure caused by relaxation, pressure changes caused by respiration, and external pressure. It moves due to massage stimulation. Therefore, the movement timing of the lymph fluid is not constant, but becomes an intermittent flow with an irregular sense, for example, once every several tens of seconds to several minutes. Even if a spectral image is acquired at a time when lymph fluid is not moving, the lymph vessels cannot be drawn because only a sufficient amount of contrast agent is not present in the lymph vessels, or only a part of the lymph vessels is drawn. I am concerned that I cannot do it. In other words, with only one frame image in the moving image, only a portion of the lymphatic vessel where the contrast agent exists may be depicted.

  Therefore, in the system according to the present embodiment, a plurality of spectral images (a plurality of first image data) are acquired in a time series in a predetermined period, and the lymphatic vessels are determined based on the acquired plurality of spectral images. The existing area (that is, the area through which the contrast agent passes) is extracted. In the present embodiment, the photoacoustic apparatus 1100 acquires a plurality of time-series spectral images in the processes of steps S500 to S800 and stores the spectral images in the storage device 1200. In addition, it is preferable that the predetermined period is longer than the cycle in which the movement of lymph occurs (for example, longer than about 40 seconds to 2 minutes).

Step S800 is a step of generating a moving image based on a plurality of spectral images.
By displaying a plurality of spectral images as moving images, a user of the apparatus can observe a state in which lymph fluid moves. However, since the lymph fluid flows intermittently in the lymphatic vessels, only a part of the spectral images that can be used for confirming the flow of the lymph fluid among the plurality of spectral images acquired in a time series is used. That is, when the spectral image is displayed only with the moving image, the user must keep watching the screen until the movement of the lymph occurs. Furthermore, since the movement of the lymph (contrast agent) at one time is short, it is difficult for the user to accurately grasp the position of the lymph vessel on the screen.

  Therefore, in the present embodiment, after executing step S800, the image processing apparatus 1300 generates a still image (second image data) indicating the position of the lymphatic vessel based on the plurality of spectral images. The spectral image of FIG. 11 shows the position of the lymphatic vessel specified in this way.

  Next, after the process of step S800 is completed, the image processing device 1300 acquires a plurality of spectral images (spectral images of a plurality of frames) stored in the storage device 1200, and displays an image representing a region where the lymphatic vessels are present. Generate.

  In this step, first, for each of a plurality of spectral images obtained in a time series, a region where the image value is within a predetermined range is extracted. In the example described above, a set of pixels in which the image value that is the calculated value of Expression (1) is a negative value is extracted. As a result, as shown in FIG. 18A, an area (shown by a black line) is extracted for each frame of the moving image, that is, for each spectral image forming the moving image. The extracted area is an area where a contrast agent exists in each frame. Note that FIG. 18 illustrates a two-dimensional image, but when the spectral image is a three-dimensional spectral image, an area may be extracted from the three-dimensional space.

  Then, the regions obtained for each frame are superimposed (combined) to generate a region corresponding to the lymphatic vessel. When the regions shown in FIG. 18A are overlapped, a region (reference numeral 1101) corresponding to the lymphatic vessel is obtained as shown in FIG. 18B.

The image processing device 1300 generates and outputs an image (second image data) representing the position of the lymphatic vessel based on the region generated in this manner. When an image representing the position of the lymphatic vessel is generated, a hue corresponding to the original image value (that is, the image value of the spectral image) may be given, or the image may be highlighted by applying a unique marking. Is also good. Further, a luminance corresponding to the absorption coefficient may be given. The absorption coefficient can be obtained from the photoacoustic image used when generating the spectral image.

  The generated image may be output on the same screen as the GUI shown in FIG. 10 or may be output on another screen. The second image may be a three-dimensional image or a two-dimensional image. Further, an interface for storing the second image data generated as described above in the image server 1210, the storage device 1200, or the like may be added to the GUI illustrated in FIG. Since the second image data has a smaller data amount than the first image data which is a moving image, even when using a terminal having a low processing capacity, it is easy to grasp the position of the lymphatic vessel. Can be.

  According to the first spectral image acquisition method, it is possible to provide a user such as a doctor with a still image representing the position of a lymph vessel. Since the lymph (contrast agent) moves periodically, simply adding (or averaging) a plurality of spectral images cannot accurately indicate the position of the lymphatic vessel. On the other hand, in the present embodiment, information in the time direction is compressed in order to extract and combine an area having an image value within a predetermined range from each frame of the spectral image. Thereby, the position of the lymphatic vessel can be accurately depicted.

  In the illustrated embodiment, the region where the image value of the spectral image is within a predetermined range is extracted, but the region may be extracted using other conditions. For example, a photoacoustic image (an image representing an absorption coefficient) corresponding to a spectral image may be referred to, and an area whose luminance value is below a predetermined threshold may be excluded. This is because even if the image value of the spectral image is within a predetermined range, the region where the absorption coefficient is small is highly likely to be noise. Further, the threshold value of the luminance value for performing the filtering may be changeable by the user.

  In the present embodiment, the wavelength (two wavelengths) of the irradiation light is selected such that the image value of the pixel corresponding to the blood vessel region becomes positive and the image value of the pixel corresponding to the contrast agent region becomes negative. Any two wavelengths may be selected such that the signs of both image values in the spectral image are reversed.

(Another way to acquire spectral images)
Another method (second acquisition method) for acquiring the spectral image shown in FIG. 11 will be described below.
In the method described above, in a process after step S800, an area extraction process is performed on each frame of the spectral image acquired in time series, and a plurality of extracted areas are combined. On the other hand, it is conceivable to directly extract a region that satisfies the condition within a predetermined period by referring to a plurality of frames of the spectral image acquired in time series.

  In the present example, in a process after step S800, a plurality of spectral images included in a predetermined period are selected, and an area where the image value falls within a predetermined range within the predetermined period (in the above-described example, the image (A region where the value is a negative value) is extracted. An area where the image value falls within the predetermined range within the predetermined period can be said to be an area where the contrast agent has passed. In addition, it is preferable that the predetermined period is longer than the cycle in which the movement of lymph occurs (for example, longer than about 40 seconds to 2 minutes).

FIG. 19 is a diagram exemplifying a temporal change of an image value at a certain pixel P (x, y) in a spectral image within a predetermined period. The illustrated pixels are to be extracted because the image values are within a predetermined range.
As described above, the contrast agent region may be extracted based on the image value that changes within a predetermined period. Note that when making the determination, a peak hold or the like of the image value of the photoacoustic image may be performed within a predetermined period.

Note that, as a countermeasure against noise, in the second acquisition method, similarly to the first acquisition method, a region where the absorption coefficient is smaller than a predetermined value may be excluded. That is, a region where the image value of the spectral image is within a predetermined range and the brightness of the corresponding photoacoustic image is above the threshold may be set as the extraction target.
Further, in order to reduce noise, an area in which a predetermined time has elapsed while the above-described condition is satisfied may be set as an extraction target. Further, the above-mentioned fixed time may be adjustable by the user.

(Example 2)
In the first embodiment, the image processing apparatus 1300 automatically classifies the lymph vessels and estimates the state of the lymph vessels by performing image analysis on image data including the area of the lymph vessels. In contrast, in the second embodiment, the user specifies a part of the lymphatic vessel region in the image data including the lymphatic vessel region, and determines the state of the specified region (hereinafter, referred to as a region of interest). The image processing apparatus 1300 displays, on the display device 1400, an input interface for the user to input information such as a determination result for the region of interest. The user can input data relating to the region of interest, such as the state of the region of interest and findings regarding the region of interest, via the input interface. The information input by the user is stored in the storage device 1200 in association with the image data. The information input by the user may be stored in the storage device 1200 in association with the corresponding region of interest. Hereinafter, an image processing method according to the second embodiment will be described with reference to a flowchart illustrated in FIG.

(Step S1221: Step of Specifying Lymphatic Vessel Region)
First, the image processing apparatus 1300 as a specifying unit extracts a region of a lymphatic vessel from image data, similarly to the process of S1211 in the first embodiment. The image processing apparatus 1300 specifies a part of the extracted lymphatic vessel region as a region of interest. The image processing apparatus 1300 can set, for example, a region of a predetermined length including a position designated by the user as the region of interest. Note that the image processing apparatus 1300 can divide the lymphatic vessel region into a plurality of regions of interest by repeating the processing shown in FIG. 15, and can receive input of information for each region of interest.

  The image processing device 1300 may specify the region of interest based on a user's instruction. For example, in the image data displayed on the display device 1400, the user can indicate the position of the region of interest by pointing the region to be specified among the regions of the lymphatic vessels with a pointing device such as a mouse. For example, the image processing apparatus 1300 may specify a region of a predetermined length including a position designated by the user as the region of interest. Further, the image processing apparatus 1300 may allow the user to specify the positions of the start point and the end point to specify the region of interest.

  A GUI for allowing the user to specify the position of the region of interest will be described with reference to FIG. The item 3100 displays image data to be analyzed. In the example of FIG. 16, the item 3100 displays a lymphatic vessel A1 and a vein A2. Note that the image data displayed on the item 3100 may be a moving image.

The user points the position to be specified as the region of interest with the mouse. In the example of FIG. 16, the position pointed by the user is indicated by an arrow 3110. The image processing apparatus 1300 enlarges and displays a square area centered on the position pointed by the user, that is, an area 3120 surrounded by a dotted line on the item 3200. The region of the lymphatic vessel included in the region 3120 is the specified region of interest. The size of the region of interest (the length of the identified lymphatic vessel) may be specified by the user, or may be determined by the image processing apparatus 1300 to a predetermined size. When the region of interest specified by the user includes a plurality of lymph vessels, the image processing apparatus 1300 may change the region of interest so that only one of the lymph vessels is included. Note that a moving image corresponding to the area 3120 may be displayed on the item 3200. Furthermore, when the image shown in the item 3100 is a moving image, the image can be synchronized with the moving image in the area 3120, so that the user can observe the image at the same time. Item 3300 and item 3400 will be described in the process of S1222.

(S1222: Step of Receiving Input of Lymphatic Vessel Classification)
The image processing apparatus 1300 as a display control unit displays on the display device 1400 an input interface that receives an input for the region of interest specified in S1221. An item 3300 and an item 3400 illustrated in FIG. 16 correspond to an input interface that receives an input for a region of interest.

  The item 3300 is an input interface for inputting the state of the region of interest displayed on the item 3200. Item 3300 includes “running lymph vessels” and “DBF” tabs. FIG. 16 shows a state where the “traveling lymphatic vessel” tab is selected. The user can select any one of Shooting Star, contraction, stay, and stop as the state of the attention area in the item 3300. When the “DBF” tab is selected, for example, the state of interstitial leakage and lymphatic dilatation are displayed as options.

  The item 3400 is an input interface for inputting a finding for the region of interest displayed on the item 3200. The input interface is not limited to the state of the region of interest and the findings for the region of interest, and may be an interface that accepts input of various information such as the degree of suitability as an anastomotic position in lymphatic venule anastomosis.

When a plurality of lymph vessels are included in the region of interest, an interface that allows the user to specify the state of the lymph vessels and the lymph vessels to be found in the item 3100 or the item 3200 is input. It may be. By storing the information of the identified lymphatic vessels along with the state of the lymphatic vessels and findings, it is possible to easily understand which lymphatic vessels were evaluated for later observation. Become.

(S1223: Step of Displaying Classification Result)
The image processing apparatus 1300 as a display control unit can display the lymph vessels A1 in the item 3100 by color-coding the regions of interest based on the state selected in the item 3300 as the classification result of the lymph vessels A1. .

  A display example of the classification result of lymph vessels according to the second embodiment will be described with reference to FIG. FIG. 17 shows an example in which the classification result is displayed on the item 3100 of the GUI shown in FIG. The item 3100 displays a lymphatic vessel A1 and a vein A2. The example of FIG. 17 shows a state in which the region of interest A121, the region of interest A122, and the region of interest A123 are specified in the lymphatic vessel A1. The region A124 is a region of an unclassified lymphatic vessel that is not specified as a region of interest. As shown in the legend, the region of interest A121 is a Shooting Star, the region of interest A122 is in a stagnant state, and the region of interest A123 is in a contracted state. The unclassified area A124 may be displayed by blinking, for example. The image processing apparatus 1300 can prompt the user to instruct the classification of the lymph vessels by blinking the unclassified area. Note that the method of displaying the region specified as the region of interest and the region not specified as the region of interest in different modes is not limited to blinking display. For example, the same effect can be obtained by displaying an unclassified region in a color different from the color assigned to the classified region, or by displaying a frame indicating the region.

(S1224: Step of saving data)
The image processing apparatus 1300 may associate the classification result of the lymphatic vessels in S1212 with the analyzed image data and the information on the patient and store the result in the storage device 1200. The image processing apparatus 1300 can display the classification result of the lymphatic vessels stored in the storage device 1200 on the display device 1400 together with the image data. When the image data is a moving image, the user can check his or her classification result while reproducing the moving image.

  The flow illustrated in FIG. 15 exemplifies a process of inputting information such as a state or a finding to one region of interest. By repeating the flow shown in FIG. 15 for an unclassified region, the region of the lymphatic vessel is divided into a plurality of regions of interest and classified according to each state. The step of displaying the classification result (S1223) and the step of saving the data (S1224) are executed for each attention area in the flow shown in FIG. 15, but are executed after the input to a plurality of attention areas is completed. Is also good.

(Other Examples)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program and reads the program. This is the process to be performed.

1100 Photoacoustic device 1300 Image processing device

Claims (29)

An image processing apparatus that processes image data generated based on a photoacoustic wave generated from within the subject by irradiating the subject with light,
Display control means for displaying a display device with the image data and an input interface that receives an input for a region of interest that is a part of the region of the lymph vessel in the subject in the image data,
Storage control means for storing the image data in the storage device in association with information input via the input interface,
An image processing apparatus comprising: 2. The apparatus according to claim 1, further comprising a specifying unit configured to specify a part of the lymphatic vessel region as the region of interest by performing image analysis on the image data including the lymphatic vessel region in the subject. 3. Image processing device. The image processing apparatus according to claim 2, wherein the specifying unit specifies the region of interest based on a user's instruction. The image processing apparatus according to claim 1, wherein the input interface receives an input of information regarding a state of the region of interest. The image processing apparatus according to claim 1, wherein the input interface receives an input of information regarding a finding regarding the region of interest. The said display control means displays the area | region which is not specified as the said area | region of attention among the area | regions of the said lymphatic vessel in a different mode from the said area | region of interest, The Claim 1 characterized by the above-mentioned. An image processing apparatus according to claim 1. 7. The display control device according to claim 1, wherein the display control unit causes the display device to display information stored in the storage device in association with the image data displayed on the display device. An image processing apparatus according to claim 1. The image data is time-series three-dimensional image data generated based on photoacoustic waves generated by a plurality of light irradiations on the subject and including images corresponding to each of the plurality of light irradiations. is there,
The image processing apparatus according to claim 1, wherein:   The image processing apparatus according to claim 8, wherein the display control unit displays the image data as a moving image. The image processing device according to claim 9, wherein the display control unit is capable of repeatedly displaying the moving image at a predetermined speed. The image processing device according to claim 9, wherein the display control unit is capable of fast-forward displaying the moving image. The image processing apparatus according to claim 9, wherein the display control unit displays a GUI for receiving a change in a frame rate of the moving image. The display control means displays information of lymph flowing through the lymph vessel in the moving image in association with an area of the lymph vessel, and displays the information by at least one of a luminance display, a color display, a graph display, and a numerical display. The image processing apparatus according to claim 9, wherein: 14. The image processing apparatus according to claim 9, wherein the display control unit highlights at least one region of the lymphatic vessel. Generating image data based on photoacoustic waves generated from within the subject by irradiating the subject with light,
A display control to display a display device with the image data and an input interface for receiving an input to a region of interest that is a part of a region of the lymph vessel in the subject in the image data;
An image processing method, comprising: associating the image data with information input via the input interface and storing the image data in a storage device. 16. The method according to claim 15, further comprising: identifying a part of the lymphatic vessel region as the region of interest by performing image analysis on the image data including the lymphatic vessel region in the subject. Image processing method. 17. The image processing method according to claim 16, wherein the region of interest is specified based on a user's instruction. 18. The image processing method according to claim 15, wherein the input interface receives an input of information on a state of the region of interest. 19. The image processing method according to claim 15, wherein the input interface receives an input of information on a finding regarding the region of interest. The display control according to any one of claims 15 to 19, wherein, in the lymphatic vessel region, a region not specified as the region of interest is displayed in a mode different from the region of interest. The image processing method described in the above. 21. The display device according to claim 15, wherein the display control displays information stored in the storage device in association with the image data displayed on the display device on the display device. The image processing method described in the above. The image data is time-series three-dimensional image data generated based on photoacoustic waves generated by a plurality of light irradiations on the subject and including images corresponding to each of the plurality of light irradiations. is there,
The image processing method according to any one of claims 15 to 21, wherein:   The image processing method according to claim 22, wherein the display control displays the image data as a moving image. The image processing method according to claim 23, wherein the display control is capable of repeatedly displaying the moving image at a predetermined speed. The image processing method according to claim 23, wherein the display control is capable of fast-forwarding display of the moving image. 26. The image processing method according to claim 23, wherein the display control displays a GUI for receiving a change in a frame rate of the moving image. The display control is to display information of lymph flowing through the lymph vessel in the moving image in association with a region of the lymph vessel, and to display at least one of a luminance display, a color display, a graph display, and a numerical display. The image processing method according to any one of claims 23 to 26, wherein: 28. The image processing method according to claim 23, wherein the display control highlights at least one region of the lymphatic vessel.   A program for causing a computer to execute the image processing method according to any one of claims 15 to 28.

Images