JP2018011928A - Control device, control method, control system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device capable of acquiring an ultrasonic image and a photoacoustic image and displaying one of the images on the basis of the other image while the other image is observed.SOLUTION: A first image is acquired on the basis of a first signal acquired by first means, and the first image is analyzed and a region of interest is detected. Acquisition of a second signal by a second means is controlled in order to acquire a second image that includes the region of interest on the basis of the acquired second signal, and the first image and the second image are displayed in a display unit in a manner enabling comparison of the first image with the second image.SELECTED DRAWING: Figure 3

Description

  The disclosure of the present specification relates to a control device, a control method, a control system, and a program.

  Ultrasonic imaging devices and photoacoustic imaging devices are used as imaging devices that image the inside of a subject in a minimally invasive manner. In Patent Document 1, in a device that acquires an ultrasonic signal and a photoacoustic signal, a process for generating a photoacoustic image from a photoacoustic signal is performed for a specific region identified based on an ultrasonic image, It is disclosed that processing for generating a photoacoustic image from a photoacoustic signal is not performed for a region other than this region.

Japanese Unexamined Patent Publication No. 2015-198888

  The photoacoustic signal is based on an acoustic wave generated by expansion generated inside the subject due to light irradiated on the subject. Therefore, only by generating a photoacoustic image from a photoacoustic signal only in a region where a photoacoustic image is desired to be acquired, light is irradiated even on a region where a photoacoustic image is not acquired.

  The control device disclosed in this specification outputs an ultrasonic signal by transmitting / receiving ultrasonic waves to / from a subject, and receives a photoacoustic wave generated by light irradiation on the subject, from a probe that outputs a photoacoustic signal. , Signal acquisition means for acquiring at least one of the ultrasonic signal and the photoacoustic signal, generation means for generating an ultrasonic image based on the ultrasonic signal, and the generated ultrasonic image. Irradiation control means for controlling the light irradiation of the probe on the basis thereof.

  According to the present invention, in a control device capable of controlling different types of imaging, it is possible to control whether or not the other type of imaging is performed based on an image obtained by one type of imaging. It is possible to perform a test that is not redundant for the user or the subject.

It is a figure which shows an example of a structure of the control apparatus which concerns on 1st Embodiment. It is a flowchart which shows an example of the process performed by the control apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the image displayed by the control apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the image displayed by the control apparatus which concerns on 2nd Embodiment. It is a flowchart which shows an example of the process performed by the control apparatus which concerns on 2nd Embodiment. It is a figure which shows an example of the image displayed by the control apparatus which concerns on 3rd Embodiment. It is a flowchart which shows an example of the process performed by the control apparatus which concerns on 3rd Embodiment. It is a figure which shows an example of the test | inspection performed by the control apparatus which concerns on 4th Embodiment. It is a flowchart which shows an example of the process performed by the control apparatus which concerns on 4th Embodiment. It is a figure which shows an example of a structure of the system containing the control apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
In the specification of the present application, an acoustic wave generated by irradiating a subject with light and expanding inside the subject is referred to as a photoacoustic wave. In addition, an acoustic wave transmitted from the transducer or a reflected wave (echo) in which the transmitted acoustic wave is reflected inside the subject is referred to as an ultrasonic wave.

  As a method for imaging the state inside the subject in a minimally invasive manner, an imaging method using ultrasonic waves and an imaging method using photoacoustic waves are used. The method of imaging using ultrasonic waves is, for example, that the ultrasonic waves oscillated from the transducer are reflected by the tissue inside the subject according to the difference in acoustic impedance, and the time until the reflected wave reaches the transducer and the reflected wave This is a method for generating an image based on intensity. An image imaged using ultrasound is hereinafter referred to as an ultrasound image. The user can operate while changing the angle of the probe and observe ultrasonic images of various cross sections in real time. Ultrasound images depict the shapes of organs and tissues and are used to find tumors. The imaging method using photoacoustic waves is a method of generating an image based on photoacoustic waves generated by adiabatic expansion of tissue inside a subject irradiated with light, for example. An image imaged using the photoacoustic wave is hereinafter referred to as a photoacoustic image. In the photoacoustic image, information related to optical characteristics such as the degree of light absorption of each tissue is depicted. In photoacoustic images, for example, it is known that blood vessels can be drawn by the optical characteristics of hemoglobin, and its use for evaluating the malignancy of tumors is being studied.

  In order to increase the accuracy of diagnosis, various information may be collected by imaging different phenomena on the same part of the subject based on different principles. For example, there is a case where diagnosis about cancer is performed by combining morphological information obtained from a CT (Computed Tomography) image and functional information related to metabolism obtained from a PET (Positron emission tomography) image. As described above, it is considered effective to improve diagnosis accuracy to perform diagnosis using information obtained by imaging different phenomena based on different principles.

  As for the above-described ultrasonic image and photoacoustic image, an imaging device for obtaining an image in which respective characteristics are combined has been studied. In particular, since both an ultrasonic image and a photoacoustic image are imaged using ultrasonic waves from a subject, it is also possible to perform imaging of an ultrasonic image and photoacoustic image with the same imaging device. . More specifically, it can be configured such that the reflected wave and the photoacoustic wave irradiated to the subject are received by the same transducer. As a result, it is possible to acquire an ultrasonic signal and a photoacoustic signal with a single probe, and realize an imaging device that captures an ultrasonic image and a photoacoustic image without complicating the hardware configuration. it can.

  In an imaging apparatus that captures an ultrasound image and a photoacoustic image, it is assumed that the user wants to operate the probe in the same manner as a conventional ultrasound image. That is, it is conceivable that the user touches the surface of the subject and operates the probe while observing an image displayed based on information acquired by the probe. At that time, if the operation mode related to signal acquisition or image display is switched via, for example, a switch provided on the probe or an input device provided on the console of the imaging device, the user observes the image. It is necessary to interrupt the probe operation. For this reason, it is conceivable that the body movement of the subject occurs during the operation input to the switch or the input device of the console, or the probe position shifts.

  For example, as in the above-described example, consider a case where an ultrasonic image and a photoacoustic image are observed in combination to evaluate the malignancy of a tumor. A user operates a probe while observing an ultrasound image, finds a site that may be a tumor, acquires a photoacoustic image, and collects blood vessel information. At this time, in order to switch to the operation mode for displaying the photoacoustic image, the probe from a position where it is possible to observe the part considered to be a tumor during the operation input to the switch or the input device of the console described above. May shift. An object of the first embodiment is to provide a control device that can switch an image to be displayed without deteriorating operability when a user observes an image.

  FIG. 10 is a diagram illustrating an example of a configuration of a system including the control device 101 according to the first embodiment. An imaging system 100 that can generate an ultrasonic image and a photoacoustic image is connected to various external devices via a network 110. Each configuration and various external devices included in the imaging system 100 do not need to be installed in the same facility, and may be connected to be communicable.

  The imaging system 100 includes a control device 101, a probe 102, a detection unit 103, a display unit 104, and an operation unit 105. The control apparatus 101 is an apparatus that acquires an ultrasonic signal and a photoacoustic signal from the probe 102, controls acquisition of the photoacoustic signal based on, for example, an ultrasonic image, and generates a photoacoustic image based on the control. . In addition, the control device 101 acquires information related to an examination including imaging of an ultrasonic image and a photoacoustic image from the ordering system 112, and controls the probe 102, the detection unit 103, and the display unit 104 when the examination is performed. The control device 101 outputs the generated ultrasonic image, photoacoustic image, and superimposed image obtained by superimposing the photoacoustic image on the ultrasonic image to the PACS 113. The control device 101 transmits and receives information to and from external devices such as the ordering system 112 and the PACS 113 in accordance with standards such as HL7 (Health level 7) and DICOM (Digital Imaging and Communications in Medicine). Details of processing performed by the control device 101 will be described later.

  The region in the subject from which an ultrasound image is captured by the imaging system 100 is, for example, a circulatory region, a breast, a liver, or a pancreas. In the imaging system 100, for example, an ultrasound image of a subject to which an ultrasound contrast agent using microbubbles is administered may be captured.

  Moreover, the area | region in the subject from which a photoacoustic image is imaged with the imaging system 100 is areas, such as a circulatory organ area | region, a breast, a diameter part, an abdominal part, extremities including fingers and toes, for example. In particular, a blood vessel region including a new blood vessel and a plaque on a blood vessel wall may be set as a target for imaging a photoacoustic image in accordance with the characteristics relating to light absorption in the subject. In the following, a case where a photoacoustic image is captured while capturing an ultrasonic image will be described as an example. However, a region in a subject where a photoacoustic image is captured by the imaging system 100 is not necessarily a region where an ultrasonic image is captured. Does not have to match. In the imaging system 100, for example, a photoacoustic image of a subject to which a dye such as methylene blue or indocyanine green, gold fine particles, or a substance obtained by integrating or chemically modifying them is administered as a contrast agent is captured. May be.

  The probe 102 is operated by a user and transmits an ultrasonic signal and a photoacoustic signal to the control device 101. The probe 102 includes a transmission / reception unit 106 and an irradiation unit 107. The probe 102 transmits an ultrasonic wave from the transmission / reception unit 106 and receives the reflected wave by the transmission / reception unit 106. Further, the probe 102 irradiates the subject with light from the irradiation unit 107, and the photoacoustic wave is received by the transmission / reception unit 106. The probe 102 converts the received reflected wave and photoacoustic wave into an electric signal, and transmits it to the control device 101 as an ultrasonic signal and a photoacoustic signal. The probe 102 is controlled so that, when information indicating contact with the subject is received, transmission of ultrasonic waves for acquiring an ultrasonic signal and light irradiation for acquiring a photoacoustic signal are executed. It is preferable.

  The transceiver 106 includes at least one transducer (not shown), a matching layer (not shown), a damper (not shown), and an acoustic lens (not shown). The transducer (not shown) is made of a material exhibiting a piezoelectric effect, such as PZT (lead zirconate titanate) or PVDF (polyvinylidene difluoride). The transducer (not shown) may be other than a piezoelectric element, such as a transducer using a capacitive micro-machined transducer (CMUT) or a Fabry-Perot interferometer. Typically, the ultrasonic signal has a frequency component of 2 to 20 MHz and the photoacoustic signal has a frequency component of 0.1 to 100 MHz, and a transducer (not shown) that can detect these frequencies is used. The signal obtained by the transducer (not shown) is a time-resolved signal. The amplitude of the received signal represents a value based on the sound pressure received by the transducer at each time. The transmission / reception unit 106 includes a circuit (not shown) or a control unit for electronic focusing. The array form of transducers (not shown) is, for example, a sector, a linear array, a convex, an annular array, or a matrix array.

  The transmission / reception unit 106 may include an amplifier (not shown) that amplifies a time-series analog signal received by a transducer (not shown). The transmission / reception unit 106 may include an A / D converter that converts a time-series analog signal received by a transducer (not shown) into a time-series digital signal. The transducer (not shown) may be divided into a transmitter and a receiver depending on the purpose of imaging an ultrasonic image. Further, the transducer (not shown) may be divided into an ultrasonic image capturing unit and a photoacoustic image capturing unit.

  The irradiation unit 107 includes a light source (not shown) for acquiring a photoacoustic signal and an optical system (not shown) that guides pulsed light emitted from the light source (not shown) to the subject. The pulse width of light emitted from a light source (not shown) is, for example, 1 ns or more and 100 ns or less. Moreover, the wavelength of the light which a light source (not shown) inject | emits is a wavelength of 400 nm or more and 1600 nm or less, for example. When imaging a blood vessel in the vicinity of the surface of the subject with a high resolution, a wavelength of 400 nm or more and 700 nm or less and a large absorption in the blood vessel is preferable. Moreover, when imaging the deep part of a test object, the wavelength of 700 nm or more and 1100 nm or less which is hard to be absorbed by tissues such as water and fat is preferable.

  The light source (not shown) is, for example, a laser or a light emitting diode. The irradiation unit 107 may use a light source that can convert wavelengths in order to acquire a photoacoustic signal using light of a plurality of wavelengths. Alternatively, the irradiation unit 107 may include a plurality of light sources that generate light of different wavelengths, and may be configured to be able to irradiate light of different wavelengths alternately from each light source. The laser is, for example, a solid laser, a gas laser, a dye laser, or a semiconductor laser. As a light source (not shown), a pulsed laser such as an Nd: YAG laser or an alexandrite laser may be used. Further, a Ti: sa laser or an OPO (optical parametric oscillators) laser that uses Nd: YAG laser light as excitation light may be used as a light source (not shown). A microwave source may be used as a light source (not shown).

  In the optical system (not shown), an optical element such as a lens, a mirror, or an optical fiber is used. When the subject is a breast, it is preferable to irradiate with the beam diameter of the pulsed light expanded, so the optical system (not shown) may include a diffusion plate that diffuses the emitted light. Alternatively, in order to increase the resolution, the optical system (not shown) may include a lens or the like so that the beam can be focused.

  The detection unit 103 acquires information regarding the position and orientation of the probe 102. The detection unit 103 transmits information related to the position of the probe 102 to the control device 101. The detection unit 103 is a motion sensor provided in the probe 102, for example. The detection unit 103 is not necessarily included in the control device 101, and the sensor may be switched between ON and OFF as appropriate based on various conditions set prior to the inspection.

  The display unit 104 displays an image captured by the imaging system 100 and information related to the inspection based on control from the control device 101. The display unit 104 provides an interface for receiving user instructions based on control from the control device 101. The display unit 104 is a liquid crystal display, for example.

  The operation unit 105 transmits information related to user operation input to the control device 101. The operation unit 105 is, for example, a keyboard, a trackball, and various buttons for performing operation inputs related to inspection.

  Note that the display unit 104 and the operation unit 105 may be integrated as a touch panel display. Moreover, the control apparatus 101, the display part 104, and the operation part 105 do not need to be separate apparatuses, and may be implement | achieved as an operation console with which these structures were integrated. The control device 101 may have a plurality of probes.

  A HIS (Hospital Information System) 111 is a system that supports hospital operations. The HIS 111 includes an electronic medical record system, an ordering system, and a medical accounting system. By HIS111, it is possible to manage from inspection order issuance to accounting. The ordering system of the HIS 111 transmits order information to the ordering system 112 for each department. The ordering system 112, which will be described later, manages the execution of the order.

  The ordering system 112 is a system that manages inspection information and manages the progress of each inspection in the imaging apparatus. The ordering system 112 may be configured for each department that performs inspection. The ordering system 112 is, for example, a RIS (Radiology Information System) in the radiation department. In response to an inquiry from the control apparatus 101, the ordering system 112 transmits information on examinations performed by the imaging system 100 to the control apparatus 101. The ordering system 112 receives information related to the progress of the inspection from the control device 101. When the ordering system 112 receives information indicating that the inspection is completed from the control device 101, the ordering system 112 transmits information indicating that the inspection is completed to the HIS 111. The ordering system 112 may be integrated into the HIS 111.

  A PACS (Picture Archiving and Communication System) 113 is a database system that holds images obtained by various imaging devices inside and outside the facility. The PACS 113 manages a storage unit (not shown) that stores medical images and imaging conditions of such medical images, additional parameters such as image processing parameters including reconstruction, and patient information, and information stored in the storage unit. A controller (not shown). The PACS 113 stores an ultrasonic image, a photoacoustic image, and a superimposed image output from the control device 101. It is preferable that communication between the PACS 113 and the control device 101 and various images stored in the PACS 113 comply with standards such as HL7 and DICOM. Various images output from the control device 101 are stored with associated information associated with various tags in accordance with the DICOM standard.

  The Viewer 114 is a terminal for image diagnosis, reads an image stored in the PACS 113 or the like, and displays it for diagnosis. The doctor displays an image on the Viewer 114 for observation, and records information obtained as a result of the observation as an image diagnosis report. The diagnostic imaging report created using the Viewer 114 may be stored in the Viewer 114, or may be output and stored in the PACS 113 or a report server (not shown).

  The printer 115 prints an image stored in the PACS 113 or the like. The Printer 115 is, for example, a film printer, and outputs an image stored in the PACS 113 or the like by printing it on a film.

  FIG. 1 is a diagram illustrating an example of the configuration of the control device 101. The control device 101 includes a CPU 131, ROM 132, RAM 133, DISK 134, USB 135, communication circuit 136, GPU 137, HDMI 138, and probe connector port 139. These are connected by BUS130 so that communication is possible.

  The BUS 130 is a data bus, and is used to transmit and receive data between connected hardware and to transmit commands from the CPU 131 to other hardware.

  A CPU (Central Processing Unit) 131 is a control circuit that integrally controls the control device 101 and each unit connected thereto. The CPU 131 performs control by executing a program stored in the ROM 132. Further, the CPU 131 executes a display driver which is software for controlling the display unit 104 and performs display control on the display unit 104. Further, the CPU 131 performs input / output control for the operation unit 105.

  A ROM (Read Only Memory) 132 stores a program and data storing a control procedure by the CPU. The ROM 132 has a boot program 140 of the control device 101 and various initial data 141. In addition, various modules 142 to 150 for realizing the processing of the control apparatus 101 are included. Various modules for realizing the processing of the control apparatus 101 will be described later.

  A RAM (Random Access Memory) 133 provides a working storage area when the CPU 131 performs control by an instruction program. The RAM 133 has a stack 151 and a work area 152. The RAM 133 stores a program for executing processing in each unit connected to the control device 101 and various parameters used in image processing. The RAM 133 stores a control program executed by the CPU 131, and temporarily stores various data when the CPU 131 executes various controls.

  The DISK 134 is an auxiliary storage device that stores various data such as ultrasonic images and photoacoustic images. The DISK 134 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

  A USB (Universal Serial Bus) 205 is a connection unit that connects to the operation unit 105.

  The communication circuit 136 is a circuit for communicating with each unit constituting the imaging system 100 and various external devices connected to the network 110. The communication circuit 136 stores output information in a transfer packet, for example, and outputs the information to an external device via the network 110 by a communication technique such as TCP / IP. The control device 101 may have a plurality of communication circuits in accordance with a desired communication form.

  The GPU 137 is included in a general-purpose graphics board including a video memory. The GPU 137 executes part or all of the image processing module 145 and performs, for example, a photoacoustic image reconstruction process. By using such an arithmetic device, it is possible to perform operations such as reconstruction processing at high speed without requiring dedicated hardware.

  An HDMI (registered trademark) (High Definition Multimedia Interface) 138 is a connection unit connected to the display unit 104.

  The probe connector port 139 is a connection port for connecting the probe 102 to the control device 101. The ultrasonic signal and photoacoustic signal output from the probe 102 are acquired by the control device 101 via the port 139.

  The CPU 131 and the GPU 137 are examples of processors. The ROM 132, RAM 133, and DISK 134 are examples of memories. The control device 101 may have a plurality of processors. In the first embodiment, the function of each unit of the control device 101 is realized by the processor of the control device 101 executing a program stored in the memory.

  Note that the control device 101 may include a CPU or GPU that performs a specific process exclusively. Further, the control device 101 may include a field-programmable gate array (FPGA) in which specific processing or all processing is programmed. The control device 101 may have both an HDD and an SSD as the DISK 134.

  The modules 143 to 150 stored in the ROM 132 will be described. Modules 143 to 150 shown in FIG. 1 are obtained by extracting modules for executing processing related to the embodiment of the present invention. The control device 101 may include modules necessary for executing the inspection and operating the control device 101 other than those illustrated. In addition, each module may be configured as a combination of one or a plurality of programs. Also, some or all of the modules 143 to 150 may be stored in a memory other than the ROM 132 such as the DISK 134. Hereinafter, each module will be described in detail.

  The inspection control module 142 controls inspection performed in the imaging system 100. The inspection control module 142 acquires inspection order information from the ordering system 112. The examination order includes information on a patient who undergoes an examination and information on imaging procedures. The inspection control module 142 controls the probe 102 and the detection unit 103 based on information on the imaging technique. Further, the examination control module 142 displays information on the examination on the display unit 104 via the output module 150 in order to present information related to the examination to the user. The information on the examination displayed on the display unit 104 includes information on the patient undergoing the examination, information on the imaging technique included in the examination, and an image already generated after imaging. Further, the inspection control module 142 transmits information regarding the progress of the inspection to the ordering system 112. For example, when the inspection is started by the user, the system 112 is notified of the start, and when imaging by all the imaging techniques included in the inspection is completed, the system 112 is notified of the completion.

  The signal acquisition module 143 acquires an ultrasonic signal and a photoacoustic signal from the probe 102. Specifically, the signal acquisition module 143 distinguishes an ultrasonic signal and a photoacoustic signal from information acquired from the probe 102 based on information from the inspection control module 142, the image processing module 145, and the position acquisition module 149. get. For example, in the imaging technique in which imaging is performed, when the acquisition timing of the ultrasonic signal and the acquisition of the photoacoustic signal is defined, based on the acquisition timing information acquired from the inspection control module 142, An ultrasonic signal and a photoacoustic signal are distinguished and acquired from information acquired from the probe 102. When the photoacoustic signal is acquired by controlling the irradiation unit 107 based on the result of analyzing the ultrasonic image by the image processing module 145 as in the example described later, the signal acquired corresponding to the control is obtained. The photoacoustic signal is acquired separately from the ultrasonic signal. The signal acquisition module 143 is an example of an acquisition unit that acquires at least one of an ultrasonic signal and a photoacoustic signal from the probe 102.

  The signal acquisition module 143 has an irradiation control module 144. The irradiation control module 144 controls the light irradiation by the irradiation unit 107 based on the information regarding the imaging conditions acquired from the inspection control module 142 and the result of analysis of the ultrasonic image by the image processing module 145.

  The image processing module 145 is a module for performing processing for generating an image based on a signal acquired by the imaging system 100. The image processing module 145 includes an ultrasonic image generation module 146, a photoacoustic image generation module 147, and a superimposed image generation module 148.

  The image processing module 145 stores the images generated by the ultrasonic image generation module 146, the photoacoustic image generation module 147, and the superimposed image generation module 148 in the DISK 134 together with accompanying information. In addition, the image is stored in the external device by outputting the image together with the supplementary information to the external device via the output module 150.

  The ultrasonic image generation module 146 generates an ultrasonic image to be displayed on the display unit 104 from the ultrasonic signal acquired by the signal acquisition module 143. The ultrasonic image generation module 146 generates an ultrasonic image suitable for the set mode based on the imaging technique information acquired from the examination control module 142. For example, when the Doppler mode is set as an imaging technique, the ultrasound image generation module 146 determines whether or not the internal frequency of the subject is based on the difference between the frequency of the ultrasound signal acquired by the signal acquisition module 143 and the transmission frequency. An image showing the flow velocity is generated. The ultrasonic image generated by the ultrasonic image generation module 146 may be generated by any other method such as A mode, M mode, or Doppler mode, or may be a harmonic image or an elastography image. May be.

  Furthermore, in the first embodiment, the ultrasonic image generation module 146 analyzes the generated ultrasonic image, acquires a photoacoustic signal, and specifies a region where a photoacoustic image is to be generated. For example, the ultrasonic image generation module 146 analyzes the ultrasonic image and identifies an area where there is a possibility of calculus. In this viewpoint, the ultrasonic image generation module 146 functions as a detection unit that analyzes an ultrasonic image and specifies a region from which a photoacoustic image is acquired.

  The photoacoustic image generation module 147 generates a photoacoustic image based on the photoacoustic signal acquired by the signal acquisition module 143. The photoacoustic image generation module 147 reconstructs an acoustic wave distribution (hereinafter referred to as an initial sound pressure distribution) when light is irradiated based on the photoacoustic signal. The photoacoustic image generation module 147 obtains the light absorption coefficient distribution in the subject by dividing the reconstructed initial sound pressure distribution by the light fluence distribution of the subject irradiated with the light. . Further, the concentration distribution of the substance in the subject is obtained from the absorption coefficient distribution for a plurality of wavelengths by utilizing the fact that the degree of light absorption in the subject varies depending on the wavelength of the light irradiated to the subject. . For example, the photoacoustic image generation module 147 acquires the concentration distribution of substances in the subject of oxyhemoglobin and deoxyhemoglobin. Further, the photoacoustic image generation module 147 acquires the oxygen saturation distribution as a ratio of the oxyhemoglobin concentration to the deoxyhemoglobin concentration. The photoacoustic image generated by the photoacoustic image generation module 147 is an image indicating information such as the above-described initial sound pressure distribution, optical fluence distribution, absorption coefficient distribution, substance concentration distribution, and oxygen saturation distribution. The photoacoustic image may be any image generated by combining these.

  That is, the image processing module 145 is an example of a generation unit that generates an ultrasonic image based on an ultrasonic signal and generates a photoacoustic image based on a photoacoustic signal.

  The superimposed image generation module 148 generates a superimposed image obtained by superimposing the photoacoustic image generated by the photoacoustic image generation module 147 on the ultrasonic image generated by the ultrasonic image generation module 146. The superimposed image generation module 148 obtains a superimposed image by aligning the ultrasonic image and the photoacoustic image. For the alignment, information regarding the imaging condition acquired from the inspection control module 142 or the position of the probe 102 acquired from the position acquisition module 149 described later may be used. In addition, the alignment may be performed based on a region that is depicted in common for the ultrasonic image and the photoacoustic image.

  The position acquisition module 149 acquires information related to the position of the probe 102 based on information from the detection unit 103. The position acquisition module 149 obtains at least one of information on the speed of movement of the probe 102 with respect to the subject, information on the speed of rotation, and information indicating the degree of pressure on the subject based on the change over time of the information on the position. You may get it. When the detection unit 103 transmits the information on the probe 102 to the control device 101, the position acquisition module 149 preferably acquires the position information on the probe 102 at regular time intervals, preferably in real time.

  Further, the position acquisition module 149 may acquire information regarding the probe 102 used for imaging. Information related to the probe 102 includes information such as the type of probe, center frequency, sensitivity, acoustic focus, electronic focus, and observation depth. The position acquisition module 149 appropriately transmits information regarding the position of the probe 102 and information regarding the probe 102 to the inspection control module 142, the image processing module 145, and the output module 150.

  The output module 150 outputs information for displaying a screen on the display unit 104, and outputs information to an external device via the network 110.

  The output module 150 controls the display unit 104 to display information on the display unit 104. The output module 150 displays information on the display unit 104 in response to an input from the inspection control module 142 or the image processing module 145 or a user operation input via the operation unit 105. The output module 150 is an example of a display control unit.

  The output module 150 outputs information from the control device 101 to an external device such as the PACS 113 via the network 110. For example, the output module 150 outputs the ultrasonic image and the photoacoustic image generated by the image processing module 145 and a superimposed image thereof to the PACS 113. The image output from the output module 150 includes incidental information attached as various tags according to the DICOM standard by the inspection control module 142. The incidental information includes, for example, patient information, information indicating the imaging device that captured the image, an image ID for uniquely identifying the image, and an examination ID for uniquely identifying the examination that captured the image Is included. Further, the incidental information includes information that associates an ultrasonic image captured in the same examination with a photoacoustic image. The information associating the ultrasonic image and the photoacoustic image is information indicating a frame having the closest timing at which the photoacoustic image is acquired, for example, among a plurality of frames constituting the ultrasonic image. Furthermore, as the incidental information, the position information of the probe 102 acquired by the detection unit 103 may be incidental to each frame of the ultrasonic image and the photoacoustic image. That is, the output module 150 outputs information indicating the position of the probe 102 that has acquired the ultrasonic signal for generating the ultrasonic image, attached to the ultrasonic image. Also, the output module 150 outputs information indicating the position of the probe 102 that has acquired the photoacoustic signal for generating the photoacoustic image, attached to the photoacoustic image. The output module 150 is an example of an output unit.

  FIG. 2 is a flowchart illustrating an example of processing of the control device 101 for controlling light irradiation based on the acquired ultrasonic image and acquiring a photoacoustic image. In the following processes, the CPU 131 or the GPU 137 is a main body that realizes the processes by the modules unless otherwise specified.

  In step S <b> 201, the ultrasonic image generation module 146 is executed to acquire an ultrasonic image. Specifically, first, the inspection control module 142 is executed prior to the inspection, whereby the inspection order is acquired from the ordering system 112. The examination order includes information on the patient to be examined, information on the part to be examined, and information on the imaging technique. The user operates the probe 102, and an ultrasonic signal is transmitted from the probe 102 to the control device 101. By executing the signal acquisition module 143, the ultrasonic signal is acquired by the control device 101. By executing the ultrasonic image generation module 146, an ultrasonic image is generated based on the ultrasonic signal. By executing the output module 150, the ultrasonic image is displayed on the display unit 104. The user can further operate the probe 102 while observing the ultrasonic image displayed on the display unit 104.

  In step S202, the inspection control module 142 is executed to acquire information indicating whether to acquire a photoacoustic signal based on the ultrasonic image. The information indicating whether or not to acquire the photoacoustic signal based on the ultrasonic image is specifically information set in advance by the user or included in the inspection order. In another example, whether to acquire a photoacoustic signal may be determined according to the state of the load of processing performed in the control apparatus 101. For example, when the processing load performed by the control device 101 is heavy and the acquisition of the photoacoustic signal affects the processing of acquiring the ultrasound image, You may determine not to acquire an acoustic signal. If the photoacoustic signal is acquired based on the ultrasound image, the process proceeds to step S203. If not, the process returns to step S201 to continue acquiring the ultrasound image.

  In step S203, the ultrasonic image generation module 146 is executed to analyze the ultrasonic image. For example, the ultrasound image generation module 146 analyzes the ultrasound image and detects a region of interest defined in advance from the regions depicted in the ultrasound image. For example, an organ such as a calculus, a tumor, or a blood vessel is set in the region of interest. The calculus includes the case of a virtual image that can be depicted by ultrasonic imaging. A computer diagnosis support system can also be used in combination to search for a site of interest. The analysis of the ultrasound image and the region to be extracted as the region of interest are not limited to the above-described examples, and any form may be used as long as it is possible to detect a region that is considered to be beneficial to acquire a photoacoustic image in addition to the ultrasound image. An example is also acceptable.

  An example of the analysis performed based on the ultrasound image generation module 146 in step S203 will be described. Here, a case where a B-mode image that is a two-dimensional tomographic image is analyzed will be described as an example. The ultrasonic image includes mottled noise called a speckle pattern. Therefore, in the analysis, first, processing for reducing speckle is performed. For example, speckle is regarded as a noise component, and speckle is reduced by a spatial filter such as moving average or median. Alternatively, speckle is reduced by a filter that uses the property of Rayleigh distribution, which is a property peculiar to speckle patterns, and the mask shape is locally variable. In addition, speckles may be reduced using a multi-resolution filter or a filter by numerical simulation. Then, segmentation for extracting a target region is performed on the image with reduced speckles by threshold processing and differentiation processing on the density value. Alternatively, focusing on the shape of the tissue, for example, segmentation using a variable shape model may be performed. As another example of the analysis, segmentation may be performed based on a speckle pattern that is a property unique to an ultrasound image. As a method focusing on the features of the speckle pattern, for example, texture analysis using a feature amount based on a co-occurrence matrix for density values, or a statistic obtained from a parameter of a logarithmic compression K distribution is used as a feature amount for a non-uniform echo. One example is a probability distribution method. In addition, in the analysis, information on internal echo, shape, boundary echo, backward echo, and outer shadow may be used.

  While the ultrasonic image is analyzed in step S203, the ultrasonic signal is continuously acquired by the signal acquisition module 143, the ultrasonic image is continuously generated by the ultrasonic image generation module 146, and the output module 150. Then, the ultrasonic image may be continuously displayed on the display unit 104.

  In step S204, by executing the ultrasonic image generation module 146, it is determined whether to acquire a photoacoustic image based on the analysis result in step S203. If the region of interest is extracted in step S203, the process proceeds to step S205. If the region of interest is not extracted, the process returns to step S201 to continue acquiring the ultrasound image.

  In step S205, the irradiation control module 144 is executed to determine whether or not to irradiate the subject with light. Specifically, the irradiation control module 144 determines whether or not the probe 102 is in contact with the subject. The irradiation control module 144 determines contact between the subject and the probe 102 based on the ultrasonic image generated by the ultrasonic image generation module 146 and the position information of the probe 102 acquired by the position acquisition module 149. The probe 102 is provided with a sensor (not shown) for detecting contact with the subject, and the irradiation control module 144 determines contact between the subject and the probe 102 based on information from the sensor (not shown). Also good. The irradiation control module 144 controls the irradiation unit 107 to emit light when it is determined that the subject and the probe 102 are in contact with each other. When it is determined that the subject and the probe 102 are not in contact, a screen for notifying the user that the subject and the probe 102 are not in contact is displayed on the display unit 104 via the output module 150. May be. The irradiation control module 144 further irradiates light based on the ultrasonic image generated by the ultrasonic image generation module 146 when the region extracted as the region of interest in step S203 is drawn on the ultrasonic image. The irradiation unit 107 may be controlled as described above. Thereby, when the probe 102 leaves | separates from the position which can draw the area | region analyzed in step S203, possibility of performing redundant light irradiation can be reduced. If it is determined in step S205 that the subject is to be irradiated with light, the process proceeds to step S206. If it is determined that the object is not irradiated, the process returns to step S201 to continue acquiring the ultrasound image.

  In step S <b> 206, the signal acquisition module 143 is executed to acquire a photoacoustic signal from the probe 102.

  In step S207, the photoacoustic image generation module 147 is executed to reconstruct a photoacoustic image from the photoacoustic signal acquired in step S206. Then, when the output module 150 is executed, the reconstructed photoacoustic image is displayed on the display unit 104. Furthermore, by controlling the photoacoustic image generation module 147 to execute the superimposed image generation module 148, a superimposed image may be generated and displayed on the display unit 104 via the output module 150. .

  For example, morphological information inside the subject is depicted in an ultrasonic image such as a B-mode image. On the other hand, functional information such as the amount of hemoglobin in the blood vessel is depicted in the photoacoustic image. In the first embodiment, a superimposed image generated according to a predetermined overlapping condition is displayed on the display unit 104. The superimposing conditions are, for example, conditions such as a color for displaying each image, a superimposing range, and transparency in a superimposed image in which an ultrasonic image is a base image and a photoacoustic image is a layer image.

  As described above, the control apparatus 101 displays the ultrasonic image and the photoacoustic image on the display unit 104 so that the image of the region of interest specified based on the form information and the function information can be referred to in association with each other. Can do. Thereby, the workflow for a user such as a doctor to observe a medical image such as an ultrasonic image or a photoacoustic image of a subject and perform a diagnosis can be improved.

  The irradiation control module 144 may irradiate the subject with light based on the user's operation input. When the light is irradiated, the photoacoustic image may be generated by the photoacoustic image generation module 147 and the photoacoustic image may be displayed on the display unit 104 via the output module 150.

  Further, the photoacoustic image generated by the photoacoustic image generation module 147 and the superimposed image generated by the superimposed image generation module 148 may be appropriately stored in the DISK 134 or the PACS 113.

  FIG. 3 is a diagram illustrating an example of analysis performed in step S203 illustrated in FIG. 2 and an example of an image displayed in step S207. Here, a case where a region with a possibility of calculus is depicted in an ultrasonic image will be described as an example.

  FIG. 3A is a diagram schematically illustrating the internal structure of the subject. An area 301 is an area drawn on an ultrasonic image generated based on the ultrasonic signal when the ultrasonic signal is acquired by bringing the probe into contact with a certain position with respect to the subject. It is assumed that a calculus 302 and a blood vessel 303 exist inside the subject. At this time, the calculus 302 is located outside the region 301.

  FIG. 3B is an example of an ultrasonic image 304 generated by imaging the region 301 illustrated in FIG. In the ultrasound image 304, a blood vessel region 306 corresponding to the blood vessel 303 inside the subject is depicted. Furthermore, an image of the calculus 302 existing outside the region 301 is depicted as a virtual image 305 in the ultrasonic image 304. A virtual image is an image in which a structure that does not originally exist inside the subject is depicted on the image. There are several possible causes for the virtual image being depicted in the ultrasound image. In the case illustrated in FIG. 3B, it is considered that this is caused by weak ultrasonic waves that are irradiated outside the ultrasonic waves in the main direction transmitted from the probe 102, called side lobes. When a reflected wave reflected by a structure such as a calculus is detected by a probe 102, an ultrasonic image 304 as if the calculus 302 is present in the region 301 depicted by the ultrasonic waves in the main direction is obtained. Generated. A user who observes the ultrasound image 304 must determine whether or not the virtual image region 305 is a virtual image. In general, the user often determines whether the image is a virtual image while operating the probe 102 and changing the imaging range of the ultrasonic image. In step S203, the picked-up ultrasonic image is analyzed to detect, for example, a region with a possibility of calculus.

  FIG. 3C is an example of a photoacoustic image 307 generated by imaging the region 301 illustrated in FIG. That is, the photoacoustic image 307 is a photoacoustic image generated based on the photoacoustic signal acquired by the processing in steps S204 to S206 illustrated in FIG. In the photoacoustic image 307, a blood vessel region 308 corresponding to the blood vessel 303 inside the subject is depicted. The photoacoustic image 307 does not depict the image caused by the calculus 302 or the influence on the blood vessel region 308. A possible reason for the feature resulting from the calculus 302 not being depicted in the photoacoustic image 307 is that, for example, the laser beam emitted from the irradiation unit 107 has higher straightness than the ultrasonic wave emitted from the transmission / reception unit 106. It is done. As described above, when the virtual image region 305 that is an image with the possibility of a calculus is a virtual image, the feature due to the calculus 302 is not drawn at the position corresponding to the virtual image region 305 in the photoacoustic image 307. Therefore, the user can refer to the information obtained from the photoacoustic image when determining whether or not the area considered to have a calculus is a virtual image.

  In addition, although FIG.3 (c) demonstrated the example which produces | generates the photoacoustic image 307 of the same area | region as the area | region 301 which imaged the ultrasonic image 304, it does not restrict to this. For example, light may be irradiated only in the vicinity of the area detected as a calculus area in step S203, and a photoacoustic image only in the vicinity may be generated. By setting only the vicinity of the virtual image region 305 as a target for acquiring the photoacoustic signal, it is possible to reduce the time for irradiating the subject with light and the load on the resources of the control apparatus 101.

  FIG. 3D is a diagram illustrating an example of a superimposed image 309 in which the photoacoustic image 307 is superimposed on the ultrasonic image 304. For example, in step S207 illustrated in FIG. 2, the photoacoustic image is displayed by displaying the superimposed image 309. FIG. 3D shows an example in which the photoacoustic image 310 of the region corresponding to the region detected as the region of interest in step S <b> 203 is superimposed on the ultrasonic image 304 and displayed. The method of displaying the photoacoustic image, here, that is, the superimposing method can be set in advance by the user. For example, the photoacoustic image is displayed in a color corresponding to the intensity of the photoacoustic wave. As a result, a blood vessel having the characteristic of absorbing the irradiated light and generating a photoacoustic wave is depicted in the photoacoustic image. In the photoacoustic image 310, only a blood vessel is depicted on a blood vessel region 311 corresponding to the blood vessel 303, and an image resulting from the calculus 302 is not depicted. Thereby, the user can compare an ultrasonic image and a photoacoustic image regarding the region of interest with the possibility of calculus. The control device 101 can assist the user in diagnosis related to the region of interest. For example, the control device 101 can assist in determining whether or not a region with a possibility of calculus is a virtual image.

  Although the example which detects the specific characteristic drawn on an ultrasonic image was demonstrated using FIG.2 and FIG.3, this invention is not limited to this. For example, in step S203, an ultrasonic image is analyzed using the technique described in “Removal of a virtual image in an ultrasonic image using fuzzy image processing” (Medical Imaging Technology, Vol. 14, No. 5, 1996). A virtual image may be detected. Further, in step S207, the detected virtual image may be removed and displayed. The user can determine whether or not the detected virtual image region is a virtual image using the photoacoustic image. Even when the virtual image is removed and the ultrasonic image is displayed, the photoacoustic image is displayed in a comparable manner, so that the user can visually recognize that the virtual image has been removed from the ultrasonic image. In addition, as described above, the control device 101 may determine whether or not a region having a possibility of a virtual image detected in the ultrasonic image is a virtual image based on information drawn in the photoacoustic image.

  According to the configuration of the first embodiment, the control apparatus 101 analyzes the acquired ultrasonic image to detect a region of interest, and generates a photoacoustic image corresponding to at least the region of interest. Thereby, an image useful for diagnosing a region of interest can be efficiently captured. Further, since the control device 101 emits light when a region of interest is detected, redundant light irradiation can be reduced.

[Second Embodiment]
In the first embodiment, the example in which the irradiation of light to the subject is controlled based on the result of analyzing the B-mode image as an example of the ultrasonic image has been described. In the second embodiment, an example in which an elastography image is used as an example of an ultrasound image will be described. It is considered that the region where imaging by elastography is performed is a region of interest focused on by the user. By generating an ultrasonic image and a photoacoustic image of a region of interest, it is useful for diagnosis of the region of interest, and the inspection workflow can be improved. Details will be described below.

  Elastography is a method for imaging tissue hardness according to the principle described below. In general, elastography takes into account hardness based on Hooke's law, and measures tissue strain due to externally applied stress. For example, when the probe 102 is pressed from the body surface, there is a property that the softer tissue is deformed more greatly. If the displacement of the tissue before and after the pressurization is measured and differentiated, the strain at each point of the tissue can be obtained. An elastography image is an image of the strain distribution at each point of tissue. For example, an elastography image is a two-dimensional image expressed by changing the hue so that a portion with a large distortion (soft portion) is red and a portion with a small distortion (hard portion) is blue via an intermediate green color. It is.

  For example, when a breast is used as a subject, it is considered that adipose tissue is soft and a site calcified by breast cancer or the like is hard. Thus, knowing the hardness of the tissue in the subject is useful information for diagnosis. Further, for example, it is said that a tumor tissue generates a lot of new blood vessels around it, and it is considered useful for diagnosis to use blood vessel information obtained by a photoacoustic image together with an elastography image. Hereinafter, an example in which the photoacoustic image is acquired by controlling the irradiation of light to the subject based on the result of analyzing the elastography image which is an example of the ultrasonic image will be described.

  FIG. 5 is a flowchart illustrating an example of processing of the control device 101 for controlling light irradiation based on the acquired ultrasonic image and acquiring a photoacoustic image. In the following processes, the CPU 131 or the GPU 137 is a main body that realizes the processes by the modules unless otherwise specified.

  In step S501, an ultrasonic image is acquired by executing the ultrasonic image generation module 146. Specifically, first, the inspection control module 142 is executed prior to the inspection, whereby the inspection order is acquired from the ordering system 112. The examination order includes information on the patient to be examined, information on the part to be examined, and information on the imaging technique. The user operates the probe 102, and an ultrasonic signal is transmitted from the probe 102 to the control device 101. By executing the signal acquisition module 143, the ultrasonic signal is acquired by the control device 101. By executing the ultrasonic image generation module 146, an ultrasonic image is generated based on the ultrasonic signal. By executing the output module 150, the ultrasonic image is displayed on the display unit 104. The user can further operate the probe 102 while observing the ultrasonic image displayed on the display unit 104.

  In step S <b> 502, the signal acquisition module 143 is executed to determine whether or not imaging by elastography has been performed. For example, while observing the ultrasonic image generated in step S501 and displayed on the display unit 104, the user finds a region where detailed observation is desired and performs imaging by elastography. For example, the user can switch the operation mode related to signal acquisition of the probe 102 to the operation mode of elastography imaging via the operation unit 105. Note that a switch or the like for switching the operation mode may be provided in the probe 102. The user switches to the elastography imaging operation mode and performs an operation of pressing the probe 102 against the subject. An ultrasonic signal is acquired from the probe 102 by the signal acquisition module 143. As a result, it is determined that elastography imaging has been performed. By executing the ultrasonic image generation module 146, based on the ultrasonic signal acquired by the signal acquisition module 143, the displacement of the tissue before and after pressing is acquired, and an elastography image reflecting the elasticity of the tissue is generated. Is done. If elastography imaging has been performed, the process proceeds to step S203. If not, the process returns to step S201, and acquisition of an ultrasonic image is continued.

  In step S503, the inspection control module 142 is executed to acquire information indicating whether to acquire a photoacoustic signal based on the ultrasonic image. Since the process of step S503 is the same as the process of step S202 illustrated in FIG. 2, the above description is used and the description is omitted here. When acquiring a photoacoustic signal based on an ultrasonic image, it progresses to step S504, and when not acquiring, it returns to step S501 and acquisition of an ultrasonic image is continued.

  In step S504, the ultrasound image generation module 146 is executed, whereby the elastography image acquired in step S502 is analyzed. For example, the ultrasound image generation module 146 analyzes the elastography image, and detects a region of interest defined in advance from the regions drawn on the elastography image.

  An example of the analysis performed based on the ultrasonic image generation module 146 in step S504 will be described. Here, an example in which an elastography image is analyzed and a region where a hard tissue may be depicted is detected as a region of interest is shown. As described above, the elastography image is obtained by imaging the strain distribution at each point of the tissue. For example, in this analysis, a set of pixels having a distortion equal to or less than a predetermined value is detected as an area where a hard tissue may be depicted. When the probe 102 is equipped with a pressure sensor (not shown) that measures the degree to which the probe 102 presses the subject, the signal acquisition module 143 or the ultrasonic image generation module 146 provides information indicating the degree of the pressing. To be acquired. In the analysis in step S504, information on the degree to which the probe 102 presses the subject may be used together. This is because the ease of displacement of each tissue changes according to the degree to which the probe 102 presses the subject. For example, the greater the degree that the probe 102 presses the subject, the smaller the predetermined distortion value may be.

  During the analysis of the elastography image in step S504, the signal acquisition module 143 continues to acquire an ultrasonic signal, and the ultrasonic image generation module 146 continues to acquire an ultrasonic wave such as a B-mode image or an elastography image. An image may be generated and an ultrasonic image may be continuously displayed on the display unit 104 by the output module 150.

  In step S505, when the ultrasonic image generation module 146 is executed, it is determined whether to acquire a photoacoustic image based on the analysis result in step S504. If a region of interest is extracted in step S504, the process proceeds to step S506, and if a region of interest is not extracted, the process returns to step S501 to continue acquiring an ultrasound image.

  In step S506, the irradiation control module 144 is executed to determine whether to irradiate the subject with light. Since the process of step S506 is the same as the process of step S205 illustrated in FIG. 2, description here is abbreviate | omitted by using the description mentioned above. If it is determined to irradiate light, the process proceeds to step S507. If it is determined not to irradiate, the process returns to step S501 to continue acquiring an ultrasound image.

  In step S507, the signal acquisition module 143 is executed to acquire a photoacoustic signal from the probe 102.

  In step S508, the photoacoustic image is generated from the photoacoustic signal acquired in step S507 by executing the photoacoustic image generation module 147. Then, when the output module 150 is executed, the reconstructed photoacoustic image is displayed on the display unit 104. Furthermore, by controlling the photoacoustic image generation module 147 to execute the superimposed image generation module 148, a superimposed image may be generated and displayed on the display unit 104 via the output module 150. .

  For example, a superimposed image obtained by superimposing a photoacoustic image on an elastography image is displayed on the display unit 104 in step S508. Here, the elastography image is generally expressed as a color image having a hue reflecting the degree of elasticity. In addition, in a photoacoustic image, for example, an image reflecting the concentration of a specific substance, for example, hemoglobin, is expressed as a color image of a hue reflecting the magnitude of the concentration. When the color images are superimposed and displayed, it is preferable to use different hues for the base image and the layer image, for example. Further, when the region expressed by the hue of the photoacoustic image and the region expressed by the hue of the elastography image overlap, the user confirms that the region is the region depicted in both images. It is preferable to make it visible.

  As described above, the control device 101 displays the ultrasonic image and the photoacoustic image on the display unit 104 so that the image of the region of interest specified based on the degree of elasticity of the tissue and the function information can be referred to. Can be displayed. For example, the user may determine whether a region that is considered to be a hard tissue when an elastography image is observed is a malignant tumor. At this time, the control apparatus 101 can assist a user's judgment by presenting, for example, information on a new blood vessel drawn in the photoacoustic image in a comparable manner. Thereby, the workflow for a user such as a doctor to observe a medical image such as an ultrasonic image or a photoacoustic image of a subject and perform a diagnosis can be improved.

  The irradiation control module 144 may irradiate the subject with light based on the user's operation input. When the light is irradiated, the photoacoustic image may be generated by the photoacoustic image generation module 147 and the photoacoustic image may be displayed on the display unit 104 via the output module 150.

  Further, the photoacoustic image generated by the photoacoustic image generation module 147 and the superimposed image generated by the superimposed image generation module 148 may be appropriately stored in the DISK 134 or the PACS 113.

  FIG. 4 is a diagram illustrating an example of the analysis performed in step S504 illustrated in FIG. Here, a case where a region having a possibility of a hard tissue is depicted in an elastography image will be described as an example.

  FIG. 4A is an example of an elastography image 401. It is the figure which illustrated typically the structure inside a subject. In the image 401, a hard tissue region 402 is displayed so as to be distinguishable from surrounding softer tissue.

  FIG. 4B is an example of the photoacoustic image 403 obtained by imaging the region depicted in the elastography image 401 shown in FIG. A blood vessel region 404 and a blood vessel region 405 are depicted in the photoacoustic image 403.

  FIG. 4C is an example of a superimposed image 406 in which the photoacoustic image 403 is superimposed on the elastography image 401. The blood vessel region 407 corresponds to the blood vessel region 404 depicted in the photoacoustic image illustrated in FIG. The tissue region 408 corresponds to the tissue region 402 depicted in the elastography image illustrated in FIG. FIG. 4C is an example of a superimposed image in which a photoacoustic image in the vicinity of a tissue region 408 that is considered to be harder than the surrounding tissue in the elastography image is superimposed. By acquiring the photoacoustic image only in the vicinity of the hard tissue region 402 that is the region of interest, the time for irradiating the subject with light and the load on the resources of the control apparatus 101 can be reduced.

  According to the configuration of the second embodiment, the control apparatus 101 analyzes the acquired ultrasonic image to detect a region of interest, and generates a photoacoustic image corresponding to at least the region of interest. Thereby, an image useful for diagnosing a region of interest can be efficiently captured. Further, since the control device 101 emits light when a region of interest is detected, redundant light irradiation can be reduced.

  In the second embodiment, the example using the elastography image that expresses the elasticity of the tissue qualitatively has been described, but the present invention is not limited to this. For example, an image generated by quantitative elastic imaging that quantitatively represents the elasticity of the tissue may be used. The propagation of sound waves is the propagation of wave energy, and a force called an acoustic radiation force is generated in the direction of sound wave propagation in an object that blocks the propagation of wave energy. Therefore, when a convergent ultrasonic pulse having a high sound pressure and a relatively long duration is radiated to a living body, a minute displacement occurs in the tissue due to the acoustic radiation force. At this time, a transverse wave is generated that propagates in a direction perpendicular to the displacement, that is, in a direction perpendicular to the ultrasonic beam. Since the propagation speed of the transverse wave is slower than that of the longitudinal wave, the propagation process of the transverse wave can be imaged by the pulse echo method, and the propagation speed can be obtained. It is considered that the propagation speed of the shear wave is higher as the tissue is harder, and thereby the hardness of the tissue can be quantitatively evaluated.

  Alternatively, an elastic modulus distribution, that is, a quantitative hardness index may be obtained and imaged based on the tissue strain distribution obtained by qualitative elastography and the tissue stress distribution. The tissue stress distribution cannot be directly measured, but may be obtained by anatomical information, simulation, or the like.

[Third Embodiment]
In the first embodiment and the second embodiment, the example in which the irradiation of light to the subject is controlled and the photoacoustic image is displayed based on the result of analyzing the ultrasonic image has been described. In the third embodiment, an example will be described in which irradiation of an ultrasonic beam to a subject is controlled based on a result of analyzing a photoacoustic image and an ultrasonic image is displayed. Below, the case where the elastography image which is an example of an ultrasonic image is acquired is demonstrated to an example.

  As described above, the photoacoustic image depicts a substance or tissue having the property of absorbing irradiated light and generating an acoustic wave (hereinafter referred to as optical characteristics). Depending on the lesion to be diagnosed, tissue features with optical properties can aid in diagnosis. For example, it is said that there are many new blood vessels in the vicinity of the tumor tissue, and there is a possibility that there is a correlation between a region where the density of thin blood vessels is high and the malignancy of the tumor. Further, there is a possibility that the concentration of a substance having optical characteristics is different in a specific lesion tissue as compared with the surrounding normal tissue. By observing the photoacoustic image, there may be a case where a region having a lesion having such characteristics can be identified. When a user observing a photoacoustic image finds a region that may have a lesion, i.e., a region of interest that requires more detailed observation, providing more detailed information about the region of interest It is considered useful for diagnosis. In the third embodiment, when a region of interest is depicted in a photoacoustic image, an ultrasonic signal can be acquired and an ultrasonic image can be displayed.

  FIG. 7 is a flowchart illustrating an example of processing of the control device 101 for controlling the irradiation of ultrasonic waves based on the acquired photoacoustic image and acquiring the ultrasonic image. In the following processes, the CPU 131 or the GPU 137 is a main body that realizes the processes by the modules unless otherwise specified.

  In step S701, the photoacoustic image generation module 147 is executed to acquire a photoacoustic image. Specifically, first, the inspection control module 142 is executed prior to the inspection, whereby the inspection order is acquired from the ordering system 112. The examination order includes information on the patient to be examined, information on the part to be examined, and information on the imaging technique. The user operates the probe 102, and a photoacoustic signal is transmitted from the probe 102 to the control device 101. By executing the signal acquisition module 143, the photoacoustic signal is acquired by the control device 101. By executing the photoacoustic image generation module 147, a photoacoustic image is generated based on the photoacoustic signal. By executing the output module 150, the photoacoustic image is displayed on the display unit 104. The user can further operate the probe 102 while observing the photoacoustic image displayed on the display unit 104.

  In step S702, the inspection control module 142 is executed to acquire information indicating whether to acquire an ultrasonic signal based on the photoacoustic image. The information indicating whether or not to acquire an ultrasonic signal based on the photoacoustic image is specifically information set in advance by the user or included in the inspection order. In another example, whether or not to acquire an ultrasonic signal may be determined according to the state of the load of processing performed by the control device 101. For example, when the processing load performed by the control device 101 is heavy and the acquisition of the photoacoustic signal affects the processing of acquiring the ultrasound image, You may determine not to acquire an acoustic signal. When acquiring a photoacoustic signal based on an ultrasonic image, it progresses to step S703, and when not acquiring, it returns to step S701 and acquisition of a photoacoustic image is continued.

  In step S703, the photoacoustic image is analyzed by executing the photoacoustic image generation module 147. For example, the photoacoustic image generation module 147 analyzes the photoacoustic image and detects a region of interest defined in advance from the region depicted in the photoacoustic image.

  An example of the analysis performed based on the photoacoustic image generation module 147 in step S703 is shown. Here, an example of analyzing an image (hereinafter referred to as an absorption coefficient image) reflecting an absorption coefficient for light of a specific wavelength will be described as an example of a photoacoustic image. For example, by irradiating light having a wavelength corresponding to the light absorption of hemoglobin, a blood vessel image is drawn on the absorption coefficient image. For example, in this analysis, the density of blood vessels in the region depicted in the photoacoustic image is analyzed. Then, a region where blood vessels are present at a predetermined density or more is detected as a region of interest. More specifically, for example, among the pixels included in the predetermined range of the photoacoustic image, a value obtained by dividing the number of pixels having a pixel value equal to or greater than a certain threshold value by the number of pixels constituting the predetermined range is used as the density. To do.

  Note that while the photoacoustic image is analyzed in step S703, the photoacoustic signal is continuously acquired by the signal acquisition module 143, the photoacoustic image is continuously generated by the photoacoustic image generation module 147, and the output module 150. Then, the photoacoustic image may be continuously displayed on the display unit 104.

  In step S704, by executing the photoacoustic image generation module 147, it is determined whether or not to acquire an ultrasonic image based on the analysis result in step S703. If the region of interest is extracted in step S703, the process proceeds to step S705. If the region of interest is not extracted, the process returns to step S701 to continue acquiring the ultrasound image. From this viewpoint, the photoacoustic image generation module 147 is an example of an analysis unit.

  In step S705, the irradiation control module 144 is executed to determine whether or not to irradiate the subject with ultrasonic waves. Specifically, the irradiation control module 144 determines whether or not the probe 102 is in contact with the subject. The irradiation control module 144 determines contact between the subject and the probe 102 based on the photoacoustic image generated by the photoacoustic image generation module 147 and the position information of the probe 102 acquired by the position acquisition module 149. The probe 102 is provided with a sensor (not shown) for detecting contact with the subject, and the irradiation control module 144 determines contact between the subject and the probe 102 based on information from the sensor (not shown). Also good. The irradiation control module 144 controls the irradiation unit 107 to irradiate ultrasonic waves when it is determined that the subject and the probe 102 are in contact with each other. When it is determined that the subject and the probe 102 are not in contact, a screen for notifying the user that the subject and the probe 102 are not in contact is displayed on the display unit 104 via the output module 150. May be. The irradiation control module 144 further irradiates the ultrasonic wave based on the photoacoustic image generated by the photoacoustic image generation module 147 when the region extracted as the region of interest in step S703 is drawn on the photoacoustic image. The transmitting / receiving unit 106 may be controlled to do so. Thereby, when the probe 102 leaves | separates from the position which can draw the area | region analyzed in step S703, possibility of performing redundant ultrasonic irradiation can be reduced. In another example, when the temperature of the probe 102 is equal to or lower than a predetermined value, it may be determined that the ultrasonic wave is irradiated. Due to the characteristics of the transmitter / receiver 106, when the probe 102 is separated from the subject, an air layer is formed between the probe 102 and the subject. The acoustic impedance of air is much larger than the acoustic impedance of the transmission / reception unit 106. If there is an air layer between the probe 102 and the subject, the ultrasonic waves are repeatedly reflected in the vicinity of the transmission / reception unit 106, and the temperature of the probe 102 may rise. The probe 102 may be provided with a temperature sensor that measures the temperature, and the irradiation control module 144 may acquire temperature information of the probe 102 from the temperature sensor. If it is determined in step S705 that the subject is to be irradiated with ultrasonic waves, the process proceeds to step S706, and if it is determined not to be irradiated, the process returns to step S701 to continue acquiring the photoacoustic image.

  In step S706, the signal acquisition module 143 is executed to acquire an ultrasonic signal from the probe 102. Then, by executing the ultrasonic image generation module 146, an ultrasonic image is generated from the ultrasonic signal. Then, when the output module 150 is executed, the generated ultrasonic image is displayed on the display unit 104. Furthermore, a superimposition image may be generated by controlling the ultrasonic image generation module 146 to execute the superimposition image generation module 148, and the superimposition image may be displayed on the display unit 104 via the output module 150. . The ultrasonic image generated in step S706 is, for example, a B mode image.

  In step S707, the ultrasonic image generation module 146 is executed, and it is determined whether or not imaging by elastography has been performed. Since the region of interest is depicted in the photoacoustic image, it is considered useful to display the elastography image on the display unit 104 as one piece of detailed information that assists diagnosis of the region of interest. For example, a screen is displayed on the display unit 104 via the output module 150 to notify the user that there is a region with high blood vessel density and that it is useful to perform elastography imaging. The user refers to the notification screen, presses the probe 102 against the subject, and performs elastography imaging. What is the process for determining whether the user has performed elastography photography? Since it is the same as the process of step S502 illustrated in FIG. 5, description here is abbreviate | omitted by using the description mentioned above. If the user has taken elastography, the process proceeds to step S708. If not, the process returns to step S701 to continue acquiring the photoacoustic image.

  In step S708, by executing the ultrasonic image generation module 146, an elastography image is generated from the ultrasonic signal acquired in step S707. Then, when the output module 150 is executed, the generated elastography image is displayed on the display unit 104. Furthermore, a superimposition image may be generated by controlling the ultrasonic image generation module 146 to execute the superimposition image generation module 148, and the superimposition image may be displayed on the display unit 104 via the output module 150. .

  As described above, the control apparatus 101 can refer to the image of the region of interest specified based on the photoacoustic image and the ultrasonic image such as the B-mode image and the elastography image in association with each other. Images can be displayed on the display unit 104. Thereby, the workflow for a user such as a doctor to observe a medical image such as an ultrasonic image or a photoacoustic image of a subject and perform a diagnosis can be improved.

  The irradiation control module 144 may irradiate the subject with ultrasonic waves based on the user's operation input. When the ultrasonic wave is irradiated, an ultrasonic image may be generated by the ultrasonic image generation module 146 and the ultrasonic image may be displayed on the display unit 104 via the output module 150.

  Further, the elastography image generated by the ultrasonic image generation module 146 and the superimposed image generated by the superimposed image generation module 148 may be appropriately stored in the DISK 134 or the PACS 113.

  Although the case where elastography imaging is performed is described as an example in FIG. 7, steps S707 and S708 are not necessarily performed. For example, an ultrasonic image such as a B-mode image may be displayed based on the result of analyzing the photoacoustic image.

  FIG. 6 is a diagram illustrating an example of analysis performed in step S703 illustrated in FIG. 7 and an example of an image displayed in step S708. Here, a case where a region having a high blood vessel density is depicted in the photoacoustic image will be described as an example.

  FIG. 6A is an example of the photoacoustic image 602 displayed on the display unit 104. In the photoacoustic image 602, a blood vessel region 603 and a blood vessel region 604 are depicted.

  FIG. 6B is an example of a screen displayed on the display unit 104 when a region considered to have a high blood vessel density is detected as a result of the analysis in step S703. A frame 605 indicates the region of interest detected in step S703. Thereby, the user can visually recognize the region of interest, that is, the region detected when the blood vessel density is high. The notification screen 606 displays information based on the analysis result. For example, a message for informing the user that there is a region with a high blood vessel density and that the elastography image is useful for further observation is displayed on the notification screen 606. For example, a message “Please perform elastography of a blood vessel dense region” is displayed on the notification screen 606.

  FIG. 6C is an example of an elastography image 607 based on the ultrasonic signal obtained by the elastography imaging performed in step S707. In the elastography image 607, a tissue region 608 that may be a hard tissue is depicted so as to be distinguishable from surrounding tissues.

  FIG. 6D is an example of a superimposed image 609 obtained by superimposing an elastography image 607 on the photoacoustic image 602. A blood vessel region 610 corresponding to the blood vessel region 603 of the photoacoustic image 602 is depicted. Thereby, the user can visually recognize the region relating to the blood vessel density around the tissue region 608 of the elastography image 607 illustrated in FIG. 6C, and the diagnosis performed by the user can be assisted.

  According to the configuration of the third embodiment, the control device 101 analyzes the acquired photoacoustic image, detects a region that may be a lesion as a region of interest, and generates an ultrasound image corresponding to at least the region of interest. To do. For example, when diagnosing a region of interest where the blood vessel density is high and the presence of a tumor is suspected, the diagnosis can be assisted by controlling to perform elastography imaging for evaluating the hardness of the region of interest. By controlling in this way, the user can perform elastography imaging centering on the region of interest, and the user workflow in the examination can be improved.

  In the third embodiment, the case where an elastography image is acquired as an ultrasonic image has been described as an example. However, the present invention is not limited to this. For example, Doppler imaging for measuring blood flow velocity or B-mode imaging for grasping the structure in the subject may be performed.

[Fourth Embodiment]
In the first to third embodiments, the example in which the other imaging is controlled based on the result obtained by analyzing either the ultrasonic image or the photoacoustic image has been described. There may be a case where a time lag occurs between the time when the region of interest is drawn on one image and the time when the other image is controlled as a result of analysis. It is conceivable that the user moves the probe 102 to a position not suitable for capturing an image of the region of interest during the time lag. In the fourth embodiment, an example will be described in which the probe 102 is guided so that the region of interest can be appropriately imaged based on position information of the probe 102 in the examination.

  FIG. 8 is a diagram schematically illustrating an inspection state in the fourth embodiment. FIG. 8A shows an example of a state in which the user brings the probe 102 into contact with the subject 803 and acquires an ultrasonic image. An ultrasonic signal from the probe 102 is transmitted to the console 801. The console 801 is an apparatus in which the control device 101, the display unit 104, and the operation unit 105 shown in FIG. The console 801 corresponds to the control device 101 in each embodiment described above. The position information of the position 802 is acquired by the position acquisition module 149 of the console 801 and stored in the RAM 133 for a predetermined period. Further, when output to the PACS 113 by the output module 150 and when stored in the DISK 134, position information is associated with the ultrasonic image and the photoacoustic image, respectively. It is assumed that the region of interest is detected in the ultrasonic image captured at the position 802.

  FIG. 8B is an example of a state of inspection at the time when the console 801 detects an area of interest by analyzing the ultrasonic image in the series of processes of FIG. The user has moved the probe 102 from the position 805 corresponding to the position 802 in FIG. Although the region of interest from which the photoacoustic image is to be acquired is detected by the console 801, the detected region of interest may not be depicted even if the photoacoustic image is generated based on the photoacoustic signal acquired at the position 804. There is.

  FIG. 8C shows an example of a state in which the probe 102 is moved to a position 806 where the user can draw the region of interest by the guide of the console 801. By executing the position acquisition module 149 of the console 801, the current position of the probe 102 is compared with the target position which is the position of the probe 102 that acquired the ultrasonic signal of the ultrasonic image in which the region of interest is detected. The Thereby, guide information for guiding the probe 102 to the target position is generated and displayed. Thereby, the user can acquire the photoacoustic image of the detected region of interest.

  FIG. 9 is a flowchart illustrating an example of processing for the guide illustrated in FIG. In the following processes, the CPU 131 or the GPU 137 is a main body that realizes the processes by the modules unless otherwise specified.

  Since the process of step S901 and step S902 is the same as the process of step S201 illustrated in FIG. 2, description here is abbreviate | omitted by using the description mentioned above.

  In step S903, the position acquisition module 149 is executed, whereby the position information of the probe 102 is acquired. More specifically, the motion sensor, which is an example of the detection unit 103, tracks the position information of the probe 102 and transmits it to the control device 101. The motion sensor is provided or embedded in a portion different from the transmitting / receiving unit 106 of the probe 102 and the light source (not shown). The motion sensor is composed of, for example, a micro electro mechanical system, and provides 9-axis motion sensing including a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetic compass. Information regarding the movement of the probe 102 sensed by the motion sensor is acquired by the position acquisition module 149 and stored for a certain period of time.

  Since the processing of step S904 and step S905 is the same as the processing of step S203 and step S204 illustrated in FIG. 2, respectively, the description here is omitted by using the above description. If it is determined in step S905 that the photoacoustic signal is acquired, the process proceeds to step S906. If it is determined that the photoacoustic signal is not acquired, the process returns to step S901 to continue acquiring the photoacoustic image.

  In step S906, the position acquisition module 149 is executed, whereby guide information is displayed on the display unit 104. Specifically, in the generation of guide information, first, the position of the probe 102 at the time when the ultrasonic signal for generating the ultrasonic image in which the region of interest is detected in step S904 is acquired is set as the target position. Then, the difference between the current position of the probe 102 indicated by the position information sequentially transmitted from the detection unit 103 and the target position is acquired. And guide information is produced | generated based on the said difference. The guide information is presented to the user via the output module 150. The guide information is a guide image, for example, and is displayed on the display unit 104. The guide image is an objective index indicating guide information such as a movement direction, a movement amount, an inclination angle, a rotation direction, and a rotation amount for moving the probe 102 to the target position. The guide image may be any image as long as it is an objective index of the guide information. For example, the guide image is an image of an arrow having a size corresponding to the amount of movement or rotation and a direction corresponding to the direction of movement, rotation, or tilt. In another example, the guide image is a figure that has a size corresponding to the amount of movement or rotation, and whose shape is deformed according to the direction of movement, rotation, or inclination. The guide image is displayed on the display unit 104 in a manner that does not interfere with observation of the region of interest when the probe 102 is moved to the target position. For example, the guide image is displayed in an area where the ultrasonic image, the photoacoustic image, and the superimposed image are not displayed. In another example, while guiding the probe 102 to move to the target position, it is displayed at a position that overlaps the area near the target area, and when the target area is rendered, it is deformed into a shape that cannot be seen. May be displayed.

  In step S907, the irradiation control module 144 is executed to determine whether to irradiate the subject with light. Specifically, in the determination by the irradiation control module 144, it is determined that light is irradiated when the probe 102 has reached the target position and the probe 102 is in contact with the subject. For example, when the current position of the probe 102 indicated by the position information transmitted from the detection unit 103 matches the target position, it is determined that the probe 102 has reached the target position. In another example, when the probe 102 reaches a predetermined range including the target position, it may be determined that the probe 102 has reached the target position. For example, the position of the probe 102 capable of acquiring the region of interest detected in step S904 and the target position are set as a predetermined range. Since the process for determining that the probe 102 is in contact with the subject is the same as the process of step S205 illustrated in FIG. 2, the description here is omitted by using the above description. If it is determined that the subject is to be irradiated with light, the process proceeds to step S908. If it is determined that the object is not irradiated with light, the process returns to step S901 to continue acquiring the ultrasound image.

  Since the processing of step S908 and step S909 is the same as the processing of step S206 and step S207 illustrated in FIG. 2, the description here will be omitted by using the above description.

  In the fourth embodiment, the case where the target position is set based on the region of interest detected by analyzing the ultrasound image has been described as an example. However, the present invention is not limited to this. For example, as shown in the third embodiment, the target position may be set based on the region of interest detected by analyzing the photoacoustic image. Moreover, the method of presenting guide information to the user is not limited to the above-described guide image. For example, the guide information may be presented to the user by generating a sound such that the sound generation interval decreases as the probe 102 approaches the target position.

  According to the configuration of the fourth embodiment, the control apparatus 101 can efficiently acquire an image depicting a region of interest, thereby improving the user workflow.

[Modification 1]
In the first to fourth embodiments, the analysis for detecting the region of interest may not be performed for all the frames of the acquired image. For example, it is possible to reduce the processing load on the control apparatus 101 by performing the analysis at a predetermined frame interval.

  In the first to fourth embodiments, when one of the ultrasonic image and the photoacoustic image is analyzed and a region of interest is detected, an example of controlling the probe 102 to capture the other image will be described. did. The present invention is not limited to this. For example, when the region of interest is included in a predetermined range of the range that can be drawn by the probe 102, the probe 102 may be controlled to capture the other image. For example, when the region of interest is included in the range near the center of one image, the probe 102 may be controlled to capture the other image. This makes it possible to obtain a medical image that makes it easier to observe the region of interest when it is later interpreted or attached to an image diagnosis report. In addition, this can reduce the possibility that the other image is inadvertently captured when the region of interest is included in a range in which one image can be drawn unintentionally.

  In the first to fourth embodiments, the example in which the superimposed image is displayed has been described. However, the superimposing method is not limited to the above-described example. In an example in which the ultrasonic image is a base image and the photoacoustic image is a layer image, the photoacoustic image may be superimposed only in the vicinity of the region of interest of the ultrasonic image, or a photoacoustic image in a desired range is superimposed. May be. The transparency of the layer image is appropriately changed according to the purpose. The layer image may be opaque, or the transparency may be increased only in the vicinity of the region of interest. A slider capable of changing the transparency of the layer image may be displayed on the display unit 104 and changed during observation by the user. The display of the superimposed image, the display in which the ultrasonic image and the photoacoustic image are arranged in parallel, and the display of any one of the ultrasonic image and the photoacoustic image may be switched by a user operation input. Good.

  In the first to fourth embodiments, the example in which the region of interest is detected by analyzing the image acquired in real time has been described. However, the present invention is not limited to this. For example, a region drawn in a past three-dimensional image may be set as a region of interest. For example, a specific region on a CT image taken in the past is set as a region of interest. And the coordinate system of the said CT image and the coordinate system of the real space which operates the probe 102 are matched. The control apparatus 101 can acquire an ultrasonic image and a photoacoustic image, and when the probe 102 is positioned at a position where a region of interest can be acquired while acquiring one image, the other image is also acquired. You may control to.

  Furthermore, in the first to fourth embodiments, the user may be notified that light irradiation is performed by the probe 102. For example, a notification image for notifying that light irradiation is performed by the probe 102 is displayed on the display unit 104. When displaying the notification image on the display unit 104, it is preferable to display the notification image in the vicinity of the image of the subject being observed by the user. In another example, the probe 102 may be provided with an LED light that is turned on during light irradiation. In yet another example, the control device 101 may generate a notification sound during light irradiation. Thereby, the user can know that light is being emitted from the probe 102, and the safety of the user and the subject can be improved.

[Modification 2]
In the above embodiment, the necessity of obtaining the photoacoustic image is determined based on the analysis result of the ultrasonic image. Therefore, when a region of interest such as a calculus is included in the ultrasonic image, a photoacoustic image is automatically captured. Therefore, a photoacoustic image is captured once for a certain region of interest. Nevertheless, there is a possibility that a photoacoustic image will be taken again if an attempt is made to confirm the region of interest again with an ultrasonic image. That is, a photoacoustic image is taken a plurality of times for the same region of interest, and an unnecessary photoacoustic image is acquired. Therefore, an object of the present modification is to prevent an unnecessary photoacoustic image from being acquired.

  In this modification, the irradiation control module 144 limits the irradiation of light for acquiring a photoacoustic image based on the position information of the probe 102 acquired by the position acquisition module 149, for example. For example, the irradiation control module 144 stores the position information of the probe 102 when it is determined that light irradiation has been performed in the past, and the position information of the current probe 102 matches or is stored with the stored position information. It is determined whether the deviation is within a predetermined threshold. And the irradiation control module 144 is the part which has already acquired the photoacoustic image when the position information of the current probe 102 matches the stored position information or the deviation from the stored position information is within a predetermined threshold. Judging that there is, it restricts light irradiation. By restricting the light irradiation, it is possible to prevent the subject from performing unnecessary light irradiation and to prevent acquisition of unnecessary photoacoustic images.

  Further, the irradiation control module 144 may limit light irradiation for acquiring a photoacoustic image based on an ultrasonic image generated by the ultrasonic image generation module 146, for example. For example, the irradiation control module 144 stores the ultrasonic image generated when it is determined that the light irradiation has been performed in the past, and compares it with the currently generated ultrasonic image. If the degree of similarity between the images as a result of the comparison is equal to or greater than a predetermined threshold value, it may be determined that the photoacoustic image has already been acquired and light irradiation may be limited.

  The irradiation control module 144 performs light irradiation for acquiring a photoacoustic image based on the ultrasonic image generated by the ultrasonic image generation module 146 and the position information of the probe 102 acquired by the position acquisition module 149. It may be limited. For example, the irradiation control module 144 stores an ultrasonic image in association with position information of the probe 102 when it is determined that light irradiation has been performed in the past. Then, the irradiation control module 144 reads an ultrasonic image associated with the current position information of the probe 102 and compares it with the currently generated ultrasonic image. If the degree of similarity between the images as a result of the comparison is equal to or greater than a predetermined threshold value, it may be determined that the photoacoustic image has already been acquired and light irradiation may be limited.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The control device in each of the above-described embodiments may be realized as a single device, or may be configured to execute the above-described processing by combining a plurality of devices so that they can communicate with each other. included. The above-described processing may be executed by a common server device or server group. The plurality of devices constituting the control device and the control system need only be able to communicate at a predetermined communication rate, and do not need to exist in the same facility or in the same country.

  In the embodiment of the present invention, a software program that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and the computer of the system or apparatus reads and executes the code of the supplied program. Includes form.

  Therefore, in order to realize the processing according to the embodiment by a computer, the program code itself installed in the computer is also one embodiment of the present invention. Further, based on instructions included in a program read by the computer, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing. .

  Embodiments appropriately combining the above-described embodiments are also included in the embodiments of the present invention.

Claims (16)

The ultrasonic signal and the photoacoustic signal are output from a probe that outputs an ultrasonic signal by transmitting and receiving ultrasonic waves to the subject and outputs a photoacoustic signal by receiving a photoacoustic wave generated by light irradiation on the subject. And signal acquisition means for acquiring at least one of
Generating means for generating an ultrasonic image based on the ultrasonic signal;
Irradiation control means for controlling the light irradiation of the probe based on the generated ultrasonic image;
A control device comprising: The generating means generates a photoacoustic image based on a photoacoustic signal acquired based on the controlled light irradiation,
The control apparatus according to claim 1, further comprising display control means for causing the generated ultrasonic image and the photoacoustic image to be displayed on a display unit in a comparable manner. By further detecting a region of interest for acquiring the photoacoustic signal by analyzing the generated ultrasonic image,
The control apparatus according to claim 2, wherein the irradiation control unit controls the light irradiation in order to acquire the photoacoustic signal from the detected region of interest. Position acquisition means for acquiring position information indicating a position of the probe with respect to the subject;
The control device according to any one of claims 1 to 3, further comprising guide means for guiding the probe to a position where the detected region of interest can be acquired.   The said irradiation control means controls to perform the said light irradiation, when the said probe exists in the position which can acquire the photoacoustic image containing the said region of interest. A control device according to claim 1.   6. The apparatus according to claim 3, further comprising an output unit that outputs the photoacoustic image acquired in response to detection of the region of interest in association with the analyzed ultrasonic image. The control device according to any one of the above.   The control device according to any one of claims 3 to 6, wherein the detection unit detects an area where a calculus is likely by analyzing the ultrasonic image.   The ultrasonic image is an elastography image reflecting the strain distribution of the tissue of the subject before and after being pressed by the probe, and the detection means detects a region where the strain is a predetermined value or less. The control device according to any one of claims 3 to 7.   The control according to any one of claims 1 to 8, wherein the irradiation control unit controls the light irradiation when the probe is in contact with the subject. apparatus. The ultrasonic signal and the photoacoustic signal are output from a probe that outputs an ultrasonic signal by transmitting and receiving ultrasonic waves to the subject and outputs a photoacoustic signal by receiving a photoacoustic wave generated by light irradiation on the subject. Acquisition means for acquiring at least one of
Generating means for generating a photoacoustic image based on the photoacoustic signal;
Irradiation control means for controlling transmission of the ultrasonic wave of the probe based on the generated photoacoustic image;
A control device comprising: By further detecting a region of interest for acquiring the ultrasonic signal by analyzing the generated photoacoustic image,
The irradiation control means controls transmission of the ultrasonic wave to obtain the ultrasonic signal from the detected region of interest;
The generating means generates an ultrasound image based on an ultrasound signal acquired based on the controlled transmission of the ultrasound,
The control apparatus according to claim 10, further comprising display control means for causing the generated ultrasonic image and the photoacoustic image to be displayed on a display unit in a comparable manner.   The control apparatus according to claim 11, wherein the irradiation control unit controls a transmission range of the ultrasonic wave based on the detected region of interest.   The control device according to any one of claims 2 to 12, wherein the display control unit displays a superimposed image obtained by superimposing the photoacoustic image on the ultrasonic image on a display unit. . A light source for irradiating the subject with light;
A transducer for transmitting and receiving ultrasound,
A signal acquisition means for acquiring a reflected wave of an ultrasonic wave transmitted by the transducer as an ultrasonic signal, and acquiring a photoacoustic wave generated by light irradiated on a subject from the light source as a photoacoustic signal;
Generating means for generating an ultrasonic image based on the ultrasonic signal;
Irradiation control means for controlling irradiation of the light from the light source based on the generated ultrasonic image;
An imaging system comprising: A light source for irradiating the subject with light;
A transducer for transmitting and receiving ultrasound,
A signal acquisition means for acquiring a reflected wave of the ultrasonic wave transmitted by the transducer as an ultrasonic signal, and acquiring a photoacoustic wave generated by light irradiated on the subject from the light source as a photoacoustic signal;
Generating means for generating a photoacoustic image based on the photoacoustic signal;
Irradiation control means for controlling transmission of the ultrasonic waves of the transducer based on the generated photoacoustic image;
An imaging system comprising: Acquiring a first image based on a first signal acquired by the first technique;
Analyzing the first image to detect a region of interest; and obtaining a second image including the region of interest based on a second signal obtained by a second method different from the first method. Controlling the acquisition of the second signal,
Displaying the first image and the second image on a display unit so that they can be compared;
A control method characterized by comprising:

Images