Classifications

• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
  • A61B3/0016—Operational features thereof
  • A61B3/0025—Operational features thereof characterised by electronic signal processing, e.g. eye models
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
  • A61B3/0016—Operational features thereof
  • A61B3/0041—Operational features thereof characterised by display arrangements
  • A61B3/0058—Operational features thereof characterised by display arrangements for multiple images
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
  • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
  • A61B3/1005—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for measuring distances inside the eye, e.g. thickness of the cornea
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
  • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
  • A61B3/1025—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for confocal scanning
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
  • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
  • A61B3/12—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes
  • A61B3/1225—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes using coherent radiation
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
  • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
  • A61B3/14—Arrangements specially adapted for eye photography
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/12—Edge-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30041—Eye; Retina; Ophthalmic
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30101—Blood vessel; Artery; Vein; Vascular

Definitions

• the present invention relates to an image processing apparatus and an image processing method, which are to be used for ophthalmic diagnosis and treatment.
• the inspection of an eye has been widely conducted for. the purpose of diagnosing and treating lifestyle-related diseases and diseases that are leading causes of blindness in early stages.
• a scanning laser ophthalmoscope SLO
• the scanning laser ophthalmoscope is an apparatus configured to perform raster scanning on a fundus of an eye with laser light that is measuring light to obtain a planar image of the fundus based on the intensity of return light of the measuring light, and the image is obtained with high resolution at high speed.
• the planar image is generated by detecting only light having passed through an aperture portion (pinhole) out of the return light. This allows only return light at a particular depth position to be imaged, and an image having a contrast higher than that of a fundus camera or the like to be acquired.
• SLO apparatus Such an apparatus configured to photograph a planar image is hereinafter referred to as "SLO apparatus", and the planar image is hereinafter
• SLO image referred to as "SLO image”.
• An adaptive optics SLO apparatus including an adaptive optics system such as the wavefront correction device has been developed to enable the acquisition of an SLO image having a high lateral resolution.
• the SLO image obtained by the adaptive optics SLO apparatus can be acquired as a moving image. Therefore, for example, in order to observe hemodynamics non-invasively, the SLO image is used for measurement of the moving speed of blood corpuscles in a capillary vessel and the like through extraction of a retinal vessel from each frame. Further, in order to evaluate a relation with a visual function through use of the SLO image, a density distribution and arrangement of photoreceptor cells P are also measured through detection of the photoreceptor cells P.
• FIG. 6B is an illustration of an example of the SLO image with a high lateral resolution obtained by the adaptive optics SLO apparatus. In the image, the photoreceptor cells P, a low brightness region Q corresponding to the position of the capillary vessel, and a high brightness region W corresponding to the position of a leukocyte can be observed.
• a focus position is set to the vicinity of an outer layer of the retina (for example, layer boundary B5 in FIG. 6A) , to thereby acquire such an SLO image as illustrated in FIG. 6B.
• retinal vessels and branching capillary vessels travel in an inner layer of the retina (from layer boundary B2 to layer boundary B4 in FIG. 6A) .
• an adaptive optics SLO image is acquired with the focus position set in the inner layer of the retina, for example, a retinal vessel wall can be observed directly.
• Non Patent Literature 1 a method involving obtaining scattering light by changing the diameter, shape, and position of a pinhole arranged in front of a photo-receiving unit and observing a nonconfocal image thus obtained has come to be used.
• Non Patent Literature 1 a method involving obtaining scattering light by changing the diameter, shape, and position of a pinhole arranged in front of a photo-receiving unit and observing a nonconfocal image thus obtained has come to be used.
• Non Patent Literature 1 a method involving obtaining scattering light by changing the diameter, shape, and position of a pinhole arranged in front of a photo-receiving unit and observing a nonconfocal image thus obtained has come to be used (Non Patent Literature 1) .
• a focus depth is large, and hence an object having irregularities in a depth direction, such as a blood vessel, can be observed easily. Further, light reflected from the nerve fiber layer is not easily received directly, and hence noise can be reduced.
• a retinal artery is an arteriole having a blood vessel diameter of from about 10 ⁇ to about 100 ⁇ , and a wall of the retinal artery is formed of an intima, a media, and an adventitia. Further, the media is formed of smooth muscle cells, and travels along a circumferential direction of the blood vessel in a coil shape.
• a smooth muscle contracts to increase a wall thickness.
• blood pressure is lowered through administration of an antihypertensive agent, the shape of the wall of the retinal artery returns to an original shape.
• Non Patent Literature 1 a technology for acquiring the nonconfocal image of the retinal vessel through use of the adaptive optics SLO apparatus and visualizing the retinal vessel wall cells is disclosed in Non Patent Literature 1.
• a technology for semiautomatically extracting a retinal vessel wall boundary from an image of an adaptive optics fundus camera through use of a variable shape model is disclosed in Non Patent Literature 2.
• a high-resolution image relating to the wall of the retinal artery is acquired through use of an SLO apparatus to which an adaptive optics technology is applied, to thereby allow the observation of the wall of the retinal artery. Further, in a position (Prl in FIG. 6H) that passes through the center of a cell that forms the blood vessel wall, a peak corresponding to each membrane that forms the blood vessel wall occurs in a brightness profile shown in FIG. 61, and hence the wall thickness and a membrane thickness can be manually measured.
• Non Patent Literature 1 the retinal vessel wall, the membrane boundary, and the wall cells are visualized from an AO- SLO image having a nonconfocal image acquisition function based on pinhole control, and the membrane thickness and a cell density are manually measured.
• a technology for automatically measuring the wall thickness and membrane thickness of the retinal vessel and the density of cells that form the wall is not disclosed.
• the technology disclosed in Non Patent Literature 1 does not solve the above-mentioned problem.
• the retinal vessel wall boundary is detected from the image of the adaptive optics fundus camera through the use of the variable shape model, and the wall thickness of the retinal artery is semiautomatically measured.
• a venous wall, or membranes or cells that form an arterial wall and a venous wall cannot be visualized from the image of the adaptive optics fundus camera. That is, a technology for measuring the wall thickness of a vein, the membrane thickness of the artery or the vein, or the distribution of cells that form the blood vessel wall is not disclosed even in Non Patent Literature 2.
• NPL 1 Chui et al.; "Imaging of Vascular
• NPL 2 Koch et al.; "Morphometric analysis of small arteries in the human retina using adaptive optics imaging: relationship with blood pressure and focal vascular changes", Journal of Hypertension, Vol. 32, No. 4, pp. 890-898, 2014.
• the present invention has been made in view of the above-mentioned problems, and has an object to accurately measure thicknesses of membranes that form a blood vessel wall of an eye.
• an image processing apparatus including:
• an image acquiring unit configured to acquire an image of an eye
• a vessel feature acquiring unit configured to acquire membrane candidate points that form a wall of a blood vessel based on the acquired image
• a cell identifying unit configured to identify a cell that forms the wall of the blood vessel based on the membrane candidate points
• a measuring position acquiring unit configured to identify a measuring position regarding the wall of the blood vessel based on a position of the identified cell
• FIG. 1 is a block diagram for illustrating a configuration example of functions of an image processing apparatus according to a first embodiment of the present invention.
• FIG. 2 is a block diagram for illustrating a configuration example of a system including the image processing apparatus according to the embodiment of the present invention.
• FIG. 3A is a diagram for illustrating an overall configuration of an SLO image acquiring apparatus according to the embodiment of the present invention.
• FIG. 3B is a diagram for illustrating an example of configurations of an aperture portion and a photosensor within the SLO image acquiring apparatus illustrated in FIG. 3A.
• FIG. 3C is a diagram for illustrating an example of the aperture portion illustrated in FIG. 3B.
• FIG. 3D is a diagram for illustrating an example of the aperture portion illustrated in FIG. 3B.
• FIG. 3E is a diagram for illustrating an example of a light shielding portion illustrated in FIG. 3B.
• FIG. 3F is a diagram for illustrating an example of the light shielding portion illustrated in FIG. 3B.
• FIG. 3G is a diagram for illustrating an example of the light shielding portion illustrated in FIG. 3B.
• FIG. 3H is a diagram for illustrating an example of the light shielding portion illustrated in FIG. 3B.
• FIG. 4 is a block diagram for illustrating a hardware configuration example of a computer including hardware corresponding to a memory portion and an image processing portion and being configured to hold and execute other respective portions as software.
• FIG. 5 is a flowchart of processing executed by the image processing apparatus according to the embodiment of the present invention.
• FIG. 6A is a diagram for illustrating details of image processing according to the embodiment of the present invention, and illustrating an imaged layer structure of a retina.
• FIG. 6B is a diagram for illustrating an example of an SLO image obtained by an adaptive optics SLO apparatus .
• FIG. 6C is a diagram for illustrating an example of an obtained confocal image.
• FIG. 6D is a diagram for illustrating an example of a nonconfocal image obtained regarding the same body part as that of the confocal image of FIG. 6C.
• FIG. 6E is a diagram for illustrating an example of the nonconfocal image obtained regarding the same body part as that of the confocal image of FIG. 6C.
• FIG. 6F is a diagram for illustrating an example of an image obtained based on FIG. 6D and FIG. 6E.
• FIG. 6G is a diagram for illustrating a relationship between a low magnification image and a high magnification image.
• FIG. 6H is a diagram for illustrating another example of the image obtained based on FIG. 6D and FIG. 6E.
• FIG. 61 is a graph for showing an example of a brightness profile along a line segment orthogonal to a blood vessel center line exhibited in respective positions on the blood vessel center line.
• FIG. 6J is a graph for showing another example of a brightness profile along a line segment orthogonal to the blood vessel center line exhibited in the respective positions on the blood vessel center line.
• FIG. 6K is a graph for showing processing for searching a corrected brightness profile for a local maximum value of a brightness value.
• FIG. 6L is a first diagram for illustrating processing for identifying a measuring position of a membrane thickness.
• FIG. 6M is a second diagram for illustrating the processing for identifying the measuring position of the membrane thickness.
• FIG. 6N is a third diagram for illustrating the processing for identifying the measuring position of the membrane thickness.
• FIG. 7A is a flowchart for illustrating details of a cell identification process illustrated in FIG. 5.
• FIG. 7B is a flowchart for illustrating details of a measuring process illustrated in FIG. 5.
• FIG. 8A is a diagram for illustrating content such as a measurement result displayed on a monitor in the processing illustrated in FIG. 5.
• FIG. 8B is a diagram for illustrating a map displayed on the monitor in the processing illustrated in FIG. 5.
• An image processing apparatus uses an image obtained by imaging a retinal vessel wall through use of an SLO apparatus configured to simultaneously acquire a confocal image and a nonconfocal image. An extreme value of a brightness profile is detected from the image along travel of the wall. Then, cells that form the blood vessel wall are detected based on the obtained extreme value, and a distribution thereof is automatically measured.
• the retinal vessel wall is imaged through use of the SLO apparatus configured to simultaneously acquire a confocal image and a nonconfocal image.
• a center line of a retinal vessel is imaged through use of the SLO apparatus configured to simultaneously acquire a confocal image and a nonconfocal image.
• blood vessel center line (hereinafter referred to also as "blood vessel center line”) is acquired from the obtained nonconfocal image by morphology filter processing.
• a membrane candidate region that forms the retinal vessel wall is further acquired based on the blood vessel center line.
• a brightness profile along the travel of a blood vessel wall is generated based on the membrane candidate region.
• a brightness value within the brightness profile is subjected to a- Fourier transform. After a high frequency component is removed from the image that has been subjected to the Fourier transform, a peak position within the brightness profile is detected as the position of the cells.
• a. membrane thickness is measured by automatically identifying a measuring position of the membrane thickness based on a relative distance between the cells calculated in respective positions along the travel direction or travel line of the blood vessel wall is described.
• FIG. 2 is a diagram of an overall configuration of a system including an image processing apparatus 10 according to this embodiment.
• the image processing apparatus 10 is connected to an SLO image acquiring apparatus 20, a data server 40, and a pulse data acquiring apparatus 50 through a local area network (LAN) 30.
• the LAN 30 is formed of an optical fiber, USB, IEEE 1394, or the like. Note that, the connection to those apparatus may be configured as the connection through an external network such as the Internet. Alternatively, the direct connection to the image processing apparatus 10 may be employed.
• the SLO image acquiring apparatus 20 is an apparatus configured to acquire a wide field angle image Dl of an eye and a confocal image Dc and a nonconfocal image Dn that are high magnification images.
• the SLO image acquiring apparatus 20 transmits the wide field angle image Dl, the confocal image Dc, the nonconfocal image Dn, and information on fixation target positions Fl and Fen used at a time of image acquisition thereof to the image processing apparatus 10 and the data server 40.
• the SLO image acquiring apparatus 20 functions as an image acquiring unit configured to acquire the image of the eye in this embodiment .
• the pulse data acquiring apparatus 50 is an apparatus configured to acquire biosignal data (pulse data) that changes autonomously, and is formed of, for example, a sphygmograph or an electrocardiograph.
• the pulse data acquiring apparatus 50 acquires pulse data. Pi simultaneously with the acquisition of the wide field angle image Dl, the confocal image Dc, and the nonconfocal image Dn in response to an operation performed by an operator (not shown) .
• the obtained pulse data Pi is transmitted to the image processing apparatus 10 and the data server 40.
• the pulse, data acquiring apparatus 50 may be directly connected to the SLO image acquiring apparatus 20.
• Dclm (Dnlm) is represented by a high magnification confocal (nonconfocal) image
• Dc2o, ... (Dn2o, ... ) is represented by a medium magnification confocal (nonconfocal) image.
• the SLO image acquiring apparatus 20 transmits the wide field angle image Dl, the confocal image Dc, the nonconfocal image Dn, the fixation target positions Fl and Fen used at the time of the image acquisition, the pulse data Pi, and the like to the data server 40.
• the data server 40 stores those pieces of information along with image features of the eye output by the image processing apparatus 10.
• the fixation target positions Fl and Fen are fixation target positions used at the time of the image acquisition, and it is preferred that other image- acquiring conditions be also stored along with those fixation target positions. Examples of the image features include features regarding the retinal vessel, the retinal vessel wall, and the cells that form the blood vessel wall.
• the wide field angle image Dl, the confocal image Dc, the nonconfocal image Dn, the pulse data Pi, and the image features of the eye are transmitted to the image processing apparatus 10.
• FIG. 1 is a block diagram for illustrating the functional configuration of the image processing apparatus 10, and the image processing apparatus 10 includes an image acquiring portion 110, a memory portion 120, an image processing portion 130, and an instruction acquiring portion 140. Further, the image acquiring portion 110 includes a confocal data acquiring portion 111, a nonconfocal data acquiring portion 112, and a pulse data acquiring portion 113.
• the image processing portion 130 includes a position alignment portion 131, a vessel feature acquiring portion 132, a cell identifying portion 133, a measuring position identifying portion 134, a measuring portion 135, and a display control portion 136.
• the SLO image acquiring apparatus 20 includes a super luminescent diode (SLD) 201, a Shack-Hartmann wavefront sensor 206, an adaptive optics system 204, a first beam splitter 202, a second beam splitter 203, an X-Y scanning mirror 205, a focus lens 209, an aperture portion 210, a photosensor 211, an image forming portion 212, and an output portion 213.
• SLD super luminescent diode
• the first beam splitter 202, the second beam splitter 203, the adaptive optics system 204, and the X-Y scanning mirror 205 are arranged in the stated order from the SLD 201 to an eye to be inspected.
• the focus lens 209, the aperture portion 210, and the photosensor 211 are arranged in the stated order in a branching direction of the first beam splitter 202.
• the image forming portion 212 is connected to the photosensor 211, and the output portion 213 is connected to the image forming portion 212.
• the Shack-Hartmann wavefront sensor 206 is arranged in a branching direction of the second beam splitter 203.
• Measuring light emitted from the SLD 201 serving as a light source passes through an optical path in which the respective optical members are arranged and a crystalline lens OL of an eye E to be inspected to reach a fundus Er of the eye E to be inspected.
• the measuring light reflected by the fundus Er of the eye follows the optical path backward as return light.
• a part of return light is split toward the Shack-Hartmann wavefront sensor 206 by the second beam splitter 203.
• the other part of the return light is further split by the first beam splitter 202 to be guided to the photosensor 211.
• the Shack-Hartmann wavefront sensor 206 is a device for measuring an aberration of the eye, and has a CCD 208 connected to a lens array 207.
• the split part of the return light is transmitted through the lens array 207 as incident light.
• the incident light transmitted through the lens array 207 appears as a group of bright spots on the CCD 208, and a wavefront aberration of the return light is measured based on a positional deviation of the projected bright spots.
• the adaptive optics system 204 drives an aberration correction device to correct the aberration based on the wavefront aberration measured by the Shack-Hartmann wavefront sensor 206.
• the aberration correction device is formed of a shape variable mirror or a spatial light phase modulator.
• the return light subjected to aberration correction and split by the first beam splitter 202 passes through the focus lens 209 and the aperture portion 210 to be received by the photosensor 211.
• the scan position of the measuring light on the fundus Er of the eye can be controlled by moving the X-Y scanning mirror 205.
• the operator acquires data on an image acquisition target region specified in advance at a specified frame rate by a specified number of frames.
• the data is transmitted to the image forming portion 212, and subjected to the correction of an image distortion ascribable to variations in scanning speed and the correction of the brightness value, and image data (moving image or still, image) is thus formed.
• the output portion 213 outputs the image data formed by the image forming portion 212 to the image processing apparatus 10 or the like.
• the part of the aperture portion 210 and the photosensor 211 illustrated in FIG. 3A may have any configuration that can acquire the confocal image Dc and the nonconfocal image Dn.
• the part of the aperture portion 210 and the photosensor 211 is formed of a light shielding portion 210-1 illustrated in FIG. 3B and FIG. 3E and photosensors 211-1, 211-2, and 211-3 illustrated in FIG. 3B.
• the return light enters the light shielding portion 210-1 arranged on an imaging surface, and partial light thereof is reflected by the light shielding portion 210-1 to enter the photosensor 211-1.
• the light shielding portion 210-1 is described with reference to the FIG. 3E.
• the light shielding portion 210-1 is formed of transmission regions 210-1-2 and 210-1-3, a light shielding region
• the center of the light shielding portion 210-1 where the reflection region 210-1-1 is formed is arranged so as to be positioned at the center of an optical axis of the return light. Further, the light shielding portion 210-1 has an elliptical pattern that is formed into a circle when viewed from an optical axis direction when the light shielding portion 210-1 is arranged diagonally with respect to the optical axis of the return light.
• the light split by being reflected by the reflection region 210-1-1 of the light shielding portion 210-1 enters the photosensor 211-1.
• the light that has passed through the transmission regions 210-1- 2 and 210-1-3 of the light shielding portion 210-1 is further split by a two-split prism 210-2 arranged on the imaging surface. Light beams obtained after the splitting enter the photosensors 211-2 and 211-3, respectively, as illustrated in FIG. 3B.
• a voltage signal obtained by each of the photosensors is converted into a digital value by an AD board included in the image forming portion 212, and then converted into a two-dimensional image.
• An image generated based on the light having entered the photosensor 211-1 becomes a confocal image focused within a particular narrow range. Further, an image generated based on the light input- to the photosensors 211-2 and 211-3 becomes a nonconfocal image focused within a wide range.
• a method of splitting the return light for extracting a nonconfocal signal is not limited thereto.
• the transmission region may be divided into four (210-1-4, 210-1-5, 210-1-6, and 210-1-7) to obtain four nonconfocal signals.
• a method of receiving a confocal signal and the nonconfocal signal is not limited thereto.
• the diameter and position of the aperture portion 210 may be made variable and adjusted so as to receive the confocal signal under the state of an opening diameter of FIG. 3C and receive the nonconfocal signal under the state of an opening diameter of FIG. 3D.
• the diameter and moving amount of the aperture portion may be set arbitrarily. For example, in FIG.
• the diameter of the aperture portion can be set to 1 airy disc diameter (ADD), while in FIG. 3D, the diameter of the aperture portion can be set to about 10 ADD, and the moving amount can be set to about 6 ADD.
• the light shielding portion 210-1 may be configured so that only a plurality of nonconfocal signals are received substantially simultaneously by arranging, for example, two aperture portions 210-1-8 as illustrated in FIG. 3G or four aperture portions 210-1-9 as illustrated in FIG. 3H. Note that, when the aperture portion is divided into four, a four-split prism is arranged on the imaging surface in place of the two-split prism, and four photosensors are arranged as well.
• nonconfocal image Dn represents both the R-channel image Dnr and the L-channel image Dnl.
• the SLO image acquiring apparatus 20 may also be instructed to increase a swing angle of the X-Y scanning mirror 205 serving as a scanning optical system in the configuration of FIG. 3A to inhibit the adaptive optics system 204 from correcting the aberration.
• Such an instruction allows the SLO image acquiring apparatus 20 to operate also as a normal SLO apparatus to acquire a wide field angle image.
• the wide field angle image Dl (Die and Din)
• the wide field angle image Dl may be an SLO image to which the adaptive optics is applied, or may be a mere SLO image.
• a confocal wide field angle image and a nonconfocal wide field angle image are represented by Die and Din, respectively, when distinguished from each other.
• the image processing apparatus 10 includes a central processing unit (CPU) 301, a memory (RAM) 302, a control memory (ROM) 303, an external memory 304, a monitor 305, a keyboard 306, a mouse 307, and an interface 308.
• Control programs for implementing image processing functions according to this embodiment and data to be used when the control programs are executed are stored in the external memory 304.
• Those control programs and the data are appropriately loaded into the RAM 302 through a bus 309 under the control of the CPU 301, and are executed by the CPU 301 to function as the respective portions described below.
• FIG. 5 is a flowchart relating to an operation performed when the image of a fundus of the eye to be inspected is processed by the image processing apparatus 10.
• the image acquiring portion 110 requests the SLO image acquiring apparatus 20 to acquire a low magnification image and a high magnification image.
• the low magnification image corresponds to the wide field angle image Dl as illustrated in FIG. 6G
• the high magnification image corresponds to the confocal image Dcj within an annular region of an optic papilla portion as indicated by a region Ptl of FIG. 6G, and two nonconfocal images Dnrk and Dnlk.
• the image acquiring portion 110 requests the SLO image acquiring apparatus 20 to acquire the fixation target positions Fl and Fen corresponding to those images as well .
• SLO image acquiring apparatus 20 acquires the wide field angle image Dl, the confocal image Dcj, the nonconfocal images Dnrk and Dnlk, corresponding attribute data, and the fixation target positions Fl and Fen. After the acquisition, those pieces of data are transmitted to the image acquiring portion 110.
• the image acquiring portion 110 receives the data such as the wide field angle image Dl, the confocal image Dcj , the nonconfocal images Dnrk and Dnlk, the fixation target positions Fl and Fen from the SLO image acquiring apparatus 20 through the LAN 30, and stores those pieces of data into the memory portion 120.
• the pulse data acquiring apparatus 50 requests the pulse data acquiring apparatus 50 to acquire the pulse data Pi relating to a biosignal.
• a sphygmograph is used as the pulse data acquiring apparatus, and the pulse wave data Pi is acquired as the pulse data from a lobulus auriculae
• the pulse wave data Pi is expressed by a point sequence having one axis indicating an acquisition time and the other axis indicating a pulse wave signal value measured by the sphygmograph.
• the pulse data acquiring apparatus 50 acquires and transmits the corresponding pulse data Pi in response to the acquisition request.
• the pulse data acquiring portion 113 receives the pulse data Pi from the pulse data acquiring apparatus 50 through the LAN 30.
• the pulse data acquiring portion 113 stores the received pulse data Pi into the memory portion 120.
• the confocal data acquiring portion 111 or the nonconfocal data acquiring portion 112 starts acquiring an image.
• Cases conceivable as modes of the image acquisition include a case where the image acquisition is started in synchronization with a given phase of the pulse data Pi and a case where the acquisition of the pulse data Pi and the image acquisition are simultaneously started immediately after the image acquisition request. In this embodiment, the acquisition of the pulse data Pi and the image acquisition are started immediately after the image acquisition request.
• Pieces of pulse data Pi on the respective images are acquired from the pulse data acquiring portion 113, and extreme values of the respective pieces of the pulse data Pi are detected to calculate a heart beat cycle and a relative cardiac cycle.
• the relative cardiac cycle is a relative value expressed by a floating-point number ranging from 0 to 1 when the heart beat cycle is set to 1.
• FIG. 6C examples of the confocal image Dc and the nonconfocal image Dnr obtained when the retinal vessel is imaged are illustrated in FIG. 6C and FIG. 6D.
• the confocal image Dc the reflection of a nerve fiber layer in a background thereof is strong, and. position alignment easily becomes difficult due to noise in the background part.
• the contrast of a blood vessel wall on the right is high.
• the nonconfocal image Dnl of the L-channel as illustrated in, for example, FIG. 6E, the contrast of a blood vessel wall on the left is high.
• any one of an addition-average image Dnr+l (FIG. 6H) and a split detector image Dns (FIG. 6F) can also be used as an image obtained by subjecting the R-channel image and the L-channel image to arithmetic operation processing. Through use of those images, the blood vessel wall may be observed, and measuring processing relating to the blood vessel wall may be performed.
• the addition- average image Dnr+l is an image obtained by subjecting the R-channel image and the L-channel image to addition averaging.
• the split detector image Dns is an image obtained by performing difference emphasis processing ( (L-R) / (R+L) ) regarding the nonconfocal image .
• the acquisition position of the high magnification image is not limited thereto, and the image in an arbitrary acquisition position may be used.
• the image in an arbitrary acquisition position may be used.
• a case of using an image acquired in a macula portion or an image acquired along a retinal vessel arcade is also included in one embodiment of the present invention.
• the position alignment portion 131 serving as a position alignment unit performs inter-frame position alignment of the acquired images. Subsequently, the position alignment portion 131 determines an exceptional frame based on the brightness value and noise of each frame and a displacement amount with respect to a reference frame. Specifically, first, the inter-frame position alignment is performed for the wide field angle image Dl and the confocal image Dc . After that, a parameter value of the inter-frame position alignment is also applied to each of the nonconfocal images Dnr and Dnl.
• the inter-frame position alignment is executed by the position alignment portion 131 with the following procedure.
• the position alignment portion 131 first sets the reference frame as the reference of the position alignment.
• the frame having the smallest frame number is set as the reference frame.
• a method of setting the reference frame is not limited thereto, and an arbitrary setting method may be used.
• the position alignment portion 131 performs rough association of positions between frames (rough position alignment) .
• An arbitrary position alignment method can be used therefor, but in this embodiment, a correlation coefficient is used as an inter-image similarity evaluation function, and affine transformation is used as a coordinate transformation method, to thereby perform the rough position alignment.
• the position alignment portion 131 performs fine position alignment based on data on a correspondence relationship of the rough positions between the frames.
• the fine position alignment between the frames is performed for a moving image obtained by being subjected to the rough position alignment in the stage (ii) through use of the free form deformation (FFD) method that is a kind of non-rigid position alignment method .
• FFD free form deformation
• a method for the fine position alignment is not limited thereto, and an arbitrary position alignment method may be used.
• a position alignment parameter obtained by performing the inter-frame position alignment of the confocal image Dc is also used as a parameter for the inter-frame position alignment of the nonconfocal image Dn.
• an execution order or the like of the position alignment is not limited thereto.
• Dc is also included in one embodiment of the present invention.
• the nonconfocal image Dn include not only Dnr and Dnl described above but also an image obtained by performing arithmetic operation processing for Dnr and Dnl .
• the position alignment portion 131 performs the position alignment of the wide field angle image Dl and the high magnification confocal image Dcj (so-called merging of images), and obtains the relative position of the confocal image Dcj on the wide field angle image Dl .
• the merging processing is performed through use of superimposed images of the respective moving images.
• the merging processing may be performed through use of, for example, the reference frames of the respective moving images.
• the position alignment portion 131 acquires the fixation target position Fen used at the time of the image acquisition of the confocal image Dcj from the memory portion 120, and sets the fixation target position Fen as an initial search point of the position alignment parameter for the position alignment of the wide field angle image Dl and the confocal image Dcj . From then on, the wide field angle image Dl and the confocal image Dcj are subjected to the position alignment while a combination of the parameter values is changed.
• the combination of the position alignment parameter values having the highest similarity between the wide field angle image Dl and the confocal image Dcj is determined as the relative position of the confocal image Dcj on the wide field angle image Dl .
• the position alignment method is not limited thereto, and an arbitrary position alignment method may be used.
• the position alignment is performed in ascending order of the magnification from the image having the lowest magnification.
• the position alignment be first performed between the wide field angle image Dl and the medium magnification image Dc2o. In this case, it is preferred that' the above-mentioned position alignment be followed by the position alignment between the medium magnification image Dc2o and the high magnification image Dclm.
• an image merging parameter value determined for the wide field angle image Dl and the confocal image Dcj is also applied to the merging of the nonconfocal images (Dnrk and Dnlk) . Therefore, the relative positions of the high magnification nonconfocal images Dnrk and Dnlk on the wide field angle image Dl are respectively determined.
• the vessel feature acquiring portion 132 that functions as a vessel feature acquiring unit and the cell identifying portion 133 that functions as a cell identifying unit identify cells that form the blood vessel wall with the following procedure. That is, the cell identifying unit identifies the cells that form the blood vessel wall based on membrane candidate points that form an arbitrary wall within the blood vessel acquired by the vessel feature acquiring portion 132.
• a morphology filter is applied to detect, a retinal artery center line.
• the brightness profile on a line segment orthogonal to the artery center line is generated.
• local maximum values are detected at three points from the center of the line segment toward each of the left side and the right side, and are set as candidates for an intima, a media, and an adventitia of the blood vessel wall in the stated order from the position closest to the blood vessel center line.
• membrane candidate points are not acquired from the brightness profile when the number of detected local maximum points is smaller than three.
• membrane candidate points for the media are interpolated along the travel direction of the wall, to thereby generate a curved line along the travel of a blood vessel wall.
• the brightness profile is generated along the curved line generated in the stage (ii) .
• the brightness profile is subjected to a Fourier transform, and then a low-pass filter is applied to a frequency domain, to thereby remove high frequency noise.
• the local maximum values are detected on the brightness profile generated along the travel of the blood vessel wall, which is generated in the stage (iii), to identify positions of the cells that form the blood vessel wall. That is, the cells are identified based on the brightness profile generated along the sequence of the acquired membrane candidate points.
• the brightness profile can also be generated along a curved line parallel with the blood vessel center line within a blood vessel wall region.
• Step S710 to Step S740 illustrated in the flowchart of FIG. 7A.
• the measuring portion 135 calculates a relative distance between the cells based on the positions of the cells that form the blood vessel wall identified in Step S530, and identifies the measuring position of the membrane thickness based on the relative distance.
• a membrane thickness of the media, a compound membrane thickness of the media and the adventitia, and a wall thickness are measured in the identified measuring position.
• Step S750 to Step S770 illustrated in the flowchart of FIG. 7B.
• the display control portion 136 displays the acquired images, the detected positions of the cells that form the blood vessel wall, and measurement results (density of the cells that form the blood vessel wall, membrane thickness, and wall thickness) on the monitor 305.
• the following items (i) to (iv) are displayed. That is,
• (iv) a map for showing the distribution (cell density and area of the cells) of the cells that form the blood vessel wall calculated for each small area (FIG. 8B) are displayed on the monitor 305. Note that, it is preferred that the item (iv) be displayed in colors after the calculated values are associated with a color bar.
• the instruction acquiring portion 140 acquires from the outside an instruction as to whether or not to store the images acquired in Step S510 and the data on the measurement result obtained in Step S540, that is, the values of the positions of the cells that form the blood vessel wall, the membrane thickness, the wall thickness, the density of the cells that form the blood vessel wall, and the like within the nonconfocal image Dnk, in the data server 40.
• the instruction is input by the operator through, for example, the keyboard 306 and the mouse 307.
• the processing advances to Step S570, and when the storing is not instructed, the processing advances to Step S580.
• the image processing portion 130 transmits an inspection date/time, information for identifying the eye to be inspected, and the images and the data on the measurement result, which are determined to be stored in Step S560, to the data server 40 in association with one another.
• the instruction acquiring portion 140 acquires from the outside an instruction as to whether or not to complete the processing relating to the high magnification nonconfocal image Dnk performed by the image processing apparatus 10.
• the instruction is input by the operator through the keyboard 306 and the mouse 307.
• the processing is brought to an end. Meanwhile, when the instruction to continue the processing is acquired, the processing returns to Step S510 to perform the processing for the next eye to be inspected (or reprocessing for the same eye to be inspected) .
• the cell identifying portion 133 In order to identify the cells that form the blood vessel wall, the cell identifying portion 133 first performs an edge preserving smoothing process for the nonconfocal image.
• An arbitrary known edge preserving smoothing process is applicable, but in this embodiment, a median value filter is applied to the nonconfocal images Dnr+Dnl.
• the morphology filter is applied to the smoothed image generated by the cell identifying portion 133 in Step S710 to detect the retinal artery center line.
• a top-hat filter is applied to detect a high brightness region having a narrow width, which corresponds to blood vessel wall reflection. Further, the high . brightness region is subjected to a thinning process to detect the blood vessel center line. Note that, a method of detecting the blood vessel center line is not limited thereto, and an arbitrary known detection method may be used.
• the 133 generates a brightness profile Cr shown in FIG. 61 along a line segment (line segment Prl in FIG. 6H) . orthogonal to the blood vessel center line in the respective positions on the blood vessel center line. Then, the brightness profile Cr is searched for the local maximum point from the center of the line segment toward the left side and the right side.
• the first local maximum point Lmi having such a brightness value that a ratio or difference with respect to the brightness value on the center line falls within a predetermined range is set as a membrane candidate point for the intima
• the second local . maximum point Lmm is set as a membrane candidate point for the media
• the last local maximum point Lmo is set as a membrane candidate point for the adventitia.
• the membrane candidate points are not acquired from the brightness profile when the number of detected local maximum points is smaller than three.
• the local maximum point Lmm for the media detected from the brightness profile obtained in the respective positions on the blood vessel center line (along the line segment orthogonal to the blood vessels center line) is subjected to an interpolation process in the vessel travel direction.
• a membrane candidate point sequence for the media is generated through use of an interpolation value and a plurality of local maximum points aligned in the extending direction of the blood vessel center line, which are obtained above.
• a method of acquiring the membrane candidate point sequence is not limited thereto, and an arbitrary known acquisition method may be used.
• two curved lines parallel with the blood vessel center line are respectively arranged on a blood vessel lumen side and a nerve fiber side as a variable shape model.
• the model may be deformed so as to match a blood vessel wall boundary by minimizing an evaluation function value regarding the shape and the brightness value on the point sequence that forms the model, and the detected blood vessel wall boundary may be acquired as the membrane candidate point sequence .
• the cell identifying portion 133 generates a curved line through the interpolation of the membrane candidate point sequence generated in Step S720, and generates a brightness profile shown in FIG. 6J along the curved line (Pr2 in FIG. 6H) .
• the high frequency component is removed in order to remove a peak component other than the cells that form the wall (noise or light reflected from a fundus tissue other than the cells that form the wall) from the profile.
• the frequency is transformed through use. of a Fourier transform, and a low-pass filter is applied to cut a signal value of the higher frequency component.
• the filtered signal is returned to a spatial domain by being subjected to an inverse Fourier transform, to generate a corrected brightness profile with the high frequency components removed therefrom.
• the cell identifying portion 133 detects the local maximum values (Lmml, Lmm2 , and Lmm3 in FIG. 6K) through the search for the brightness value on the corrected brightness profile generated in Step S730. Based on the obtained local maximum values, the cell positions along the vessel travel direction are identified .
• the measuring position identifying portion 134 calculates a conformance degree of the measuring position.
• a relative distance Ph obtained when a distance between the cell positions is set as 1 is calculated in the respective positions on a curved line obtained by interpolating the cell positions. This corresponds to a relative phase value obtained when an interval between the center positions of cells distributed at regulating intervals is set as 1 cycle. Specifically, assuming that the center of a cell is 0, the edge of the cell is 0.5, and the center of the adjacent cell is 1. Such a relative distance Ph between the cell positions is calculated for all the membranes whose cells have been detected.
• a conformance degree Cf of the measuring position is calculated based on the above-mentioned relative distance Ph.
• the conformance degree Cf is calculated as follows.
• the conformance degree of the measuring position is not limited to the above-mentioned expression for Cf, and an arbitrary expression may be used for the calculation as long as an evaluation value becomes higher in a position closer to the center of a cell and becomes lower in a position closer to the end of the cell.
• the compound membrane thickness for example, compound membrane thickness of the media and the adventitia or blood vessel wall thickness
• a sum of values obtained by weighting the conformance degrees Cf based on a cell size ratio is calculated as the conformance degree of the measuring position.
• a value determined by an arbitrary known method may be used as the cell size ratio.
• a value set in advance based on the kind of membrane is used as the cell size ratio.
• the measuring position identifying portion 134 identifies the . measuring position based on the conformance degree Cf of the measuring position calculated in Step S750. That is, in this embodiment, the measuring position identifying portion 134 functions as a measuring position acquiring unit configured to identify the measuring position regarding the membrane or the wall of the blood vessel based on the positions of identified cells.
• the membrane thickness is measured by selecting a plurality of positions in which the conformance degree Cf is maximum in each membrane as indicated by the white dashed lines in FIG. 6L, and the mean value, standard deviation, maximum value, and minimum value of the membrane thickness are calculated as statistics.
• the membrane thickness is measured in a plurality of positions close to the centers of the cells because the conformance degree becomes higher in the position closer to the center of a cell.
• the conformance degree of the measuring position is calculated by a procedure including steps (i) and (ii) described below.
• Cfm represents the conformance degree of the measuring position for the media
• Cfa represents the conformance degree of the measuring position for the adventitia.
• the positions indicated by the white dashed lines are determined as the measuring positions based on the conformance degrees of the measuring positions used for the measurement of the compound membrane thickness.
• the measuring position identifying portion 134 identifies or determines the measuring position based on the distance between predetermined positions within the cells, in this embodiment, between the centers of the cells, identified in at least one of a plurality of membranes that form a blood vessel wall.
• the predetermined position may be allowed to be appropriately acquired from an image being displayed or the like.
• the measuring portion 135 measures the respective membrane thicknesses of the blood vessel, the compound membrane thickness obtained by summing up the thicknesses of the plurality of membranes, and the wall thicknesses of the blood vessel formed of the plurality of membranes, as measurement values regarding the wall of the blood vessel in the measuring position identified in Step S760. Note that, those measurement items may be at least one of those exemplified above.
• the mean values, standard deviations, maximum values, and minimum values are respectively calculated for the membrane thickness of the media, the compound membrane thickness of the media and the adventitia, and the wall thickness in the measuring positions identified in Step S760.
• Those index values are calculated not only as statistics for the entire image, but also in units of a blood vessel branch, units of one side within the blood vessel branch (right side or left side in terms of the vessel travel direction), or units of a small region.
• indices regarding the thicknesses of the membranes that form the blood vessel wall are not limited thereto, and the index values may be calculated by subjecting the values of the membrane thicknesses calculated for the plurality of membranes to an arithmetic operation.
• the following methods (a) or (b) may be exemplified.
• the compound membrane thickness of the media and the adventitia that easily alter or undergo hypertrophy is standardized with the density of the cells in the intima that relatively hardly change.
• a ratio of the membrane thickness (of the same kind) between the left side wall and the right side wall in terms of the vessel travel direction is set as an index.
• the wall cells travel in a coil shape, and when a membrane thickness abnormality occurs, the membrane thickness abnormality is considered to be liable to occur on both sides. Therefore, the ratio of the membrane thickness is used as the index of reliability regarding the measurement values of the membrane thickness.
• a specification value based on the membrane thicknesses measured for different membranes of the blood vessel or a new index value obtained by subjecting the index values regarding the membrane thicknesses to an arithmetic operation. Further, such a specification value and an index value can also be used for, for example, determination as to appropriateness of the calculation of the thickness of the actual blood vessel wall .
• Step S740 when the distance between the cells that form the blood vessel wall detected in Step S740 falls out of a predetermined range, there may be a case where the cells that form the blood vessel wall have altered or died and hence an appropriate measuring position cannot be identified.
• the cell positions have been detected in a plurality of kinds of membranes and when a cell interval in at least one kind of membrane falls within a predetermined range, it is preferred to identify the measuring position through use of only the conformance degree of the measuring position calculated for the membrane having an appropriate cell interval.
• the cell interval is not appropriate in any kind of membrane, it is preferred to identify the measuring position at predetermined intervals along the travel of the blood vessel wall.
• the image processing apparatus 10 performs the following processing for the image acquired by imaging the wall of the retinal artery through the use of the SLO apparatus configured to simultaneously acquire the confocal image and the nonconfocal image. That is, after detecting the cells that form the retinal vessel wall, the image processing apparatus 10 measures the membrane thickness by identifying the measuring position of the membrane thickness based on the relative distance between the cells calculated in the respective positions along the travel of the blood vessel wall.
• the description of the above-mentioned embodiment is directed to the case where the image acquiring portion 110 includes both the confocal data acquiring portion 111 and the nonconfocal data acquiring portion 112.
• the image acquiring portion 110 does not necessarily include the confocal data acquiring portion 111 as long as the configuration allows the acquisition of at least two kinds of non-confocal data.
• Embodiment ( s ) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium' ) to perform the functions of one or more of the above-described embodiment ( s ) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC) ) for performing the functions of one or more of the above-described embodiment ( s ) , and by a method performed by the .
• ASIC application specific integrated circuit
• the computer may comprise one or more processors (e.g., central processing unit (CPU) , micro processing unit (MPU) ) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.
• the computer executable instructions may be provided to the computer, for example, from a network or the storage medium.
• the storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM) , a read only memory (ROM) , a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM) , a flash memory device, a memory card, and the like.
• a hard disk such as a hard disk (RAM) , a read only memory (ROM) , a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM) , a flash memory device, a memory card, and the like.
• RAM random-access memory
• ROM read only memory
• BD Blu-ray Disc

Landscapes

• Health & Medical Sciences (AREA)
• Engineering & Computer Science (AREA)
• Life Sciences & Earth Sciences (AREA)
• Physics & Mathematics (AREA)
• Medical Informatics (AREA)
• General Health & Medical Sciences (AREA)
• Surgery (AREA)
• Biomedical Technology (AREA)
• Molecular Biology (AREA)
• Ophthalmology & Optometry (AREA)
• Animal Behavior & Ethology (AREA)
• Biophysics (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)
• Heart & Thoracic Surgery (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Physics & Mathematics (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Quality & Reliability (AREA)
• Signal Processing (AREA)
• Eye Examination Apparatus (AREA)

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=55806736

Family Applications (1)

Country Status (3)

Families Citing this family (3)

Family Cites Families (12)

• 2015
  • 2015-03-25 JP JP2015062511A patent/JP6468907B2/ja not_active Expired - Fee Related
• 2016
  • 2016-03-23 WO PCT/JP2016/060285 patent/WO2016153074A1/en not_active Ceased
  • 2016-03-23 US US15/541,755 patent/US20180000338A1/en not_active Abandoned

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events