JP6711617B2 - Image forming apparatus, image forming method and program thereof - Google Patents

Description

The present invention relates to an image forming apparatus, an image forming method and a program thereof.

2. Description of the Related Art Eye examinations are widely performed for the purpose of early diagnosis of lifestyle-related diseases and diseases that are the leading causes of blindness. Scanning laser ophthalmoscope (SLO: Scanning Laser Ophthalmoscope) is used as an ophthalmologic apparatus utilizing the principle of a confocal laser microscope in the examination of the eye. The SLO is a device that raster-scans a fundus as a measurement light on the fundus and obtains a plane image with high resolution and high speed from the intensity of the returned light. Further, in SLO, by detecting only the light that has passed through the opening (pinhole), only the return light at a specific depth position can be imaged, and an image with a higher contrast than that of a fundus camera or the like can be acquired.

Hereinafter, an apparatus that captures such a plane image will be referred to as an SLO apparatus, and the plane image will be referred to as an SLO image.

In recent years, it has become possible to obtain an SLO image of the retina with improved lateral resolution by increasing the beam diameter of measurement light in an SLO device. However, with the increase in the beam diameter of the measurement light, a decrease in the S/N ratio and the resolution of the SLO image due to the aberration of the eye to be examined has become a problem in acquiring the SLO image of the retina.

In order to solve this problem, the adaptive optics having an adaptive optics system in which the aberration of the eye to be inspected is measured in real time by the wavefront sensor and the aberration of the measurement light or its returning light generated in the eye is corrected by the wavefront correction device. SLO devices have been developed. By using an adaptive optics (AO: Adaptive Optics) SLO device (AO-SLO device), a high lateral resolution SLO image can be acquired.

Such a high lateral resolution SLO image can be acquired as a moving image. Therefore, for example, in order to observe the blood flow dynamics non-invasively, it is possible to extract the retinal blood vessels from each frame of the moving image and then measure the blood cell migration speed in the capillaries. Further, in order to evaluate the relationship between the capillaries and the like of the eye to be examined and the visual function using the SLO image, it is possible to detect the photoreceptor cells P and then measure the density distribution and arrangement of the photoreceptor cells P. FIG. 1 shows a schematic example of such a high lateral resolution SLO image. Referring to FIG. 1, a low brightness area Q corresponding to the positions of photoreceptor cells P and capillaries and a high brightness area W corresponding to the positions of white blood cells can be observed.

Here, each layer which comprises a retina is shown in FIG. When observing the photoreceptor cells P in the SLO image, the focus position is set near the outer layer of the retina (B5 in FIG. 2) and an SLO image as shown in FIG. 1 is captured. On the other hand, retinal blood vessels and branched capillaries run in the inner layer of the retina (B2 to B4 in FIG. 2). Therefore, when the adaptive optics SLO image is acquired with the focus position set to the inner layer of the retina, for example, the retinal blood vessel wall can be directly observed.

However, in the confocal image of the inner layer of the retina, the noise signal is strong due to the influence of the light reflected from the nerve fiber layer, which may make it difficult to observe the blood vessel wall or detect the wall boundary.

Therefore, in recent years, a method (Offset Pinhole method) for observing a non-confocal image based on scattered light acquired by changing the diameter, shape, and position of the pinhole in front of the light receiving unit has been used. Since the depth of focus is large in a non-confocal image, it is easy to observe an object having irregularities in the depth direction such as a blood vessel, and it becomes difficult to directly receive the reflected light from the nerve fiber layer, so that noise is reduced.

Non-Patent Document 1 discloses a non-confocal imaging technique that visualizes the retinal blood vessel wall by receiving scattered light after manually adjusting the position and diameter of the opening of the pinhole of such an AO-SLO device. Disclosure.

For the same purpose, a non-confocal imaging method (Split Detection method) has also been developed in which an opening such as a pinhole is divided into two and two types of scattered light are acquired substantially at the same time. In this regard, Non-Patent Document 2 discloses a technique of moving a pinhole of an AO-SLO device at eight equal intervals in the circumferential direction to photograph photoreceptor cells.

On the other hand, the retinal artery observed by the AO-SLO device is an arteriole having a blood vessel diameter of about 10 to 100 μm, and its wall is composed of an intima, a media and an adventitia. Furthermore, the media is composed of smooth muscle cells, and runs in a coil shape in the circumferential direction of blood vessels. When the pressure applied to the retinal artery wall increases due to hypertension or the like, the smooth muscle contracts and the wall thickness of the retinal artery wall increases. At this point, the shape of the retinal artery wall will return to its original shape if the blood pressure is lowered by taking an antihypertensive agent. However, if left as it is for a long time with hypertension, smooth muscle cells composing the media are necrotized and fibrous thickening of the media adventitia is caused to increase the wall thickness of the artery. In this case, organic (irreversible) damage has already occurred in the retinal artery wall, and continuous treatment must be performed to prevent further progression of arteriole damage.

The retinal artery is the only directly observable tissue in the arterioles of the whole body. Therefore, in order to estimate the presence or absence and the degree of organic change of arterioles in the body of a person suffering from hypertension, diabetes, etc., it is desired to clearly delineate the walls and membranes of retinal arteries with the AO-SLO device. There is.

Chui et al.; “The use of forward scatter to improve retinal vascular imaging with an adaptive optics scanning laser ophthalmoscope”, BIOMEDICAL OPTICS EXPRESS, Vol.3, No.10, pp.2537-2549, 2012. Rossi et al.; “Adaptive optics imaging of putative cone inner segments within geographic atrophy lesions”, IOVS, Vol.56, No.7, 4931, 2015.

However, the Offset Pinhole method has a problem that only one blood vessel wall among the blood vessel walls existing on the left and right when viewed from the running direction of the blood vessel can be clearly drawn in one imaging. In this regard, Non-Patent Document 1 also does not disclose a technique for clearly depicting the blood vessel walls on the left and right sides. In the following, the left and right with respect to the blood vessel wall will be described as meaning the left and right as seen from the running direction of the blood vessel in the captured image.

Similarly, in the Split Detection method, there is a problem in that one type of non-confocal signal can clearly depict only one blood vessel wall of the left and right blood vessel walls. In this regard, Non-Patent Document 2 also does not disclose a technique for clearly depicting the blood vessel walls on the left and right sides.

The present invention has been made in view of the above problems, and provides an image forming apparatus, an image forming method, and a program for forming an image in which a blood vessel wall region is clearly drawn on both left and right sides.

The image forming apparatus according to an embodiment of the present invention is an image forming apparatus for forming an image, each of which forms a non-confocal image of the eye, the first that area dividing returned light from the eye A signal acquisition unit that acquires a non-confocal signal and a second non-confocal signal, a first non-confocal image generated from the first non-confocal signal, and a second non-confocal signal And a correction unit that corrects the positional relationship between the first non-confocal signal and the second non-confocal signal so that the positions of the pixels of the second non-confocal image are adjusted. A calculation unit that performs a maximum value calculation using the first non-confocal signal and the second non-confocal signal, and a determination that determines each pixel value of the image to be formed based on the calculation result by the calculation unit. The first non-confocal signal and the second non-confocal signal, which are provided with a section and whose positional relationship is corrected, are signals acquired at the same time.

The image processing method according to an embodiment of the present invention is an image forming method for forming an image, each of which forms a non-confocal image of the eye, the said divided into regions the return light from the eye Acquiring a first non-confocal signal and a second non-confocal signal, and generating a first non-confocal image generated from the first non-confocal signal and the second non-confocal signal And correcting the positional relationship between the first non-confocal signal and the second non-confocal signal so that the positions of the respective pixels of the second non-confocal image are adjusted. Performing a maximum value calculation using the first non-confocal signal and the second non-confocal signal, and determining each pixel value of an image to be formed based on the calculation result of the maximum value calculation. And the first non-confocal signal and the second non-confocal signal whose positional relationship is corrected are signals that are simultaneously acquired.

According to the present invention, there are provided an image forming apparatus, an image forming method, and a program thereof capable of forming an image in which a blood vessel wall region is clearly drawn on both left and right sides.

An example of a high lateral resolution SLO image is shown. An example of the layer which comprises the retina of a fundus is shown. 1 shows a schematic configuration example of a system including an image forming apparatus according to a first embodiment of the present invention. 1 shows a schematic configuration example of an SLO device according to a first embodiment of the present invention. 1 shows a schematic configuration example of a diaphragm and an optical sensor in an SLO device according to a first embodiment of the present invention. 1 shows a schematic configuration example of a diaphragm in an SLO device according to a first embodiment of the present invention. 1 shows a schematic hardware configuration example of an image forming apparatus according to a first embodiment of the present invention. FIG. 1 is a block diagram showing a schematic functional configuration example of an image forming apparatus according to a first embodiment of the present invention. 6 is a flowchart of processing by the image forming apparatus according to the first embodiment of the present invention and its modification. FIG. 3 is a diagram illustrating image processing performed by the image forming apparatus according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating image processing content by the image forming apparatus according to the first embodiment of the present invention. 6 is a flowchart showing details of image forming processing by the image forming apparatus according to the first embodiment of the present invention and its modification. FIG. 6 is a block diagram showing a schematic functional configuration example of an image forming apparatus according to a second embodiment of the present invention. 9 is a flowchart showing details of image forming processing by the image forming apparatus according to the second embodiment of the present invention and its modification. FIG. 9 is a block diagram showing a schematic functional configuration example of an image forming apparatus according to another embodiment of the present invention. 9 is a flowchart showing details of image forming processing by an image forming apparatus according to another embodiment of the present invention.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, dimensions, materials, shapes, relative positions of constituent elements, and the like described in the following embodiments are arbitrary and can be changed according to the configuration of the apparatus to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used in the drawings to denote the same or functionally similar elements. Further, hereinafter, the divided non-confocal signals are referred to as different types of non-confocal signals, and non-confocal images based on the different types of non-confocal signals are referred to as different types of non-confocal images. In this regard, for example, the four non-confocal signals divided into four are referred to as four types of non-confocal signals. Further, for example, a non-confocal signal obtained by combining two out of four non-confocal signals is also treated as one type of non-confocal signal. Further, each of the divided non-confocal signals is also referred to as a signal type. Further, a calculation for selecting (determining) a larger value among the calculated values is called maximum value calculation.

[First Embodiment]
The image forming apparatus according to the first embodiment of the present invention performs the following processing when forming an image of a retinal blood vessel wall using an SLO device that acquires a confocal image and two types of non-confocal images. That is, both images are aligned in order to reduce the parallax between the two types of non-confocal images, and then an image is formed based on the result of the maximum value calculation of the pixel values of both images at each pixel position. As a result, the image forming apparatus according to the present embodiment can form an image with high visibility of blood vessel walls on the left and right sides while suppressing blurring and false images in the image.

(overall structure)
FIG. 3 illustrates a schematic configuration example of a system including the image forming apparatus 10 according to the present exemplary embodiment. As shown in FIG. 3, the image forming apparatus 10 is connected to the SLO device 20, the data server 40, and the temporal data acquisition device 50 via a LAN (Local Area Network) 30. The LAN 30 is composed of an optical fiber, USB, IEEE 1394, or the like. The connection of these devices may be via an external network such as the Internet, or may be a direct connection.

The SLO device 20 captures a wide-angle image Dl of the eye part, and a high-magnification confocal image Dc and a non-confocal image Dn. The SLO device 20 transmits the wide-angle image Dl, the confocal image Dc, the non-confocal image Dn, and the fixation target positions Fl and Fcn used at the time of imaging to the image forming device 10 and the data server 40.

The time phase data acquisition device 50 is a device that acquires biological signal data (time phase data) that changes autonomously, and includes, for example, a pulse wave meter or an electrocardiograph. The time-phase data acquisition device 50 acquires the time-phase data Pi at the same time when the SLO device 20 acquires the wide-angle image Dl, the confocal image Dc, and the non-confocal image Dn according to an operation by an operator (not shown). .. The time phase data acquisition device 50 transmits the obtained time phase data Pi to the image forming apparatus 10 and the data server 40. The time phase data acquisition device 50 may be directly connected to the SLO device 20.

Here, when each image is acquired at a different shooting position, the image acquired at each imaging position is represented as a wide-angle image Dli, a confocal image Dcj, a non-confocal image Dnk, or the like. That is, these reference symbols i, j, and k are variables indicating the shooting position (shooting position number), and i=1, 2,. ． ． , Imax, j=1, 2,. ． ． , Jmax, k=1, 2,. ． ． , Kmax.

The data server 40 includes imaging condition data such as the wide-angle image Dl of the eye to be inspected, the confocal image Dc, the non-confocal image Dn output by the SLO device 20, and the fixation target positions Fl and Fcn used during the imaging. And so on. The data server 40 also holds the time phase data Pi output by the time phase data acquisition device 50, the image of the eye part output by the image forming apparatus 10, the image feature of the eye part, and the like. In this embodiment, the image features of the retinal blood vessel, the retinal blood vessel wall, the center line of the retinal blood vessel, and the membranes and parietal cells constituting the blood vessel wall are handled as the image features of the eye part. In addition, the data server 40, in response to a request from the image forming apparatus 10, images the wide-angle image Dl, the confocal image Dc, the non-confocal image Dn, the time phase data Pi, and the image features of the eye portion that are held. It is transmitted to the forming device 10.

The image forming apparatus 10 uses the confocal image Dc and the non-confocal image Dn acquired by the SLO apparatus 20, the data held in the data server 40, the time phase data Pi acquired by the time phase data acquisition apparatus 50, and the like. To form an image including the retinal blood vessel wall. The formed image can be displayed on an image display device (not shown) or stored in the data server 40.

Next, an SLO device 20 to which an adaptive optics system is applied will be described with reference to FIGS. The SLO device 20 is provided with an SLD (Super Luminescent Diode) 201, beam splitters 202 and 203, an adaptive optical system 204, and an XY scanning mirror 205. The SLO device 20 is further provided with a Shack-Hartmann wavefront sensor 206, a focus lens 209, a diaphragm 210, an optical sensor 211, an image forming unit 212, and an output unit 213.

The SLD 201 is a light source, and the light emitted from the SLD 201 is reflected by the fundus Er of the eye E to be examined. Part of the return light from the fundus Er is input to the Shack-Hartmann wavefront sensor 206 via the second beam splitter 203, and the other return light is input to the optical sensor 211 via the first beam splitter 202. Is entered.

The XY scanning mirror 205 can control the scanning position on the fundus of the light emitted from the SLD 201.

The Shack-Hartmann wavefront sensor 206 is provided with a lens array 207 and a CCD (Charge-Coupled Device) 208, and the CCD 208 is connected to the lens array 207.

When the light incident on the Shack-Hartmann wavefront sensor 206 passes through the lens array 207, a bright spot group appears on the CCD 208. The Shack-Hartmann wavefront sensor 206 measures the wavefront aberration of the subject's eye E based on the displacement of the bright spot projected on the CCD 208 from the predetermined position. The Shack-Hartmann wavefront sensor 206 also sends information about the measured wavefront aberration to the adaptive optics 204.

The adaptive optics 204 is provided with a known aberration correction device such as a deformable mirror or a spatial light phase modulator. The adaptive optics system 204 drives the aberration correction device based on the information about the wavefront aberration received from the Shack-Hartmann wavefront sensor 206, and corrects the aberration of the eye E included in the return light from the eye E. The return light in which the aberration of the subject's eye E is corrected by the adaptive optical system 204 is received by the optical sensor 211 via the focus lens 209 and the diaphragm 210.

The optical sensor 211 receives the return light from the eye E to be examined, generates an electric signal, and sends the electric signal to the image forming unit 212.

The SLO device 20 acquires the data of the imaging target area and time (frame rate×number of frames) designated in advance by the operator via an input device (not shown) or the like.

The image forming unit 212 performs correction of image distortion due to variations in scanning speed and luminance value correction using the data of the imaging target area and the like, and the image data (from the electric signal received from the optical sensor 211 A moving image or a still image). The image forming unit 212 sends the formed image to the output unit 213.

The output unit 213 outputs the image data received from the image forming unit 212 to the image forming apparatus 10, the data server 40, or the like. The output unit 213 may display the image on an image display device (not shown) provided in the SLO device 20.

As long as the SLO device 20 has a configuration capable of acquiring the confocal image Dc and the non-confocal image Dn, the diaphragm 210 and the optical sensor 211 as shown in FIG. 4 may have arbitrary configurations. In the present embodiment, the diaphragm 210 and the optical sensor 211 portion include a light shielding unit 210-1 and optical sensors 211-1 211-2, 211-3 as shown in FIG. Here, FIG. 5 shows a schematic configuration of the diaphragm 210 and the optical sensor 211 of the SLO device 20 in the present embodiment.

The light shielding unit 210-1 in this embodiment will be described with reference to FIG. 6A to 6H show a configuration example of the diaphragm 210 (light blocking portion 210-1). 6A to 6H, the light shielding unit 210-1 according to the present embodiment is shown in FIG. 6A, and a modified example of the diaphragm 210 (light shielding unit 210-1) is shown in FIG. 6B. To (h).

The light shielding unit 210-1 in the present embodiment is formed of a reflection region 210-1-1, transmission regions 210-1-2 and 210-1-3, and a light shielding region (thick line portion) dividing the transmission region. As shown in FIG. 5, the light shielding unit 210-1 is arranged on the image plane of the SLO optical system, and is arranged so that the center of the light shielding unit 210-1 is located at the optical axis center of the return light RL. The light blocking unit 210-1 has an elliptical pattern that is circular when viewed obliquely from the optical axis of the return light RL when viewed from the optical axis direction.

As shown in FIG. 5, a part of the return light RL that is incident on the light blocking section 210-1 arranged on the image plane is reflected by the reflection region 210-1-1 located in the central portion of the light blocking section 210-1. It is incident on the optical sensor 211-1. On the other hand, the return light RL that has passed through the transmissive regions 210-1-2 and 210-1-3 of the light blocking unit 210-1 is split by the prism 210-2 arranged on the image plane, as shown in FIG. , And enters the optical sensors 211-2 and 211-3, respectively.

Each of the optical sensors 211-1, 211-2, 211-3 receives the return light RL and generates a voltage signal. The voltage signal generated by each of the optical sensors 211-1, 211-2, 211-3 is sent to the image forming unit 212, converted into a digital value by the AD board in the image forming unit 212, and then converted into a two-dimensional image. To be converted.

Here, the digital signal generated based on the light incident on the optical sensor 211-1 becomes a confocal signal focused on a specific narrow range in the depth direction, and an image formed from the confocal signal is co-focused. It is a focused image. Further, the digital signal generated based on the light incident on the optical sensors 211-2 and 211-3 becomes a non-confocal signal focused in a wide range in the depth direction, and is formed from the non-confocal signal. The image is a non-confocal image. Therefore, the SLO device 20 divides the return light from the eye E by using the light shielding unit 210-1, and acquires a confocal image and two types of non-confocal images from the divided light.

The method for dividing the non-confocal signal is not limited to this, and the non-confocal signal may be divided into an arbitrary number of 2 or more and the number of photosensors according to the division number may be used. For example, as shown in FIG. 6B, a non-confocal signal is divided into four by using a light-shielding portion obtained by dividing the transmission area into four, or eight transmission areas as shown in FIG. The non-confocal signal may be divided into eight using the light-shielding portion divided into. Further, without providing a reflective area, the transmissive area is divided into two, four, or eight as shown in FIGS. 6D, 6E, and 6F, respectively, and a plurality of types of non-confocal signals are obtained. It may be configured such that only the above are acquired substantially at the same time.

Furthermore, the method of acquiring the confocal signal and the non-confocal signal is not limited to this. For example, the diameter and position of the diaphragm 210 (opening) may be variable. In this case, for example, the diameter of the opening is reduced to obtain a confocal signal as shown in FIG. 6(g), or the diameter of the opening is increased to obtain a non-confocal signal as shown in FIG. 6(h). The diameter and position of the openings can be adjusted to obtain. The diameter of the opening and the amount of movement may be set arbitrarily. For example, as shown in FIG. 6(g), the diameter of the opening is set to about 1 ADD (Airy Disc Diameter), or as shown in FIG. 6(h), the diameter of the opening is set to about 10 ADD. The amount can be set to about 6 ADD.

In this embodiment, since the non-confocal signal is divided into two, there are two types of non-confocal signals. Therefore, in the following, for convenience, the non-confocal image formed from the non-confocal signal on one side will be referred to as the R channel non-confocal image Dnr, and the non-confocal image formed from the other non-confocal signal will be referred to as the L channel. Non-confocal image Dnl. In addition, when described as a non-confocal image Dn, it refers to both the R-channel non-confocal image Dnr and the L-channel non-confocal image Dnl.

Further, by increasing the swing angle of the scanning optical system (XY scanning mirror 205) shown in FIG. 4 and instructing the adaptive optical system 204 not to perform aberration correction, the SLO device 20 operates as a normal SLO device. However, it is also possible to acquire a wide-angle image.

In the following, the image with the lowest magnification among the images acquired from the SLO device 20 by the image forming apparatus 10 has a lower magnification than the confocal image Dc and the non-confocal image Dn that are high-magnification images. A wide-angle image Dl (Dlc, Dln) is set. The wide-angle image Dl may be an SLO image to which adaptive optics is applied, or may be a simple SLO image.

Next, the hardware configuration of the image forming apparatus 10 will be described with reference to FIG. As shown in FIG. 7, the image forming apparatus 10 includes a central processing unit (CPU) 301, a memory (RAM) 302, a control memory (ROM) 303, an external storage device 304, a monitor 305, a keyboard 306, a mouse 307, and An interface 308 is provided.

The external storage device 304 stores a control program for realizing the image processing function according to the present embodiment and data used when the control program is executed. The CPU 301 functions as each unit described below by fetching these control programs and data stored in the external storage device 304 into the RAM 302 via the bus 309 and executing the control programs.

An example of the functional configuration of the image forming apparatus 10 according to this embodiment will be described with reference to FIG. 8. FIG. 8 is a block diagram showing a schematic functional configuration example of the image forming apparatus 10. The image forming apparatus 10 includes a signal acquisition unit 110, a storage unit 120, an image processing unit 130, and an instruction acquisition unit 140.

The signal acquisition unit 110 is provided with a confocal data acquisition unit 111, a non-confocal data acquisition unit 112, and a time phase data acquisition unit 113. The confocal data acquisition unit 111 requests the SLO device 20 to acquire a confocal image Dcj that is a high-magnification image. Similarly, the non-confocal data acquisition unit 112 requests the SLO device 20 to acquire non-confocal images Dnrk and Dnlk, which are high-magnification images. The signal acquisition unit 110 also requests the SLO device 20 to acquire the fixation target positions Fl and Fcn corresponding to these images. The time phase data acquisition unit 113 requests the time phase data acquisition device 50 to acquire the time phase data Pi regarding the biological signal. The signal acquisition unit 110 stores these acquired images and data in the storage unit 120.

The image processing unit 130 is provided with a registration unit 131, a correction unit 132, a calculation unit 133, a feature amount calculation unit 134, a determination unit 135, a display control unit 136, and a storage unit 137.

The alignment unit 131 aligns the frames of each moving image stored in the storage unit 120.

The correction unit 132 is provided with a heterogeneous signal alignment unit 1321 and a brightness correction unit 1322. The heterogeneous signal alignment unit 1321 performs alignment between the respective superimposed images generated from the moving images in which the alignment between the frames has been performed by the alignment unit 131 with respect to the acquired two types of non-confocal images. To do. The brightness correction unit 1322 adjusts the brightness distributions of both images to be uniform when the brightness distributions of these images are significantly different.

The calculation unit 133 performs calculation processing between the non-confocal signals corresponding to the pixel positions at each pixel position of the two types of non-confocal images whose positions have been adjusted by the correction unit 132. In this embodiment, the maximum value calculation is performed between the non-confocal signals.

The determination unit 135 determines the pixel value at each pixel position based on the calculation result of the calculation unit 133, and forms an image in which the blood vessel walls on the left and right sides are clearly drawn.

The display control unit 136 includes a moving image acquired by the SLO device 20, an image formed by determining the pixel value by the determination unit 135, and a confocal image Dc that is a high-magnification image for the wide-angle image Dl. An image obtained by combining the non-confocal images Dn is displayed on the monitor 305.

The storage unit 137 controls the storage of these images and the values of the alignment parameters used during image formation in the data server 40 and the like.

The feature amount calculation unit 134 calculates a feature amount such as a brightness contrast and a spatial differential value regarding the non-confocal signals that form each of the superimposed images generated from the moving images that are aligned by the alignment unit 131. .. In the present embodiment, the calculation unit 133 calculates the maximum value of the signal value (luminance value) between the non-confocal signals, and thus the feature amount calculation unit 134 may not be provided. On the other hand, when the calculation unit 133 performs the calculation on the feature amount, the determination unit 135 performs the calculation between the non-confocal signals using the feature amount calculated by the feature amount calculation unit 134.

The instruction acquisition unit 140 acquires an instruction input by the operator via the keyboard 306 or the mouse 307. The instruction includes, for example, whether to save the image or the like in the data server 40.

Hereinafter, the processing by the image forming apparatus 10 will be described with reference to the flowchart of FIG.

When the process by the image forming apparatus 10 is started, step S910 is executed.

<Step S910>
In step S910, the confocal data acquisition unit 111 and the non-confocal data acquisition unit 112 of the signal acquisition unit 110 generate high-magnification images (the confocal image Dcj and the non-confocal images Dnrk and Dnlk) for the SLO device 20. Request acquisition. The signal acquisition unit 110 also requests acquisition of the fixation target positions Fl and Fcn corresponding to these images.

In response to the acquisition request, the SLO device 20 receives the wide-angle image Dl, the confocal image Dcj, the non-confocal images Dnrk, Dnlk, the attribute data corresponding to these images such as the angle of view and the frame rate, and the fixation target position Fl, Fcn is acquired and transmitted to the image forming apparatus 10. The signal acquisition unit 110 receives the wide-angle image Dl, the confocal image Dcj, the non-confocal images Dnrk and Dnlk, the attribute data and the fixation target positions Fl and Fcn from the SLO device 20 via the LAN 30, and the storage unit. It is stored in 120.

Here, examples of the confocal image Dc and the non-confocal images Dnr and Dnl when the retinal blood vessels are imaged are shown in FIGS. 10A to 10C, respectively. FIGS. 10A to 10C show the captured confocal image Dc and the non-confocal images Dnr and Dnl, respectively. As shown in FIG. 10A, in the confocal image Dc, the reflection of the nerve fiber layer in the background is strong and the contrast in the vicinity of the blood vessel wall tends to be low. Further, as shown in FIG. 10B, in the R channel non-confocal image Dnr, the contrast of the right blood vessel wall is high. On the other hand, as shown in FIG. 10C, in the non-confocal image Dnl of the L channel, the contrast of the blood vessel wall on the left side is high.

The confocal image Dc or the non-confocal image Dn, which is a high-magnification image, may be captured at any position on the fundus Er. For example, the confocal image Dc and the non-confocal image Dn may be acquired around the optic papilla or may be acquired along the retinal blood vessel arcade.

Further, the time phase data acquisition unit 113 requests the time phase data acquisition device 50 to acquire the time phase data Pi regarding the biological signal. In the present embodiment, a pulse wave meter is used as the time phase data acquisition device 50, and pulse wave data is acquired from the ear lobe (ear lobe) of the subject. Here, the pulse wave data is expressed as a sequence of points having the acquisition time on one axis and the pulse wave signal value measured by the pulse wave meter on the other axis.

The time phase data acquisition device 50 acquires the time phase data Pi in response to the acquisition request and transmits it to the image forming apparatus 10. The time phase data acquisition unit 113 receives the pulse wave data, which is the time phase data Pi, from the time phase data acquisition device 50 via the LAN 30. The time phase data acquisition unit 113 stores the received time phase data Pi in the storage unit 120.

Here, in the present embodiment, immediately after the image acquisition request by the confocal data acquisition unit 111 and the non-confocal data acquisition unit 112 and the acquisition of the time phase data Pi by the time phase data acquisition unit 113, the image and the time phase data are immediately acquired. Start to get Pi. On the other hand, the confocal data acquisition unit 111 and the non-confocal data acquisition unit 112 perform SLO in accordance with a certain phase of the time phase data Pi acquired by the time phase data acquisition device 50 according to a request from the time phase data acquisition unit 113. Image acquisition from the device 20 may be started.

The signal acquisition unit 110 acquires the time phase data Pi at the time of acquiring each image from the time phase data acquisition unit 113 or the storage unit 120, detects the extreme value of each time phase data Pi, and detects the heartbeat cycle and heart time. The relative value of the phase (relative cardial cycle) is calculated. The relative value of the cardiac phase is a relative value represented by a floating point number from 0 to 1 when the heartbeat cycle is 1. Here, since each time phase data Pi is acquired at substantially the same time as each acquired image, they have a corresponding relationship. Therefore, the relative value of the cardiac time phase indicates the timing at which the image and the time phase data are acquired in the heartbeat cycle. The signal acquisition unit 110 stores the calculated relative value of the cardiac cycle and the cardiac phase in the storage unit 120.

<Step S920>
In step S920, the alignment unit 131 performs inter-frame alignment of the moving images acquired in step S910. In the present embodiment, the following method is executed as a specific interframe alignment method.

i) The alignment unit 131 sets a reference frame that serves as a reference for alignment. In this embodiment, the frame having the smallest frame number is used as the reference frame. The reference frame setting method is not limited to this, and any setting method may be used.

ii) The alignment unit 131 uses the set reference frame to roughly associate the frames (coarse alignment). Although any registration method can be used, in this embodiment, rough registration is performed using a correlation coefficient as an inter-image similarity evaluation function and Affine conversion as a coordinate conversion method.

iii) The alignment unit 131 performs precision alignment between frames based on the data of the rough positional correspondence between frames obtained in ii). In the present embodiment, the coarse registration-completed moving image obtained in ii) is subjected to precise registration between frames using an FFD (Free Form Deformation) method, which is a kind of non-rigid registration method. The precision alignment method is not limited to this, and any alignment method may be used.

Further, in the present embodiment, the inter-frame alignment is performed for each moving image, but the inter-frame alignment method is not limited to this. For example, the alignment unit 131 may use the alignment parameter obtained by performing the inter-frame alignment on the confocal image Dc as the inter-frame alignment parameter of the non-confocal image Dn. Alternatively, the alignment parameter obtained by the inter-frame alignment of the non-confocal images Dn (including the non-confocal images Dnr, Dnl and the images obtained by the arithmetic processing between these images) is used as the frame of the confocal image Dc. It may be used as an inter-position alignment parameter.

Note that the alignment unit 131 determines the exceptional frame in each acquired moving image based on the luminance value and noise of each frame and the displacement amount between each frame and the reference frame during the inter-frame alignment. It can also be determined. For example, a frame in which the luminance value of the entire frame is lower than a predetermined threshold value, a frame in which noise is higher than a predetermined threshold value, or a frame whose displacement amount from the reference frame is higher than a predetermined threshold value is determined from a moving image used in the processing described below. Judge as an exception frame to be excluded. In this case, the frame determined as the exceptional frame by the alignment unit 131 is excluded from the processing described below.

The alignment unit 131 stores, in the storage unit 120, the moving image thus aligned between the frames.

<Step S930>
In step S930, the image processing unit 130 forms an image Df with high visibility of the blood vessel walls on both sides by the following procedure.

i) Overlaid image generation for each moving image The image processing unit 130 generates each overlaid image by projecting each moving image that has undergone inter-frame registration in step S920 on an average value. In the present embodiment, a superimposed image of confocal images and two types of non-confocal superimposed images are generated.

ii) Positioning/brightness adjustment between heterogeneous images The heterogeneous signal alignment unit 1321 reduces the parallax between the two types of non-confocal superimposed images generated in i), so Align the. Specifically, the heterogeneous signal alignment unit 1321 corrects the non-confocal signals of both images so that the positions of the non-confocal superimposed images are aligned. In this specification, the term “parallax” is used to mean “difference in imaging position of an object caused by difference in observation position”.

This is because the relative positions of both transmission regions (210-1-2, 210-1-3) of the light shielding unit 210-1 with respect to the eye E to be inspected are different, which causes non-confocal image Dnr of R channel and L channel. Parallax occurs between the non-confocal image Dnl. Therefore, parallax also exists between the non-confocal superimposed image obtained by superimposing the R channel non-confocal image Dnr and the non-confocal superimposed image obtained by superimposing the L channel non-confocal image Dnl. Occurs.

As an example, the luminance profile acquired at the same position (dotted line portion) of the R channel non-confocal image Dnr shown in FIG. 10B and the L channel non-confocal image Dnl shown in FIG. It shows in (a). In FIG. 11A, the solid line shows the luminance profile for the R channel non-confocal image Dnr, and the dotted line shows the luminance profile for the L channel non-confocal image Dnl. With reference to FIG. 11A, it can be seen that parallax occurs between both luminance profiles.

As described above, in the non-confocal images Dnr and Dnl, only the corresponding blood vessel wall on one side has high contrast. For this reason, it is conceivable to perform, for example, the arithmetic mean calculation of the R-channel non-confocal superimposed image and the L-channel non-confocal superimposed image as a calculation for making the visibility of the blood vessel walls on the left and right sides comparable. However, even if only the averaging calculation is performed on both images, the parallax between the images due to the relative positions of the transmissive regions (210-1-2, 210-1-3) in the aperture 210 being different causes , A blurred image is generated. Further, as another calculation for making the visibility of the blood vessel walls on the left and right sides comparable, for example, it is conceivable to calculate the maximum value of the R channel non-confocal superimposed image and the L channel non-confocal superimposed image. .. However, even if the maximum value is simply calculated for both images, the blood column reflection (high-intensity region extending to the center of the blood vessel) is rendered thicker than it actually is due to the parallax between the images. In order to prevent "blurring" and false images (artifacts) caused by the calculation of images including such parallax, it is necessary to reduce the parallax between both non-confocal images.

Therefore, the heterogeneous signal alignment unit 1321 reduces the parallax by performing alignment between the non-confocal superimposed images before performing the arithmetic processing between the images. As described above, the R channel non-confocal image Dnr shown in FIG. 10B and the L channel non-confocal image Dnl shown in FIG. 10C are acquired at the same position (dotted line portion). Parallax occurs in the luminance profile as shown in (a). On the other hand, the parallax is reduced as shown in FIG. 11B by performing the alignment between the non-confocal superimposed images. FIG. 11( b) shows the brightness profile corresponding to FIG. 11( a) after alignment of both non-confocal superimposed images. Although any known registration method can be applied as the registration method, the same method as the non-rigid body registration method used in step S920 is used in this embodiment.

The heterogeneous signal alignment unit 1321 stores the non-confocal superimposed image that has undergone the alignment in the storage unit 120.

In addition, as shown in FIG. 11C, the brightness correction unit 1322 performs the aligned non-confocal overlay when the brightness distributions of the different types of non-confocal overlay images are largely different from each other for some reason. The brightness distribution characteristics of the image are adjusted (gradation conversion processing). Accordingly, the brightness correction unit 1322 can prevent the influence of a specific type of non-confocal signal from becoming excessively large in the calculation result between non-confocal superimposed images described below. The brightness correction unit 1322 stores the non-confocal superimposed image with adjusted brightness distribution in the storage unit 120. Note that FIG. 11C shows an example of the brightness profile of each image when the brightness distributions are greatly different between different types of non-confocal superimposed images. In FIG. 11C, the solid line shows the luminance profile for the R channel non-confocal image Dnr, and the dotted line shows the luminance profile for the L channel non-confocal image Dnl.

iii) Calculation between non-confocal signals at the target pixel position The calculation unit 133 corresponds to each pixel position of the formed image in order to generate an image in which the blood vessel walls on the left and right sides are highly visible. Yes, the arithmetic processing is performed between the non-confocal signals of the two types of non-confocal superimposed images. In the present embodiment, the calculation unit 133 determines the maximum between the non-confocal signals of the two types of non-confocal superimposed images aligned in ii) at the pixel (pixel of interest) position where the pixel value in the image to be formed is to be determined. A value calculation (first calculation) is performed. After that, the determination unit 135 determines the pixel value at each pixel position based on the calculation result of the maximum value calculation by the calculation unit 133. Thereby, the image forming apparatus 10 forms the image Df in which the blood vessel walls on the left and right sides are highly visible. Note that specific processing in the image forming processing will be described later using steps S1210 to S1250 shown in FIG. The determination unit 135 stores the formed image Df in the storage unit 120.

<Step S940>
Next, in step S940, the alignment unit 131 aligns the wide-angle image Dl and the high-magnification image Df generated in step S930, and the high-magnification image Df on the wide-angle image Dl is aligned. Find the relative position. Then, the alignment unit 131 attaches the wide-angle image Dl and the image Df based on the obtained relative position. Specifically, the relative position is obtained and the images are pasted as follows.

The alignment unit 131 acquires from the storage unit 120 the fixation target position Fcn used when capturing the high-magnification confocal image Dcj, and the alignment parameter in the alignment between the wide-angle image Dl and the high-magnification image Df. Is the initial point of the search. The alignment unit 131 aligns the wide-angle image Dl and the high-magnification image Df while changing the combination of the parameter values.

The alignment unit 131 determines the combination of the alignment parameter values having the highest similarity between the wide-angle image Dl and the image Df as the relative position of the image Df with respect to the wide-angle image Dl. The alignment method is not limited to this, and any alignment method may be used.

The alignment unit 131 attaches the image Df on the wide-angle image Dl based on the value of the alignment parameter thus obtained. The alignment unit 131 stores the combined image in the storage unit 120.

In addition, the alignment unit 131 uses the alignment parameter values obtained for the wide-angle image Dl and the image Df to set the confocal image Dc and the non-confocal image Dn acquired in step S910 on the wide-angle image Dl. It may be attached to.

The wide-angle image Dl and the confocal image Dcj (or the non-confocal images Dnr and Dnl) obtained in step S910 may be obtained by aligning the alignment parameter values and may be combined. Further, in this case, the high-magnification image on the wide-angle-of-view image Dl is obtained by applying the alignment parameter value obtained for the wide-angle-of-view image Dl and the image acquired in step S910 to the bonding of the images Df. The relative position of the image Df may be determined.

<Step S950>
In step S950, the display control unit 136 displays on the monitor 305 the image acquired in step S910, the image Df generated in step S930, and the image subjected to the combining process in step S940.

In the present embodiment, the display control unit 136 selects a frame corresponding to a specific phase of a pulse wave in a moving image subjected to inter-frame alignment in step S920 and a moving image subjected to inter-frame alignment, which will be described later. Then, the superimposed image is displayed on the monitor 305.

The display control unit 136 also displays the image Df generated in step S930 on the monitor 305.

Further, the display control unit 136 combines the combined image (the image obtained by combining the wide-angle image Dl and the image acquired in step S910, or the wide-angle image Dl and the image Df generated in step S930). The processed image) is displayed on the monitor 305.

<Step S960>
In step S960, the instruction acquisition unit 140 externally acquires an instruction as to whether or not to save the image acquired in step S910, the image Df generated in step S930, and the values of alignment parameters in the data server 40. This instruction is input by the operator via the keyboard 306 or the mouse 307, for example. The image processing unit 130 advances the processing to step S970 when the saving is instructed, and advances the processing to step S980 when the saving is not instructed.

<Step S970>
In step S970, the storage unit 137 associates the examination date/time and information for identifying the eye to be examined with the image to be stored, the fixation position, and the alignment parameter value determined in step S960, and transmits them to the data server 40.

<Step S980>
In step S980, the instruction acquisition unit 140 externally acquires an instruction as to whether or not to end the image forming processing by the image forming apparatus 10. This instruction is input by the operator via the keyboard 306 or the mouse 307. When the processing end instruction is acquired, the processing ends. On the other hand, when the instruction to continue the process is acquired, the process is returned to step S910, and the process for the next eye to be inspected (or the re-process for the same eye to be inspected) is performed.

Further, the details of the image forming process executed in step S930 will be described with reference to FIGS. 9, 11, and 12A. FIG. 12A is a flowchart of the image forming process executed in step S930. The image forming process will be described below with reference to the flowchart shown in FIG.

<Step S1210>
When the image forming process is executed in step S930, in step S1210, the image processing unit 130 averages the moving images aligned in the frames in step S920 to generate a superimposed image. When performing the superimposing process, in order to avoid changing the position of the membrane boundary forming the blood vessel wall due to the influence of heartbeat, the superimposing process is performed by selecting the frame corresponding to the specific phase of the pulse wave. It is desirable to do. In this case, the image processing unit 130 can select the frame corresponding to the specific phase of the pulse wave based on the relative value of the cardiac phase calculated in step S910.

In the present embodiment, the image processing unit 130 generates a superimposed image Dac of confocal images and two types of non-confocal superimposed images (Dar, Dal). The image processing unit 130 also stores the generated image in the storage unit 120.

<Step S1220>
As described above, parallax occurs between the superimposed images (Dac, Dar, Dal) generated in step S1210. Therefore, in step S1220, the heterogeneous signal alignment unit 1321 performs non-rigid body alignment processing between different types of non-confocal superimposed images Dar and Dal in order to reduce the parallax. Specifically, the heterogeneous signal alignment unit 1321 corrects the non-confocal signals of both images so that the non-confocal superimposed images Dar and Dal are aligned. As described in step S930, any known registration method can be applied as the registration method, but in the present embodiment, the same method as the non-rigid body registration method used in step S920 is used. The heterogeneous signal alignment unit 1321 stores the non-confocal superimposed image that has been aligned in the storage unit 120.

In this embodiment, the non-confocal images (Dar and Dal) are aligned with the reference image using the superimposed image of the confocal images as the reference image. The invention is not limited to this, and one of the two types of non-confocal images may be used as a reference image to perform direct alignment between different types of non-confocal images.

In addition, as shown in FIG. 11C, the brightness distributions may differ greatly between different types of non-confocal superimposed images Dar and Dal. In this case, the brightness correction unit 1322 performs calculation between the non-confocal signals of both non-confocal superimposed images in order to prevent the signal component of the specific non-confocal superimposed image from excessively affecting the calculation result. Before, the gradation conversion processing is performed to make the luminance distribution characteristics of the non-confocal superimposed image uniform.

Any known method can be applied to the method of correcting the luminance distribution. In the present embodiment, a method of linearly converting the average luminance Xa and the standard deviation Sx of the luminance of one image X so as to match the average luminance Ya and the standard deviation Sy of the luminance of the other image Y is used. In this method, for example, the linear conversion Y=aX+b is performed such that the coefficient a is a=Sy/Sx and the coefficient b is b=Ya−(Sy/Sx)·Xa.

For example, the brightness profiles as shown in FIG. 11C are aligned with each other as shown in FIG. 11D by the non-rigid body alignment processing between the non-confocal signals by the heterogeneous signal alignment unit 1321. Further, the brightness correction processing by the brightness correction unit 1322 corrects the brightness distribution as shown in FIG. By selecting the maximum value between the signals of both non-confocal superimposed images (maximum value calculation) from the brightness profile shown in FIG. 11E, an image with improved visibility of the blood vessel walls on the left and right sides is obtained. I understand. Here, FIG. 11D shows that the luminance profiles shown in FIG. 11C are aligned with each other by the non-rigid body alignment processing, and FIG. 11E is shown in FIG. 11D. The corrected luminance distribution is shown for the generated luminance profile.

The brightness correction unit 1322 stores the image thus subjected to the brightness correction (gradation conversion) processing in the storage unit 120.

<Step S1230>
In step S1230, in order to form an image in which the blood vessel walls on both the left and right sides are highly visible, the calculation unit 133 sets the pixel positions of the images to be formed between the non-confocal signals of different types of non-confocal superimposed images. Then, the calculation processing is performed and the calculation result is sent to the determining unit 135. In the present embodiment, the maximum value calculation is performed between the signal values of the non-confocal signals of the two types of non-confocal superimposed images that have undergone the positioning and the luminance correction in step S1220. Then, the determination unit 135 determines the pixel value at each pixel position based on the calculation result of the calculation unit 133. By performing the processing for all pixel positions, it is possible to form an image Df with high visibility of the left and right blood vessel walls in which blurring and false images in the image are suppressed, as shown in FIG. .. FIG. 10E shows an example of an image Df in which the blood vessel walls on the left and right sides are highly visible.

<Step S1240>
In step S1240, the image processing unit 130 advances the process to step S940 if the arithmetic processing between the non-confocal signals has been completed for all pixels, and if there are still pixels for which the arithmetic processing is not completed, The process proceeds to step S1250.

<Step S1250>
In step S1250, the image processing unit 130 moves to an adjacent pixel position for which the calculation processing between the non-confocal signals has not been completed, and then advances the processing to step S1230.

As described above, the image forming apparatus 10 according to the present embodiment includes the signal acquisition unit 110 that acquires a plurality of types of non-confocal signals, each of which forms the non-confocal image Dn of the subject. The image forming apparatus 10 also includes a correction unit 132 that corrects a plurality of types of non-confocal signals so that the positions of the non-confocal images Dn are matched. Furthermore, the image forming apparatus 10 calculates the maximum value using a plurality of types of non-confocal signals that have been corrected, and calculates each pixel value of the image to be formed based on the calculation result by the calculation unit 133. And a determining unit 135 for determining. Here, the maximum value operation is an operation for selecting a larger value among the values used for the operation. It should be noted that the plurality of types of non-confocal signals are acquired using a diaphragm in which the area of the opening is divided into a plurality of areas.

According to the above configuration, the image forming apparatus 10 forms an image of the retinal blood vessel wall based on the two types of non-confocal images Dnr and Dnl acquired using the SLO device 20. At this time, the image forming apparatus 10 aligns both images in order to reduce the parallax between the two types of non-confocal images Dnr and Dnl, and then, between the non-confocal signals for each pixel position of the image to be formed. An image is formed on the basis of the result of the maximum value calculation in. As a result, the image forming apparatus 10 can form an image in which the blood vessel walls on the left and right sides are highly visible.

Further, the correction unit 132 in the image forming apparatus 10 according to the present embodiment changes the non-confocal signals of the non-confocal images Dnr and Dnl so that the luminance distribution characteristics between the non-confocal images Dnr and Dnl are aligned (similar to each other). to correct. For example, the brightness correction unit 1322 corrects the non-confocal signals of the non-confocal images Dnr and Dnl so that the scale of the brightness distribution between the non-confocal images Dnr and Dnl is made uniform. Accordingly, the image forming apparatus 10 can prevent the signal component of the specific non-confocal superimposed image from excessively affecting the calculation result by the calculation unit 133.

In the image forming apparatus 10 according to the present embodiment, the pixel value is determined based on the result of maximum value calculation at each pixel position after aligning two types of non-confocal superimposed images. On the other hand, even when the maximum value calculation is performed at each pixel position without performing alignment between the two types of non-confocal superimposed images, an image with improved visibility in a region having a high luminance value is formed. can do. Therefore, even in this case, it is possible to form an image with improved visibility of the blood vessel walls on the left and right sides.

However, as described above, since there is a parallax between the two types of non-confocal superimposed images, both images are displaced from each other based on the parallax. Therefore, if the maximum value calculation is performed at each pixel position without performing alignment between the two types of non-confocal superimposed images, the maximum value calculation is performed on high-luminance regions that are offset from each other. The area may be drawn thicker than it actually is. Therefore, in this case, for example, in the image shown in FIG. 10B, a region having high brightness due to so-called blood column reflection, which extends to the center of the blood vessel, is rendered thick by the maximum value calculation, and is actually drawn. An image suitable for the state of the eye E to be inspected may not be obtained.

On the other hand, in the image forming apparatus 10 according to this embodiment, the parallax between the two images can be reduced by performing the alignment between the two types of non-confocal superimposed images. Therefore, in the image forming apparatus 10, for example, it is possible to suppress the occurrence of false images (artifacts) and blurring due to the high-intensity region being drawn thick based on the shift between the two images, and to obtain a clearer image. Can be formed.

Further, in the image forming process according to the present embodiment, the pixel value at each pixel position is determined by the maximum value calculation between the non-confocal signals. However, the calculation between the non-confocal signals by the calculation unit 133 for determining the pixel value is not limited to this. For example, arithmetic mean calculation may be performed between non-confocal signals. In this case, one of the blood vessel walls, which is rendered with low brightness, can be rendered more clearly, and the occurrence of blurring in the image due to the image forming process is suppressed as compared with the conventional image forming process. However, it is possible to form an image in which the blood vessel walls on both the left and right sides are clearly depicted.

Note that, in the present embodiment, a case has been described in which images based on a plurality of types of non-confocal signals acquired at substantially the same time by the Split Detection method are used to form an image in which the visibility of blood vessel walls on the left and right sides is improved. However, the method of acquiring a plurality of types of non-confocal signals is not limited to this. For example, different types of non-confocal signals may be acquired by the Offset Pinhole method.

Further, in the present embodiment, the signal value (luminance value) of the non-confocal signal itself is used for the calculation between the non-confocal signals of different non-confocal superimposed images to improve the visibility of the left and right blood vessel walls. The case of forming the allowed image has been described. However, what is used for the calculation between the non-confocal signals is not limited to the signal value of the non-confocal signal. For example, in the procedure as shown in FIG. 12B, the pixel value is determined by selecting the signal having the maximum feature amount (contrast of brightness or absolute value of spatial differentiation) of different types of non-confocal signals, and determining the pixel value. You may form the image which improved the visibility of the wall.

FIG. 12B shows a flowchart of the image forming process when the feature amount in the non-confocal superimposed image is used as the target for the maximum value calculation. The flowchart shown in FIG. 12B is the same as the flowchart shown in FIG. 12A except step S1231, and therefore the description will be centered on step S1231, which is the difference, and the description of other parts will be omitted.

<Step S1231>
In step S1231, the feature amount calculation unit 134 calculates the feature amount (luminance contrast or spatial derivative) for the two types of non-confocal superimposed images Dar and Dal subjected to the alignment and the luminance correction in step S1221. The feature amount is stored in the storage unit 120.

The calculation unit 133 performs maximum value calculation on the feature amounts of the non-confocal signals of the two types of non-confocal images calculated by the feature amount calculation unit 134 at each pixel position of the image to be formed, and determines the calculation result. Send to section 135. Then, the determining unit 135 determines the pixel value of the non-confocal image having a larger feature amount as the pixel value of each pixel position in the image to be formed, based on the calculation result of the calculating unit 133. By performing the process for all pixel positions, it is possible to form an image Df with high visibility of the blood vessel walls on the left and right sides as shown in FIG.

For example, when the absolute value of the spatial differential value of the brightness value is used as the feature amount, the non-confocal images Dnr and Dnl shown in FIG. 10B and FIG. Then, when the brightness profile of the differential image is generated, the brightness profile shown in FIG. Then, when non-rigid body alignment and brightness correction processing are performed in step S1221, the brightness profile becomes as shown in FIG. 11G, and the signal value having the larger absolute value of the differential is selected as the pixel value and determined. To do. By performing the process for all pixel positions of the image to be formed, the image forming apparatus 10 can form an image with high visibility of the blood vessel walls on the left and right sides. 11F and 11G, the solid line shows the brightness profile of the differential image of the R channel non-confocal image, and the dotted line shows the brightness profile of the differential image of the L channel non-confocal image.

Further, when the contrast of the brightness value is used as the feature amount, when the contrast is calculated at the dotted line portions of the non-confocal images Dnr and Dnl in FIG. 10B and FIG. 10C, the contrast shown in FIG. Become a profile. Then, when non-rigid body alignment and luminance correction processing is performed in step S1221, the contrast profile becomes as shown in FIG. 11I, and the signal value having the larger contrast value is selected and determined as the pixel value. By performing the process for all pixel positions of the image to be formed, the image forming apparatus 10 can form an image in which the blood vessel walls on the left and right sides are highly visible. 11(h) and 11(i), the solid line shows the contrast profile of the R channel non-confocal image, and the dotted line shows the contrast profile of the L channel non-confocal image.

In such a configuration, the calculation unit 133 performs the maximum value calculation using the characteristic amounts related to the corrected values of the plurality of types of non-confocal signals. Instead of the signal value of the non-confocal signal, the feature values of the images are used to perform the calculation between the non-confocal superimposed images to determine the pixel value at each pixel position in the image to be formed. An image with high visibility of the wall can be formed.

[Second Embodiment]
The image forming apparatus 10 according to the first embodiment has dealt with the problem that only the blood vessel wall on one side can be clearly drawn in the image forming apparatus using the conventional Offset Pinhole method or the Split Detector method. On the other hand, in the conventional Offset Pinhole method, in addition to the above problem, when there are blood vessels having different traveling directions due to meandering, bifurcation, crossover, etc. in the imaging target region, the blood vessel wall is clearly defined by the traveling direction of the blood vessel branch. There was a problem that it could not be drawn. Similarly, the conventional Split Detecotr method has a problem that it is difficult to clearly depict the blood vessel wall when the dividing direction of the opening such as a pinhole and the traveling direction of the blood vessel are greatly different from each other.

In this regard, Non-Patent Document 1 does not disclose a non-confocal imaging/image generation technique suitable for a meandering, branched, or crossed blood vessel. Non-Patent Document 2 also discloses that the opening of the pinhole of the SLO device is moved at eight equal intervals in the circumferential direction to photograph photoreceptor cells. However, meandering, bifurcating, or intersecting blood vessels are disclosed. No technique is disclosed for adaptively selecting eight types of images in order to generate an image suitable for.

In view of the above circumstances, the second embodiment of the present invention provides an image forming apparatus and an image forming method for clearly depicting a blood vessel wall region regardless of the traveling direction of the blood vessel.

The image forming apparatus according to the present embodiment performs a calculation process between non-confocal signals of different types of non-confocal images captured by an SLO device that acquires three or more types of non-confocal images, and calculates the calculation result. Select the combination of signal types that maximizes the value. Further, the image forming apparatus executes maximum value calculation between the selected combinations of signal types. As a result, the image forming apparatus according to the present embodiment can form an image with high wall visibility regardless of the blood vessel traveling direction.

The image forming apparatus 60 according to the present exemplary embodiment will be described below with reference to FIGS. The configuration of the device connected to the image forming apparatus 60 according to the present embodiment is the same as the configuration of the device connected to the image forming apparatus 10 according to the first embodiment, and thus the description thereof will be omitted.

FIG. 13 shows a functional configuration example of the image forming apparatus 60 according to the present exemplary embodiment. FIG. 13 is a block diagram showing a schematic functional configuration example of the image forming apparatus 60. The image forming apparatus 60 according to the present embodiment includes a selection unit 638 in the image processing unit 630 instead of the correction unit 132, and the calculation unit 633 causes the selection unit 638 to select different types of non-confocal superimposed images. The difference from the first embodiment is that arithmetic processing is performed between the non-confocal signals. The other components such as the signal acquisition unit 110 are the same as those of the image forming apparatus 60 according to the first embodiment, and therefore, the same reference numerals are given and the description thereof will be omitted. Hereinafter, the image forming apparatus 60 according to the present exemplary embodiment will be described focusing on differences from the image forming apparatus 10 according to the first exemplary embodiment.

The image forming apparatus 60 according to the present embodiment includes an arithmetic unit 633 and a selection unit 638 in the image processing unit 630.

The calculation unit 633 performs a calculation process such as a difference enhancement process between the non-confocal signals of different types of non-confocal images, and sends the calculated value as the calculation result to the selection unit 638. In addition, the calculation unit 633 calculates at each pixel position of the image to be formed for the combination of different types of non-confocal signals selected by the selection unit 638, and sends the calculation result to the determination unit 135. Note that the determination unit 135, based on the calculation result for the combination of different types of non-confocal signals selected by the selection unit 638, similarly to the determination unit 135 in the first embodiment, each pixel position of the image to be formed. To determine the pixel value at.

The selection unit 638 selects a combination of types of non-confocal signals, which is used in the calculation for determining the pixel value at each pixel position of the image to be formed, based on the calculation value received from the calculation unit 633. Specifically, the selection unit 638 compares the calculation values received from the calculation unit 633, and the calculation value is the maximum among the combinations of the types (signal types) of the non-confocal signals processed by the calculation unit 633. Select the combination of signal types.

Next, an image processing flow in the image forming apparatus 60 according to this embodiment will be described. The image processing flow is the same as that of the first embodiment, and is the same as that of the first embodiment except for the image forming processing in step S930 shown in FIG. 9, so description thereof will be omitted.

<Step S930>
In step S930 according to the present embodiment, the image processing unit 630 forms an image with high visibility of the blood vessel wall regardless of the traveling direction of the blood vessel by the following procedure.

i) Overlaid image generation in moving image units As in the case of the first embodiment, the image processing unit 630 projects the overlaid images by projecting the moving images subjected to the inter-frame alignment in step S920 on an average value. To generate.

ii) Calculation of Calculation Value Between Non-confocal Signals The calculation section 633 determines that the pixels in the small area including each pixel (pixel of interest) of the image to be formed are different from each other in the non-confocal superimposed images of different types. Arithmetic processing is performed between the corresponding non-confocal signals. Here, the "small area including the pixel of interest" refers to an area near the position of the pixel (pixel of interest) whose pixel value is to be determined, and an area of any size or shape is set as the "small area including the pixel of interest". be able to. For example, the “small area including the pixel of interest” may be a circular area having a radius r or a rectangular area. Alternatively, the “small area including the pixel of interest” may be the “pixel of interest and its four neighboring pixels” or the “pixel of interest and its eight neighboring pixels”. Alternatively, the “small area including the target pixel” may be only the target pixel (the area is 1).

The SLO device 20 and the image forming device 60 according to the present embodiment use four types of non-confocal signals NC1, NC2, NC3 by using a diaphragm having four transmission regions as shown in FIG. 6B. , NC4. The diaphragm shown in FIG. 6B divides the reflection area 210-1-1, four transmission areas 210-1-4, 210-1-5, 210-1-6, 210-1-7 and the transmission area. The light-shielding area (thick line portion) is provided. Note that the non-confocal signals NC1, NC2, NC3, NC4 are respectively transmission regions 210-1-4, 210-1-5, 210-1-6, 210-1-7 of the diaphragm shown in FIG. 6B. Is a non-confocal signal acquired based on the light that has passed through.

As a calculation process for selecting a combination of non-confocal signals, the calculation unit 633 performs difference enhancement processing between different non-confocal signals (NC1 to NC4) in each pixel in the small region including each target pixel. (Split Detection processing) is performed. The calculation unit 633 calculates the absolute value of the difference (Split Detection) value obtained by the difference emphasis process as a calculated value for each combination of the signal types used in the difference emphasis process, and calculates the sum of the calculated values in the small region. To do. However, it is not limited to the difference emphasis process, and any known arithmetic process may be applied.

iii) Selection of signal type based on comparison of calculated values calculated between the respective non-confocal signals The selection unit 638 compares the values calculated in ii) (sum of calculated values) to determine different types of non-confocal signals. Of these, the combination of signal types that maximizes the calculated value is selected for each pixel of interest.

iv) Determination of Pixel Value by Calculation Between Selected Signal Types The calculation unit 633 calculates between the signal types selected by the selection unit 638. Then, the determining unit 135 determines the pixel value of each target pixel in the image formed by the image forming apparatus 60 based on the calculation result of the calculating unit 633. In the present embodiment, the calculation unit 633 obtains a value obtained by performing the maximum value calculation between the signal types selected in iii), and the determination unit 135 determines the pixel value of the pixel of interest in the image forming the calculation result of the maximum value calculation. To decide.

In the present embodiment, the sum of the absolute values of the differences between the signal types in the small area calculated in ii) is compared in iii) and the combination of signal types that maximizes the sum is selected. However, the operation between the signal types compared for selecting the combination of the signal types is not limited to this. For example, in ii), the calculation unit calculates the absolute value of the difference value without calculating the total sum of the absolute values of the differences in the small area. Then, in iii), the selection unit tentatively selects a signal type at each pixel position in the small region based on the magnitude of the absolute value of the difference value calculated by the calculation unit, and selects the most tentatively selected signal type. A combination is selected for each pixel of interest. After that, in iv), the calculation unit performs a calculation on the selected combination of signal types, and the determination unit determines the pixel value of each target pixel based on the calculation result. A pixel value of the target pixel may be obtained by selecting a combination of signal types by using such arithmetic processing between signal types.

The image forming process performed in step S930 of the image processing flow in the image forming apparatus 60 will be described below with reference to FIG. FIG. 14A is a flowchart showing the image forming process executed by the image forming apparatus 60. However, step S1410 is the same as step S1210 according to the first embodiment, so description thereof will be omitted.

<Step S1420>
In step S1420, operation unit 633 calculates the operation value between each non-confocal signal in each pixel in the small area including each target pixel. The calculation unit 633 according to the present embodiment performs a non-confocal signal smoothed on the same scale as a difference calculation between different non-confocal signals (NC1 to NC4) in each pixel in the small area including each target pixel. Difference enhancement processing is performed using (NC1 to NC4). The calculation unit 633 calculates, for each combination of signal types used for the difference enhancement processing, the sum of absolute values of the difference values obtained by the difference enhancement processing between different non-confocal signals in the small region.

When parallax occurs between the non-confocal images (R, L) as in the first embodiment, difference enhancement processing ((LR)/(R+L)) is performed between both images. Then, an image similar to the case where spatial differentiation processing (brightness value difference processing between different coordinates) is performed in the same image is obtained. Here, as an example of the image subjected to the difference enhancement processing by the calculation unit 633, FIG. 10D illustrates a smoothed Split Detector image.

The calculation unit 633 according to the present embodiment performs a difference emphasis calculation process between the following four types of different non-confocal signals, and calculates the sum of absolute values in the small area including the pixel of interest.

a) Lateral difference (transparent area dividing line: vertical direction)
((NC1+NC2)/2-(NC3+NC4)/2)
/((NC1+NC2)/2 + (NC3+NC4)/2)

b) Diagonal difference (upper left and lower right) (division line of transparent area: direction from upper right to lower left)
(NC1-NC3)/(NC1+NC3)

c) Vertical difference (transparent area dividing line: horizontal direction)
((NC1+NC4)/2-(NC2+NC3)/2)
/((NC1+NC4)/2 + (NC2+NC3)/2)

d) Difference between diagonal directions (upper right and lower left) (transparent area dividing line: direction from lower right to upper left)
(NC4-NC2)/(NC4+NC2)

Note that the non-confocal signals used for the arithmetic processing between the non-confocal signals may be any combination. In this regard, the transmissive portions of the diaphragm corresponding to the non-confocal signal used in the arithmetic processing need not be at the diametrically opposite positions and have the same transmissive area. For example, based on the result of the arithmetic processing between the non-confocal signals based on the light transmitted through the transmissive regions having different sums of areas, such as NC1 and (NC3+NC4)/2, the image formation is not performed. The combination of signal types used may be selected. Here, the combination of signal types used for image formation refers to a combination of non-confocal signals calculated by the calculation unit 633 in order to determine the pixel value by the determination unit 135.

The calculation unit 633 sends the calculated sum of absolute values of the difference values between the non-confocal signals to the selection unit 638.

<Step S1430>
In step S1430, the selection unit 638 compares the values calculated in step S1420 and selects the combination of signal types that maximizes the value. In the present embodiment, the sum of the absolute values of the difference values between the non-confocal signals described above is compared with each other, and the combination of the maximum signal types is selected. For example, the sum of the absolute values of the difference values between the non-confocal signals in the small area including the pixel of interest obtained by performing the operations a) to d) and the operation b) is the maximum among the four operation values. If so, the combination of the non-confocal signal NC1 and the non-confocal signal NC3 is selected.

Here, in the present embodiment, the sum of the calculated values is used as the value used for selecting the combination of signal types used for image formation, but the present invention is not limited to this. For example, as a value used for selecting a combination of signal types used for image formation, a maximum value or an average value of calculated values, or a sum of calculated values equal to or larger than a predetermined threshold may be used.

The selection unit 638 sends the combination of the types (signal types) of the selected non-confocal signals to the calculation unit 633.

<Step S1440>
In step S1440, the calculation unit 633 calculates between the combinations of signal types selected in step S1430 and sends the calculation result to the determination unit 135. The determination unit 135 determines the pixel value of the target pixel of the image formed by the image forming apparatus 10 based on the calculation result of the calculation unit 633.

In the present embodiment, the calculation unit 633 sends to the determination unit 135 the maximum-value-calculated value of the signal value of the non-confocal signal corresponding to the pixel of interest between the combinations of signal types selected in step S1430. The determining unit 135 determines the value calculated by the maximum value as the pixel value of the target pixel.

<Step S1450>
In step S1450, the image processing unit 630 advances the processing to step S940 shown in FIG. 9 if the calculation processing between the non-confocal signals is completed for all the pixels, and the pixels for which the calculation processing is not completed remain. If so, the process advances to step S1460.

<Step S1460>
In step S1460, the image processing unit 630 moves to an adjacent pixel position for which calculation between the non-confocal signals has not been completed, and then advances the process to step S1420.

As described above, the image forming apparatus 60 according to the present embodiment includes the signal acquisition unit 110 that acquires a plurality of types of non-confocal signals NC1 to NC4, each of which forms a non-confocal image of a subject. The image forming apparatus 60 also includes a selection unit 638 that selects a combination of non-confocal signals used when determining the pixel value of the image to be formed from a plurality of types of non-confocal signals NC1 to N4. Further, the image forming apparatus 60 includes a calculation unit 633 that performs maximum value calculation using a plurality of types of non-confocal signals for the combination of non-confocal signals selected by the selection unit 638. In addition, the image forming apparatus 60 includes a determination unit 135 that determines each pixel value of the image to be formed based on the calculation result of the maximum value calculation by the calculation unit 633. In addition, the calculation unit 633 performs difference emphasis processing between the plurality of types of non-confocal signals to calculate a calculation value, and the selection unit 638 based on the calculation value between the plurality of types of non-confocal signals. , Select a combination of non-confocal signals.

With such a configuration, the image forming apparatus 60 performs arithmetic processing (difference enhancement processing) between the non-confocal signals of three or more types of non-confocal images captured by the SLO apparatus 20. The image forming apparatus 60 selects a combination of signal types that maximizes the calculated value by the calculation process, and then performs maximum value calculation on the selected combination of signal types. As a result, the image forming apparatus 60 according to the present embodiment can form an image with high wall visibility regardless of the blood vessel traveling direction.

In the present embodiment, a case has been described in which four types of non-confocal signals are acquired using a diaphragm as shown in FIG. 6B, but any number of types of non-confocal signals can be used as long as there are three or more types. May be obtained. For example, eight kinds of non-confocal signals (NC1 to NC8) are acquired by using a diaphragm as shown in FIG. 6C or 6F, and the selecting unit 638 selects between the eight kinds of non-confocal signals. A combination of signal types used for image formation may be selected based on the calculated value.

Here, the non-confocal signals NC1 to NC8 are non-confocal signals acquired from the light transmitted through the transmission regions 210-1-9 to 210-1-16 of the diaphragm shown in FIG. 6C, respectively. In this case, the pixel value of the target pixel of the image to be formed can be determined by any one of the following methods i) to iii).

i) The calculation unit performs difference calculation between the non-confocal signals NC1 and NC5, NC2 and NC6, NC3 and NC7, and NC4 and NC8. The selection unit selects the combination having the largest sum of the absolute values of the differences as a combination of the signal types used for image formation, the calculation unit performs the maximum value calculation between the selected signal types, and the determination unit determines the calculation result. The pixel value of the pixel of interest is determined based on

ii) The calculation unit is (NC1+NC2)/2 and (NC5+NC6)/2, (NC2+NC3)/2 and (NC6+NC7)/2, (NC3+NC4)/2 and (NC7+NC8)/2, (NC4+NC5)/2 and (NC8+)NC. )/2 is calculated. The selection unit selects the combination having the largest sum of the absolute values of the differences as a combination of the signal types used for image formation, the calculation unit performs the maximum value calculation between the selected signal types, and the determination unit determines the calculation result. The pixel value of the pixel of interest is determined based on

iii) The calculation unit performs difference calculation between (NC1+NC2+NC3+NC4)/4 and (NC5+NC6+NC7+NC8)/4, and (NC2+NC3+NC4+NC5)/4 and (NC6+NC7+NC8+NC1)/4. In addition to this, the calculation unit performs difference calculation between (NC3+NC4+NC5+NC6)/4 and (NC7+NC8+NC1+NC2)/4, and (NC4+NC5+NC6+NC7)/4 and (NC8+NC1+NC2+NC3)/4. The selection unit selects the combination having the largest sum of the absolute values of the differences as a combination of the signal types used for image formation, the calculation unit performs the maximum value calculation between the selected signal types, and the determination unit determines the calculation result. The pixel value of the pixel of interest is determined based on

Note that the non-confocal signals used for the arithmetic processing between the non-confocal signals may be any combination. In this regard, the transmissive part of the diaphragm corresponding to the non-confocal signal used in the arithmetic processing is at the opposite positions, such as the transmissive region 210-1-9 and the transmissive region 210-1-13, and is in the transmissive region. The areas need not be the same. For example, when image formation is performed based on the result of calculation processing between signal types based on light transmitted through transmissive regions that are not in opposite positions such as (NC1+NC2)/2 and (NC4+NC5+NC6)/3, or have different sums of areas. The combination of signal types used may be selected. From this, it is understood that the calculation between the combinations of the eight types of non-confocal signals is not limited to the above i) to iii), and the calculation between any combination may be performed.

Further, in the present embodiment, the case has been described in which the signal values (luminance values) of different types of non-confocal signals themselves are used to form an image in which the visibility of the blood vessel wall is improved regardless of the traveling direction of the blood vessel. .. However, the values used for the selection of the combination of non-confocal signals and the calculation for determining the pixel value are not limited to the signal values of the non-confocal signal. For example, according to the procedure as shown in FIG. 14B, it is used for image formation based on a combination of signal types having the largest feature amount (contrast of brightness or absolute value of spatial differentiation) regarding different types of non-confocal signals. A combination of signal types may be selected. In addition, for the combination of the signal types, a pixel value may be determined based on a feature amount related to a non-confocal signal, and an image with improved visibility of the blood vessel wall may be formed.

FIG. 14B shows a flowchart of the image forming process when the feature amount related to the non-confocal signal is used for the maximum value calculation for selecting the combination of signal types and determining the pixel value. The flowchart shown in FIG. 14B is similar to the flowchart shown in FIG. 14A except for steps S1421 to S1441. Therefore, description will be given centering on steps S1421 to S1441 which are different points, and other parts will be described. Omit it.

<Step S1421>
In step S1421, the feature amount calculation unit 134 calculates the feature amount (contrast of brightness or absolute value of spatial differentiation) regarding the non-confocal signal of each non-confocal superimposed image, and the calculated feature amount is provided to the calculation unit 633. send. The feature amount may be stored in the storage unit 120.

The calculation unit 633 performs calculation processing using the feature amount regarding each non-confocal signal received from the feature amount calculation unit 134 in the small area including the target pixel, and calculates the calculated value. As the calculation process, any calculation process suitable for selection of a combination of non-confocal signals may be performed, and for example, feature amounts may be added together.

<Step S1431>
In step S1431, the selection unit 638 compares the values calculated in step S1421 and selects the combination of signal types having the maximum value. In the present embodiment, the sum totals of the absolute values of the above-described calculated values are compared with each other, and the combination of signal types having the maximum sum is selected.

Here, the sum of the calculated values is used as the value used for selecting the combination of the signal types used for image formation, but the present invention is not limited to this. For example, as a value used for selecting a combination of signal types used for image formation, a maximum value or an average value of calculated values, or a sum of calculated values equal to or larger than a predetermined threshold may be used.

The selection unit 638 sends the combination of the types (signal types) of the selected non-confocal signals to the calculation unit 633.

<Step S1441>
In step S1441, the calculation unit 633 calculates the maximum value between the feature amounts of the non-confocal signals between the combinations of signal types selected in step S1431 in the target pixel, and sends the calculation result to the determination unit 135. Then, the determining unit 135 determines the pixel value of the non-confocal image having a larger feature amount as the pixel value of the target pixel in the image to be formed, based on the calculation result of the calculating unit 633.

As described above, even when the calculation process between the non-confocal signals is performed by using the feature amount related to the non-confocal signal instead of the signal value of the non-confocal signal, and the combination of the signal types used for image formation is selected, An image with high visibility of the blood vessel wall can be formed regardless of the running direction of the blood vessel. Further, even when the pixel value at each pixel position in the image to be formed is determined using the feature amount related to the non-confocal signal, it is possible to form an image with high visibility of the blood vessel wall regardless of the traveling direction of the blood vessel.

[Other Embodiments]
In the first embodiment, the case where the image forming apparatus 10 including the correction unit 132 forms an image in which the visibility of the blood vessel walls on the left and right sides is improved is described. Further, in the second embodiment, the case where the image forming apparatus 60 including the selection unit 638 forms an image in which the visibility of the blood vessel wall is improved regardless of the traveling direction of the blood vessel has been described. In the above, these embodiments have been described as different embodiments, but these may be combined to form an image. That is, by providing both the correction unit and the selection unit in the image processing unit, it is possible to form an image with high visibility of the blood vessel walls on both sides regardless of the blood vessel traveling direction.

FIG. 15 shows a functional configuration example of the image forming apparatus 70 according to such a modification. FIG. 15 is a block diagram showing a schematic functional configuration example of the image forming apparatus 70. In FIG. 15, the same components as those in the image forming apparatuses 10 and 60 according to the first and second embodiments are designated by the same reference numerals. Further, in the following, the description of the configuration and the image processing flow similar to the configuration and the image processing flow according to the first embodiment and the second embodiment will be omitted, and the differences between them will be mainly described according to the present modification. The image forming apparatus 70 will be described.

The image forming apparatus 70 has a configuration in which the selecting unit 638 according to the second embodiment is added to the image forming apparatus 10 according to the first embodiment. Note that these functions are similar to the constituent elements in the first embodiment and the second embodiment, so description thereof will be omitted. Note that the calculation unit 733 of the image processing unit 730 selects the non-confocal signal or the non-confocal signal for which the selection of the combination of the signal types by the selection unit 638 and the non-confocal superimposed images by the correction unit 132 have been performed or the brightness adjustment has been performed. Perform calculations on confocal images.

The image forming process by the image forming apparatus 70 will be described below with reference to FIG. FIG. 16A is a flowchart showing the image forming process executed by the image forming apparatus 70. Note that steps S1610, S1660, and S1670 in the image forming process are the same as steps S1210, S1240, and S1250 of the image forming process executed in the image forming apparatus 10 according to the first embodiment, and thus description thereof will be omitted.

<Step S1620>
In step S1620, the calculation unit 733 performs difference enhancement processing (calculation processing) between different non-confocal signals in each pixel in the small area including the target pixel, as in the calculation unit 633 in the second embodiment. .. The calculation unit 733 calculates the absolute value of the difference value calculated by the difference emphasis process as a calculated value for each combination of the signal types used for the difference emphasis process (second calculation), and sums the calculated values in the small region. Is calculated and sent to the selection unit 638.

<Step S1630>
In step S1630, the selection unit 638 compares the sums of the absolute values of the difference values calculated in step S1620, selects the combination of signal types that maximizes the sum, and sends the combination to the correction unit 132 and the calculation unit 733. ..

<Step S1640>
In step S1640, the heterogeneous signal alignment unit 1321 aligns the non-confocal superimposed images corresponding to the combination of the signal types selected by the selection unit 638, as in step S1220 according to the first embodiment. By doing so, the parallax between both images is reduced. The heterogeneous signal alignment unit 1321 stores the non-confocal superimposed image that has undergone the alignment in the storage unit 120.

In addition, the brightness correction unit 1322 determines that the non-confocal overlapped images that have been registered are matched when the brightness distributions of the non-confocal overlapped images corresponding to the combination of the signal types are significantly different from each other for some reason. Adjust so that the brightness distribution characteristics are uniform. The method of adjusting is the same as step S1220 according to the first embodiment. The brightness correction unit 1322 stores the non-confocal superimposed image for which brightness correction has been performed in the storage unit 120.

<Step S1650>
In step S1650, the calculation unit 733 performs the maximum value calculation (first calculation) between the combinations of the non-confocal signals of the corrected non-confocal superimposed images for the target pixel, and the calculation result is determined by the determination unit 135. Send to. The determination unit 135 determines the pixel value of the target pixel in the image to be formed based on the calculation result.

As described above, in the image forming apparatus 70 according to the present modification, in addition to the configuration of the image forming apparatus 10 according to the first embodiment, a plurality of types of non-confocal signal combinations used when determining pixel values are used. A selection unit 638 that selects from confocal signals is further provided. The calculation unit 733 performs difference emphasis processing between the plurality of types of non-confocal signals to calculate the absolute value of the difference value as a calculation value, and the selection unit 638 determines the calculated difference between the plurality of types of non-confocal signals. A combination of non-confocal signals is selected based on the calculated value of. Then, the correction unit 132 corrects the selected combination of non-confocal signals so that the positions of the non-confocal images formed by the non-confocal signals are matched. After that, the calculation unit 733 performs maximum value calculation on the combination of the non-confocal signals selected by the selection unit 638 using the corrected plurality of types of non-confocal signals, and the determination unit based on the calculation result. 135 determines the pixel value of the pixel of interest.

As described above, in the image forming apparatus 70 according to the present modification, the selection unit 638 performs difference processing between different signal types and selects a combination of signal types that maximizes the sum of absolute values of the differences. Then, the correction unit 132 corrects the alignment and the brightness distribution between the non-confocal superimposed images corresponding to the selected combination of signal types. Then, the calculation unit 733 determines the pixel value by calculating the maximum value between the corrected signal pairs.

With this configuration, the image forming apparatus 70 can form an image in which the visibility of the left and right blood vessel walls is improved regardless of the traveling direction of the blood vessel.

In the image forming apparatus 70 according to the modified example, after the selection unit 638 selects a combination of signal types, the correction unit 132 performs alignment of non-confocal superimposed images corresponding to the combination of signal types. On the other hand, the correction unit 132 may correct the parallax or luminance distribution between different non-confocal images, and then the selection unit 638 may select the combination of signal types.

In this case, the calculation unit 733 performs difference enhancement processing between the plurality of types of corrected non-confocal signals to calculate the absolute value of the difference value as a calculation value, and the selection unit 638 causes the corrected plurality of corrected non-confocal signals. A combination of non-confocal signals is selected based on the calculated value between the non-confocal signals of

With this configuration as well, the image forming apparatus 70 can form an image in which the visibility of the left and right blood vessel walls is improved regardless of the traveling direction of the blood vessel.

Further, in the present modification, the signal value (luminance value) itself of the non-confocal signal is used for the calculation process for selecting the signal type and the calculation between the non-confocal signals for determining the pixel value. The case of forming an image in which the visibility of the blood vessel wall is improved regardless of the traveling direction of the blood vessel has been described. However, what is used for the calculation process for selecting the signal type and the calculation between the non-confocal signals for determining the pixel value is not limited to the signal value of the non-confocal signal. For example, select a combination of signal types that maximizes the calculated value of the feature amount (contrast of brightness or absolute value of spatial differentiation) relating to different types of non-confocal signals, and select the non-confocal signal with the maximum feature amount. The pixel value may be determined based on the above.

In this case, the calculation unit 733 performs difference emphasis processing, addition of the characteristic amounts, and the like using the characteristic amounts regarding the values of the plurality of types of non-confocal signals, and calculates the calculated value as the calculation result. Further, when the correction unit 132 corrects the parallax and the luminance distribution between different non-confocal images, and the selection unit 638 selects a combination of signal types, the calculation unit 733 corrects a plurality of types of non-corrected images. The same calculation is performed using the feature amount related to the value of the confocal signal, and the calculated value is calculated. Then, with respect to the selected combination of non-confocal signals, the calculation unit 733 performs maximum value calculation on the feature amount regarding the non-confocal signals, and the determination unit 135 determines the pixel value based on the calculation result. Even in these cases, the image forming apparatus 70 can form an image with improved visibility of the left and right blood vessel walls regardless of the running direction of the blood vessel.

Further, in the above-described second embodiment and modified example, the case where the selection unit 638 selects a combination of signal types for each pixel (pixel of interest) whose pixel value should be determined has been described, but the present invention is not limited to this. .. For example, the present invention also includes a case where a combination of signal types is selected for each small area in an image, or a signal type is uniformly selected for an image using the brightness value of the entire image.

In addition, in the above-described second embodiment and modification, the selection unit 638 performs a selection process of a combination of signal types for pixel values (luminance values) of the entire image to improve the visibility of the blood vessel wall in the entire image. Formed image. However, the selection processing of the combination of signal types and the image formation area in which the visibility of the blood vessel wall is improved are not limited to this. For example, in the procedure as shown in FIG. 16B, the image processing unit performs the feature extraction processing to specify one or more regions of interest, and the calculated value between the non-confocal signals calculated in the region of interest is the maximum. The present invention also includes the case of selecting a combination of the following signal types and forming an image in the region of interest. FIG. 16B is a flowchart showing the image forming process in such a case. Note that step S1611 is the same as step S1610 of the image forming process in the image forming apparatus 70 according to the modification shown in FIG.

The image forming process shown in FIG. 16B is the same as the image forming process shown in FIG. 16A except for steps S1612, S1621, and S1661, and therefore the difference will be mainly described.

<Step S1612>
In step S1612 in the modified example, the image processing unit extracts a blood vessel feature by a well-known feature extraction process, and determines a region of interest (ROI) based on the extracted feature. The image processing unit sends the determined ROI to the calculation unit.

<Step S1621>
In step S1621, the calculation unit performs difference enhancement processing between different non-confocal signals for a small region including the pixel of interest in the determined ROI, as in step S1420 according to the second embodiment. The calculation unit, for each combination of signal types used in the difference emphasis process, obtains the absolute value of the difference value obtained by the difference emphasis process as a calculation value, calculates the sum of the calculation values in the small area, and outputs it to the selection unit. send.

Steps S1631 to S1651 are the same as steps S1630 to S1650 of the image forming process in the image forming apparatus 70 according to the modification shown in FIG.

<Step S1661>
In step S1661, the image processing unit advances the processing to step S940 shown in FIG. 9 if the calculation processing between the non-confocal signals is completed for all the pixels in the ROI, and the pixels for which the calculation processing is not completed yet. If is left, the process proceeds to step S1671. Note that step S1671 is the same as step S1670 of the image forming process in the image forming apparatus 70 according to the modification shown in FIG.

In this way, a combination of signal types may be selected not for the entire image but for pixels in a specific region of interest to form an image with improved visibility of the blood vessel wall. The same applies to a configuration in which the combination of signal types is not selected as in the image forming apparatus 10 according to the first embodiment.

In the modification, the image processing unit performs the feature extraction process to specify one or more regions of interest. However, for example, the operator manually specifies the regions of interest using the keyboard 306, the mouse 307, or the like. Good.

Further, in the above-described embodiment, a case has been described in which an image is formed using the result of arithmetic processing between the signal types selected by the selection unit 638 for the previously designated imaging target area. However, the determination of the shooting target area is not limited to this. For example, the imaging target area may be determined based on the blood vessel traveling direction information after acquiring the blood vessel traveling direction information based on the signal type selected by the selection unit 638.

In the above-described embodiment, the case where the signal acquisition unit 110 includes both the confocal data acquisition unit 111 and the non-confocal data acquisition unit 112 has been described. However, as long as two or more types of non-confocal data can be acquired, The signal acquisition unit 110 may not include the confocal data acquisition unit 111.

Furthermore, although the image forming apparatus according to each embodiment has been described above, the embodiment of the present invention is not limited to the image forming apparatus. The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. A storage medium storing such a program is also included in the scope of the present invention.

In the above embodiment, the case where the confocal image and the non-confocal image are moving images has been described. However, the confocal image and the non-confocal image may be still images. In this case, these images are used for image formation without aligning each frame. Therefore, the calculation unit calculates not the superimposed images but the respective still images, and calculates the pixel of interest of the image to be formed, and the determination unit determines the pixel value based on the calculation result. Even the image forming apparatus configured in this case can achieve the same effects as those of the image forming apparatuses according to the first and second embodiments.

Moreover, in the said embodiment, the case where the to-be-measured object was an eye was described. However, the image forming apparatus and the image forming method according to the present invention can also be applied to an inspection apparatus that inspects skin, organs, and the like other than the eye as an object to be measured. Therefore, the image forming apparatus according to the present invention can be used for medical equipment such as an endoscope other than the ophthalmologic apparatus.

Although the present invention has been described with reference to the exemplary embodiments, the present invention is not limited to the above exemplary embodiments. The present invention includes inventions modified within a range not departing from the spirit of the present invention and inventions equivalent to the present invention. Further, the above-described respective embodiments can be appropriately combined without departing from the spirit of the present invention.

10: image forming apparatus, 110: signal acquisition unit, 132: correction unit, 135: determination unit

Claims (13)

An image forming apparatus for forming an image,
Each of which forms a non-confocal image of the eye, a signal acquisition unit that acquires a first non-confocal signal and the second non-confocal signal returning light was divided into regions from the subject's eye,
The first non-confocal image generated from the first non-confocal signal and the second non-confocal image generated from the second non-confocal signal are aligned in position between pixels, A correction unit that corrects the positional relationship between the first non-confocal signal and the second non-confocal signal,
An arithmetic unit that performs a maximum value operation using the corrected first non-confocal signal and the second non-confocal signal;
A determination unit that determines each pixel value of an image to be formed based on a calculation result by the calculation unit, and the first non-confocal signal and the second non-confocal signal whose positional relationship is corrected. Is an image forming apparatus, which is a signal acquired at the same time. The correction unit corrects one of the first non-confocal signal and the second non-confocal signal so that the luminance distribution characteristics between the non-confocal images are made uniform. Image forming device. The first non-confocal signal and the second non-confocal signal used for the maximum value calculation are selected from four non-confocal signals obtained by dividing the return light into four regions. Further provided with a selection unit,
The image forming apparatus according to claim 1, wherein the calculation unit performs the maximum value calculation using the first non-confocal signal and the second non-confocal signal selected by the selection unit. .. The correction unit corrects the positional relationship of the four non-confocal signals so that the positions of the pixels in the four non-confocal signals are matched.
The calculation unit performs a difference emphasis processing calculation between the corrected four non-confocal signals to calculate a calculation value,
The image forming apparatus according to claim 3, wherein the selection unit selects the first non-confocal signal and the second non-confocal signal based on the calculated value. The calculation unit performs a difference emphasis processing calculation between the four non-confocal signals to calculate a calculation value,
The selecting unit selects the first non-confocal signal and the second non-confocal signal based on the calculated calculation value;
The image forming apparatus according to claim 3, wherein the correction unit corrects a positional relationship between the first non-confocal signal and the second non-confocal signal. The calculating unit calculates the calculated value by performing the difference emphasis processing calculation with respect to the non-confocal signal corresponding to a predetermined region including a pixel whose pixel value is to be determined,
The said selection part selects the said 1st non-confocal signal and the said 2nd non-confocal signal based on the average value of the said calculation value in the said predetermined area|region. Image forming apparatus. The calculating unit calculates the calculated value by performing the difference emphasis processing calculation with respect to the non-confocal signal corresponding to a predetermined region including a pixel whose pixel value is to be determined,
The selection unit selects the first non-confocal signal and the second non-confocal signal based on the sum of the calculated values that exceed a predetermined threshold value among the calculated values in the predetermined region. The image forming apparatus according to claim 4, wherein the image forming apparatus comprises: The calculation unit performs the maximum value calculation by using one of a value of the corrected first non-confocal signal and the second non-confocal signal and a feature amount related to the value. The image forming apparatus according to any one of 1 to 7. The image forming apparatus according to claim 4, wherein the calculation unit performs the difference emphasis processing calculation using one of the corrected values of the four non-confocal signals and the feature amount related to the values. The image forming apparatus according to claim 5, wherein the calculation unit performs the difference emphasis processing calculation by using one of the value of the four non-confocal signals and the feature amount related to the value. An image forming method for forming an image,
And that each forms a non-confocal image of the eye to obtain a first non-confocal signal and the second non-confocal signal returning light was divided into regions from the subject's eye,
The first non-confocal image generated from the first non-confocal signal and the second non-confocal image generated from the second non-confocal signal are aligned in position between pixels, Correcting the positional relationship between the first non-confocal signal and the second non-confocal signal;
Performing a maximum value calculation using the corrected first non-confocal signal and the second non-confocal signal;
Determining each pixel value of the image to be formed based on the calculation result of the maximum value calculation;
Including,
The image forming method, wherein the first non-confocal signal and the second non-confocal signal whose positional relationship is corrected are signals that are simultaneously acquired. The first non-confocal signal and the second non-confocal signal used for the maximum value calculation are selected from four non-confocal signals obtained by dividing the return light into four regions. Further including
The image according to claim 11, wherein performing the maximum value calculation includes performing the maximum value calculation using the selected first non-confocal signal and the second non-confocal signal. Forming method. A program that causes a computer to realize each unit of the image forming apparatus according to claim 1.