JP2022038751A - Image processing apparatus, image processing method and program - Google Patents

Abstract

To highly accurately detect a disease.SOLUTION: An image processing apparatus comprises: setting means which sets a two-dimensional coordinate system based on a first coordinate axis with a yellow spot and nipple as a reference in first ocular fundus image information and second ocular fundus image information of right and left eyes of a subject; specification means which specifies a first point or region in the first ocular fundus image information, a second point or region symmetrical to the first coordinate axis and a third point or region corresponding to a relative relation between the yellow spot and nipple and the first point or region in the second ocular fundus image information; and determination means which determines a characteristic value about the first point or region on the basis of a value about the first point or region in the first ocular fundus image information, a value about a second point or region in the first ocular fundus image information and a value about a third point or region in the second ocular fundus image information.SELECTED DRAWING: Figure 5

Description

The disclosure of this specification relates to an image processing apparatus, an image processing method, and a program.

In order for doctors to diagnose ophthalmic diseases, ophthalmologic devices such as fundus cameras that acquire ophthalmologic photographs are widely used. Further, it is known that a tomographic image to be measured can be obtained non-invasively by an apparatus (hereinafter referred to as an OCT apparatus) using an optical coherence tomography (hereinafter referred to as OCT). Using an OCT device, the degree of discrepancy between the OCT image of the subject and the OCT image of a healthy person or a normal eye database is evaluated, and a diagnosis is made to find the ophthalmic disease of the subject.

Here, it is known that the thickness (layer thickness) of the retina decreases as glaucoma progresses, and glaucoma can be detected by evaluating the layer thickness with an OCT image. Patent Document 1 discloses a technique for calculating the displacement of the layer thickness value at the corresponding position with respect to the standard layer thickness obtained based on the statistical values of a large number of measured values of the layer thickness in a standard eye (normal eye, etc.). Has been done. Further, Patent Document 1 discloses a technique for comparing the displacements at positions symmetrical with respect to the horizontal axis passing through the fovea centralis.

Japanese Unexamined Patent Publication No. 2009-89792

If the method of detecting the disease in comparison with the normal eye database deviates from the normal value due to individual differences, it may be erroneously determined to be the disease even in normal health. For example, when the axial length is long due to myopia or the like, the retina thickness generally tends to be thin, so that even a healthy eye may be erroneously determined as glaucoma. Further, the method using the normal eye database has a problem that it costs a lot to generate or obtain the normal eye database. Further, the analysis method for evaluating the vertical symmetry of the layer thickness has a problem that the progression of glaucoma is overlooked when the layer thickness becomes as small as the upper and lower layers.

Therefore, one of the purposes of the disclosure of the present specification is to detect a disease with high accuracy.

The image processing apparatus according to one embodiment of the disclosure of the present specification is
A setting means for setting a two-dimensional coordinate system based on the first coordinate axis based on the macula and the nipple in the first fundus image information and the second fundus image information of the subject's left and right eyes.
The first point or region in the first fundus image information, the second point or region symmetrical with respect to the first coordinate axis, and the first point or region in the second fundus image information and the macula. And the specific means of identifying the third point or region to which the relative relationship with the nipple corresponds.
The value regarding the first point or region in the first fundus image information, the value regarding the second point or region in the first fundus image information, and the third in the second fundus image information. It has a determination means for determining a characteristic value with respect to the first point or region based on the value with respect to the point or region.

According to at least one embodiment of the disclosure herein, the disease can be detected with high accuracy.

An example of the overall configuration of the image processing apparatus according to the first embodiment is schematically shown. An example of a flowchart of the operation of image processing according to the first embodiment is shown. An example of the flowchart of the symmetry map generation process according to the first embodiment is shown. An example of the process of the geometric correction processing according to the first embodiment is shown. An example of the process of generating the symmetry map according to the first embodiment is shown. An example of the effect of the symmetry map according to the first embodiment is shown. An example of the flowchart of the symmetry map generation process according to the second embodiment is shown. The schematic diagram of the symmetry map generation process which concerns on 2nd Embodiment is shown. An example of the layer thickness profile according to the modification is shown.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. In the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar. In each drawing, some of the components, members, and processes that are not important for explanation may be omitted.

Further, in the present specification, images of the fundus acquired by each imaging technique (modality) such as OCT, fundus camera, and scanning laser ophthalmoscope (SLO) are collectively referred to as “fundus image”. Further, in the present specification, the depth direction of the subject is the Z direction, the direction perpendicular to the Z direction is the X direction, and the direction perpendicular to the Z direction and the X direction is the Y direction.

This embodiment will be described with reference to FIGS. 1 to 9. As illustrated in FIG. 1, the image processing apparatus 100 according to the present embodiment includes an input unit 101, a control unit 102, a display unit 103, an operation unit 104, and a storage unit 105.

The input unit 101 is an interface using, for example, a USB (Universal Serial Bus) cable, and inputs data from an ophthalmic device such as an OCT device and outputs the data to the control unit 102. Further, the input unit 101 may input various data including images from an external device (not shown) such as a server connected to the control unit 102. Here, the control unit 102 and an external device (not shown) may be connected via an arbitrary network such as the Internet. Further, the input unit 101 may input various data stored in the storage unit 105.

The control unit 102 executes a program stored in the storage unit 105 for the input data, and outputs the result to the display unit 103. Here, the control unit 102 may be a built-in (internal) computer of the OCT device 100, or may be a separate (external) computer to which the OCT device 100 is communicably connected. Further, the control unit 102 may be, for example, a personal computer, or a desktop PC, a notebook PC, or a tablet PC (portable information terminal) may be used. At this time, the communication connection between the control unit 102 and the ophthalmic device may be a connection by wire communication or a connection by wireless communication. The processor may be a CPU (Central Processing Unit). Further, the processor may be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), an FPGA (Field-Programmable Gate Array), or the like.

The display unit 103 is a display or the like, and displays information to the examiner based on the data input from the control unit 102. For example, the display unit 103 can display patient information regarding the eye to be inspected, various images, estimation results regarding ophthalmic diseases, and the like.

The operation unit 104 is a mouse, a keyboard, a touch panel, or the like, and can input the process instructed by the operation of the examiner to the control unit 102.

The storage unit 105 stores a program for realizing various application software including an operating system (OS), a device driver of a peripheral device, and a program for performing processing described later. In addition, data necessary for various operations is stored.

Hereinafter, the first embodiment and the second embodiment will be described. The first embodiment performs a process for ophthalmic diagnosis based on an image acquired in a single time. The second embodiment performs a process for ophthalmologic diagnosis based on images acquired at a plurality of different times.

[First Embodiment]
The image processing according to the first embodiment will be described step by step with reference to the flowchart of FIG. In particular, the case of estimating glaucoma using the OCT image of the fundus will be described as an example.

[Step S1: Image input]
The input unit 101 inputs tomographic image (OCT image) data of the fundus acquired by the OCT device (step S1). This tomographic image data includes those taken at a plurality of B scan positions, whereby the input unit 101 can acquire three-dimensional data regarding the fundus of the eye to be inspected. Here, the A scan means to acquire tomographic information from one point of the eye to be inspected, and by performing the A scan a plurality of times in an arbitrary crossing direction (main scanning direction), the crossing direction and the depth of the eye to be inspected E are obtained. Acquiring information on a two-dimensional fault in the vertical direction is called B scan. In this example, tomographic images of both the right eye and the left eye are input to the same patient. If there is a difference in axial length between the right eye and the left eye, processing may be performed to correct this.

The input unit 101 is connected to an ophthalmic device by a cable, and an image is input through the cable. Further, an image taken at another place may be input to the input unit 101 through a communication network or the like. The input unit 101 outputs the input image to the control unit 102.

Further, the input unit 101 can also input images and data acquired by other photographing techniques, such as a fundus photograph taken by a fundus camera and a frontal view of the fundus image (SLO image) taken by using an SLO optical system. ..

The input unit 101 may also input patient data such as a patient identification number for the eye to be inspected, an axial length, age, visual acuity, race, medical history, and at least one of the non-severe myopia. can. The input unit 101 may also input these images and data from an external device (not shown). The control unit 102 may extract patient data from the image of the medical record input by the input unit 101 by using a text mining technique or the like.

[Step S2: Image processing]
The control unit 102 processes the image input from the input unit 100 and generates an image to be output to the display unit 103 (step S2). An example of image processing according to this embodiment will be described with reference to the flowchart of FIG.

[Step S21: Thickness map generation]
The control unit 102 generates a thickness map using the tomographic image data (step S21). The thickness map is the thickness (layer thickness) of the anatomically determined observation target layer in the retina for each XY position in the XY plane perpendicular to the depth direction (Z direction) of the eye to be inspected, such as a brightness value. It is a map (map image) represented by, and is also called a layer thickness map. An example of the observation target layer is a nerve fiber layer (NFL), a ganglion cell layer (GCL), and an inner plexiform layer (IPL), and a thickness map can be generated by integrating these layer thicknesses. .. Further, the thickness map may be generated from the thickness of only the nerve fiber layer (NFL), or the thickness map may be generated based on the layer thickness of any layer. The thickness map can be represented by converting the layer thickness at each XY position into a grayscale luminance value. Further, the thickness map may be represented by converting the layer thickness at each XY position into a luminance value of a pseudo color scale. Further, the pixel size of the input OCT data may be changed in consideration of the visibility of the map and the processing load.

Specifically, the control unit 102 performs segmentation processing on the input tomographic image, extracts the layer structure in the tomographic area of the eye to be inspected, and calculates the thickness of each layer structure. As the method of segmentation processing and the method of calculating the thickness, any known method may be used. The control unit 102 generates a thickness map of the eye to be inspected using the thickness calculated for the observation target layer. The thickness map of the left eye and the right eye thus obtained is an example of a pair of left and right fundus image information of the subject, that is, the first fundus image information and the second fundus image information. Further, the control unit 102 is an example of the image information acquisition means for acquiring the image information in this way.

[Step S22: Correction process]
Next, the control unit 102 performs correction processing on the thickness map. An example of the correction process will be described below.

An example of the correction process is a process (error correction process) for correcting the luminance value of an inappropriate portion of the thickness map. For example, if an error occurs in the layer segmentation process, the thickness at that position becomes a specific value and is represented as overexposure or underexposure on the thickness map. The control unit 102 performs a process of replacing the luminance value of such an inappropriate portion having a specific value in the thickness map with, for example, an average value of the periphery. Alternatively, the control unit 102 may remove such an inappropriate portion from the thickness map. Further, the control unit 102 may similarly replace the luminance value of a region considered unnecessary for diagnosis, such as the inside of the optic nerve papilla (hereinafter, also simply referred to as “papillary head”), or remove the region from the map. You may.

Further, when it is determined that the image quality of the image data input from the input unit 101 is low and inappropriate for use in diagnosis, the control unit 102 may display a message prompting re-shooting on the display unit 103. can. This judgment may be made visually by the examiner (user). Further, automatic determination based on rules based on image brightness, success rate of segmentation, etc. may be performed, or image quality determination by machine learning may be performed.

Another example of the correction process is a process of correcting the layer thickness value based on the axial length data of the eye to be inspected when the thickness map is generated. Here, the axial length data of the eye to be inspected may be acquired in step S1 or may be input by the examiner. When the axial length of the eye to be examined is longer than that of the normal eye database, the retina is stretched, which has the effect of thinning the entire layer thickness. Therefore, the control unit 102 can correct the thickness map so as to reduce the influence of the layer thickness according to the axial length. The layer thickness may be corrected according to the axial length by using any known method. By such correction, it is possible to distinguish between the case where the layer thickness is thinned due to the lesion and the case where the layer thickness is thinned from normal. Further, since the photographing range of the fundus with respect to the scan angle changes when the axial length is different, the control unit 102 may perform a process of correcting the scale of the thickness map. Also, when the patient has severe myopia, the axial length tends to be long. Therefore, if the patient has severe myopia, the control unit 102 may make similar corrections to the thickness map. The axial length of the eye to be inspected and the non-existence of severe myopia may be determined based on the input patient data.

Another example of the correction process is a process of geometrically correcting the thickness map (geometric correction process). An example of the geometric correction process will be described with reference to FIGS. 4 (a) and 4 (b). Here, FIG. 4A shows an example of a thickness map before the geometric correction process, and FIG. 4B shows an example of a thickness map after the geometric correction process.

As a geometric correction process, the control unit 102 refers to, for example, reference information about the thickness map stored in the storage unit 105, and expands / contracts (enlarges or reduces), rotates, and translates the generated thickness map. Perform geometric correction processing. This correction process may be performed using an affine transformation.

An example of the reference information stored in the storage unit 105 is the center position of the macula, the center position of the papilla, and the macula to the papilla in the image as shown by the dotted lines in FIGS. 4 (a) and 4 (b). The distance and angle to part B. The angle from the macula A to the nipple B may be, for example, the angle of the line connecting the macula A and the nipple B with respect to the horizontal direction (horizontal direction) of the image.

Specifically, first, the control unit 102 translates the thickness map so that the center position of the macula portion A comes to the center of the thickness map. The center position information of the macula A can be obtained by analyzing a tomographic image, a thickness map, or the like. Further, the thickness map may be displayed on the display unit 103, and the examiner may select the center position of the macula using the operation unit 104. Next, the expansion / contraction and rotation of the thickness map are adjusted so that the line connecting the macula A and the papilla B shown by the dotted line in FIG. 4A has the reference length and angle (horizontal). The reference length here can be set arbitrarily, but as an example, the distance from the macula (A) to the papilla (B) may be set to be equivalent to 4500 μm, and the image may be scaled. .. By such adjustment, the thickness map as shown in FIG. 4B can be obtained. note that. For example, the average value of the peripheral pixel values can be used as the pixel value at the edge portion where the pixel value is lost by the geometric correction process. Further, a process of cutting out the entire image may be performed so that the edges are sufficiently filled. Further, the aspect ratio may be corrected.

The control unit 102 may perform these correction processes in combination.

[Step S23: Symmetry map generation]
Next, the control unit 102 uses the thickness map to generate a symmetry map showing symmetry with respect to the eye to be inspected (S23). Hereinafter, an example of a method for generating a symmetry map will be described with reference to FIGS. 5 (a) and 5 (b). FIG. 5A is an example of a thickness map obtained from a tomographic image of the right eye of a subject, and FIG. 5B is a thickness map obtained from a tomographic image of the left eye of the same subject. be. In FIGS. 5A and 5B, the central lines H1 and H2 extending in the horizontal direction and the central lines V1 and V2 extending in the vertical direction of the thickness map are single-point chain lines as criteria for the symmetry of the eye to be examined. It is shown. In this embodiment, the center lines H1, H2, V1, and V2 are defined with reference to the center of the macula in the thickness map. That is, the central lines H1 and H2 extending in the horizontal direction are the first standards of symmetry of the eye to be inspected, which are set in the first fundus image information and the second fundus image information with respect to the macula and the papilla. This is an example of the coordinate axes of. Further, the central lines V1 and V2 extending in the vertical direction are examples of the second coordinate axis perpendicular to the first coordinate axis. A two-dimensional coordinate system is set from the first coordinate axis and the second coordinate axis. The control unit 102 is an example of the coordinate system setting means for setting the two-dimensional coordinate system in this way. The first coordinate axis and the second coordinate axes (center lines H1, H2, V1, V2) that are the criteria for the symmetry of the eye to be inspected are not limited to these, and the object that can secure the symmetry of the eye to be inspected is used as a reference. It may be determined.

The symmetry map of this embodiment has the luminance values I1 and I2 of the pixel P1 of interest in the thickness map of FIG. 5A and the pixels P2, P3, and P4 shown in FIGS. 5A and 5B, respectively. , I3, I4. P2 is a pixel at a position vertically symmetrical with respect to P1 with respect to the horizontal center line H1 of the thickness map. P3 is an example of a position in the map of FIG. 5B that is symmetrical to the position corresponding to P1 and the center line V2 in the vertical direction (the relative relationship between the macula and the papilla corresponds to P1). ) Pixel. P4 is a pixel at a position vertically symmetrical with respect to P3 and the horizontal center line H2 of the thickness map. I1, I2, I3, and I4 may be the average value of the luminance values of the peripheral pixels, respectively. That is, the pixel of interest P1 is an example of the first point in the coordinate system of the first fundus image information, and the peripheral pixels thereof are an example of the first region. The control unit 102 is an example of a point or area designating means for designating a point or area in this way. In one example of this embodiment, each point in the thickness map is sequentially designated as a first point or region. In addition, a specific point or area may be specified by the examiner. Further, P2, P3, P4 and their peripheral pixels are examples of a second point or region, a third point or region, and a fourth point or region, respectively. The control unit 102 is an example of the target position specifying means for specifying the point of symmetry or the region in this way. Further, the luminance values at P1, P2, and P3 of the thickness map are the values relating to the first point or region in the first fundus image information, the values relating to the second point or region in the first fundus image information, and the second. It is an example of the value regarding the third point or region in the fundus image information of. Further, the control unit 102 is an example of information acquisition means for acquiring values related to the first to third points or regions.

As an example of the method of generating the symmetry map, first, the ratios of the luminance values of the other three points to I1 are calculated as R12 = I1 / I2, R13 = I1 / I3, and R14 = I1 / I4, respectively. Next, the minimum value of these ratios, RM1 = Min (R12, R13, R14) is selected. Next, the characteristic value IS1 is determined by performing logarithmic conversion as in the equation (1).

IS1 = A × Log10 (RM1) + B ... (1)
Here, A is a coefficient for adjusting the gradation range so that the symmetry map is easy to see. B is an offset and can be, for example, an intermediate luminance value. For example, when a symmetry map is generated with 8-bit (256 gradations) data, A = 30 and B = 128 can be set as an example. Especially in glaucoma, it is necessary to pay attention to the area where the retina thickness is thin. Therefore, by performing logarithmic conversion as in Eq. (1), it is possible to emphasize the part with a small ratio more sensitively. By performing this process for each pixel in the thickness map, a symmetry map is generated. The symmetry map is an example of a characteristic map generated based on the characteristic value determined as described above and the coordinate value of the specified point or region.

Further, the characteristic value IS1 may be determined based only on I1, I2, and I3, and by setting RM1 = Min (R12, R13), the vertical and horizontal symmetry can be evaluated together. As described above, the control unit 102 has the value regarding the first point or region in the first fundus image information, the value regarding the second point or region in the first fundus image information, and the second fundus image information. It is an example of a determination means for determining a characteristic value for a first point or region based on a value for a third point or region. Further, as described above, the determining means is the first comparison result obtained by comparing the value relating to the first point or region with the value relating to the second point or region, and the value relating to the first point or region and the first. The third comparison result obtained by comparing with the second comparison result obtained by comparing with the value relating to the point or region of 3 is determined as a characteristic value.

The edge portion where the pixel value is lost by the geometric correction process can be filled with, for example, an intermediate luminance value or a luminance value based on the luminance value of a nearby pixel without performing the calculation of the equation (1). In addition, based on the statistical data of the rotation variation of the captured image, the edge area of the thickness map is uniformly defined from the periphery to the number of pixels, and the pixels with zero brightness value of the thickness map within this edge area are symmetrical. In the map, it may be replaced with an intermediate brightness value. When the macula or transpapillary head is included in the edge, these parts may be displayed with an intermediate luminance value on the symmetry map, or may be displayed with the luminance value set to zero or the like.

Further, the control unit 102 may perform a process of smoothing the thickness map before calculating the ratio of the luminance values. The smoothing process can be performed by a moving average filter process, a Gaussian filter process, a median filter process, or the like. By performing the smoothing process, it is possible to reduce the influence of the position shift and the rotation shift for each thickness map. The size and strength of the filter may be determined based on the thickness of the blood vessels in the thickness map. Further, a process of reducing the number of pixels may be performed.

In addition, the macula and the border of the papilla are susceptible to misalignment because the value on the thickness map becomes steeper than the surrounding area. Therefore, a process of masking a region susceptible to these misalignments may be performed. Further, the smoothing treatment as described above may be applied only to these regions.

Further, the effect of evaluating both the vertical and horizontal symmetries in this way will be described with reference to FIG. In (a), (b), (c), and (d) of each row of FIG. 6, the left column is the map of the right eye and the right column is the map of the left eye.

FIG. 6A schematically shows a thickness map. The black-painted elliptical regions E1, E2, and E3 indicate that the thickness is smaller (thinner) than the background region indicated by the dot pattern due to glaucoma. Further, the thicknesses of E1, E2, and E3 have the same value. E2 is distributed at a position vertically symmetrical with E1. Further, E3 is distributed at a position symmetrical to E2 in the left eye.

FIG. 6B is a vertical symmetry map in which the symmetry of only the vertical is evaluated. In the vertical symmetry map, the luminance value IUD1 at P1 and the luminance value IUD2 at P2 in FIG. 5A are set to IUD1 = A × Log10 (I1 / I2) + B ... (2) using the above I1 and I2.
IUD2 = A × Log10 (I2 / I1) + B ... (3)
It is a map generated by performing the processing of the above for each pixel in the thickness map.

In the left eye of FIG. 6 (a), the area where the thickness was small was only in the lower half, so that the upper half had a higher luminance value than the surroundings and the lower half had a higher luminance value than the surroundings, as shown in FIG. 6 (b). A small, vertical symmetry map is obtained. On the other hand, since E1 and E2 were distributed in vertically symmetrical positions, the ratio of these positions was the same as that of the surroundings, and no structure was seen in the vertically symmetrical map of the right eye in FIG. 6 (b). As described above, if glaucoma progresses almost symmetrically in the vertical direction only with the vertical symmetry map, it may be overlooked that it is glaucoma.

FIG. 6C is a left-right symmetry map in which left-right symmetry is evaluated. In the left-right symmetry map, the luminance value ILR1 at P1 and the luminance value ILR3 at P3 in FIG. 5 (a) are set to ILR1 = A × Log10 (I1 / I3) + B ... (4) using the above I1 and I3.
ILR3 = A × Log10 (I3 / I1) + B ... (5)
It is a map generated by performing the processing of the above for each pixel in the thickness map.

The upper half of FIG. 6 (a) was thinner than the left eye in the right eye, and the lower half was thinner than the surroundings in both the right eye and the left eye and had the same thickness. As shown above, a left-right symmetry map is obtained in which the brightness value of the upper half of the left eye is smaller than that of the surroundings and the brightness value of the upper half of the right eye is larger than that of the surroundings.

On the other hand, since E2 and E3 were distributed in symmetrical positions, the ratio of these positions was the same as that of the surroundings, and the structure was formed in the lower half of the left-right eye and left-eye symmetry map in FIG. 6 (c). can not see. Thus, in the left-right symmetry map, if glaucoma progresses almost symmetrically between the right eye and the left eye, it may be overlooked as glaucoma.

Further, when both the vertical symmetry map of FIG. 6 (b) and the horizontal symmetry map of FIG. 6 (c) are viewed individually, the region of E2 has the same luminance value as the surroundings and is not extracted as a thin portion. .. In this case, it may be overlooked that glaucoma is progressing.

FIG. 6D is a symmetry map according to the present embodiment generated by the above method. In this symmetry map, it is possible to extract that the region of E2 is thin. The symmetry map according to the present embodiment refers to the diagonal positions that are vertically and horizontally symmetrical in addition to the vertical and horizontal directions, and by taking these minimum values, the pixel of interest can be compared with other regions from more conditions. I have been able to obtain information on whether it is not as thin as it is. By referring to more conditions, a map closer to the conventional Deviation Map can be obtained. Furthermore, the method of comparing the thickness map with the normal eye database has the following two problems. First, the comparison map of the normal eye database for glaucoma is generally designed to respond only to areas thinner than the normal range, so if the original layer thickness is thick, it may become slightly thinner. The problem is that the initial glaucoma cannot be detected because it falls within the normal range. The other is that the running direction of medium and large blood vessels and nerve fiber bundles seen from the center of the nipple differs from individual to individual, so if the thick part is different from other people, it will be detected as false positive even in healthy eyes. Is the issue. These problems can be improved by the present embodiment. Moreover, since the symmetry can be expressed by one map, the abnormal part can be shown more easily than displaying the vertical symmetry map and the left-right symmetry map individually. When the symmetry map is generated with RM1 = Min (R12, R13), it may be overlooked that the region of E2 in FIG. 6A is thin as described above.

Also, the symmetry may be converted to a pseudo color scale. This makes it easier to see the asymmetric feature area.

Furthermore, the presence or absence of symptoms and the stage may be quantitatively evaluated based on the distribution of the luminance values of the symmetry map. For example, a standard deviation can be used as the distribution of the luminance values of the symmetry map. Since the symmetry map of a healthy person has almost uniform luminance values, the standard deviation is small. On the other hand, as glaucoma progresses and an asymmetric structure is formed, the standard deviation increases. The threshold value of the standard deviation for determining the presence or absence of symptoms and the stage may be determined by a rule base, or a machine learning model may be generated from patient data to determine the threshold value.

In anterior visual field glaucoma, which is an early symptom of glaucoma, such a region with a small layer thickness is partially generated. Since anterior visual field glaucoma often has no subjective symptoms, such symptoms can be detected by observation with a symmetry map. On the other hand, it is known that as glaucoma progresses, the retina becomes thinner overall. Therefore, if the retina is thinned as a whole from the thickness map, the display unit 103 may suggest the possibility of glaucoma.

[Step S3: Display symmetry map]
The control unit 102 causes the display unit 103 to display the symmetry map generated in this way. A thickness map or the like may be displayed together. When displaying the symmetry map and the thickness map on the display unit 103, the geometric correction process performed in step S22 may be restored and then displayed. The examiner can diagnose whether or not glaucoma is present and the degree of progression (stage) of glaucoma based on the symmetry breaking. In addition, the diagnosis of age-related macular degeneration can be similarly performed from the local collapse of the symmetry of the retinal film thickness. The standard deviation for symmetry may be displayed with the symmetry map. The symmetry map may be displayed on a shooting screen displayed during shooting, or may be displayed on a report screen displaying shooting results and analysis results after shooting. Further, it may be displayed on both of them.

The symmetry map may be superimposed on at least one of a fundus photograph, an SLO image, an OCT En-face image, and an OCT angiography (OCTA) image. The En-Face image of the OCT may be a front image generated by using the data (three-dimensional OCT data) acquired by the OCT in at least a part of the depth direction of the imaging target. .. Further, the OCTA image may be a frontal image (OCTA En-Face image) that is generated from motion contrast data obtained from two or more sets of OCT data and measures the movement of blood flow. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , Can be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. By displaying the symmetry map on top of these images, it is possible to clearly show the site where the symptoms are occurring. At this time, the geometric correction processing performed in step S22 may be restored and then the processing may be performed.

By generating the symmetry map in this way, it is possible to make a diagnosis without using a normal eye database and without depending on individual differences in the overall retinal thickness.

[Modified example of the first embodiment]
The control unit 102 may perform estimation by a machine learning model using the generated symmetry map. A machine learning model can be generated by associating a symmetry map with a healthy person or the stage of glaucoma. When generating a machine learning model, a thickness map may also be used as an input image. The control unit 102 causes the display unit 103 to display at least one of the learning result and the symmetry map of the subject.

Further, in the above example, the thickness map was generated using the minimum value RM1 = Min (R1, R2, R3) of R1, R2, R3, but instead, the average value IM of I2, I3, I4 was calculated. , RM1 = I1 / IM, and a symmetry map may be generated from the equation (1). Further, a symmetry map may be generated by using the value of RM1 as the luminance value in P1 without performing logarithmic transformation. Further, the symmetry map may be generated by taking the difference instead of taking the ratio of the luminance values. However, when the evaluation is based on the difference, it is affected by individual differences in the overall net thickness. In that case, the influence of the overall net thickness may be corrected from the data such as the axial length and visual acuity.

In the above example, the symmetry map is generated from the OCT data including the macula and the papilla, but the symmetry map may be generated from the OCT data scanned around the papilla. This has the effect of improving the detection accuracy of diseases in which changes occur around the nipple. At this time, the horizontal central axis for evaluating the vertical symmetry may be set by extracting the positional information of the macula and the central portion from the separately acquired information such as the OCT data and the fundus photograph. At this time, the central axis in the vertical direction for evaluating the left-right symmetry may be set so as to pass through the center of the papilla.

[Second Embodiment]
In order to predict the prognosis of an ophthalmic disease, it is required to detect the progression of the disease with high accuracy from a plurality of fundus images (time series data) taken at different times with the same eye to be inspected. Therefore, the image processing apparatus according to the present embodiment presents data for detecting the progression of the disease without using the normal eye database by generating a symmetry map from the time series data. Since the image processing apparatus according to the present embodiment has the same configuration as that of FIG. 1, the same reference numerals are used and the description thereof will be omitted. Further, since the image processing apparatus according to the present embodiment processes the same flowchart as in FIG. 2, it will be described with reference to the same reference numerals. Hereinafter, the control unit 102 according to the present embodiment will be described with a focus on the differences from the first embodiment.

[Step S1: Image input]
The input unit 101 according to the present embodiment inputs a plurality of fundus images taken at different times for the same eye to be inspected (step S1). The plurality of fundus images input by the input unit 101 are output to the control unit 102.

For example, at least one three-dimensional tomographic image taken in the past is input to the input unit 101 from an existing database through a communication network or the like. At the same time, the newly captured tomographic image is input from the OCT device to the input unit 101 through a cable or the like. These tomographic images are output to the control unit 102. The past tomographic image may be stored in the storage unit 105 in advance and input from the storage unit 105 to the control unit 102.

Further, the input unit 101 can also acquire a fundus photograph, an SLO image, an En-Face image, an OCTA image, patient data, and the like, as in the first embodiment.

[Step S2: Image processing]
The control unit 102 according to the present embodiment compares the fundus images of the same eye to be inspected taken at different times to generate a symmetry map. (Step S2). An example of symmetry map generation will be described with reference to the flowchart of FIG. 7 and the schematic diagram of FIG.

First, the control unit 102 generates the first thickness map 811 from the newly captured tomographic image (first OCT image 801), and the second from the previously captured tomographic image (second OCT image 802). The thickness map 812 of is generated (step S31). The thickness map is generated in the same manner as in the first embodiment. As an example, the second thickness map is generated using normal data and the oldest acquired time data. The first thickness map 811 and the second thickness map 812 obtained in this way are examples of the fundus image information of the subject at different times, that is, the first fundus image information and the second fundus image information. be. Further, the control unit 102 is an example of the image information acquisition means for acquiring the image information in this way.

Next, the control unit 102 performs correction processing on the first thickness map 811 and the second thickness map 812 (step S32).

An example of the correction process of the present embodiment includes a geometric correction process. Similar to the first embodiment, the control unit 102 scales (enlarges or reduces) the first thickness map 811 and the second thickness map 812 while referring to the reference information stored in the storage unit 105. , Rotation, translation, and other geometric correction processes. The reference information may be set uniformly as in the first embodiment, or the distance from the macula A to the papilla B of the thickness map generated from the tomographic image taken at the oldest time. It may be the angle and the position of the macula A and the papilla B. In this case, the thickness map generated from the tomographic images taken at other times is corrected with reference to the reference information. Further, the image used for acquiring the reference information is not limited to the oldest one, and may be an image taken at an arbitrary time.

Further, an example of the correction process of the present embodiment can include a process of smoothing the thickness map. The smoothing process can reduce the effects of misalignment and rotational misalignment between tomographic images taken at different times. In addition, a process of masking a region susceptible to misalignment, such as the macula or the border of the nipple, may be performed. Further, the correction processing may include other image processing such as correction of the luminance value of the inappropriate portion and correction according to the axial length.

In one example of the present embodiment, the symmetry map is the luminance of the pixel P1 of interest in the first thickness map of FIG. 8, and the luminance of each of P2, P3, and P4 in the first thickness map and the second thickness map. Generated from the values I1, I2, I3, I4. P2 is a pixel at a position vertically symmetrical to P1 with respect to the horizontal center line of the thickness map. P3 is a pixel at a position corresponding to P1 in the second thickness map. P4 is a pixel whose position is vertically symmetrical with respect to the horizontal center line of P3 and the thickness map. I1, I2, I3, and I4 may be the average value of the luminance values of the peripheral pixels, respectively. That is, the pixel of interest P1 is an example of the first point in the coordinate system of the first fundus image information, and the peripheral pixels thereof are an example of the first region. The control unit 102 is an example of a point or area designating means for designating a point or area in this way. In an example of this embodiment, each point in the thickness map is sequentially designated as a first point or region by the control unit 102. Further, a specific point or area may be designated as a first point or area by the examiner. Further, P2, P3, P4 and their peripheral pixels are examples of a second point or region, a third point or region, and a fourth point or region, respectively. The control unit 102 is an example of the target position specifying means for specifying the second point or region, the third point or region, and the fourth point or region in this way. Further, the luminance values at P1, P2, and P3 of the thickness map are the values relating to the first point or region in the first fundus image information, the values relating to the second point or region in the first fundus image information, and the first. It is an example of the value regarding the third point or region in the fundus image information of 2. Further, the control unit 102 is an example of information acquisition means for acquiring values related to the first to third points or regions.

As an example of the method of generating the symmetry map, first, the ratios of the luminance values of the other three points to I1 are calculated as R12 = I1 / I2, R13 = I1 / I3, and R14 = I1 / I4, respectively. Next, the minimum value of these ratios, RM1 = Min (R12, R13, R14) is selected. Next, the characteristic value IS1 is determined by performing logarithmic conversion as in the equation (1). By performing this process for each pixel in the thickness map, a symmetry map is generated. The symmetry map is an example of a characteristic map generated based on the characteristic value determined as described above and the coordinate value of the specified point or region.

Further, the characteristic value IS1 may be determined based on I1, I2, and I3. At this time, by setting RM1 = Min (R12, R13), the vertical symmetry of the current thickness map and the symmetry obtained by comparing the luminance value of the current thickness map with the luminance value of the past thickness map. The two symmetries of and can be evaluated together. As described above, the control unit 102 has the value regarding the first point or region in the first fundus image information, the value regarding the second point or region in the first fundus image information, and the second fundus image information. It is an example of a determination means for determining a characteristic value for a first point or region based on a value for a third point or region. Further, as described above, the determining means is the first comparison result obtained by comparing the value relating to the first point or region with the value relating to the second point or region, and the value relating to the first point or region and the first. The third comparison result obtained by comparing with the second comparison result obtained by comparing with the value relating to the point or region of 3 is determined as a characteristic value.

In addition, a thickness map is generated from each of the plurality of tomographic images taken in the past, and a plurality of symmetry maps are generated by performing the calculation described in the equation (1) on the thickness map of the oldest time. You may. As a result, the control unit 102 can generate time-series data of the correlation map. Further, the object referred to in the calculation of the equation (1) is not limited to the thickness map obtained from the image taken at the oldest time, but also the thickness map obtained from the image at another time and the thickness obtained from the normal image. It may be a map.

[Step S3: Display symmetry map]
The control unit 102 causes the display unit 103 to display the symmetry map generated in this way. By observing this symmetry map, it is possible to evaluate the progression of the disease compared to the past, in addition to the vertical symmetry at that time.

[Modified example of the second embodiment]
The control unit 102 may estimate the presence or absence of a symptom and its probability from the time change of the symmetry map by a machine learning model, and output the result to the display unit 103. Further, it is possible to estimate how much the symptom has progressed in a predetermined future and output the result to the display unit 103. In the process of machine learning, a caution map (heat map) showing the points where the learning model paid attention may be generated, and the time series data of the caution map may be displayed on the display unit 103.

[Modification 1]
In the present modification 1, the control unit 102 determines whether or not the symmetry map can be generated (or the characteristic value is determined).

When the tomographic image data that can be acquired from the input unit 101 exists in only one of the left and right eyes (for example, when the first fundus image information exists but the second fundus image information does not exist), the control unit 102 It is judged that the symmetry map generation is "impossible". Further, even when the tomographic image data of the left and right eyes are present, if the scan patterns (position, angle of view, spacing, etc.) are different on the left and right, the control unit 102 determines that the symmetry map generation is "impossible".

Further, even when the tomographic image data that can be acquired from the input unit 101 includes data for both the left and right eyes, if the control unit 102 determines that the quality of the data is not good (for example, with the first fundus image information). (When the quality of at least one of the second fundus image information is not good), the control unit 102 determines that the symmetry map generation is "impossible". Examples of poor data quality here are "dark image", "major failure in segmentation of tomographic image", "no macular or papilla", and "missing data". To do "and so on. The quality of the data may be evaluated by evaluating the tomographic image data or by evaluating the generated symmetry map.

As a result, when the control unit 102 determines that the symmetry map generation is "impossible", the control unit 102 causes the display unit 103 to display that the symmetry map generation is impossible. At this time, if the data of the left and right eyes is input but the quality is not good, the control unit 102 may generate and display the symmetry map and then display that the quality is not good.

Further, the control unit 102 may display a message on the display unit 102 prompting the user to take a picture of the data necessary for generating the symmetry map, such as the eye or the scan pattern in which the data does not exist. That is, a message urging the control unit 102 to acquire at least one of the first fundus image information and the second fundus image information based on the result obtained by determining whether or not the characteristic value can be determined. May be displayed on the display unit 102. At this time, as a message display method, a pop-out format may be used, for example, when the examiner consents to capture the data necessary for generating the symmetry map, the "OK" button is pressed. If the examiner approves this, the control unit 102 prepares for data capture and acquires the data necessary for generating the symmetry map.

[Modification 2]
In this modification 2, when the tomographic image data acquired from the input unit 101 is aligned with the left and right eyes and there are a plurality of data that can be used for symmetry map generation, the examiner or the examiner or the data used for symmetry map generation is used. A step selected by the control unit 102 is provided.

In the example in which the examiner selects the image data to be used for generating the symmetry map, the display unit 103 is made to display the candidate image data and the information related to the data. Information about the data is, for example, a shooting time or a quality evaluation value. The data quality evaluation value is set by the brightness of the image, the success rate of segmentation, the signal-to-noise ratio (SNR), and the like. The examiner selects the data to be used for the symmetry map generation by using the operation unit 104 while looking at the displayed image data and information.

In the example in which the control unit 102 selects the image data to be used for generating the symmetry map, the control unit 102 selects the data based on the shooting time, the quality evaluation value, and the like. As an example based on the shooting time, the one with the latest shooting time or the one with the closest shooting time of the left and right eye data can be selected as a selection candidate. The quality evaluation value is set by the brightness of the image, the success rate of segmentation, the signal-to-noise ratio (SNR), and the like, and those having excellent ones can be selected as selection candidates.

The control unit 102 generates a symmetry map using the tomographic image data selected in this way.

[Modification 3]
The symmetry map generated in the above-described embodiment and modification is displayed on the display unit 103 in combination with other related images or maps. This makes it easier to recognize the information on the symmetry at that position in correspondence with other information such as the blood vessel running state and the layer thickness. In this case, the symmetry map may be displayed after the geometric correction applied at the time of generation is restored.

In one example of this variant, the symmetry map and other related images and maps are displayed side by side. For example, at least one of an SLO image, an En-face image, an OCTA image, a tomographic image, and a thickness map may be displayed side by side with a symmetry map. At this time, the corresponding shooting range may be indicated by a mark such as a square frame or a line.

Further, the symmetry map may be displayed on the display unit 103 by superimposing it on another related image or map. For example, at least one of the SLO image, the En-face image, the OCTA image, and the thickness map may be superimposed and displayed on the symmetry map. In this case, the control unit 102 may include a superimposition ratio adjusting means in order for the examiner to specify the superimposition ratio of each image or map by the operation unit 104.

[Other variants]
In the above-described embodiment and modification, an example in which a symmetry map is generated from a thickness map acquired by OCT to diagnose glaucoma has been described, but the embodiment is not limited to this.

For example, a symmetry map may be generated from the thickness map to diagnose diseases that cause retinal deformity such as age-related macular degeneration (AMD).

In addition, a symmetry map may be similarly generated from a fundus photograph, a fluorescent monochromatic frontal image of the fundus, and an SLO image to diagnose a disease such as diabetic retinopathy and glaucoma in which a loss of symmetry appears in terms of light and shade and color difference. .. Further, instead of the SLO image, an En-face image obtained from the OCT image may be used.

Alternatively, a blood vessel density map may be obtained from the OCTA image and a symmetry map thereof may be generated in the same manner. This symmetry map may be used to diagnose diseases such as diabetic retinopathy that have symptoms in vascular density.

In addition, an En-face image of the choroid is generated from an OCT image acquired by Swept Source OCT (SS-OCT) and the like, and the symmetry of the vascular running pattern is evaluated to diagnose central serous chorioretinosis. You may go.

In addition, tomographic images of the left and right eyes are acquired as shown in FIGS. 9 (a) and 9 (b), and the one-dimensional network film thickness profile and the vertical-left-right symmetry of the retinal topology shape as shown in FIGS. 9 (c) and 9 (d) are obtained. You may evaluate it. For example, even if the same processing as in the first embodiment is performed on the vertically symmetrical position P2 with respect to a certain point P1, the vertically symmetrical position P3, and the vertically symmetrical position P4 with P1 of the opposite eye. good. That is, one-dimensional symmetry is obtained by plotting RM1 = Min (R12, R13, R14) by taking the ratios R12, R13, and R14 of the thicknesses of P2, P3, and P4 to the thickness of P1. A map (FIG. 9 (e)) can be generated. Then, the diagnosis of age-related macular degeneration and glaucoma may be made.

Further, the symmetry map is obtained from the image information obtained by calculating at least two of the fundus photograph, the SLO image, the fluorescent monochromatic frontal image of the fundus, the En-face image, the OCTA image, and the layer thickness map obtained by the fundus camera. May be generated. One example of the image information is a superimposed image, and as an example, a symmetry map may be generated from the superimposed image of the fundus photograph and the layer thickness map.

Needless to say, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

The present invention is also realized by executing the following processing. That is, the present invention supplies software (program) that realizes one or more functions of the various embodiments and modifications described above to a system or device via a network or storage medium, and the computer of the system or device ( Alternatively, it can also be realized by a process in which a CPU, MPU, etc.) reads and executes a program. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

At this time, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims (10)

A setting means for setting a two-dimensional coordinate system based on the first coordinate axis based on the macula and the nipple in the first fundus image information and the second fundus image information of the subject's left and right eyes.
The first point or region in the first fundus image information, the second point or region symmetrical with respect to the first coordinate axis, and the first point or region in the second fundus image information and the macula. And the specific means of identifying the third point or region to which the relative relationship with the nipple corresponds.
The value regarding the first point or region in the first fundus image information, the value regarding the second point or region in the first fundus image information, and the third in the second fundus image information. A determination means for determining the characteristic value for the first point or region based on the value for the point or region, and
Image processing device with. A setting means for setting a two-dimensional coordinate system based on the first coordinate axis with respect to the macula and the papilla in the first fundus image information and the second fundus image information obtained by photographing the eye to be inspected at different times. When,
The first point or region in the first fundus image information, the second point or region symmetrical with respect to the first coordinate axis, and the first point or region in the second fundus image information and the macula. And the specific means of identifying the third point or region to which the relative relationship with the nipple corresponds.
The value regarding the first point or region in the first fundus image information, the value regarding the second point or region in the first fundus image information, and the third in the second fundus image information. A determination means for determining the characteristic value for the first point or region based on the value for the point or region, and
Image processing device with. The fundus image information is a color, fluorescent monochromatic frontal fundus image, En-face image, OCTA image, layer thickness map, retinal topology, or at least two of these, obtained from at least one of the fundus camera, SLO, and OCT. The image processing apparatus according to claim 1 or 2, which is the image information obtained by calculating. The specifying means identifies the third point or region in the second fundus image information and the fourth point or region symmetrical with respect to the first coordinate axis, and also identifies the fourth point or region.
The determining means includes a value related to the first point or region in the first fundus image information, a value related to the second point or region in the first fundus image information, and the second fundus image information. In any one of claims 1 to 3, the characteristic value relating to the first point or region is determined based on the value relating to the third point or region and the value relating to the fourth point or region. The image processing device described. The image processing apparatus according to any one of claims 1 to 4, wherein a characteristic map is generated based on the determined characteristic value and the coordinate value of the first point or region. Based on the result obtained by determining whether or not the characteristic value can be determined, a message prompting the acquisition of at least one of the first fundus image information and the second fundus image information is displayed on the display unit. The image processing apparatus according to any one of claims 1 to 5, further comprising a control means. The determining means includes a first comparison result obtained by comparing a value relating to the first point or region and a value relating to the second point or region, a value relating to the first point or region, and the first. In any one of claims 1 to 6, the third comparison result obtained by comparing with the second comparison result obtained by comparing with the value relating to the point or region of 3 is determined as the characteristic value. The image processing device described. A step of setting a two-dimensional coordinate system based on the first coordinate axis based on the macula and the nipple in the first fundus image information and the second fundus image information of the subject's left and right eyes, and
The first point or region in the first fundus image information, the second point or region symmetrical with respect to the first coordinate axis, and the first point or region in the second fundus image information and the macula. And the step of identifying the third point or region to which the relative relationship with the nipple corresponds.
The value regarding the first point or region in the first fundus image information, the value regarding the second point or region in the first fundus image information, and the third in the second fundus image information. The step of determining the characteristic value for the first point or region based on the value for the point or region, and
Image processing method having. A step of setting a two-dimensional coordinate system based on the first coordinate axis based on the macula and the papilla in the first fundus image information and the second fundus image information obtained by photographing the eye to be inspected at different times. ,
The first point or region in the first fundus image information, the second point or region symmetrical with respect to the first coordinate axis, and the first point or region in the second fundus image information and the macula. And the step of identifying the third point or region to which the relative relationship with the nipple corresponds.
The value regarding the first point or region in the first fundus image information, the value regarding the second point or region in the first fundus image information, and the third in the second fundus image information. The step of determining the characteristic value for the first point or region based on the value for the point or region, and
Image processing method having. A program that causes a computer to execute each step of the image processing method according to claim 8 or 9.

Images