JPWO2019073962A1 - Image processing apparatus and program - Google Patents

Abstract

Learn the relationship between fundus photo image data and information about eye symptoms using learning data that includes information that correlates image data of fundus photos and information about eye symptoms corresponding to each fundus photo Holds the machine learning result in the processed state, accepts the image data of the fundus photo to be processed, and uses the input data based on the received image data and the machine learning result to process the fundus photo An image processing apparatus that estimates information related to eye symptoms for the eyes.

Description

  The present invention relates to an image processing apparatus and a program for processing an image for ophthalmic medicine.

  Early detection is required for diseases with irreversible loss of visual function, such as glaucoma. However, in glaucoma, since the examination for the definitive diagnosis is generally time-consuming and burdensome, there is a need for a method for simply detecting the possibility of glaucoma by screening in advance.

  An apparatus that provides information for diagnosis using a three-dimensional measurement result of the fundus is disclosed in Patent Document 1.

JP 2017-74325 A

  However, it is difficult to set conditions for detecting the presence or absence of glaucoma symptoms based on fundus information in the above-described conventional devices. This is because the information used by the doctor for the diagnosis is the interrelationship between the color of the optic disc in and around the nipple, thinning of the rim, deepening of the recess, laminar dot sign, nasal deviation of the papillary vessels, PPA (papillae This is because it is based on comprehensive judgment of information such as peripheral network choroidal atrophy), papillary marginal hemorrhage, retinal nerve fiber layer defect, etc. ("Glaucomatous optic nerve head / retinal nerve fiber layer change guidelines" vol. 110, No. 10, p810-, (2006)). In recent years, there is a device that takes a fundus photograph with a non-mydriatic, but it is difficult to obtain three-dimensional information with such a device, and three-dimensional information is not always obtained.

  In addition, the color and blood vessel shape of fundus photographs vary greatly depending on the shooting conditions and the individual differences of the target. For this reason, for example, segmentation processing based simply on pixel values or the like is not practical for detecting an image portion that is useful for diagnosing glaucoma, such as a concave edge of the optic disc.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus and a program capable of detecting the presence or absence of glaucoma relatively easily based on a fundus image. .

  The present invention that solves the problems of the conventional example is an image processing device, and includes learning data including information that correlates image data of fundus photographs and information about eye symptoms corresponding to each fundus photograph. A holding means for holding a machine learning result in a state in which the relationship between image data of a fundus photograph and information on eye symptoms is machine-learned, and a receiving means for receiving image data of a fundus photograph to be processed Using the input data based on the received image data and the machine learning result, estimating means for estimating information about the eye symptom about the eye related to the fundus photograph that has been processed, and the estimation And means for outputting the result.

  According to the present invention, the possibility of glaucoma can be detected relatively easily based on the fundus image.

It is a block diagram showing the example of a structure of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of the image data of the fundus photograph which the image processing apparatus which concerns on embodiment of this invention processes. It is a functional block diagram showing the example of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of the image which the image processing apparatus which concerns on embodiment of this invention outputs.

  Embodiments of the present invention will be described with reference to the drawings. As illustrated in FIG. 1, the image processing apparatus 1 according to the embodiment of the present invention includes a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and an input / output unit 15. Composed.

  Here, the control unit 11 is a program control device such as a CPU, and operates according to a program stored in the storage unit 12. In the example of the present embodiment, the control unit 11 uses the learning data including information in which the image data of the fundus photograph and the information related to the eye symptom corresponding to each fundus photograph are associated with each other. The processing using the machine learning result in a state in which the relationship between the image data and the information regarding the eye symptoms is learned is performed.

  Note that the learning data does not necessarily have to be information that correlates image data of fundus photographs and information about eye symptoms corresponding to each fundus photograph. Image data of fundus photographs that are not associated with information on symptoms may be included.

  The machine learning result may be a result of machine learning using, for example, a neural network, SVM (Support Vector Machine), a Bayesian method, or a method based on a tree structure, or semi-supervised learning. The model etc. which are obtained by these may be sufficient.

  Specifically, the control unit 11 receives the image data of the fundus photograph to be processed, acquires the output of the neural network when the input data based on the received image data is input, and the input data and the machine learning result Is used to estimate information about a predetermined symptom about the eye related to the fundus photograph that is the target of processing. Then, the control unit 11 outputs the estimation result. Details of the operation of the control unit 11 will be described later.

  The storage unit 12 is a disk device, a memory device, or the like, and holds a program executed by the control unit 11. The program may be provided by being stored in a computer-readable non-transitory recording medium and stored in the storage unit 12. Further, in the present embodiment, the storage unit 12 uses the learning data in which the image data of the fundus photograph and the information regarding the eye symptom corresponding to each fundus photograph are associated with each other, and the image data of the fundus photograph It also functions as a holding means for holding a machine learning result in a state in which the relationship between the eye symptom and information related to eye symptoms is learned. Details of the machine learning result will be described later.

  The operation unit 13 is a keyboard, a mouse, or the like, and accepts a user instruction and outputs it to the control unit 11. The display unit 14 is a display or the like, and displays and outputs information in accordance with an instruction input from the control unit 11.

  The input / output unit 15 is an input / output interface such as USB (Universal Serial Bus), for example. In one example of the present embodiment, the fundus photo image data to be processed is transferred to an external device (for example, a photographing device or a card reader). Etc.) and output to the control unit 11.

  Next, the machine learning result used by the control unit 11 of the present embodiment will be described. In an example of the present embodiment, the machine learning result used by the control unit 11 is a neural network. The neural network used by the control unit 11 in this example uses the learning data in which the image data of the fundus photograph and the information related to the eye symptom corresponding to each fundus photograph are associated with each other, This is a machine learning state of the relationship with information about eye symptoms.

  As an example, this neural network uses a Residual network (ResNet: Kaiming He, et.al., Deep Residual Learning for Image Recognition, https://arxiv.org/pdf/1512.03385v1.pdf). It is formed. This neural network learning process can be performed using a general computer.

  More specifically, the learning process is performed on the residual network, such as image data of a fundus photograph (image data that is known to be a fundus photograph, for example, human artifacts) included in the learning data, as illustrated in FIG. That can be collected in advance). This fundus photograph may be a two-dimensional fundus photograph taken with mydriasis or non-mydriasis, but at least an image Y of the optic disc is included. The image Y of the optic nerve head is generally imaged as a region having a relatively higher brightness than the image of other parts (parts other than the part corresponding to the optic nerve head). Note that many blood vessels B are imaged in the fundus photograph.

  In addition, as information regarding eye symptoms to be included in the learning data, information indicating the presence or absence of suspected glaucoma by an ophthalmologist referring to the fundus photograph is used. In this case, for example, for each fundus photograph that is image data input in the learning process, among the plurality of ophthalmologists referring to the fundus photograph, the ratio of the ophthalmologist diagnosed as suspected of glaucoma is used. Information relating to eye symptoms may be learning data in association with image data of the fundus photograph.

  In this example of the present embodiment, for example, the dimension N of the output vector of a neural network such as a residual network is set to “1”, which is a parameter indicating the degree of suspected glaucoma. Then, learning processing is performed using the output of the neural network when the fundus photo image data of each learning data is input, and information indicating the presence or absence of suspected glaucoma associated with the input fundus photo image data. . Since this learning process can be performed by a widely known process using a back-propagation process or the like, a detailed description thereof is omitted here.

  Further, in the example of the present embodiment, information for support may be presented instead of directly determining whether or not there is a suspicion of glaucoma. For example, in the example of the present embodiment, the position of the optic disc recess edge is drawn and presented. In this case, in each fundus photograph, a curve (X in FIG. 2 (a); generally, the optic disc recessed edge is the optic disc, which is artificially drawn by a doctor (ophthalmologist) and represents the position of the optic disc recessed edge. Information representing a closed curve in the image Y) is used as information regarding eye symptoms included in the learning data. The information representing the curve may be a set of coordinate information representing the position of the pixel in the image data of the fundus photograph through which the curve itself passes.

  In addition, as support information for determining the presence or absence of suspected glaucoma, images used for diagnosis of glaucoma, such as the position of the wedge-shaped defect part of the retinal nerve fiber layer, as well as the curve representing the position of the optic disc recessed edge It may also indicate other characteristic parts above.

  In this example of the present embodiment, for example, a pixel through which an artificially drawn closed curve that passes through the dimension N of the output vector of the residual network matches the number N of pixels of the image data and represents the optic nerve crevice edge passes. The residual network is subjected to a learning process, with a vector that is “1” for “0” and “0” for the other pixels as a correct answer. This learning process can also be performed by a widely known process such as a back-propagation process, and a detailed description thereof will be omitted here.

  Further, using the learning data in which the image data of the fundus photograph and the information about the eye condition corresponding to each fundus photograph of the present embodiment are associated with each other, the image data of the fundus photograph and the eye condition The neural network that has learned the relationship with the information is not limited to the residual network.

  This neural network may be a general convolutional network (CNN). Further, the final layer of a general convolution network may be SVM (Support Vector Machine). Further, in a general convolution network, the activation function is Leaky Relu (activation function φ (x) = max (0.01x, x), where max (a, b) is the larger of a and b. May be used. Alternatively, Dense Net (Gao Huang, et. Al., Densely Connected Convolutional Network, arXiv: 1608.06993) may be used.

  Furthermore, in this embodiment, the neural network formed in this way is used as a reference network, and the same learning data is input to another neural network having a relatively small number of layers, for example, two fully connected layers. The network obtained by the so-called “distillation” process learned using the output of the reference network as a correct answer may be used.

  Further, in the present embodiment, for one piece of learning data A, image data Pa obtained by inverting or rotating the learning data A, or noise (random dots or the like) is added to the image data Pa. The processed image data may be input as image data different from the image data Pa.

  In this case, the information indicating the presence or absence of suspicion of glaucoma corresponding to the correct answer is the original learning whether the image data obtained by inverting or rotating the image data Pa is used as learning data or when noise is added. Information corresponding to the data is used as it is.

  In addition, image data Pa obtained by reversing or rotating the image data Pa, which is the learning data A, or image data obtained by adding noise (random dots or the like) to the image data Pa is referred to as image data Pa. In the case of inputting as different image data, when using an artificially drawn closed curve representing the optic papular depression edge as the information about the eye symptom, the following is performed.

  In other words, an artificially drawn closed curve representing the optic disc dip corresponding to the correct answer is the same deformation as the image data Pa when image data obtained by inverting or rotating the image data Pa is used as learning data. The information subjected to (inversion or rotation) is used, and when noise is added, the information corresponding to the original learning data is used as it is.

  In the present embodiment, information on the neural network in such a state after the learning process is stored in the storage unit 12. The information of the neural network is provided via, for example, a network or stored in a computer-readable and non-transitory recording medium, and stored in the storage unit 12.

[Other examples of machine learning results]
Further, the machine learning result information of the present embodiment is not limited to an example of a neural network. The machine learning result information here may be information learned by applying a method based on SVM (Support Vector Machine), Bayesian posterior distribution information or tree structure, for example.

  When machine learning results are obtained by applying machine learning based on a method based on SVM or Bayesian posterior distribution information or a tree structure, a vector in which pixel values of fundus photographic image data are arrayed is input information. Or information on a predetermined characteristic amount obtained from image data of the fundus photograph (for example, the size of the area surrounded by the ridge of the optic disc), and information on a predetermined symptom of the eye related to the fundus photograph Information indicating whether or not there is a suspicion of glaucoma.

  By using these pieces of information as learning data, for example, information for identifying an identification boundary surface that identifies the presence or absence of suspicion of glaucoma is obtained by machine learning of SVM. Also, machine learning is performed on a parameter for obtaining a posterior distribution of Bayesian estimation as a probable probability of glaucoma and a clustering model for determining whether or not there is a suspicion of glaucoma based on a tree structure.

[Semi-supervised learning]
Further, in an example of the present embodiment, part of the learning data may include image data of a fundus photograph that is not associated with information related to a predetermined symptom about the eye related to the fundus photograph.

  Since the machine learning method using such learning data is widely known as semi-supervised learning, detailed description thereof is omitted here.

[Operation of control unit]
Next, operation | movement of the control part 11 of this Embodiment is demonstrated. As illustrated in FIG. 3, the control unit 11 according to the present embodiment functionally includes a reception unit 21, an estimation unit 22, and an output unit 23.

  The accepting unit 21 accepts image data of a fundus photograph to be processed and outputs it to the estimating unit 22. Here, the image data of the fundus photograph received by the receiving unit 21 may also be a two-dimensional fundus photograph captured with a mydriatic or non-mydriatic pupil, but at least an image of the optic disc is included. Note that the fundus photograph may be taken with a general camera (including a camera such as a smartphone) instead of a specialized medical camera as long as at least an image of the optic disc is included. .

  The estimation unit 22 acquires an output using the machine learning result stored in the storage unit 12 when input data based on the image data received by the reception unit 21 is input. Based on the output using the machine learning result acquired here, the estimation unit 22 estimates information related to a predetermined symptom about the eye related to the fundus photograph that is the target of processing.

  As an example, when the machine learning result is a neural network as in the example described above, and the information on the predetermined symptom is a curve representing the position of the optic disc recessed edge, the estimation unit 22 determines the optic disc recessed edge. The group of pixels through which the curve representing the position passes is estimated.

  The output unit 23 outputs the estimation result. Specifically, when the estimating unit 22 estimates a group of pixels through which a curve representing the position of the optic disc recessed edge passes, the output unit 23 adds the result of the estimation to the image data of the fundus photograph received by the receiving unit 21. An image in which each pixel included in the pixel group is highlighted is displayed in a superimposed manner (FIG. 4). FIG. 4 shows an example in which the optic papilla is enlarged.

  In addition, the estimation unit 22 estimates information representing the probability of diagnosing glaucoma as information about a predetermined symptom about the eye related to the fundus photograph that is the target of processing based on the output using the machine learning result. In this case, the output unit 23 may output a numerical value that is a result of the estimation.

[Operation]
The image processing apparatus 1 according to the present embodiment basically has the above configuration and operates as follows. In an example of the image processing apparatus 1 according to the present embodiment, using a residual network, image data of the fundus photograph illustrated in FIG. 2A and an optic disc defect artificially drawn on the image data. The data representing the position of each pixel through which the closed curve representing the concave edge passes is used as learning data, and the information representing the position of each pixel of the closed curve is obtained as a vector as an output when inputting image data of the fundus photograph. The learned neural network is stored in the storage unit 12.

  When a user inputs image data of a fundus photograph that is an object of estimation processing (an object of processing for estimating the presence or absence of glaucoma symptoms) to the image processing apparatus 1, the image processing apparatus 1 accepts the image data and accepts the image data. When the input data based on the image data is input, the output of the neural network stored in the storage unit 12 is acquired.

  As a result of the neural network acquired here, the image processing apparatus 1 estimates a group of pixels through which a curve representing the position of the optic disc recessed edge passes through the eye related to the fundus photograph that is the processing target. Get. Then, the image processing apparatus 1 superimposes and displays an image in which each pixel included in the group of pixels obtained as a result of estimation is superimposed on the received fundus photographic image data (FIG. 4).

  The user determines the presence or absence of a glaucoma symptom based on the shape of the depressed edge of the optic nerve head shown in the displayed image.

  According to this example of the present embodiment, the presence or absence of the possibility of glaucoma can be detected relatively easily based on the two-dimensional fundus image.

[Preprocessing]
Further, the machine learning result held by the storage unit 12 of the image processing apparatus 1 according to the present embodiment may be a learning process that is performed as follows. That is, as illustrated in FIG. 2B as learning data, a range of an image corresponding to the optic papilla in the fundus photographic image data is specified, and partial image data including the specified image range is extracted. May be used.

  Specifically, in the fundus photograph, the optic papilla is captured as an area having a relatively higher brightness than the image of other parts (parts other than the part corresponding to the optic papilla). The pixel values of a pair of adjacent pixels are compared to find a pair of pixels that are larger than a predetermined threshold (so-called contour detection processing). And the computer which performs learning processing is good also as detecting the pixel of the comparatively low brightness | luminance among the found pixels as an outline of an optic nerve head.

  The computer that performs the learning process generates square information circumscribing the detected outline of the optic papilla, and reduces the image data so that the image portion within the square has a predetermined size (for example, 32 × 32 pixels). Enlarge conversion. Next, a square range (for example, a range of 64 × 64 pixels) having a size larger than the predetermined size centered on the center of the generated square is cut out. At this time, if there is a portion that is not included in the original image data in the portion to be cut out, the portion is padded with black pixels and used as input data (FIG. 2B).

  In order to normalize the contrast, the computer that performs the learning process further calculates an average pixel value of the converted image data (image data before padding) and subtracts it from the value of each pixel of the converted image data. Processing may be performed.

  In addition, the computer that performs the learning process is a fundus photograph such as information indicating the presence or absence of suspected glaucoma in the image data of the fundus photograph that has been subjected to the above process, and information of a closed curve that represents an artificially drawn optic disc recessed edge. Get information about a given symptom about the eye.

  In addition, information that can be overlaid on the image data of the fundus photographic image is used as information about a predetermined symptom regarding the eye related to the fundus photographic image, such as a closed curve representing an artificially drawn optic disc dip. In this case, the same conversion as the enlargement / reduction conversion and cut-out of the image data of the fundus photo is performed on the image data that is information about a predetermined symptom of the eye related to the fundus photo. For example, for the data representing the position of each pixel through which a closed curve representing an artificially drawn optic disc recessed edge passes, the same conversion as the enlargement / reduction conversion and extraction of the image data is performed, and the converted image data Conversion into corresponding pixel position data. Since this conversion process can be performed by widely known methods such as enlargement / reduction conversion and cut-out process, detailed description thereof is omitted here.

  The computer that performs the learning process includes, for example, a portion that includes an image corresponding to the optic nerve head as input data for a residual network that receives 64 × 64-dimensional data and outputs a 32 × 32-dimensional vector. The image data is input, and the learning process is executed, for example, the output is subjected to the learning process with information of the closed curve representing the optic disc depression edge corresponding to the input data. Such a learning process can also be performed by a widely known process such as backpropagation according to the machine learning mode.

  The residual network obtained by the learning process according to this example, when the partial image data including the image corresponding to the optic papilla is input, the partial image data including the image corresponding to the optic papilla in the partial image data including the image corresponding to the optic papilla The result of estimating the pixel through which the recessed edge passes is output.

  In this example, the control unit 11 performs the following operation as the processing of the estimation unit 22. That is, in the estimation unit 22 of this example of the present embodiment, an image range corresponding to the optic papilla is specified from the image data received by the receiving unit 21, and the received image data including the specified image range. The partial image data that is a part of the image is extracted, and the extracted partial image data is input as input data to the neural network stored in the storage unit 12, and the estimation result of the information on the closed curve representing the optic disc recessed edge is obtained. obtain.

  At this time, the partial image data is extracted so that the specified image range is a predetermined position in the image.

  That is, the estimation unit 22 performs a contour detection process on the image data of the fundus photograph received by the receiving unit 21, and is a pair of pixels adjacent to each other, and the difference between the pixel values is greater than a predetermined threshold value. A pair of pixels as a difference is found, and a pixel having a relatively low luminance among the found pixels is detected as an outline of the optic papilla.

  The estimation unit 22 generates square information circumscribing the detected outline of the optic papilla, and reduces or enlarges the image data so that the image portion in the square has a predetermined size (for example, 32 × 32 pixels). Convert. Next, a square range (for example, a range of 64 × 64 pixels) having a size larger than the predetermined size centered on the center of the generated square is cut out. At this time, if there is a portion that is not included in the original image data in the portion to be cut out, the portion is padded with black pixels and used as input data (similar to FIG. 2B). Thereby, the partial image data including the range of the image corresponding to the optic nerve head and in which the range is a predetermined position in the image is extracted.

  The estimation unit 22 inputs the partial image data extracted here to the neural network stored in the storage unit 12 and acquires the output. Based on the output of the neural network acquired here, the estimation unit 22 estimates information related to a predetermined symptom about the eye related to the fundus photograph that is the target of processing. In this example, since the information regarding the predetermined symptom is a curve representing the position of the optic disc recessed edge, the estimation unit 22 estimates a group of pixels through which the curve representing the position of the optic disc recessed edge passes. Will be. The output unit 23 outputs the estimation result. Specifically, the output unit 23 superimposes and displays an image in which each pixel included in the group of pixels as a result of estimation is highlighted on the image data of the fundus photograph received by the receiving unit 21 (FIG. 4). ).

  In addition, a computer that performs learning processing using information representing the probability of being diagnosed with glaucoma instead of information about a closed curve representing the optic disc recessed edge receives, for example, 64 × 64-dimensional data as a one-dimensional scalar quantity. Is input to the neural network that outputs the partial image data including the image corresponding to the optic nerve head, and the output is information indicating the probability of diagnosing glaucoma corresponding to the input data (for example, The learning process is executed, for example, by performing a learning process on the number of doctors diagnosed with glaucoma when a fundus photograph as the input data is presented to a plurality of ophthalmologists. This learning process can also be performed by a well-known process such as back propagation according to the machine learning mode.

  The neural network obtained by the learning process according to this example outputs a result of estimating the probability of diagnosing glaucoma when partial image data including an image corresponding to the optic nerve head is input.

  In this example, the control unit 11 performs the following operation as the processing of the estimation unit 22. That is, in the estimation unit 22 of this example of the present embodiment, an image range corresponding to the optic papilla is specified from the image data received by the receiving unit 21, and the received image data including the specified image range. Is extracted, and the extracted partial image data is input as input data to the neural network stored in the storage unit 12 to obtain an estimation result of the probability of diagnosing glaucoma.

  Also at this time, partial image data is extracted so that the specified image range is a predetermined position in the image.

  That is, the estimation unit 22 performs a contour detection process on the image data of the fundus photograph received by the receiving unit 21, and is a pair of pixels adjacent to each other, and the difference between the pixel values is greater than a predetermined threshold value. A pair of pixels as a difference is found, and a pixel having a relatively low luminance among the found pixels is detected as an outline of the optic papilla.

  The estimation unit 22 generates square information circumscribing the detected outline of the optic papilla, and reduces or enlarges the image data so that the image portion in the square has a predetermined size (for example, 32 × 32 pixels). Convert. Next, a square range (for example, a range of 64 × 64 pixels) having a size larger than the predetermined size centered on the center of the generated square is cut out. At this time, if there is a portion that is not included in the original image data in the portion to be cut out, the portion is padded with black pixels and used as input data (similar to FIG. 2B). Thereby, the partial image data including the range of the image corresponding to the optic nerve head and in which the range is a predetermined position in the image is extracted.

  The estimation unit 22 inputs the partial image data extracted here to the neural network stored in the storage unit 12 and acquires the output. Based on the output of the neural network acquired here, the estimation unit 22 estimates information related to a predetermined symptom about the eye related to the fundus photograph that is the target of processing. In this example, the information related to the predetermined symptom is the probability that glaucoma is diagnosed, and therefore the estimation unit 22 estimates the probability that glaucoma is diagnosed. The output unit 23 displays and outputs a numerical value that is a result of the estimation.

[Other examples of pre-processing]
Further, instead of an example of specifying a range of an image corresponding to the optic papilla as learning data, not only the optic papilla but also a range including the optic papilla and the macula is specified, and the specified image range is included. The partial image data extracted may be used as learning data and input data to be processed.

  In general, changes in the fundus corresponding to glaucoma increase from the optic papilla to the macula, so that the range including the optic papilla and the macula is cut out as the target of the learning process. It is possible to estimate based on more information by inputting image data obtained by cutting out the range including the optic papilla and macula as input data of machine learning results, such as a neural network Become.

  Furthermore, in an example of the present embodiment, processing for emphasizing a blood vessel portion may be performed on image data included in learning data and image data of input data to be processed. For example, the blood vessel part is subjected to enhancement processing by performing line segment extraction processing from a continuous contour line. When the image data in which the blood vessel part is emphasized is used as learning data or input data, information on the two-dimensional shape of the blood vessel (the shape of the blood vessel image projected on the plane) in the vicinity of the optic disc recess is obtained. The probability of diagnosing glaucoma and the estimation of the optic disc recess edge will be used to improve the learning efficiency and the accuracy of the estimation result.

[Example using 3D fundus photography]
In the description so far, two-dimensional image data is used as the image data of the fundus photograph, but the present embodiment is not limited to this.

  In other words, in an example of the present embodiment, three-dimensional information (information such as film thickness) of the fundus may be included together with image data of the fundus photograph as learning data for machine learning. In this example, for example, fundus photograph image data and fundus three-dimensional information (film thickness information) are input to a neural network, and the fundus photograph image data and fundus three-dimensional information are referenced. Based on the information, the probability that the ophthalmologist diagnoses glaucoma (the ratio of the number of ophthalmologists diagnosed with glaucoma among a plurality of ophthalmologists) is used as a correct answer to learn the neural network.

[Pre-processing in the receiving department]
The image processing apparatus 1 according to an example of the present embodiment captures image data that is an object of estimation processing that does not have sufficient image quality for estimation processing, or a fundus structure such as an optic nerve head in the first place. If it is determined that the image data cannot be estimated, such as data that has not been estimated, and if it is determined that estimation cannot be performed, the estimation process is not performed or the estimation process is performed and the result is obtained. At the same time, information indicating that sufficient estimation cannot be performed may be presented to the user.

  As an example, the image processing apparatus 1 according to the present embodiment receives input of image data to be processed as the processing of the receiving unit 21, and whether the image data has sufficient image quality, an optic disc or the like. Whether or not the fundus structure is photographed is determined by clustering processing. When it is determined that the image quality is sufficient and the fundus photograph can be sufficiently estimated, the input image data is received as image data of the fundus photograph and output to the estimation unit 22; To do.

  If it is determined that sufficient estimation cannot be performed, the receiving unit 21 outputs information indicating that estimation cannot be performed.

  Here, the image quality can be determined by measuring the S / N ratio, the ratio of the area close to white when binarized, and the like. Specifically, the S / N ratio is, for example, PSNR (Peak Signal-to-Noise Ratio), and the receiving unit 21 receives input image data and an image obtained by performing predetermined noise removal processing on the image data. The mean square error is calculated, and the common logarithm of the maximum pixel value (or 255 if the image is expressed in 256 levels from 0 to 255) divided by this mean square error (or (The specific calculation method is widely known, and a detailed description thereof is omitted here). If the mean square error is “0” or the PSNR exceeds a predetermined threshold (for example, the value of the common logarithm is equal to or greater than 0.8), the receiving unit 21 determines the image quality related to the S / N ratio. Is determined to be sufficient.

  Further, the ratio of the area close to white when binarized occupies the entire area, that is, the ratio of the pixels whose luminance is excessively high to the entire area. The receiving unit 21 determines the input image data as The pixel value is converted to a gray scale by a known method, and the pixel value is larger (closer to white) than the threshold value, with a point that is α times the maximum pixel value (0 <α <1) as a threshold value. The pixel value of the pixel is set to the maximum pixel value (white). At this time, by setting the value of α to a value relatively close to 1, for example, a value larger than 0.95, only the region close to white is set to white. Further, the pixel value of the pixel having a pixel value lower than the threshold value is set to the lowest pixel value (black). The receiving unit 21 divides the number of pixels set to white included in the binarization result by the number of pixels of the entire image data, and obtains the ratio of the area close to white to the whole. The receiving unit 21 determines that the image quality is sufficient when the value of the ratio obtained here is below a predetermined threshold value.

  The receiving unit 21 may execute the above-described processing after performing correction for normalizing the color tone of the input image data. Here, normalization is performed by, for example, a pixel value closest to the maximum pixel value (white) when converted to gray scale (using a known method) as the maximum pixel value, and a pixel value closest to the minimum pixel value (black) as the minimum. This is done by converting the pixel values so as to correspond to the pixel values. Since this color correction method is widely known, detailed description thereof is omitted.

  Whether or not the fundus structure such as the optic nerve head has been photographed can be determined based on, for example, whether or not an image of the optic nerve head is included in the image data. Specifically, the receiving unit 21 performs a contour extraction process on the input image data, and then detects a circle from the extracted contour image using a method such as Hough transform. Here, well-known methods can be adopted for the contour line extraction and circle detection processing.

  The receiving unit 21 checks whether the number and size of the detected circles satisfy a predetermined condition. Specifically, the receiving unit 21 has a number of detected circles of “1” (only a circle considered to be an optic nerve head), and a size (for example, a short side of a circumscribed rectangle, that is, a short axis of the circle) is a predetermined value. When it is within the range of values, it is determined that an image of the optic disc is included. Alternatively, the receiving unit 21 has the number of detected circles “2” (a circle that is considered to be an optic disc and a circle that is considered to be a boundary line of the entire visual field), and among the detected circles, a relatively small circle is compared. The size of a circle (for example, the short side of a circumscribed rectangle) that is contained within a relatively large circle and that has a relatively small circle size (for example, the short side of the circumscribed rectangle, that is, the minor axis of the circle), That is, it is determined that an image of the optic disc is included (a fundus structure such as the optic disc is photographed) when it is within a range of a predetermined ratio with respect to the minor axis of the circle.

  Alternatively, the receiving unit 21 may determine whether or not a fundus structure such as an optic nerve head has been imaged based on whether or not a blood vessel image can be detected from the input image data. For example, Kazuaki Sugio et al., “Research on blood vessel analysis in fundus photos-Extraction of blood vessels and their intersections”, Journal of Medical Image Information Society, Vol.16, No.3 (1999) Can be adopted. In the present embodiment, the accepting unit 21 attempts to extract a blood vessel image from image data input by a widely known method such as the above method. As a result of the extraction, the ratio of the number of significant pixels included in the obtained image (the number of pixels determined to be blood vessel images) to the total number of pixels is within a predetermined value range. At some point, it is determined that a blood vessel can be detected, and it is estimated that a fundus structure such as an optic nerve head is captured.

  Further, the receiving unit 21 may use a neural network that has been machine-learned so as to determine whether or not a fundus structure such as an optic nerve head has been imaged (hereinafter referred to as a prior determination neural network). As an example, such a prior determination neural network is realized by using a CNN (convolution network), a residual network, or the like, and image data of a fundus photograph in which a fundus structure such as an optic nerve head is photographed. A plurality of pieces of image data (for example, image data that is not a fundus photograph) in which the fundus structure such as the optic disc is not photographed are input, and when the fundus structure such as the optic disc is photographed, the fundus structure such as the optic disc is photographed. If the fundus structure such as the optic nerve head is not photographed, supervised machine learning is performed so that the fundus structure such as the optic nerve head is not photographed (such a machine Since the learning process can adopt a widely known method, detailed description is omitted). The receiving unit 21 converts the input image data into data that can be input to the pre-determining neural network (changes the size, etc.), and inputs the converted data to the pre-determining neural network. With reference to the output, when the output is an output indicating that the fundus structure such as the optic nerve head is photographed, the input image data may be determined to be a fundus photograph and accepted.

  In addition, here, in addition to the neural network used by the estimation unit 22, an example in which a prior determination neural network is used separately has been described as an example, but the neural network used by the estimation unit 22 may also serve as the prior determination neural network.

  In this case, the neural network used by the estimation unit 22 inputs a plurality of image data that are known in advance as fundus photographs, and diagnoses that each of the image data has a predetermined symptom (for example, glaucoma). It is assumed that machine learning is performed using teacher data such as a ratio of doctors and the like and a probability of not having a predetermined symptom (for example, a ratio of doctors diagnosed as not having glaucoma) as teacher data. In this way, the neural network has a probability that the image data is a fundus photograph of the eye with respect to the input image data, and a probability that the image is not in the predetermined symptom. Both probabilities are estimated.

  In this example, the accepting unit 21 outputs the input image data as it is to the estimating unit 22, and the probability Pp that there is a predetermined symptom, which is information output by the estimating unit 22, and the predetermined symptom are not present. The probability Pn of the effect is referred to, and when these probabilities Pp and Pn are both below a predetermined threshold (for example, both are less than 40%), or the absolute value | Pp of the difference between these probabilities -Pn | is less than a predetermined threshold value (that is, the difference between these probabilities Pp and Pn is smaller than the predetermined threshold value). It is good also as judging and outputting the information to that effect.

  Further, when the predetermined condition is not satisfied (when it can be determined that the estimation can be performed), the output of the estimation unit 22 may be output to the output unit 23.

  Further, the receiving unit 21 may use a combination of these determinations. For example, the receiving unit 21 checks the S / N ratio of the input image data, and when it is determined that the S / N ratio is larger than a predetermined value (with relatively little noise), the image data is further reduced. Try to extract the optic nerve head from among them. The receiving unit 21 may output the image data to the estimating unit 22 when it is determined that the optic nerve head has been extracted.

  As described above, in the example of the present embodiment, before outputting the estimation by the estimation unit 22, it is determined whether the estimation unit 22 has received image data that can be sufficiently estimated, and it can be determined that sufficient estimation can be performed. In this case, since the estimation result by the estimation unit 22 is output, the estimation result for image data that cannot be sufficiently estimated is not output.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 11 Control part, 12 Storage part, 13 Operation part, 14 Display part, 15 Input / output part, 21 Reception part, 22 Estimation part, 23 Output part

Claims (6)

Using learning data including information that correlates image data of fundus photos and information about eye symptoms corresponding to each fundus photo, the relationship between image data of fundus photos and information about eye symptoms is machine Holding means for holding machine learning results in a learned state;
A receiving means for receiving image data of a fundus photograph to be processed;
Using the input data based on the received image data and the machine learning result, estimating means for estimating information about the eye symptom about the eye related to the fundus photograph that has been processed;
Means for outputting the result of the estimation;
An image processing apparatus. The image processing apparatus according to claim 1,
The estimation unit specifies a range of an image corresponding to the optic nerve head from the image data received by the receiving unit, and includes partial image data that is a part of the received image data including the specified image range. An image processing apparatus that extracts information and estimates information about the eye symptom about an eye related to a fundus photograph that is a target of processing using the extracted partial image data and the machine learning result. The image processing apparatus according to claim 2,
The estimation means is an image processing apparatus that extracts the partial image data so that the specified image range is a predetermined position in the image. The image processing apparatus according to any one of claims 1 to 3,
Information about eye symptoms
An image processing apparatus, which is information of a curve representing a optic papular depression edge in a fundus photograph to be processed. The image processing apparatus according to any one of claims 1 to 4, wherein:
The receiving means determines whether or not a predetermined condition is satisfied with respect to input image data, and accepts the image data satisfying the condition as image data of a fundus photograph to be processed. Computer
Using learning data including information that correlates image data of fundus photos and information about eye symptoms corresponding to each fundus photo, the relationship between image data of fundus photos and information about eye symptoms is machine Holding means for holding machine learning results in a learned state;
A receiving means for receiving image data of a fundus photograph to be processed;
Using the input data based on the received image data and the machine learning result, estimating means for estimating information about the eye symptom about the eye related to the fundus photograph that has been processed;
Means for outputting the result of the estimation;
Program to function as.

Images