KR102533962B1 - System and method for constructing eye-related database - Google Patents

Abstract

Embodiments include: a data acquisition unit receiving a medical report of a subject in the form of an image from a photographing device that photographs an eyeball, wherein the medical report includes a photographed image, report items, and measurement results for the report items; a data reader configured to set a plurality of recognition areas from which letters or numbers are to be extracted from the medical report, recognize letters or numbers included in each of the plurality of recognition areas, and extract text data or number data; A fundus-related database construction system and method including a database management unit generating a fundus data set of a subject composed of extracted character data or numeric data and report items related to a recognition region.

Description

Eye-related database construction system and method {SYSTEM AND METHOD FOR CONSTRUCTING EYE-RELATED DATABASE}

Embodiments of the present invention relate to a system for building an eye-related database, and more particularly, to a system for building an eye-related database that extracts letters or numbers from a medical report in the form of an image to automatically form a fundus data set.

A fundus image is an image obtained by measuring the thickness seen in the retina in ophthalmology, and is used to observe various eye-related diseases, such as cataract, glaucoma, retinal complications caused by hypertension and diabetes, and other retinal diseases. Therefore, database creation of reports containing fundus images is very important in eye-related research.

1A and 1B are diagrams illustrating reports including fundus images and measurement values in a conventional viewer format.

Referring to FIGS. 1A and 1B , the commonly used fundus image capture device of Heidelberg or Zeiss does not provide measurement values and measurement results as numeric data or text data, but supports them in a viewer format such as a PDF file, It takes a lot of manpower and time to convert measurement values and measurement results into text data or numeric data and build a database.

Patent Publication No. 10-2017-0051227 (2015.10.02.)

According to one aspect of the present invention, it is possible to provide an eye-related database construction system that automatically forms an eye fundus data set of a subject by extracting character or numeric data from a medical report in the form of an image.

In the embodiments of the present invention, in one aspect, a data acquisition unit receiving a medical report of the subject in the form of an image from a photographing device that has photographed the subject's eyeball - the medical report is a photographed image, a medical report of the subject Include report items and eye-related measurement results; a data reader configured to set a plurality of recognition areas from which letters or numbers are to be extracted from the medical report, recognize letters or numbers included in each of the plurality of recognition areas, and extract text data or number data; and a database management unit generating an ocular fundus data set of a subject composed of extracted text data or numeric data and report items associated with a recognition region.

In one embodiment, the medical report may further include at least one of manufacturer information of the photographing device, identification information of the photographing device, and report type information. The data reading unit loads a template corresponding to the received medical report based on at least one of a manufacturer of the photographing device, device information, and report type information, and sets a recognition area for each report item through the template. may be configured.

In one embodiment, the data reading unit recognizes the manufacturer of the photographing device in the medical report through a pre-learned first recognition model, and the first recognition model extracts features from an input image to determine the manufacturer of the input image. It may also include an artificial neural network that has been trained to recognize.

In one embodiment, the data reading unit may be configured to extract patches for each set recognition area, recognize letters or numbers included in the extracted patches, and extract text data or numeric data for a report item related to the recognition area. there is.

In one embodiment, the data reading unit applies a result of recognizing letters or numbers included in the extracted patch to a pre-learned second recognition model to generate a report related to the recognition area to be used to generate the fundus data set. It may be further configured to extract character data or numeric data for the item. The second recognition model includes an artificial neural network trained to recognize the corresponding letter or number by extracting features from an input image including the letter or number.

In one embodiment, the second recognition model may include a 2-1 model for character string recognition and a 2-2 model for number string recognition. Also, the data reading unit may input an input patch including characters to the 2-1 model and input an input patch including numbers to the 2-2 model.

In one embodiment, the data reading unit may be further configured to set an image display area in the medical report using the template, and extract a patch formed of the corresponding area as a captured image patch. The captured image includes a first fundus image taken centered on the macula, a second fundus image taken centered on the optic nerve head, an OPTICAL COHERENCE TOMOGRAPHY (OCT) image, or a retinal layer image, and the medical Some of the report forms have captured images located in different positions from other forms.

In one embodiment, the report type is a report of a first type including ETDRS measurement results obtained by imaging the macular area, and a second type report including ETDRS measurement results obtained by imaging the nerve head area. , or a third type of report including OCT images or retinal layer images.

In one embodiment, the fundus data set may include identification information of the subject and attribute information of the medical report including the measurement date. In addition, the database management unit may be further configured to search the ocular fundus data set of the subject based on at least one of input identification information and measurement date of the subject.

In one embodiment, when receiving a new medical report of the subject, the data reading unit responds to receiving the new medical report to an item of identification information of the subject to update a measurement result of the new medical report. It may be further configured to retrieve the fundus data set by extracting identification information of the subject from an associated recognition region.

In embodiments of the present invention, in another aspect, in the fundus-related database construction method performed by a computing device including a processor and a memory, receiving a medical report of a subject in the form of an image from a photographing device that photographs an eyeball step - the medical report includes a photographed image, report items, and measurement results for the report items; setting a plurality of recognition areas to extract letters or numbers from the medical report, recognizing letters or numbers included in each of the plurality of recognition areas, and extracting the letters or numbers as text data or numeric data; and generating an ocular fundus data set of a subject composed of extracted text data or numeric data and report items associated with a recognition region.

A fundus-related database construction method performed by a computing device including a processor and memory includes: receiving a medical report of a subject in the form of an image from a photographing device that photographs an eyeball - the medical report includes a photographed image, report items, and the report contains measurement results for items; setting a plurality of recognition areas to extract letters or numbers from the medical report, recognizing letters or numbers included in each of the plurality of recognition areas, and extracting the letters or numbers as text data or numeric data; and generating an ocular fundus data set of the subject composed of the extracted text data or numeric data and report items related to the recognition region.

In one embodiment, the medical report may further include at least one of manufacturer information of the photographing device, identification information of the photographing device, and report type information. The extracting of the text data or numeric data may include loading a template corresponding to the received medical report based on at least one of a manufacturer of the photographing device, device information, and report type information; and setting a recognition area for each report item through the template.

In one embodiment, the step of extracting the text data or numeric data may further include recognizing a manufacturer of the photographing device from the medical report through a first recognition model trained in advance to load the template. there is. The first recognition model may include an artificial neural network trained to recognize a manufacturer of the input image by extracting features from the input image.

In an embodiment, the extracting of the text data or numeric data may include extracting patches for each set recognition area; and recognizing letters or numbers included in the extracted patch and extracting them as text data or numeric data for a report item related to the recognition area.

In one embodiment, the extracting of the character data or number data may include applying a result of recognizing the character or number included in the extracted patch to a pre-learned second recognition model to be used to generate the fundus data set. , extracting text data or numeric data for a report item related to the recognition area. The second recognition model includes an artificial neural network trained to recognize the corresponding letter or number by extracting features from an input image including the letter or number.

In one embodiment, the second recognition model may include a 2-1 model for character string recognition and a 2-2 model for number string recognition. Also, the data reading unit may input an input patch including characters to the 2-1 model and input an input patch including numbers to the 2-2 model.

In one embodiment, the extracting of the text data or numeric data may include setting an image display area in the medical report using the template; and extracting a patch comprising the corresponding region as a captured image patch.

In one embodiment, the method may further include: receiving a user request and searching for an ocular fundus data set matching the user request. The fundus data set includes attribute information of the medical report including identification information of the subject and measurement date, and the user request may include at least one of identification information and measurement date of the subject to be searched.

In one embodiment, the method further comprises: receiving a new medical report of the subject; and in order to update a measurement result of the new medical report, in response to receiving the new medical report, the subject's identification information is extracted from a recognition area associated with the subject's identification information item to retrieve the fundus data set. It may further include steps to do.

The construction system according to an embodiment of the present invention may form a fundus data set of a subject by automatically recognizing and extracting text and fundus images from a medical report in the form of an image.

Such a subject's fundus data set can be stored in a database in the form of a medical file, enabling easier lesion prediction and follow-up.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS To describe the technical solutions of the embodiments of the present invention or the prior art more clearly, drawings required in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for the purpose of explaining the embodiments of the present specification and not for limiting purposes. In addition, for clarity of explanation, some elements applied with various modifications, such as exaggeration and omission, may be shown in the drawings below.
1A and 1B are diagrams illustrating reports including fundus images and measurement values in a conventional viewer format.
2 is a conceptual diagram of a fundus-related database construction system according to an embodiment of the present invention.
3A to 3C are diagrams illustrating recognition areas for each medical report according to a template, according to an embodiment of the present invention.
4A and 4B are diagrams illustrating results of extracting numeric data according to an embodiment of the present invention.
5 is a diagram illustrating an fundus data set for each subject constituting an fundus-related database according to an embodiment of the present invention.
6 is a flowchart of a fundus-related database construction method according to an embodiment of the present invention.

The terminology used herein is only for referring to specific embodiments and is not intended to limit the present invention. As used herein, the singular forms also include the plural forms unless the phrases clearly indicate the opposite. The meaning of "comprising" as used herein specifies particular characteristics, regions, integers, steps, operations, elements and/or components, and the presence or absence of other characteristics, regions, integers, steps, operations, elements and/or components. Additions are not excluded.

Although not defined differently, all terms including technical terms and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with related technical literature and currently disclosed content, and are not interpreted in ideal or very formal meanings unless defined.

Hereinafter, with reference to the drawings, look at the embodiments of the present invention in detail.

2 is a conceptual diagram of a fundus-related database construction system according to an embodiment of the present invention.

Referring to FIG. 2 , system 1 for constructing a fundus-related database includes a data acquisition unit 100; data reading unit 300; and a database management unit 500 .

The system 1 according to embodiments may have aspects that are entirely hardware, entirely software, or partly hardware and partly software. For example, the system may collectively refer to hardware equipped with data processing capability and operating software for driving the hardware. In this specification, terms such as "unit", "module", "device", or "system" are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device including a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), or another processor. Also, software may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

The data acquisition unit 100 is a component that receives a medical report containing information about the eye condition of the subject. The medical report is a report including a photographed image of a subject's eyeball and measurement results, and is data in the form of an image. The fundus-related measurement result is information expressed as a measurement value and/or text, and is associated with each report item. In the medical report, letters meaning report items may be recorded. The measurement result expressed as a character may include, for example, normal, out of normal, border line, and the like.

As shown in FIGS. 1A and 1B , the photographed image included in the medical report may include a macular-centered fundus image, an optic disc-centered fundus image, an OCT image, or a retinal layer image.

The medical report includes various measurement results of measuring the fundus state, which is substantial information related to the subject's eyeball. The measurement results are expressed in numbers and/or letters.

In one embodiment, the medical report may have measurement values including one or more of ETDRS measurement values, retinal thickness, and volume. Also, the medical report may include measurement values for the left eye (OS) and the right eye (OD) for the same measurement item.

For example, a medical report with OCT images taken, Average RNFL thickenss Eye, OCT temporal RNFL thickness Eye, OCT superotemporal RNFL thickness Eye, OCT superior RNFL thickness Eye, OCT superonasal RNFL thickness Eye, OCT nasal RNFL thickness Eye, OCT inferonasal It may include one or more of RNFL thickness Eye, OCT inferior RNFL thickness Eye, OCT inferotemporal RNFL thickness Eye, and OCT RNFL classification.

The medical report includes information other than substantive information related to the eye of the subject. In one embodiment, the other information includes attribute information describing the report in a formal aspect. The attribute information may include, for example, manufacturer information of the photographing device, device information, subject identification information, report type, measurement date, etc., but is not limited thereto.

Manufacturer information of the photographing device may include a manufacturer's logo and/or text indicating the manufacturer's name. An image of the manufacturer's logo may be included in the medical report image.

The photographing device information may include, for example, a device (or product) name, device number, and the like. The manufacturer information is information of a manufacturer of the corresponding photographing device. In one embodiment, manufacturer information may include at least one of Heidelberg and Zeiss.

A report type is a protocol that constitutes a medical report. For example, according to the report type, medical reports may be classified as RNFL reports, Macular reports, and Thickness Map single Exam reports.

Also, the report type may be further classified based on the shooting part. For example, a report of the first type including the ETDRS measurement result obtained by imaging the macular portion, a report of the second type including the ETDRS measurement result obtained by imaging the nerve head portion, or an OCT image or retina It may also be classified as a third type of report including layer images. A first type of report may include a fundus image centered on the macula. A report of the second type may include a fundus image centered on a portion of the nerve head.

The identification information of the subject may include a name of the subject, a code (eg, ID) of the subject, date of birth, gender, and the like.

In one embodiment, the data acquisition unit 100 may receive a medical report from a photographing device (not shown) that photographs an eyeball and generates a medical report in a viewer format. The data acquisition unit 100 acquires viewer data (ie, image data) of the medical report through the interface unit 110 and/or the data communication unit 130.

The interface unit 110 may transmit/receive analog signals and digital data with imaging devices having various interface standards. The interface unit 110 transmits medical report data of the subject using one or more of, for example, a serial interface, a parallel interface, a transmission control protocol/internet protocol (TCP/IP) interface, and a medical digital image and communication standard (DICOM) interface. Acquire

The data communication unit 130 is a component that exchanges data with an external device capable of communication. The data communication unit 130 may include a wired and/or wireless network module. The wired network module may include a wired LAN, Ethernet, USB, power line communication (PLC), and the like, and the wireless network module may include a wireless LAN, infrared communication (IrDA), and code division multiple access (CDMA). ), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Ubiquitous Sensor Network (USN), RFID, NFC, ZigBee, Bluetooth, LTE, WiBro, High Speed Downlink Packet Access (HSDPA), Broadband Code Division It may include multiple access (WCDMA), ultra-wideband communication (UWB), and the like. The data communication unit 130 obtains medical report data from a communicable photographing device through electrical communication, or obtains medical report data pre-stored by an external device (eg, server, user terminal, computer, etc.) other than the photographing device through electronic communication. may also be obtained.

The medical report obtained by the data acquisition unit 100 is supplied to the data reading unit 300 .

The data reading unit 300 may recognize characters included in the image of the medical report and output them as text data. Also, the data reading unit 300 may recognize numbers included in the image of the medical report and output them as numeric data. The data reading unit 300 may acquire character data (ie, text data) and numeric data by recognizing characters or numbers included in the image through an Optical Character Recognition (OCR) algorithm.

Also, the data reading unit 300 may set an area (hereinafter referred to as “recognition area”) to recognize characters (or numbers) in the medical report. In some embodiments, the operation of setting the recognition area is performed before the operation of extracting text data (or numeric data).

In one embodiment, the data reading unit 300 may set a recognition area using a pre-stored template.

The template is a structure in which the location and/or shape of a recognition area in which characters are to be extracted from an image of an obtainable medical report is structured. Report items related to text or displayed images in a medical report are structured through templates. For example, a character (eg, a measurement value) recorded at a specific position in an image of a medical report on a template is related to a specific report item (eg, the corresponding measurement item).

In the medical report, an area in which the manufacturer, photographing device, report type, subject identification information, or measurement value is recorded is pre-specified according to the medical report format. The format of the medical report depends on the manufacturer of the imaging device (or device information), the imaging device, and the type of report. For example, even if it is the same type of report, the recorded location of the measurement value, etc. may be different depending on the manufacturer of the photographing device, Heidelberg company, or Zeiss sign.

The system 1 stores various templates according to various forms of medical reports in advance. The system 1 acquires the manufacturer of the imaging device, device information, and/or report type information from the image of the medical report (e.g., by the data reader 300), and obtains the manufacturer, device information, and/or report type information of the imaging device. Alternatively, a template corresponding to the medical report is loaded based on the report type.

In one embodiment, the data reading unit 300 may obtain manufacturer information of the photographing device from the medical report through the pre-learned first recognition model.

The first recognition model includes an artificial neural network trained to recognize a manufacturer, device information, and report type by extracting features from an input image and determining which manufacturer, device, or report type the input image belongs to. It is a machine learning model that The first recognition model may have, for example, a structure of a CNN-based artificial neural network, but is not limited thereto.

The first recognition model is learned using a training data set consisting of a plurality of training data. Each training data in this set may include a shape image indicating a manufacturer of the photographing device and label data. For example, each training data may include a logo image of Heidelberg or Zeiss.

The data reading unit 300 is used to recognize a manufacturer from the manufacturer logo in the image of the medical report through the first recognition model and to load a template corresponding to the medical report related to the recognized manufacturer.

In some embodiments, the data reading unit 300 may generate a patch consisting of a specific recognition area in the image of the medical report to obtain manufacturer information. The data reading unit 300 may crop the recognition area to generate a patch of the recognition area.

Then, the data reading unit 300 may apply the generated patch as an input image to the first recognition model instead of the entire area of the medical report. The patch consists of a recognition area at a location where a manufacturer's logo is statistically frequently displayed. For example, the data reading unit 300 may extract left/right upper/lower parts from the entire image of the medical report as patches and input the extracted patches to recognize the manufacturer of the imaging device that generated the image of the medical report. there is.

In addition, the data reading unit 300 may obtain the device identifier and/or report type information by extracting a device identifier (eg, device name) and/or report type text recorded in a specific recognition area of the medical report. The specific recognition area is a recognition area of a location where a device identifier and a report type are statistically frequently displayed.

For example, the data reading unit 300 recognizes some or all of the edges of the entire area, such as the left/right top/bottom of the medical report, through an OCR algorithm, and extracts text to obtain device identifier and/or report type information. Acquire The obtained device identifier and/or report type is used to load a template corresponding to the medical report.

Then, the data reading unit 300 loads a template corresponding to the obtained medical report based on at least one of manufacturer information obtained through image recognition and device identifier and/or report type information obtained through text extraction. .

The data reader 300 sets a recognition area on the image of the medical report using the template loaded in this way. Character or number data is extracted from the character or number image recorded in the set recognition area.

3A to 3C are diagrams illustrating recognition areas for each medical report according to a template, according to an embodiment of the present invention.

The data acquisition unit 300 recognizes Heidelberg's logo by inputting a patch including a specific part (eg, top) of the medical report image to the first image recognition model, and acquires manufacturer information. In addition, the data acquisition unit 300 may acquire the report type and the name of the photographing device by performing OCR processing on a specific part (eg, upper portion) of the medical report image. The data acquisition unit 300 may load a template based on the acquired manufacturer information, report type, and name of the photographing device, and may set a recognition area as shown in FIGS. 3A to 3C .

Report items structured as templates of FIGS. 3A to 3C include property information items as shown in Table 1 below.

The template of FIG. 3A includes measurement result items as shown in Table 2 below.

Alternatively, the data reading unit 300 may load a template different from that of FIG. 3A even though the same manufacturer information is acquired. When at least one of the report type and the photographing device is different, a template different from that of FIG. 3A may be loaded as shown in FIG. 3B.

Using the template of Fig. 3b, the template position, namely the recognition area, and the item are related to each other as follows.

This use of a template associates an item with a location on the medical report, i.e., a recognition area. As a result, when data is extracted from the recognition area, items related to the extracted data can be known. And, as shown in FIGS. 3A and 3B , measurement values are related to different measurement items according to a loaded template.

Also, the data reading unit 300 may extract a captured image patch included in the medical report by using a template. The position of the image is fixed according to the format of the medical report. Some forms of multiple medical report forms may be configured such that a photographed image is positioned at a location different from other forms. For example, a report of the first type including the ETDRS measurement results obtained by imaging the macular area, a report of the second type including the results of the ETDRS measurement obtained by imaging the nerve head area, or an OCT image or retinal layer image In the third type of report including , the photographed images may be located in different locations.

As shown in FIGS. 3A to 3C , the data reading unit 300 may set the image display area structured in the template as an image recognition area and extract a patch made of the corresponding area.

Through this, the system 1 does not use the entire image of the medical report to construct the fundus-related database, but uses only the portion used for prediction of the lesion and follow-up observation of the fundus disease, so that storage space can be efficiently utilized.

In this way, the data reader 300 recognizes report item data, numeric data for the item, or numbers representing item information such as letters representing report items and measurement values through an OCR algorithm in a recognition area set using a template. Get text data. Items related to the obtained numeric or text data are determined through the template.

In addition, the data reading unit 300 further performs a reading operation other than simple OCR processing, so that numbers or characters can be more accurately read compared to the case of directly extracting a recognition result (ie, simple OCR processing result) as numeric or text data. Recognizable.

In one embodiment, the data reading unit 300 obtains report item data and/or numeric data or text data of a corresponding item through a pre-learned second recognition model. A result of recognizing characters or numbers in the recognition area (ie, OCR-processed data) is input to the second recognition model. That is, data primarily recognized through the OCR algorithm through the second recognition model is re-recognized.

The second recognition model is a machine learning model including an artificial neural network trained to extract features from an input image including letters or numbers and recognize corresponding letters or numbers. The second recognition model may have, for example, a structure of a CNN-based artificial neural network, but is not limited thereto.

The data reading unit 300 crops the recognition area set using the template to generate a recognition area patch, inputs the recognition area patch to the recognition model, and converts characters or numbers displayed in the recognition area into character data or text data. Extract as numeric data.

In one embodiment, the second recognition model may include a 2-1 model for character string recognition and a 2-2 model for number string recognition. The data reading unit 300 may determine whether the language type in the patch is a character or a number through OCR processing. When the input patch includes characters, the data reading unit 300 inputs the input patch to the 2-1 model. On the other hand, when the input patch includes a number, the data reading unit 300 inputs the input patch to the 2-2 model. The 2-1 model and the 2-2 model are specialized in recognizing the corresponding text type. As a result, System 1 using the second recognition model has a high recognition rate even for a single character, and misrecognizes a number or character at a low resolution (eg, misrecognizing a real 7 as a 1 or misrecognizing a 2 as a 7). It does not misrecognize numbers as character string data (for example, misrecognizing 99 as gg or misrecognizing 97 as ar).

The second recognition model is learned using a training data set. The training data set may be subsetted by number or letter. For example, the training data set may be sorted into 250 subsets by number and by letter. The training data in the set includes training image data and label data of the corresponding letter or number.

4A and 4B are diagrams illustrating results of extracting numeric data according to an embodiment of the present invention.

4A is a result of extracting data from the medical report of FIG. 3A, and FIG. 4B is a result of extracting data from the medical report of FIG. 3B.

Referring to FIG. 4A , the data readout unit 300 obtains numeric data representing measurement values recorded in each recognition area by inputting patches of recognition areas to the second recognition model.

Also, referring to FIG. 4B , the data reading unit 300 obtains each number or character data recorded in each recognition area by inputting the patch of the recognition area to the second recognition model. The data reading unit 300 may associate numbers or letters extracted based on the structure of the template with corresponding items.

The database management unit 500 forms an ocular fundus data set of the subject consisting of recognized numeric or text data and report items related thereto.

Recognized numeric or character data means the attribute information and the measurement result, and are respectively associated with report items. Accordingly, the fundus data set includes the identification information of the subject and the measurement date.

5 is a diagram illustrating an fundus data set for each subject constituting an fundus-related database according to an embodiment of the present invention.

Referring to FIG. 5 , the database manager 500 may build a database by forming fundus data sets for each subject. In one embodiment, the system 1 may further include a database storage (not shown) to build a database.

The database management unit 500 forms ocular fundus data sets for a plurality of subjects.

In one embodiment, the database management unit 500 may form a medical file consisting of fundus data sets of each of a plurality of subjects. The medical file may be a csv format file.

In addition, the database management unit 500 may search the fundus data set of the subject from the constructed database. In one embodiment, the database manager may search the ocular fundus data set of the subject based on at least one of input identification information and measurement date of the subject.

In addition, when receiving a new medical report of a subject, the database management unit 500 may extract data from the new medical report and update the fundus data set of the subject.

In one embodiment, the retrieval of the fundus data set may be performed in response to receipt of a new medical report. The data reading unit 300 recognizes and extracts data (eg, identification information of a subject) to be used for search in a new medical report. The data reading unit 300 uses data extracted from a recognition area associated with an item of identification information of a subject in a template for a search. Then, whenever a new medical report is input, the fundus data set of the subject is automatically updated.

It will be apparent to those skilled in the art that the system 1 may include other components not described herein. For example, the system 1 may include a network interface, an input device for data entry, a memory for storing data, and an output device for display, printing, or other presentation of data necessary for the operations described herein. It may also contain other hardware elements.

A fundus-related database construction method according to another aspect of the present invention may be performed by a computing device including a processor. For example, the fundus-related database construction method may be performed by at least some components of an eyeball-related database construction system.

6 is a flowchart of a fundus-related database construction method according to an embodiment of the present invention.

Referring to FIG. 6 , the fundus-related database construction method includes receiving a medical report of the subject in the form of an image from a photographing device that has photographed the eyeball (S610). The medical report includes a photographed image, report items, and measurement results for the report items.

Since the operation of step S610 is similar to that of the data acquisition unit 100, a detailed description thereof will be omitted.

The fundus-related database construction method comprises setting a plurality of recognition areas from which letters or numbers are to be extracted from the medical report after step S610, and recognizing the letters or numbers included in each of the plurality of recognition areas to character data or numeric data. and extracting to (S630).

The step S630 includes at least a part of the operation steps of the data reading unit 300.

In one embodiment, the step (S630) may include loading a template corresponding to the received medical report based on at least one of a manufacturer of the photographing device, device information, and report type information; and setting a recognition area for each report item through the template.

In addition, the step (S630) may further include recognizing the manufacturer of the imaging device in the medical report through a pre-learned first recognition model to load the template. The first recognition model includes an artificial neural network trained to recognize a manufacturer of the input image by extracting features from the input image.

In an embodiment, the step (S630) may include extracting patches for each set recognition area; and recognizing letters or numbers included in the extracted patch and extracting them as text data or numeric data for a report item related to the recognition area. A recognition area for extracting a patch is preset through a template.

In addition, in the step (S630), a result of recognizing letters or numbers included in the extracted patch is applied to a pre-learned second recognition model to generate a report item related to the recognition region to be used to generate the fundus data set. The method may further include extracting text data or numeric data. The second recognition model includes an artificial neural network trained to recognize the corresponding letter or number by extracting features from an input image including the letter or number.

Since the operation steps of the data reading unit 300 that can be included in the step S630 have been described in detail above, a detailed description thereof will be omitted.

After step S630, the fundus-related database construction method includes: generating a fundus data set of the subject including the extracted text data or numeric data and report items related to the recognition region (S650).

In addition, the fundus-related database construction method may further include receiving a user request and searching for a fundus data set of a subject matching the user request ( S670 ). The user request may include identification information and/or measurement date of the subject. The searched fundus data set may be provided to the user.

In addition, the fundus-related database construction method may further include: updating the fundus data set of the subject when a new medical report is received (S690). A search of the fundus data set may be performed for update.

The steps S650 to S690 include at least some of the operation steps of the database management unit 500.

In one embodiment, step S670 may be performed in response to receiving the medical report. Whenever a new medical report is received, a fundus data set of a corresponding subject may be automatically searched and contents of the new medical report may be updated.

In the above embodiment, when a new medical report is received (S610), character or numeric data for the new medical report is extracted (S630). In step S630, identification information of the subject is recognized.

As a result of the search based on the subject identification information, when there is no ocular fundus data set for the subject in the new medical report, step S650 is performed.

On the other hand, as a search result based on the identification information of the subject, when the fundus data set for the subject of the new medical report is already stored, a new fundus data set is generated based on the new medical report (S650), and the fundus data set and the previously stored fundus data set An already stored fundus data set may be updated by combining a new fundus data set (S690).

The operation steps of the database management unit 500 that can be included in the steps S650 to S690 have been described in detail above, and detailed descriptions thereof will be omitted.

Operations by the fundus-related database construction system and method according to the embodiments described above may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. For example, implemented together with a program product consisting of a computer-readable medium containing program code, which may be executed by a processor to perform any or all steps, operations, or processes described.

The computer may be any device that may be integrated into or may be a computing device such as a desktop computer, laptop computer, notebook, smart phone, or the like. A computer is a device that has one or more alternative and special purpose processors, memory, storage, and networking components (whether wireless or wired). The computer may run, for example, an operating system compatible with Microsoft's Windows, Apple's OS X or iOS, a Linux distribution, or an operating system such as Google's Android OS.

The computer-readable recording medium includes all types of recording and identification devices in which data readable by a computer is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage and identification devices, and the like. In addition, computer-readable recording media may be distributed in computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention reviewed above has been described with reference to the embodiments shown in the drawings, but this is only exemplary, and those skilled in the art will understand that various modifications and variations of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Claims (20)

a data acquiring unit receiving a medical report of the subject in the form of an image from a photographing device that has photographed the subject's eyeball - the medical report includes a photographed image, report items of the subject's medical report, and eyeball-related measurement results;
a data reader configured to set a plurality of recognition areas from which letters or numbers are to be extracted from the medical report, recognize letters or numbers included in each of the plurality of recognition areas, and extract text data or number data; and
A database management unit generating a fundus data set of the subject consisting of extracted character data or numeric data and report items related to the recognition area;
The medical report further includes at least one of manufacturer information of the photographing device, identification information of the photographing device, and report type information;
The report type includes a first type report including ETDRS measurement results obtained by imaging the macula, a second type report including ETDRS measurement results obtained by imaging centered on the nerve head, and an OCT image or retina at least one of a third type of report including a layer image;
The data reading unit loads a template corresponding to the medical report based on at least one of a manufacturer, device information, and report type information of a photographing device recognized and classified through a pre-learned first recognition model, and through the template Characterized in that it is configured to set the recognition area for each report item, fundus-related database construction system.
delete The method of claim 1, wherein the data reading unit,
recognizing the manufacturer of the photographing device in the medical report through a pre-learned first recognition model;
The first recognition model comprises an artificial neural network trained to recognize a manufacturer of the input image by extracting features from the input image.
The method of claim 1, wherein the data reading unit,
Extract patches for each set recognition area,
An ocular fundus-related database construction system, characterized in that it is configured to recognize letters or numbers included in the extracted patch and extract them as text data or numeric data for a report item related to the recognition area.
The method of claim 4, wherein the data reading unit,
By applying a result of recognizing letters or numbers included in the extracted patch to a pre-learned second recognition model, text data or numeric data for a report item associated with the recognition area to be used to generate the fundus data set is extracted. It is further configured to
The second recognition model comprises an artificial neural network trained to extract features from an input image including letters or numbers and recognize the corresponding letters or numbers.
The method of claim 5, wherein the second recognition model includes a 2-1 model for character string recognition and a 2-2 model for number string recognition,
wherein the data reading unit inputs input patches including characters to the 2-1 model, and inputs input patches including numbers to the 2-2 model.
The method of claim 1, wherein the data reading unit,
setting an image display area in the medical report using the template; and
It is further configured to extract a patch consisting of the corresponding area as a photographed image patch,
The photographic image includes a first fundus image photographed centering on the macula, a second fundus image photographed centering on the optic nerve head, an OPTICAL COHERENCE TOMOGRAPHY (OCT) image, or a retinal layer image,
Fundus-related database construction system, characterized in that some of the forms of the medical report are located in a location different from other forms.
delete According to claim 1,
The fundus data set includes identification information of the subject and attribute information of the medical report including a measurement date,
The database management unit is further configured to search the fundus data set of the subject based on at least one of the input identification information and the measurement date of the subject.
According to claim 9,
When a new medical report of the subject is received, the data reading unit responds to the reception of the new medical report to update the measurement result of the new medical report in a recognition area associated with an item of identification information of the subject. The fundus-related database construction system, characterized in that it is further configured to search the fundus data set by extracting the identification information of the subject.
A fundus-related database construction method performed by a computing device including a processor and a memory,
Receiving a medical report of the subject in the form of an image from a photographing device that has photographed the eyeball - the medical report includes a photographed image, report items, and measurement results for the report items;
setting a plurality of recognition areas to extract letters or numbers from the medical report, recognizing letters or numbers included in each of the plurality of recognition areas, and extracting the letters or numbers as text data or numeric data; and
generating an ocular fundus data set of the subject consisting of the extracted character data or numeric data and report items associated with the recognition area;
The medical report further includes at least one of manufacturer information of the photographing device, identification information of the photographing device, and report type information;
The report type includes a first type report including ETDRS measurement results obtained by imaging the macula, a second type report including ETDRS measurement results obtained by imaging centered on the nerve head, and an OCT image or retina at least one of a third type of report including a layer image;
The step of extracting the text data or numeric data,
loading a template corresponding to the medical report based on at least one of a manufacturer, device information, and report type information of a photographing device recognized and classified through a pre-learned first recognition model; and setting a recognition area for each report item through the template.
delete 12. The method of claim 11, wherein the step of extracting text data or numeric data comprises:
Recognizing a manufacturer of the photographing device in the medical report through a first recognition model trained in advance to load the template;
The first recognition model comprises an artificial neural network trained to recognize a manufacturer of the input image by extracting features from the input image.
12. The method of claim 11, wherein the step of extracting text data or numeric data comprises:
extracting patches for each set recognition area; and
A fundus-related database construction method comprising recognizing letters or numbers included in the extracted patch and extracting them as text data or numeric data for a report item related to the recognition area.
15. The method of claim 14, wherein the extracting into text data or numeric data comprises:
By applying a result of recognizing letters or numbers included in the extracted patch to a pre-learned second recognition model, text data or numeric data for a report item associated with the recognition area to be used to generate the fundus data set is extracted. Including more steps to do,
The second recognition model comprises an artificial neural network trained to extract features from an input image including letters or numbers and recognize the corresponding letters or numbers.
The method of claim 15, wherein the second recognition model includes a 2-1 model for character string recognition and a 2-2 model for number string recognition,
The extracting of the character data or numeric data may include inputting an input patch including characters to the 2-1 model; and inputting an input patch including numbers into the 2-2 model.
12. The method of claim 11, wherein the step of extracting text data or numeric data comprises:
setting an image display area in the medical report using the template; and
A fundus-related database construction method, further comprising extracting a patch consisting of the corresponding region as a photographed image patch.
According to claim 11,
Further comprising receiving a user request and searching for a fundus data set of a subject matching the user request,
The fundus data set includes attribute information of the medical report including identification information and measurement date of the subject,
The user request comprises at least one of identification information and a measurement date of a subject to be searched.
According to claim 18,
receiving a new medical report of the subject; and
Searching the fundus data set by extracting identification information of the subject from a recognition area associated with an item of identification information of the subject in response to receiving the new medical report, in order to update a measurement result of the new medical report. A fundus-related database construction method further comprising steps.
A computer-readable recording medium recording a program for performing the fundus-related database construction method according to any one of claims 11 and 13 to 19.

Images