KR102604122B1 - Apparatus and method for classifying images - Google Patents

Abstract

The present disclosure relates to an image classification device and method. In particular, an image classification device capable of generating standardized data from images by classifying images according to standard types and extracting text using keywords and coordinates in medical image images. A method can be provided. In addition, it is possible to provide an image classification device and method that can generate standardized data that can be used as big data by classifying, itemizing, and storing medical image images using a deep learning algorithm.

Description

Image classification apparatus and method {APPARATUS AND METHOD FOR CLASSIFYING IMAGES}

These embodiments provide an image classification apparatus and method.

Globally, respiratory diseases have been one of the leading causes of death even before the emergence of coronavirus disease 19 (COVID-19). According to a report by the World Health Organization (WHO), the five leading causes of serious lung-related diseases are chronic obstructive pulmonary disease (COPD), asthma, acute lower respiratory tract infections, tuberculosis (TB), and lung cancer, with 4 million people suffering from chronic lung disease every year. It is analyzed that premature death occurs due to respiratory disease. Moreover, due to the recent COVID-19 epidemic, many studies are being conducted to predict the occurrence of lung diseases.

Recently, the importance of big data is increasing due to developments in the field of artificial intelligence. In addition, the trend in medical services is changing from treatment-centered to prevention-centered and health care-centered, and artificial intelligence (AI) technology using medical data is developing. As cutting-edge medical technology using artificial intelligence develops rapidly in the medical field, the learning data required for the production and learning of artificial intelligence is a more important asset than ever. In particular, due to the COVID-19 epidemic, medical data obtained from tests are widely used in disease analysis to predict the occurrence of lung diseases, which requires standardized big data related to tests.

However, medical data includes a lot of data such as imaging test results, medical records, surgical records, nursing records, and blood tests, but most imaged data cannot be extracted directly from images and has the problem of making it difficult to normalize the data. . In other words, even if data is stored in an EMR (electronic medical record), it is difficult to utilize it as big data for applied research if it is stored in a non-standardized format or a format that cannot perform appropriate statistical analysis.

In this situation, manually classifying or itemizing thousands or tens of thousands of pieces of high-quality medical data for learning is very time-consuming and costly. Therefore, in order to use test data obtained from patients as big data for research on disease analysis and disease prediction, technology to extract text from medical image data is required. Specifically, there is a need for medical image processing technology to classify and itemize medical image data.

Against this background, the present embodiments provide an image classification device and method that can generate standardized data by classifying and itemizing medical images.

In order to achieve the above-described object, in one aspect, in the image classification device, a reference keyword and reference coordinates are set according to a standard type of a medical image image, and a reference keyword and reference coordinates are input based on the reference keyword and reference coordinates. Based on the image classification unit that classifies the type of image, the text conversion unit that extracts detailed images and coordinates from the classified images and converts the extracted detailed images into text using a deep learning algorithm, and the columns and rows extracted from the target file An image classification device is provided that includes a database storage unit that inputs text to create a final target file and stores the final target file in a database.

In another aspect, in the image classification method, this embodiment is an image classification step of setting reference keywords and reference coordinates according to the standard type of medical image image and classifying the type of input image based on the reference keyword and reference coordinates. , extract detailed images and coordinates from the classified images, convert the extracted detailed images into text using a deep learning algorithm, and enter text based on the columns and rows extracted from the target file to create the final target file. An image classification method is provided, which includes a database storage step of generating and storing the final target file in a database.

According to the present embodiments, an image classification device and method capable of generating standardized data by classifying and itemizing medical images can be provided.

1 is a diagram illustrating the configuration of an image classification device according to an embodiment of the present disclosure.
FIG. 2 is a flowchart illustrating an operation of generating a final target file in an image classification device according to an embodiment of the present disclosure.
Figure 3 is a flowchart for explaining the overall operation of generating a final target file in an image classification device according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating an example of a standard type of medical image used in an image classification device according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an example to explain an operation of classifying the type of an image in an image classification device according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating an example to explain the operation of preprocessing an input image in an image classification device according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating an example to explain an operation of inputting converted text in an image classification device according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating an example for explaining the structure of a database in an image classification device according to an embodiment of the present disclosure.
Figure 9 is a flowchart of an image classification method according to another embodiment of the present disclosure.

This disclosure relates to an image classification device and method.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to illustrative drawings. In adding reference numerals to components in each drawing, the same components may have the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present technical idea, the detailed description may be omitted. When “comprises,” “has,” “consists of,” etc. mentioned in the specification are used, other parts may be added unless “only” is used. When a component is expressed in the singular, it can also include the plural, unless specifically stated otherwise.

Additionally, in describing the components of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, order, or number of the components are not limited by the term.

In the description of the positional relationship of components, when two or more components are described as being “connected,” “coupled,” or “connected,” the two or more components are directly “connected,” “coupled,” or “connected.” ", but it should be understood that two or more components and other components may be further "interposed" and "connected," "combined," or "connected." Here, other components may be included in one or more of two or more components that are “connected,” “coupled,” or “connected” to each other.

In the description of temporal flow relationships related to components, operation methods, production methods, etc., for example, temporal precedence relationships such as “after”, “after”, “after”, “before”, etc. Or, when a sequential relationship is described, non-continuous cases may be included unless “immediately” or “directly” is used.

On the other hand, when a numerical value or corresponding information (e.g., level, etc.) for a component is mentioned, even if there is no separate explicit description, the numerical value or corresponding information is related to various factors (e.g., process factors, internal or external shocks, It can be interpreted as including the error range that may occur due to noise, etc.).

The optical character recognition module (Optical character reader, OCR) in this specification is used to receive handwritten or printed documents as input and convert them into data that a computer can understand. Application areas of OCR include vehicle license plates, receipts, passports, etc. This technology is applied to various fields such as recognition of checks, etc. In other words, the optical quasi-character recognition module allows you to conveniently convert images, PDF files or other types of files (if they contain text) into text in machine encoded format. For example, Tesseract OCR can be an open source based optical character recognition engine.

Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

1 is a diagram illustrating the configuration of an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 1, the image classification device 100 according to an embodiment of the present disclosure sets reference keywords and reference coordinates according to the standard type of medical image image, and selects the image input based on the reference keyword and reference coordinates. An image classification unit 110 that classifies the type, a text conversion unit 120 that extracts detailed images and coordinates from the classified images and converts the extracted detailed images into text using a deep learning algorithm, and extracts from the target file. An image classification device 100 is provided that includes a database storage unit 130 that inputs text based on the selected columns and rows to generate a final target file and stores the final target file in a database.

According to one embodiment, the image classification unit 110 may classify the type of input image according to the standard type of medical image. As an example, the image classification unit 110 may set each reference keyword and reference coordinate according to the standard type of medical image, and classify the type of input image based on the set reference keyword and reference coordinate. This is because medical imaging images, which are classified into various types, include data (text and images) in different locations depending on the type, so classifying the images by type before text conversion can provide the effect of maximizing efficiency.

For example, the image classification unit 110 can set unique words that do not overlap each other for each standard type of medical imaging image as each reference keyword, and drag the position of the set reference keyword to extract the coordinates and set them as the reference coordinates. there is. For a specific example, when setting reference keywords that do not overlap each other for each standard type of video image, the image classification unit 110 can extract the coordinates of the start and end points of the reference keyword using a mouse handler and set them as reference coordinates. there is.

For another example, if the word and the coordinates of the word included in the input image match the reference keyword and reference coordinate set for each standard type, the image classification unit 110 may classify the input image into the standard type set as the reference keyword. You can. Accordingly, the image classification unit 110 can classify the input image into the same standard type if the keyword and coordinates match.

According to one embodiment, the text conversion unit 120 may extract detailed images and coordinates from input images classified by standard type, and convert the extracted detailed images into text using a deep learning algorithm. For example, when the input image is classified as a specific type among standard types, the text conversion unit 120 repeats the process of extracting detailed images and coordinates and converting them into text until all data included in the classified image is extracted. can do.

As another example, the text conversion unit 120 may preprocess the input image to increase the accuracy of extracting detailed images and coordinates from the classified images. For example, the text converter 120 may enlarge the size of the input image by an arbitrary magnification, adjust it to a grayscale image, and then apply a whitelist to each field to preprocess the input image. This may be a method that integrates the gray scale method and the rescale method. For a specific example, the text conversion unit 120 doubles the size of the input original image and then adjusts it to a grayscale image, and then changes the characteristics of the input image, which generally includes only alphabet characters and no special characters, to The considered whitelist can be applied to each field. However, the rate at which the image is resized is not limited to doubling.

As another example, the text conversion unit 120 may convert the extracted detailed image into text in a standardized data format using a deep learning algorithm. For example, the deep learning algorithm may be an optical character recognition module (OCR) that utilizes Long Short-Term Memory (LSTM). For a specific example, LSTM may be an improved algorithm of RNN (Recurrent Neural Networks) that allows a neural network network to make decisions by simultaneously considering current input values and past input values. In addition, LSTM includes a back propagation process that corrects the error by updating each coefficient by passing the error obtained at the output stage back through the neural network, which can improve the recognition rate compared to the recognition process of RNN. For another example, the text conversion unit 120 converts an image into text in a standardized data format using CTC (Connectionist Temporal Classification), an algorithm that learns unsegmented sequence data in which only class labels are in order and the location of each class is unknown. It can be converted. However, the deep learning algorithm of the present disclosure is not limited to this and may consist of various algorithms that can be used now or in the future.

According to one embodiment, the database storage unit 130 may generate a final target file by inputting text converted based on columns and rows extracted from the target file. And the database storage unit 130 may store the generated final target file in the database. For example, the database storage unit 130 may extract columns and rows from a target file into which text is to be input and generate a final target file consisting of columns containing variables and rows containing case descriptions. Additionally, the database storage unit 130 may generate the final target file by repeating the input process based on the extracted columns and rows until all text converted from the image is input. Accordingly, the database storage unit 130 can generate a final target file in highly compatible CSV (Common-Separated Value) or Excel format and store it in the database.

As an example, the database storage unit 130 may classify items to be extracted from the input image into an Entity Relationship Model structure and store them in the database. For example, the database storage unit 130 may be composed of a table containing test results, a table containing patient information, and other test-related tables.

According to one embodiment, the image classification device may further include a verification tool that can evaluate the accuracy of the algorithm. Here, the verification tool may be a validation tool (Validator). For example, algorithms may require validation because their accuracy is not always consistent and is greatly affected by the application of various preprocessing and whitelists. Here, the algorithm explains Tesseract OCR as an example, but is not limited thereto.

FIG. 2 is a flowchart illustrating an operation of generating a final target file in an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 2, the image classification unit 110 of the image classification device according to an embodiment of the present disclosure may set reference keywords and reference coordinates for each standard type (S210). As an example, the image classification unit 110 may set reference keywords and reference coordinates according to the standard type of medical image. Here, the medical imaging image is an original PFT image file in which the results of pulmonary function tests (PFT) are recorded, and may be a PFT result image included in various types of PFT image sheets. However, the lung function test image is an example of a medical imaging image and is not limited thereto. For example, the image classification unit 110 sets a standard for the type because the location of the data (text and image) included is different depending on the type of medical image image, and sets reference keywords and reference coordinates for each set standard type. there is. For another example, the image classifier 110 may set the standard types of medical imaging images to eight types, including an unknown type. Specifically, when the medical imaging image is a PFT result image, each PFT result image may have different number of characters, size, thickness, color, and space between letters. In addition, the larger the number of characters, the thicker the characters included in the PFT result image, and the greater the color difference between the characters and the background, the higher the accuracy. Additionally, the space between characters can affect accuracy. Accordingly, the image classification unit 110 can maximize efficiency by classifying the PFT result image by standard type before algorithmic processing for text conversion.

However, the number of standard types is set according to the type of medical image, and is not limited to this.

As another example, the image classification unit 110 may set unique words that do not overlap each other for each standard type as reference keywords, and set coordinates extracted by dragging the position of the reference keyword as reference coordinates. For example, the image classifier 110 may set the standard keyword as a unique word that does not overlap with keywords set in images of other standard types among text included in the image for each standard type of medical image image. Accordingly, the image classification unit 110 may set a unique word representing each standard type as each standard keyword. For another example, the image classification unit 110 may extract the coordinates of the reference keyword by dragging the location of the reference keyword set for each standard type with a mouse handler and set it as the reference coordinate.

The image classification unit 110 of the image classification device according to one embodiment may classify the type of the input image based on the reference keyword and reference coordinates (S220). For example, the image classification unit 110 may determine whether a word and its coordinates included in any input image match the reference keyword and reference coordinates set for each standard type. And, as a result of the determination, the image classification unit 110 may classify the input image into a standard type set with a reference keyword and reference coordinates that match the words included in the input image and the coordinates of the word. Accordingly, the image classification unit 110 may classify the PFT result images included in the input PFT image sheet by standard type.

The text conversion unit 120 of the image classification device according to one embodiment may extract detailed images and coordinates from each image classified by standard type (S230). As an example, the text conversion unit 120 may perform preprocessing to extract detailed images and coordinates from the PFT result image. For example, the text converter 120 may enlarge the size of the input image by an arbitrary magnification, adjust it to a grayscale image, and then apply a whitelist to each field to preprocess the input image. For a specific example, the text converter 120 can double the size of the input image through a method that combines the rescale method and the gray scale method and then adjust it to a grayscale image. Additionally, the text conversion unit 120 may apply a whitelist to each field, taking into account the characteristics of the PFT result image, which does not include special characters and includes only alphabet characters.

As another example, the text conversion unit 230 may extract detailed images and coordinates from the preprocessed PFT result image. For example, the text converter 230 may extract a detailed image corresponding to a specific area from the preprocessed PFT result image and extract coordinates of both ends that can specify the location of the detailed image.

The text conversion unit 120 of the image classification device according to one embodiment may convert the extracted detailed image into text using a deep learning algorithm (S240). As an example, the text converter 120 may convert the extracted detailed image into text using a deep learning algorithm based on an optical character recognition module (Optical character reader, OCR) utilizing LSTM (Long Short-Term Memory). . Here, the optical character recognition module may be based on the Tesseract OCR library. For example, the text conversion unit 120 may detect and convert text included in the extracted detailed image using a deep learning algorithm and OpenCV to maximize accuracy.

As another example, the text conversion unit 120 may repeat the process of extracting detailed images and coordinates and converting them into text until all data included in the classified image is extracted. For example, the text converter 120 may extract detailed images and coordinates by moving the area of the PFT result image until all data included in the PFT result image classified according to the standard type is extracted. And the text conversion unit 120 can convert each detailed image extracted from the PFT result image into text using a deep learning algorithm.

The database storage unit 130 of the image classification device according to one embodiment may extract the columns and rows of the target file (S250). As an example, the database storage unit 130 may extract the columns and rows of the target file into which the converted text is to be input from the PFT result image. For example, the database storage unit 130 may extract columns and rows into which converted text is input based on variables or case numbers from an Excel format target file. Additionally, the database storage unit 130 may repeat the process of extracting columns and rows of the target file until all data included in the PFT result image is input.

The database storage unit 130 of the image classification device according to one embodiment may generate a final target file by inputting text based on the extracted columns and rows (S260). As an example, the database storage unit 130 may create a final target file consisting of columns containing variables and rows containing case defenses and store the final target file in the database. For example, the database storage unit 130 may input the text converted from the PFT result image into a highly compatible CSV (Common-Separated Value) or Excel format target file and store the final target file generated in the database.

As another example, the database storage unit 130 may classify items to be extracted from the input image into an Entity Relationship Model structure and store them in the database. For example, the database storage unit 130 classifies items to be extracted from the PFT result image into an entity relationship model structure consisting of a table containing PFT result information, a table containing patient information, and other PFT result information related tables. It can be saved in the database. Here, other PFT result information related tables may be tables related to at least one of spirometry, lung diffusing capacity, lung volume, and airway resistance.

Figure 3 is a flowchart for explaining the overall operation of generating a final target file in an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 3, the image classification unit 110 of the image classification device according to an embodiment of the present disclosure may read the original image file of the medical test result (S301). For example, the image classification unit 110 can read the original image capture file of the PFT test result through the OpenCV (Open Source Computer Vision) library.

According to one embodiment, the image classification unit 110 may classify the type of image (S302). For example, the image classification unit 110 may classify the PFT result image into a standard type through the Tesseract OCR library.

According to one embodiment, the image classification unit 110 may determine whether an existing folder corresponding to the type of the classified image exists (S303). For example, the image classification unit 110 may determine whether an existing folder exists in which the PFT result image can store the corresponding type.

According to one embodiment, the image classification unit 110 may create a new type of folder if an existing folder does not exist (S304). For example, the image classification unit 110 may create a new type of folder in which the PFT result image of the corresponding type can be stored.

According to one embodiment, if a folder of a classified type exists, the image classification unit 110 may store the image in the corresponding folder (S305). For example, the image classification unit 110 can move the PFT result image from its existing location to a folder of the corresponding type and store it. Here, the folder may be an existing folder or a newly created type of folder. Additionally, the image classification unit 110 can move the PFT result image to a folder of the corresponding type and set a memory pointer to create a database in real time.

According to one embodiment, the text conversion unit 120 may preprocess images classified by type (S306). For example, the text converter 120 can maximize reading accuracy by preprocessing the PFT result image by adjusting its size and grayscale.

According to one embodiment, the text converter 120 may apply a whitelist to the PFT result image (S307). For example, the text converter 120 may apply a whitelist to each field to generate an image containing only desired types of characters in the PFT result image.

According to one embodiment, the text converter 120 may extract text from the PFT result image (S308). For example, the text converter 120 can extract and convert all text from the PFT result image using a deep learning algorithm. Here, the deep learning algorithm may be Tesseract OCR, an optical character recognition module (OCR) that utilizes Long Short-Term Memory (LSTM).

According to one embodiment, the database storage unit 130 may read a target file in Excel format (S311). And the database storage unit 130 can determine whether a target file in Excel format already exists (S312). The database storage unit 130 may create a new Excel format target file if there is no existing Excel format target file (S313).

According to one embodiment, the database storage unit 130 may input and store the converted text in an Excel format target file (S309). As an example, the database storage unit 130 may repeatedly input and store the text of the PFT result image according to the columns and rows of the target file in Excel format until all text is entered.

According to one embodiment, the database storage unit 130 may store a CSV (Common-Separated Value) format file along with an Excel format target file (S310). As an example, the database storage unit 130 may separately save all text in the PFT result image as a CSV format file that stores the text by separating fields with commas.

FIG. 4 is a diagram illustrating an example of a standard type of medical image used in an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 4, the standard type set to classify medical images in the image classification unit 110 of the image classification device according to an embodiment of the present disclosure can be described. As an example, the image classification unit 110 may classify the type of image based on a set standard type before extracting text from a medical image. For example, the image classification unit 110 may set the standard type of the PFT result image to eight types, including an unknown type. At this time, the unknown type can include all types that do not contain text (ex. only graphs). Accordingly, the image classification unit 110 can classify the input PFT result image into eight types according to the set standard type. However, the standard type is set according to the type of medical image, and the type or number of standard types is not limited to this.

For another example, the image classification unit 110 may set the standard type of the PFT result image based on the number of characters, size, thickness, color, and space between characters. For example, a set standard type of medical image may consist of different numbers of extracted text data. This may be because the indicators and presence or absence of stimulants in the test items included in the PFT result image are different. Specifically, the standard types can include 116 types of type 1, 164 types of type 2, 21 types of type 3, 130 types of type 4, 37 types of type 5, 73 types of type 6, and 130 types of data. For another example, among standard types, type1 and type2 formats may be similar. Type1 and type2 generally include features such that the characters included in the PFT result image are thicker and larger than other types, and there is a space between characters. Additionally, among the standard types, the formats of type3, type4, and type7 may be similar. Type3, type4, and type7 may include features in which the characters included in the PFT result image are thin and small, and the margins between characters are narrow. Additionally, among the standard types, the formats of type5 and type6 may be similar. In types 5 and 6, the characters included in the PFT result image are generally smaller than other types, but may include features such as thick characters and a space between them.

FIG. 5 is a diagram illustrating an example to explain an operation of classifying the type of an image in an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 5 , the operation of classifying the type of image based on a reference keyword and reference coordinates in the image classification unit 110 of the image classification device according to an embodiment of the present disclosure can be described. As an example, the image classification unit 110 may set unique words that do not overlap each other for each standard type as the standard keyword 510. And the image classification unit 110 can set reference coordinates by dragging the location of the reference keyword 510 with a mouse handler. For example, the image classification unit 110 drags the mouse handler to the Y') coordinates can be extracted and set as reference coordinates. Here, Mouse Handler is an abbreviation for mouse event handler, and can refer to a module programmed to perform a specified coordinate measurement function by calling a specific function when various events or incidents related to the mouse occur. there is.

As another example, when an arbitrary image is input, the image classification unit 110 may classify the type based on reference keywords and reference coordinates set for each standard type. For example, when a random PFT result image is input, the image classification unit 110 can extract words and their coordinates included in the image using a mouse handler. And, if the extracted word and its coordinates match the reference keyword and the reference coordinate, the image classification unit 110 may classify the input PFT result image into the standard type of the reference keyword.

FIG. 6 is a diagram illustrating an example to explain the operation of preprocessing an input image in an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 6, the operation of preprocessing an image input by the text conversion unit 120 of the image classification device according to an embodiment of the present disclosure can be described. As an example, the text converter 120 may increase the reading accuracy of the deep learning algorithm by preprocessing the input image. For example, the text converter 120 may convert the input image 610 into an enlarged image 620 by enlarging the size of the input image 610 at an arbitrary magnification using a rescale method. Here, the arbitrary magnification may be 2 times, but is not limited thereto. Additionally, the text converter 120 can adjust the enlarged image 620, which is a multi-channel image, to a grayscale image 630, which is a single-channel image, through the grayscale method. Additionally, the text converter 120 may generate the final input image 640 by applying a whitelist to the grayscale image 630 for each field. Additionally, the text converter 120 may generate the final input image 640 by applying a whitelist considering the characteristics of the PFT result image to each field to the grayscale image 630. Here, the final input image 640 may be an image containing only characters of a specific type (eg, numbers, alphabets, special symbols) selected according to the characteristics of the PFT result image. Additionally, the text converter 120 may additionally perform preprocessing methods such as noise removal, thresholding, expansion, erosion, opening, Canny, and distortion correction.

FIG. 7 is a diagram illustrating an example to explain an operation of inputting converted text in an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 7, an operation of inputting text extracted from an image into a target file in the database storage unit 130 of the image classification device according to an embodiment of the present disclosure can be described. As an example, the text conversion unit 120 may extract coordinates from the original PFT result image 710. For example, the text conversion unit 120 may extract a detailed image and coordinates of a specific area containing data from the original PFT result image 710. Additionally, the text conversion unit 120 can convert the image within the corresponding coordinates into text using a deep learning algorithm.

As an example, the database storage unit 130 may extract columns and rows in which to input text from the target file 720. For example, the database storage unit 130 may extract columns and rows into which text is to be input from the target file 720 in a standardized format. Here, the target file 720 may be a highly compatible CSV (Common-Separated Value) or Excel format file.

As an example, the database storage unit 130 may generate the final target file 730 by inputting the converted text into the target file 720. For example, the database storage unit 130 may generate the final target file 730 by inputting text converted based on the columns and rows extracted from the target file. Additionally, the database storage unit 130 may repeatedly generate the final target file 730 until all text in the original PFT result image 710 is extracted and input.

As an example, the database storage unit 130 may store the final target file 730 in the database. For example, the database storage unit 130 may generate text automatically classified and extracted from the PFT result image 710 as a final target file 730 in a standardized data format and store it in the database. Additionally, the database storage unit 130 may classify and store items included in the PFT result image 710 in an Entity Relationship Model structure. Accordingly, the PFT result image 710 is compatible with future clinical result data and past accumulated data, and can be used as big data for future research and disease prediction.

FIG. 8 is a diagram illustrating an example for explaining the structure of a database in an image classification device according to an embodiment of the present disclosure.

Referring to FIG. 8, the structure of an entity relationship model stored in the database storage unit 130 of the image classification device according to an embodiment of the present disclosure can be described. As an example, the database storage unit 130 may generate relationship information between patient information and related PFT result information centered on patient information and classify items based on an Entity Relationship Model structure. For example, the PFT result information may include PFT result image information, spirometry information, lung diffusion capacity information, lung volume information, and airway resistance information. Additionally, the entity relationship model structure is a type of conceptual model or semantic data model, and may be a technique for analyzing commonalities and differences between each entity and generating relationship information between each entity based on the analysis results.

As another example, the database storage unit 130 may classify items to be extracted from the PFT result image into an Entity Relationship Model structure consisting of a plurality of tables and store them in the database. For example, the plurality of tables may be composed of a table containing PFT result information, a table containing patient information, and other tables related to PFT result information.

Additionally, the database storage unit 130 can classify the PFT result image into an Entity Relationship Model structure and store the generated relationship information in the database to utilize it as big data.

Hereinafter, an image classification method that can be performed by the image classification device described with reference to FIGS. 1 to 8 will be described.

9 is a flowchart of an image classification method according to another embodiment of the present disclosure.

Referring to FIG. 9, the image classification method of the present disclosure may include an image classification step of classifying the type of the input image (S910). According to one embodiment, an image classification device may classify the type of input image according to a standard type of medical image. As an example, the image classification device may set reference keywords and reference coordinates according to the standard type of medical image, and classify the type of input image based on the set reference keywords and reference coordinates. This is because medical imaging images, which are classified into various types, include data (text and images) in different locations depending on the type, so classifying the images by type before text conversion can provide the effect of maximizing efficiency.

For example, the image classification device can set unique words that do not overlap each other for each standard type of medical image image as each reference keyword, extract the coordinates by dragging the position of the set reference keyword, and set it as the reference coordinate. For a specific example, if the image classification device sets standard keywords that do not overlap each other for each standard type of video image, the coordinates of the start and end points of the standard keyword can be extracted using a mouse handler and set as the reference coordinates.

For another example, if the word and the coordinates of the word included in the input image match the reference keyword and reference coordinate set for each standard type, the image classification device may classify the input image into the standard type set as the standard keyword. Accordingly, the image classification device can classify input images into the same standard type as long as the keywords and coordinates match.

The image classification method of the present disclosure may include a text conversion step of converting the image into text (S920). According to one embodiment, an image classification device may extract detailed images and coordinates from classified images and convert the extracted detailed images into text using a deep learning algorithm. For example, if an input image is classified as a specific type among standard types, the image classification device may repeat the process of extracting detailed images and coordinates and converting them to text until all data included in the classified image is extracted. .

As another example, an image classification device may preprocess input images to increase the accuracy of extracting detailed images and coordinates from classified images. For example, an image classification device can preprocess the input image by enlarging the size of the input image at an arbitrary magnification, adjusting it to a grayscale image, and then applying a whitelist to each field. This may be a method that integrates the gray scale method and the rescale method. For example, the image classification device doubles the size of the original input image, adjusts it to a grayscale image, and then white-scales it, taking into account the characteristics of the input image, which generally contains only alphabetic characters and no special characters. The list can be applied to each field. However, the rate at which the image is resized is not limited to doubling.

As another example, an image classification device can convert extracted detailed images into text in a standardized data format using a deep learning algorithm. For example, the deep learning algorithm may be an optical character recognition module (OCR) that utilizes Long Short-Term Memory (LSTM). For a specific example, LSTM may be an improved algorithm of RNN (Recurrent Neural Networks) that allows a neural network network to make decisions by simultaneously considering current input values and past input values. In addition, LSTM includes a back propagation process that corrects the error by updating each coefficient by passing the error obtained at the output stage back through the neural network, which can improve the recognition rate compared to the recognition process of RNN. For another example, an image classification device can convert images into text in a standardized data format using CTC (Connectionist Temporal Classification), an algorithm that learns unsegmented sequence data in which only class labels are in order and the location of each class is unknown. there is. However, the deep learning algorithm of the present disclosure is not limited to this and may consist of various algorithms that can be used now or in the future.

The image classification method of the present disclosure may include a database storage step of generating a final target file and storing it in the database (S930). According to one embodiment, the image classification device may generate a final target file by inputting text converted based on columns and rows extracted from the target file. And the image classification device can store the generated final target file in the database. For example, an image classification device can extract columns and rows from a target file into which to input text and generate a final target file consisting of columns containing variables and rows containing case descriptions. Additionally, the image classification device can generate the final target file by repeating the input process based on the extracted columns and rows until all text converted from the image is input. Accordingly, the image classification device can generate a final target file in highly compatible CSV (Common-Separated Value) or Excel format and save it in the database.

As an example, an image classification device may classify items to be extracted from an input image into an Entity Relationship Model structure and store them in a database. For example, an image classification device may consist of a table containing test results, a table containing patient information, and other test-related tables.

As described above, according to the present disclosure, an image classification device and method can be provided. In particular, it is possible to provide an image classification device and method that can generate standardized data from images by classifying images according to standard types and extracting text using keywords and coordinates in medical image images. In addition, it is possible to provide an image classification device and method that can generate standardized data that can be used as big data by classifying, itemizing, and storing medical image images using a deep learning algorithm.

In the above, the image classification method according to the embodiment of the present disclosure has been described as being performed through the same procedure as in FIG. 9, but this is only for convenience of explanation, and within the scope of the essential concept of the present disclosure, the method of implementation Accordingly, the performance procedure of each step may be changed, two or more steps may be integrated, or one step may be performed separately into two or more steps.

In addition, terms such as “include,” “comprise,” or “have” as used above mean that the corresponding component may be included, unless specifically stated to the contrary, and thus do not exclude other components. Rather, it should be interpreted as being able to include other components. All terms, including technical or scientific terms, unless otherwise defined, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which this disclosure pertains. Commonly used terms, such as terms defined in a dictionary, should be interpreted as consistent with the context meaning of the related technology, and should not be interpreted in an idealized or overly formal sense unless explicitly defined in the present disclosure.

The above description is merely an illustrative explanation of the technical idea of the present disclosure, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present disclosure. Accordingly, the embodiments disclosed in the present disclosure are not intended to limit the technical idea of the present disclosure but are for illustrative purposes, and the scope of the technical idea of the present disclosure is not limited by these embodiments. The scope of protection of this disclosure should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this disclosure.

Claims (16)

an image classification unit that sets reference keywords and reference coordinates according to the standard type of medical image image, and classifies the type of input image based on the reference keyword and the reference coordinates;
a text conversion unit that extracts detailed images and coordinates from the classified images and converts the extracted detailed images into text using a deep learning algorithm; and
A database storage unit that generates a final target file by inputting the text based on the columns and rows extracted from the target file and stores the final target file in a database,
The image classification unit,
A unique word that does not overlap with each other for each standard type is set as the reference keyword, coordinates extracted by dragging the position of the keyword are set as the reference coordinates, and a word included in the input image and the coordinates of the word are set as the reference keyword. If the keyword and the reference coordinate match, the image classification device classifies the input image into the standard type set by the reference keyword. delete delete According to claim 1,
The text conversion unit,
An image classification device characterized by enlarging the size of the input image at a random magnification, adjusting it to a grayscale image, and then applying a whitelist to each field to preprocess the input image. According to claim 1,
The deep learning algorithm is,
An image classification device characterized by an optical character recognition module (Optical character reader, OCR) utilizing LSTM (Long Short-Term Memory). According to claim 1,
The text conversion unit,
An image classification device that repeats the process of extracting the detailed image and the coordinates and converting them to the text until all data included in the classified image is extracted. According to claim 1,
The database storage unit,
and generating the final target file consisting of the columns containing variables and the rows containing case descriptions. According to claim 1,
The database storage unit,
An image classification device that classifies items to be extracted from input images into an Entity Relationship Model structure and stores them in the database. An image classification step of setting reference keywords and reference coordinates according to the standard type of medical image image and classifying the type of input image based on the reference keyword and reference coordinates;
A text conversion step of extracting detailed images and coordinates from the classified images and converting the extracted detailed images into text using a deep learning algorithm; and
A database storage step of generating a final target file by inputting the text based on the columns and rows extracted from the target file, and storing the final target file in a database,
The image classification step is,
A unique word that does not overlap each other for each standard type is set as the reference keyword, the coordinates extracted by dragging the position of the reference keyword are set as the reference coordinates, and the word included in the input image and the coordinates of the word are set as the reference keyword. An image classification method characterized in that, if the reference keyword and the reference coordinate match, the input image is classified into the standard type set by the reference keyword. delete delete According to clause 9,
The text conversion step is,
An image classification method characterized by enlarging the size of the input image by an arbitrary magnification, adjusting it to a grayscale image, and then applying a whitelist to each field to preprocess the input image. According to clause 9,
The deep learning algorithm is,
An image classification method characterized by an optical character recognition module (Optical character reader, OCR) utilizing LSTM (Long Short-Term Memory). According to clause 9,
The text conversion step is,
An image classification method characterized by repeating the process of extracting the detailed image and the coordinates and converting them to the text until all data included in the classified image is extracted. According to clause 9,
The database storage step is,
An image classification method, characterized in that generating the final target file consisting of the columns containing variables and the rows containing case numbers. According to clause 9,
The database storage step is,
An image classification method characterized by classifying items to be extracted from an input image into an Entity Relationship Model structure and storing them in the database.

Images