KR20230019523A - Apparatus and method for classifying images - Google Patents

Abstract

The present disclosure relates to a device and method for classifying an image. Specifically, by classifying the image and extracting a text according to a standard type using a keyword and coordinate in a medical-image image, the present invention enables in providing the device and method for classifying the image that can generate the standardized data from the image. In addition, by classifying, itemizing, and storing the medical-image image using a deep learning algorithm, the present invention enables to provide the device and method for classifying the image that can generate the standardized data that can be used as big data. The device comprises: an image classification part; a text conversion part; and a database storage part.

Description

Image classification apparatus and method {APPARATUS AND METHOD FOR CLASSIFYING IMAGES}

The present embodiments provide an image classification apparatus and method.

Globally, respiratory illness is one of the leading causes of death even before the advent of coronavirus disease 19 (COVID-19). According to a report by the World Health Organization (WHO), the five leading causes of severe lung-related diseases are chronic obstructive pulmonary disease (COPD), asthma, acute lower respiratory tract infections, tuberculosis (TB) and lung cancer, which affect 4 million people annually. It is analyzed that premature death due to respiratory disease is analyzed. Moreover, many studies are being conducted to predict the occurrence of lung diseases due to the recent COVID-19 epidemic.

Recently, the importance of big data is growing due to the development of artificial intelligence. In addition, the trend of medical service is changing from treatment-oriented to prevention-oriented and health management-oriented, and artificial intelligence (AI) technology using medical data is developing. As cutting-edge medical technology using artificial intelligence develops rapidly in the medical field, learning data required for production and learning of artificial intelligence is more important than ever. In particular, due to the COVID-19 epidemic, medical data obtained from tests to predict the occurrence of lung diseases is widely used in disease analysis, which requires standardized big data related to tests.

However, medical data includes a lot of data such as image test results, medical records, surgical records, nursing records, and blood tests, but most imaged data cannot be directly extracted from images and it is difficult to normalize the data. . In other words, even if the data is stored in EMR (electronic medical record), it is difficult to utilize it as big data for applied research if it is stored in a non-standardized format or in a format in which appropriate statistical analysis cannot be performed.

In this situation, manually classifying or itemizing thousands or tens of thousands of high-quality medical data for learning is very time-consuming and costly. Therefore, in order to utilize test data obtained from patients as big data for disease analysis research and disease prediction, a technology for extracting text from medical image data is required. Specifically, medical image processing technology that classifies and itemizes medical image data is required.

Against this background, the present embodiments provide an image classification apparatus and method capable of generating standardized data by classifying and itemizing medical images.

In order to achieve the above object, in one aspect, in the image classification apparatus, a reference keyword and reference coordinates are set according to a standard type of a medical imaging image, and input based on the reference keyword and reference coordinates is inputted. An image classification unit that classifies the type of image, a text conversion unit that extracts detailed images and coordinates from the classified images and converts the extracted detailed images into text using a deep learning algorithm, and a column and row extracted from the target file. An image classification device including a database storage unit for generating a final target file by inputting text and storing the final target file in a database is provided.

In another aspect, in the image classification method according to the present embodiment, the image classification step of setting a reference keyword and reference coordinates according to a standard type of a medical imaging image, and classifying the type of an input image based on the reference keyword and reference coordinates. , A text conversion step of extracting detailed images and coordinates from classified images and converting the extracted detailed images into text using a deep learning algorithm, and inputting text based on columns and rows extracted from the target file to obtain the final target file. It provides an image classification method comprising a database storage step of generating and storing a final target file in a database.

According to the present embodiments, it is possible to provide an image classification apparatus and method capable of generating standardized data by classifying and itemizing medical images.

1 is a diagram showing the configuration of an image classification apparatus according to an embodiment of the present disclosure.
2 is a flowchart illustrating an operation of generating a final target file in an image classification apparatus according to an embodiment of the present disclosure.
3 is a flowchart illustrating an entire operation of generating a final target file in an image classification apparatus according to an embodiment of the present disclosure.
4 is a diagram illustrating examples of standard types of medical imaging images used in an image classification apparatus according to an embodiment of the present disclosure.
5 is a diagram illustrating an example for explaining an operation of classifying an image type in an image classification apparatus according to an embodiment of the present disclosure.
6 is a diagram illustrating an example for explaining an operation of pre-processing an input image in an image classification apparatus according to an embodiment of the present disclosure.
7 is a diagram illustrating an example for explaining an operation of inputting converted text in the image classification apparatus according to an embodiment of the present disclosure.
8 is a diagram illustrating an example for explaining the structure of a database in an image classification apparatus according to an embodiment of the present disclosure.
9 is a flowchart of an image classification method according to another embodiment of the present disclosure.

The present disclosure relates to an image classification apparatus and method.

DETAILED DESCRIPTION Some embodiments of the present disclosure are described in detail below with reference to exemplary drawings. In adding reference numerals to components of each drawing, the same components may have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present technical idea, the detailed description may be omitted. When "comprises", "has", "consists of", etc. mentioned in this specification is used, other parts may be added unless "only" is used. In the case where a component is expressed in the singular, it may include the case of including the plural unless otherwise explicitly stated.

Also, terms such as first, second, A, B, (a), and (b) may be used in describing the components of the present disclosure. These terms are only used to distinguish the component from other components, and the nature, sequence, order, or number of the corresponding component is not limited by the term.

In the description of the positional relationship of components, when it is described that two or more components are "connected", "coupled" or "connected", the two or more components are directly "connected", "coupled" or "connected". ", but it will be understood that two or more components and other components may be further "interposed" and "connected", "coupled" or "connected". Here, other components may be included in one or more of two or more components that are “connected”, “coupled” or “connected” to each other.

In the description of the temporal flow relationship related to components, operation methods, production methods, etc., for example, "after", "continued to", "after", "before", etc. Alternatively, when a flow sequence relationship is described, it may also include non-continuous cases unless “immediately” or “directly” is used.

On the other hand, when a numerical value or corresponding information (eg, level, etc.) for a component is mentioned, even if there is no separate explicit description, the numerical value or its corresponding information is not indicated by various factors (eg, process factors, internal or external shocks, noise, etc.) may be interpreted as including an error range that may occur.

The optical character reader (OCR) in this specification is used to receive handwritten or printed documents as input and convert them into data that a computer can understand. The application fields of OCR include vehicle license plates, receipts, passports, It is a technology applied to various fields such as recognition of checks, etc. In other words, with the optical quasi-java recognition module, you can conveniently convert images, PDF files, or other types of files (if they contain text) into text in machine-encoded format. For example, Tesseract OCR can be an open source based optical character recognition engine.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a diagram showing the configuration of an image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 1 , the image classification apparatus 100 according to an embodiment of the present disclosure sets standard keywords and standard coordinates according to standard types of medical imaging images, and sets an input image based on the standard keywords and standard coordinates. An image classification unit 110 that classifies the type of , a text conversion unit 120 that extracts detailed images and coordinates from the classified images, and converts the extracted detailed images into text using a deep learning algorithm, and extracts from a target file Provided is an image classification apparatus 100 including a database storage unit 130 that generates a final target file by inputting text based on columns and rows and stores the final target file in a database.

According to an embodiment, the image classification unit 110 may classify the type of the input image according to the standard type of the medical imaging image. For example, the image classification unit 110 may set each reference keyword and reference coordinates according to the standard type of the medical imaging image, and classify the type of the input image based on the set reference keyword and reference coordinates. Since medical imaging images classified into various types include data (text and image) at different positions depending on the type, classification of images by type before text conversion can provide an effect of maximizing efficiency.

For example, the image classification unit 110 may set a unique word that does not overlap with each other for each standard type of medical imaging image as each standard keyword, extract coordinates by dragging the location of the set standard keyword, and set them as standard coordinates. there is. For example, if the image classification unit 110 sets non-overlapping standard keywords for each standard type of video image, the coordinates of the start and end points of the standard keyword can be extracted using a mouse handler and set as standard coordinates. there is.

For another example, the image classification unit 110 classifies the input image into the standard type set as the standard keyword when words included in the input image and the coordinates of the word match standard keywords and standard coordinates set for each standard type. can Accordingly, the image classification unit 110 may classify the input image into the same standard type when the keyword and the coordinates match.

According to an embodiment, the text conversion unit 120 may extract detailed images and coordinates from input images classified according to standard types, and convert the extracted detailed images into text using a deep learning algorithm. For example, when an input image is classified as a specific type among standard types, the text conversion unit 120 repeats a process of extracting detailed images and coordinates and converting them into text until all data included in the classified images are extracted. can do.

As another example, the text conversion unit 120 may pre-process the input image to increase the accuracy of extracting detailed images and coordinates from the classified images. For example, the text conversion unit 120 may preprocess the input image by enlarging the size of the input image at an arbitrary magnification and adjusting the size to a grayscale image, and then applying a whitelist for each field. This may be a method combining the gray scale method and the rescale method. For example, the text conversion unit 120 doubles the size of the input original image, adjusts it to a grayscale image, and then generally converts the characteristics of an input image that does not include special characters but only alphabetic characters. The considered whitelist can be applied for each field. However, the ratio at which the size of the image is adjusted is not limited to doubling.

As another example, the text conversion unit 120 may convert the extracted detailed image into text in a standardized data format using a deep learning algorithm. For example, the deep learning algorithm may be an optical character reader (OCR) that utilizes long short-term memory (LSTM). As a specific example, LSTM may be an improved algorithm of recurrent neural networks (RNNs) that allows a neural network to simultaneously consider and determine a current input value and a past input value. In addition, LSTM includes a back propagation process of correcting the error by updating each coefficient while going back through the neural network with the error obtained at the output stage, so the recognition rate can be improved compared to the recognition process of RNN. For another example, the text conversion unit 120 transforms an image into text in a standardized data format using CTC (Connectionist Temporal Classification), an algorithm that learns unsegmented sequence data in which only class labels are in order and the position of each class is not known. can be converted However, the deep learning algorithm of the present disclosure is not limited thereto and may be composed of various algorithms that can be used now or in the future.

According to an embodiment, the database storage unit 130 may generate a final target file by inputting converted text based on columns and rows extracted from the target file. Also, the database storage unit 130 may store the generated final target file in the database. For example, the database storage unit 130 may extract columns and rows from a target file to which text is to be input, and generate a final target file composed of columns including variables and rows including case codes. Then, the database storage unit 130 may create a final target file by repeating the input process based on the extracted columns and rows until all texts converted from images are input. Accordingly, the database storage unit 130 may generate a final target file in a highly compatible CSV (Common-Separated Value) or Excel format and store it in the database.

For example, the database storage unit 130 may classify items to be extracted from an input image into an Entity Relationship Model structure and store the items in the database. For example, the database storage unit 130 may include a table including test results, a table including patient information, and other test-related tables.

According to an embodiment, the image classification apparatus may further include a verification tool capable of evaluating the accuracy of the algorithm. Here, a validator may be used as the verification tool. For example, algorithms may require verification because their accuracy is not always constant and is highly dependent on the application of various pre-processing and whitelisting. Here, the algorithm describes Tesseract OCR as an example, but is not limited thereto.

2 is a flowchart illustrating an operation of generating a final target file in an image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 2 , the image classification unit 110 of the image classification apparatus according to an embodiment of the present disclosure may set standard keywords and standard coordinates for each standard type (S210). For example, the image classification unit 110 may set standard keywords and standard coordinates according to standard types of medical imaging images. Here, the medical imaging image is an original PFT image file in which Pulmonary Function Tests (PFT) results are recorded, and may be PFT result images included in various types of PFT image sheets. However, the pulmonary function test image describes an example of a medical imaging image, but is not limited thereto. For example, the image classification unit 110 may set a type standard and set standard keywords and reference coordinates for each standard type since the location of included data (text and image) is different depending on the type of the medical image image. there is. For another example, the image classification unit 110 may set standard types of medical imaging images to 8 types including unknown types. Specifically, when the medical imaging image is a PFT result image, each PFT result image may have different numbers of characters, size, thickness, color, and space between characters. In addition, the characters included in the PFT result image increase in size and become thicker as the number of characters increases, and accuracy may increase as the color difference between the characters and the background increases. Also, white space between characters can affect accuracy. Therefore, the image classification unit 110 can maximize efficiency by classifying the resulting PFT images according to standard types before algorithm processing for text conversion.

However, the number of standard types is set according to the type of medical imaging image, but is not limited thereto.

As another example, the image classification unit 110 may set a unique word that does not overlap with each other for each standard type as a standard keyword, and set coordinates extracted by dragging the position of the standard keyword as the standard coordinates. For example, the image classifier 110 may set a unique word that does not overlap with a keyword set in other standard types of images among text included in the image for each standard type of medical imaging image as a standard keyword. Accordingly, the image classification unit 110 may set a unique word representing each standard type as each standard keyword. For another example, the image classification unit 110 may extract coordinates of the standard keyword by dragging the position of the standard keyword set for each standard type with a mouse handler, and set the coordinates as the standard coordinates.

The image classification unit 110 of the image classification apparatus according to an embodiment may classify the type of the input image based on the reference keyword and reference coordinates (S220). For example, the image classification unit 110 may determine whether a word included in an input image and the coordinates of the word coincide with a reference keyword and reference coordinates set for each standard type. Further, the image classification unit 110 may classify the input image into a standard type set as a word included in the input image and a reference keyword matching the coordinates of the word and reference coordinates as a result of the determination. Accordingly, the image classification unit 110 may classify the PFT result images included in the input PFT image sheet according to standard types.

The text conversion unit 120 of the image classification apparatus according to an embodiment may extract detailed images and coordinates from each image classified by standard type (S230). For example, the text conversion unit 120 may perform preprocessing to extract detailed images and coordinates from the resulting PFT image. For example, the text conversion unit 120 may preprocess the input image by enlarging the size of the input image at an arbitrary magnification and adjusting the size to a grayscale image, and then applying a whitelist for each field. As a specific example, the text conversion unit 120 doubles the size of the input image through a method combining the rescale method and the gray scale method, and then adjusts the size of the input image to a grayscale image. In addition, the text conversion unit 120 may apply a whitelist for each field in consideration of the characteristics of the PFT result image including only alphabetic characters without including special characters.

As another example, the text conversion unit 230 may extract detailed images and coordinates from the preprocessed PFT result image. For example, the text conversion unit 230 may extract a detailed image corresponding to a specific region from the preprocessed PFT result image, and extract coordinates of both ends capable of specifying the location of the detailed image.

The text conversion unit 120 of the image classification apparatus according to an embodiment may convert the extracted detailed image into text using a deep learning algorithm (S240). For example, the text conversion unit 120 may convert the extracted detailed image into text using a deep learning algorithm based on an optical character reader (OCR) using long short-term memory (LSTM). . Here, the optical character recognition module may be based on the Tesseract OCR library. For example, the text conversion unit 120 may detect and convert text included in the extracted detailed image by using a deep learning algorithm and OpenCV to maximize accuracy.

As another example, the text conversion unit 120 may repeat a process of extracting detailed images and coordinates and converting them into text until all data included in the classified images are extracted. For example, the text conversion unit 120 may extract detailed images and coordinates while moving an area of the PFT result image until all data included in the PFT result image classified according to the standard type is extracted. Further, the text conversion unit 120 may convert each detailed image extracted from the PFT result image into text using a deep learning algorithm.

The database storage unit 130 of the image classification apparatus according to an embodiment may extract columns and rows of the target file (S250). For example, the database storage unit 130 may extract columns and rows of a target file to input converted text from the PFT result image. For example, the database storage unit 130 may extract columns and rows to input converted text based on variables or case numbers from an Excel format target file. In addition, the database storage unit 130 may repeat the process of extracting columns and rows of the target file until all data included in the PFT result image are input.

The database storage unit 130 of the image classification apparatus according to an embodiment may generate a final target file by inputting text based on the extracted columns and rows (S260). For example, the database storage unit 130 may generate and store a final target file composed of columns including variables and rows including case numbers in the database. For example, the database storage unit 130 may store a final target file generated by inputting text converted from a PFT result image into a target file in a highly compatible CSV (Common-Separated Value) or Excel format, in the database.

As another example, the database storage unit 130 may classify items to be extracted from an input image into an Entity Relationship Model structure and store the items in the database. For example, the database storage unit 130 classifies items to be extracted from the PFT result image into an object relation model structure composed of a table including PFT result information, a table including patient information, and other tables related to PFT result information, can be stored in a database. Here, the table related to other PFT result information may be a table related to at least one of spirometry, lung diffusing capacity, ling volume, and airway resistance.

3 is a flowchart illustrating an entire operation of generating a final target file in an image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 3 , the image classification unit 110 of the image classification apparatus according to an embodiment of the present disclosure may read an original image file of a medical examination result (S301). For example, the image classification unit 110 may read an original image capture file of a PFT examination result through an Open Source Computer Vision (OpenCV) library.

According to an embodiment, the image classification unit 110 may classify the type of image (S302). For example, the image classification unit 110 may classify the resulting PFT image into a standard type through the Tesseract OCR library.

According to an embodiment, the image classification unit 110 may determine whether an existing folder corresponding to the type of the classified image exists (S303). For example, the image classification unit 110 may determine whether an existing folder in which a type corresponding to a PFT result image is stored exists.

According to an embodiment, the image classification unit 110 may create a new type of folder if no existing folder exists (S304). For example, the image classification unit 110 may create a new type of folder in which a type corresponding to a PFT result image may be stored.

According to an embodiment, if a folder of a classified type exists, the image classification unit 110 may store an image in the corresponding folder (S305). For example, the image classification unit 110 may move and store the PFT result image from the existing location to a folder of a corresponding type. Here, the folder may be an existing folder or a newly created type of folder. In addition, the image classification unit 110 may set a memory pointer to create a database in real time while moving the PFT result image to a corresponding type of folder.

According to an embodiment, the text conversion unit 120 may pre-process the type-classified image (S306). For example, the text conversion unit 120 may maximize reading accuracy by pre-processing the resulting PFT image through scaling and grayscale adjustment.

According to an embodiment, the text conversion unit 120 may apply a whitelist to the resulting PFT image (S307). For example, the text conversion unit 120 may apply a whitelist for each field in order to generate an image including only desired types of characters in the PFT result image.

According to an embodiment, the text conversion unit 120 may extract text from the resulting PFT image (S308). For example, the text conversion unit 120 may extract and convert all texts from the PFT result image using a deep learning algorithm. Here, the deep learning algorithm may be Tesseract OCR, which is an optical character reader (OCR) using long short-term memory (LSTM).

According to an embodiment, the database storage unit 130 may read a target file in Excel format (S311). In addition, the database storage unit 130 may determine whether a target file in Excel format already exists (S312). The database storage unit 130 may create a new target file in Excel format if there is no existing target file in Excel format (S313).

According to an embodiment, the database storage unit 130 may input and store the converted text in an Excel format target file (S309). For example, the database storage unit 130 may repeatedly input and store text in the PFT result image according to columns and rows of the target file in Excel format until all texts are input.

According to an embodiment, the database storage unit 130 may store a CSV (Common-Separated Value) format file together with an Excel format target file (S310). For example, the database storage unit 130 may separately store all texts of the PFT result image as a CSV format file that stores text by separating fields with commas.

4 is a diagram illustrating examples of standard types of medical imaging images used in an image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 4 , standard types set to classify medical imaging images in the image classification unit 110 of the image classification apparatus according to an embodiment of the present disclosure will be described. For example, the image classification unit 110 may classify the image type based on a standard type set before text is extracted from the medical imaging image. For example, the image classification unit 110 may set standard types of PFT result images to 8 types including unknown types. At this time, the unknown type can include all types (ex. only graphs) that do not include text. Accordingly, the image classification unit 110 may classify the input PFT result image into 8 types according to the set standard type. However, the standard type is set according to the type of medical imaging image, and the type or number of standard types is not limited thereto.

For another example, the image classification unit 110 may set the standard type of the PFT result image based on the number of characters, size, thickness, color, and space between characters. For example, the set standard types of medical imaging images may consist of different numbers of extracted text data. This may be because the presence or absence of indicators and stimulants in the test items included in the PFT result image is different. Specifically, 116 data of type 1, 164 data of type 2, 21 data of type 3, 130 data of type 4, 37 data of type 5, 73 data of type 6, and 130 data of type 7 are standard types. For another example, types of type1 and type2 among standard types may be similar. In general, the type1 and type2 types may include characters included in the PFT result image that are thicker and larger than other types, and that blanks exist between characters. Also, among the standard types, types of type3, type4, and type7 may be similar. The type3, type4, and type7 types may include the characteristics that the characters included in the PFT result image are thin and small, and the margins between characters are narrow. Also, among the standard types, the formats of type5 and type6 may be similar. In general, the type5 and type6 types may include the characteristics that the characters included in the PFT result image are smaller than other types but thicker and there is a blank space between them.

5 is a diagram illustrating an example for explaining an operation of classifying an image type in an image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 5 , an operation of classifying an image type based on a reference keyword and reference coordinates in the image classification unit 110 of the image classification apparatus according to an embodiment of the present disclosure will be described. For example, the image classification unit 110 may set a unique word that does not overlap with each other for each standard type as the standard keyword 510 . The image classification unit 110 may set reference coordinates by dragging the location of the reference keyword 510 to a mouse handler. For example, the image classification unit 110 drags a mouse handler along the X-axis 520 and the Y-axis 530 to determine the start point (X, Y) and end point (X', The coordinates of Y') can be extracted and set as reference coordinates. Here, the mouse handler (Mouse Handler) is an abbreviation of mouse event handler, and may refer to a module programmed to perform a predetermined coordinate measurement function by calling a specific function when various events or events related to the mouse occur. there is.

As another example, the image classification unit 110 may classify a type based on standard keywords and standard coordinates set for each standard type when an arbitrary image is input. For example, the image classification unit 110 may extract a word included in the image and the coordinates of the word using a mouse handler when an arbitrary PFT result image is input. Then, the image classification unit 110 may classify the input PFT result image into the standard type of the standard keyword when the extracted word and the coordinates of the word coincide with the standard keyword and the standard coordinates.

6 is a diagram illustrating an example for explaining an operation of pre-processing an input image in an image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 6 , an operation of preprocessing an input image in the text conversion unit 120 of the image classification apparatus according to an embodiment of the present disclosure will be described. For example, the text conversion unit 120 may increase the reading accuracy of the deep learning algorithm by pre-processing the input image. For example, the text conversion unit 120 may enlarge the size of the image 610 input through a rescaling method at an arbitrary magnification and convert the image 620 into an enlarged image 620 . Here, an arbitrary magnification may be 2 times, but is not limited thereto. Also, the text conversion unit 120 may adjust the enlarged image 620 corresponding to a multi-channel image into a grayscale image 630 that is a single-channel image through a grayscale method. The text conversion unit 120 may generate the final input image 640 by applying a whitelist to the grayscale image 630 for each field. In addition, the text conversion unit 120 may generate the final input image 640 by applying a whitelist considering characteristics of the PFT result image for each field to the grayscale image 630 . Here, the final input image 640 may be an image including only characters of a specific type (ex, numbers, alphabets, special symbols) selected according to the characteristics of the PFT result image. In addition, the text conversion unit 120 may additionally perform preprocessing methods such as noise removal, threshold value, expansion, erosion, opening, canny and distortion correction.

7 is a diagram illustrating an example for explaining an operation of inputting converted text in the image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 7 , an operation of inputting text extracted from an image into a target file in the database storage unit 130 of the image classification apparatus according to an embodiment of the present disclosure will be described. For example, the text conversion unit 120 may extract coordinates from the original PFT result image 710 . For example, the text conversion unit 120 may extract a detailed image and coordinates of a specific area including data from the original PFT result image 710 . In addition, the text conversion unit 120 may convert an image within corresponding coordinates into text using a deep learning algorithm.

For example, the database storage unit 130 may extract columns and rows to input text from the target file 720 . For example, the database storage unit 130 may extract columns and rows to input text from the target file 720 in a standardized format. Here, the target file 720 may be a CSV (Common-Separated Value) or Excel format file with high compatibility.

For example, the database storage unit 130 may generate a final target file 730 by inputting converted text into the target file 720 . For example, the database storage unit 130 may generate the final target file 730 by inputting converted text based on columns and rows extracted from the target file. In addition, the database storage unit 130 may generate the final target file 730 repeatedly until all texts of the original PFT result image 710 are extracted and input.

For example, the database storage unit 130 may store the final target file 730 in a database. For example, the database storage unit 130 may generate text automatically classified and extracted from the PFT result image 710 as a final target file 730 in a standardized data format and stored in the database. In addition, the database storage unit 130 may classify and store items included in the PFT result image 710 in an Entity Relationship Model structure. Accordingly, the PFT result image 710 is compatible with future clinical result data and past accumulated data, and can be used as big data for future research and disease prediction.

8 is a diagram illustrating an example for explaining the structure of a database in an image classification apparatus according to an embodiment of the present disclosure.

Referring to FIG. 8 , the structure of an entity relationship model stored in the database storage unit 130 of the image classification apparatus according to an embodiment of the present disclosure can be described. For example, the database storage unit 130 may create relationship information with PFT result information related to patient information based on patient information based on an entity relationship model structure and classify items. For example, the PFT result information may include PFT result image information, spirometry information, lung diffusion capacity information, lung volume information, and airway resistance information. Also, the entity relation model structure is a type of conceptual model or semantic data model, and may be a technique of analyzing commonalities and differences between entities and generating relational information between entities based on the analysis result.

As another example, the database storage unit 130 may classify items to be extracted from the PFT result image into an Entity Relationship Model structure composed of a plurality of tables and store them in the database. For example, the plurality of tables may include a table including PFT result information, a table including patient information, and tables related to other PFT result information.

In addition, the database storage unit 130 may store relationship information generated by classifying the PFT result image into an Entity Relationship Model structure in a database and utilize it as big data.

Hereinafter, an image classification method that can be performed by the image classification apparatus described with reference to FIGS. 1 to 8 will be described.

9 is a flowchart of an image classification method according to another embodiment of the present disclosure;

Referring to FIG. 9 , the image classification method of the present disclosure may include an image classification step of classifying the type of an input image (S910). According to an embodiment, the image classification apparatus may classify the type of the input image according to the standard type of the medical imaging image. For example, the image classification apparatus may set reference keywords and reference coordinates according to standard types of medical imaging images, and classify the type of an input image based on the set reference keywords and reference coordinates. Since medical imaging images classified into various types include data (text and image) at different positions depending on the type, classification of images by type before text conversion can provide an effect of maximizing efficiency.

For example, the image classification apparatus may set a unique word that does not overlap with each other for each standard type of medical imaging image as each standard keyword, extract coordinates by dragging the location of the set standard keyword, and set them as standard coordinates. For example, if the image classification apparatus sets non-overlapping standard keywords for each standard type of video image, the coordinates of the start and end points of the standard keyword can be extracted and set as standard coordinates using a mouse handler.

For another example, the image classification apparatus may classify the input image into the standard type set as the standard keyword when words included in the input image and coordinates of the word match standard keywords and standard coordinates set for each standard type. Accordingly, the image classification device may classify input images into the same standard type as long as keywords and coordinates match.

The image classification method of the present disclosure may include a text conversion step of converting into text (S920). According to an embodiment, the image classification apparatus may extract detailed images and coordinates from the classified images, and convert the extracted detailed images into text using a deep learning algorithm. For example, when an input image is classified as a specific type among standard types, the image classification device may repeat a process of extracting detailed images and coordinates and converting them into text until all data included in the classified images are extracted. .

As another example, the image classification device may pre-process input images to increase accuracy of extracting detailed images and coordinates from the classified images. For example, the image classification apparatus may pre-process the input image by enlarging the size of the input image at an arbitrary magnification, adjusting the size to a grayscale image, and then applying a whitelist for each field. This may be a method combining the gray scale method and the rescale method. As a specific example, the image classification device doubles the size of the original input image, adjusts it to a grayscale image, and generally considers the characteristics of the input image, which does not contain special characters but only alphabetic characters, to make it white. Lists can be applied field by field. However, the ratio at which the size of the image is adjusted is not limited to doubling.

As another example, the image classification device may convert the extracted detailed image into text in a standardized data format using a deep learning algorithm. For example, the deep learning algorithm may be an optical character reader (OCR) that utilizes long short-term memory (LSTM). As a specific example, LSTM may be an improved algorithm of recurrent neural networks (RNNs) that allows a neural network to simultaneously consider and determine a current input value and a past input value. In addition, LSTM includes a back propagation process of correcting the error by updating each coefficient while going back through the neural network with the error obtained at the output stage, so the recognition rate can be improved compared to the recognition process of RNN. As another example, an image classification device can convert an image into text in a standardized data format using CTC (Connectionist Temporal Classification), an algorithm that learns unsegmented sequence data in which only class labels are in order and the position of each class is unknown. there is. However, the deep learning algorithm of the present disclosure is not limited thereto and may be composed of various algorithms that can be used now or in the future.

The image classification method of the present disclosure may include a database storage step of generating and storing a final target file in a database (S930). According to an embodiment, the image classification apparatus may generate a final target file by inputting converted text based on columns and rows extracted from the target file. Also, the image classification device may store the generated final target file in a database. For example, the image classification apparatus may extract columns and rows from a target file to input text to generate a final target file composed of columns including variables and rows including case numbers. Then, the image classification device may create a final target file by repeating the input process based on the extracted columns and rows until all texts converted from images are input. Accordingly, the image classification device may generate a final target file in a highly compatible CSV (Common-Separated Value) or Excel format and store it in a database.

For example, the image classification device may classify an item to be extracted from an input image into an entity relationship model structure and store it in a database. For example, the image classification device may include a table including test results, a table including patient information, and other test-related tables.

As described above, according to the present disclosure, an image classification apparatus and method can be provided. In particular, it is possible to provide an image classification apparatus and method capable of generating standardized data from images by classifying images according to standard types using keywords and coordinates in medical imaging images and extracting text. In addition, it is possible to provide an image classification apparatus and method capable of generating standardized data that can be used as big data by classifying, itemizing, and storing medical imaging images using a deep learning algorithm.

In the above, the image classification method according to an embodiment of the present disclosure has been described as being performed in the same procedure as in FIG. Accordingly, the procedure for performing each step may be changed, two or more steps may be integrated, or one step may be separated into two or more steps.

In addition, terms such as "include", "comprise" or "have" described above mean that the corresponding component may be inherent unless otherwise stated, excluding other components. It should be construed as being able to further include other components. All terms, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs, unless defined otherwise. Commonly used terms, such as terms defined in a dictionary, should be interpreted as consistent with the meaning in the context of the related art, and are not interpreted in an ideal or excessively formal meaning unless explicitly defined in the present disclosure.

The above description is merely an example of the technical idea of the present disclosure, and various modifications and variations may be made to those skilled in the art without departing from the essential characteristics of the present disclosure. Therefore, the embodiments disclosed in this disclosure are not intended to limit the technical spirit of the present disclosure, but to explain, and the scope of the technical spirit of the present disclosure is not limited by these embodiments. The protection scope of the present disclosure should be construed by the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present disclosure.

Claims (16)

an image classification unit configured to set reference keywords and reference coordinates according to standard types of medical imaging images, and to classify input image types based on the reference keywords and reference coordinates;
a text conversion unit that extracts detailed images and coordinates from the classified images and converts the extracted detailed images into text using a deep learning algorithm; and
and a database storage unit for generating a final target file by inputting the text based on columns and rows extracted from the target file, and storing the final target file in a database. According to claim 1,
The image classification unit,
The image classification apparatus characterized in that for each standard type, a unique word that does not overlap with each other is set as the reference keyword, and coordinates extracted by dragging a location of the keyword are set as the reference coordinates. According to claim 2,
The image classification unit,
If a word included in the input image and the coordinates of the word match the reference keyword and reference coordinates, the input image is classified into the standard type set as the reference keyword. According to claim 1,
The text conversion unit,
An image classification apparatus characterized by preprocessing the input image by enlarging the size of the input image at an arbitrary magnification and adjusting the size to a grayscale image, and then applying a whitelist for each field. According to claim 1,
The deep learning algorithm,
An image classification device characterized in that it is an optical character reader (OCR) that utilizes LSTM (Long Short-Term Memory). According to claim 1,
The text conversion unit,
The image classification apparatus, characterized in that repeating the process of extracting the detailed image and the coordinates and converting them into the text until all data included in the classified image are extracted. According to claim 1,
The database storage unit,
and generating the final target file composed of the columns including variables and the rows including case numbers. According to claim 1,
The database storage unit,
An image classification apparatus characterized in that an item to be extracted from an input image is classified into an entity relationship model structure and stored in the database. an image classification step of setting reference keywords and reference coordinates according to standard types of medical imaging images, and classifying types of input images based on the reference keywords and reference coordinates;
a text conversion step of extracting detailed images and coordinates from the classified images and converting the extracted detailed images into text using a deep learning algorithm; and
and a database storage step of generating a final target file by inputting the text based on columns and rows extracted from the target file, and storing the final target file in a database. According to claim 9,
The image classification step,
An image classification method characterized in that for each standard type, a unique word that does not overlap with each other is set as the reference keyword, and coordinates extracted by dragging a location of the reference keyword are set as the reference coordinates. According to claim 10,
The image classification step,
and classifying the input image into the standard type set as the standard keyword when a word included in the input image and coordinates of the word match the reference keyword and the reference coordinates. According to claim 9,
The text conversion step,
An image classification method characterized by preprocessing the input image by enlarging the size of the input image at an arbitrary magnification and adjusting the size to a grayscale image, and then applying a whitelist for each field. According to claim 9,
The deep learning algorithm,
An image classification method characterized in that it is an optical character reader (OCR) that utilizes LSTM (Long Short-Term Memory). According to claim 9,
The text conversion step,
The image classification method characterized in that repeating the process of extracting the detailed image and the coordinates and converting them into the text until all data included in the classified image are extracted. According to claim 9,
The database storage step,
and generating the final target file composed of the columns including variables and the rows including case numbers. According to claim 9,
The database storage step,
An image classification method characterized in that items to be extracted from an input image are classified into an Entity Relationship Model structure and stored in the database.

Images