WO2022255674A1

WO2022255674A1 - Exponible biometric image reading support device and method

Info

Publication number: WO2022255674A1
Application number: PCT/KR2022/006536
Authority: WO
Inventors: 박상민
Original assignee: 자이메드 주식회사
Priority date: 2021-05-31
Filing date: 2022-05-09
Publication date: 2022-12-08
Also published as: KR20220161672A

Abstract

Provided are a biometric image reading support device and method, the device comprising: a processor; and a memory including one or more instructions implemented to be performed by the processor, wherein the processor extracts first attribute information from a first biometric image on the basis of a first machine learning model, transforms the style of a second biometric image paired with the first biometric image on the basis of a second machine learning model to generate a third biometric image having second attribute information comparable with the first attribute information, and displays the first biometric image having the first attribute information and the third biometric image having the second attribute information on a display unit.

Description

Describeable biometric image reading support device and method

The present application relates to an explanatory reading support method and an apparatus and system using the method in reading a biometric image in the medical field.

Recently, with the development of artificial intelligence learning models, many machine learning models are being used to read medical images. For example, learning models such as convolutional neural networks, deep neural networks, recurrent neural networks, and deep belief networks can be used to detect, classify, and feature medical image images. It is applied to learning.

A machine learning model is used to support image reading (finding/diagnosis) in order to predict a test subject's disease. More specifically, a medical image of a subject is acquired, attribute information is extracted based on a machine learning model, provided to a diagnoser, and a disease is predicted based on the attribute information. At this time, the attribute information includes various information included in the medical image.

However, even if the attribute information of the image is extracted based on the learning model as described above, if the learning information input to the learning model is different, an entity such as a medical staff may give wrong information. For example, differences in learning information include lack of learning data input to the learning model, differences in imaging environments (e.g., health checkup center, private eye hospital, general hospital specializing in ophthalmology), and group (e.g., only normal people, only abnormal people). It may be due to the difference between normal and abnormal people), differences in imaging equipment, and the like. These various factors cause medical staff to mispredict the disease of the test subject.

Therefore, even if there is a difference in learning information, there is a need for more accurate disease prediction for medical images and methods and devices for explanations that can support disease prediction.

Embodiments of the present invention are intended to provide a biometric image reading support method and apparatus capable of accurately and reliably reading a biometric image based on a machine learning model.

An embodiment of the present invention is to provide a biometric image read support method and apparatus capable of explaining the reason for reading the biometric image.

The objects of the present application are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

According to one aspect of the invention, a processor; and a memory including one or more instructions implemented to be executed by the processor, wherein the processor extracts first attribute information from a first biological image based on a first machine learning model; A third biometric image having second attribute information comparable to the first attribute information is generated by modifying a style of a second biometric image paired with the first biometric image based on a machine learning model, and the first attribute information The first biological image having information and the third biological image having the second attribute information are displayed on the display unit.

The second machine learning model includes a first sub machine learning model and a second sub machine learning model, and based on the first sub machine learning model, the style of the second biometric image is modified to be generated as an adversarial style image. can

The style may be modified by mapping hostile noise to the second biological image.

The hostile noise may control values for adjusting gradation levels of R, G, and B pixels of the second living body image, and changes in color of R, G, and B pixels of the second living body image. value, and a value for adjusting the contrast ratio of the second living body image.

The second sub-machine learning model may generate the third biological image by fusing the hostile style image based on the second biological image.

The screen displayed on the display unit may include a biometric image screen for displaying the third biological image and an image style adjusting screen for displaying a scroll bar for adjusting the style of the second biological image.

According to an embodiment of the present invention, in reading a biometric image based on a machine learning model, there is an effect of supporting more accurate reading.

In addition, based on the machine learning model, in reading the biometric image, it is possible to explain the reading result by generating comparable biometric images and comparing them, and there is an effect of enabling more reliable biometric image reading. .

Effects of the present application are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram exemplarily illustrating a process of generating attribute information of a biometric image by a biometric image reading device according to an embodiment of the present invention.

2A is a diagram exemplarily illustrating a process of generating a plurality of adversarial style images by a biometric image reading device according to an embodiment of the present invention.

2B is a diagram illustratively illustrating a process of generating a single adversarial style image according to predicted values by a biometric image reading device according to an embodiment of the present invention.

FIG. 3 exemplarily illustrates a process of generating a third living body image to be compared with a first living body image by a living body image reading device according to an embodiment of the present invention.

4 is a diagram illustrating a second biological image generated by a biometric image reading device and an adversarial style according to an exemplary embodiment of the present invention.

5 is a diagram schematically illustrating an exemplary configuration of an apparatus for supporting reading of a biometric image that performs a method of supporting reading of a biometric image according to an embodiment of the present invention.

6 is a diagram schematically illustrating a biometric image reading support system that performs a method of supporting reading of a biometric image according to an embodiment of the present invention.

7 is a diagram illustrating a screen displayed on a display unit of an apparatus for supporting reading of a biometric image according to an embodiment of the present invention.

8 is a flowchart exemplarily illustrating a biometric image read support method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the accompanying drawings are only described in order to more easily disclose the contents of the present invention, and those skilled in the art can easily understand that the scope of the present invention is not limited to the scope of the accompanying drawings. You will know.

In addition, terms used in the detailed description and claims of the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In the detailed description and claims of the present invention, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, It should be understood that it does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In the detailed description and claims of the present invention, terms such as "learning" or "learning" refer to performing machine learning through procedural computing, such as human educational activities. It should be understood that it is not intended to refer to mental processes.

The term "image" used in the detailed description and claims of the present invention can be defined as a digital copy or imitation of the shape or specific characteristics of a person or object, and the image is a JPEG image, PNG image, GIF image, It may be, but is not limited to, a TIFF image or any other digital image format known in the art. Also, "image" may be used in the same sense as "photograph".

The "attribute" used in the detailed description of the present invention and claims may be defined as a group of one or more descriptive characteristics of a subject capable of distinguishing a disease (lesion), and the "attribute" may be expressed as a numerical characteristic. .

The apparatus, method, system, and device disclosed in the present invention may be applied to and used for medical images of fundus and osteoarthritis structures or any other biological tissue images capable of supporting the diagnosis of disease states, but are not limited thereto, and computed tomography scans. (CT), magnetic resonance imaging (MRI), computed radiography, magnetic resonance, angioscopy, optical coherence tomography, color flow Doppler, cystoscopy, diaphanography, echocardiography, fluorescein angiography ( fluoresosin angiography), laparoscopy, magnetic resonance angiography, positron emission tomography, single photon emission computed tomography, X-ray angiography, nuclear medicine, biomagnetic imaging, culposcopy, double Doppler, digital microscopy, endoscopy , laser, surface scanning, magnetic resonance spectroscopy, radiographic imaging, thermal imaging and radiofluorescence.

Moreover, the present invention covers all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment in another embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

In general, when a biometric image such as a fundus image is acquired from an arbitrary object, disease attribute information is derived from the biometric image using a machine learning model. However, as described above, the attribute information of such a biometric image cannot derive reliable attribute information due to the difference in learning information input to the machine learning model. In addition, even if the attribute information of the biometric image is primarily derived, support for reliable reading and diagnosis of the biometric image cannot be provided due to a lack of explanation that can be supported in predicting and judging a disease through this.

In order to solve such unreliable biometric image reading support according to an embodiment of the present invention, the attribute information of the derived biometric image has reliability and can assist medical staff in accurately reading the biometric image. Image attribute information can be automatically generated. That is, the biometric image reading support apparatus 100 according to the present invention finds attribute information from the first biological image 1 primarily photographed through a camera or the like with respect to the biological tissue of an object, and secondarily acquires attribute information. It may be helpful in determining or explaining whether the first biological image 1 of the target object is an actual reliable biological image.

In order to generate the attribute information of the biometric image as described above, the biometric image reading support apparatus 100 uses a first machine learning model (Machine Learning Model, 3) from the captured first biometric image (1) to obtain a first biometric image. The first attribute information 5 of the biometric image 1 is extracted, and the extracted first attribute information 5 of the first biometric image 1 is stored in a system memory unit 113 or a storage device 115 to be described later. can be stored in The first machine learning model 3 may be input to the processor 111 and executed, or may be input to a computer readable recording medium (not shown) and executed. Also, the machine learning model may be input to the memory unit 113 or the storage device 115 and operated by the processor 111 to be performed.

In addition, the biometric image reading support apparatus 100 may additionally store clinical information of an object, eg, an object (patient), in the memory unit 113 or the storage device 115, which will be described later. The processor 111 may extract the first attribute information 5 of the first biological image 1 based on a machine learning model by utilizing clinical information of an object stored in a memory or a storage device. Clinical information may include, but is not limited to, age, gender, medical history, medical examination information, test measurement values, exercise habits, eating habits, family history related to the above medical history, alcohol consumption and smoking status of the subject (patient). The medical history information may include a neurological medical examination that a doctor may perform on a patient, and may mean an ideal finding currently observed, unlike a medical history. In addition, as the test measurement value, for example, a value obtained by measuring intraocular pressure, blood pressure, blood glucose level, and the like may be considered.

The first attribute information 5 is information that can be supported by an entity such as a medical staff in predicting and diagnosing a disease. For example, when the first living body image 1 is a fundus image, the first attribution information 5 is information on an increased C/D ratio (Cup-to-Disc ratio) included in the fundus image, disc rim thinning information, information on retinal nerve fiber layer defect, information on retinal hemorrhage, etc., and these information may be tissue position, thickness, shape, pixel information It is not limited thereto, and any information capable of identifying the attributes of each tissue may be included. Based on this tissue information, the device 100 may predict and diagnose glaucoma in the fundus image of the subject.

In addition, when the first biological image 1 is a bone joint image, the first attribute information 5 includes joint space narrowing, osteophyte, sclerosis, and bone end defects included in the bone joint image. deformity) and the like.

The second biometric image 10 is an image paired with the first biometric image 1, and any image of which the first biometric image 1 can be a target and which can be the basis of the first biometric image 10 All can be used, the style of the second biometric image 10 is modified based on the second machine learning model 30, and the third biometric image 50 can be generated by the processor 111. .

Specifically, the second machine learning model 30 may include a first sub machine learning model 30a and a second sub machine learning model 30b. The style of the second biological image 10 is transformed by the processor 111 based on the first sub-machine learning model 30a, and accordingly, a plurality of adversarial style images from the second biological image 10 (Adversarial Style image) , 10a) can be generated. The generated hostile style image 10a may be stored in a system memory unit 113 or a storage device 115 to be described later. The first sub machine learning model 30a may be input to the processor 111 and executed, or may be input to a computer readable recording medium (not shown) and executed. In addition, the first machine learning model 30a may be input to the memory unit 113 or the storage device 115 and operated by the processor 111 to be performed. The hostile style image 10a is an image having second attribute information comparable to the first attribute information 1a of the first biological image 1 by mapping noise to the second biological image 10, and is a hostile style image ( 10a) The production method will be described later. The second attribute information is similar to the first attribute information described above, so a description thereof will be omitted.

The second sub-machine learning model 30b selects one of the plurality of adversarial style images 10a and fuses the input second body image 10 to form a third body image 50. can create Accordingly, second attribute information comparable to the first attribute information 1a of the first biological image 1 may be reflected in the third biometric image 50 .

2A is a diagram illustrating a process of generating a plurality of adversarial style images by a biometric image reading device according to an embodiment of the present invention, and FIG. 2B is a diagram illustrating a biometric image reading device according to an embodiment of the present invention. It is a diagram showing the process of generating a single adversarial style image according to predicted values by exemplarily.

Referring to FIG. 2A , the processor 111 may generate hostile style images 10a1, 10a2, and 10a3 by mapping

hostile noises

11a, 11b, and 11c to the second biological image 10. have. At this time, the

hostile noises

11a, 11b, and 11c are generated in a specific region of the second living image 10, for example, a Cup-to-Disc ratio region, a Disc Rim Thinning (DRT) region, and a retinal nerve fiber layer (Retinal Nervous Fiber Layer). It is noise that can be mapped identically or independently differently to the Nerve Fiber Layer, RNFL) area, etc., and the hostile style images (10a1, 10a2, 10a3, ) are mapped by each hostile noise (11a, 11b, 11c, ) It is an image created by

Referring to FIG. 2B, it includes second attribute information 40 obtained by changing the attribute information of the second biological image by mapping adversarial noise 11 to the second biological image 10. Creates an adversarial style image (AS Image, 10a) that At this time, mapping the hostile noise 11 may be performed by the processor 111 .

The hostile noise 11 controls the gradation level of R, G, and B pixels representing each labeled tissue (eg, retina, nerve fiber layer, cup, disc, etc.) in the second living body image 10. values for adjusting the color of R, G, and B pixels, values for adjusting the contrast ratio locally in the second biometric image, and the like, but are not limited thereto. 2 Any element that can change the attribute information of the biometric image can be included.

The hostile style image 10a including the second attribution information may be generated by mapping the hostile noise once, but a prediction value appearing for the second attribution information 40 obtained based on the machine learning model It can be generated by repeatedly mapping to converge to this set value.

As shown in the drawing, in the mapping method, when the set value is 1 and the predicted value according to the attribute information 15 of the second biometric image 10 obtained based on the machine learning model 13 is 0.58, the second biometric image After mapping the hostile noise for (10), mapping is repeated so that the predicted values (0.60, 0.66, 0.68, ?) of the hostile style image 10a obtained based on the same machine learning model converge to the set value 1.

In addition, although not shown, when the set value is 0 and the predicted value of the second biometric image 10 obtained based on the machine learning model is 0.58, hostile noise is repeatedly mapped to the second biometric image to create an adversarial style. Mapping is repeated so that the predicted values (0.56, 0.54, 0.52, ) of the image 10a converge to the set value 0. In more detail, the hostile noise can be mapped to the labeled tissue in the second living body image.

At this time, the meaning that the set value is 1 is a biometric image that is almost abnormal, and may correspond to a biometric image that has a finding that diseases (eg, glaucoma, osteoarthritis, etc.) can be predicted and diagnosed, and the set value is 0. is an almost normal biometric image, and may correspond to a biometric image that does not show that a disease cannot be predicted or diagnosed. Accordingly, the hostile style image 10a may be repeatedly reproduced according to the number of set values and may be generated in various forms. For example, the first unit set value specified from 0.1 to 0.3, the second unit set value specified from 0.31 to 0.6, the third unit set value specified from 0.61 to 0.9, and the fourth unit set value specified from 0.91 to 1.0 An adversarial style image can be generated by dividing into a plurality of unit setting value intervals as shown in FIG.

As such, the hostile style image 10a obtained by mapping the hostile noise to the second biological image 10 based on the machine learning model is generated by the processor 111 based on the above-described second sub-machine learning model 30b. The third biological image 50 is created by merging with the second biological image 10 again. The generated third biometric image 50 is displayed on the display unit, or provided to an external entity through a transmission module such as a display adapter 118 or a network adapter 119 to be described later, or linked to the computing device 110 through an Internet network. It may be provided to a remote computing device or other device that becomes a device.

Since the provided third biological image 50 may reflect the second attribute information, which is a biometric image close to normal or abnormal, the style of the third biological image 50 can be freely changed. Therefore, when the entity reads the first biological image 1, it is easy to compare and read the third biological image, and it is possible to explain the result of the reading because there is a comparison standard, and accordingly, the first attribute information 1a Reliability of reading the first living body image 1 having ? can be increased.

FIG. 3 exemplarily illustrates a process of generating a third biological image to be compared with a first biological image by a biometric image reading device according to an embodiment of the present invention. FIG. It is a diagram showing a second biometric image generated by the biometric image reading device and a hostile style by way of example.

3 and 4, the second biometric image 10 is input to the style machine learning model (Style Encoder, 31a) of the first sub machine learning model 30a, and the second biometric image 10 is encoded by a processor (not shown). Style data (Intermediate Style Data) may be generated from the image 10 . At this time, the generated style data may be image data. The second biological image 10 is the first biological image 1 captured by the biological image reading device 100 described in FIG. 1 or a fake image similar to the first biological image 1 to generate style data. ) can be. Based on the predictor 31b of the first sub-machine learning model 30a, a processor (not shown) may extract a prediction value for attribute information of the style data. The attribute information may be any information capable of supporting reading of the aforementioned biometric image. Thereafter, the adversarial style image 10a may be generated by comparing the predicted value of attribute information with a desired disease prediction value. The hostile style image 10a is generated in the same or similar manner as described above with reference to FIG. 2 and, as shown in FIG. 4 , may be expressed differently according to style data (eg, Style A and Style B).

The hostile style image 10a and the second biometric image 10 are input to the second sub-machine learning model 30b. The second sub-machine learning model 30b may generate various third biological images 50 by fusing the hostile style image 10a based on the second biological image 10 . The second sub-machine learning model (30b) is implemented with various learning algorithms including CycleGAN, GANMOOK, StarGAN, Real-time Style Transfer model learning algorithm based on deep learning, and Generative Adversarial Network or Style Transfer model depending on the amount of training data. It can be. The second sub machine learning model 220 according to an embodiment may calculate a loss using an identity loss function proposed by the present invention. This identity loss function can enable style conversion without damaging the structure of image data by selecting a semantically similar region during image data-style mapping.

Referring to FIG. 5 , a biometric image reading device 100 according to an embodiment of the present invention may include a computing device 110, a display device 130, and a camera 150. have. The computing device 110 includes a processor 111, a memory unit 113, a storage device 115, an input/output interface 117, a network adapter 118, and a display adapter. (Display Adapter, 119), and a system bus (System bus, 112) that connects various system components including a processor to the memory unit 113, but is not limited thereto. In addition, the biometric image reading support device may include a system bus 112 for transferring information as well as other communication mechanisms.

A system bus or other communication mechanism may include a processor, a memory that is a computer-readable recording medium, a short-range communication module (eg, Bluetooth or NFC), a network adapter including a network interface or mobile communication module, and a display device (eg, CRT or LCD, etc.), input devices (eg, keyboard, keypad, virtual keyboard, mouse, trackball, stylus, touch sensing means, etc.), and/or subsystems.

The processor 111 may be a processing module that automatically processes using the machine learning model 13, and may be a CPU, an application processor (AP), a microcontroller, etc., but is not limited thereto.

The processor 111 may display the operation of the biometric image reading support apparatus and the user interface on the display device 130 by communicating with the display adapter 119 , for example, a hardware controller for the display device.

The processor 111 controls the operation of the biometric image reading support apparatus according to an embodiment of the present invention to be described later by accessing the memory unit 113 and executing one or more sequences of commands or logic stored in the memory unit.

These instructions may be read into memory from static storage or other computer readable media such as a disk drive. In other embodiments, hard-wired circuitry may be used in place of or combined with software instructions to implement the present disclosure. Logic may refer to any medium that participates in providing instructions to the processor and may be loaded into the memory unit 113 .

System bus 112 is one of several possible types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. indicates an abnormality. For example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standard Association (VESA) local bus, Accelerated Graphics Port (AGP) Buses and Peripheral Component Interconnects (PCI), PCI-Express buses, Personal Computer Memory Card Industry Association (PCMCIA), and Universal Serial Bus (USB).

The system bus 112 can be implemented as a wired or wireless network connection. Processor (111), mass storage device (Mass Storage Device), operating system (Operating System, 113c, 115a), imaging software (Imaging Software, 113b, 115b), imaging data (Imaging Data, 113a, 115c), network A subsystem including an adapter (Network Adapter, 118), system memory, input/output interface (Input/Output Interface, 117), display adapter (119), and display device (Display Device, 130) Each may be included in one or more remote computing devices (Remote Computing Devices, 200, 300, 400) to be described later at physically separate locations, and may be connected through these types of buses to efficiently run a distributed system. .

Transmission media including the wires of the bus may include coaxial cable, copper wire, and optical fibers. In one example, transmission media may take the form of acoustic or light waves generated during radio wave communication or infrared data communication.

The apparatus 100 for supporting biometric image reading according to an embodiment of the present invention includes messages, data, information, and one or more programs (ie, application code) through a network link and a network adapter 118. Commands may be sent and received.

The network adapter 118 may include a separate or integrated antenna for enabling transmission and reception over a network link. The network adapter 118 may access a network and communicate with the remote computing device (Remote Computing Device, 200, 300, 400). The network may include, but is not limited to, at least one of a LAN, WLAN, PSTN, and cellular phone network.

The network adapter 118 may include at least one of a network interface and a mobile communication module for accessing the network. The mobile communication module can access a mobile communication network for each generation (eg, 2G to 5G mobile communication networks).

The program code may be executed by the processor 111 when received and/or may be stored for execution in a disk drive of the memory unit 113 or in a non-volatile memory of a type other than the disk drive.

The computing device 110 may be a variety of computer readable recording media. A readable medium can be any of a variety of media that can be accessed by a computing device, including, for example, volatile or non-volatile media, removable media, and non-removable media. removable media), but is not limited thereto.

The memory unit 113 may store an operating system, driver, application program, data, and database necessary for the operation of the device for supporting biometric image reading according to an embodiment of the present invention, but is not limited thereto. In addition, the memory unit 113 may include computer readable media in the form of volatile memory such as RAM (Random Access Memory), ROM (Read Only Memory) and non-volatile memory such as flash memory, and may also include a disk drive. For example, it may include, but is not limited to, a hard disk drive, a solid state drive, an optical disk drive, and the like. In addition, the memory unit 113 and the storage device 115 typically store data such as imaging data (Imaging Data, 113a, 115a) such as a biometric image of an object, which can be immediately accessed to be operated by the processor 111. It may include program modules such as

imaging software

113b and 115b and

operating systems

113c and 115c.

The machine learning model 13 may be inserted into a processor, a memory unit 113 or a storage device 115 . The machine learning model at this time may include a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc., which are one of the machine learning algorithms. Not limited.

A deep neural network (DNN) is an artificial neural network (ANN) consisting of several hidden layers between an input layer and an output layer. A deep neural network (DNN) can model complex non-linear relationships, just like a general artificial neural network. For example, in a deep neural network structure for an object identification model, each object may be represented as a hierarchical composition of basic elements of an image. In this case, the additional layers may consolidate the characteristics of the gradually gathered lower layers. This feature of deep neural networks allows complex data to be modeled with fewer units (units, nodes) compared to similarly performed artificial neural networks.

A convolutional neural network (CNN) is a type of multilayer perceptron designed to use minimal preprocessing. A convolutional neural network (CNN) consists of one or several convolutional layers and general artificial neural network layers placed on top of them, and additionally utilizes weights and pooling layers. Thanks to this structure, convolutional neural networks (CNNs) can fully utilize input data with a two-dimensional structure. Compared to other deep learning architectures, convolutional neural networks (CNNs) show good performance in both video and audio fields. Convolutional Neural Networks (CNNs) can also be trained via standard back-propagation. Convolutional neural networks (CNNs) are easier to train than other feedforward artificial neural network techniques and have the advantage of using fewer parameters.

In deep learning, a Convolutional Deep Belief Network (CDBN) has been developed. It is structurally very similar to the existing Convolutional Neural Network (CNN), so it can use the two-dimensional structure well, and at the same time, the Deep Belief Network (Deep Belief Network) You can also take advantage of pre-training in Belief Network (DBN). Convolutional deep neural networks (CDBNs) provide a general structure that can be used in a variety of image and signal processing techniques, and are used in several benchmark results for standard image data such as CIFAR.

A recurrent neural network (RNN) refers to a neural network in which connections between units constituting an artificial neural network constitute a directed cycle. A recurrent neural network can utilize memory inside the neural network to process arbitrary inputs. Due to these characteristics, the recurrent neural network is used in fields such as handwriting recognition and shows a high recognition rate.

The camera unit 150 includes an image sensor (not shown) that captures an image of an object and photoelectrically converts the image into an image signal, and captures a biometric image of the object. The captured biometric image of the object may be provided to the processor 111 through the input/output interface 117 and processed based on the machine learning model 13 or stored in the memory unit 113 or the storage device 115. . In addition, the photographed biometric image of the object may be provided to the

remote computing devices

200, 300, and 400 to be described later through an Internet network.

Referring to FIG. 6 , a biometric image reading support system 500 according to an embodiment of the present invention includes a computing device 310, a display device 330, a camera 350, and one or more It may include a remote computing device (Remote Compting Device, 200, 300, 400). The computing device 310 and the

remote computing devices

200, 300, and 400 may be connected to each other through an Internet network. Components included in the computing device (Computing Device) 310 are similar to the corresponding components in FIG. 5 described above, so descriptions of operations and functions thereof will be omitted. Also, components included in each of the

remote computing devices

200 , 300 , and 400 are similar to components of the computing device 310 .

The computing device 310 and the

remote computing devices

200, 300, and 400 may be configured to perform one or more of the methods, functions, and/or operations presented in the embodiments of the present invention. These

computing devices

310, 200, 300, and 400 may include an application running on at least one computing device. In addition,

computing devices

310, 200, 300, and 400 may include one or more computers and one or more databases, and may be single devices, distributed devices, cloud-based computers, or combinations thereof.

The biometric image reading device according to the present invention is not limited to laptop computers, desktop computers and servers, and can be implemented in a computing device or system capable of processing data and executing arbitrary commands, and can be used for other computing devices through an Internet network. It can be implemented as devices and systems. In addition, the biometric image reading device may be implemented in various ways including software including firmware, hardware, or a combination thereof. For example, functions to execute in various ways may be performed by components implemented in various ways, including discrete logic components, one or more application specific integrated circuits (ASICs), and/or program control processors.

Referring to FIG. 7 , the screen displayed on the display unit 130 may be provided as a graphical user interface (GUI) screen capable of adjusting attribute information of a biological image (eg, a third biological image, 50). , Property information of the biometric image created by the user may be obtained through the graphic user interface screen. The displayed screen may include a biometric image screen unit 130a and an image style control screen unit 130b. A first biological image 1 for reading and diagnosing based on attribute information and a third biological image 50 for comparison and determination with the first biological image 1 may be displayed on the biometric image display unit 130a. .

A scroll bar 131 for adjusting the style of the second biological image (not shown) may be displayed on the image style adjusting screen unit 130b. Styles (adversarial style images) may be created in various ways by the methods described above in FIGS. 2 to 4 , and various styles may have respective attribute information. At this time, the attribute information may be tissue attribute information labeled according to the type of style, and the style of the second biological image (not shown) may be changed according to the adjustment of the attribute information. As an embodiment, as shown, when the second living body image is an fundus image, various information including C/D ratio, retinal nerve fiber layer, disc rim thinning, and the like as attribute information are displayed on the style adjustment screen 130b. can be displayed

In this way, the third biometric image 50 expressed by freely adjusting the style (attribute information) of the second biometric image by the biometric image reading device 100 according to the present invention and the first biometric image 1 to be read ) is displayed on the screen of the display unit 130, since the first biological image 1 can be compared and read with the third biological image 50, more reliable reading is possible, and the reading result is logical. can be explained, so accurate reading can be made.

Referring to FIG. 8 , in step S810, when a first biological image of an object is acquired from an external device, for example, a camera or another device (not shown) interworking with the computing device 110, the processor 111 ) extracts the first attribute information from the first biological image based on the first machine learning model (3). In this case, the first biological image may be stored in the memory unit 113 or the storage device 115 .

In step S920, the style of the second biological image 10 is modified based on the second machine learning model 30, and the third biological image 50 is generated by the processor 111. At this time, the second biometric image 10 is an image paired with the first biometric image, and the second biometric image 10 is based on the first sub machine learning model 30a of the second machine learning model 30. The style is modified by the processor 111, and accordingly, a plurality of adversarial style images 10a may be generated from the second biological image 10. In addition, the second sub-machine learning model 30b selects one of the plurality of adversarial style images 10a and fuses the inputted second biological image 10 (Fusion & Aggregation) to form the third biological image 50. ) can be created. Accordingly, second attribute information comparable to the first attribute information 1a of the first biological image 1 may be reflected in the third biometric image 50 .

In step S930, the first biometric image having the first attribute information and the third biometric image having the second attribute information are displayed on the display unit. The displayed screen may include a biometric image screen unit 130a and an image style control screen unit 130b. On the biometric image display unit 130a, a first biometric image 1 for reading and diagnosing based on attribute information and a third biometric image 50 for comparison and determination with the first biometric image 1 may be displayed. , A scroll bar 131 for adjusting the style of the second biological image (not shown) may be displayed on the image style adjustment screen 130b.

As described above, the device and method for supporting biometric image reading according to the present invention can support explanatory biometric image reading in reading a biometric image based on a machine learning model, enabling medical staff to read biometric images more reliably. have.

As in the above embodiment, it can be clearly understood that the present invention can be achieved through a combination of software and hardware or can be achieved only by hardware. Objects of the technical solution of the present invention or parts contributing to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-readable recording medium. The machine-readable recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the machine-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software.

Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler.

The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa. The hardware device may include a processor such as a CPU or GPU coupled to a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, and may transmit and receive signals with external devices. A communication unit may be included. In addition, the hardware device may include a keyboard, mouse, and other external input devices for receiving commands written by developers.

As described above, the embodiments according to the present invention have been reviewed, and the fact that the present invention can be embodied in other specific forms in addition to the above-described embodiments without departing from the spirit or scope is apparent to those skilled in the art. It is self-evident to Therefore, the embodiments described above are to be regarded as illustrative rather than restrictive, and thus the present invention is not limited to the above description, but may vary within the scope of the appended claims and their equivalents.

Claims

processor; and

a memory comprising one or more instructions implemented to be executed by the processor, the processor comprising:

Extracting first attribute information from a first biological image based on a first machine learning model;

generating a third biological image having second attribute information comparable to the first attribute information by modifying a style of a second biological image paired with the first biological image based on a second machine learning model;

An apparatus for supporting biometric image reading that displays a first biological image having the first attribute information and a third biological image having the second attribute information on a display unit.
According to claim 1,

The second machine learning model,

a first sub-machine learning model and a second sub-machine learning model, wherein based on the first sub-machine learning model, the second biometric image is style-transformed and generated as an adversarial style image. support device.
According to claim 2,

The style is

An apparatus for supporting biometric image reading, characterized in that hostile noise is mapped to and transformed into the second biometric image.
According to claim 3,

The hostile noise,

a value for adjusting gradation levels of R, G, and B pixels of the second living body image, a value for adjusting color change of R, G, and B pixels of the second living body image; An apparatus for supporting biometric image reading, characterized in that it includes at least one of values adjusting a contrast ratio of the second biometric image.
According to claim 2,

The second sub machine learning model,

and generating the third body image by fusing the hostile style image based on the second body image.
According to claim 1,

The screen displayed on the display unit includes a biometric image screen for displaying the third biometric image and an image style adjusting screen for displaying a scroll bar for adjusting the style of the second biometric image. .
A method for supporting reading of a biometric image of an object by a processor, the method comprising:

extracting first attribute information from a first biological image of the object based on a first machine learning model;

generating a third biological image having second attribute information comparable to the first attribute information by modifying a style of a second biological image paired with the first biological image based on a second machine learning model; and

Displaying a first biological image having the first attribute information and a third biological image having the second attribute information

Biometric image reading support method comprising a.