KR102046720B1 - Method for adjusting color of image and apparatus using the same - Google Patents

Abstract

The present invention relates to a method for adjusting the color of an image and an apparatus using the same. Specifically, according to the method of the present invention, a computing apparatus obtains an image, classifies a class for each of the image elements, which are individual elements constituting the image and obtains a result image by adjusting the color of each of the image elements by referring to the class. Therefore, the accuracy and efficiency of image classification can be increased.

Description

METHOOD FOR ADJUSTING COLOR OF IMAGE AND APPARATUS USING THE SAME}

The present invention relates to a method for adjusting the color of an image and an apparatus using the same. Specifically, according to the method according to the present invention, the computing device acquires an image, classifies a class for each of the image elements which are individual elements constituting the image, and with reference to the class corresponding to each of the image elements The resultant image is obtained by adjusting the color of each of the image elements.

Medical images, such as those that can be preserved as a general RGB image, will change color somewhat depending on the characteristics of the imaging device and the influence of the surrounding environment. For example, in the case of pathological images, there is a color difference between the sample itself or the medical institution that generated the sample due to the difference in staining method, which is one of factors that may affect the judgment of the machine learning algorithm.

Scanning or microscopic images of lesions that can be seen with the naked eye from outside the skin, including skin diseases, or whole slides of sections of the body that contain cells or tissues. The diagnosis used plays a decisive role in the determination of treatment options and the prediction of prognosis for the patient. This diagnosis is mainly made by considering the type of tissue and cells. This diagnosis is one of the problems that is so difficult that there is a difference between not only medical practitioners in the same country but also between countries. The difficulty of this diagnosis largely depends on the doctor's experience and intuition. In practice, even an experienced doctor is often ambiguous to diagnose alone. This is one of the main reasons for the inconsistency of diagnosis between doctors and the waste of medical services such as advice and reread.

Therefore, in order to increase the efficiency of artificial intelligence diagnosis aid using medical images, it is necessary to reduce the color difference between images or construct a robust model for such color difference.

To this end, in the present disclosure, a model capable of distinguishing structures (eg, cell body, cytoplasm, background, etc.) included in the image to standardize or adjust the color of the image using a plurality of patches included in the color medical image. In this paper, we propose a system for constructing an artificial neural network model that learns how to standardize or adjust colors using them, and how to diagnose lesions by patch.

The present invention helps to prepare a robust model for the change of the dyeing or photographing method used in the image in learning by artificial intelligence, and can improve the reliability in the classification process, and the labeling data It is an object of the present invention to enhance the applicability of class classification on an image by allowing unsupervised learning to be performed even when there is no labeling data as well as supervised learning.

Accordingly, the present invention aims to ultimately improve the efficiency and accuracy in the determination of artificial intelligence based images, such as the diagnosis of medical images.

The characteristic constitution of the present invention for achieving the object of the present invention as described above and realizing the characteristic effects of the present invention described below is as follows.

According to an aspect of the present invention, there is provided a method of adjusting the color of an image, the method comprising: (a) causing a computing device to acquire an image or cause another device to be linked to the computing device; Assisting to obtain; (b) classifying, by the computing device, or classifying the classifier for each of the image elements that are individual elements constituting the image or allowing the other device to classify the class; And (c) the computing device obtaining a resultant image by adjusting the color of each of the image elements with reference to a class corresponding to each of the image elements, or obtaining the resultant image by allowing the other device to adjust the color. Supporting the steps.

According to another aspect of the present invention, there is also provided a computer program, stored on a machine-readable non-transitory recording medium comprising instructions implemented to perform a method according to the invention.

According to another aspect of the invention, there is provided a computing device for adjusting the color of an image, the device comprising: a communication unit for acquiring an image; And (i) classifying a class for each of the image elements, which are individual elements constituting the image, or supporting another device linked with the communication unit. And (ii) obtaining a result image by adjusting the color of each of the image elements with reference to a class corresponding to each of the image elements, or supporting the other device to obtain the result image by adjusting the color. It includes a processor to perform.

According to the present invention, there is an effect of facilitating an increase in accuracy and efficiency of classification of images.

In addition, according to the present invention, it is possible to store the classification information of the image with the image, which has a convenient effect.

Such an effect according to an embodiment of the present invention can be applied not only to a specific modality image but also to a wide variety of formats of 2D or 3D images, and thus the method of the present invention is not dependent on a specific format or platform. Of course not.

BRIEF DESCRIPTION OF THE DRAWINGS The following drawings, which are attached to be used in the description of the embodiments of the present invention, are merely some of the embodiments of the present invention, and to those skilled in the art to which the present invention pertains based on these drawings without efforts to reach a novel invention. Other drawings can be obtained.
1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device for performing a method of adjusting color of an image (hereinafter referred to as "image color adjusting method") according to the present invention.
2 is a block diagram schematically illustrating hardware or software components of an apparatus for performing a method for adjusting color of an image according to the present invention.
3 is a block diagram illustrating a process of obtaining an output image of which color is adjusted for each class from an input image according to the image color adjusting method according to the present invention.
FIG. 4 is a block diagram illustrating two embodiments of a class classification module shown in FIG. 3.
FIG. 5A is a block diagram illustrating an example of a module based on statistics as an example of the color adjustment module for each image class shown in FIG. 2.
FIG. 5B is a block diagram illustrating a machine learning module including an artificial neural network as another example of the image class color adjusting module shown in FIG. 2, and FIG. 5C is an overview of the modules shown in FIGS. 5A and 5B together. It is a block diagram.
6 is a view conceptually showing the configuration of the resultant image retained or provided according to the present invention as compared to the conventional image.

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced to clarify the objects, technical solutions and advantages of the invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

The terms "image", "image", "image data" or "image data" as used throughout the description and claims of the present invention refer to discrete image elements (e.g. pixels, two-dimensional images in two-dimensional images). Refers to multidimensional data consisting of voxels, or in other words, refers to a visually visible object (eg, displayed on a video screen) or a digital representation of that object (eg, a file corresponding to a pixel or voxel output). It is a term to refer to.

For example, an "image" may be an image of a subject collected according to any imaging method known in the art, such as imaging a microscope image. However, the image is not necessarily provided in a medical context, but may be provided in a non-medical context.

In the drawings provided for convenience of description, some exemplary image modalities are illustrated, but a person skilled in the art includes images captured by a microscope, although the image formats used in various embodiments of the present disclosure are necessarily included in the format. It will be appreciated that it is possible to include various images that are not limited and require scanning.

And throughout the description and claims of this invention, the word 'comprises' and variations thereof are not intended to exclude other technical features, additives, components or steps. In addition, "one" or "one" is used in more than one sense, and "other" is limited to at least a second or more.

A user referred to in the present disclosure may not only refer to singular but may also refer to a plurality of users whose purpose is to obtain an image, especially a pathological image and its related data in a rich and overlapping manner, including training of images. ) Or in order to secure the integrity of the image in actual use.

Other objects, advantages and features of the present invention will become apparent to those skilled in the art, in part from this description and in part from the practice of the invention. The following examples and drawings are provided by way of illustration and are not intended to limit the invention. Accordingly, the details disclosed in this disclosure with respect to particular structures or functions are not to be interpreted in a limiting sense, but merely to provide a guide for a person skilled in the art to variously implement the present invention with any suitable structures. It should be interpreted as representative basic data.

Moreover, the present invention encompasses all possible combinations of the embodiments indicated in this disclosure. It is to be understood that the various embodiments of the invention are different, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

Unless otherwise indicated in the present disclosure or clearly contradicted by context, the item referred to in the singular encompasses the plural unless the context otherwise requires. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device for performing an image color adjustment method according to the present invention.

Referring to FIG. 1, the computing device 100 according to an embodiment of the present invention includes a communication unit 110 and a processor 120, and directly or indirectly with an external computing device (not shown) through the communication unit 110. Can communicate with each other.

Specifically, the computing device 100 may comprise typical computer hardware (eg, a computer processor, memory, storage, input and output devices, devices that may include components of other existing computing devices; routers, switches, etc.). Electronic communication devices; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SANs) and computer software (ie, enabling computing devices to function in a particular manner). Combination of instructions) to achieve the desired system performance.

The communication unit 110 of such a computing device may transmit and receive a request and a response with another computing device to which the computing device is interworked. As an example, the request and response may be made by the same transmission control protocol (TCP) session. However, the present invention is not limited thereto, and may be transmitted and received as, for example, a user datagram protocol (UDP) datagram. In addition, in a broad sense, the communication unit 110 may include a keyboard, a mouse, another external input device, a printing device such as a printer, a display device, and other external output devices for receiving commands or instructions.

In addition, the processor 120 of the computing device may include a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU) or a tensor processing unit (TPU), and a cache memory. ), And a hardware configuration such as a data bus. In addition, the operating system may further include a software configuration of an application performing a specific purpose.

2 is a block diagram schematically illustrating hardware or software components of an apparatus for performing an image color adjusting method according to the present invention.

2, the computing device 100 may include a color adjustment module 200 for each image class as a component thereof. The color adjustment module 200 for each image class is configured to acquire an image to which the method according to the present invention is applied, classify the obtained image, and adjust the color of the image for each class according to the result. The color-adjusted result image may be input to the medical image diagnosis neural network module 300 and used for image diagnosis.

Information calculated in the processing process, such as a classification result of the class according to the present invention, a color-adjusted result image, or the like may be provided to at least a part of the information in an external entity, which is referred to as the external entity. It is understood that the subject includes a user, an administrator of the computing device 100, a medical person in charge of the contents shown in the image, and any subject that requires classification (determination, reading, etc.) of the image. Should be. The external entity may be an artificial intelligence entity using the information, or may be a storage device or an output device for storing the information.

As described above, the modules illustrated in FIG. 2 may be implemented by, for example, the communication unit 110 or the processor 120 included in the computing device 100, or the interworking of the communication unit 110 and the processor 120. It will be understood by those skilled in the art and will be described in more detail below with respect to the function and effect thereof. Although the modules illustrated in FIG. 2 are illustrated as being implemented in one computing device for convenience of description, the apparatus 100 for performing the method of the present invention may be configured such that a plurality of devices are interlocked with each other, which is attached to the present disclosure. It will be understood that it may be covered by the appended claims.

3 is a block diagram illustrating a process of obtaining an output image of which color is adjusted for each class from an input image according to the image color adjusting method according to the present invention.

Referring to FIG. 3, in the image color adjusting method according to the present invention, an image acquisition module (not shown) implemented by the computing device 100 acquires an image 1 or communicates with the communication unit 110 of the computing device. (S100) (not shown) for supporting another device (not shown) interworking with each other to acquire the image.

Next, in the image color adjustment method according to the present invention, the class classification module 220 of the color adjustment module 200 for each image class implemented by the computing device 100 constitutes the individual elements constituting the image 1. The method may further include classifying a class for each of the in-picture elements, or supporting the classifying device (S200) (not shown). The result of the classification is as illustrated by reference numeral 2 in FIG. 3. Classes are exemplified herein as, for example, cell bodies, cytoplasm, and the like.

FIG. 4 is a block diagram illustrating two embodiments 220a and 220b of the class classification module 220 shown in FIG. 3.

Referring to FIG. 4, the class classification module 220a according to the first embodiment of step S200 may include an autoencoder including a product multiplication layer 222a and an upsampling layer 224a; 222a and 224a and clustering module 228a. In this embodiment, the convolutional layer 222a and the upsampling layer 224a are trained through a training loss value, and the class classification module 220a is trained using a clustered module trained using intermediate feature values, which are values obtained from the convolutional layer. The classification information for the image 1 is obtained from the result value of 228a. Size adjustment 226a may be made to reconstruct the original size image from the feature value of the convolutional product layer. As described above, a database 240 for storing the classified results of the classes is illustrated as an example, but it is a matter of course that the database 240 is not essential.

4, the class classification module 220b according to the second embodiment of step S200 may be trained using U-Net or U-Net trained using a plurality of training images and training labeling information. And similar structure 222b. This training labeling information is also called ground truth in that it is correct data. In this embodiment, structure 222b is trained through a training loss value, and class classification module 220b performs segmentation using structure 222b. Likewise, database 240 is not essential.

Next, after the class is classified (2) in step (S200), the image color adjustment method according to the present invention, the statistics-based image of the color adjustment module 200 for each image class implemented by the computing device 100 The standardization module 280a or the class-specific color change module 280b adjusts the color of each of the image elements by referring to the class corresponding to each of the image elements, thereby obtaining the resultant image 3 or causing the other device to perform the operation. The method may further include assisting to obtain the resultant image 3 by adjusting the color (S300; not shown).

FIG. 5A is an exemplary block diagram illustrating a module 200a based on a statistic based method as an embodiment of the color class adjusting module for each image class shown in FIG. 2, and FIG. 5B is a color class adjusting module for each image class shown in FIG. 2. As another embodiment of the present invention, a block diagram showing an example of a machine learning module 200b including an artificial neural network is illustrated. FIG. 5C is a block diagram of the modules shown in FIGS. 5A and 5B together.

Referring to FIG. 5A, an embodiment in which the statistics-based image standardization module 280a is used in step S300 is illustrated. In this embodiment, the statistics-based image standardization module 280a uses statistics of color information for each class. Adjust the color.

In detail, when each image is obtained from a database including a medical image, the color adjustment module 200a for each image class may classify each class from the database 260a using the classification information obtained by the class classification module 220. A statistical value of color information is obtained and image normalization is performed using the image information (280a).

As an example, for all pixels included in the image 1, the color adjustment module 200a for each image class classifies the class using an RGB channel, a LAB channel, or another channel, and converts the class into a LAB channel. After that, the average and variance of each channel can be calculated for each class. Using these averages and variances, a statistical-based color standardization method known as Color Transfer (Reinhard et al., 2001) can be applied, and images can be normalized by class.

In addition, the user can specify the number of classes (natural numbers) as necessary. When the number of classes is 1, the class classification is equivalent to the pixel-wise classification.

Next, referring to FIG. 5B, another embodiment in which the color random change module 280b for each class is used in step S300 is illustrated. In this embodiment, the color random change module 280b for each class is determined by a user. The color of each class of the image is changed in response to the designated class-specific adjustment variables or the class-specific adjustment variables provided by the trained designated module.

In detail, when each image is obtained from a database including a medical image, the color adjustment module 200b for each image class randomly changes colors for each class by using classification information obtained by the class classification module 220. Image normalization is performed (280b).

As an example, for every pixel included in the image 1, the color adjustment module 200b for each image class divides the class, and randomly changes the color within a range specified by the user for each class. The user can specify the number of classes (natural numbers) as needed. If the number of classes is 1, this is equivalent to the pixel-wise classification. The images of which the color is arbitrarily changed may be used for training of the medical imaging diagnostic neural network module 300.

For reference, FIG. 5C illustrates the embodiment of FIG. 5A and the embodiment of FIG. 5B in one drawing.

Referring back to FIG. 3, the image color adjustment method of the present invention may further include the step of retaining or providing the obtained result image to an external entity, or supporting the other device to retain or provide the image (S400). Can be.

6 is a view conceptually showing the configuration of the resultant image retained or provided according to the present invention as compared to the conventional image. Typically, the image 600a includes R channel images 610a, B channel images 620a, and G channel images 630a corresponding to respective color channels, for example, R channels, B channels, and G channels. For example, the resultant image 3 (600b) may include R, B, and G channel images 610b, 620b, and 630b, as well as classification information about the image, for convenience in holding or providing in operation S400. (2) may be included as at least one channel 640b to 660b. In the case of storing as a probability or confidence corresponding to each class without dealing with the class as a binary classification or a chopped classification, that is, when each channel corresponds to each class, the at least One channel may be provided as many as the number of classes. If the number of classes is three, the C1 channel image 640b, the C2 channel image 650b, and the C3 channel image 660b may be included.

On the other hand, the image color adjustment method of the present invention, the computing device 100 to train a predetermined machine learning model, for example, artificial neural network model configured to classify the class using the resultant image or to let the other device to train. Supporting step (S500) may further include. The predetermined machine learning model may be, for example, a model included in the aforementioned medical imaging diagnostic neural network module 300 implemented by the computing device 100, for diagnosing a medical image.

According to the description so far, the present invention has the effect of facilitating the improvement of the classification and the efficiency of the classification of the image by performing the color adjustment for the image for each class in all the embodiments and modifications thereof. Having been described, based on the description of the above embodiments, one of ordinary skill in the art will appreciate that the method and / or processes of the present invention and the steps of the hardware and software are suitable for the specific application. It can be clearly understood that it can be realized in combination.

The functions and process steps described above may be performed automatically or in response to all or some user commands. Activity (including steps) performed automatically is performed in response to one or more executable instructions or device operation without direct initiation by the user of the action. .

The hardware may include a general purpose computer and / or a dedicated computing device or a particular aspect or component of a particular computing device or a specific computing device. The processes may be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices having internal and / or external memory. In addition, or alternatively, the processes may be configured to process an application specific integrated circuit (ASIC), a programmable gate array, a programmable array logic (PAL), or electronic signals. It can be implemented in any other device or combination of devices that are present. Furthermore, the objects of the technical solution of the present invention or the portions contributing to the prior art can be embodied in the form of program instructions that can be executed by various computer components and recorded in a machine-readable recording medium. The machine-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the machine-readable recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in the computer software arts. Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, Blu-rays, and magnetic-optical media such as floptical disks. (magneto-optical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include, but are not limited to storing and compiling or interpreting for execution on any one of the foregoing devices, as well as on a machine capable of executing a heterogeneous combination of a processor, processor architecture or combinations of different hardware and software, or any other program instructions. Can be made using a structural programming language such as C, an object-oriented programming language such as C ++, or a high or low level programming language (assembly, hardware description languages and database programming languages and technologies), including machine code, This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter.

Thus, in one aspect according to the present invention, when the methods and combinations described above are performed by one or more computing devices, the methods and combinations of methods can be implemented as executable code that performs each step. In another aspect, the method may be practiced as systems that perform the steps, and the methods may be distributed in various ways across the devices or all the functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any hardware and / or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware device may be configured to operate as one or more software modules to perform a process according to the present invention, and vice versa. The hardware device may include a processor, such as an MPU, CPU, GPU, TPU, coupled with a memory, such as a ROM / RAM for storing program instructions, and configured to execute the instructions stored in the memory, the external device and the signal It may include a communication unit that can send and receive. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving instructions written by developers.

Although the present invention has been described in terms of specific embodiments such as specific components and the like, and limited embodiments and drawings, the systems and processes of the accompanying drawings are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the present invention to achieve the same purpose. Although the present invention has been described with reference to specific embodiments, it will be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the design of the present disclosure may be implemented by those skilled in the art without departing from the scope of protection of the present invention. Various systems, ancillary systems, agents, managers, and processes as described in this disclosure can be implemented using hardware components, software components, and / or combinations thereof. .

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the appended claims, fall within the scope of the spirit of the present invention. I will say.

Such equivalent or equivalent modifications will include, for example, logically equivalent methods capable of producing the same results as those of the method according to the invention, the spirit and scope of the invention being Is not to be limited by the foregoing examples, but is to be understood in the broadest sense acceptable by law.

Claims (13)

In the method of adjusting the color of the image,
(a) the image obtaining module implemented by the computing device, obtaining an image or supporting another device linked to the computing device through a communication unit of the computing device to acquire the image;
(b) a class classification module implemented by the computing device, classifying a class for each of the image elements which are individual elements constituting the image, or supporting the other device to classify the class;
(c) the image element in a direction in which a statistical based image standardization module or a color change module for each class implemented by the computing device refers to a class corresponding to each of the image elements to perform image normalization corresponding to the class. Acquiring a result image from the image or the other device to obtain the result image from the image; And
(d) the communication unit of the computing device retaining or providing the resultant image to an external entity or assisting the other device to hold or provide it.
Including,
In step (c),
Performed using statistics of color information per class, or
Performed by changing the color of a class in response to class-specific adjustment variables provided by a user-designated or trained assignment module
Characterized by,
And the resultant image includes color channels and at least one channel representing classification information about the image. delete The method of claim 1,
(e) the processor of the computing device training or assisting the other device to train a machine learning model configured to classify the class using the resulting image.
Image color adjustment method further comprising. The method of claim 1,
In step (b),
Performed using an autoencoder and a clustering module comprising a multiplication product layer and an upsampling layer, the classification of the class using an intermediate feature value that is a value obtained in the multiplication product layer. And classifying information on the image from the trained clustering module result. The method of claim 1,
In step (b),
And a segmentation using a U-Net trained using a plurality of training images and training labeling information or a similar structure of the U-Net. delete delete delete The method of claim 1,
And the at least one channel corresponds to one class each. A computer program stored on a machine-readable non-transitory recording medium comprising instructions implemented to cause a computing device to perform the method of any one of claims 1, 3-5 and 9. . In a computing device that adjusts the color of an image,
Communication unit for obtaining an image; And
(i) a process of classifying a class for each of the image elements, which are individual elements constituting the image, or allowing other devices to be interlocked through the communication unit to classify; And (ii) obtaining a resultant image from the image or the other device as a result of adjusting the color of each image element in a direction in which image standardization corresponding to the class is performed by referring to a class corresponding to each of the image elements. A processor that performs a process for assisting the user to obtain the resultant image from the image
Including,
The processor,
(iii) further performing a process of retaining or providing the resultant image to an external entity, or assisting the other device to retain or provide,
The process (ii),
Performed using statistics of color information per class, or
Performed by changing the color of a class in response to class-specific adjustment variables provided by a user-designated or trained assignment module
Characterized by,
The resulting image,
And color classification information of the image as at least one channel. delete The method of claim 11,
The processor,
And training a predetermined machine learning model configured to classify a class using the resultant image, or assisting the other apparatus to train.

Images