CN113706400A

CN113706400A - Image correction method, image correction device, microscope image correction method, and electronic apparatus

Info

Publication number: CN113706400A
Application number: CN202110361857.6A
Authority: CN
Inventors: 张军; 田宽
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-04-02
Filing date: 2021-04-02
Publication date: 2021-11-26

Abstract

The disclosure provides an image correction method and device, a microscope image correction method and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an image to be corrected, and performing feature extraction on the image to be corrected through a light field prediction model to acquire a three-channel background light field image corresponding to the image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information; correcting the image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the image to be corrected; the light field prediction model is trained on the basis of a plurality of non-standard simulation image samples and three-channel background light field image samples corresponding to the non-standard simulation image samples. The method and the device can realize synchronous background light field correction and color white balance on a single image, and improve the image quality.

Description

Image correction method, image correction device, microscope image correction method, and electronic apparatus

Technical Field

The present disclosure relates to the field of artificial intelligence technologies, and in particular, to an image correction method, an image correction apparatus, a microscope image correction method, a computer-readable medium, and an electronic device.

Background

Due to the fact that light sources are not uniform and camera imaging is achieved, the photographed picture is often uneven in brightness, brightness distortion of the image is caused, and meanwhile color distortion is caused due to the difference of cold and warm colors of the light sources. For example, in the image taken by the microscope, due to uneven light source, different cold and warm colors and imaging reasons, the taken microscope image has uneven brightness and color distortion, and as shown in fig. 1, the microscope image has dark shadow regions at four corners and the color of the whole image is dark, which has a great influence on the accuracy of the analysis result of the microscope image.

At present, in order to correct the background and the shadow in the image, the image correction is usually performed by using an associated image correction tool, for example, a BaSic tool or the like, but this correction method needs a plurality of different images with the same shooting condition to estimate the background field, and cannot correct a single image. The method of correcting the image with uneven brightness and color distortion through the machine learning model is also provided, but the model is usually trained by adopting a standard image sample without uneven brightness and color distortion and a corresponding image sample with uneven brightness and color distortion, and then the trained model processes the image to be processed with uneven brightness and color distortion to directly obtain a corresponding corrected image.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

Embodiments of the present disclosure provide an image correction method, an image correction apparatus, a microscope image correction method, a computer-readable medium, and an electronic device, which can further achieve background light field correction and image white balance of an image synchronously at least to a certain extent, thereby further improving image quality.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.

According to an aspect of an embodiment of the present disclosure, there is provided an image rectification method including: acquiring an image to be corrected, and performing feature extraction on the image to be corrected through a light field prediction model to acquire a three-channel background light field image corresponding to the image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information; correcting the image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the image to be corrected; the light field prediction model is trained on the basis of a plurality of non-standard simulation image samples and three-channel background light field image samples corresponding to the non-standard simulation image samples.

According to an aspect of an embodiment of the present disclosure, there is provided a method of correcting a microscope image, including: acquiring a microscope image to be corrected, and performing feature extraction on the microscope image to be corrected through a light field prediction model to acquire a three-channel background light field image corresponding to the microscope image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information; correcting the microscope image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the microscope image to be corrected; the light field prediction model is trained on a plurality of non-standard simulation microscope image samples and three-channel background light field image samples corresponding to the non-standard simulation microscope image samples.

According to an aspect of an embodiment of the present disclosure, there is provided an image rectification apparatus including: the model processing module is used for acquiring an image to be corrected, and performing feature extraction on the image to be corrected through a light field prediction model to acquire a three-channel background light field image corresponding to the image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information; the image correction module is used for correcting the image to be corrected according to the three-channel background light field image so as to obtain a corrected image corresponding to the image to be corrected; the light field prediction model is trained based on a simulation image sample and a three-channel background light field image sample corresponding to the simulation image sample.

In some embodiments of the present disclosure, based on the above scheme, the model processing module is configured to: extracting the background light field image information of three color channels in the image to be corrected through an end-to-end full convolution neural network model, and determining the three-channel background light field image according to the extracted background light field image information of the three color channels.

In some embodiments of the present disclosure, based on the above scheme, the image rectification module includes: and the correcting unit is used for correcting the image information corresponding to the three color channels in the image to be corrected according to the three-channel background light field image so as to obtain a corrected image corresponding to the image to be corrected.

In some embodiments of the present disclosure, the image to be rectified includes R-channel image information, G-channel image information, and B-channel image information; the three-channel background light field image comprises R channel prediction background light field information, G channel prediction background light field information and B channel prediction background light field information; based on the scheme, the correcting unit is configured to: dividing the R channel image information by the R channel prediction background light field information to obtain R channel correction image information; dividing the G channel image information by the G channel predicted background light field information to obtain G channel corrected image information; dividing the B channel image information by the B channel prediction background light field information to obtain B channel correction image information; and acquiring the corrected image according to the R channel corrected image information, the G channel corrected image information and the B channel corrected image information.

In some embodiments of the present disclosure, based on the above solution, the image rectification apparatus further includes: the information acquisition module is used for acquiring a standard image set containing a plurality of standard images and acquiring a light field brightness variation range and a white balance coefficient value range; the parameter determining module is used for determining a simulated light field based on the light field brightness variation range and determining white balance sample information according to the white balance coefficient value range, wherein the simulated light field and the standard image have the same size; the three-channel background light field image sample generation module is used for determining the three-channel background light field image sample according to the simulation light field and the white balance sample information; and the training module is used for performing superposition processing on the standard images in the standard image set according to the three-channel background light field image sample to obtain the non-standard simulation image sample, and training a light field prediction model to be trained according to the non-standard simulation image sample and the three-channel background light field image sample.

In some embodiments of the present disclosure, based on the above scheme, the simulated light field is a randomly generated image with gaussian-like distribution with a non-fixed center position and a non-fixed variance; the white balance sample information includes white balance coefficient samples corresponding to three color channels, and each of the white balance coefficient samples is a random number that conforms to uniform distribution and is independent of each other.

In some embodiments of the present disclosure, the white balance sample information includes an R channel white balance coefficient, a G channel white balance coefficient, and a B channel white balance coefficient; based on the above scheme, the three-channel background light field image sample generation module is configured to: multiplying brightness information corresponding to each pixel in the simulated light field by the R channel white balance coefficient, the G channel white balance coefficient and the B channel white balance coefficient respectively to obtain R channel background light field information, G channel background light field information and B channel background light field information; and determining the three-channel background light field image sample based on the R-channel background light field information, the G-channel background light field information and the B-channel background light field information.

In some embodiments of the present disclosure, based on the above scheme, the training module is configured to: inputting the non-standard simulation image sample into the to-be-trained light field prediction model, and performing feature extraction on the non-standard simulation image sample through the to-be-trained light field prediction model to obtain a three-channel background light field prediction image; determining a first loss function according to the predicted three-channel background light field image and a three-channel background light field image sample corresponding to the non-standard simulation image sample; and optimizing parameters of the light field prediction model to be trained based on the first loss function so as to obtain the light field prediction model.

In some embodiments of the present disclosure, based on the above solution, the image rectification apparatus is further configured to: constructing a second loss function according to image information corresponding to three color channels of each pixel in the predicted three-channel background light field image; and optimizing the parameters of the light field prediction model to be trained on the basis of the first loss function and the second loss function so as to obtain the light field prediction model.

In some embodiments of the present disclosure, based on the above scheme, the first loss function is calculated according to formula (1):

wherein i is an RGB triple channel, i is 1,2,3, (x, y) is a position coordinate of any pixel in the non-standard analog image sample or the predicted triple channel background light field image, M is a maximum value of an abscissa of the non-standard analog image sample or the predicted triple channel background light field image, N is a maximum value of an ordinate of the non-standard analog image sample or the predicted triple channel background light field image, F is a maximum value of an ordinate of the non-standard analog image sample or the predicted triple channel background light field image, and F is a maximum value of an abscissa of the predicted triple channel background light field image_i(x, y) is a three-channel background light field image sample, F ', corresponding to the non-standard analog image sample'_i(x, y) is the predicted three-channel background light field image.

In some embodiments of the present disclosure, based on the above scheme, the second loss function is calculated according to formula (2):

wherein i is an RGB triple channel, i is 1,2,3, (x, y) is a position coordinate of any pixel in the non-standard simulation image sample or the predicted triple channel background light field image, M is a maximum value of an abscissa of the non-standard simulation image sample or the predicted triple channel background light field image, N is a maximum value of an ordinate of the non-standard simulation image sample or the predicted triple channel background light field image, and F'_i(x, y) is the predicted three-channel background light-field image,

is the difference operator.

In some embodiments of the present disclosure, based on the above solution, the image rectification apparatus is further configured to: performing feature extraction on the corrected image through the light field prediction model to obtain a three-channel background light field image corresponding to the corrected image; re-correcting the corrected image according to the three-channel background light field image corresponding to the corrected image to obtain an optimized corrected image; and repeating the steps, and performing iterative correction on the optimized corrected image until an optimal corrected image is obtained.

According to an aspect of an embodiment of the present disclosure, there is provided an apparatus for correcting a microscope image, including: the characteristic extraction module is used for acquiring a microscope image to be corrected, and performing characteristic extraction on the microscope image to be corrected through a light field prediction model so as to acquire a three-channel background light field image corresponding to the microscope image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information; the microscope image correction module is used for correcting the microscope image to be corrected according to the three-channel background light field image so as to obtain a corrected image corresponding to the microscope image to be corrected; the light field prediction model is trained on a plurality of non-standard simulation microscope image samples and three-channel background light field image samples corresponding to the non-standard simulation microscope image samples.

According to an aspect of an embodiment of the present disclosure, there is provided a computer storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the image rectification method or the microscope image rectification method provided in the above-mentioned alternative implementation.

According to an aspect of an embodiment of the present disclosure, there is provided a computer program product or a computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image rectification method or the microscope image rectification method provided in the above-described alternative implementation.

According to an aspect of an embodiment of the present disclosure, there is provided an electronic device including: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method provided in the above-described alternative implementations.

In the technical solutions provided in some embodiments of the present disclosure, a light field prediction model to be trained is trained first to obtain a stable light field prediction model; then, carrying out feature extraction on a single image to be corrected with brightness distortion and color distortion through a light field prediction model to obtain a three-channel background light field image corresponding to the single image to be corrected, wherein the three-channel background light field image correspondingly comprises brightness information and white balance information; and finally, correcting the image to be corrected according to the three-channel background light field image, and acquiring a corrected image corresponding to the image to be corrected. The technical scheme disclosed by the invention can realize synchronous background light field correction and color white balance on a single image, and improves the image quality.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty. In the drawings:

fig. 1 shows a schematic diagram of an exemplary system architecture to which technical aspects of embodiments of the present disclosure may be applied;

FIG. 2 schematically illustrates a flow diagram of an image rectification method according to an embodiment of the present disclosure;

FIG. 3 schematically shows a structural diagram of a light field prediction model according to one embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow diagram for acquiring a rectified image according to one embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow diagram for training a light field prediction model to be trained, according to one embodiment of the present disclosure;

FIG. 6 schematically illustrates a flowchart for training a light field prediction model to be trained from a non-standard simulation image sample and a three-channel background light field image sample, according to one embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow diagram of a method of rectification of a microscope image according to one embodiment of the present disclosure;

FIG. 8 schematically illustrates an interface diagram of a method of rectification of a microscope image according to one embodiment of the present disclosure;

9A-9C schematically illustrate interface schematic diagrams of a non-standard analog microscope image sample and a three-channel background light field image sample generated based on a standard slide image, according to one embodiment of the present disclosure;

10A-10C schematically illustrate interface views of pre-and post-corrective microscope images according to one embodiment of the present disclosure;

FIG. 11 schematically illustrates an interface diagram of a microscope image corrected by a method of correction in the present disclosure and a related art method of correction, according to one embodiment of the present disclosure;

FIG. 12 schematically illustrates a block diagram of an image rectification apparatus according to an embodiment of the present disclosure;

FIG. 13 schematically illustrates a block diagram of a rectification apparatus for microscope images according to one embodiment of the present disclosure;

FIG. 14 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

Fig. 1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present disclosure may be applied.

As shown in fig. 1, system architecture 100 may include terminal device 101, network 102, and server 103. The terminal device 101 may be a terminal device having a shooting unit, such as a smart phone, a portable computer, a tablet computer, a video camera, a camera, and a photographing microscope; the network 102 is a medium used for providing a communication link between the terminal device 101 and the server 103, and the network 102 may include various connection types, such as a wired communication link, a wireless communication link, and the like, and in the embodiment of the present disclosure, the network 102 between the terminal device 101 and the server 103 may be a wireless communication link, and particularly may be a mobile network.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. It is worth mentioning that the server in the present disclosure may be an independent server or a server cluster formed by a plurality of servers.

In an embodiment of the present disclosure, after a user shoots a target scene through a shooting unit in the terminal device 101, a corresponding image can be obtained, and because there may be uneven light sources, cool and warm colors of the light sources, and camera imaging reasons of the terminal device during shooting, the shot image has problems of brightness distortion and color distortion more or less. The image to be corrected having this problem may be transmitted from the terminal apparatus 101 to the server 103 via the network 102, so that the image correction module mounted in the server 103 corrects the image to obtain a corrected image in which the luminance distortion and the color distortion are corrected. Specifically, a light field prediction model is arranged in the image correction module of the server 103, and the light field prediction model can perform feature extraction on the received image to be corrected to output a three-channel background light field image corresponding to the received image to be corrected, and then remove the three-channel background light field image from the image to be corrected to obtain a corrected image. Because the three-channel background light field image comprises the brightness information and the white balance information, when the three-channel background light field image is removed from the image to be corrected, the brightness and the chromaticity in the image to be corrected can be adjusted simultaneously, and the corrected image without brightness distortion and color distortion is obtained.

Specifically, the image to be corrected may be a microscope image obtained by shooting a slide image displayed in an eyepiece by a photographable microscope, and after the microscope image is subjected to feature extraction by using a light field prediction model to obtain a three-channel background light field image corresponding to the microscope image, the microscope image may be corrected according to the three-channel background light field image to obtain a standard microscope image. The standard microscope image refers to that the brightness in the microscope image can be restored to the brightness of a common scanning image, and the chromaticity can reach uniform chromaticity.

The image correction method and the microscope image correction method provided by the embodiments of the present disclosure are generally executed by a server, and accordingly, the image correction device and the microscope image correction device are generally installed in the server. However, in other embodiments of the present disclosure, the image correction method and the microscope image correction apparatus provided in the embodiments of the present disclosure may be executed by a terminal device. That is to say, after the terminal device obtains the image to be corrected, the three-channel background light field image in the image to be corrected can be extracted through the built-in light field prediction model, and then the three-channel background light field image is removed from the image to be corrected, so that the corresponding corrected image can be obtained.

As described in the background art, in the related art in this field, the luminance and the chromaticity in the image are mainly corrected by the image correction tool, but this correction method requires a plurality of different images of the same shooting condition to estimate the background field, and cannot correct a single image. Meanwhile, in the related art in the field, image processing is performed through a machine learning model to correct an image with brightness distortion and color distortion, but during the model processing, an input image is firstly downsampled, and then prediction is performed based on the downsampled image, so that even if the output image is upsampled to the original size, the precision of the output image is poor, the image quality is reduced, namely the output corrected image still has the problems of brightness distortion and color distortion, and in addition, due to the fact that the number of training samples is not large enough, the stability of the trained model is poor, the corrected image still has the conditions of uneven brightness and color distortion, and the correction effect is poor.

In view of the problems in the related art, the embodiments of the present disclosure provide an image correction method and a microscope image correction method, which are implemented based on machine learning, which is one of Artificial Intelligence (AI), which is a theory, method, technique, and application system that simulates, extends, and expands human Intelligence, senses an environment, acquires knowledge, and uses the knowledge to obtain an optimal result using a digital computer or a machine controlled by a digital computer. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (CV) Computer Vision is a science for researching how to make a machine "see", and further refers to that a camera and a Computer are used to replace human eyes to perform machine Vision such as identification, tracking and measurement on a target, and further image processing is performed, so that the Computer processing becomes an image more suitable for human eyes to observe or transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technologies, virtual reality, augmented reality, synchronous positioning, map construction, and other technologies, and also include common biometric technologies such as face recognition and fingerprint recognition.

Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal education learning.

With the research and progress of artificial intelligence technology, the artificial intelligence technology is developed and applied in a plurality of fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, unmanned aerial vehicles, robots, smart medical care, smart customer service, and the like.

The scheme provided by the embodiment of the disclosure relates to an artificial intelligence image processing technology, and is specifically explained by the following embodiment:

the embodiment of the present disclosure first provides an image rectification method, and details of implementation of the technical solution of the embodiment of the present disclosure are set forth in the following:

fig. 2 schematically illustrates a flow diagram of an image rectification method according to an embodiment of the present disclosure, which may be performed by a server, which may be the service 103 shown in fig. 1. Referring to fig. 2, the image rectification method at least includes steps S210 to S220, and the following details are described as follows:

in step S210, an image to be corrected is obtained, and feature extraction is performed on the image to be corrected through a light field prediction model to obtain a three-channel background light field image corresponding to the image to be corrected, where the three-channel background light field image includes luminance information and white balance information.

In an embodiment of the present disclosure, when an image is captured, no matter in an indoor scene or in an outdoor scene, there may be a phenomenon that a captured image has uneven brightness due to uneven brightness of a light source and uneven brightness due to camera imaging, which causes brightness distortion of the image, that is, an abnormality of a background light field, and at the same time, due to a difference of cold and warm colors of the light source, there is color distortion of the image, that is, an abnormality of white balance. In the embodiment of the present disclosure, an image with brightness distortion and color distortion is named as an image to be corrected, and the image to be corrected not only has low image quality and affects the appearance, but also has disadvantages in other aspects, for example, when the image to be corrected is a pathological image, a device damage image, and the like, a final pathological analysis result and a damage detection result may have a large error due to the brightness distortion and the color distortion existing in the image, and a final diagnosis conclusion and a detection result may be wrong. Therefore, correction is necessary for an image having luminance distortion and color distortion.

In one embodiment of the present disclosure, the rectification of the image to be rectified may be divided into two steps: the first step is as follows: extracting a three-channel background light field image in an image to be corrected; the second step is that: and correcting the image to be corrected according to the three-channel background light field image. For the first step, in the embodiment of the present disclosure, a light field prediction model is used to extract a three-channel background light field image, where the light field prediction model may be an end-to-end full convolution network model with any network structure, such as U-Net, LinkNet, and the like, where end-to-end means that an image to be corrected is input, and a three-channel background light field image corresponding to the image to be corrected and having the same image size is output, instead of information such as prediction classification.

After the image to be corrected is input into the light field prediction model, the light field prediction model can perform feature extraction on the image to be corrected to obtain background light field image information of three color channels, namely R, G, B background light field image information of the three channels, and further determine a three-channel background light field image to be output according to the extracted background light field image information of R, G, B three channels. Next, the structure and the operation principle of the light field prediction model will be described by taking LinkNet as a light field prediction model as an example.

Fig. 3 shows a schematic structural diagram of a light field prediction model, as shown in fig. 3, the light field prediction model includes a convolutional layer 301, a max-pooling layer 302, a coding layer 303, a decoding layer 304, a full convolutional layer 305, a convolutional layer 306, and a full convolutional layer 307, where the coding layer 303 includes a plurality of coding blocks (coding block 1, coding block 2, coding block 3, and coding block 4), the decoding layer 304 includes the same number of decoding blocks (decoding block 1, decoding block 2, decoding block 3, and decoding block 4) as the coding blocks, and each coding block is connected with a decoding block, and an input of a coding block is connected to an output of a corresponding decoding block. In addition, fig. 3 also shows parameters in the convolutional layer 301, the max-pooling layer 302, the full-convolutional layer 305, the convolutional layer 306, and the full-convolutional layer 307, and it is worth to say that the parameter setting is only an exemplary illustration, and may also be set as other parameter values, which is not specifically limited in this disclosure.

When the light field prediction model shown in fig. 3 is used to extract the features of the image to be corrected, the feature extraction and downsampling are performed on the image to be corrected through the convolution layer 301, then the feature map output by the convolution layer 301 is pooled and downsampled through the maximum pooling layer 302, and then the input pooled feature map is encoded and compressed through the encoding layer 303, so that low-layer semantic feature information with low dimensionality, such as color information and luminance information, is obtained. After the coding layer 303 finishes coding and compressing the pooled feature map, the low-layer semantic feature information may be input into the decoding layer 304, so that the decoding layer 304 performs decoding operation on the low-layer semantic feature information, and feature extraction and upsampling are sequentially performed on the feature information output by the decoding layer 304 through the full convolution layer 305, the convolution layer 306 and the full convolution layer 307, wherein the times of upsampling and downsampling are the same, so as to output a three-channel background light field image with the same original size as that of the image to be corrected. Because each coding block is connected with the decoding block, and the input of the coding block is connected with the output of the corresponding decoding block, the coding block can integrate the low-layer semantic feature information into the decoding block, so that the decoding block integrates the low-layer semantic feature information and the high-layer semantic feature information, the lost space information during the down-sampling operation can be effectively reduced, and the decoding block shares the parameters learned from each layer of the coding block, thereby effectively reducing the parameters of the decoding block.

The structure and the working principle of the light field prediction model adopted by the embodiment of the invention are analyzed, the three-channel background light field information of the down-sampled image is predicted after the input image to be corrected is down-sampled in the embodiment of the invention, then the up-sampling is carried out to restore the original size, and the image to be corrected is corrected based on the three-channel background light field information after the up-sampling, because the three-channel background light field information is smooth and gradually changed, even if the down-sampling and the up-sampling exist in the process of model processing, the corrected image obtained by correcting the image to be corrected according to the three-channel background light field image after the up-sampling also has higher image quality.

In step S220, the image to be corrected is corrected according to the three-channel background light field image, so as to obtain a corrected image corresponding to the image to be corrected.

In an embodiment of the present disclosure, after acquiring the three-channel background light field image, the image to be corrected may be corrected according to the three-channel background light field image, so as to acquire a corrected image with luminance distortion and color distortion corrected. Specifically, when the image to be corrected is corrected, image information corresponding to three color channels in the image to be corrected can be corrected according to the three-channel background light field image, so as to obtain a corrected image corresponding to the image to be corrected.

In the embodiment of the present disclosure, the three-channel background light field image is obtained by predicting, by the light field prediction model, the image to be corrected according to the image to be corrected, where the three-channel background light field image includes predicted background light field image information corresponding to R, G, B three color channels, that is, R-channel predicted background light field information, G-channel predicted background light field information, and B-channel predicted background light field information.

The brightness unevenness and the brightness variation are generally linear transformation relations as shown in formula (1):

I_imaging(x,y)＝I_{Reality (reality)}(x,y)×S(x,y)+D(x,y) (1)

Wherein, I_ImagingFor photographed images of light-field inhomogeneities, I_{Reality (reality)}For an ideal brightness-uniform image, S is the multiplicative illumination intensity, D is the additive dark field, and (x, y) is the pixel in the image at coordinate (x, y).

Since the image processing is pixel-by-pixel, from a linear transformation perspective, for I_{Reality (reality)}Both the multiplication and addition operations performed by (x, y) can be converted into a multiplication relation, so that the formula (1) can be faded to be controlled by a uniform background light field (luminance information) M, as shown in the formula (2):

I_imaging(x,y)＝I_{Reality (reality)}(x,y)×M(x,y) (2)

Comparing the formula (2) with the formula (1), it can be known that the transformation between the captured image with non-uniform light field and the ideal image with uniform brightness is still linear.

The above formula only considers the case of brightness distortion, that is, for a single gray scale image, there is a mathematical relationship as shown in formulas (1) and (2) between the captured image with non-uniform light field and the ideal image with uniform brightness, and it is desired to obtain I_{Reality (reality)}(x, y) is only required to be according to I_Imaging(x, y)/M (x, y) may be calculated.

Since the luminance variation varies from pixel to pixel, each pixel corresponds to its own specific linear transform coefficient m (x). While color distortion is related to white balance, which means that a white object can be restored to white regardless of any light source. When a camera takes a bright field image, white generally refers to a visual response formed by light reflected to human eyes due to the fact that the proportion of blue, green and red lights is the same and the light has certain brightness. The gray values corresponding to the RGB color image, i.e., the three channels R, G, B of the RGB image, are similar, for example, when the gray values of R, G, B of the three channels are all 255, the image appears pure white. As can be seen from the above description, the difference of white balance is mainly the coefficient difference of each pixel corresponding to different color channels, so that it can be considered that for any pixel in the image, the background light fields (luminance information) of R, G, B three channels are uniform, and there is only the difference of white balance coefficients, so that when there are both luminance distortion and color distortion, there is still a linear correlation between the captured image (image to be corrected) and the ideal image (corrected image).

When the brightness distortion and the color distortion are considered at the same time, the image information corresponding to the three color channels in the image to be corrected needs to be corrected according to the background light field information related to the three color channels to obtain a corrected image. In the embodiment of the disclosure, the three-channel background light field image output by processing the image to be corrected by the light field prediction model includes R-channel prediction background light field information, G-channel prediction background light field information, and B-channel prediction background light field information, and meanwhile, the image to be corrected includes R-channel image information, G-channel image information, and B-channel image information, so that the corrected image information of each color channel can be obtained based on the linear relationship of the formula (2), and further the corrected image is determined according to the corrected image information of each color channel. Compared with the image to be corrected, the corrected image obtained through correction corrects the part with uneven brightness and color distortion, so that the image quality is higher, and the image is more in line with the real and ideal image effect.

Fig. 4 is a schematic flowchart of acquiring a corrected image, and as shown in fig. 4, in step S401, R-channel image information is divided by R-channel predicted background light field information to acquire R-channel corrected image information; in step S402, dividing G channel image information by G channel predicted background light field information to obtain G channel corrected image information; in step S403, dividing the B-channel image information by the B-channel predicted background light field information to obtain B-channel corrected image information; in step S404, a corrected image is acquired from the R-channel corrected image information, the G-channel corrected image information, and the B-channel corrected image information.

It can be seen from the foregoing embodiments that, in the embodiment of the present disclosure, when image correction is performed, only one to-be-corrected image with luminance distortion and color distortion needs to be input into the light field prediction model, a three-channel background light field image corresponding to the to-be-corrected image can be obtained, and then the to-be-corrected image is corrected according to the three-channel background light field image to obtain a corrected image.

In an embodiment of the present disclosure, before feature extraction is performed on an image to be corrected by using a light field prediction model, the light field prediction model to be trained needs to be trained to obtain a stable light field prediction model. In the embodiment of the present disclosure, fig. 5 shows a schematic flowchart of a process for training a light field prediction model to be trained, as shown in fig. 5, in step S501, a standard image set including a plurality of standard images is obtained, and a light field brightness variation range and a white balance coefficient value range are obtained; in step S502, a simulated light field is determined based on the light field brightness variation range, and white balance sample information is determined according to the white balance coefficient value range, where the simulated light field is the same as the standard image in size; in step S503, determining a three-channel background light field image sample according to the analog light field and white balance sample information; in step S504, the standard images in the standard image set are processed according to the three-channel background light field image sample to obtain a non-standard simulation image sample, and the light field prediction model to be trained is trained according to the non-standard simulation image sample and the three-channel background light field image sample.

In the embodiment of the present disclosure, the light field luminance variation range and the white balance coefficient value range in step S501 may be set according to actual needs, and in the embodiment of the present disclosure, the light field luminance variation range may be set to [0.5,1], and the white balance coefficient value range may be set to [0.7,1.2], which may also be other value ranges, which is not specifically limited by the present disclosure. In step S502, when determining the simulated light field based on the light field brightness variation range, a gaussian distribution-like image with an unfixed center position and an unfixed scale (variance) may be randomly generated as the simulated light field according to the value in the light field brightness variation range, and when determining the white balance sample information according to the white balance coefficient value range, three different random numbers may be arbitrarily determined from the white balance coefficient value range as white balance coefficient samples of three color channels, and then the white balance sample information is obtained according to the white balance coefficient samples of the three color channels, and it is noted that the white balance coefficient samples are random numbers that are uniformly distributed and are independent of each other. After the information of the analog light field and the white balance sample is determined, a three-channel background light field image sample can be determined according to the information of the analog light field and the white balance sample, and then a standard image is processed according to the three-channel background light field image sample to obtain a non-standard analog image sample.

In the above embodiment, it is mentioned that, for any pixel in the image, the background light fields (luminance information) of the R, G, B three channels are uniform, and there is only a difference in white balance coefficient, so that when there are both luminance distortion and color distortion, there is still a linear correlation between the captured image (image to be corrected) and the ideal image (corrected image). In view of this, when constructing a three-channel background light field image sample, the background light field information of the R, G, B channel may be obtained based on the white balance coefficient of the R, G, B channel and the luminance information corresponding to each pixel, and then the three-channel background light field image sample is obtained according to the background light field information of the R, G, B channel. Specifically, the luminance information corresponding to each pixel in the simulated light field may be multiplied by the R-channel white balance coefficient, the G-channel white balance coefficient, and the B-channel white balance coefficient in the white balance sample information, respectively, to obtain R-channel background light field information, G-channel background light field information, and B-channel background light field information, and the specific calculation method is as shown in equations (3) - (5):

M^R(x,y)＝a1×M(x,y) (3)

M^G(x,y)＝a2×M(x,y) (4)

M^B(x,y)＝a3×M(x,y) (5)

wherein M is^R(x, y) is R channel background light field information, M^G(x, y) is G channel background light field information, M^B(x, y) is B-channel background light field information, a1 is an R-channel white balance coefficient, a2 is a G-channel white balance coefficient, a3 is a B-channel white balance coefficient, and M (x, y) is a uniform background light field, i.e., luminance information.

And then, determining a three-channel background light field image sample based on the calculated R channel background light field information, G channel background light field information and B channel background light field information.

The purpose of training the light field prediction model to be trained is to enable the model to extract a three-channel background light field image in an image to be corrected, that is, the image to be corrected is a non-standard image with brightness distortion and color distortion, so that when the light field prediction model to be trained is trained, it is necessary to process the standard image to generate a non-standard simulation image sample so as to train the light field prediction model to be trained. In the embodiment of the disclosure, when the standard image in the standard image set is processed according to the three-channel background light field image sample, firstly, R-channel image information, G-channel image information, and B-channel image information in the standard image may be respectively processed based on the R-channel background light field information, the G-channel background light field information, and the B-channel background light field information in the three-channel background light field image sample to obtain non-standard analog image information corresponding to the R, G, B channel, and then, a non-standard analog image sample containing luminance distortion and color distortion may be obtained according to the non-standard analog image information corresponding to the R, G, B channel. Specifically, the mathematical expression formula for obtaining the non-standard simulated image information corresponding to the R, G, B channel is shown in equations (6) to (8):

I^R _imaging(x,y)＝I^R _{Reality (reality)}(x,y)×M^R(x,y)＝I^R _{Reality (reality)}(x,y)×(a1×M(x,y)) (6)

I^G _Imaging(x,y)＝I^G _{Reality (reality)}(x,y)×M^G(x,y)＝I^G _{Reality (reality)}(x,y)×(a2×M(x,y)) (7)

I^B _Imaging(x,y)＝I^B _{Reality (reality)}(x,y)×M^B(x,y)＝I^B _{Reality (reality)}(x,y)×(a3×M(x,y)) (8)

Wherein, I^R _{Reality (reality)}(x, y) is R-channel image information in the standard image, I^G _{Reality (reality)}(x, y) is G-channel image information in the standard image, I^B _{Reality (reality)}(x, y) is B-channel image information in the standard image, I^R _Imaging(x, y) is R-channel non-standard analog image information, I^G _Imaging(x, y) is G-channel nonstandard analog image information, I^B _ImagingAnd (x, y) is B-channel non-standard analog image information.

Finally, mixing I^R _Imaging(x,y)、I^G _Imaging(x, y) and I^B _ImagingAnd (x, y) integrating to obtain a non-standard analog image sample.

As can be seen from the above description, when the image to be corrected is corrected, the inverse transformation of equations (6) - (8) is adopted, and the correction method also corresponds to the flowchart of obtaining the corrected image shown in fig. 4, that is, the image information of R, G, B channels in the image to be corrected is divided by the predicted background light field information of R, G, B channels in the three-channel background light field image corresponding to the image to be corrected, and the specific mathematical expression equations are shown in equations (9) - (11):

I^R _Correction(x,y)＝I^R _{to be corrected}(x,y)/M^R’(x,y)＝I^R _{To be corrected}(x,y)/(a1’×M’(x,y)) (9)

I^G _Correction(x,y)＝I^G _{To be corrected}(x,y)/M^G’(x,y)＝I^G _{To be corrected}(x,y)/(a2’×M’(x,y)) (10)

I^B _Correction(x,y)＝I^B _{To be corrected}(x,y)/M^B’(x,y)＝I^B _{To be corrected}(x,y)/(a3’×M’(x,y)) (11)

Wherein, I^R _{To be corrected}(x, y) is R-channel image information in the image to be corrected, I^G _{To be corrected}(x, y) is G-channel image information in the image to be rectified, I^B _{To be corrected}(x, y) is B-channel image information in the image to be corrected, M^R’(x, y) predicting background light field information for the R channel, M^G’(x, y) predicting background light field information for the G channel, M^B’(x, y) is B-channel predicted background light field information, a1 'is an R-channel white balance coefficient in the image to be corrected, a 2' is a G-channel white balance coefficient in the image to be corrected, a3 'is a B-channel white balance coefficient in the image to be corrected, and M' (x, y) is luminance information.

It should be noted that, in order to reduce the data amount during model training, the standard image with a larger size may be cropped, so as to reduce the image size and further reduce the data processing amount. In addition, the gaussian-like distribution in the disclosure may be gaussian distribution, or may also be multi-peak mild gaussian distribution, and certainly, the simulated light field may also be other gradually-changed distributions, which is not described herein again in the embodiments of the disclosure.

Fig. 6 is a schematic diagram illustrating a process of training a to-be-trained light field prediction model according to a non-standard simulation image sample and a three-channel background light field image sample, where as shown in fig. 6, in step S601, the non-standard simulation image sample is input to the to-be-trained light field prediction model, and feature extraction is performed on the non-standard simulation image sample by the to-be-trained light field prediction model to obtain a predicted three-channel background light field image; in step S602, a first loss function is determined according to the predicted three-channel background light field image and the three-channel background light field image sample corresponding to the non-standard analog image sample; in step S603, parameters of the light field prediction model to be trained are optimized based on the first loss function to obtain the light field prediction model.

When the light field prediction model to be trained is trained, an Adam optimization method can be adopted to train according to a preset learning rate. The learning rate may be arbitrarily set, for example, to 0.001, or the like. In an embodiment of the present disclosure, the first loss function may specifically be a minimum mean square error, and a specific expression is shown in equation (12):

wherein i is an RGB three channel, i is 1,2,3, (x, y) is a position coordinate of any pixel in a non-standard simulation image sample or a predicted three-channel background light field image, M is a maximum value of an abscissa of the non-standard simulation image sample or the predicted three-channel background light field image, N is a maximum value of an ordinate of the non-standard simulation image sample or the predicted three-channel background light field image, and F is a maximum value of an ordinate of the non-standard simulation image sample or the predicted three-channel background light field image_i(x, y) is a three-channel background light field image sample, F ', corresponding to the non-standard analog image sample'_i(x, y) is a predicted three-channel background light field image.

In an embodiment of the present disclosure, since the simulated light field is gaussian-like distributed, the light field changes gradually, and even an uneven light field should be gradual rather than jump, in order to prevent light field jump, a smooth constraint may be added to the three-channel background light field image before outputting the three-channel background light field image, and the smooth constraint may be implemented by controlling the adjacent pixel difference minimization of the three-channel background light field image. If the light field changes corresponding to all the pixels are consistent, namely a constant value, the difference result is 0, and the three-channel background light field image is the most smooth; if adjacent pixels have abrupt changes, the difference result is too large, and the three-channel background light field image is not smooth. In the embodiment of the present disclosure, the difference coefficient c may be set so as to control the light field variation by the difference coefficient c. The optimized value of the difference coefficient c can be determined by training the light field prediction model to be trained, and specifically, a second loss function can be constructed according to image information corresponding to three color channels of each pixel in the predicted three-channel background light field image; and then optimizing parameters of the light field prediction model to be trained based on the first loss function and the second loss function to obtain the light field prediction model. The expression of the second loss function is shown in equation (13):

wherein i is RGB three channels, i is 1,2,3, (x, y) is the position coordinate of any pixel in the non-standard simulation image sample or the predicted three-channel background light field image, M is the maximum value of the abscissa of the non-standard simulation image sample or the predicted three-channel background light field image, N is the maximum value of the ordinate of the non-standard simulation image sample or the predicted three-channel background light field image, and F'_i(x, y) for predicting three-channel background light-field images,

is the difference operator.

Further, F 'in formulae (12) to (13)'_i(x, y) is determined based on the R, G, B channel background light field information determined according to equations (3) - (5).

It should be noted that, in the embodiment of the present disclosure, in addition to performing the smoothing constraint by using a simple difference between adjacent pixels, a sobel operator may also be used to perform the smoothing constraint, and certainly, other methods may also be used to perform the smoothing constraint, which is not specifically limited in the embodiment of the present disclosure.

After iterative training is carried out on the light field prediction model to be trained, optimized model parameters can be obtained, the three-channel background light field image in the image to be corrected can be accurately extracted based on the light field prediction model with the optimized model parameters, and the quality of the corrected image is further ensured.

In an embodiment of the present disclosure, when the corrected image obtained through one round of correction does not achieve the ideal effect, the corrected image may be further corrected, and the number of times of correction may be one or more times until the optimal corrected image is obtained. Specifically, feature extraction can be performed on the corrected image through a light field prediction model to obtain a three-channel background light field image corresponding to the corrected image; then, re-correcting the corrected image according to the three-channel background light field image corresponding to the corrected image to obtain an optimized corrected image; and judging whether the optimized correction image achieves the ideal effect, if not, repeating the steps, and carrying out iterative correction on the optimized correction image until the optimal correction image is obtained.

The image correction method in the embodiment of the disclosure can be applied to a plurality of fields, and all images with brightness distortion and color distortion can be corrected by the method. In particular, the image correction method can be used for correcting microscope images, for example, after a pathological tissue is made into a pathological slide by a pathological analyst, the pathological slide can be photographed by a photographable microscope, and the acquired pathological slide image can be analyzed to acquire a pathological analysis result. If brightness distortion and color distortion exist in the pathological slide image, the pathological analysis result may have deviation, which has great influence on the correctness of the final diagnosis result, so it is necessary to acquire a high-quality pathological slide image to ensure the correctness of the diagnosis result.

The embodiment of the present disclosure further provides a microscope image correction method, where the flow of the correction method is the same as that of the image correction method in the above embodiment, and the difference is only that the analysis object is a microscope image to be corrected. Fig. 7 shows a schematic flow chart of a method for correcting a microscope image, as shown in fig. 7:

in step S710, obtaining a microscope image to be corrected, and performing feature extraction on the microscope image to be corrected through a light field prediction model to obtain a three-channel background light field image corresponding to the microscope image to be corrected, where the three-channel background light field image includes luminance information and white balance information;

in step S720, correcting the microscope image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the microscope image to be corrected;

the light field prediction model is trained on a plurality of non-standard simulation microscope image samples and three-channel background light field image samples corresponding to the non-standard simulation microscope image samples.

When the microscope image is corrected, a trained light field prediction model can be adopted to perform feature extraction on the microscope image to be corrected so as to obtain a three-channel background light field image corresponding to the microscope image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information; and finally, correcting the microscope image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the microscope image to be corrected, wherein the image A is the microscope image to be corrected with brightness distortion and color distortion, a three-channel background light field image B can be obtained after the characteristics of the image A are extracted through a light field prediction model, and finally the three-channel background light field image B is removed from the microscope image A to be corrected to obtain a corrected image C.

In an embodiment of the present disclosure, a specific method and details for correcting a microscope image to be corrected are the same as those in the above embodiment, and are not described herein again. However, the training samples used in the model training phase may vary from one corrective object to another. When a light field prediction model for correcting a microscope image is trained, a pair of non-standard simulation microscope image sample and a three-channel background light field image sample can be constructed through simulation data for training. When a non-standard simulation microscope image sample is constructed, firstly, a scanner is used for scanning a slide to obtain a standard slide image, and the scanner has the characteristics of stable image imaging performance and uniform light field, so that the slide image scanned by the scanner does not have brightness distortion and color distortion and can be considered as an ideal standard slide image; then, a pair of brightness non-uniform images and corresponding background light fields thereof are constructed through computer simulation data, white balance coefficients (a1, a2 and a3) corresponding to three channels are randomly generated, and the three-channel background light fields can be determined according to the background light fields and the randomly generated white balance coefficients corresponding to the three channels; then, superposing the standard slide image and the three-channel background light field to obtain a non-standard analog microscope image sample with brightness distortion and color distortion, and generating a three-channel background light field image sample according to the brightness non-uniform image and a white balance coefficient corresponding to three channels; and finally, training the light field prediction model to be trained according to the non-standard simulation microscope image sample and the three-channel background light field image sample.

FIGS. 9A-9C are schematic diagrams illustrating an interface between a sample of a non-standard simulated microscope image generated based on a standard slide image and a sample of a three-channel background light field image, as shown in FIG. 9A, where FIG. a-1 is a standard slide image in which there is no luminance distortion and color distortion, FIG. b-1 is a sample of a three-channel background light field image generated by simulation, and FIG. C-1 is a sample of a non-standard simulated microscope image generated by processing the standard slide image a-1 according to the three-channel background light field generated by simulation; as shown in fig. 9B, wherein a-2 is the same standard slide image as a-1, B-2 is a three-channel background light field image sample generated by simulation, B-2 is different from B-1 only in white balance coefficient, and c-2 is a non-standard simulation microscope image sample generated by processing the standard slide image a-2 according to the three-channel background light field generated by simulation; as shown in fig. 9C, a-3 is a standard slide image identical to fig. a-1 and a-2, a b-3 is a three-channel background light field image sample generated by simulation, b-3 is different from b-1 and b-2 only in white balance coefficient, and C-3 is a non-standard simulation microscope image sample generated by processing the standard slide image a-3 according to the three-channel background light field generated by simulation. It should be noted that when three-channel background light field image samples are generated by simulation, different brightness variation ranges, different central positions and different sizes can be selected, and when three-channel white balance coefficients are generated randomly, the three-channel background light field image samples can be generated randomly, so that different three-channel background light field image samples can be generated based on different parameter combinations, and different non-standard simulation microscope image samples can be generated for the same standard slide image. Furthermore, when the model is trained according to a large number of non-standard simulation microscope image samples and three-channel background light field image samples, the non-standard simulation microscope image samples and the three-channel background light field image samples corresponding to the non-standard simulation microscope image samples are not strictly adopted for model training, and the non-standard simulation microscope image samples and the three-channel background light field image samples can be in a disorganized matching mode for training, so that the performance of the model can be improved.

The trained light field prediction model can be used to correct any microscope image, and fig. 10A-10C show schematic interface diagrams of microscope images before and after correction, as shown in fig. 10A, wherein, the left image is a microscope image before correction, four corners of the microscope image have deeper shadows, the integral chromaticity of the images is inconsistent, the middle image is a three-channel background light field image extracted by a light field prediction model from the microscope image before correction, the right image is a corrected microscope image, the corrected image is obtained after the microscope image before correction is processed according to the three-channel background light field image, and as can be seen from the image, the shadow of the four corners of the corrected microscope image becomes less and can be almost ignored, the chromaticity of the whole image tends to be consistent, and substances such as cell nucleuses and the like in the image are clearer; similarly, fig. 10B and 10C also show a set of microscope images before correction, three-channel background light field images corresponding to the microscope images, and a set of microscope images after correction, respectively, and like fig. 10A, the microscope images after correction by the method for correcting microscope images in the embodiment of the present disclosure have higher image quality, and the degree of brightness distortion and color distortion is very low and almost negligible.

Compared with the prior art in which the microscope image to be corrected is directly processed by the machine learning model to output the corrected microscope image, the image correction method and the microscope image correction method in the disclosure have better effects. Fig. 11 shows an interface schematic diagram of a microscope image corrected by the correction method in the present disclosure and the correction method in the related art, as shown in fig. 11, where the image on the left side is a microscope image to be corrected, the image in the middle is a microscope image corrected by the correction method in the related art, and the image on the right side is a microscope image corrected by the method in the present disclosure, it can be found by comparing the three images that the microscope image corrected by the correction method in the embodiment of the present disclosure has more uniform brightness, more uniform color, and better image quality than the microscope image corrected by the correction method in the related art.

In the image correction method or the microscope image correction method in the embodiment of the disclosure, a light field prediction model to be trained is trained to obtain a stable light field prediction model; then, carrying out feature extraction on the image to be corrected with uneven brightness and color distortion through a light field prediction model to obtain a three-channel background light field image corresponding to the image to be corrected, wherein the three-channel background light field image correspondingly comprises brightness information and white balance information; and finally, correcting the image to be corrected according to the three-channel background light field image, and acquiring a corrected image corresponding to the image to be corrected. It can be known from the analysis of the technical solution of the present disclosure that as long as an image with luminance distortion and color distortion is input, the image correction method of the present disclosure can be used to correct the image to obtain a standard image with the luminance distortion and the color distortion corrected, that is, the technical solution of the present disclosure can implement background light field correction and color white balance synchronously on a single image, thereby improving the image quality. In addition, in the embodiment of the disclosure, when the model is trained, the model training can be completed by simulating the background light field image and the nonstandard simulation image through three channels without acquiring light field data, so that manpower and material resources are saved.

Embodiments of the apparatus of the present disclosure are described below, which may be used to perform the image rectification method in the above-described embodiments of the present disclosure. For details that are not disclosed in the embodiments of the apparatus of the present disclosure, please refer to the embodiments of the image rectification method described above in the present disclosure.

Fig. 12 schematically illustrates a block diagram of an image rectification apparatus according to an embodiment of the present disclosure.

Referring to fig. 12, an image rectification apparatus 1200 according to an embodiment of the present disclosure includes: a model processing module 1201 and an image rectification module 1202.

The model processing module 1201 is configured to acquire an image to be corrected, perform feature extraction on the image to be corrected through a light field prediction model, so as to acquire a three-channel background light field image corresponding to the image to be corrected, where the three-channel background light field image includes luminance information and white balance information; an image correction module 1202, configured to correct the image to be corrected according to the three-channel background light field image, so as to obtain a corrected image corresponding to the image to be corrected; the light field prediction model is trained based on a simulation image sample and a three-channel background light field image sample corresponding to the simulation image sample.

In some embodiments of the present disclosure, the model processing module 1201 is configured to: extracting the background light field image information of the three color channels in the image to be corrected through an end-to-end full convolution neural network model, and determining the three-channel background light field image according to the extracted background light field image information of the three color channels.

In some embodiments of the present disclosure, the image rectification module 1202 includes: and the correcting unit is used for correcting the image information corresponding to the three color channels in the image to be corrected according to the three-channel background light field image so as to obtain a corrected image corresponding to the image to be corrected.

In some embodiments of the present disclosure, the image to be rectified includes R-channel image information, G-channel image information, and B-channel image information; the three-channel background light field image comprises R channel prediction background light field information, G channel prediction background light field information and B channel prediction background light field information; the correction unit is configured to: dividing the R channel image information by the R channel prediction background light field information to obtain R channel correction image information; dividing the G channel image information by the G channel predicted background light field information to obtain G channel corrected image information; dividing the B channel image information by the B channel prediction background light field information to obtain B channel correction image information; and acquiring the corrected image according to the R channel corrected image information, the G channel corrected image information and the B channel corrected image information.

In some embodiments of the present disclosure, the image rectification apparatus 1200 further includes: the information acquisition module is used for acquiring a standard image set containing a plurality of standard images and acquiring a light field brightness variation range and a white balance coefficient value range; the parameter determining module is used for determining a simulated light field based on the light field brightness variation range and determining white balance sample information according to the white balance coefficient value range, wherein the simulated light field and the standard image have the same size; the three-channel background light field image sample generation module is used for determining the three-channel background light field image sample according to the simulation light field and the white balance sample information; and the training module is used for processing the standard images in the standard image set according to the three-channel background light field image samples to obtain the non-standard simulation image samples and training a light field prediction model to be trained according to the non-standard simulation image samples and the three-channel background light field image samples.

In some embodiments of the present disclosure, the simulated light field is a randomly generated gaussian-like distributed image with a center position that is not fixed and a variance that is not fixed; the white balance sample information includes white balance coefficient samples corresponding to three color channels, and each of the white balance coefficient samples is a random number that conforms to uniform distribution and is independent of each other.

In some embodiments of the present disclosure, the white balance sample information includes an R channel white balance coefficient, a G channel white balance coefficient, and a B channel white balance coefficient; the three-channel background light field image sample generation module is configured to: multiplying brightness information corresponding to each pixel in the simulated light field by the R channel white balance coefficient, the G channel white balance coefficient and the B channel white balance coefficient respectively to obtain R channel background light field information, G channel background light field information and B channel background light field information; and determining the three-channel background light field image sample based on the R-channel background light field information, the G-channel background light field information and the B-channel background light field information.

In some embodiments of the disclosure, the training module is configured to: inputting the non-standard simulation image sample into the to-be-trained light field prediction model, and performing feature extraction on the non-standard simulation image sample through the to-be-trained light field prediction model to obtain a three-channel background light field prediction image; determining a first loss function according to the predicted three-channel background light field image and a three-channel background light field image sample corresponding to the non-standard simulation image sample; and optimizing parameters of the light field prediction model to be trained based on the first loss function so as to obtain the light field prediction model.

In some embodiments of the present disclosure, the image rectification apparatus 1200 is further configured to: constructing a second loss function according to image information corresponding to three color channels of each pixel in the predicted three-channel background light field image; and optimizing the parameters of the light field prediction model to be trained on the basis of the first loss function and the second loss function so as to obtain the light field prediction model.

In some embodiments of the present disclosure, the first loss function is calculated according to equation (1):

wherein i is an RGB triple channel, i is 1,2,3, (x, y) is a position coordinate of any pixel in the non-standard analog image sample or the predicted triple channel background light field image, M is a maximum value of an abscissa of the non-standard analog image sample or the predicted triple channel background light field image, N is a maximum value of an ordinate of the non-standard analog image sample or the predicted triple channel background light field image, F is a maximum value of an ordinate of the non-standard analog image sample or the predicted triple channel background light field image, and F is a maximum value of an abscissa of the predicted triple channel background light field image_i(x, y) is a three-channel background light field corresponding to the non-standard analog image sampleImage sample, F'_i(x, y) is the predicted three-channel background light field image.

In some embodiments of the present disclosure, the second loss function is calculated according to equation (2):

is the difference operator.

In some embodiments of the present disclosure, the image rectification apparatus 1200 is further configured to: performing feature extraction on the corrected image through the light field prediction model to obtain a three-channel background light field image corresponding to the corrected image; re-correcting the corrected image according to the three-channel background light field image corresponding to the corrected image to obtain an optimized corrected image; and repeating the steps, and performing iterative correction on the optimized corrected image until an optimal corrected image is obtained.

Fig. 13 schematically illustrates a block diagram of a rectification apparatus of a microscope image according to one embodiment of the present disclosure.

Referring to fig. 13, an apparatus 1300 for correcting a microscope image according to an embodiment of the present disclosure includes: a feature extraction module 1301 and a microscope image rectification module 1302.

The feature extraction module 1301 is configured to obtain a microscope image to be corrected, and perform feature extraction on the microscope image to be corrected through a light field prediction model to obtain a three-channel background light field image corresponding to the microscope image to be corrected, where the three-channel background light field image includes luminance information and white balance information; the microscope image correcting module 1302 is configured to correct the microscope image to be corrected according to the three-channel background light field image, so as to obtain a corrected image corresponding to the microscope image to be corrected; the light field prediction model is trained on a plurality of non-standard simulation microscope image samples and three-channel background light field image samples corresponding to the non-standard simulation microscope image samples.

In some embodiments of the present disclosure, based on the above scheme, the feature extraction module 1301 is configured to: extracting the background light field image information of three color channels in the microscope image to be corrected through an end-to-end full convolution neural network model, and determining the three-channel background light field image according to the extracted background light field image information of the three color channels.

In some embodiments of the present disclosure, based on the above scheme, the microscope image rectification module 1302 includes: and the correcting unit is used for correcting image information corresponding to three color channels in the microscope image to be corrected respectively according to the three-channel background light field image so as to obtain a corrected image corresponding to the microscope image to be corrected.

In some embodiments of the present disclosure, the microscope image to be rectified includes R-channel image information, G-channel image information, and B-channel image information; the three-channel background light field image comprises R channel prediction background light field information, G channel prediction background light field information and B channel prediction background light field information; based on the scheme, the correcting unit is configured to: dividing the R channel image information by the R channel prediction background light field information to obtain R channel correction image information; dividing the G channel image information by the G channel predicted background light field information to obtain G channel corrected image information; dividing the B channel image information by the B channel prediction background light field information to obtain B channel correction image information; and acquiring the corrected image according to the R channel corrected image information, the G channel corrected image information and the B channel corrected image information.

In some embodiments of the present disclosure, based on the above scheme, the apparatus 1300 for correcting a microscope image further includes: the information acquisition module is used for acquiring a standard microscope image set containing a plurality of standard microscope images and determining a light field brightness change range and a white balance coefficient value range; the parameter determining module is used for determining a simulated light field based on the light field brightness variation range and determining white balance sample information according to the white balance coefficient value range, wherein the simulated light field and the standard microscope image have the same size; the three-channel background light field image sample generation module is used for determining the three-channel background light field image sample according to the simulation light field and the white balance sample information; and the training module is used for processing the standard microscope image in the standard microscope image set according to the three-channel background light field image sample to obtain the non-standard simulation microscope image sample, and training the light field prediction model to be trained according to the non-standard simulation microscope image sample and the three-channel background light field image sample.

In some embodiments of the present disclosure, based on the above scheme, the training module is configured to: inputting the non-standard simulation microscope image sample to the light field prediction model to be trained, and performing feature extraction on the non-standard simulation microscope image sample through the light field prediction model to be trained to obtain a three-channel background light field image to be predicted; determining a first loss function according to the predicted three-channel background light field image and a three-channel background light field image sample corresponding to the non-standard simulation microscope image sample; and optimizing parameters of the light field prediction model to be trained based on the first loss function so as to obtain the light field prediction model.

In some embodiments of the present disclosure, based on the above scheme, the apparatus 1300 for correcting a microscope image is further configured to: constructing a second loss function according to image information corresponding to three color channels of each pixel in the predicted three-channel background light field image; and optimizing the parameters of the light field prediction model to be trained on the basis of the first loss function and the second loss function so as to obtain the light field prediction model.

wherein i is RGB triple channel, i is 1,2,3, (x, y) is a position coordinate of any pixel in the non-standard analog microscope image sample or the predicted triple channel background light field image, M is a maximum value of abscissa of the non-standard analog microscope image sample or the predicted triple channel background light field image, N is a maximum value of ordinate of the non-standard analog microscope image sample or the predicted triple channel background light field image, F is a maximum value of ordinate of the non-standard analog microscope image sample or the predicted triple channel background light field image, and F is a maximum value of y, and y is a maximum value of y, F_i(x, y) is a three-channel background light field image sample, F ', corresponding to the non-standard analog microscope image sample'_i(x, y) isAnd predicting the three-channel background light field image.

wherein i is RGB three channels, i is 1,2,3, (x, y) is a position coordinate of any pixel in the non-standard analog microscope image sample or the predicted three-channel background light field image, M is a maximum value of abscissa of the non-standard analog microscope image sample or the predicted three-channel background light field image, N is a maximum value of ordinate of the non-standard analog microscope image sample or the predicted three-channel background light field image, and F'_i(x, y) is the predicted three-channel background light-field image,

is the difference operator.

In some embodiments of the present disclosure, based on the above scheme, the apparatus 1300 for correcting a microscope image is further configured to: performing feature extraction on the corrected image through the light field prediction model to obtain a three-channel background light field image corresponding to the corrected image; re-correcting the corrected image according to the three-channel background light field image corresponding to the corrected image to obtain an optimized corrected image; and repeating the steps, and performing iterative correction on the optimized corrected image until an optimal corrected image is obtained.

It should be noted that the computer system 1400 of the electronic device shown in fig. 14 is only an example, and should not bring any limitation to the functions and the scope of the application of the embodiments of the present disclosure.

As shown in fig. 14, a computer system 1400 includes a Central Processing Unit (CPU)1401, which can perform various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) 1402 or a program loaded from a storage portion 1408 into a Random Access Memory (RAM) 1403, implementing the search string Processing method described in the above-described embodiments. In the RAM 1403, various programs and data necessary for system operation are also stored. The CPU 1401, ROM 1402, and RAM 1403 are connected to each other via a bus 1404. An Input/Output (I/O) interface 1405 is also connected to the bus 1404.

The following components are connected to the I/O interface 1405: an input portion 1406 including a keyboard, a mouse, and the like; an output portion 1407 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage portion 1408 including a hard disk and the like; and a communication section 1409 including a Network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1409 performs communication processing via a network such as the internet. The driver 1410 is also connected to the I/O interface 1405 as necessary. A removable medium 1411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1410 as necessary, so that a computer program read out therefrom is installed into the storage section 1408 as necessary.

In particular, the processes described below with reference to the flowcharts may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 1409 and/or installed from the removable medium 1411. The computer program performs various functions defined in the system of the present disclosure when executed by a Central Processing Unit (CPU) 1401.

It should be noted that the computer readable medium shown in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

As another aspect, the present disclosure also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method described in the above embodiments.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image rectification method, comprising:

acquiring an image to be corrected, and performing feature extraction on the image to be corrected through a light field prediction model to acquire a three-channel background light field image corresponding to the image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information;

correcting the image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the image to be corrected;

the light field prediction model is trained on the basis of a plurality of non-standard simulation image samples and three-channel background light field image samples corresponding to the non-standard simulation image samples.

2. The method according to claim 1, wherein the performing feature extraction on the image to be corrected through a light field prediction model to obtain a three-channel background light field image corresponding to the image to be corrected comprises:

extracting the background light field image information of three color channels in the image to be corrected through an end-to-end full convolution neural network model, and determining the three-channel background light field image according to the extracted background light field image information of the three color channels.

3. The method according to claim 1, wherein the correcting the image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the image to be corrected comprises:

and respectively correcting image information corresponding to three color channels in the image to be corrected according to the three-channel background light field image so as to obtain a corrected image corresponding to the image to be corrected.

4. The method according to claim 3, wherein the image to be rectified includes R-channel image information, G-channel image information, and B-channel image information; the three-channel background light field image comprises R channel prediction background light field information, G channel prediction background light field information and B channel prediction background light field information;

the correcting the image information corresponding to three color channels in the image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the image to be corrected includes:

dividing the R channel image information by the R channel prediction background light field information to obtain R channel correction image information;

dividing the G channel image information by the G channel predicted background light field information to obtain G channel corrected image information;

dividing the B channel image information by the B channel prediction background light field information to obtain B channel correction image information;

and acquiring the corrected image according to the R channel corrected image information, the G channel corrected image information and the B channel corrected image information.

5. The method of claim 1, further comprising:

acquiring a standard image set comprising a plurality of standard images, and acquiring a light field brightness variation range and a white balance coefficient value range;

determining a simulated light field based on the light field brightness variation range, and simultaneously determining white balance sample information according to the white balance coefficient value range, wherein the simulated light field and the standard image have the same size;

determining the three-channel background light field image sample according to the simulated light field and the white balance sample information;

and processing the standard image according to the three-channel background light field image sample to obtain the non-standard simulation image sample, and training a light field prediction model to be trained according to the non-standard simulation image sample and the three-channel background light field image sample.

6. The method of claim 5, wherein the simulated light field is a randomly generated Gaussian-like distribution image with a center position that is not fixed and a variance that is not fixed; the white balance sample information includes white balance coefficient samples corresponding to three color channels, and each of the white balance coefficient samples is a random number that conforms to uniform distribution and is independent of each other.

7. The method according to claim 5, wherein the white balance sample information includes an R-channel white balance coefficient, a G-channel white balance coefficient, and a B-channel white balance coefficient;

the determining the three-channel background light field image sample according to the simulated light field and the white balance sample information includes:

multiplying brightness information corresponding to each pixel in the simulated light field by the R channel white balance coefficient, the G channel white balance coefficient and the B channel white balance coefficient respectively to obtain R channel background light field information, G channel background light field information and B channel background light field information;

and determining the three-channel background light field image sample based on the R-channel background light field information, the G-channel background light field information and the B-channel background light field information.

8. The method of claim 5, wherein training a light field prediction model to be trained from the non-standard simulation image sample and the three-channel background light field image sample comprises:

inputting the non-standard simulation image sample into the to-be-trained light field prediction model, and performing feature extraction on the non-standard simulation image sample through the to-be-trained light field prediction model to obtain a three-channel background light field prediction image;

determining a first loss function according to the predicted three-channel background light field image and a three-channel background light field image sample corresponding to the non-standard simulation image sample;

and optimizing parameters of the light field prediction model to be trained based on the first loss function so as to obtain the light field prediction model.

9. The method of claim 8, further comprising:

constructing a second loss function according to image information corresponding to three color channels of each pixel in the predicted three-channel background light field image;

and optimizing the parameters of the light field prediction model to be trained on the basis of the first loss function and the second loss function so as to obtain the light field prediction model.

10. The method of claim 1, further comprising:

performing feature extraction on the corrected image through the light field prediction model to obtain a three-channel background light field image corresponding to the corrected image;

re-correcting the corrected image according to the three-channel background light field image corresponding to the corrected image to obtain an optimized corrected image;

and repeating the steps, and performing iterative correction on the optimized corrected image until an optimal corrected image is obtained.

11. An image rectification apparatus, characterized by comprising:

the model processing module is used for acquiring an image to be corrected, and performing feature extraction on the image to be corrected through a light field prediction model to acquire a three-channel background light field image corresponding to the image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information;

the image correction module is used for correcting the image to be corrected according to the three-channel background light field image so as to obtain a corrected image corresponding to the image to be corrected;

the light field prediction model is trained based on a simulation image sample and a three-channel background light field image sample corresponding to the simulation image sample.

12. A method for correcting a microscope image, comprising:

acquiring a microscope image to be corrected, and performing feature extraction on the microscope image to be corrected through a light field prediction model to acquire a three-channel background light field image corresponding to the microscope image to be corrected, wherein the three-channel background light field image comprises brightness information and white balance information;

correcting the microscope image to be corrected according to the three-channel background light field image to obtain a corrected image corresponding to the microscope image to be corrected;

13. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the image rectification method according to any one of claims 1 to 10 or the microscope image rectification method according to claim 12.