CN114926567A

CN114926567A - Image rendering method, electronic device, storage medium, and computer program product

Info

Publication number: CN114926567A
Application number: CN202210444846.9A
Authority: CN
Inventors: 吴凡
Original assignee: Beijing Kuangshi Technology Co Ltd
Current assignee: Beijing Kuangshi Technology Co Ltd
Priority date: 2022-04-24
Filing date: 2022-04-24
Publication date: 2022-08-19

Abstract

The disclosure relates to an image rendering method, an electronic device, a storage medium, and a computer program product. The image coloring method comprises the following steps: acquiring a gray level image to be colored; respectively extracting the characteristics of the gray level image based on at least two branches of a first neural network model to obtain at least two characteristic graphs; wherein, the receptive fields corresponding to different branches in the at least two branches are different, and the at least two characteristic graphs correspond to the at least two branches one by one; channel splicing is carried out on the at least two characteristic graphs, and a two-channel color image which is consistent with the size of the gray image is obtained based on the spliced characteristic graphs; and performing channel splicing on the two-channel color image and the gray image to obtain a first color image for coloring the gray image. The color overflowing problem of coloring the gray-scale image can be improved through the present disclosure.

Description

Image rendering method, electronic device, storage medium, and computer program product

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image rendering method, an electronic device, a storage medium, and a computer program product.

Background

Image rendering is one of means of image processing, and aims to convert a single-channel gray scale image into a three-channel color image, and meanwhile, the obtained color image is expected to conform to a real scene of the physical world as much as possible.

In the related art, a color image obtained by converting a grayscale image generally has a problem of color bleeding.

Disclosure of Invention

To overcome the problems in the related art, the present disclosure provides an image rendering method, an electronic device, a storage medium, and a computer program product.

According to a first aspect of embodiments of the present disclosure, there is provided an image coloring method, including:

acquiring a gray level image to be colored; respectively extracting the characteristics of the gray level image based on at least two branches of a first neural network model to obtain at least two characteristic graphs; wherein the receptive fields corresponding to different branches in the at least two branches are different, and the at least two characteristic maps correspond to the at least two branches one to one; channel splicing is carried out on the at least two characteristic graphs, and a two-channel color image which is consistent with the size of the gray image is obtained based on the spliced characteristic graphs; and performing channel splicing on the two-channel color image and the gray image to obtain a first color image for coloring the gray image.

In one embodiment, the method further comprises: acquiring a plurality of different three-dimensional display lookup tables configured in advance; the plurality of different three-dimensional display lookup tables at least comprise three-dimensional display lookup tables corresponding to color channels in the three-primary-color channels; and performing color adjustment on the first color image based on the plurality of different three-dimensional display lookup tables to obtain a second color image.

In one embodiment, the color adjusting the first color image based on the plurality of different three-dimensional display look-up tables to obtain a second color image includes: respectively coloring the first color image based on the plurality of different three-dimensional display lookup tables to obtain a plurality of different colored images; and determining a first weight for weighting the plurality of different colored images based on the gray level image, and weighting the plurality of different colored images by the first weight to obtain the second color image.

In one embodiment, the color adjusting the first color image based on the plurality of different three-dimensional display look-up tables to obtain a second color image includes: respectively performing coloring treatment on the first color image based on the plurality of different three-dimensional display lookup tables to obtain a plurality of different colored images; and determining second weights for weighting the plurality of different colored images and the first color image based on the gray-scale image, and weighting the plurality of different colored images and the first color image by the second weights to obtain the second color image.

In one embodiment, the target weight is determined by the following method, and the target weight comprises a first weight or a second weight: calling a pre-trained second neural network model, wherein the input of the second neural network model is a gray image, and the output of the second neural network model is a target weight; determining the target weight based on an output result of the second neural network model.

In one embodiment, the at least two branches of the first neural network model include at least a first branch and a second branch; wherein the receptive field of the first branch is larger than the receptive field of the second branch, and the difference between the receptive field of the first branch and the receptive field of the second branch is larger than the target difference.

In one embodiment, the obtaining a two-channel color image in accordance with the size of the grayscale image based on the feature map after the stitching includes: and carrying out inter-channel information fusion and/or spatial information fusion on the spliced feature images, and carrying out size adjustment and channel number adjustment on the feature images subjected to inter-channel information fusion and/or spatial information fusion to obtain a two-channel color image consistent with the size of the gray image.

According to a second aspect of the embodiments of the present disclosure, there is provided an image coloring apparatus including:

an acquisition unit configured to acquire a grayscale image to be colored; the processing unit is used for respectively extracting the features of the gray level image to be colored based on at least two branches of the first neural network model to obtain at least two feature maps; wherein the receptive fields corresponding to different branches in the at least two branches are different, and the at least two characteristic maps correspond to the at least two branches one to one; the splicing unit is used for splicing the channels of the at least two characteristic graphs, and adjusting the size and the number of the channels of the spliced characteristic graphs to obtain a two-channel color image with the size consistent with that of the gray image; and the color image processing unit is used for performing channel splicing on the two-channel color image and the gray image to obtain a first color image for coloring the gray image.

In one embodiment, the obtaining unit is further configured to: acquiring a plurality of different three-dimensional display lookup tables configured in advance; the plurality of different three-dimensional display lookup tables at least comprise three-dimensional display lookup tables corresponding to color channels in the three-primary-color channels; the processing unit is further to: and performing color adjustment on the first color image based on the plurality of different three-dimensional display lookup tables to obtain a second color image.

In one embodiment, the processing unit performs color adjustment on the first color image based on the plurality of different three-dimensional display look-up tables to obtain a second color image by: respectively coloring the first color image based on the plurality of different three-dimensional display lookup tables to obtain a plurality of different colored images; and determining a first weight for weighting the plurality of different colored images based on the gray level image, and weighting the plurality of different colored images by the first weight to obtain the second color image.

In one embodiment, the processing unit performs color adjustment on the first color image based on the plurality of different three-dimensional display look-up tables to obtain a second color image by: respectively performing coloring treatment on the first color image based on the plurality of different three-dimensional display lookup tables to obtain a plurality of different colored images; and determining second weights for weighting the plurality of different colored images and the first color image based on the gray-scale image, and weighting the plurality of different colored images and the first color image by the second weights to obtain a second color image.

In one embodiment, the processing unit determines the target weight, which includes the first weight or the second weight, by: calling a pre-trained second neural network model, wherein the input of the second neural network model is a gray image, and the output of the second neural network model is a target weight; determining the target weight based on an output result of the second neural network model.

In one embodiment, the stitching unit obtains a two-channel color image in accordance with the size of the grayscale image based on the stitched feature map by: and carrying out inter-channel information fusion and/or spatial information fusion on the spliced feature images, and carrying out size adjustment and channel number adjustment on the feature images subjected to the inter-channel information fusion and/or spatial information fusion to obtain a two-channel color image consistent with the size of the gray image.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device including:

a processor; a memory for storing processor-executable instructions;

wherein the processor is configured to: the image rendering method described in the first aspect or any one of the embodiments of the first aspect is performed.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a storage medium having instructions stored therein, where the instructions when executed by a processor enable the processor to execute the image rendering method of the first aspect or any one of the embodiments of the first aspect.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product, the computer program product comprising a computer program, which when executed by a processor, implements the image rendering method of the first aspect or any one of the embodiments of the first aspect.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: the gray level image to be colored can be obtained, and the characteristic extraction is respectively carried out on the gray level image through a plurality of different branches corresponding to a plurality of different receptive fields. On the basis, at least two characteristic maps obtained by at least two branches can be spliced, and the spliced characteristic maps are adjusted to be two-channel color images with the same size as the gray image. Further, the gray image is regarded as image data of a corresponding L channel in a color-opponent space (Lab) format, the obtained two-channel color image is regarded as image data of a corresponding ab channel in the Lab format, and the two channel color image and the image data are subjected to channel splicing to obtain a first color image for coloring the gray image. Because in two at least branches that correspond a plurality of different receptive fields, the wider region of scope can be paid attention to the great branch of receptive field to catch senior semantic information, the less branch of receptive field can be positioned out comparatively accurate colour value in the sub-range, with this accuracy of guaranteeing the coloring result. Therefore, the method for coloring the gray-scale image can effectively improve the color overflow problem of the color image, so that the colored image obtained after coloring has a better visual effect.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a flow diagram illustrating a method of image rendering according to an exemplary embodiment.

FIG. 2 is a flow diagram illustrating another method of image rendering according to an example embodiment.

FIG. 3 is a schematic diagram illustrating a process for obtaining a two-channel color image consistent with the size of a grayscale image according to an exemplary embodiment.

FIG. 4 is a flow chart illustrating yet another method of image rendering according to an exemplary embodiment.

FIG. 5 is a flow diagram illustrating another method of image rendering according to an example embodiment.

FIG. 6 is a flowchart illustrating yet another method of image rendering according to an example embodiment.

FIG. 7 is a flow chart illustrating a method for determining target weights based on a grayscale image according to an exemplary embodiment.

Fig. 8 is a schematic flow chart illustrating a process of performing color adjustment on a first color image by weighting a plurality of color images and the first color image to obtain a second color image according to an exemplary embodiment.

FIG. 9 is a schematic diagram illustrating the structure of a second neural network model, in accordance with an exemplary embodiment.

FIG. 10 is a block diagram illustrating an image rendering device according to an exemplary embodiment.

FIG. 11 is a block diagram illustrating an electronic device for image rendering in accordance with an exemplary embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure.

In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are only a subset of the embodiments of the present disclosure, and not all embodiments. The embodiments described below with reference to the drawings are exemplary and intended to be illustrative of the present disclosure, and should not be construed as limiting the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure. Embodiments of the present disclosure are described in detail below with reference to the drawings.

In recent years, technical research based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition, has been actively developed. Artificial Intelligence (AI) is an emerging scientific technology for studying and developing theories, methods, techniques and application systems for simulating and extending human Intelligence. The artificial intelligence subject is a comprehensive subject and relates to various technical categories such as chips, big data, cloud computing, internet of things, distributed storage, deep learning, machine learning and neural networks. Computer vision is used as an important branch of artificial intelligence, particularly a machine is used for identifying the world, and the computer vision technology generally comprises the technologies of face identification, living body detection, fingerprint identification and anti-counterfeiting verification, biological feature identification, face detection, pedestrian detection, target detection, pedestrian identification, image processing, image identification, image semantic understanding, image retrieval, character identification, video processing, video content identification, behavior identification, three-dimensional reconstruction, virtual reality, augmented reality, synchronous positioning and map construction (SLAM), computational photography, robot navigation and positioning and the like. With the research and development of artificial intelligence technology, the technology is applied to many fields, such as security protection, city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone images, cloud services, smart homes, wearable equipment, unmanned driving, automatic driving, intelligent medical treatment, face payment, face unlocking, fingerprint unlocking, human evidence verification, smart screens, smart televisions, cameras, mobile internet, network, beauty, makeup, medical beauty, intelligent temperature measurement and the like.

The image coloring method provided by the embodiment of the disclosure can be applied to a scene for performing coloring processing on a gray image.

In the related art, image rendering is mainly based on deep learning, and includes a convolutional neural network (a codec network formed by connecting a plurality of most basic convolutional layers in series), example segmentation is introduced (a foreground and a background are segmented, and the foreground is distinguished in categories, so that colors of the background and the foreground are influenced mutually in a coloring process), or a antagonistic neural network (GAN) is used for generating colors of an image. In the related art, a color image converted from a grayscale image usually has a large area of a solid color region and/or an irregular color region, which is usually caused by color bleeding of the color image.

In view of this, the present disclosure provides an image coloring method, which can obtain a grayscale image to be colored, and perform feature extraction on the grayscale image through at least two branches corresponding to different receptive fields, respectively. On the basis, at least two feature maps obtained by at least two branches can be spliced, and the spliced feature maps are adjusted to be two-channel color images with the same size as the gray image. Further, the gray image is regarded as image data of a corresponding L channel in a color-opponent space (Lab) format, the obtained two-channel color image is regarded as image data of a corresponding ab channel in the Lab format, and the two channels are spliced to obtain a color image for coloring the gray image. Because in two at least branches that correspond a plurality of different receptive fields, the wider region of scope can be paid attention to the great branch of receptive field to catch senior semantic information, the less branch of receptive field can be positioned out comparatively accurate colour value in the sub-range, with this accuracy of guaranteeing the coloring result. Therefore, the method for coloring the gray-scale image can effectively improve the color overflowing problem of the color image, so that the colored image obtained after coloring has a better visual effect. For convenience of description, a neural network model for extracting features of a grayscale image is referred to as a first neural network model, and a color image obtained by channel stitching a two-channel color image and the grayscale image is referred to as a first color image.

Fig. 1 is a flowchart illustrating an image coloring method according to an exemplary embodiment, as shown in fig. 1, including the following steps S11 through S13.

In step S11, a grayscale image to be colored is acquired.

In step S12, feature extraction is performed on the grayscale image based on at least two branches of the first neural network model, respectively, to obtain at least two feature maps.

In the embodiment of the disclosure, the first neural network model is used for feature extraction of a grayscale image and includes at least two branches. Wherein, the receptive fields corresponding to different branches in the at least two branches are different, and the at least two characteristic maps correspond to the at least two branches one to one.

For example, a plurality of different branches may be configured to correspond to a plurality of different receptive fields by adjusting the number of convolution layers and/or the size of the convolution kernel.

In step S13, at least two feature maps are channel-stitched, and a two-channel color image having a size that matches the size of the grayscale image is obtained based on the stitched feature maps.

The size of at least two feature maps subjected to channel splicing is consistent, and the channel splicing of at least two feature maps is understood to be the channel dimension data superposition of image data of at least two feature maps. Further, the size of the spliced feature map is adjusted to splice the adjusted feature map and the gray scale image through a channel, and the number of channels of the spliced feature map is adjusted to represent image data of a color channel and b color channel in the Lab-format color image through the adjusted two-channel color image.

In step S14, the two-channel color image and the grayscale image are channel-stitched to obtain a first color image obtained by coloring the grayscale image.

The image coloring method is suitable for coloring the gray-scale image into a color image comprising the gray-scale image data in multi-channel image data. In practical application scenarios, the method is not only applicable to color images in Lab format, but also applicable to color images in true color space (YUV) format, for example. For the color image in YUV format, the image data of the Y color channel corresponds to the image data of the grayscale image in the above embodiment, and the image data of the U color channel and the image data of the V color channel correspond to the image data of the two-channel color image in the above embodiment.

The image coloring method provided by the embodiment of the disclosure can perform characteristics on the gray-scale image by corresponding to at least two branches of a plurality of different receptive fields, and splice at least two obtained characteristic maps. Furthermore, a two-channel color image with the same size as the gray image can be obtained through the spliced characteristic diagram, and a first color image with a better visual effect can be obtained through a channel splicing mode of the two-channel color image and the gray image.

For example, under the condition that inter-channel information fusion and/or spatial information fusion is performed on the spliced feature map, size adjustment and channel number adjustment are performed on the feature map after inter-channel information fusion and/or spatial information fusion, so as to obtain a two-channel color image with the size consistent with that of the gray-scale image.

Fig. 2 is a flowchart illustrating another image coloring method according to an exemplary embodiment, and as shown in fig. 2, steps S21, S22, and S24 in the embodiment of the present disclosure are similar to the execution methods of steps S11, S12, and S14 in fig. 1, and are not repeated herein.

In step S23, channel splicing is performed on at least two feature maps, inter-channel information fusion and/or spatial information fusion is performed on the spliced feature maps, and size adjustment and channel number adjustment are performed on the feature maps after inter-channel information fusion and/or spatial information fusion, so as to obtain a two-channel color image whose size is consistent with that of the grayscale image.

According to the image coloring method provided by the embodiment of the disclosure, information fusion between channels can be performed on the spliced feature map through a Channel Attention mechanism module (CA), so that features among multiple channels can be better fused. In another example, Spatial information fusion may be performed on the stitched feature map through a Spatial Attention mechanism (SA), so that consistent colors may be maintained between pixels with close brightness in the feature map. And the spatial attention mechanism module is used for carrying out spatial information fusion on the spliced characteristic diagram, so that the color overflowing problem of the color image can be further improved.

For example, a first and second receptive field may be configured for a first neural network model. The receptive field of the first branch is larger than that of the second branch, and the difference between the receptive field of the first branch and the receptive field of the second branch is larger than the target difference, so that at least two branches of the first neural network have branches corresponding to a larger receptive field and branches corresponding to a smaller receptive field. On this basis, the wider region of scope can be paid attention to the great first branch of receptive field to catch senior semantic information, the less second branch of receptive field can be positioned out comparatively accurate colour value in the sub-range, with this accuracy of guaranteeing the result of coloring, and then guarantee the actual effect of color image.

FIG. 3 is a schematic diagram illustrating a process for obtaining a two-channel color image consistent with the size of a grayscale image according to an exemplary embodiment. For example, as shown in fig. 3, the first neural network model may include two branches (a first branch and a second branch). Under the condition that the gray level image is input into the first neural network model, the gray level image respectively flows through the first branch and the second branch to obtain corresponding characteristic maps. On the basis, the obtained feature maps can be subjected to channel splicing through a stack (stack) module to obtain spliced feature maps. Further, the spliced feature maps can be input into the channel attention module and the space attention module in sequence, and the feature maps are processed into a two-channel color image with the size consistent with that of the gray-scale image in a rolling and up-sampling manner (as shown by a white square in fig. 3), and the obtained two-channel color image is the image data of the color channel a and the color channel b in the Lab format color image corresponding to the gray-scale image. In other words, a color image in Lab format in which a gray image is colored can be obtained by channel-superimposing the obtained two-channel color image and the gray image. In addition, it should be noted that, the residual block (ResBlock) module, the 3x3 convolution module, the 1x1 convolution module, and the batch normalization layer involved in the drawings are used for performing feature extraction on a grayscale image in a convolution calculation manner, and the corresponding modules may be added or deleted according to actual use requirements.

In general, when a gradation image is subjected to a coloring process, there is a problem that the color image obtained is poor in color vividness.

In one embodiment, when the first color image is obtained, the color of the first color image may be adjusted by using a plurality of different pre-configured three-dimensional Look up tables (3-dimensional Look up tables, 3D LUTs). For convenience of description, a color image obtained by color-adjusting a first color image will be referred to as a second color image.

Fig. 4 is a flowchart illustrating still another image coloring method according to an exemplary embodiment, and as shown in fig. 4, steps S31, S32, S33 and S34 in the embodiment of the present disclosure are similar to the steps S11, S12, S13 and S14 in fig. 1, and are not repeated herein.

In step S35, a plurality of different three-dimensional display look-up tables configured in advance are acquired.

The preset multiple different three-dimensional display lookup tables at least comprise three-dimensional display lookup tables corresponding to color channels in the three primary color channels. In addition, besides the three-dimensional display lookup tables corresponding to the color channels of the three primary colors, other types of three-dimensional display lookup tables such as a purple three-dimensional display lookup table and a black and white three-dimensional display lookup table may be included, and the type of the obtained three-dimensional display lookup table may be adjusted according to actual requirements, which is not limited in the present disclosure.

In step S36, the first color image is color adjusted based on a plurality of different three-dimensional display look-up tables to obtain a second color image.

Therein, it is understood that the first color image is currently an image in Lab format. In one embodiment, the first color image may be converted to an image in a three primary color space (RGB) format prior to color adjustment of the first color image by the plurality of three-dimensional display look-up tables for color adjustment of the RGB format first color image by the three-dimensional display look-up tables.

According to the image coloring method provided by the embodiment of the disclosure, when the first color image is obtained, the color of the first color image can be adjusted through a plurality of different three-dimensional display lookup tables configured in advance, and the second color image with better color vividness can be obtained.

In one embodiment, the first color image may be rendered by a plurality of different three-dimensional display look-up tables, and the second color image may be obtained by weighting the plurality of different rendered images.

Fig. 5 is a flowchart illustrating another image coloring method according to an exemplary embodiment, and as shown in fig. 5, the execution methods of step S41, step S42, step S43, step S44 and step S45 in the embodiment of the present disclosure are similar to the execution methods of step S31, step S32, step S33, step S34 and step S35 in fig. 4, and are not repeated herein.

In step S46, the first color image is colored based on the plurality of different three-dimensional display look-up tables, respectively, to obtain a plurality of different colored images.

In step S47, a first weight for weighting the plurality of different color images is determined based on the grayscale image, and the plurality of different color images are weighted by the first weight to obtain a second color image.

In another embodiment, the first color image may be colored by a plurality of different three-dimensional display look-up tables, and the second color image may be obtained by weighting the plurality of differently colored images and the first color image.

Fig. 6 is a flowchart illustrating still another image coloring method according to an exemplary embodiment, and as shown in fig. 6, the execution methods of step S51, step S52, step S53, step S54 and step S55 in the embodiment of the present disclosure are similar to the execution methods of step S31, step S32, step S33, step S34 and step S35 in fig. 4, and are not repeated herein.

In step S56, the first color image is colored based on the plurality of different three-dimensional display look-up tables, respectively, to obtain a plurality of different colored images.

In step S57, a second weight for weighting the plurality of different color images and the first color image is determined based on the grayscale image, and the plurality of different color images and the first color image are weighted by the second weight to obtain a second color image.

According to the image coloring method provided by the embodiment of the disclosure, since the plurality of different three-dimensional display lookup tables can convert the first color image into a plurality of different color style results (i.e., a plurality of different colored images), the second color image obtained by weighting the plurality of results is more colorful. In addition, the second color image mode is obtained by weighting the plurality of different colored images and the first color image, which is equivalent to taking the first color image as a reference, so that the method can ensure the color correctness of the second color image. Compared with the method of weighting a plurality of different colored images to obtain the second color image, the obtained second color image has better visual effect.

For example, the first weight or the second weight may be determined by a pre-trained neural network model. The present disclosure hereinafter refers to a neural network model for determining the first weight or the second weight as a second neural network model for convenience of description.

FIG. 7 is a flowchart illustrating a method for determining target weights based on grayscale images, as shown in FIG. 7, including the following steps, according to an exemplary embodiment.

In step S61, a pre-trained second neural network model is invoked.

The input of the second neural network model is a gray image, and the output is a target weight. Wherein the target weight comprises a first weight or a second weight.

In step S62, a target weight is determined based on the output result of the second neural network model.

Wherein, the first weight and the second weight can be understood as different results output by the second neural network model. In other words, the second neural network model used is different for the first weight and the second weight. For example, in the case of performing a coloring process on a grayscale image by using a predetermined number of three-dimensional display look-up tables, if a second color image is obtained by weighting a plurality of different colored images, the number of channels corresponding to the output result of the second neural network model should be a predetermined number. And if the second color image is obtained by weighting the plurality of different colored images and the first color image, the number of channels corresponding to the output result of the second neural network model is + 1. It can be understood that, for the output result of the second neural network model, the data corresponding to each channel is the weight corresponding to the corresponding shading image (or the first color image) when weighted. Also, the correspondence relationship between the plurality of channels of the output result and the plurality of colored images (or the plurality of colored images and the first color image) may be configured in advance so that the weight configured when weighting each image is determined directly according to the correspondence relationship when obtaining the output result of the second neural network model.

The process of obtaining the second color image by weighting the plurality of color images and the first color image is described as follows.

In the embodiment of the present disclosure, as shown in fig. 8, a red three-dimensional display lookup table, a blue three-dimensional display lookup table, a green three-dimensional display lookup table, a purple three-dimensional display lookup table, and a black and white three-dimensional display lookup table are configured in advance. For example, when the first color image is obtained by channel splicing, the first color image may be respectively input into a plurality of three-dimensional display lookup tables configured in advance, so as to obtain a plurality of colored images corresponding to each three-dimensional display lookup table. The gray-scale image may be input to the second neural network model, and a second weight for weighting the plurality of color images and the first color image may be obtained. Further, the plurality of colored images and the first color image may be weighted according to the obtained second weight to obtain a second color image. Compared with the first color image, the second color image which is subjected to color adjustment through the three-dimensional display lookup table has more bright colors and better visual effect.

For example, as shown in fig. 9, a second neural network model may be constructed by a plurality of convolution layers and a plurality of deconvolution layers, so as to obtain the first weight or the second weight by changing the number of channels without changing the size of the image. For example, after the grayscale image is input into the second neural network, the grayscale image passes through a plurality of convolution layers to increase the number of channels of the image data. Further, the image data after passing through the convolution layer is further processed through a plurality of deconvolution layers to reduce the number of channels of the image data, and finally an output image with the number of channels matching the number of weighted images is obtained. On the basis, the image output by the second neural network model is the first weight or the second weight. Specifically, the output image includes a plurality of channels, and the plurality of channels are in one-to-one correspondence with the plurality of weighted images, and the image data of each channel is used as the calculation weight of the corresponding weighted image. In one embodiment, a linear rectification (Relu) function may be used as the activation function for the second neural network model. In addition, for passingWhen the second color image is obtained by weighting the plurality of color images and the first color image, the weight may be obtained by lose ═ MSE (GT, FinalRes) +0.1 mean (weight) ² )+0.1*MSE(1,(weight[0]+weight[1]+…+weight[5]) As a loss function of the second neural network model. Wherein, weight [0 ]]To weight [5 ]]Respectively representing a second weight corresponding to each colored image in a plurality of different colored images, or representing a second weight corresponding to the first color image, wherein MSE (GT, FinalRes) represents a mean square error value, MSE (1), (weight [0 ]) between image data of the color image used for training and image data of the second color image obtained by coloring a gray level image of the color image]+weight[1]+…+weight[5]) For limiting the sum value between the plurality of different second weights to 1.

Based on the same conception, the embodiment of the disclosure also provides an image coloring device.

It is understood that the image rendering device provided by the embodiments of the present disclosure includes a hardware structure and/or a software module for performing the above functions. The disclosed embodiments can be implemented in hardware or a combination of hardware and computer software, in combination with the exemplary elements and algorithm steps disclosed in the disclosed embodiments. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

FIG. 10 is a block diagram illustrating an image rendering device according to an exemplary embodiment. Referring to fig. 10, the apparatus 100 includes an acquisition unit 101, a processing unit 102, and a splicing unit 103.

An acquisition unit 101 is used for acquiring a grayscale image to be colored. The processing unit 102 is configured to perform feature extraction on the grayscale image to be colored respectively based on the at least two branches of the first neural network model to obtain at least two feature maps. Wherein, the receptive fields corresponding to different branches in the at least two branches are different, and the at least two characteristic maps correspond to the at least two branches one to one. And the splicing unit 103 is used for performing channel splicing on the at least two feature maps, and performing size adjustment and channel number adjustment on the spliced feature maps to obtain a two-channel color image with the size consistent with that of the gray image. And the color filter is used for splicing the two-channel color image and the gray image to obtain a first color image for coloring the gray image.

In one embodiment, the obtaining unit 101 is further configured to: and acquiring a plurality of different pre-configured three-dimensional display lookup tables. The plurality of different three-dimensional display lookup tables at least comprise three-dimensional display lookup tables corresponding to color channels in the three primary color channels. The processing unit 102 is further configured to: and performing color adjustment on the first color image based on a plurality of different three-dimensional display lookup tables to obtain a second color image.

In one embodiment, the processing unit 102 performs color adjustment on the first color image based on a plurality of different three-dimensional display look-up tables to obtain a second color image as follows: and respectively coloring the first color image based on a plurality of different three-dimensional display lookup tables to obtain a plurality of different colored images. And determining a first weight for weighting the plurality of different colored images based on the gray level image, and weighting the plurality of different colored images by the first weight to obtain a second color image.

In one embodiment, the processing unit 102 performs color adjustment on the first color image based on a plurality of different three-dimensional display look-up tables to obtain a second color image as follows: and respectively coloring the first color image based on a plurality of different three-dimensional display lookup tables to obtain a plurality of different colored images. And determining second weights for weighting the plurality of different colored images and the first color image based on the gray level image, and weighting the plurality of different colored images and the first color image by the second weights to obtain a second color image.

In one embodiment, the processing unit 102 determines the target weight by the following method, and the target weight includes a first weight or a second weight: and calling a pre-trained second neural network model, wherein the input of the second neural network model is a gray image, and the output of the second neural network model is the target weight. Target weights are determined based on the output results of the second neural network model.

In one embodiment, the at least two branches of the first neural network model include at least a first branch and a second branch. Wherein the receptive field of the first branch is larger than the receptive field of the second branch, and the difference between the receptive field of the first branch and the receptive field of the second branch is larger than the target difference.

In one embodiment, the stitching unit 103 obtains a two-channel color image with a size consistent with the size of the grayscale image based on the stitched feature map by the following method: and performing inter-channel information fusion and/or spatial information fusion on the spliced feature map, and performing size adjustment and channel number adjustment on the feature map subjected to the inter-channel information fusion and/or spatial information fusion to obtain a two-channel color image consistent with the size of the gray image.

With regard to the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

FIG. 11 is a block diagram illustrating an electronic device 200 for image rendering according to an example embodiment.

As shown in fig. 11, one embodiment of the present disclosure provides an electronic device 200. The electronic device 200 includes a memory 201, a processor 202, and an Input/Output (I/O) interface 203. The memory 201 is used for storing instructions. And a processor 202 for calling the instructions stored in the memory 201 to execute the image coloring method according to the embodiment of the disclosure. The processor 202 is connected to the memory 201 and the I/O interface 203, respectively, for example, via a bus system and/or other connection mechanism (not shown). The memory 201 may be used to store programs and data, including the programs of the image rendering method involved in the embodiments of the present disclosure, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the programs stored in the memory 201.

In the embodiment of the present disclosure, the processor 202 may be implemented in at least one hardware form of a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), and the processor 202 may be one or a combination of several Central Processing Units (CPUs) or other forms of Processing units with data Processing capability and/or instruction execution capability.

Memory 201 in embodiments of the present disclosure may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile Memory may include, for example, a Random Access Memory (RAM), a cache Memory (cache), and/or the like. The nonvolatile Memory may include, for example, a Read Only Memory (ROM), a Flash Memory (Flash Memory), a Hard Disk Drive (HDD), a Solid State Drive (SSD), or the like.

In the embodiment of the present disclosure, the I/O interface 203 may be used to receive input instructions (e.g., numeric or character information, and generate key signal inputs related to user settings and function control of the electronic apparatus 200, etc.), and may also output various information (e.g., images or sounds, etc.) to the outside. The I/O interface 203 in the disclosed embodiment may include one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel, among others.

In some embodiments, the present disclosure provides a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, perform any of the methods described above.

In some embodiments, the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, performs any of the methods described above.

Although operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in serial order, or that all illustrated operations be performed, to achieve desirable results. In certain environments, multitasking and parallel processing may be advantageous.

The methods and apparatus of the present disclosure can be accomplished with standard programming techniques with rule-based logic or other logic to accomplish the various method steps. It should also be noted that the words "means" and "module," as used herein and in the claims, is intended to encompass implementations using one or more lines of software code, and/or hardware implementations, and/or equipment for receiving inputs.

Any of the steps, operations, or procedures described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer readable medium embodying computer program code, which is executable by a computer processor to perform any or all of the described steps, operations, or procedures.

The foregoing description of the implementations of the disclosure has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosure. The embodiments were chosen and described in order to explain the principles of the disclosure and its practical application to enable one skilled in the art to utilize the disclosure in various embodiments and with various modifications as are suited to the particular use contemplated.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

It is understood that "a plurality" in this disclosure means two or more, and other words are analogous. "and/or" describes the association relationship of the associated object, indicating that there may be three relationships, for example, a and/or B, which may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. The singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It will be further understood that the terms "first," "second," and the like, are used to describe various information and should not be limited by these terms. These terms are only used to distinguish one type of information from another and do not denote a particular order or importance. Indeed, the terms "first," "second," and the like are fully interchangeable. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure.

It is further understood that, unless otherwise specified, "connected" includes direct connections between the two without other elements and indirect connections between the two with other elements.

It will be further appreciated that while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in serial order, or that all illustrated operations be performed, to achieve desirable results. In certain environments, multitasking and parallel processing may be advantageous.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the scope of the appended claims.

Claims

1. An image rendering method, comprising:

acquiring a gray level image to be colored;

respectively extracting the characteristics of the gray level image based on at least two branches of a first neural network model to obtain at least two characteristic graphs; wherein the receptive fields corresponding to different branches in the at least two branches are different, and the at least two characteristic maps correspond to the at least two branches one to one;

channel splicing is carried out on the at least two characteristic graphs, and a two-channel color image which is consistent with the size of the gray image is obtained based on the spliced characteristic graphs;

and carrying out channel splicing on the two-channel color image and the gray image to obtain a first color image for coloring the gray image.

2. The image coloring method according to claim 1, further comprising:

acquiring a plurality of different three-dimensional display lookup tables configured in advance; the plurality of different three-dimensional display lookup tables at least comprise three-dimensional display lookup tables corresponding to color channels in the three-primary-color channels;

and performing color adjustment on the first color image based on the plurality of different three-dimensional display lookup tables to obtain a second color image.

3. The image rendering method of claim 2, wherein said color adjusting said first color image based on said plurality of different three-dimensional display look-up tables to obtain a second color image comprises:

respectively performing coloring treatment on the first color image based on the plurality of different three-dimensional display lookup tables to obtain a plurality of different colored images;

and determining a first weight for weighting the plurality of different colored images based on the gray level image, and weighting the plurality of different colored images by the first weight to obtain the second color image.

4. The image rendering method of claim 2, wherein said color adjusting said first color image based on said plurality of different three-dimensional display look-up tables to obtain a second color image comprises:

and determining second weights for weighting the plurality of different colored images and the first color image based on the gray-scale image, and weighting the plurality of different colored images and the first color image by the second weights to obtain the second color image.

5. An image rendering method according to claim 3 or 4, wherein the target weight is determined in the following way, the target weight comprising a first weight or a second weight:

calling a pre-trained second neural network model, wherein the input of the second neural network model is a gray image, and the output of the second neural network model is a target weight;

determining the target weight based on an output result of the second neural network model.

6. The image coloring method according to any one of claims 1 to 5, wherein at least two branches of the first neural network model, including at least a first branch and a second branch;

wherein the receptive field of the first branch is larger than the receptive field of the second branch, and the difference between the receptive field of the first branch and the receptive field of the second branch is larger than the target difference.

7. The image rendering method according to any one of claims 1 to 6, wherein the obtaining a two-channel color image in accordance with the size of the grayscale image based on the stitched feature map comprises:

and carrying out inter-channel information fusion and/or spatial information fusion on the spliced feature images, and carrying out size adjustment and channel number adjustment on the feature images subjected to the inter-channel information fusion and/or spatial information fusion to obtain a two-channel color image consistent with the size of the gray image.

8. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: -performing the image rendering method of any of claims 1 to 7.

9. A storage medium having stored therein instructions which, when executed by a processor, enable the processor to carry out the image rendering method of any one of claims 1 to 7.

10. A computer program product, characterized in that it comprises a computer program which, when being executed by a processor, implements the image rendering method of any one of claims 1 to 7.