CN108921916B - Method, device and equipment for coloring multi-target area in picture and storage medium - Google Patents

Abstract

The application discloses a method, a device, equipment and a storage medium for coloring multi-target areas in a picture, wherein the method comprises the following steps: carrying out graying processing on a picture to be colored to obtain a grayscale image; selecting a plurality of color standard diagrams for carrying out different color designs on the picture to be colored; coloring the gray level images respectively through the selected multiple color standard images; identifying and segmenting a plurality of colored target areas; and fusing the divided target areas to the corresponding positions of the picture to be colored. According to the method and the device, a plurality of target areas in the picture to be colored are colored by utilizing a plurality of existing color standard pictures with coordinated colors, the color information of the plurality of target areas in the picture to be colored is changed, a large amount of time is saved, the colors of the pictures are not too monotonous, the coloring diversity and flexibility are realized, and different design requirements are met.

Description

Coloring method, device, device and storage medium for multi-target areas in pictures

technical field

The present invention relates to the field of image processing, in particular to a coloring method, device, device and storage medium for multi-target areas in a picture.

Background technique

In the design of posters, leaflets, etc., it is often necessary to design colors for multiple objects in a picture, and the color design of an object is not purely a pure color replacement, but also must consider the various parts of the object. Whether the colors are coordinated or not, it takes a lot of time for designers to manually design the colors of various parts of the object.

At present, there are technologies for coloring the entire grayscale image. For example, given a color image, the color information of the color image is defined by the global histogram and saturation, and the Lab color model is used to pass the bilinear difference value. Resize the color to quarter resolution Encode each pixel in quantized ab space and average spatially Calculate the global histogram of the colormap by converting the colormap to HSV color space and then spatially averaging Saturation is used to calculate saturation, and finally the calculated global histogram and saturation are integrated into the coloring network, which applies the color information of the color image to the entire grayscale image. Such a method cannot freely determine the color of an object in the picture, and is not flexible enough for the color design of the picture.

Therefore, how to realize the design of different colors for multiple objects of a picture is a technical problem to be solved urgently by those skilled in the art.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a coloring method, device, device and storage medium for a multi-target area in a picture, which can use a plurality of existing color-coordinated color standard pictures to color a plurality of targets in the picture. Color design. Its specific plan is as follows:

A method for coloring multiple target areas in a picture, comprising:

Perform grayscale processing on the to-be-colored image to obtain a grayscale image;

Selecting a plurality of color standard maps for carrying out different color designs on the picture to be colored;

Coloring the grayscale image by selecting a plurality of the color standard images;

Identify and segment multiple colored target areas;

The segmented target regions are fused to corresponding positions of the picture to be colored.

Preferably, in the method for coloring the multi-target areas in the above picture provided by the embodiment of the present invention, the grayscale images are respectively colored by using a plurality of the selected color standard images, which specifically includes:

Train a convolutional neural network for coloring and a target network for obtaining color information;

Inputting the picture to be colored and the grayscale image into the convolutional neural network for training;

Inputting a plurality of the selected color standard maps into the target network, acquiring the color information of the color standard maps and inputting them into the convolutional neural network for training;

A loss function is calculated until the convolutional neural network outputs a plurality of colormaps with the color information.

Preferably, in the method for coloring multiple target areas in the above picture provided by the embodiment of the present invention, the convolutional neural network includes ten convolutional layers; the first three convolutional layers are used for downsampling; the last three convolutional layers are used for downsampling; the convolutional layers are used for upsampling;

The target network includes four convolutional layers; the output of the target network is added to the fourth convolutional layer of the convolutional neural network.

Preferably, in the coloring method for multiple target areas in the above picture provided by the embodiment of the present invention, identifying and segmenting the colorized multiple target areas specifically includes:

The colored target regions are identified and segmented through the Mask R-CNN network.

Preferably, in the method for coloring multiple target areas in the above-mentioned pictures provided by the embodiment of the present invention, a MaskR-CNN network is used to identify and segment a plurality of target areas after coloring, which specifically includes:

Extract the feature map from the picture to be colored by using the ResNet-FPN network;

Use the RPN network to generate a candidate frame on the feature map, and mark the specific position of the target area through the candidate frame;

The candidate frame is input into RoIAlign to extract features, and a mask corresponding to each of the target regions is obtained;

Use the softmax function to output the probability to obtain multiple example classes and 1 background class;

perform linear regression on the mask and the candidate frame;

The colored grayscale image is input into the Mask R-CNN network for training, and multiple colored target regions are segmented.

Preferably, in the method for coloring multiple target areas in the above picture provided by the embodiment of the present invention, an RPN network is used to generate a candidate frame on the feature map, and the specific position of the target area is marked by the candidate frame, Specifically include:

slide the kernel on the feature map;

Mapping the center of the core back to the picture to be colored, generating multiple candidate frames with a set size at the center, and judging whether the candidate frame includes the target area;

If so, fine-tune the candidate frame, so that the candidate frame marks the specific position of the target area.

Preferably, in the method for coloring multiple target areas in the above picture provided by the embodiment of the present invention, the multiple target areas after segmentation are fused to the corresponding positions of the picture to be colored, which specifically includes:

A smoothing filter is used to fuse the segmented target regions into the corresponding positions of the to-be-colored picture and smooth the edge transition.

The embodiment of the present invention also provides a coloring device for multiple target areas in a picture, including:

The grayscale processing module is used to perform grayscale processing on the to-be-colored image to obtain a grayscale image;

A standard picture selection module, used for selecting a plurality of color standard pictures that carry out different color designs on the picture to be colored;

a grayscale image coloring module, configured to colorize the grayscale image respectively by selecting a plurality of the color standard images;

The target area segmentation module is used to identify and segment multiple target areas after coloring;

The target area fusion module is used to fuse the segmented target areas into corresponding positions of the picture to be colored.

An embodiment of the present invention further provides a coloring device for multiple target areas in a picture, including a processor and a memory, wherein the processor implements the above-mentioned as provided in the embodiment of the present invention when the processor executes the computer program stored in the memory Coloring method for multi-target areas in images.

Embodiments of the present invention further provide a computer-readable storage medium for storing a computer program, wherein, when the computer program is executed by a processor, the method for coloring multi-target areas in the above picture as provided by the embodiments of the present invention is implemented .

A coloring method, device, device and storage medium for multi-target areas in a picture provided by the present invention include: performing grayscale processing on the to-be-colored picture to obtain a grayscale image; selecting the to-be-colored picture for different Multiple color standard images for color design; color the grayscale image through the selected multiple color standard images; identify and segment multiple colored target areas; fuse the segmented multiple target areas into The corresponding position of the picture to be colored. The invention uses a plurality of existing color-coordinated color standard pictures to colorize a plurality of target areas in the picture to be colored, changes the color information of the plurality of target areas in the picture to be colored, saves a lot of time, and also It can make the color of the picture not too monotonous, realize the diversity and flexibility of coloring, and meet different design needs.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

1 is a flowchart of a method for coloring multiple target areas in a picture according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an apparatus for coloring multiple target areas in a picture according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present invention provides a method for coloring multi-target areas in a picture, as shown in Figure 1, comprising the following steps:

S101. Perform grayscale processing on the to-be-colored image to obtain a grayscale image;

S102, selecting multiple color standard images for different color designs of the image to be colored;

S103, coloring the grayscale images respectively through the selected multiple color standard images;

S104, identifying and segmenting the colored target areas;

S105 , fuse the divided multiple target regions into corresponding positions of the picture to be colored.

In the coloring method for multiple target areas in the above-mentioned picture provided by the embodiment of the present invention, firstly, grayscale processing (OpenCV can be used) is performed on the picture to be colored to obtain a grayscale image; The target area is designed with multiple color standard maps in different colors; the color standard map is fused into the grayscale map; then the colored target area is identified and segmented; finally, the segmented target area is fused into the original color map . In this way, multiple existing color-coordinated color standard images are used to colorize multiple target areas in the image to be colored, and the color information of multiple target areas in the image to be colored can be changed, which saves a lot of time, and can also save a lot of time. Make the color of the picture not too monotonous, realize the variety and flexibility of coloring, and meet different design needs.

Further, in the specific implementation, in the coloring method of the multi-target area in the above-mentioned picture provided by the embodiment of the present invention, step S103 colorizes the grayscale image by selecting a plurality of color standard images, which may specifically include the following: step:

Step 1: Train a convolutional neural network M for coloring and a target network U for acquiring color information; the above-mentioned convolutional neural network M includes ten convolutional layers, the first three convolutional layers are used for downsampling, and the latter are used for downsampling. Three convolutional layers are used for upsampling; the target network U above includes four convolutional layers, and the output of the target network U is added to the fourth convolutional layer of the convolutional neural network M; the two networks trained can make The color coordination between the parts of the picture to be colored does not change the content information of the picture;

Step 2: Input the image to be colored and the grayscale image into the convolutional neural network M for training;

Step 3. Input the selected multiple color standard maps into the target network U, obtain the color information of the color standard maps and input them into the convolutional neural network M for training;

Step 4: Calculate the loss function L until the convolutional neural network outputs multiple color maps (colored grayscale images) with color information.

Specifically, train the network M to minimize the following objective function, where the loss function L represents the distance between the output of the network and the true value, update θ to minimize the loss function, X∈R ^H×W×1 denotes the grayscale image, Y∈ R ^H×W×2 represents the picture to be colored (i.e. color image), C∈R ^H×W×2 represents the selected color standard image, for each group D includes X, C and Y, E represents the mean value, H and W represent the height and width of the image, respectively, and R represents the real number domain. These mappings are learned by the convolutional neural network M:

Similar to the above formula, E represents the mean, D represents the distribution of X and Y, and the objective function of minimizing the network U is used to obtain the color:

The above two objective functions are defined by Huber(smooth-l ₁ ) loss, which can solve the situation of blurred pictures and provide end-to-end learning, that is, the general expressions of the above two loss functions are as follows:

Where x and y are variables, the objective functions of networks M and U can be obtained by substituting x=M(X, C; θ) or x=M(X, C; θ _U ) and y=Y into the above formula.

Further, during specific implementation, in the method for coloring multiple target areas in the above-mentioned picture provided by the embodiment of the present invention, step S104 identifies and divides a plurality of target areas after coloring, which may specifically include: using Mask R -CNN (Regions with CNN features, edge neural network) to identify and segment multiple target regions after coloring.

It should be noted that there are many ways to divide pictures. The present invention adopts the Mask R-CNN network, which can be divided into three sub-networks, including a feature extraction network, a candidate frame generation network and a classification network. The feature maps are shared among the sub-networks, which can accurately identify and classify the colored target regions. Specifically, the network is divided into three losses: class, box and mask. Class represents the classification of the target, box represents adding a frame to the target, and mask represents a target with the same pixel.

Further, during specific implementation, in the coloring method for multiple target areas in the above-mentioned pictures provided by the embodiment of the present invention, the Mask R-CNN network is used to identify and segment a plurality of target areas after coloring, which may specifically include: The following steps:

The first step is to use the ResNet-FPN network to extract the feature map from the image to be colored;

The second step is to use the RPN network (Region Proposal Network) to generate a candidate frame (Region of Interest, RoI) on the feature map, and mark the specific position of the target area through the candidate frame;

The third step is to input the candidate frame into RoIAlign to extract features, and obtain the mask corresponding to each target area;

The fourth step, use the softmax function to output the probability to obtain multiple example classes and one background class;

The fifth step is to perform linear regression on the mask and candidate frame to make it more suitable for the target;

The sixth step is to input the colored grayscale image into the Mask R-CNN network for training, and segment the colored target regions.

Further, in the specific implementation, in the method for coloring multiple target areas in the above-mentioned pictures provided by the embodiment of the present invention, in the above-mentioned second step, the RPN network is used to generate a candidate frame on the feature map, and the target area is marked by the candidate frame. The specific location of the image can include the following steps: slide the 3×3 kernel (kernel) on the feature map; map the center of the kernel back to the image to be colored, and generate a variety of set sizes (such as 9 sizes at the center) , 3 ratios: 128 ² , 256 ² , 512 ² ; 3 kinds of candidate frames with aspect ratios: 1, 0.5, 2), determine whether the candidate frame includes the target area; if so, fine-tune the candidate frame to make The candidate frame marks the specific location of the target area.

Further, during specific implementation, in the method for coloring multiple target areas in the above-mentioned pictures provided by the embodiment of the present invention, step S105 fuses the divided multiple target areas into corresponding positions of the picture to be colored, which may specifically include: : Integrate the segmented target areas into the corresponding positions of the image to be colored through the smoothing filter and smooth the edge transition.

It should be noted that, using a smoothing filter, the segmented target area (foreground image) can be smoothly merged into the image to be colored (background image), so that the foreground image and the background image are seamlessly connected, including the following steps:

An edge-oriented smoothing filter a _{(u, v)} is proposed to filter the fused image h, and b(u) represents the filtered image, which is expressed as follows:

Among them, T _s represents the set of the origin in the center space of the smoothing filter, and u=[u _x , u _y ] ^T , v=[v _x , v _y ] ^T is the spatial position represented in the form of a vector.

The filter a _(u,v) is represented as follows:

where γ is a normalization parameter satisfying the condition ∑a=0, σ is a diffusion parameter, g(u) and θ(u) are the edge strength and edge direction at the spatial position u, respectively, E(g( u)) is a monotonically increasing function, satisfying E(0)≥1, where E(g(u))=β(g(u))+1, β>0 is the edge strength parameter, and the matrix G _(u) makes The weight value along the edge direction is greater than the vertical edge direction weight value.

When calculating the coefficients of the edge-oriented smoothing filter, firstly calculate the edge intensity and edge direction at a certain spatial position of the image, these two parameters can be obtained by the Sobel operator. h` <sub>x</sub> , h` _y are the result of Sobel operator processing the image h in the horizontal and vertical directions. The edge direction θ and edge strength g are expressed as follows:

The smoothing filter coefficients can be obtained by substituting the above formula into the above formula.

In the present invention, a smoothing filter is applied when merging the divided target areas and the picture to be colored to make the connection between the two more harmonious. The smoothing filter can be divided into two parts, and one part is used to find the colorized target area. In the corresponding position in the picture to be colored, the other part makes a smooth transition between the two.

Based on the same inventive concept, the embodiment of the present invention also provides a coloring device for multi-target areas in a picture, because the principle of solving the problem of the coloring device for multi-target areas in the picture is the same as that of the above-mentioned one for the multi-target areas in the picture. The coloring methods are similar, so the implementation of the coloring device for the multi-target areas in the picture can refer to the implementation of the coloring method for the multi-target areas in the picture, and the repetition will not be repeated.

During specific implementation, the coloring device for multi-target areas in a picture provided by the embodiment of the present invention, as shown in FIG. 2 , specifically includes:

The grayscale processing module 11 is used to perform grayscale processing on the to-be-colored picture to obtain a grayscale image;

The standard map selection module 12 is used to select a plurality of color standard maps for different color designs of the pictures to be colored;

The grayscale image coloring module 13 is used for coloring the grayscale image through the selected multiple color standard images;

The target area segmentation module 14 is used to identify and segment a plurality of target areas after being colored;

The target area fusion module 15 is used to fuse the divided multiple target areas into corresponding positions of the picture to be colored.

In the coloring device of the multi-target area in the above-mentioned picture provided by the embodiment of the present invention, through the interaction of the above-mentioned four modules, a plurality of existing color-coordinated color standard pictures can be used to treat a plurality of targets in the colored picture Coloring the area saves a lot of time, and also makes the color of the picture not too monotonous, realizes the diversity and flexibility of coloring, and meets different design needs.

For more specific working processes of the above-mentioned modules, reference may be made to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

Correspondingly, the embodiment of the present invention also discloses a coloring device for multi-target areas in a picture, including a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the multi-target in the picture disclosed in the foregoing embodiments is realized. The coloring method of the area.

For a more specific process of the above method, reference may be made to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

Further, the present invention also discloses a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the aforementioned method for coloring multiple target areas in a picture is implemented.

For a more specific process of the above method, reference may be made to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. For the apparatuses, devices, and storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and reference may be made to the descriptions of the methods for related parts.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the possibilities of hardware and software. Interchangeability, the above description has generally described the components and steps of each example in terms of functionality. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

The steps of a method or algorithm described in conjunction with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by a processor, or a combination of the two. A software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

An embodiment of the present invention provides a coloring method, device, device, and storage medium for multiple target areas in a picture. The method includes: performing grayscale processing on a to-be-colored picture to obtain a grayscale image; Multiple color standard images designed with different colors; color the grayscale image through the selected multiple color standard images; identify and segment multiple colored target areas; fuse the segmented multiple target areas Go to the corresponding position of the picture to be colored. The invention uses a plurality of existing color-coordinated color standard pictures to colorize a plurality of target areas in the picture to be colored, changes the color information of the plurality of target areas in the picture to be colored, saves a lot of time, and also It can make the color of the picture not too monotonous, realize the diversity and flexibility of coloring, and meet different design needs.

Finally, it should also be noted that in this document, relational terms are only used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual existence between these entities or operations. relationship or order. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The method, device, device and storage medium for coloring multi-target areas in pictures provided by the present invention have been described in detail above. Specific examples are used in this paper to illustrate the principles and implementations of the present invention. It is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific embodiments and application scope. In summary, The contents of this specification should not be construed as limiting the present invention.

Claims (9)

1. a coloring method of multiple target areas in a picture, is characterized in that, comprises: Perform grayscale processing on the to-be-colored image to obtain a grayscale image; Selecting a plurality of color standard maps for carrying out different color designs on the picture to be colored; Coloring the grayscale image by selecting a plurality of the color standard images; Identify and segment multiple colored target areas; Fusing the segmented multiple target regions into the corresponding positions of the picture to be colored; The coloring of the grayscale image through the selected multiple color standard images, specifically includes: Train a convolutional neural network for coloring and a target network for obtaining color information; Inputting the picture to be colored and the grayscale image into the convolutional neural network for training; Inputting a plurality of the selected color standard maps into the target network, acquiring the color information of the color standard maps and inputting them into the convolutional neural network for training; A loss function is calculated until the convolutional neural network outputs a plurality of colormaps with the color information. 2. The coloring method for multiple target areas in a picture according to claim 1, wherein the convolutional neural network comprises ten convolutional layers; the first three convolutional layers are used for downsampling; three of the convolutional layers for upsampling; The target network includes four convolutional layers; the output of the target network is added to the fourth convolutional layer of the convolutional neural network. 3. the coloring method of multi-target areas in the picture according to claim 1, is characterized in that, identifying and dividing out multiple target areas after coloring, specifically comprises: The colored target regions are identified and segmented through the Mask R-CNN network. 4. the coloring method of multi-target area in the picture according to claim 3, is characterized in that, identify and divide out multiple target areas after coloring by Mask R-CNN network, specifically comprise: Extract the feature map from the picture to be colored by using the ResNet-FPN network; Use the RPN network to generate a candidate frame on the feature map, and mark the specific position of the target area through the candidate frame; The candidate frame is input into RoIAlign to extract features, and a mask corresponding to each of the target regions is obtained; Use the softmax function to output the probability to obtain multiple example classes and 1 background class; perform linear regression on the mask and the candidate frame; The colored grayscale image is input into the Mask R-CNN network for training, and multiple colored target regions are segmented. 5. The coloring method of multiple target areas in a picture according to claim 4, wherein a candidate frame is generated on the feature map by using an RPN network, and the specific position of the target area is marked by the candidate frame , including: slide the kernel on the feature map; Mapping the center of the core back to the picture to be colored, generating multiple candidate frames with a set size at the center, and judging whether the candidate frame includes the target area; If so, fine-tune the candidate frame, so that the candidate frame marks the specific position of the target area. 6. The coloring method of multiple target areas in a picture according to claim 1, wherein the divided multiple target areas are fused to the corresponding positions of the picture to be colored, specifically comprising: A smoothing filter is used to fuse the segmented target regions into the corresponding positions of the to-be-colored picture and smooth the edge transition. 7. A coloring device for multi-target areas in a picture, comprising: The grayscale processing module is used to perform grayscale processing on the to-be-colored image to obtain a grayscale image; A standard picture selection module, used for selecting a plurality of color standard pictures that carry out different color designs on the picture to be colored; a grayscale image coloring module, configured to colorize the grayscale image through the selected plurality of the color standard images; The target area segmentation module is used to identify and segment multiple target areas after coloring; a target area fusion module, which is used to fuse the segmented target areas to the corresponding positions of the picture to be colored; The grayscale image coloring module is specifically used for training a convolutional neural network for coloring and a target network for acquiring color information; Inputting the picture to be colored and the grayscale image into the convolutional neural network for training; Inputting a plurality of the selected color standard maps into the target network, acquiring the color information of the color standard maps and inputting them into the convolutional neural network for training; A loss function is calculated until the convolutional neural network outputs a plurality of colormaps with the color information. 8. A coloring device for multi-target areas in a picture, comprising a processor and a memory, wherein the processor implements any one of claims 1 to 6 when the processor executes a computer program saved in the memory The coloring method of the multi-target area in the picture. 9. A computer-readable storage medium, characterized in that it is used for storing a computer program, wherein, when the computer program is executed by a processor, the multi-target area in the picture according to any one of claims 1 to 6 is implemented. Coloring method.

Images