RU2524869C1 - Device for colouring black and white image - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: device comprises RGB→NTSC converter units (1), (20), a label mask selection unit (2), multiplier units (3), (18), (19), a breakdown unit (4), a dispersion calculation unit (6), a weight function generating unit (10), a label coordinate determining unit (5), buffer units (7), (8), (12), (21) delay units (11), (13), (14), (15), (16), a sparse matrix generating unit (17), a control unit (9), a clock-pulse generator (22).

EFFECT: reduced colour distortion when colouring monochromatic images.

1 dwg

Description

The invention relates to the field of computer technology and can be used in digital, television and photo systems.

The main task to be solved is the colorization of black and white images.

Colorization (coloring) is a computer process of adding color to a monochromatic (black and white) image or video. This process typically involves segmenting the image in areas and tracking these areas through a sequence of images. Modern methods and devices do not allow the implementation of reliable segmentation in practice, therefore, coloring requires operator intervention, requiring a large amount of time.

Currently, in Russia and abroad, a lot of work is underway to restore the color of old films and photographic materials. In addition to cinema, color restoration methods for digital grayscale images can be applied in areas such as restoration of archive photographs and film videos, recovery from incorrect gamma correction.

Simplified methods for staining monochromatic images can be divided into the following groups:

1) Methods using color coding.

2) Automatic staining methods using reference images.

Analysis of existing staining methods shows that they

require a priori information about the object. The use of automatic staining methods fills all local areas of the image with color and requires that the colored areas in the image have different brightness groups. In images where these parameters are similar, a visual change in the color of image pixels is observed, i.e. non-coloring of image pixels. In addition, to use automatic staining methods, a reference image is required, the colors of which are similar to the desired colors of the processed image. To use methods based on color marking, it is necessary to apply color markers in local areas of the image, the colors of which can be selected based on expert estimates, or the “desired” colors can be specified. It should be noted that the staining result directly depends on the method of choosing color markers, as well as on the amount of their application.

A known method and system for digital colorization of images [Patent No. 5,093,717, USA, IPC H04N 1/46]. The method is based on manually setting masks of image objects having similar shades, and defining for each mask the function of color transfer, converting the information of the image brightness scale into unique values of hue, brightness and saturation. The value of the brightness scale within each area is processed using the color transfer function. The result of operations is saved for later retrieval and display.

The signs of the method and system - analogue, coinciding with the signs of the proposed technical solution, the following: staining a monochromatic image.

The disadvantages of the known method and system are: in images where multi-colored areas have similar brightness groups, a color change is possible; The manual task of the areas of colorization is used.

A known method for restoring color grayscale images based on a neural network [Varlamov AD “Restoration of color of grayscale images by a neural network” // Algorithms, methods and data processing systems - Murom: publishing house Murom Institute (branch) of the State educational institution of higher professional education "Vladimir State University named after Alexander Grigoryevich and Nikolay Grigoryevich Stoletov", 2010. - p.6]. This method is based on machine learning, that is, a technology is used in which an algorithm for converting a grayscale image to a color image will be automatically generated based on examples selected by a person. Colorization is carried out by the following steps: on color images with a homogeneous content type (for example, sunset, road asphalt and others) objects of the same type have approximately the same color. To evaluate the color of each point of the original grayscale image, the following information is used regarding it and the image itself:

1. Type of image scene (determines what is depicted on the scene).

2. The brightness of the point.

3. A set of values of local features that will allow more accurately determine the type of object and, accordingly, more accurately "pick" the color.

The scene type of the image determines which objects can be present on the scene. The brightness of the point is taken from the input grayscale image; values of local features are calculated by deterministic image processing algorithms. The “selection” of color is carried out directly by a neural network, the input of which receives a set of values of local signs and the brightness of the point, and the intensities of the color components (red, green, and blue) are formed at the output. The network is pre-trained for each type of scene.

The signs of the method and system - analogue, coinciding with the signs of the proposed technical solution, the following: staining a monochromatic image.

The disadvantages of the known method and system are: in images where multi-colored areas have similar brightness groups, a color change is possible; the impossibility of the task and training of all types of scenes; errors in the training of the neural network.

A known method and device for coloring images [Patent No.US 2009/0096807 A1]. This method of coloring images includes: at the first step, the first color is assigned from the color map to the maximum brightness pixel to determine the first graphic element; in the next step, the second color of the color map is assigned to a pixel of a different brightness; recalculating the brightness of the first graphic element and the second graphic element; in the next step, the brightness associated with the first graphic element is adjusted relative to the brightness in the first predetermined brightness range and the saturation of the graphic element is corrected; the procedure is repeated for the second graphic element with respect to brightness in the second predetermined range.

The structural diagram of a device that implements the considered algorithm contains a block for determining the first graphic element; a block for determining a second graphic element; block for calculating the first brightness; block for calculating the second brightness; brightness control unit; condition check block; saturation correction unit; feed unit for the next data point.

A known method of converting a black and white image into color [T. Welsh, M. Ashikhmin, K. Mueller "Transferring Color to Greyscale Images" // IJCSNS International Journal of Computer Science and Network Security, 2007 p. 1-4.]. This method uses a color (source) image represented in RGB space and black and white (target) image. At the first step, they are transformed into a decorrelated lαβ color space for subsequent analysis. In order to transfer color values from a color image to a monochromatic image, it is necessary that each pixel in a black-and-white image be matched with a pixel in a color image. The comparison is based on brightness values and neighborhood statistics for a given pixel. The brightness value is determined by the l channel in the lαβ color space. In order to reduce the differences in brightness between the two images, the brightness conversion is performed by linearly moving and scaling the histogram of the brightness of the color image.

Since most of the visually significant differences between the pixel values are associated with differences in brightness, it is possible to limit the number of samples used as the source color pixels. This reduces the number of comparisons made for each pixel in the monochromatic image and reduces the calculation time. For each pixel of the monochromatic image in the raster line, the most suitable color matching sample is selected, which is selected depending on the weighted average brightness (50%) and standard deviation (50%). As soon as the best pixel match is found, α and β the color values are transmitted to the target pixel, while the original brightness value is stored.

The features of the method is an analogue that coincides with the features of the proposed technical solution, the following: staining black and white images.

The disadvantages of this method are: if the areas in the reference image do not have similar brightness values for the corresponding structures in the original image, then there is a color change of local areas of the image; high computing costs; as a result of using the iterative algorithm, a large processing time is required.

Closest to the invention is a method and device for colorization using optimization according to color intensity [Patent No.US 2010/0085372 A1]. The present invention produces the addition of color to an image or video sequence or changes the color of one or more elements in the image.

в соседних пикселях s:The colorization method is implemented using the following algorithm. In the first step, the sum of the squares of the difference between the color U (r) in pixel r and the weighted average of the color is minimized $$\underset{s\in N\left(r\right)}{\Sigma }{w}_{rs}U\left(s\right)$$

in adjacent pixels s:

$$J(U)=\underset{r}{\Sigma }{\left(U(r)-\underset{s\in N\left(r\right)}{\Sigma }U\left(s\right)\right)}^2\left(\mathrm{one}\right)$$

where w _rs is the weight function taking large values at Y (r) ~ Y (s), and small, when these two intensities are different, Y (r) is the monochromatic intensity of the pixel r, s∈N (r) is the region of pixels around r.

The solution of the objective function (1) is carried out using two weight functions w _rs .

The first weighting function w _rs uses image segmentation algorithms and is based on the squared difference between the two intensities:

$$w_{rs}\raisebox{1ex}{$\infty e-{\left(Y\left(r\right)-Y\left(s\right)\right)}^2$}\!\left/ \!\raisebox{-1ex}{$2{\sigma }_r^2$}\right.\left(2\right)$$

The second weight function w _{rs is} based on the normalized correlation between two intensities:

$$w_{rs}\infty \mathrm{one}+\frac{\mathrm{one}}{{\sigma }_r^2}\left(Y\left(r\right)-{\mu }_r\right)\left(Y\left(s\right)-{\mu }_r\right)\left(3\right)$$

where µ _r and σ _r are the average value and the variance of the intensity in the region N (r).

The correlation is found by adopting a local linear relationship between color and intensity. It is assumed that the color of the pixel U (r) is a linear function of the intensity Y (r):

$$U\left(r\right)=a_{i}Y\left(r\right)+b_i\left(\mathrm{four}\right)$$

where a _i , b _i are linear coefficients that are the same for all pixels in a small neighborhood of N (r).

Given the number of locations r _i , where the colors are given by the source markers u (r _i ) = u _i ; ν (r _i ) = ν _{i is} minimized by J (U), J (V) of expression (1) taking into account the restrictions introduced by equations (2) - (4).

The signs of the method is an analogue that coincides with the features of the proposed technical solution, the following: the use of color marking, staining a monochromatic image.

The disadvantages of the known method and device that implements it are: the result of the restoration of colors to a greater extent depends on the choice of colors of markers, local areas for their application and the area of staining; a slight change in the initial parameters can lead to significant color distortions; high computing costs.

The block diagram of a device that implements the considered algorithm contains a block for determining the intensity values of images, a block for entering constraints, a block for determining coordinates of markers, and a block for minimizing the objective function.

The proposed device for the colorization of black and white images allows you to color black and white images, to replace colors on the image. The device implements the following algorithm. At the first step, the operator manually applies color markers to known areas and converts the original image I with color markers applied from RGB color space to YUV color space.

The basic equations that are used to convert between RGB and YUV space are:

Y = 0.299 R + 0.587 G + 0.114 V;

$$\begin{array}{l}Y=0.299\cdot R+0.587\cdot G+0.114\cdot B;\hfill \\ U=0.492\cdot \left(B-Y\right)=-0.147\cdot R-0.289\cdot G+0.436\cdot B;\left(5\right)\hfill \\ V=0.877\cdot \left(R-Y\right)=0.615\cdot R-0.515\cdot G+\mathrm{0,100}\cdot B.\hfill \end{array}$$

As a result of the conversion, three channels are formed: the channel of the luminance component Y; the channel of the first color difference component U; channel of the second color difference component V.

At the second step, the marker mask is selected, which is a binary array m whose elements take the value 1 if they correspond to the marked areas, otherwise the value 0 is accepted.

At the next step, the initial image is sequentially divided by a 3 × 3 sliding window into overlapping subregions. In all local areas, the variance is calculated and the absolute x, y coordinates of the markers are extracted.

In the fourth step, based on the calculated variance, the weighting coefficients are calculated for each element of the subblocks defined by expression (2).

At the next step, a sparse matrix A of dimension N × n is formed based on weighting coefficients and their absolute x, y coordinates:

$$N=h\cdot w\left(6\right)$$

where h is the image height, w is the image width.

At the sixth step, the rarefied matrix A is multiplied by the vector of the color difference channel U and V.

At the end, the painted image is converted from the YUV color space to the RGB color space.

The device for colorization of black-and-white images (Fig. 1) contains an RGB → NTSC 1 converter unit and marker mask extraction unit 2, the inputs of which are information inputs of the device, the output of marker mask extraction unit 2 is connected to the first input of multiplier unit 3; the first output of the RGB → NTSC converter block 1 is connected to the second input of the multiplier block 3, the output of which is connected to the first input of the partition block 4, the output of which is connected to the input of the dispersion calculation block 6, to the second input of the weight function formation block 10, to the input of the coordinate determination block markers 5; the output of the dispersion calculation unit 6 is connected to the first input of the weight function forming unit 10, the output of which is connected to the first input of the buffer 12, the output of which is connected to the first input of the sparse matrix forming unit 17; the first output of the marker coordinate determining unit 5 is connected to the first input of the buffer unit 7, the output of which is connected to the input of the delay unit 13, the output of which is connected to the second input of the sparse matrix forming unit 17; the second output of the marker coordinate determining unit 5 is connected to the first input of the buffer unit 8, the output of which is connected to the input of the delay unit 14, the output of which is connected to the third input of the sparse matrix forming unit 17; the first output of the RGB → NTSC converter block 1 is connected to the input of the control unit 9, the first output of which is connected to the second input of the buffer block 8, to the second input of the buffer block 7, to the second input of the buffer block 12; the second output of the control unit 9 is connected to the second input of the partitioning unit 4, to the fourth input of the sparse matrix forming unit 17; the second output of the RGB → NTSC 1 converter block is connected to the input of the delay block 11, the output of which is connected to the second input of the multiplier block 18; the third output of the RGB → NTSC 1 converter block is connected to the input of the delay block 15, the output of which is connected to the second input of the multiplier block 19; the output of the sparse matrix forming unit 17 is connected to the first input of the multiplier unit 18, to the first input of the multiplier unit 19; the output of the multiplier 18 is connected to the second input of the NTSC → RGB 20 converter block; the output of the multiplier 19 is connected to the third input of the NTSC → RGB 20 converter block; the first output of the RGB → NTSC converter block 1 is connected to the input of the delay block 16, the output of which is connected to the first input of the NTSC → RGB 20 converter block, the output of which is connected to the input of the buffer block 21, the output of which is the information output of the device. The synchronization of the device is provided by the clock generator 22.

The device colorization of black and white images works as follows. At the same time, the input image with the color markers applied to it enters the input of the RGB → NTSC 1 converter block and the input of the marker mask selection block 2. In the marker mask highlighting block 2, the mask of color markers applied to the image is selected, which is a binary array whose elements, which take the value 1, correspond to the marked areas, otherwise 0. In the RGB → NTSC 1 conversion block, the input image is converted from RGB to NTSC format. Three NTSC format image channels are formed at the outputs of the RGB → NTSC 1 conversion unit: the channel of the luminance component Y is formed at the first output, the first color-difference component U is formed at the second output, and the second color-difference component V is at the third output. A signal is sent to the inputs of the multiplier unit 3 the luminance channel Y of the RGB → NTSC 1 conversion unit and the marker mask from the output of the marker mask selection block 2, the output of which is a matrix of areas marked with color. The result of the multiplication from the output of the block of the multiplier 3 goes to the partition block 4, where the initial image is sequentially divided by the sliding window 3 × 3 into overlapping subregions. Information about the size of the sliding window comes from the second output of the control unit 9 to the second input of the partition unit 4. From the output of the partition unit 4, subblocks are sequentially transferred to the dispersion calculation block 6, in which the dispersion values for each subblock are calculated. In the block for generating the weight function 10, the weighting coefficients for each element of the subblocks are calculated based on the calculated variance, the value of which is supplied to the first input of the block for the formation of the weighting function 10, and its second input subblocks of the original image from the output of the partition block 4. Values of the calculated weighting coefficients from the output of the block for the formation of the weight function 10, they arrive at the block of the buffer 12. From the output of the block 4, the block-by-block values are transferred to the block for determining the coordinates of markers 5 for calculating lute x, y coordinate values included in the current sub-block, relative to the original processed image. The values of the absolute x coordinates from the first output of the block for determining the coordinates of the markers 5 go to the first input of the block of the buffer 7. At the same time, the values of the absolute coordinates from the second output of the block for determining the coordinates of the markers 5 go to the first input of the block of the buffer 8. The sizes of the blocks of buffers 7, 8 and 12 are set the value from the control unit 9, determined on the basis of the channel of the luminous component Y of the original processed image, which is fed to the input of the control unit 9 from the first output of the converter block RGB → NTSC 1.

The calculated values of x coordinates from the output of the buffer block 7 go to the delay block 13. The calculated values of the coordinates from the output of the buffer 8 block go to the delay block 14. The storage time of the results in the delay blocks 13 and 14 is set in such a way as to ensure synchronization of the arrival of weight values and their absolute x, y coordinates to the first, second and third input of the sparse matrix forming unit 17. The size of the generated sparse matrix is determined by the control unit 9, the second output of which is connected to the fourth input of the form block sparse matrix 17, where a sparse matrix is formed on the basis of weighting coefficients and their absolute x, y coordinates. From the second output of the converter block RGB → NTSC 1, the signal in the form of a vector of values of the first color-difference component U is supplied to the delay unit 11, the output of which is transmitted to the second input of the multiplier unit 18, the first input of which is the sparse matrix from the output of the sparse matrix forming unit 17. In the block of the multiplier 18, matrix dilution of the sparse matrix by the vector of the color difference channel U is carried out. From the third output of the converter block RGB → NTSC 1, the signal in the form of a vector of values of the second color of the real component V enters the delay block 15, from the output of which is transmitted to the second input of the multiplier block 19, the first input of which receives the sparse matrix from the output of the sparse matrix forming unit 17. In the multiplier block 19, the matrix of the sparse matrix is multiplied by the color-difference channel V The information storage time in the delay blocks 11, 15 is set in such a way as to ensure the synchronization of data from the output of the sparse matrix forming unit 17 and the second, third ode of the converter block RGB → NTSC 1. At the outputs of the blocks of the multipliers 18 and 19, signals of the first U and second V color-difference component of the colorized image are generated, which are fed to the second and third input of the converter block NTSC → RGB 20. To restore the full image, the signal of the luminance channel Y with the first output of the converter block RGB → NTSC 1 is fed to the input of the delay block 16, the output of which is fed to the first input of the converter block NTSC → RGB 20. The storage time of the information in the delay block 16 is set so that both to synchronize the arrival of the signal of the luminance channel Y, the signal of the first color-difference component U of the colorized image and the signal of the second color-difference component V of the colorized image to the first, second and third inputs of the NTSC → RGB 20 converter block. In the NTSC → RGB 20 converter block, the colorized image is converted from the format NTSC to RGB format. From the output of the NTSC → RGB 20 converter block, the colorized image enters the buffer block 21, the output of which is the information output of the device. The synchronization of the device is provided by the clock generator 22.

EFFECT: colorization of monochromatic images under limited initial conditions.

Claims (1)

A device for colorization of black-and-white images containing a block for determining the coordinates of markers (5), characterized in that the information inputs of the device are the input of the converter block RGB → NTSC (1) and the input of the block marking mask masks (2), the output block marking mask markers (2) connected to the first input of the multiplier block (3); the first output of the RGB → NTSC converter block (1) is connected to the second input of the multiplier block (3), the output of which is connected to the first input of the splitting block (4), the output of which is connected to the input of the dispersion calculation block (6), to the second input of the weight formation block functions (10); the output of the dispersion calculation unit (6) is connected to the first input of the weight function forming unit (10), the output of which is connected to the first input of the buffer (12), the output of which is connected to the first input of the sparse matrix forming unit (17); the output of the buffer unit (7) is connected to the input of the delay unit (13), the output of which is connected to the second input of the sparse matrix forming unit (17); the output of the buffer unit (8) is connected to the input of the delay unit (14), the output of which is connected to the third input of the sparse matrix forming unit (17); the first output of the RGB → NTSC converter block (1) is connected to the input of the control unit (9), the first output of which is connected to the second input of the buffer block (8), to the second input of the buffer block (7), to the second input of the buffer block (12); the second output of the control unit (9) is connected to the second input of the partition unit (4), to the fourth input of the sparse matrix forming unit (17); the second output of the RGB → NTSC converter unit (1) is connected to the input of the delay unit (11), the output of which is connected to the second input of the multiplier unit (18); the third output of the RGB → NTSC converter unit (1) is connected to the input of the delay unit (15), the output of which is connected to the second input of the multiplier unit (19); the output of the sparse matrix forming unit (17) is connected to the first input of the multiplier unit (18), to the first input of the multiplier unit (19); the output of the multiplier (18) is connected to the second input of the NTSC → RGB converter block (20); the output of the multiplier (19) is connected to the third input of the NTSC → RGB converter block (20); the first output of the RGB → NTSC converter block (1) is connected to the input of the delay block (16), the output of which is connected to the first input of the NTSC → RGB converter block (20), the output of which is connected to the input of the buffer block (21), the output of which is an information output devices, the synchronization of the device is provided by the clock generator (22).