RU2534005C2 - Method and system for converting screenshot into metafile - Google Patents

Abstract

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to means of converting a raster image into a metafile. The method comprises detecting fragments of smoothed text on a screenshot, evaluating the background colour and colour of the characters of the smoothed text on each fragment based on determining outermost columns of a histogram of the fragment, vectorising the edge of the characters of the smoothed text, modifying the screenshot by replacing fragments of the smoothed text with the background colour, creating a metafile based on the modified screenshot and vectorised edges, filled with the colour of the characters of the smoothed text, and rendering the formed metafile.

EFFECT: high definition of text characters on a screenshot.

9 cl, 9 dwg

Description

The invention relates to the field of digital signal processing, and in particular to methods for converting a raster image to a metafile, in particular, to converting a screen shot (screenshot) to a metafile by segmenting and vectorizing a screenshot.

Vectorization of textual information on a screenshot bitmap can significantly improve the quality of text for printing and visualization. Partial vectorization of a screenshot is a promising approach to solving the problem and lies in the fact that the detected areas of the drawings and text areas are processed in different ways. The text part is translated into a sequence of straight and curved segments with information about the color of the characters and the surrounding background. The rest of the information in the original screenshot is stored as a bitmap.

The method disclosed in US patent No. 8,270,722 [1], provides for image processing with selective vectorization of characters and graphic areas. The aim of the invention is to achieve the best degree of compression with high quality images containing both areas of text and graphics. If the pixels of the symbol in the test area are overlapped by the graphics area, then the graphics area is vectorized, if there is no overlap, the text area is vectorized first.

The method disclosed in US patent No. 7,873,218 [2], is to approximate a symbol to convert a binary image into contour data using Bezier curves. The method includes raster image segmentation for extracting symbol regions and extracting symbols from these regions. The vectorization process is performed independently of other non-text parts of the image.

The above methods can improve the visual quality of the image by vectorization, but cannot be directly used to vectorize a bitmap image of a screenshot that has a number of features: namely, low resolution of characters and the application of the effect of smoothing (anti-aliasing) of text characters. This anti-aliasing improves the image quality on the display, but distorts the screenshot when printing.

The method disclosed in US patent No. 7,079,686 [3] describes an approach based on the classification of pixels in an image of a document for enlarging an image. A feature vector is generated for each pixel in the image. Each pixel is classified as text or picture depending on the corresponding feature vector. Further processing may include pixel-level enhancement, which consists of a border sharpening filter for text pixels and a smoothing filter for picture pixels.

The method disclosed in US patent No. 7,177,049 [4], is reduced to the processing of digital images, and the process of improving the quality of the text includes the processing of black text on a white background with increasing sharpness and contrast of such text due to the redistribution of brightness between dark and light pixels within predefined masks.

The methods described above are aimed at improving the image, especially areas of the text exclusively before printing. These methods are also not suitable for screenshots, as they are designed for scanned images.

The problem to which the claimed invention is directed is to develop an improved method for processing smoothed text (text with anti-aliasing) in the screenshot. Moreover, this method should provide sharpening / clarity and contrast of characters and be applicable both for printing screenshots and for saving screenshots in a metafile.

The technical result is achieved through the application of the proposed method, which includes operations to detect smoothed characters in the screenshot and improve the visual presentation of these characters through vectorization and color estimation of the fill. In this case, the claimed method of converting a screenshot into a metafile includes the following steps:

- detect fragments of text in the screenshot;

- evaluate the background color and the color of the characters on each fragment;

- vectorize the outline of the characters;

- modify the screenshot by filling in the text fragments with the background color;

- create a metafile based on a modified screenshot and vectorized outlines filled with text color.

The inventive method provides increased sharpness / sharpness and contrast of characters and can be applied both when printing screenshots, and when saving screenshots to a metafile, such as PDF or XPS.

Further, the essence of the claimed invention is explained in detail with reference to the relevant graphic materials.

Figure 1. Illustration of the proposed method in comparison with existing approaches.

Figure 2. A block diagram of a method for converting a screenshot into a metafile.

Figure 3. A block diagram of a system for converting a screenshot into a metafile.

Figure 4. Illustration of a method for detecting fragments with smoothed text.

Figure 5. Illustration of computing signs of smoothed text.

6. Block diagram of vectorized symbol.

7. Illustration of a vectorized symbol.

Fig. 8. Create a segmented text map.

Fig.9. Illustration of the result of the proposed method.

Figure 1 illustrates the difference between the existing approach (prototype), known from the prior art, and the claimed method. The screenshot 103 is captured from the display screen by copying the video memory to an intermediate buffer. The result of such copying is to present the current frame displayed on the display as a bitmap. The display may be a visualization module of any digital computing device, for example: a personal computer 101, a laptop, a smartphone 102, a tablet computer, etc. The peculiarity of the screenshot is that the text information presented on it is usually visualized using antialiasing technology, which is used to make the borders of characters visually smoother, removing the “teeth” that arise when rasterizing at the edges of objects. In this case, the pixels adjacent to the boundary pixel of the image take an intermediate value between the image color and the background color, visually blurring the border. The anti-aliasing technologies used to render the text on the display directly depend on the size of the displayed characters and the resolution of the display, so when zooming in / out they rebuild the pixel display of the characters. In addition, such smoothing differs significantly from similar technologies used in printing text. A screenshot in the form of a bitmap image is saved in any file / format 104; accordingly, the correct application of anti-aliasing technologies when displaying or printing becomes inaccessible for text areas. This leads to undesirable visual distortions in the appearance of the text. In accordance with the foregoing, the following disadvantages of the approach known from the prior art can be noted:

- since the screenshot is saved to a file as a raster image, therefore, such advantages of the usual presentation of textual information on the display as invariance to scale and smooth display of the outline of characters regardless of their size will be lost. Instead, scaling the bitmap of the text will be accompanied by such interfering factors as the appearance of “jagged” edges and blurred borders;

- the negative effect of anti-aliasing of characters appears, which is characteristic for text displayed on the display. The result of this effect is a distortion of the original color and shape of the characters when printing or zooming out.

The inventive method 105 is based on the segmentation of the screenshot into text and non-text areas and vectorization of text areas. The approach allows you to display and print the metafile at a zoomed-in without loss of quality of text information.

Figure 2 illustrates the main steps of the proposed method. At step 201, fragments of text in the screenshot are detected and a segmented text map is created. In a preferred embodiment of the inventive method, these fragments include a sequence of characters that correspond to one or more words and part of the surrounding background, and the map is a bitmap image of the same size as the original screenshot, where each pixel is encoded as related or not related to the area text. The color assessment for the characters and background of the indicated text fragments is performed at step 202. In a preferred embodiment of the inventive method, the screenshot is presented in the RGB color space and, accordingly, each pixel is described by three color components of red, green, and blue. Evaluation of each of the three color components (red, green, blue) occurs separately according to the histograms of these components. The rightmost or leftmost column (group of columns) of the histogram with the highest value (total value) corresponds to the background color opposite to the color of the symbol. At step 203, the outlines of the characters on the text map are vectorized by converting them into a sequence of line segments and curves. The process is illustrated in Fig.6 and Fig.7. At step 204, the screenshot is modified by filling in the areas of the detected text with the corresponding background color. For a complex background, it is preferable to use a more advanced way of filling in the text area, for example, the inpainting technique, i.e. shading ("Bertalmio M., Sapiro G., Caselles V., Ballester C. Image inpainting // In Proc. ACM Conf. Comp. Graphics (SIGGRAPH), pages 417-424, 2000) [5]. At step 205 is created metafile from the corresponding metafile records saved in a certain sequence.The metafile can be presented in the format: PDF, XPS, PS, EMF, etc. The modified image of the screenshot is saved to the metafile as a bitmap. Symbols are saved in vector form by the corresponding graphic metafile commands that define the parameters closed sequences of approximating segments of lines and cr It’s possible to use lossless compression algorithm for vector metafile records and lossy or lossless compression algorithm for bitmap images. The claimed approach provides efficient storage of metafile from the point of view of visual quality / size ”, Since each type of visual information is stored in an optimal way. Moreover, the claimed approach allows you to prevent distortion of the smoothed text in the screenshot when printing or rendering at an enlarged scale. This is made possible by replacing the smoothed screenshot characters with vector counterparts.

Figure 3 shows a block diagram of a system for converting a screenshot into a metafile. The text segmentation module 301 segments the text, creates a marked-up map of the segmented text, and determines the coordinates of the fragments containing the text. The input bitmap image of the screenshot is input to this module. The segmented text map is transmitted to the vectorization module 304 and the text retouching module 303, the coordinates of the fragments are transmitted to the color estimation module 302. The color evaluation module is configured to determine the background color and text for each fragment. The input of this module receives the coordinates of the fragments and the initial screenshot. Information about the background color is transmitted to the retouching unit 303, information about the text color is transmitted to the metafile creation unit 305. Vectorization module 304 approximates the contours of characters on a segmented text map with a closed sequence of line segments and curves. The input from the module from the segmentation module 301 is fed to the input of this module. The coordinates of the fragments, the sequence of line segments and curves are transferred to the metafile creation module. The text retouching unit 303 modifies the original bitmap image of the screenshot by filling in the area of the fragments with the text with the corresponding background color. The input of this module receives information about the background color for each fragment. The modified screenshot is transmitted to the metafile creation module 305, configured to generate a metafile from the following records: a modified screenshot, closed sequences of straight line segments and curves filled with the corresponding symbol color.

All of the listed system modules can be implemented as SoC, FPGA or ASIC. The functions of the modules are clear from their descriptions and descriptions of existing methods. Only those features that are mentioned in the description are illustrated. However, it should be understood that a computing system may have additional features that have not been illustrated.

It makes sense to consider in detail two possible approaches for detecting text. The first algorithm is faster and is designed to detect smoothed text, which is typical for displays. The second algorithm is more general, suitable for detecting both smoothed text and text without smoothing.

The first approach to detect text fragments includes the following steps:

- detect adjacent pixels with a large derivative in the horizontal direction;

- combine adjacent pixels into fragments;

- analyze histograms of fragments in order to detect smoothed text;

- extract fragments with signs of a smoothed text.

403 является конъюнкцией положительного $$D_{i,j}^{pos}$$

и отрицательного $$D_{i,j}^{neg}$$

градиентов, которые указывают на то, что пиксель ярче/темнее соседних:4 illustrates a first approach to detecting fragments of smoothed text in a screenshot for subsequent analysis. The illustration shows the result of detecting a horizontal gradient for one small fragment 401 of the screenshot. The calculated total horizontal gradient value $$D_{i,\mathrm{<figure-callout\; id="403"\; label="j\; \Sigma "\; filenames="00000031.png"\; state="\{\{state\}\}">j\; </figure-callout>}}^{\Sigma }$$

403 is a conjunction of positive $$D_{i,j}^{pos}$$

and negative $$D_{i,j}^{neg}$$

gradients that indicate that the pixel is brighter / darker than neighboring ones:

$$D_{i,j}^{pos}=\{\begin{array}{c}\begin{array}{cc}\mathrm{one,}& I_{i,j-\mathrm{one}}+\Delta <I_{i,j}<I_{i,j+\mathrm{one}}-\Delta \end{array}\\ \begin{array}{cc}0& otherwise\end{array}\end{array}$$

$$D_{i,j}^{neg}=\{\begin{array}{c}\begin{array}{cc}\mathrm{one,}& I_{i,j-\mathrm{one}}-\Delta >I_{i,j}>I_{i,j+\mathrm{one}}+\Delta \end{array}\\ \begin{array}{cc}0& otherwise\end{array}\end{array}$$

$$D_{i,j}^{\Sigma }=D_{i,j}^{pos}\cup D_{i,j}^{neg}$$

Where I _{i, j} is the brightness of the image at the point i, j; Δ is a given threshold.

The result of the gradient detection is a binary image 402, non-zero elements of which indicate the presence of a gradient. Due to the weak connectivity of neighboring elements in the binary image, the expansion (dilatation) procedure is performed with a horizontal structuring element 403. In the next step, individual fragments 404 are selected at the extreme points of the connected nonzero pixels.

Next, a histogram of each selected fragment is analyzed in order to detect signs of smoothed text and highlight fragments containing symbols. Figure 5 shows an illustration of the analysis of the histogram of all three color components (red, green, blue) of the RGB color space to detect signs of smoothed text: the number of nonzero b ₊ histogram columns and zero b ₀ . The histogram of fragment 504 containing non-textual data is characterized by a completely continuous distribution of 501-503 for each color component. The histogram of fragment 508 containing smoothed text consists of only a few b ₊ (for example, b ₊ = 7 was experimentally obtained for MS ClearType technology) non-zero columns that are evenly distributed between the background and text color (SOS-SOT). In a preferred embodiment of the invention, if b ₀ > 6, then the fragment is classified as text, otherwise as non-text 509.

The second approach to detecting fragments with text includes the following steps:

- divide images into rectangular blocks with overlapping;

- calculate the characteristics for each block;

- classify each block;

- combine adjacent blocks into fragments of text. In a preferred embodiment, the block size is 7 × 7 or 9 × 9 pixels. Overlapping blocks can be 1, or 2, or 3 pixels.

For each block, the following characteristics are calculated:

- the number of non-zero columns and zero columns of the histogram separately for each channel;

, где I_i(r, c) - яркость пикселя в ряду r и колонке c, N - число пикселей в блоке;- average block brightness: $${\overline{I}}_i=\frac{{\displaystyle \sum _{r=\mathrm{one}}^N{\displaystyle \sum _{c=\mathrm{one}}^N{I}_i\left(r,c\right)}}}{N^2}$$

where I _i (r, c) is the pixel brightness in the row r and column c, N is the number of pixels in the block;

;- the average difference in the average brightness of the block the average difference in the average brightness of the blocks I _k in a 4-connected neighborhood with the block I _i : $${\overline{dI}}_i=\frac{{\displaystyle \sum _{k=\mathrm{one}}^{\mathrm{four}}|\; |\; |{\overline{I}}_i-{\overline{I}}_k|\; |\; |}}{\mathrm{four}}$$

;

и горизонтальных $$d{I}_x^i$$

производных по блокам: $${\overline{d_{x,y}I}}_i=\frac{{\displaystyle \sum _{r=1}^N{\displaystyle \sum _{c=`}^{N-1}d{I}_x^i\left(r,c\right)+{\displaystyle \sum _{r=1}^{N-1}{\displaystyle \sum _{c=`}^{N}d{I}_y^i\left(r,c\right)+}}}}}{2N\left(N-1\right)}$$

;- average value of vertical $$d{I}_y^i$$

and horizontal $$d{I}_x^i$$

derivatives in blocks: $${\overline{d_{x,y}I}}_i=\frac{{\displaystyle \sum _{r=\mathrm{one}}^N{\displaystyle \sum _{c=`}^{N-\mathrm{one}}d{I}_x^i\left(r,c\right)+{\displaystyle \sum _{r=\mathrm{one}}^{N-\mathrm{one}}{\displaystyle \sum _{c=`}^{N}d{I}_y^i\left(r,c\right)+}}}}}{2N\left(N-\mathrm{one}\right)}$$

;

, где N_d - это нормированная матрица вхождений, d - определяет пространственную связь;- homogeneity of the block: $$H={\displaystyle \sum _{i,j}\frac{N_d\left(i,j\right)}{\mathrm{one}+|\; |\; |i-j|\; |\; |}}$$

where N _d is the normalized matrix of occurrences, d - defines the spatial relationship;

, где ∇I _i(r, c) вычисляется как квадратный корень из суммы квадратов горизонтальных и вертикальных производных;- percentage of pixels with a gradient above the threshold: $$P_g={\displaystyle \sum _{\forall \left(r,c\right)\in Bi}\{\mathrm{one}|\; |\; |\nabla I_i\left(r,c\right)>T\}/N^2}$$

where ∇I _i (r, c) is calculated as the square root of the sum of the squares of the horizontal and vertical derivatives;

, полученном в результате применения операции морфологического открытия к бинарному изображению $$I_i^b$$

, полученному бинаризацией с пороговым значением 128: $$P_m={\displaystyle \sum _{\forall \left(r,c\right)\in Bi}\left\{1\right|I_i^o\left(r,c\right)\ne I_i^b\left(r,c\right)\}/N^2}$$

.- percentage of changes in brightness of pixels in the image $$I_i^o$$

obtained by applying the morphological discovery operation to a binary image $$I_i^b$$

obtained by binarization with a threshold value of 128: $$P_m={\displaystyle \sum _{\forall \left(r,c\right)\in Bi}\{\mathrm{one}|\; |\; |I_i^o\left(r,c\right)\ne I_i^b\left(r,c\right)\}/N^2}$$

.

One of the following methods can be used to classify into two subsets (text and figure) based on the above criteria: boosting the decision tree committee (Yoav Freund and Robert E. Schapire. 1997. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 1 (August 1997), 119-139.), random forest of decision trees (Leo Breiman. 2001. Random Forests. Mach. Learn. 45, 1 (October 2001), 5-32.), A support vector machine (Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks. Mach. Learn. 20, 3 (September 1995), 273-297.), Method of K-nearest neighbors ( D.Coomans, DLMassart, Alternative k-nearest neighbor rules in supervised pattern recognition: Part 1. k-Nearest neigh bour classification by using alternative voting rules, Analytica Chimica Acta, Volume 136, 1982, Pages 15-27).

Figure 6 shows a block diagram of the vectorization process. After detecting screenshot fragments containing text, color fragments are transformed into grayscale images 601 and increased by a factor of 602. In a preferred embodiment of the inventive method, bilinear interpolation is used, since the initial resolution of the symbol in the screenshot is small. For example, at a screen resolution of 1920 × 1080 pixels, the height of the lowercase character of the Times New Roman font of the 12th size at 100% scale is 7-8 pixels. Next, high-resolution halftone fragments are binarized 603. In the preferred embodiment, the Father method is used to determine the segmentation threshold (N. Otsu, "A threshold selection method from gray level histogram", IEEE Transactions on System Man Cybernetics, vol. 9 no. 1, 1979 , pp. 62-66.). All binarized fragments are combined in accordance with their coordinates in the screenshot to form a map of segmented text. In a preferred embodiment, a coefficient k of three is used.

At step 604, the external and internal contours of each text fragment are monitored to obtain the path paths of the given fragment. At this stage, the contour is a closed sequence of points (vertices) connected by linear segments of minimum length equal to one pixel. The tracking procedure goes from the starting vertex along the contour line in a given direction until the starting vertex is reached again. At the next step 605, the number of vertices decreases due to the approximation of the contour path by a polygon. The polygon is converted to a smooth contour at step 606 due to the approximation by a closed sequence of line segments and curves. In a preferred embodiment of the inventive method, cubic Bezier curves are used (Piegl, L. Fundamental Developments of Computer Aided Geometric Design. San Diego, CA: Academic Press, 1993). In the general case, the approximation by linear segments is based on determining the coordinates of the ends of the segments. The approximation by curved segments requires the determination of the ends of the segments and the corresponding reference points. For example, in accordance with the illustration in FIG. 7, the angle 703 between the edges 701-703 and 703-706 of the polynomial can be approximated by a cubic Bezier curve 705 bounded by points 702 and 704. An example of the approximation of the symbol 707 is shown in 708.

An illustration of the conversion of the screenshot text from the initial raster view to a vector one is shown in Fig. 8 in accordance with steps 201, 203 and 204. The smoothed text fragment 801 is enlarged (802) to improve resolution and converted to binary image 803. The resulting vectorized fragment is represented by 804.

Figure 9 shows the result of applying the claimed method. The first fragment 901 corresponds to the original screenshot. The second is to fragment 902 of the printed screenshot. The third fragment 903 was obtained by printing a screenshot with a higher resolution using bilinear interpolation. And for comparison, the fourth fragment of 904 printed screenshots after processing the claimed method is shown. The text looks sharper and sharper.

Further aspects of the invention can be obtained from consideration of the illustrations and description of preferred embodiments. It is clear to those skilled in the art that various embodiments of the invention, additions and substitutions are possible, without departing from the claims and meaning of the present invention disclosed in the attached claims.

The inventive method and system are intended to be implemented in a printer driver or supporting software for black-and-white and color printers or MFPs. In addition, the method can be implemented as a software application for printing.

Claims (9)

1. A method of converting a screenshot into a metafile, including the following operations:
- detect fragments of smoothed text in a screenshot;
- evaluate the background color and the color of the characters of the smoothed text on each fragment based on the definition of the extreme columns of the fragment histogram;
- vectorize the outline of the characters of the smoothed text;
- modify the screenshot by replacing fragments of the smoothed text with the background color;
- create a metafile based on a modified screenshot and vectorized outlines filled with the color of the characters of the smoothed text;
- visualize the created metafile by displaying or by printing. 2. Method according to claim 1, characterized i in that the anti-aliased text fragments on the screen shot is detected by performing the following operations:
- detect neighboring pixels with a high value of the horizontal gradient;
- combine adjacent pixels into fragments;
- analyze the histograms of fragments for the detection of signs of a smoothed text;
- extract fragments with signs of a smoothed text. 3. The method according to claim 2, characterized in that adjacent pixels are combined into fragments by means of morphological expansion (dilatation) and determination of bounding rectangles for each associated group of pixels. 4. The method according to claim 2, characterized in that in the process of analyzing the histogram of fragments, the detection of smoothed text is carried out by counting the number of isolated groups with non-zero columns for each color and checking this number for exceeding a predetermined threshold. 5. The method according to claim 1, characterized in that fragments of the smoothed text in the screenshot are detected by performing the following operations:
- divide the screenshot into intersecting rectangular blocks;
- calculate the characteristics for each block;
- classify each block as text or non-text;
- combine adjacent blocks into fragments of text. 6. The method according to claim 5, characterized in that for each block the following characteristic features are calculated:
- the number of non-zero columns b ₊ and zero columns b ₀ separately for each channel of the RGB color space;
- average brightness of the block:

where I _i (r, c) is the brightness of the pixel located in row r and column c , N is the number of pixels in the block;
- the average difference in the average brightness of the blocks I _k in a 4-connected neighborhood with the block

;
- average value of vertical

and horizontal

derivatives in blocks:

;
- homogeneity of the block:

where N _d is the normalized matrix of occurrences, d - defines the spatial relationship;
- percentage of pixels with a gradient above the threshold:

calculated as the square root of the sum of the squares of the horizontal and vertical derivatives;
- percentage of changes in brightness of pixels in the image

obtained by applying the morphological discovery operation to a binary image

obtained by binarization with a threshold value of 128:

. 7. The method according to claim 5, characterized in that each block is classified as text or non-text according to one of the following methods: boosting of the decision tree committee, random forest of decision trees, support vector machine, K-nearest neighbors method. 8. The method according to claim 1, characterized in that the vector outlines of the characters of the smoothed text by performing the following operations:
- convert a fragment of text into a grayscale image;
- increase the resolution of the grayscale image;
- binarize a fragment of high resolution;
- track the points of the contours of the characters within the fragments to describe the path of the contours;
- reduce the number of vertex contours;
- approximate the path of the contours with a reduced number of vertices by a sequence of line segments and curves; 9. A system for converting a screenshot into a metafile, including:
- text segmentation module, configured to segment text, create a marked-up map of segmented text and determine the coordinates of fragments containing smoothed text; moreover, the input bitmap image of the screenshot is input to the module, the outputs of the text segmentation module are connected to the inputs of the vectorization and retouching modules, where the map is transferred, as well as to the input of the color estimation module, where the coordinates of the fragments and the original screenshot are transmitted;
- color evaluation module, configured to determine the background color and text for each fragment; the coordinates of the fragments and the initial screenshot are received from the output of the text segmentation module; the outputs of the color evaluation module are connected to the input of the retouching module and the input of the metafile creation module;
- a vectorization module, configured to approximate the contours of characters on a segmented text map with a closed sequence of line segments and curves; the input of the module is connected to the output of the segmentation module with the possibility of obtaining a card; the output of the vectorization module is connected to the input of the metafile creation module;
- text retouching module, configured to modify the initial bitmap image of the screen shot by filling in the area of fragments with the text with the corresponding background color; the input of the module is connected to the output of the color evaluation module; the output of the module is connected to the input of the metafile creation module, where the modified screenshot is transmitted;
- a metafile creation module, configured to generate a metafile from the following entries: a modified screen shot, closed sequences of line segments and curves filled with the corresponding symbol color.

Images