RU2647671C2 - Segmentation of multicolumn document - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: group of inventions refers to image processing technologies, in particular, to the analysis of documents with complex spatial pattern. Method for analyzing the logical structure of a document image is proposed. Method comprises the step of detecting objects in the document image. Further, according to the method, the separating graph of the identified objects is constructed on the basis of construction of diagram for the regions on the detected objects in the document image. Also, at least one optimal path in the dividing graph is detected by assigning the weights to the edges of the separating graph to determine the location of the regions in the document image.

EFFECT: technical result is the expansion of the arsenal of technical means aimed at analyzing the logical structure of the document's image.

26 cl, 11 dwg

Description

FIELD OF TECHNOLOGY

[0001] The subject of this application relates to a method and system in the field of image processing, in particular to the analysis of documents with complex spatial markup with a complex spatial arrangement.

BACKGROUND

[0002] At present, recognizing the internal content of photographed or scanned documents is a complex task. The basic principle of existing segmentation methods is to search for the main (logical) structure of the document, such as regions of columns, frames and headers, as well as to analyze the internal content of the identified regions (for example, identifying lines of text in the regions of columns). Known methods of segmentation are often able to identify only rectangular blocks in a document (for example, rectangular columns of text and inset primitive shapes). Such methods require that the gaps between columns and insets be large enough to correctly determine whether a given fragment of a document belongs to a column or inset. If there are isolated objects inside columns or insets (for example, sparse cells of the table) or if the shape of the regions is not rectangular, then the classification is incorrect.

[0003] Existing recognition methods are unable to accurately and reliably segment images of documents. Therefore, there is a need to develop a new method for analyzing the internal content of documents.

SUMMARY OF THE INVENTION

[0004] On the one hand, the present invention relates to a method for detecting a basic (logical) image structure of a document. The method includes identifying objects in an image of a document, plotting a chart for areas on identified objects, building a dividing graph based on a diagram for areas on identified objects, and building an adjacency graph based on a dividing graph. In some embodiments, constructing an adjacency graph includes assigning vertices of the graph corresponding to the identified objects, determining pairs of adjacent objects, corresponding pairs of objects separated by at least one edge in the separating graph, and connecting each pair of adjacent objects with an edge of the adjacency graph. This method further includes identifying the vertices of the graph and edges of the graph in the diagram for regions and constructing a minimal graph based on the identified vertices of the graph and edges of the graph. This method further includes detecting the optimal path in the graph to locate areas in the document image and dividing the document image into regions, at least partially based on the found optimal path.

[0005] In some embodiments, this method further includes assigning weights and / or penalties to the edges of the separating graph. In one embodiment, the weight values are based on a comparison of at least two identified objects. This method further includes summing the assigned weight values to determine the total weight value to find the optimal path. In one implementation example, the optimal path is the path with the best total weight. The method further includes constructing a subgraph for the dividing graph in order to limit the search area of the desired path, identifying the connected components in the subgraph, building at least one path relative to the identified connected components, and filtering the paths. In some embodiments, the implementation of this method further includes correcting the constructed path by adding one or more patches (patches). This method further includes removing at least one edge of the graph from the adjacency graph corresponding to the optimal path, identifying connectivity components in the adjacency graph, building regions corresponding to connectivity components in the adjacency graph, and dividing the document image into the constructed regions.

[0006] On the other hand, the present invention relates to a system for detecting the basic structure of a document. This system includes a memory configured to store instructions executed by the processor, and a processor operably associated with the memory. In some embodiments, this processor further has the ability to identify objects in the image of the document, build a diagram for areas on the identified objects, build a dividing graph based on a diagram for regions, and construct an adjacency graph based on a dividing graph. In some embodiments, this processor further has the ability to assign vertices of the graph corresponding to the identified objects, identify pairs of adjacent objects corresponding to pairs of objects separated by at least one edge in the separating graph, and the ability to connect each pair of adjacent objects by an edge of the adjacency graph. This processor additionally has the ability to identify the vertices of the graph and edges of the graph in the diagram for areas and build a minimal graph based on the identified vertices of the graph and edges of the graph. This method further includes detecting the optimal path in the graph to determine the location of the regions in the image of the document and dividing the image of the document into regions, at least partially based on the found optimal path.

[0007] In some embodiments, this processor further has the ability to assign weights and / or penalties to the edges of the separating graph. In one embodiment, the weight values are based on a comparison of at least one pair of identified objects. This processor additionally has the ability to sum the assigned weight values to determine the total weight value to find the optimal path. In some embodiments, the optimal path is the path with the best total weight. The processor is additionally configured to construct a subgraph for the dividing graph in order to limit the search area of the desired path, identify the connected components in the subgraph, build at least one path relative to the identified connected components and filter the paths. In some implementations, the processor further has the ability to adjust the constructed path by adding one or more patches. This processor additionally has the ability to remove at least one edge of the graph from the adjacency graph corresponding to the optimal path, determine the connectivity components in the adjacency graph, construct regions corresponding to the connectivity components in the adjacency graph, and divide the document image into the constructed regions.

[0008] On the other hand, the present invention relates to a permanent computer-readable storage medium containing computer-readable instructions stored therein, and these instructions may be executed by a processor of a computing system. These commands additionally include commands for identifying objects in the image of the document, building a diagram for areas on identified objects, commands for constructing a separating graph based on a diagram for regions, and commands for constructing an adjacency graph based on a separating graph. In some embodiments, these commands further include commands for assigning vertices of the graph corresponding to the identified objects, commands for identifying pairs of adjacent objects corresponding to pairs of objects separated by at least one edge in the separating graph, and also commands for connecting each pair of adjacent objects with an edge of the graph adjacency. These commands additionally include commands to identify the vertices of the graph and edges of the graph in the diagram for areas and teams to build a minimal graph based on the identified vertices of the graph and edges of the graph. This method further includes searching for the optimal path in the graph to determine the location of the regions in the image of the document and dividing the image of the document into identified areas at least partially based on the constructed optimal path.

[0009] In some embodiments, these instructions further include instructions for assigning weights and / or penalties to the edges of the separating graph. In one embodiment, the implementation of these penalties is based on a comparison of the mutual characteristics of at least one pair of identified objects. These commands additionally include commands for summing the assigned weight values to determine the total weight value in order to find the optimal path. In some embodiments, the optimal path is the path with the best total weight. These commands additionally include commands for constructing a subgraph for the dividing graph in order to limit the search area of the desired path, commands for identifying connected components in the subgraph, commands for constructing at least one path relative to the identified connected components, and commands for filtering these paths. In some embodiments, the implementation of these commands further include commands to correct the constructed path by adding one or more patches. These commands further include commands for removing at least one edge of the graph from the adjacency graph corresponding to at least one optimal path, commands for identifying connectivity components in the adjacency graph, commands for constructing regions corresponding to connectivity components in the adjacency graph, and commands for splitting images of the document on the constructed regions.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The foregoing and other objects, aspects, features and advantages of this disclosure of the invention will become more apparent and more understandable when referring to the following description, taken together with the accompanying drawings, in which:

[0011] FIG. 1 is a flowchart of a method for detecting a logical structure of a document in accordance with an illustrative embodiment of the invention;

[0012] FIG. 2 is a flowchart of a method of plotting for regions in accordance with an illustrative embodiment of the invention;

[0013] FIG. 3 is a flowchart of a method for constructing a dividing graph in accordance with an illustrative embodiment of the invention;

[0014] FIG. 4 is a flowchart of a method for detecting an optimal path in accordance with an illustrative embodiment of the invention;

[0015] FIG. 5 is a flowchart of a method for dividing a document into regions in accordance with an illustrative embodiment of the invention;

[0016] FIG. 6 illustrates an example image of a document with distorted columns in accordance with an illustrative embodiment of the invention (This figure was illustrative. The text provided therein did not carry a semantic load, it is not possible to translate its text and the text part has been replaced with arbitrary illustrative Russian text. Neither illustrative English the text (photo), nor the replacement of the Russian text are not part of the application and are given as an example of illustration);

[0017] FIG. 7 illustrates an example of an image of a paper with inserts in accordance with an illustrative embodiment of the invention (This figure was illustrative. The text provided therein did not carry a semantic load, it is not possible to translate its text and the text part has been replaced with arbitrary illustrative Russian text. Neither illustrative English text (photo), nor a replacement for the Russian text are not part of the application and are given as an example of illustration);

[0018] FIG. 8 illustrates examples of objects in an image of a document in accordance with an illustrative embodiment of the invention;

[0019] FIG. 9 illustrates a first example of a graph of objects in an image of a document in accordance with an illustrative embodiment of the invention;

[0020] FIG. 10 illustrates a second example of a graph of objects in an image of a document in accordance with an illustrative embodiment of the invention;

[0021] FIG. 11 is a block diagram of a system for detecting a logical structure of a document in accordance with an illustrative embodiment of the invention.

DETAILED DESCRIPTION

[0022] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that this invention can be practiced without these specific details. In other cases, structures and devices are shown only in block diagrams so as not to obscure the invention.

[0023] A reference in this description to “one embodiment” or “embodiment” means that a particular feature, structure, or characteristic described in connection with an embodiment of the invention is included in at least one embodiment of the present invention. The repeated occurrence of the expression “in one embodiment” at various places in the description does not necessarily refer to the same embodiment, and individual or alternative embodiments of the invention are not mutually exclusive with respect to other embodiments. In addition, a description is given of various features that may be used in some embodiments of the invention and not used in other embodiments. Similarly, descriptions are given of the various requirements that may be presented to some embodiments of this invention, but not to other embodiments.

[0024] Segmentation is an important step in image recognition of a document. The segmentation process allows you to identify different regions on the images of documents. Regions can be columns, figures, tables, text fragments, headings, etc.

[0025] The present invention relates to methods and systems for detecting (or detecting) the basic structure of a document. In some embodiments, the implementation of the document may contain columns of text, images, tables, etc. The present invention also relates to the detection (detection) of the boundaries of objects of any complexity located in a document, for example, the boundaries of objects in a document can partially or completely cut into columns. In some embodiments, boundary detection includes detecting gaps between columns, for example, in a document image. Gaps in document images may be curved, narrow or distorted. In some embodiments, the objects in the document may be arbitrary in shape, which increases the complexity of defining boundaries. An example of a document with curved columns 62 is shown in FIG. 6. The document may contain various regions, such as fragments of text, pictures, columns, tables, diagrams, etc. The methods and systems described herein may divide a document into such regions in order to correctly understand their logical relationship.

[0026] Two types of regions may be present in a document: column regions and non-column regions. Regions of columns can be considered as sections of a document containing text that resembles a column in shape. A document may contain one or more columns of text. Non-column regions in some embodiments may be portions of a document containing pictures, text frames, headings or tables, hereinafter referred to as “boxes”.

[0027] The box may be a region of a document that contains text, a frame with text, a title, a table, and / or a picture. In some embodiments, the main feature of the frame is that it partially or fully cuts into the main text column. In the future, a frame that partially cuts into a column may be called a “partial” frame. In addition, a frame that fully cuts into a column can be called a “full” frame. Note that the main text of the column can be located either to the left or to the right of the partial inset. Thus, two types of partial taps are possible: the right partial tapping and the left partial tapping.

[0028] In FIG. 7 shows an example image of a document containing frames. The image of the document in FIG. 7 includes a box 71, a first column 72, a middle column 73, a third column 74, full bars 75 and a border of a middle column 76. A complex box 71 in FIG. 7 is a sidebar for each column. For the first column, inset 71 is the right partial (rectangular) inset. For the middle column, frame 71 is a full frame. For the third column 74, inset 71 is a partial left (not rectangular) inset.

[0029] In some embodiments, the insets can be located at the top, bottom, or middle of the text column. For the top frame, the body text may be located below the frame. Thus, this sidebar is adjacent to the top of the context. For the bottom inset, the main text can be positioned above the inset, the inset can be adjacent to the bottom of the context. In the case of an average inset, the text can be located above, below and / or on one side (right or left), depending on whether this inset is a right inset or a left inset. For example, the inset “IN THE WORLD OF LIVING SOUNDS” (75) in FIG. 7 is the top (full) sidebar for all columns. In some embodiments, the middle full cuts may divide the column into two unrelated parts. In one embodiment, the title may be considered as a special case of the top full frame.

[0030] In FIG. 1 is a flowchart of a method for determining a logical structure of a document. This method can be implemented in a computing device (for example, in a user device). In one embodiment, this method is encoded on a computer-readable medium containing instructions that, when executed on a computing device, cause the computing device to perform operations of this method. In block 100, an image of the document is received. In some embodiments, this image of a document (or frame) can be obtained electronically using any of the known methods. In some embodiment, this image can be obtained from the memory of an electronic device or from any other available sources. Image of the document can be obtained by the processor. This processor can be configured to receive an image of a document and to execute the methods described herein.

[0031] In block 105, objects in this image of the document are detected. In some embodiments, in order to find boundaries between regions, objects may initially be defined in document 105. Objects in a document can be, for example, fragments of lines of text, lines, fragments of drawings, pictures, words, delimiters, etc. In FIG. 8 shows examples of text objects 82 in a document image. In FIG. 8 shows fragments of lines of text that can be objects 82. The boundaries of objects 84 are indicated by a black line.

[0032] In the present disclosure, a document image is segmented into regions using a Voronoi diagram for regions (hereinafter referred to as "" AVD "from the English area Voronoi diagram), and the identified objects are used as Voronoi regions. Returning to FIG. 1, in step 110, an AVD is built. The Voronoi diagram for regions can be constructed in any convenient way. For segmentation purposes, an approximate Voronoi diagram for regions can be constructed using, for example, the method shown in FIG. 2.

[0033] In FIG. 2 shows a flowchart of a method for constructing an approximate Voronoi diagram for regions. In block 200, an image with selected objects is obtained. In block 210, each object is first approximated using a set of points 210. In some embodiments, the set of points may be referred to as “approximating points”. In FIG. Figure 8 shows examples of objects 82 and one way of approximating them using a set of points 86. After approximating points have been determined for all objects, a block Voronoi 220 is constructed for them in block 220. In the constructed Voronoi diagram, there are many points, each of which is located to to a given approximating point closer than to all others, can be called a Voronoi cell for an approximating point. In block 230, boundaries between Voronoi cells belonging to the same objects (230) are excluded. In block 240, the resulting diagram is an approximate Voronoi diagram for regions. It should be noted that AVD contains useful additional information related to the topology of objects. For example, it can be used to clarify the boundaries between objects. In some embodiments, a special data structure called a “dividing graph” is used for this.

[0034] FIG. 3 is a flowchart of an embodiment for constructing a dividing graph (120 in FIG. 1). In block 300, a Voronoi diagram for regions is constructed. In some embodiments, the edges of the separating graph indicate continuous sections of the boundaries between the Voronoi cells, for example, the Voronoi cells in the Voronoi diagram for regions. The edges in the dividing graph can begin and end at the branch points of the corresponding boundary lines. The branch point may be the point at which three or more Voronoi cells meet. In some embodiments, the edge is provided with additional information, for example, the edge may indicate a pair of objects separated by a corresponding boundary line.

[0035] In FIG. 9 schematically illustrates a first example of a dividing graph. In FIG. 9 shows an edge Eab 91 that separates adjacent cells A 92 and B 93. The vertices in the separating graph may be branch points 102 and end points 104 on the AVD, as schematically shown in FIG. 10, which illustrates a second example of a dividing graph. In some embodiments, the implementation of the vertices corresponding to the end points may be called "end vertices." End vertices may be vertices located along the boundary of the graph. Only one edge can exit the end vertex. In some embodiments, if the object has a more or less random shape, then exactly 3 edges will come from each non-terminal vertex. This follows from the properties of the Voronoi diagram.

[0036] Returning to FIG. 3, in blocks 310 and 320, branch points and end points on the AVD can be used to construct the dividing graph. In some embodiments, each of these points corresponds to a vertex of graph 310, and to each segment separating Voronoi cells there corresponds an edge of graph 320. In block 330, the resulting graph can be reduced to a minimal homeomorphic graph, that is, nodes of degree 2 (vertices with only two outgoing ribs). In some embodiments, in order to bring the resulting graph to a minimal graph, the vertices from which only two edges come out can be removed, and two corresponding edges can be combined into one edge 330. In one embodiment, the edges are combined only when they share the same pair of objects. The resulting graph is a separating graph 340.

[0037] Now, referring to FIG. 1, where an adjacency graph is constructed in block 120. In some embodiments, an object adjacency graph may be a dual representation of AVD. Unlike the separating graph, the vertices of the adjacency graph can have a well-defined physical meaning and correspond to objects in the image of the document. The edges of the adjacency graph connect adjacent objects, while the edges of the separating graph separate them. In some embodiments, a partitioning graph may be used to construct an adjacency graph. In one embodiment, the procedure for constructing an adjacency graph using a dividing graph may include identifying vertices of the adjacency graph. In some embodiments, the vertices of the adjacency graph can correspond to objects found in the image of the document, and two vertices of the adjacency graph can be connected by an edge if the objects corresponding to these vertices are separated by at least one edge in the separating graph (additional information of the edge of the separating graph indicates objects which this edge shares). In some embodiments, it is not possible to uniquely recover a separating graph from an adjacency graph of objects. This is due to the fact that it is not always possible to find out from an adjacency graph whether two edges in a separating graph should be adjacent (that is, have a common vertex) or not.

[0038] Returning again to FIG. 1, block 140 searches for paths. The search for paths can be performed after constructing the separating graph and the adjacency graph, and the search for paths is carried out in the separating graph.

[0039] FIG. 4 is a flowchart of a method for finding an optimal path. In some embodiments, regions in a document may be represented by text fragments, pictures, columns, tables, diagrams, etc. Most often, the borders between regions in documents are fairly large areas of empty space, for example, the separation between columns or frame borders, which are usually continuous. From the point of view of the dividing graph, such a space can be a rather long continuous path (i.e., a sequence of adjacent edges) in the dividing graph.

[0040] In one embodiment, the path in the dividing graph may be a sequence of edges in which the end of one edge (vertex) is the beginning of another edge. In some embodiments, the implementation of the path in the dividing graph may terminate either at the end vertex, or at the top of some other path. In FIG. 7 illustrates an example image of a document, it being easily seen that a set of paths 76 will “cut” the document, thereby dividing its context into the required regions.

[0041] In one embodiment, the optimal path is determined to correctly divide a document into regions. For this, we return to FIG. 4, in block 400, the edges of the separating graph are weighted. In some embodiments, the edges of the dividing graph are weighted based on an analysis of the compatibility of the objects between which the edges of the dividing graph pass. Such an analysis may also include comparing the characteristics of the selected objects in the image of the document. For example, such an analysis may include, without limitation, comparing the text quality of objects, the positions of baselines, heights, etc. In some embodiments, based on the results, the edge separating the objects in question may be assigned a weight or a penalty. In one embodiment, the penalty (weight) may have some numerical value. In some embodiments, the implementation of the values of the weights may be based on the analysis of mutual and / or similar characteristics of at least one pair of identified objects. For example, if two objects are very similar to pieces of a string, then the edge separating the two objects can be given a large fine. In other embodiments, if the types of the two objects are incompatible (for example, a text object and a picture), then the edges between the two objects can be assigned a smaller penalty.

[0042] In some embodiments, the size of the fine may depend on the distance between the objects. In general, fines can vary for different subtasks. For example, in one embodiment, when searching for specific types of frames, it can be expected that there is a column of text on at least one side of the edge, so that edges that share non-text objects can be assigned a higher penalty. In another embodiment, a large fine would be impractical when searching for paths between columns (intercolumn paths), because it is possible that an edge separates two pictures, each of which belongs to its own column.

[0043] In some embodiments, searching for paths in a dividing graph assumes that the path must go along the border between adjacent objects in the document. Since the distance between the boundaries of the regions is greater than, for example, the distance between the lines of text in one column, after assigning the weights to the edges of the separating graph, it turns out that the edges separating neighboring regions have a relatively small weight (fine) compared to edges separating objects inside one region. Thus, in one embodiment, in order to find the path that runs along the border between regions, it is desirable to find the path with the least penalty. In some embodiments, the path with the least penalty may be the best path. In one embodiment, the optimal path may be the path with the best total weight. The best total weight value may have the lowest total penalty or the highest total score.

[0044] In one embodiment, the process of constructing an optimal path may begin by searching for “good” edges or edges with end vertices. A “good” rib can mean the rib with the least penalty. In one embodiment, an edge with a penalty that is less than a predetermined threshold can be considered “good”. Initially, the path has no fines, because it does not contain any edges. When adding each edge, the path penalty increases by the amount of the penalty of the added edge. In some embodiments, for example, Dijkstra's algorithm can be used to find the path with the least penalty. Thus, in one embodiment, the path obtained by adding edges with the least penalties may be the boundary between two or more regions of the image of the document. In some embodiments, the implementation of the path obtained by adding edges with the least penalty, can pass between the end vertices and / or parts of other paths in the image of the document. In the general case, any rating system can be selected in order to summarize, add, summarize and / or quantify the value for the rib. For example, in one embodiment, instead of fines, the quality of the rib and / or path can be evaluated. Thus, it is possible to build a path by adding ribs with the highest value of weight (evaluation) of quality. The method described below uses the penalty system as an example.

[0045] In some embodiments, a subgraph in a dividing graph can be constructed. A subgraph can be a graph that contains a subset of the vertices of the separating graph and a subset of the corresponding edges in the separating graph. In some embodiments, a subgraph may be defined in the image area of the document along which the desired path is supposed to go. In such embodiments, a path will only be searched within the subgraph. One of the advantages of using subgraphs is that the process of locating regions in a document can be accelerated.

[0046] The proposed method allows you to segment documents of arbitrary complexity having the most diverse logical structure. Building paths based on a multi-column page with insets, similar to the document shown in FIG. 7 may be difficult. This kind of structure is typical of newspapers and magazines; it is one of the most difficult segmentation tasks. An example of constructing paths between columns and paths for selecting frames is described below.

[0047] In some embodiments, the implementation of the path may include several steps. First, one or more intercolumn graphs (i.e., subgraphs) can be constructed. In some embodiments, intercolumn graphs are constructed using a separating graph. To construct an intercolumn subgraph, various hypotheses regarding how to split a page into columns can be tested. It should be borne in mind that the columns can have the same or different widths. In some embodiments, an edge column may be searched for edges with small fines, for example, fines whose values are less than a certain threshold value. In one embodiment, the fins with low penalties may be most suitable for use as the boundary between two columns. In some embodiments, the fins found with slight penalties may be parts of a future optimal path.

[0048] In one embodiment, to optimize the search for the separating path, an undirected graph can be converted to a directed graph. Initially, a subgraph may be non-oriented, for example, the edges of a subgraph may have no direction. To obtain a directed graph from an undirected graph, each non-oriented edge can be replaced by two oriented edges, in which two oriented edges have opposite directions. In some embodiments, the search is performed, for example, for paths that are directed only from top to bottom. In one embodiment, ribs that are directed in the opposite direction can be omitted. If for this hypothesis of columnization no reliable sections of the intercolumnar paths were found (i.e., sections of paths having edges with small fines), then another way of breaking the document into columns can be considered.

[0049] In some embodiments, in addition to intercolumn paths, it is possible to search for full and partial cuts. In one embodiment, partial taps may be located where there are gaps in the intercolumnar paths. The vertices at which the intercolumn path is interrupted can be considered as the initial vertices of the partial inset paths.

[0050] In those places where partial taps are not found, there is a chance to find the path of full taps. In some embodiments, in order to find the full insertion path, it may be desirable to check the ribs located between the rupture boundaries of the intercolumn paths. For example, you can find edges whose weights exceed the average weight (or whose fines are less than the average fine) of the edges in the analyzed column (since the distance between the inset and the main text of the column is greater than the distance between the lines of text inside the column).

[0051] In some embodiments, for constructing paths in a dividing graph, the method may include examining the starting vertices and ending vertices (ie, where the path ends) of already constructed paths and attempting to start a new path from one of these vertices. In other embodiments, “good” ribs can be identified and used as a plot to build a new path from them by successively adding edges adjacent to them.

[0052] We now turn to FIG. 4, where an inter-column graph is constructed in block 440. In some embodiments, implementation, as already noted, the pre-built intercolumn graph 440 can be used in the construction of intercolumn paths. An intercolumn graph may contain only those edges that can be elements of the path between columns. In some embodiments, “horizontal edges” (that is, edges that separate horizontally overlapping objects) may be excluded from the graph. In addition, in some embodiments, the implementation of the ribs that are not located on the boundaries of the analyzed columns can be excluded.

[0053] At block 450, a search for all connected components can be performed in the intercolumn graph. A connected component can relate to a set of vertices of a graph such that for any two vertices from this set there is a path from one vertex to another, and there is no path from a vertex from this set to a vertex that is not included in this set. Then, at block 460, the intercolumn paths can be matched to the found connectivity components. In some embodiments, intercolumn paths can be mapped to connectivity components using the Dijkstra algorithm.

[0054] At block 470, all or part of the paths may be filtered. In some embodiments, the paths may be filtered based on absolute and / or relative characteristics. In one embodiment, when using absolute characteristics, short, curved paths and / or paths with large fines can be considered suspicious. Short curved paths and paths with large fines can be filtered. In some embodiments, relative filtering takes into account the fact that a given interval may contain several paths that are good but incompatible with each other. For example, in one embodiment, if the table is under the intercolumn path, then it may contain a good long vertical path that is horizontally incompatible with the intercolumn path. In some embodiments, determining which of the two paths is false may be possible, for example, based on a comparative analysis that takes into account which path is closer horizontally to the center of the intercolumn division.

[0055] Still referring to FIG. 4, in block 410, a pre-built column subgraph, including only those edges that can separate the frame from the text column, can be used to construct partial frame paths. In some embodiments, when creating such a subgraph, it can be taken into account that at least one object adjacent to the edge of the search path is most likely to be a text object (text column).

[0056] The process for constructing partial cut-in paths 420 seeks to find a path between the torn parts of a found annular path. Process 420 may sequentially add edges located between the vertices of the interrupted intercolumn path, thereby constructing a path connecting the vertices. In one embodiment, if a top or bottom frame is searched, the process seeks to find a path from the start or end vertex of the corresponding intercolumn path to one of the end vertices in the column subgraph.

[0057] First, the partial insertion path is searched using the Dijkstra algorithm. In some embodiments, the path proposed by the Dijkstra algorithm may be considered the primary path. After the main path 420 has been found, in step 430, the main path can be corrected. In some embodiments, correcting the main path may include expanding the main path 430. In some embodiments, it may be necessary to adjust the main path because the inset may be sparse, for example, it may consist of objects that are far apart, or it may be complex sidebar (for example, a picture and a signature to it). If there is a sparse inset, the main path may be detected incorrectly, that is, this path may erroneously pass through the inset.

[0058] To combat this phenomenon, it is proposed to use the patch method. In some embodiments, patches can be used to correctly adjust the plotted path, for example, in the case of sparse frames. The patch can be a rib or part of the path with a small penalty, which is a candidate for inclusion in the path. In some embodiments, an example indicating that an edge or several edges should be added as a patch may be the fact that the edges or edges separate objects that differ significantly (for example, a portion of a picture and a text object or text objects having different height).

[0059] After building the patches, you can try to add the patches to the main frame path. The main path with the addition of one or more patches may be called the “final path”. In some embodiments, a system may add multiple patches to find the final path. In one embodiment, a set of hypotheses of patch paths can be created, and the best ones can be selected. This choice can be based on two criteria. Firstly, the choice can be based on the concept that it is better to capture as large a territory as possible for a tie-in. Secondly, the choice can be based on patch fines, since the final path should be “good”, i.e. must contain ribs with small fines.

[0060] In general, the method for creating full cut-in paths may be similar to the method for constructing partial cut-in paths. In some embodiments, the search for tie-in candidates may be based on enumerating the end or start vertices of the tie-in. In one embodiment, the final candidate may be heuristically selected based on the following criteria: a) the distance from the break point of the intercolumn path (the smaller the distance, the more preferred the candidate path), b) the quality of the path itself (for example, the less likely that the candidate for the path “cuts” the paragraph of the column text, the better).

[0061] In some embodiments, the detected text columns can be checked for frames that do not protrude from the borders of the columns. For this, in one embodiment, the contents of the column can be analyzed for sections of the dividing paths with small fines. If the text column contains a “good” portion of the path, then in some embodiments, this column may contain a partial inset. In one embodiment, an attempt to construct a partial insertion path may be based on this portion.

[0062] FIG. 5 is a flowchart of a method for dividing a document into regions. The constructed system of paths is insufficient to understand the logical structure of the document and to obtain the result of the segmentation of the document, so it is necessary to perform additional steps to identify the regions in the document. In some embodiments, when a path system is found in the dividing graph, objects can be assigned to the regions to which they correspond. In one embodiment, an adjacency graph 120 may be used for this purpose. Those edges that correspond to the constructed path system are removed from the adjacency graph. Each edge of the separating graph stores information about the objects that it shares (that is, about objects located on either side of this edge). For each edge of paths 480, information about objects shared by this edge can be extracted, and the corresponding edge of the adjacency graph that connects these objects can be deleted (510). Then, in block 520, any known method can be used to identify the connected components in the resulting adjacency graph with the corresponding corresponding edges removed (520). Connectivity components correspond to the desired regions and contain exactly those objects that belong to these regions. In block 530, the resulting regions are identified on the image of the document (530) and displayed in the segmentation results. As a result, the image of the document can be divided into regions (160).

[0063] The foregoing description of illustrative embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or limiting with respect to the exact disclosed form of the invention, changes and variations are possible in light of the above teachings, or they can be obtained from the practical implementation of the disclosed options. The scope of the invention is intended to be determined by the appended claims and their equivalents.

[0064] FIG. 11 shows a system 1100 that is capable of determining the logical structure of a document using the methods described above, in accordance with some embodiments of the invention. System 1100 typically includes at least one processor 1102 connected to memory 1104. Processor 1102 may be one or more processors (eg, microprocessors), and memory 1104 may be random access memory (RAM) containing a main storage device 1100 and / or any additional memory levels, for example, cache memory, non-volatile or backup storage devices (e.g., programmable or flash memory), read-only memory devices, etc. In addition, memory 1104 may include a storage device physically located in some other part of system 1100 (e.g., any cache in processor 1102), as well as any storage device used as virtual memory (e.g., recording in a storage device 1110).

[0065] In some embodiments, the system 1100 has multiple inputs and outputs for external communications. System 1100 may include one or more user input devices 1006 (e.g., a keyboard, mouse, scanner, etc.) as well as a display 1108 (e.g., a panel with a liquid crystal display (LCD)) for interacting with a user or operator. For additional storage, the equipment of the system 1100 may also include one or more removable storage devices 1110, for example, a disk drive or other removable disk, a hard disk drive, direct access storage device (DASD), an optical disk drive (e.g. a compact disk (CD) drive, a universal digital drive for DVD, etc.) and / or a tape drive, etc. In addition, system 1100 may include an interface to one or more networks 1112 (e.g. , with lok network (LAN), wide area network (WAN), wireless network, and / or Internet, etc.) to ensure the exchange of information with other computers connected to these networks. It will be appreciated that system 1100 typically includes corresponding analog and / or digital interfaces between processor 1102 and each of components 1104, 1106, 1108, and 1112, as is well known to those skilled in the art.

[0066] The system 1100 is running an operating system 1114, it executes various software applications, components, programs, objects, modules, etc., which are collectively tagged with reference number 1116 to carry out the correction methods described above.

[0067] In general, the procedures performed for embodiments of the invention can be used as part of an operating system or as a specific application, program component, object, module, or sequence of instructions called “computer programs”. Typically, a computer program contains one or more instructions recorded at different times in various computer storage devices and storage systems, which, after being read and executed by one or more computer processors, cause the computer to perform the operations necessary to execute elements, including various aspects of the invention. Moreover, although this invention has been described in the context of fully functional computers and computer systems, those skilled in the art will understand that various embodiments of the invention can be distributed as a software product in various forms, and that this invention is equally applicable regardless a particular type of device or computer-readable medium used for actual distribution. Examples of computer-readable media include, but are not limited to, the following types of writable media: non-volatile and volatile memory devices, floppy disks and other removable disks, hard disks, optical disks (for example, read-only media on a compact disc (CD ROM), universal digital DVD format disc, etc.), and data transfer media such as digital and analogue communication channels. Computer-readable media included in the present disclosure include only persistent media (i.e., they do not include transient signals in space).

[0068] Although the present description of the invention is provided with reference to specific exemplary embodiments, it should be understood that various modifications of these options are possible without changing the original nature of the invention. Accordingly, the specifications and drawings should be considered as illustrations and not as limitations.

Claims (82)

1. The method of analysis of the logical structure of the image of the document, including: identification of objects in the image of the document; building a separating graph of identified objects based on the construction of a diagram for areas on identified objects in the image of the document; detection of at least one optimal path in the dividing graph by assigning weight values to the edges of the dividing graph to determine the location of regions in the image of the document; division of the image of the document into regions, at least partially based on the detected optimal path. 2. The method according to claim 1, further comprising constructing an adjacency graph based on a dividing graph, wherein constructing an adjacency graph includes: assignment of graph vertices corresponding to identified objects; identifying pairs of adjacent objects corresponding to pairs of objects separated by at least one edge in the separating graph, and connecting each pair of adjacent objects by an edge of the adjacency graph. 3. The method according to p. 1, characterized in that the construction of the separating graph further comprises: identification of graph vertices and graph edges in the diagram for areas, as well as creating a minimal graph relative to the identified vertices of the graph and edges of the graph. 4. The method according to claim 1, characterized in that the values of the weights assigned to the edges of the separating graph are based on an analysis of the mutual characteristics of at least one pair of identified objects. 5. The method according to claim 4, further comprising summing the assigned weight values to determine the total weight value for the paths in the dividing column, where at least one optimal path is a path with the best total weight value. 6. The method according to p. 1, characterized in that the detection of at least one optimal path includes: constructing at least one subgraph for a dividing graph; identification of connectivity components in a subgraph; building at least one path relative to the identified connected components and filtering at least one path. 7. The method according to p. 1, characterized in that the detection of the optimal path further comprises correcting at least one optimal path by adding one or more edges or one or more parts of the path that are candidates for inclusion in the path, with low weight and / or fine. 8. The method according to p. 2, characterized in that the separation of the image of the document into regions further comprises: removing at least one edge of the graph from the adjacency graph, corresponding to at least one optimal path; identification of connected components in the adjacency graph; construction of regions corresponding to the connectedness components in the adjacency graph; and division of the image of the document into the built regions. 9. The system of analysis of the logical structure of the image of the document, including: memory configured to store instructions executed by the processor; a processor functionally connected to a storage device, characterized in that the processor has the ability to: identify objects in the image of the document; build a separating graph of identified objects on the image of the document; detect at least one optimal path in the dividing column to determine the location of regions in the image of the document and divide the document image into regions at least partially based on the detected optimal path. 10. The system according to p. 9, characterized in that the processor additionally has the ability to build a diagram for areas on the identified objects. 11. The system according to p. 9, characterized in that the processor further has the ability to: build an adjacency graph based on a dividing graph, characterized in that for constructing an adjacency graph, the processor additionally has the ability to: assign the vertices of the graph corresponding to the identified objects; identify pairs of adjacent objects corresponding to pairs of objects separated by at least one edge in the separating graph, and connect each pair of adjacent objects by an edge of the adjacency graph. 12. The system according to p. 10, characterized in that the processor additionally has the ability to identify the vertices of the graph and edges of the graph in the diagram for areas. 13. The system of claim 9, wherein the processor further has the ability to assign weight values to the edges of the separating graph, in which the weight values are based on an analysis of the mutual characteristics of at least one pair of identified objects. 14. The system of claim 13, wherein the processor further has the ability to sum the assigned weight values to determine the total weight value for the paths in the dividing graph, in which at least one optimal path is the path with the best total weight value. 15. The system according to p. 9, characterized in that the processor additionally has the ability to: construct at least one subgraph for the separating graph; identify connectivity components in a subgraph; build at least one path relative to the identified connected components and filter out at least one path. 16. The system of claim 9, wherein the processor further has the ability to correct at least one optimal path by adding one or more edges or one or more parts of the path that are candidates for inclusion in the path, with a small weight and / or a fine. 17. The system according to paragraph 11, characterized in that the processor further has the ability to: remove at least one edge of the graph from the adjacency graph corresponding to at least one optimal path; identify connectivity components in the adjacency graph; build regions corresponding to the connectedness components in the adjacency graph; and divide the image of the document into the built regions. 18. A permanent computer-readable storage medium in which computer-readable instructions executed by a processor of a computing system are recorded, characterized in that these instructions include: commands for identifying objects in the image of the document; commands for constructing a dividing graph of identified objects on the image of the document; commands for identifying the optimal path in the dividing graph to determine the location of regions in the image of the document, commands for dividing the document image into regions at least partially based on the detected optimal path. 19. A permanent computer-readable storage medium according to claim 18, further comprising: commands for plotting areas for identified objects. 20. A permanent computer-readable storage medium according to claim 18, further comprising: instructions for constructing an adjacency graph based on a dividing graph, characterized in that the construction of an adjacency graph further includes: commands for assigning the vertices of the graph corresponding to the identified objects; commands for identifying pairs of adjacent objects corresponding to pairs of objects separated by at least one edge in the separating graph, and commands for connecting each pair of adjacent objects with an edge of the adjacency graph. 21. A permanent computer-readable storage medium according to claim 19, further comprising: commands for identifying graph vertices and graph edges in the diagram for areas, as well as commands for constructing a minimal graph based on the identified vertices of the graph and edges of the graph. 22. The permanent computer-readable storage medium according to claim 18, further comprising instructions for assigning weight values to the edges of the separating graph, in which the weight values are based on an analysis of the mutual characteristics of at least one pair of identified objects. 23. The permanent computer-readable storage medium according to claim 22, further comprising instructions for summing the weight values assigned to determine the total weight value for the paths in the dividing column, where at least one optimal path is the path with the best total weight value. 24. A permanent computer-readable storage medium according to claim 19, further comprising: commands for constructing at least one subgraph for the separating graph; commands for identifying connectivity components in a subgraph; commands for constructing at least one path relative to the identified related components, as well as commands for filtering at least one path. 25. A permanent computer readable storage medium according to claim 18, further comprising: commands for correcting at least one optimal path by adding one or more edges or one or more parts of the path that are candidates for inclusion in the path, with little weight and / or a fine. 26. A permanent computer readable storage medium according to claim 20, further comprising: instructions for removing at least one edge from an adjacency graph corresponding to at least one optimal path; commands for identifying connected components in an adjacency graph; commands for constructing regions corresponding to the connectivity components in the adjacency graph; and commands for dividing a document image into built regions.

Images