CN115995091B

CN115995091B - Method and device for reading flow chart, electronic equipment and storage medium

Info

Publication number: CN115995091B
Application number: CN202310084775.0A
Authority: CN
Inventors: 庄家宾; 张颖; 李彦夫
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2023-02-09
Filing date: 2023-02-09
Publication date: 2023-08-25
Anticipated expiration: 2043-02-09
Also published as: CN115995091A

Abstract

The disclosure relates to the field of standard file digitization, and in particular relates to a method and a device for reading a flow chart, electronic equipment and a storage medium. The flow chart reading method comprises the following steps: performing binarization processing on the flow chart image to obtain a first processed image; performing a negation operation on the first processed image to obtain a second processed image, wherein the negation operation comprises setting black pixels in the first processed image to be white and setting white pixels to be black; and determining the corresponding relation between the graphics and the characters in the flow chart image according to the connected domain in the second processing image, wherein the connected domain is a region with connected pixels being white. The process converts the flow chart image into the simple binarization image of the second processing image to acquire the flow chart information, so that the processing speed of the flow chart image is improved, and meanwhile, the process has higher accuracy rate due to the fact that the second processing image contains all connected domains formed by characters and graphics, and further the robustness of the flow chart reading process is improved.

Description

Method and device for reading flow chart, electronic equipment and storage medium

Technical Field

The disclosure relates to the field of standard file digitization, and in particular relates to a method and a device for reading a flow chart, electronic equipment and a storage medium.

Background

With the continuous advancement of industrial digitization, digital technology is applied on a large scale, and industrial efficiency is improved unprecedentedly. The requirements of industry development on the aspects of standard formulation efficiency, use modes and the like are met under the background of digital transformation, and the method becomes a problem of general concern in the standardized field at home and abroad. The flow chart is an important part in the standard file, and the realization of the digital reading of the standard flow chart has important significance on the standard making efficiency and the using mode.

Disclosure of Invention

In view of this, the present disclosure proposes a flowchart reading technical solution.

According to an aspect of the present disclosure, there is provided a flowchart reading method, including: performing binarization processing on the flow chart image to obtain a first processed image, wherein the binarization processing comprises setting pixels with gray values smaller than a pixel threshold value in the flow chart image as black and pixels with gray values larger than the pixel threshold value as white; performing a negation operation on the first processed image to obtain a second processed image, wherein the negation operation comprises setting black pixels in the first processed image to be white and setting white pixels to be black; and determining the corresponding relation between the graphics and the characters in the flow chart image according to the connected domain in the second processing image, wherein the connected domain is a region with connected pixels being white.

In a possible implementation manner, the determining, according to the connected domain in the second processing image, a correspondence between graphics and text in the flowchart image includes: dividing a text layer and a first graphic layer from the second processed image, wherein the text layer is a connected domain formed by text in the second processed image, and the first graphic layer is a connected domain formed by a graphic frame and a flow line in the second processed image; a first picture frame layer and a flow line layer are segmented from the first picture layer, wherein the first picture frame layer is a connected domain formed by a picture frame in the second processing image, and the flow line layer is a connected domain formed by a flow line in the second processing image; determining characters in each frame in the first frame layer according to the character layer and the first frame layer; determining an input flow line and an output flow line of each frame in the first frame layer according to the flow line layer and the first frame layer; and determining the type of each frame in the first frame layer according to the first frame layer and a database comprising flow chart standard frames.

In one possible implementation manner, the segmenting the text layer and the first graphics layer from the second processed image includes: acquiring a minimum rectangular frame capable of containing each connected domain in the second processed image; and dividing the text layer and the first graphic layer from the second processed image according to the area and the length-width ratio of each minimum rectangular frame.

In one possible implementation manner, the dividing the first frame layer and the flow line layer from the first graphics layer includes: performing inverting operation on the first graph layer to obtain a second graph layer; dividing a first picture frame layer from the second picture layer; performing a first expansion operation on each connected domain in the first frame layer to obtain a second frame layer, wherein the first expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a first target distance along the center of each connected domain, and the first target distance is larger than the width of a frame line in the first image layer; performing inverse operation on the second frame layer to obtain a third frame layer; and obtaining a flow line layer according to the first graphic layer and the third graphic frame layer.

In one possible implementation manner, the determining, according to the text layer and the first frame layer, the text in each frame in the first frame layer includes: the following operations are sequentially executed for each frame in the first frame layer: determining a target position of an operated frame from the first frame layer; performing inverse operation on the first frame layer except the target position to obtain a fourth frame layer; and obtaining target characters in the operated frame according to the fourth frame layer and the character layer.

In one possible implementation manner, the determining, according to the flow line layer and the first frame layer, an input flow line and an output flow line of each frame in the first frame layer includes: performing a second expansion operation on each connected domain in the first frame layer to obtain a fifth frame layer, wherein the second expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a second target distance along the center of each connected domain, and the second target distance is larger than the first target distance; performing inverse operation on the fifth frame layer to obtain a sixth frame layer; and determining an input flow line and an output flow line of each frame in the first frame layer according to the sixth frame layer and the flow line layer.

In one possible implementation manner, the determining, according to the first frame layer and the database including the standard frames of the flowchart, a type of each frame in the first frame layer includes: the following operations are sequentially executed for each frame in the first frame layer: acquiring graphic features of an operated frame; determining a corresponding standard frame of the operated frame in the database according to the graphic features; and taking the type of the corresponding standard frame as the type of the operated frame.

According to another aspect of the present disclosure, there is provided a flowchart reading apparatus including: the first processing image acquisition module is used for carrying out binarization processing on the flow chart image to obtain a first processing image, wherein the binarization processing comprises setting the pixel points of which the gray values are smaller than the pixel threshold value in the flow chart image as black and the pixel points of which the gray values are larger than the pixel threshold value as white; the second processing image acquisition module is used for carrying out inverse operation on the first processing image to obtain a second processing image, wherein the inverse operation comprises setting black pixel points in the first processing image as white and setting white pixel points as black; and the corresponding relation determining module is used for determining the corresponding relation between the graphics and the characters in the flow chart image according to the connected domain in the second processing image, wherein the connected domain is a region with connected pixels being white.

In one possible implementation manner, the correspondence determining module includes: the first segmentation submodule is used for segmenting a text layer and a first graphic layer from the second processed image, wherein the text layer is a connected domain formed by text in the second processed image, and the first graphic layer is a connected domain formed by a frame and a flow line in the second processed image; the second segmentation submodule is used for segmenting a first picture frame layer and a flow line layer from the first picture layer, wherein the first picture frame layer is a connected domain formed by a picture frame in the second processing image, and the flow line layer is a connected domain formed by a flow line in the second processing image; the text determination submodule is used for determining the text in each frame in the first frame layer according to the text layer and the first frame layer; the flow line determining submodule is used for determining an input flow line and an output flow line of each frame in the first frame layer according to the flow line layer and the first frame layer; and the frame determining submodule is used for determining the type of each frame in the first frame layer according to the first frame layer and a database comprising the standard frames of the flow chart.

In one possible implementation manner, the first dividing sub-module is configured to: acquiring a minimum rectangular frame capable of containing each connected domain in the second processed image; and dividing the text layer and the first graphic layer from the second processed image according to the area and the length-width ratio of each minimum rectangular frame.

In one possible implementation manner, the second dividing sub-module is configured to: performing inverting operation on the first graph layer to obtain a second graph layer; dividing a first picture frame layer from the second picture layer; performing a first expansion operation on each connected domain in the first frame layer to obtain a second frame layer, wherein the first expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a first target distance along the center of each connected domain, and the first target distance is larger than the width of a frame line in the first image layer; performing inverse operation on the second frame layer to obtain a third frame layer; and obtaining a flow line layer according to the first graphic layer and the third graphic frame layer.

In one possible implementation, the text determination submodule is configured to: the following operations are sequentially executed for each frame in the first frame layer: determining a target position of an operated frame from the first frame layer; performing inverse operation on the first frame layer except the target position to obtain a fourth frame layer; and obtaining target characters in the operated frame according to the fourth frame layer and the character layer.

In one possible implementation, the flow line determining submodule is configured to: performing a second expansion operation on each connected domain in the first frame layer to obtain a fifth frame layer, wherein the second expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a second target distance along the center of each connected domain, and the second target distance is larger than the first target distance; performing inverse operation on the fifth frame layer to obtain a sixth frame layer; and determining an input flow line and an output flow line of each frame in the first frame layer according to the sixth frame layer and the flow line layer.

In one possible implementation, the frame determination submodule is configured to: the following operations are sequentially executed for each frame in the first frame layer: acquiring graphic features of an operated frame; determining a corresponding standard frame of the operated frame in the database according to the graphic features; and taking the type of the corresponding standard frame as the type of the operated frame.

According to another aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions stored by the memory.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, implement the above-described method.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, which when run in a processor of an electronic device, performs the above method.

In the embodiment of the disclosure, a first processing image including a connected domain formed by a blank region in a flowchart image is obtained by performing binarization processing on the flowchart image, a second processing image including a connected domain formed by characters, frames, flow lines and the like in the flowchart image is obtained by performing inversion operation on the first processing image, and information in the flowchart image is read according to the second processing image. The process obtains the flow chart information by converting the flow chart image into the simple binarized image of the second processed image through binarization processing and inverse operation, so that the processing speed of the flow chart image is improved, and meanwhile, the graphic processing mode of obtaining the corresponding relation between the graph and the text of the flow chart image through the connected domain is realized because the second processed image contains all connected domains formed by the text and the graph, so that the method has higher accuracy and further improves the robustness of the flow chart reading process.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features and aspects of the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 shows a flowchart of a flowchart reading method according to an embodiment of the present disclosure.

Fig. 2 shows a schematic diagram of determining correspondence of graphics and text in a flowchart image, according to an embodiment of the present disclosure.

Fig. 3 shows a schematic diagram of an example of an application according to the present disclosure.

Fig. 4 shows a comparison diagram of a second processed image before and after noise removal according to an application example of the present disclosure.

Fig. 5 shows a schematic diagram of a determination flow line layer according to an application example of the present disclosure.

Fig. 6 illustrates a schematic diagram of a dot product operation on a fourth frame layer and a text layer according to an application example of the present disclosure.

Fig. 7 shows a schematic diagram of a flow chart image to be identified according to an application example of the present disclosure.

Fig. 8 shows a schematic diagram of determining a type of a frame from a flowchart image according to an application example of the present disclosure.

FIG. 9 illustrates a schematic diagram of accuracy results of a key portion of a flowchart read according to an application example of the present disclosure.

Fig. 10 shows a block diagram of a flow chart reading apparatus according to an embodiment of the present disclosure.

Fig. 11 shows a block diagram of an electronic device according to an embodiment of the disclosure.

Fig. 12 shows a block diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the disclosure will be described in detail below with reference to the drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Although various aspects of the embodiments are illustrated in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

In addition, numerous specific details are set forth in the following detailed description in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order not to obscure the present disclosure.

Fig. 1 shows a flowchart of a flowchart reading method according to an embodiment of the present disclosure, which may be applied to a flowchart reading apparatus, which may be a terminal device, a server, or other processing device, or the like. The terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle mounted device, a wearable device, etc.

In some possible implementations, the flowchart reading method may be implemented by a processor invoking computer readable instructions stored in a memory.

As shown in fig. 1, the flowchart reading method may include:

Step S11, binarizing the flow chart image to obtain a first processed image, wherein the binarizing comprises setting the pixel points with gray values smaller than the pixel threshold value in the flow chart image as black and the pixel points with gray values larger than the pixel threshold value as white.

The flow chart image may be any image having a flow chart required for reading. Further, in an example, in a case where the image includes both the flowchart and other images, an image segmentation algorithm may be used to segment the flowchart in the image including more than the flowchart, so as to obtain the flowchart image, and the kind and execution steps of the image segmentation algorithm are not specifically limited in this disclosure. The format of the flowchart image may be TIFF, JPEG, PNG, etc., and the present disclosure does not specifically limit the format of the flowchart image, and may be selected according to practical situations. The manner in which the flowchart image is obtained may also be selected according to the actual situation, and in an example, the flowchart image may be read in by software such as OCR.

The specific size of the pixel threshold may be determined according to experience and/or image information of the flowchart image, and the manner of determining the pixel threshold is not specifically limited in the present disclosure. In an example, the size of the pixel threshold may be set to 100pt. Further, after determining the pixel threshold, in step S11, the flowchart image may be subjected to binarization processing by the pixel threshold. Specifically, the binarization processing may be performing binarization processing on each pixel point in the flow chart image. In an example, the binarization process may be performed according to formula (1), that is, pixel points smaller than and not smaller than the pixel threshold value are replaced by "0" and "1" respectively, and each pixel point after conversion has only two values of "0" and "1", where "0" represents black and "1" represents white.

(1)

Wherein, the method comprises the following steps ofx,y) For the position of a pixel in the flow chart image,f(x,y) For the gray value of a pixel in the flow chart image,g(x,y) Is the gray value of the pixel after binarization,Tis the pixel threshold.

Through step S11, the flowchart image is converted from the RGB image to the first processed image with only two colors of black and white, so that true color information irrelevant to content information in the flowchart image is omitted, complexity of information reading in the flowchart image in the subsequent process is reduced, and efficiency of the flowchart image reading is improved.

And step S12, performing a reversing operation on the first processed image to obtain a second processed image, wherein the reversing operation comprises setting black pixels in the first processed image as white and setting white pixels as black.

The connected domain (Connected Component) generally refers to an image region formed by foreground pixels having the same pixel value and adjacent to each other in the image. By marking white pixels with pixels of 1 in the binarized image, each individual connected region is formed into an identified block, so that geometric parameters such as the number, the position, the area, the outline, the circumscribed rectangle, the mass center, the invariant moment and the like of the blocks are further obtained. In an example, in step S12, the first processed image may be inverted according to the formula (2), and the black pixel point in the first processed image is changed to white, and the white pixel point is changed to black, so as to obtain a second processed image carrying information such as characters, frames, flow lines, etc., so as to read the flow chart image information with richer and more accurate content.

( 2）

Wherein, the method comprises the following steps ofx,y) For the position of a pixel in the flow chart image,z(x,y) To take the gray value of the pixel point in the inverted image,g(x,y) The gray value of the pixel point after the binarization processing is carried out on the image.

When noise exists in the first processing image, the noise can generate a white connected domain with a pixel of 1 in the second processing image, and then interference is generated on the subsequent acquisition of the flow chart image information through the connected domain. In general, the area occupied by a single noise in the flowchart image tends to be small, and in one possible implementation, before determining the correspondence between the graphics and the text in the flowchart according to the connected domain in the second processed image, the method includes: taking a connected domain with the area of the connected domain smaller than the area threshold of the connected domain in the second processing image as the connected domain to be modified; and setting the pixel points in the connected domain to be modified to be black.

Specifically, the method for eliminating noise in the second processed image by using the connected domain area threshold may be performed according to formula (3), sequentially traversing the areas of all the connected domains in the second processed image, comparing the traversed areas of the connected domains with the connected domain area threshold, and changing the pixel value of the traversed connected domain to black when the traversed areas of the connected domains are smaller than the connected domain area threshold, that is, eliminating the connected domain generated by noise, and keeping the pixel values of other connected domains unchanged.

(3)

Wherein,,g(x,y) To the connected domainS _i Noise-eliminated connected domainS _i The gray value of each pixel point in the image,i∈N，Nfor the number of connected domains in the second processed image,S _i is the first connected domain in the connected domain set in the second processed imageiThe area of the individual connected regions is defined,T ₁ is a connected domain area threshold.

In the embodiment of the disclosure, through setting the area threshold of the connected domain, traversing the areas of all the connected domains in the second processed image by adopting a traversal method, screening out the connected domains to be modified, which are smaller than the area threshold of the connected domains, and assigning the connected domains to be modified as 0 in the pixel value of the second processed image according to the position information of the connected domains to be modified, namely changing the pixel value into black pixels, thereby eliminating noise in the second processed image. Through the process, the connected domain formed by noise in the second processed image is effectively removed, and the subsequent utilization of the connected domain in the second processed image is facilitated, so that more accurate flow chart information is obtained.

And step S13, determining the corresponding relation between the graphics and the characters in the flow chart image according to the connected domain in the second processing image, wherein the connected domain is a region with connected pixels being white.

Generally, a flowchart includes elements such as graphics (including a flowchart line and a diagram frame), and text, where the shape of the diagram frame indicates the type of operation, the text in the diagram frame indicates the content of the operation, the flowchart line indicates the execution order of the operation, and the flowchart line is generally indicated by an arrow. Generally, the information in the flowchart mainly includes the specific content executed by each frame and the execution sequence of each frame, so the corresponding relationship may include the type corresponding to the frame, the corresponding relationship between the frame and the text, and the corresponding relationship between the frame and the flowchart. Specifically, the type (i.e., shape) of the frame may represent the operation type of the step represented by the frame, the correspondence between the frame and the text may be a specific text included in each frame, and the correspondence between the frame and the flow line may be an input flow line and an output flow line of each frame. The information of the flow chart image can be obtained through the determination of each frame type, the determination of specific characters included in each frame and the determination of the input and output flow lines of each frame. That is, since all connected areas including characters, frames, flow lines, and the like in the second processed image can be obtained by the correspondence between the graphics and the characters, the information of the flow chart image can be obtained entirely by the second processed image in step S13. The method of acquiring information of the flowchart image by the second processing image will be described in detail in the later embodiments, and will not be expanded here.

In the embodiment of the disclosure, a first processing image including a connected domain formed by a blank region in a flowchart image is obtained by performing binarization processing on the flowchart image, a second processing image including a connected domain formed by characters, frames, flow lines and the like in the flowchart image is obtained by performing inversion operation on the first processing image, and information in the flowchart image is read according to the second processing image. The process obtains the flow chart information by converting the flow chart image into the simple binarized image of the second processed image through binarization processing and inverse operation, so that the processing speed of the flow chart image is improved, and meanwhile, the image processing mode of obtaining the corresponding relation between the graph and the text of the flow chart image through the connected domain is higher in accuracy due to the fact that the second processed image contains all connected domains formed by the text and the graph, and the robustness of the whole flow chart reading process is further improved.

In an example, flow line information, frame information, and text information in the flow chart image may be acquired, respectively, and the flow chart image information may be acquired through a combination of the flow line information, the frame information, and the text information. In a possible implementation manner, as shown in fig. 2, the determining, according to the second connected domain in the second processed image, a correspondence between graphics and text in the flowchart image includes:

Step S201, a text layer and a first graphics layer are segmented from the second processed image, where the text layer is a connected domain formed by text in the second processed image, and the first graphics layer is a connected domain formed by a frame and a flow line in the second processed image.

Generally, for the text and the graphics (including the flow lines and the frames, etc.) in the flow chart, as shown in the following table 1, the overall appearance of the text and the graphics has a large difference, for example, the length, the width, the area occupied by the text and the graphics have a large difference, which can be used as a distinguishing point for dividing the text layer and the graphics layer.

TABLE 1 Length, width, area and aspect ratio distribution of text and graphics in a flow chart

	Long length	Wide width of	Area of	Aspect ratio
					Text with a character pattern	Smaller size	Smaller size	Smaller size	Smaller size
Arrow (straight)	Larger size	Smaller size	Smaller size	Larger size
					Arrow (turning)	Larger size	Larger size	Larger size	Smaller size
Picture frame	Larger size	Larger size	Larger size	Smaller size

In one possible implementation, step S201 includes: acquiring a minimum rectangular frame capable of containing each connected domain in the second processed image; and dividing the text layer and the first graphic layer from the second processed image according to the area and the length-width ratio of each minimum rectangular frame.

Specifically, since the connected domains formed by the characters and the graphics in the second processed image are different in overall appearance, a minimum rectangular frame may be set for each connected domain, and the minimum rectangular frame may include the connected domain. Specifically, the appearance of the smallest rectangular frame of each connected domain in the second processed image can be traversed in sequence, the text is segmented according to the formula (4), and the connected domain formed by the text is segmented from the second processed image when the area of the smallest rectangular frame is smaller than the area threshold and the aspect ratio is smaller than the aspect ratio threshold. After traversing, a text layer is constructed according to the connected domain formed by the segmented text.

(4)

Wherein,,i∈N，Nfor the number of connected domains in the second processed image,C _t is a matrix of a character layer,C _il andC _iw connected domain sets of the second processed image respectivelyCMiddle (f)iThe length and width of the smallest rectangular frame of the sub-connected domain,T ₁ as a threshold value for the area,T ₂ is the aspect ratio threshold.

Specifically, the appearance of the smallest rectangular frame of each connected region in the second processed image may be sequentially traversed, the pattern may be divided according to the formula (5), the pattern may not be divided when the area of the smallest rectangular frame is smaller than the area threshold and the aspect ratio is smaller than the aspect ratio threshold, and the pattern may be divided from the second processed image otherwise. After traversing, constructing a first graph layer according to the connected domain formed by the divided graph.

(5)

Wherein,,i∈N，Nfor the number of connected domains in the second processed image,C _g in the form of a first matrix of graphics layers,C _il andC _iw connected domain sets of the second processed image respectivelyCMiddle (f)iThe length and width of the smallest rectangular frame of the sub-connected domain,T ₁ as a threshold value for the area,T ₂ is the aspect ratio threshold.

In one example, the threshold valueT ₁ AndT ₂ the calculation of (a) is shown in the following formula (6) and formula (7), respectively:

（6）

wherein,,T ₁ as a threshold value for the area,A _p andA _a respectively the mode and average number of all connected domain areas in the second processed image,nthe coefficients preset for preference according to risk (typically ranging between 3-15).

（7）

Wherein,,T ₁ as a threshold value for the area,T ₂ is the aspect ratio threshold.

In the embodiment of the disclosure, by setting a minimum rectangular frame for each connected domain in the second processed image, the text layer and the first graphics layer are segmented from the second processed image according to the difference in area and aspect ratio between the minimum rectangular frame containing the text-formed connected domain and the minimum rectangular frame containing the graphics-formed connected domain. The process utilizes the minimum rectangular frame containing the connected domain to realize the effective and rapid division of the characters and the graphics in the second processed image, thereby being beneficial to further rapidly reading the information of the flow chart image according to the characters and the graphics.

Step S202, a first frame layer and a flow line layer are separated from the first graphic layer, where the first frame layer is a connected domain formed by a frame in the second processed image, and the flow line layer is a connected domain formed by a flow line in the second processed image.

In the first graphics layer divided in step S201, a flow line and a frame are included, and the flow line and the frame are divided respectively, so as to help to obtain an operation type represented by each frame, and a connection relationship between text and the flow line included in each frame and the frame.

The difference of the picture frames and the flow lines in the picture layers in the appearance is large, specifically, if the first picture layer is inverted, a blank connected domain which is symmetrical in shape and regular is arranged in each picture frame in the inverted image, the picture frames can be directly separated according to the blank connected domain, and then the flow lines are further separated according to the separated picture frames. In one possible implementation, step S202 includes: performing inverting operation on the first graph layer to obtain a second graph layer; dividing a first picture frame layer from the second picture layer; performing a first expansion operation on each connected domain in the first frame layer to obtain a second frame layer, wherein the first expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a first target distance along the center of each connected domain, and the first target distance is larger than the width of a frame line in the first image layer; performing inverse operation on the second frame layer to obtain a third frame layer; and obtaining a flow line layer according to the first graphic layer and the third graphic frame layer.

Specifically, in the second graphic layer obtained by reversing the first graphic layer, besides the connected domain formed by each frame, the connected domain with the largest area formed by the background is also formed. In an example, the connected domain with the largest area may be removed from the second graphics layer, and then other connected domains are separated from the second graphics layer, so as to obtain the first frame layer. In the process of dividing the first frame layer, the flow line may be overlapped with the frame line of the frame due to the non-standardization of the flow line, so that the frame is not completely divided. In one example, after the first frame layer is initially separated, a basic structure, namely a horizontal line segment and a vertical line segment, can be created again, and by stacking the two structures along the frame edges of the frame, the contour of the interior of the frame is smoothed to further ensure the integrity of the frame.

When the first frame layer is separated from the second graphic layer, the flow line may form a closed space together with other adjacent flow lines and frames to form a pseudo frame. Therefore, the dummy frame needs to be removed to improve the accuracy of the information read by the flowchart image. In one example, the frames may be distinguished from the pseudo frames by convexity, symmetry of the enclosed space. Specifically, convexity refers to the proportion of the area of the point set of the area to be analyzed to the area of the convex hull, the convexity of the true frame is 1, and the area of the false frame is reduced by 0.25-0.5 times due to the existence of the arrow of the flow line. The calculation of the convexity value can be performed according to the formula (8):

（8）

Wherein,,Sthe value of the convexity is given by,C _G.area as the area of the point set,CH _area is the convex hull area of the point set.

The vertical symmetry is equal to the ratio of the areas of the left and right parts of one connected domain, and in one example, the vertical symmetry value can be set, and then the frame and the pseudo frame are distinguished according to the threshold value. The calculation of the vertical symmetry value can be performed according to formula (9), and the ratio of the areas of the two parts of the connected domain at the point of distinguishing the connected domain is obtained, wherein the value is the value of the vertical symmetry:

（9）

wherein,,VSis a value for the vertical symmetry and,Xdistinguishing between one connected domainThe values of the points of the left and right partsx _i ,y _i ) For the pixel coordinate values,x _i as the abscissa of the pixel(s),C _G is a set of connected domains.

According to the convexity and symmetry, the frames in the first frame layer can be traversed in sequence, and separation of the frames and the pseudo frames is carried out. Since the shape of each frame may be different, so may the value of the vertical symmetry of each frame, in one example, the type of each frame may be determined first, and then a different vertical symmetry threshold may be set for each type of frame. Specifically, the separation of the frame and the pseudo frame may be performed according to the following formula (10), where the frame is divided when the convexity of the connected domain is greater than the threshold of the convexity of the connected domain and the vertical symmetry of the connected domain is less than the threshold of the vertical symmetry of the connected domain, and the frame is not divided otherwise:

（10）

Wherein,,C _N in order to separate out the frames of the picture,i∈N，Nfor the number of connected domains in the second processed image,C _is is the first in the connected domain setiThe value of convexity of the sub-connected domain,T ₄ as a threshold for convexity of connected domain in the connected domain set,C _ivs is the first in the general domain setiThe value of the vertical symmetry of the sub-connected domain,T _ivs is the first in the connected domain setiThreshold of vertical symmetry of sub-connected domains.

It is apparent that there is a frame line width gap between each pair of corresponding frames in the first frame layer as compared to the frames in the flowchart image. In an example, a first expansion operation may be performed on each frame in the first frame layer, and then the expanded second frame layer may be removed from the second graphic layer, thereby obtaining the flow line layer. In order to ensure that each frame obtained by the first expansion operation can correspond to a frame in the first graphics layer, in an example, the first expansion operation may be to expand each edge of each connected domain in the first frame layer by a first target distance along the center of each connected domain. Further, to ensure that the frame is completely removed from the first graphics layer, in one example, the outwardly extending first target distance may be greater than the width of the frame line in the first graphics layer.

It should be noted that, in this embodiment, assuming that the thicknesses of all the frames are the same, for the flowchart image with uneven thicknesses of the frames, expansion may be performed according to this embodiment, for example, the first target distance may be determined according to the thickest line frame in the flowchart image, or the first target distance corresponding to each frame may be defined according to the thickness of the line of each frame in the flowchart image, which is not specifically limited in this disclosure.

In one example, a dot-product operation of the image matrix may be utilized to remove frames in the first graphics layer to partition flow lines in the first graphics layer. In this case, the pixels of the frame in the second frame layer need to be set to "0", that is, the second frame layer is subjected to the inverting operation, so as to obtain the third frame layer. And obtaining the flow line layer through the dot multiplication operation of the third picture frame layer and the first picture layer.

In the embodiment of the disclosure, the maximum connected domain is sequentially inverted and removed from the first graphic layer, the first frame layer is separated from the first graphic layer, and then the flow line layer is separated from the first graphic layer through the first frame layer after the expansion operation. The process realizes the rapid and accurate segmentation of the picture frame and the flow line from the first picture layer through the inverse operation, the expansion operation and the combination of the first picture layer and the second picture frame layer, and improves the efficiency and the accuracy of reading the flow chart information.

Step S203, determining the text in each frame in the first frame layer according to the text layer and the first frame layer.

For each frame in the flowchart image, the text in the frame represents the specific content of the operation, which is the important content of the flowchart image information. Thus, the text in each frame needs to be determined. Since each frame and the text in the frames are in one-to-one correspondence, the determination of each frame in the first frame layer and the text corresponding to the frame can be performed in a traversing manner. In one possible implementation, step S203 includes: the following operations are sequentially executed for each frame in the first frame layer: determining a target position of an operated frame from the first frame layer; performing inverse operation on the first frame layer except the target position to obtain a fourth frame layer; and obtaining target characters in the operated frame according to the fourth frame layer and the character layer.

Specifically, after the target frame is determined, other frames except the target frame in the first frame layer are inverted, that is, all the other frames are assigned to be "0", so as to obtain a fourth frame. Further, the fourth frame layer and the text layer of the target frame can be used to obtain the target text in the target frame. In an example, the fourth frame layer and the text layer may be subjected to a dot product operation to obtain the target text in the target frame. The method for obtaining the target text in the target frame through the fourth frame layer and the text layer is not particularly limited, and other methods for obtaining the target text in the target frame through the fourth frame layer and the text layer can be adopted.

In an example, when the connected domain analysis is performed on the frames in the first frame layer, all the connected domains in the first frame layer may be marked by numbers, that is, the value of "1" in each connected domain is replaced by the serial number of the connected domain. Only one connected domain is reserved in the node matrix at a time, and all connected domains with the rest serial numbers are assigned to be 0. And then, the reserved connected domains are stored in a cell array of the corresponding serial number node, and the node matrix point only containing one connected domain is multiplied by the inverted character matrix to obtain all character information contained in the node. Further, after character recognition can be performed by using an OCR tool in MATLAB, the character information can be stored in the cell array of the corresponding sequence number node. The purpose of character recognition is to recognize node characters in the form of pictures. Since OCR can only be applied to portions that meet the default arrangement rules, to ensure accuracy, in one example, text can be output in words and then all words can be concatenated sequentially using spaces as spacers.

In the embodiment of the disclosure, only one frame connected domain is reserved at a time, the rest frame connected domains are subjected to inverse operation, then characters in the target frame are obtained by only reserving a fourth frame layer and a character layer of one frame connected domain, and characters corresponding to all frames in the first frame layer can be obtained by sequentially carrying out the operation on all frames in the first frame layer. The process can realize the determination of the specific content executed by each frame in the flow chart image by traversing each frame in the first frame layer in sequence and simultaneously carrying out the inverse operation on other frames and combining the frames with the text layer, thereby improving the efficiency and the accuracy of reading the flow chart image information.

Step S204, determining an input flow line and an output flow line of each frame in the first frame layer according to the flow line layer and the first frame layer.

In the flow chart image, one frame is often closest to the flow line connected with the frame, so that the flow line influenced by the expanded frame can be obtained by expanding the frame, and the flow line connected with the frame can be obtained. In one possible implementation, step S204 includes: performing a second expansion operation on each connected domain in the first frame layer to obtain a fifth frame layer, wherein the second expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a second target distance along the center of each connected domain, and the second target distance is larger than the first target distance; performing inverse operation on the fifth frame layer to obtain a sixth frame layer; and determining an input flow line and an output flow line of each frame in the first frame layer according to the sixth frame layer and the flow line layer.

Specifically, when the second expansion operation is performed on each connected domain in the first frame layer to obtain the fifth frame layer, since the area of each connected domain in the fifth frame layer is larger than the area of the frame in the first frame layer corresponding to the connected domain, and the frames in the flowchart are often tightly connected to the flow lines, the connected domain in the fifth frame layer affects the shape of the flow lines connected to the connected domain. In order to ensure that each frame obtained by the second expansion operation can correspond to a frame in the first graphics layer, in an example, the second expansion operation may be to expand each edge of each connected domain in the first frame layer by a second target distance along the center of each connected domain. The flow line layer is obtained through expansion operation of the frames in the first frame layer, and further, in order to enable the connected domain in the fifth frame layer obtained through the second expansion operation to influence the shape of the flow line adjacent to the frames, the second target distance is larger than the first target distance in the first expansion operation.

In an example, the frame bodies of the frames in the first frame layer may be sequentially expanded according to the serial numbers of the frames in the first frame layer to obtain a fifth frame layer with only one expanded frame, and then the expanded fifth frame layer is inverted again to obtain a sixth frame layer, where the flow lines in the flow line layer affected by the frame bodies of the expanded frames are flow lines connected with the expanded frame according to the sixth frame layer and the flow line layer. Specifically, each flow line can be randomly given a serial number, so that two frames connected with each flow line and inflow and outflow directions of the two frames can be identified, and the overall flow direction of data in the flow chart image can be obtained. Specifically, the area of each flow line before and after the second expansion operation can be counted, and the flow line with the changed area is the flow line connected with the expansion frame.

After determining the flow line connected to the frames, the direction of the flow line needs to be determined, so as to further determine the execution sequence of each frame in the flow chart image. For a straight line flow, in one example, the direction of the straight line flow may be inferred by calculating the bias of the centroid of the straight line flow. For turn and multi-turnout flow lines, in one example, the determination may be made by determining the distribution of pixels near the portion of the frame that contacts the flow line. For example, for a flow line of an arrow, the direction of the arrow may be determined by determining the distribution of arrow pixels near the portion of the frame in contact with the flow line.

In the embodiment of the disclosure, the frame in the first frame layer is expanded to obtain a fifth frame layer, and the flow line affected by the expanded frame is determined by inverting the fifth frame layer to obtain a sixth frame layer and a flow line layer, so that the flow line connected with the frame can be determined. The process realizes the rapid determination of the flow lines connected with each frame in the flow chart image through the two simple operations of expansion and inversion, and further rapidly determines the connection modes of all frames in the flow chart image so as to further determine the circulation sequence of data among the frames in the flow chart image, thereby realizing the effective reading of the flow chart image information and further improving the efficiency and the accuracy of reading the flow chart information.

Step S205, determining the type of each frame in the first frame layer according to the first frame layer and the database including the standard frames of the flowchart.

For each frame in the flowchart image, the shape of the frame represents the type of various operations, which is important content of the flowchart image information. Thus, the shape of each frame needs to be determined.

In one possible implementation, step S205 may include: the following operations are sequentially executed for each frame in the first frame layer: acquiring graphic features of an operated frame; determining a corresponding standard frame of the operated frame in the database according to the graphic features; and taking the type of the corresponding standard frame as the type of the operated frame.

Specifically, the standard frame matched with the frame in the first frame layer can be obtained by obtaining the graphic features such as the area, the mass center and the shape of the frame in the first frame layer and comparing the graphic features with the graphic features of the standard frame in the database comprising the standard frame body of the flow chart. In one example, a method of pattern recognition may be employed to determine the pattern features of the frames. In particular, pattern recognition may be performed using a geometric moment based shape descriptor. The geometric moment is an operator describing the image features. If it is toThe connected domain formed by one picture frame is regarded as a flat object, and the pixel points are #i,j) The value of (1) is regarded as the density here and is recorded asV(i,j) The desired moment at a point is the moment of the connected domain at that point. The zero-order moment is calculated by the area of a connected domain formed by the frame, and the calculation process of the zero-order moment is shown in a formula (11):

（11）

wherein,,M ₀₀ is a connected domain set in the first picture frame layerCThe area of a certain sub-connected domain in the liquid crystal display device,V(i,j) Is pixel point [ ]i,j) Is a density of (3).

The first moment is often used to calculate the centroid of the connected domain formed by the frame, and the calculation process of the first moment is shown in formula (12) and formula (13):

（12）

（13）

Wherein,,M ₁₀ andM ₀₁ connected domain sets in the first frame layerCThe abscissa of the centroid of a certain sub-connected domain,V(i,j) Is pixel point [ ]i,j) Is used for the density of the (c) in the (c),i、jrespectively pixels are [ ]i,j) And the abscissa of (2).

The second moment, also called moment of inertia, is used to calculate the shape direction of the connected domain formed by the frame, and the calculation process is shown in the formula (14), the formula (15) and the formula (16):

（14）

（15）

（16）

wherein,,M ₂₀ 、M ₀₂ andM ₁₁ connected domain sets in the first frame layerCThe moment of inertia of a certain sub-connected domain in the matrix,V(i,j) Is pixel point [ ]i,j) Is used for the density of the (c) in the (c),i、jrespectively pixels are [ ]i,j) And the abscissa of (2).

Further, the zero-order moment, the first-order moment and the second-order moment are calculated for the connected domain formed by each frame of the first frame layer, and moment vectors shown in formula (17) are formed:

（17）

wherein ƒ is a collection of connected domains in the first frame layerCMoment vector of a certain sub connected domain in the matrix,M ₀₀ is a connected domain set in the first picture frame layerCThe area of a certain sub-connected domain in the liquid crystal display device,M ₁₀ andM ₀₁ connected domain sets in the first frame layerCThe abscissa of the centroid of a certain sub-connected domain,M ₂₀ 、M ₀₂ andM ₁₁ connected domain sets in the first frame layerCThe moment of inertia of a certain sub-connected domain.

Specifically, the value of the moment may be normalized according to the value range of each dimension, and in one example, the moment vector may be normalized to be between [0,1 ]. Further, the normalized moment vector is matched with the moment vector of the standard shape in the database, and the shape of each frame is determined. Wherein, the database stores all frames with standard shapes which may exist in the connected image.

In the embodiment of the disclosure, the shape feature of each frame in the first frame layer is obtained first, and then the shape feature is matched with the shape feature of a standard frame in the database to determine the shape of each frame in the first frame layer. The process realizes the accurate determination of the frame shape in the flow chart image by fast matching the frame in the first frame layer with the standard frame in the database, and improves the efficiency and accuracy of reading the flow chart information.

Through the steps S201 to S205, a text layer and a first graphics layer are separated from the second processed image, and a first frame layer and a flow line layer are separated from the first graphics layer, so that the recognition of the frame shape of each frame in the flow chart image, the determination of the Chinese in each frame, the determination of the input flow line and the output flow line of each frame, and the information reading of the flow chart image are realized according to the text layer, the first frame layer and the flow line layer.

After the flow chart reading is completed, in order to quantify the accuracy of the reading method, an accuracy measurement method integrating the performances of each link can be defined. In an example, a method of fractional evaluation may be used to evaluate and evaluate four parts, including preprocessing (including noise removal), element segmentation (including segmentation of text and graphics, segmentation of frames and flow lines), element recognition (frame type recognition and text recognition), and element reorganization (including correspondence of text and frames, correspondence of frames and flow lines), respectively.

Wherein, the preprocessing part can observe the image quality after binarization processing and the effect of noise filtering; the element segmentation part can observe the degree of correct segmentation of various elements, including segmentation of characters/graphics and segmentation of frames/flow lines; an element recognition section that can observe the precision of frame type recognition and character recognition; and the element reorganization part can observe whether the correspondence between the characters and the frames and the correspondence between the frames and the flow lines are comprehensive and accurate.

Further, the accuracy of each portion may be normalized to between 0,1 to determine the accuracy of the flowchart reading at that portion.

Fig. 3 is a schematic diagram illustrating an application example according to the present disclosure, as shown in fig. 3, an embodiment of the present disclosure proposes a method for reading a flowchart, where the method may implement reading of flowchart information by processing the flowchart, and a process of reading the flowchart may be:

as shown in fig. 3, the flowchart reading process can be roughly divided into four steps.

And the first step, binarizing the flow chart image to obtain a first processed image. Specifically, the pixels in the flow chart image with gray values smaller than the pixel threshold value are set to be black, and the pixels in the flow chart image with gray values larger than the pixel threshold value are set to be white. Wherein the pixel threshold is set to 100.

And secondly, performing inverse processing on the first processed image to obtain a second processed image. Specifically, the pixel point of black in the first processed image is set to white, and the pixel point of white is set to black.

And thirdly, preprocessing the second processed image. Specifically, the pixel points of the connected domain in the second processed image with the area smaller than the area threshold of the connected domain are set to be black, so that noise in the second processed image is removed. Fig. 4 is a comparison diagram of the second processed image before and after noise removal. As shown in fig. 4, there is a distinct noise point in the lower right side of the second processed image before noise removal, and the noise point in the second processed image after preprocessing disappears.

And fourthly, determining the corresponding relation between the graphics and the characters in the flow chart image through the connected domain in the second processing image. Specifically, the process includes:

and the second processing image is used for dividing the text layer and the first graphic layer. Specifically, the process includes: and dividing the text layer and the first graphic layer from the second processed image according to the area and the length-width ratio of the smallest rectangular frame containing each connected domain.

And dividing a first frame layer and a flow line layer in the first graphic layer. Specifically, the process includes: performing inverse operation on the first graph layer, and removing the largest connected domain to obtain a first graph frame layer; performing a first expansion operation on each frame in the first frame layer to obtain a second frame layer; performing inverse operation on the second frame layer to obtain a third frame layer; and performing dot multiplication on the third picture frame layer and the first picture layer to obtain a flow line layer. FIG. 5 is a schematic diagram of a flow line layer obtained by a dot product operation of a third frame layer and a first graphics layer. As can be seen from fig. 5, the flow lines in the first graphics layer are segmented by a dot product operation.

The determination of the Chinese in the frames of the flow chart image. Specifically, the process includes: determining a target frame from the first frame layer; performing inverse operation on the first frame layer except the target frame to obtain a fourth frame layer; and performing dot multiplication on the fourth frame layer and the text layer to obtain the text in the target frame. Fig. 6 is a schematic diagram of performing a dot product operation on a fourth frame layer and a text layer. As can be seen from fig. 6, by the dot multiplication operation, the text in the frame of the fourth frame layer, which is subjected to the negation operation, is discarded, while the other text is retained.

TABLE 2 results of recognition of frame shape and text in frame for flowchart images

Frame number	Frame shape	The frame containing text
			1	Rectangle shape	Education and training
2	Rectangle shape	Engineer capability
			3	Rectangle shape	RAMS theory/math
4	Rectangle shape	RAMS method/data
			5	Rectangle shape	RAMS procedure
6	Rectangle shape	Process implementation
			7	Rectangle shape	RAMS phase/task
8	Rectangle shape	RAMS engineer toolbox

Input flow lines for boxes in the flow chart image and determination of the input flow lines. Specifically, the process includes: acquiring the area of each flow line in the flow line layer; performing a second expansion operation on the connected domain in the first frame layer to obtain a fifth frame layer; performing inverse operation on a fifth frame layer outside the target frame to obtain a sixth frame layer; performing point multiplication on the sixth frame layer and the flow line layer, and determining a flow line of the area affected by the target frame; and determining an input flow line and an output flow line of the target frame according to the deviation of the centroid of the flow line. Fig. 7 is a schematic diagram of a to-be-identified flowchart image, in which, for convenience of displaying the identification result, serial numbers are marked for each frame and each flow line of the flowchart image (frame serial numbers refer to numbers (1) to (8) in fig. 7, and flow line serial numbers refer to numbers 1 to 10 in fig. 7), and it should be understood that, since the numbers of the frames and the flow lines are randomly assigned during the processing of the flowchart image, the serial numbers of the frames and the flow lines in fig. 7 may not be the same as the actual numbers during the processing of the flowchart image. Table 2 is a result of recognizing the shape of each frame and the text contained in the frame from the flowchart image of fig. 7. As can be seen from table 2, the shape of each frame and the text in the frame in the flowchart image of fig. 7 are identified.

TABLE 3 results of input and output frames for a flowsheet image recognition flowsheet

Flow line number	Inputting frame number	Outputting the picture frame number
			1	1	2
2	2	5
			3	-	7
4	3	4
			5	4	6
6	6	5
			7	5	7
8	7	-
			9	3、4、6	8
10	8	5

Table 3 is the result of identifying the input and output frames for each flow line from the flow chart image of FIG. 7. As can be seen from table 3, the input flow line and the output flow line of each frame in the flow chart image are determined by the recognition process. Further, the input flow line and the output flow line of each frame may be stored together with the shape of the frame, the text in the frame, and the like in the cell array corresponding to the frame number.

The shape of the frame in the flowchart image is identified. Specifically, the process includes: acquiring graphic features of each frame in the first frame layer; and comparing the image characteristics with the graphic characteristics of standard frames in a database comprising the standard frames of the flow chart, and determining the standard frames corresponding to each frame in the first frame layer. FIG. 8 is a schematic diagram of determining the type of each frame from a flowchart image.

Fig. 9 is a schematic diagram of the results of the examination and evaluation of four parts, i.e., preprocessing (including noise removal), element segmentation (including segmentation of characters and graphics, segmentation of frames and flow lines), element recognition (frame type recognition and character recognition), and element recombination (including correspondence of characters and frames, correspondence of frames and flow lines), and the accuracy of each part obtained. It can be seen that the identification accuracy of the flow chart reading method is higher, and the flow chart reading method can be directly applied to calculation of the similarity of the flow charts. Specifically, the method for calculating the similarity of the graphs is not particularly limited, and can be determined according to the situation.

In the embodiment of the disclosure, a first processing image including a connected domain formed by a blank region in a flowchart image is obtained by performing binarization processing on the flowchart image, a second processing image including a connected domain formed by characters, frames, flow lines and the like in the flowchart image is obtained by performing inversion operation on the first processing image, and information in the flowchart image is read according to the second processing image. The process reads the information of the flow chart image by acquiring the binarized image of the second processing image, and the efficiency and the accuracy of reading the flow chart information can be improved by the process due to the simple mode of the binarized image, so that the robustness of the flow chart reading process is improved.

It will be appreciated that the above-mentioned method embodiments of the present disclosure may be combined with each other to form a combined embodiment without departing from the principle logic, and are limited to the description of the present disclosure. It will be appreciated by those skilled in the art that in the above-described methods of the embodiments, the particular order of execution of the steps should be determined by their function and possible inherent logic.

In addition, the disclosure further provides a flowchart reading device, an electronic device, a computer readable storage medium, and a program, where the foregoing may be used to implement any one of the flowchart reading methods provided in the disclosure, and corresponding technical schemes and descriptions and corresponding descriptions referring to method parts are not repeated.

Fig. 10 shows a block diagram of a flow chart reading apparatus according to an embodiment of the present disclosure. The flow chart reading means may be a terminal device, a server or other processing device, etc. The terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle mounted device, a wearable device, etc.

In some possible implementations, the flowchart reading means may be implemented by a processor invoking computer readable instructions stored in a memory.

As shown in fig. 10, the flowchart reading apparatus 100 may include:

the first processed image obtaining module 101 is configured to perform binarization processing on the flowchart image to obtain a first processed image, where the binarization processing includes setting a pixel point in the flowchart image with a gray value smaller than a pixel threshold to be black and a pixel point with a gray value greater than the pixel threshold to be white.

The second processed image obtaining module 102 is configured to perform a negation operation on the first processed image to obtain a second processed image, where the negation operation includes setting black pixels in the first processed image to white and setting white pixels to black.

And the correspondence determining module 103 is configured to determine a correspondence between graphics and text in the flowchart image according to a connected domain in the second processed image, where the connected domain is a connected region with white pixels.

The disclosed embodiments also provide a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method. The computer readable storage medium may be a non-volatile computer readable storage medium.

The embodiment of the disclosure also provides an electronic device, which comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.

Embodiments of the present disclosure also provide a computer program product comprising computer readable code which, when run on a device, causes a processor in the device to execute instructions for implementing the flowchart reading method provided in any of the embodiments above.

The disclosed embodiments also provide another computer program product for storing computer readable instructions that, when executed, cause a computer to perform the operations of the flowchart reading method provided in any of the above embodiments.

The electronic device may be provided as a terminal, server or other form of device.

Fig. 11 illustrates a block diagram of an electronic device 800, according to an embodiment of the disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.

Referring to fig. 11, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output interface 812, a sensor component 814, and a communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interactions between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen between the electronic device 800 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operational mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 further includes a speaker for outputting audio signals.

Input/output interface 812 provides an interface between processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.

The sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of the electronic device 800. For example, the sensor assembly 814 may detect an on/off state of the electronic device 800, a relative positioning of the components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in position of the electronic device 800 or a component of the electronic device 800, the presence or absence of a user's contact with the electronic device 800, an orientation or acceleration/deceleration of the electronic device 800, and a change in temperature of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices, either wired or wireless. The electronic device 800 may access a wireless network based on a communication standard, such as WiFi,2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 804 including computer program instructions executable by processor 820 of electronic device 800 to perform the above-described methods.

Fig. 12 shows a block diagram of an electronic device 1900 according to an embodiment of the disclosure. For example, electronic device 1900 may be provided as a server. Referring to fig. 12, electronic device 1900 includes a processing component 1922 that further includes one or more processors and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by processing component 1922. The application programs stored in memory 1932 may include one or more modules each corresponding to a set of instructions. Further, processing component 1922 is configured to execute instructions to perform the methods described above.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output interface 1958. Electronic device 1900 may operate an operating system based on memory 1932, such as a Windowserver ^TM ，Mac OS X ^TM ，Unix ^TM , Linux ^TM ，FreeBSD ^TM Or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 1932, including computer program instructions executable by processing component 1922 of electronic device 1900 to perform the methods described above.

The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.

Computer program instructions for performing the operations of the present disclosure can be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the improvement of technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method of flowchart reading, comprising:

performing binarization processing on the flow chart image to obtain a first processed image, wherein the binarization processing comprises setting pixels with gray values smaller than a pixel threshold value in the flow chart image as black and pixels with gray values larger than the pixel threshold value as white;

Performing a negation operation on the first processed image to obtain a second processed image, wherein the negation operation comprises setting black pixels in the first processed image to be white and setting white pixels to be black;

determining the corresponding relation between the graphics and the characters in the flow chart image according to the connected domain in the second processing image, wherein the connected domain is a connected area with white pixels;

the determining, according to the connected domain in the second processed image, a correspondence between graphics and text in the flowchart image includes:

dividing a text layer and a first graphic layer from the second processed image, wherein the text layer is a connected domain formed by text in the second processed image, and the first graphic layer is a connected domain formed by a graphic frame and a flow line in the second processed image;

a first picture frame layer and a flow line layer are segmented from the first picture layer, wherein the first picture frame layer is a connected domain formed by a picture frame in the second processing image, and the flow line layer is a connected domain formed by a flow line in the second processing image;

determining characters in each frame in the first frame layer according to the character layer and the first frame layer;

Determining an input flow line and an output flow line of each frame in the first frame layer according to the flow line layer and the first frame layer;

determining the type of each frame in the first frame layer according to the first frame layer and a database comprising flow chart standard frames;

the step of dividing the first graphic layer into a first frame layer and a flow line layer comprises the following steps:

performing inverting operation on the first graph layer to obtain a second graph layer;

dividing a first picture frame layer from the second picture layer;

performing a first expansion operation on each connected domain in the first frame layer to obtain a second frame layer, wherein the first expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a first target distance along the center of each connected domain, and the first target distance is larger than the width of a frame line in the first image layer;

performing inverse operation on the second frame layer to obtain a third frame layer;

and obtaining a flow line layer according to the first graphic layer and the third graphic frame layer.

2. The method of claim 1, wherein the segmenting the text layer and the first graphics layer from the second processed image comprises:

Acquiring a minimum rectangular frame capable of containing each connected domain in the second processed image;

and dividing the text layer and the first graphic layer from the second processed image according to the area and the length-width ratio of each minimum rectangular frame.

3. The method of claim 1, wherein the determining text in each frame of the first frame layer from the text layer and the first frame layer comprises:

the following operations are sequentially executed for each frame in the first frame layer:

determining a target position of an operated frame from the first frame layer;

performing inverse operation on the first frame layer except the target position to obtain a fourth frame layer;

and obtaining target characters in the operated frame according to the fourth frame layer and the character layer.

4. The method of claim 1, wherein determining the input flow line and the output flow line for each frame in the first frame layer based on the flow line layer and the first frame layer comprises:

performing a second expansion operation on each connected domain in the first frame layer to obtain a fifth frame layer, wherein the second expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a second target distance along the center of each connected domain, and the second target distance is larger than the first target distance;

Performing inverse operation on the fifth frame layer to obtain a sixth frame layer;

and determining an input flow line and an output flow line of each frame in the first frame layer according to the sixth frame layer and the flow line layer.

5. The method of claim 1, wherein determining the type of each frame in the first frame layer from the first frame layer and a database including flow chart standard frames comprises:

acquiring graphic features of an operated frame;

determining a corresponding standard frame of the operated frame in the database according to the graphic features;

and taking the type of the corresponding standard frame as the type of the operated frame.

6. A flow chart reading apparatus, comprising:

the first processing image acquisition module is used for carrying out binarization processing on the flow chart image to obtain a first processing image, wherein the binarization processing comprises setting the pixel points of which the gray values are smaller than the pixel threshold value in the flow chart image as black and the pixel points of which the gray values are larger than the pixel threshold value as white;

the second processing image acquisition module is used for carrying out inverse operation on the first processing image to obtain a second processing image, wherein the inverse operation comprises setting black pixel points in the first processing image as white and setting white pixel points as black;

The corresponding relation determining module is used for determining the corresponding relation between the graphics and the characters in the flow chart image according to the connected domain in the second processing image, wherein the connected domain is a connected area with white pixel points;

the correspondence determining module includes: the first segmentation submodule is used for segmenting a text layer and a first graphic layer from the second processed image, wherein the text layer is a connected domain formed by text in the second processed image, and the first graphic layer is a connected domain formed by a frame and a flow line in the second processed image; the second segmentation submodule is used for segmenting a first picture frame layer and a flow line layer from the first picture layer, wherein the first picture frame layer is a connected domain formed by a picture frame in the second processing image, and the flow line layer is a connected domain formed by a flow line in the second processing image; the text determination submodule is used for determining the text in each frame in the first frame layer according to the text layer and the first frame layer; the flow line determining submodule is used for determining an input flow line and an output flow line of each frame in the first frame layer according to the flow line layer and the first frame layer; a frame determining submodule, configured to determine a type of each frame in the first frame layer according to the first frame layer and a database including a flowchart standard frame;

The second dividing sub-module is used for: performing inverting operation on the first graph layer to obtain a second graph layer; dividing a first picture frame layer from the second picture layer; performing a first expansion operation on each connected domain in the first frame layer to obtain a second frame layer, wherein the first expansion operation comprises expanding each edge of each connected domain in the first frame layer outwards by a first target distance along the center of each connected domain, and the first target distance is larger than the width of a frame line in the first image layer; performing inverse operation on the second frame layer to obtain a third frame layer; and obtaining a flow line layer according to the first graphic layer and the third graphic frame layer.

7. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to implement the method of any one of claims 1 to 5 when executing the instructions stored by the memory.

8. A non-transitory computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1 to 5.