JP6030915B2 - Image rearrangement method, image rearrangement system, and image rearrangement program - Google Patents

Description

  The present invention relates to an image rearrangement method, an image rearrangement system, and an image rearrangement program that cut out and arrange characters from a document image.

  As a method of extracting characters from a document image, there is a method of detecting a row / column from an image and specifying a character position for each row. In character segmentation, character recognition is generally performed (see, for example, Patent Document 1). However, character recognition requires a lot of processing time and has a problem that characters are replaced by recognition errors.

  In view of this, a technique has been studied in which characters are cut out from the image without recognizing the characters in the document image, and the characters are rearranged (reflowed) and displayed according to the size of the display area.

JP 2006-235817 A

  When document data is scanned into a document image, the document data may be scanned in a tilted state. In this case, if it is assumed that a column or row exists in the document image in the vertical or horizontal direction with the characters of the document image tilted and slanted, the column region or row cut out as a column is cut out. An extra area is included in the area. Therefore, when the characters cut out from the row area or the row area are rearranged and displayed as an image, there is a possibility that the letters will not be rattled or look bad. In the conventional rearrangement technique, the inclination of the document image is not taken into consideration.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image rearrangement method, an image rearrangement system, and an image rearrangement program for cutting out a row area or a column area in consideration of the inclination of a document image. Is to provide.

In order to achieve the above object, the present invention provides an image rearrangement method performed by a computer, an area extracting step for extracting a character area including a plurality of character rectangles from a read document image, and a predetermined number of the character areas. A foreground histogram is generated for each divided area, and the inclination estimation step for estimating the inclination of the image using the positional deviation of the histogram created in each divided area, and each character corresponding to the inclination The position of the rectangle is shifted to generate a histogram of the character area, the row area or the column area having the width of the histogram is extracted, and the height or width of the character rectangle included in the line area or the column area is and row and column acquisition step of bringing the width of the height or the column region region, the row region or from said line region, each character to fit the width of the height or the column region of the line region Performing a character segmentation step for cutting out shapes, the.

The present invention is an image rearrangement system, wherein a region extraction unit that extracts a character region including a plurality of character rectangles from a read document image, the character region is divided into a predetermined number, and the foreground is divided into each divided region. The character region is generated by shifting the position of each character rectangle by an amount corresponding to the inclination, and an inclination estimation means for estimating the inclination of the image using the shift of the position of the histogram created in each divided region. A histogram is generated, a row area or a column area having a width of the histogram is extracted, and the height or width of the character rectangle included in the row area or the column area is set as the height of the row area or the width of the column area. And a character cutout means for cutting out each character rectangle in accordance with the height of the row area or the width of the column area from the row area or the column area .

The present invention is an image rearrangement program executed by a computer, wherein an area extraction step of extracting a character area including a plurality of character rectangles from a document image read into the computer, and dividing the character area into a predetermined number Then, a foreground histogram is generated for each divided area, and an inclination estimation step for estimating the inclination of the image using a shift of the position of the histogram created in each divided area, and for each character rectangle by the inclination, A position is shifted to generate a histogram of the character region, a row region or a column region having a width of the histogram is extracted, and a height or width of a character rectangle included in the row region or the column region is set to the height of the row region. and row and column acquisition step of bringing the height or width of the column region, from the row area or the row region, tailored to the width of the height or the column region of the line region A character segmentation step of cutting out the character rectangle, is executed.

  According to the present invention, it is possible to provide an image rearrangement method, an image rearrangement system, and an image rearrangement program for cutting out a row area or a column area in consideration of the inclination of a document image.

It is a block diagram which shows the image rearrangement system which concerns on embodiment of this invention. It is a figure which shows the image of a reflow display. It is a figure which shows the relationship between a class and an object. It is a figure which shows the expression method of a rectangular coordinate. It is a flowchart of a reflow data generation process. It is an enlarged view of a shaded area. It is a figure which shows the result of a binarization process. It is a figure which shows integration of an initial labeling result. It is a figure which shows a caption area candidate and a histogram. It is a figure which shows creation of the histogram using a label. It is a figure which shows creation of the histogram by the scanning of a pixel. It is a figure which shows the setting of a caption area. It is a figure which shows discrimination | determination with a caption and a text. It is a figure which shows acquisition of a character area. It is a figure which shows the histogram of a character area. It is a figure for demonstrating inclination estimation. It is a figure for demonstrating acquisition of the column which considered inclination. This shows the rearrangement of the reading order of vertical writing and horizontal writing. It is a figure which shows integration of ruby. It is a figure which shows the example of shaping of a character space | interval. It is a figure which shows the shaping example of the character space | interval of the column end. It is a figure which shows the final cutting-out result. It is a figure which shows an example of reflow data. It is a flowchart of a reflow display process. It is a figure which shows the method of dividing reflow data into a separate volume. It is a figure which shows the arrangement | sequence image of a character and a chart.

  Hereinafter, embodiments of the present invention will be described.

  FIG. 1 is a diagram showing an overall configuration of an image rearrangement system according to an embodiment of the present invention. The illustrated image rearrangement system includes a layout analysis unit 1, a data storage unit 2, and a reflow (rearrangement) display unit 3. The layout analysis unit 1 reads an image (document image) such as scanned document data as an image, and generates reflow data for realizing comfortable reading on devices of various screen sizes. The data storage unit 2 stores the read image and the generated reflow data. The reflow display unit 3 uses the data stored in the data storage unit 2 to rearrange and display characters and diagrams cut out from the image according to the screen size. FIG. 2 is a diagram showing an image of reflow display.

  The layout analysis unit 1 shown in FIG. 1 includes an image reading unit 11 that reads data to be processed as an image, a binarization processing unit 12 that performs binarization processing on the read image, and a binarized image. Extracts a continuous area of pixels recognized as characters or charts in Fig. 1, a labeling unit 13 for labeling, a chart recognition unit 14 for extracting a region indicating a chart using a labeling result, and a character not recognized as a chart A character cutout unit 15 that recognizes a row or column from an area and cuts out character information, a layout conversion unit 16 that converts a layout such as making the entire page a chart area in a predetermined case, and the last page of the read image The barcode analysis unit 17 for recognizing the barcode and acquiring the ISBN code and the storage unit 18 for storing each data are provided.

  The illustrated reflow display unit 3 includes a rendering unit 31, a display 32, and an operation receiving unit 33 such as a touch panel. Using the image and reflow data stored in the data storage unit 2, the rendering unit 31 rearranges characters and diagrams cut out from the image according to the screen size and displays them on the display 32. For example, the user touches the operation reception unit 32 such as a touch panel formed on the display 32 to input instructions such as page turning and page movement to the reflow display unit 3.

  For the image rearrangement system described above, for example, a general-purpose computer system including a CPU, a memory, an external storage device such as an HDD, an input device, and an output device can be used. In this computer system, each function of the image rearrangement system is realized by the CPU executing a program for the image rearrangement system loaded on the memory. The program for the image rearrangement system can be stored in a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, a DVD-ROM, or can be distributed via a network.

  The layout analysis unit 1 may be mounted on a computer such as a PC or a server, and the reflow display unit 3 and the data storage unit 2 may be mounted on a mobile terminal such as a smartphone or a tablet. However, the present invention is not limited to this. For example, a single computer such as a portable terminal may implement the layout analysis unit 1, the data storage unit 2, and the reflow display unit 3.

  Next, the data structure of this embodiment will be described.

  FIG. 3 shows the relationship between main classes and objects. The IR Layout class 31 holds layout information relating to the entire page of the read image, and is associated with one or more IR Area class 32 data. For example, the IR Layout class 31 uses the binarized area, horizontal writing area, vertical writing area, chart area area and their sum (CHARH AREA, CHARV AREA, FIGURE AREA, SUM AREA), and Y used for character area acquisition. Holds direction histogram data, initial labeling result holding list (srcArea), character candidate list (charCandidate), character area list (charArea), chart area list (imageArea), and the like.

  The IR Area class 32 holds information regarding a plurality of areas present in the image. There are three types of areas: “vertical writing area”, “horizontal writing area”, and “chart area”. The “vertical writing area” and “horizontal writing area” are related to the data of the IR Column class 33 in a finer unit. The IR Area class 32 includes, for example, an area type (vertical writing, horizontal writing, chart), rectangular coordinate data of the area, an X-direction histogram in the area for vertical / horizontal writing determination of the area, and vertical / horizontal writing determination of the area. Holds the Y-direction histogram in the area and the row / column data contained in the area.

  The IR Column class 33 holds information related to “column” or “row” included in the “vertical writing area” and “horizontal writing area”. For example, the IR Column class 33 holds rectangular coordinate data of rows and columns, character rectangle data included in the rows and columns, and the like. The IR Column class 33 is associated with one or more IR Cordinate classes 34.

  The IR Cordinate class 34 holds the coordinate information of the “character rectangle” included in the “column” or “row”. The paragraph position is also managed. In addition, it is also used to manage the rectangular coordinates of "vertical writing", "horizontal writing", "chart" area and "row", "column". As shown in FIG. 4, the rectangular coordinates are managed using the width and height of the rectangle based on the coordinates at the upper left corner of the rectangle. For example, the IR Cordinate class 34 stores the type of character or paragraph, the upper left X coordinate of the character rectangle, the upper left Y coordinate of the character rectangle, the width of the character rectangle, the height of the character rectangle, and the like.

  In this embodiment, the document structure is hierarchically managed as described above. Further, these data are stored in the storage unit 18 of the layout analysis unit 1.

  Next, the processing of this embodiment will be described.

  The reflow data generation process and the reflow display process will be described below.

<Reflow data generation processing>
FIG. 5 is a flowchart of the reflow data generation process performed by the layout analysis unit 1. Here, a process from reading an image to generating reflow data will be described.

[S11: Read image]
First, the image reading unit 11 reads data to be processed (such as document data) as an image and stores it in the storage unit 18. In this embodiment, for example, OpenCV is used for primitive image processing, and OpenCV supports various image codecs, so it can support various formats (for example, JPEG format, PNG format, TIFF format, etc.). Shall. When reading, it is assumed to be forcibly read as a grayscale image because it is used in the binarization processing of the image. Then, the process from S12 to S17 is performed for each page (image file) of the read image.

[S12: Binarization processing of image]
The binarization processing unit 12 performs binarization processing on the image read in S11. The binarization process is a process of converting an image with shading (0 to 255 in the case of an 8-bit image) to be represented by a binary value of 0 or 1 according to a certain condition. Here, binarization is performed so that a part with a character or a diagram is recognized as 1, and a part without a character, that is, a background is recognized as 0. An algorithm for separating the character / chart portion (foreground) and background of the image by the binarization processing of the image will be described below.

(1) Preprocessing In order to obtain the optimal binarization result, preprocessing is important in addition to the binarization algorithm itself. In the present embodiment, two processes, “background adjustment” and “shaded area processing”, are performed as preprocessing.

Regarding background adjustment, binarization processing described later utilizes a discriminant analysis method to dynamically determine a threshold value and perform binarization. For this reason, even if it is a blank page, foreground may be detected due to a problem during scanning. Therefore, in the present embodiment, it is assumed that the luminance value of the background is sufficiently bright, and for example, for pixels having a luminance value lum [x, y] of a certain level or higher before binarization as shown in Equation 1. Then, correction processing for changing to the maximum luminance value is performed. In Equation 1, 230 is set as an example of the threshold value.

  With regard to the processing of the shaded area, many printed expressions can be seen in the printed document. In the case of monochrome printing, the shaded area is expressed by alternately repeating black and white dots as shown in FIG. 6, and the entire shaded area may not be detected as a foreground during binarization processing. .

Therefore, in the present embodiment, a shaded area can be detected as a foreground by applying a box filter and smoothing the image in advance. The box filter kernel uses n × n as shown in Equation 2, and the anchor performs processing with the kernel center. n is set to a predetermined value (for example, 3) according to the resolution of the screen.

(2) Binarization processing
Binarization is a process of dividing each pixel of an image into a background class and a foreground class. In the present embodiment, the threshold value is determined using the discriminant analysis method in the binarization process, but the binarization process may be performed using a method other than the discriminant analysis method. In the discriminant analysis method, a threshold value that maximizes the degree of separation T expressed by the ratio between the interclass variance ω ₀ and the intraclass variance ω _i is determined.

  FIG. 7 shows an image after the binarization processing. It can be seen that the chart and characters are recognized as the foreground (white). The background is expressed in black.

[S13: Labeling process]
The labeling unit 13 performs a labeling process in which a different label is attached to each component connected to pixels in the foreground detected by the binarization process in S12. That is, a rectangle surrounding the connected components is extracted as a labeling result. Ideally, one character or one chart is recognized as one connected component. For example, the letter “day” can be obtained as one connected component. The labeling result is stored in the storage unit 18 as an initial labeling result holding list (srcArea).

  However, it is not uncommon for Japanese characters to be divided into a plurality of connected components. For example, the word “present” is divided into two connected components. Also, since charts cannot always be acquired as one connected component, various integration processes are required. When one character is separated into a plurality of labels, the rectangles may overlap.

  Therefore, for example, by integrating the overlapping rectangles as shown in FIG. 8, the subsequent processing can be simplified and the processing speed can be increased. Here, it is assumed that the rectangle of label A and the rectangle of label B are represented by (xa, ya, wa, ha) and (xb, yb, wb, hb), respectively, before integration. As shown in FIG. 4, the rectangle of each label is represented by (the upper left x coordinate of the rectangle, the upper left y coordinate of the rectangle, the width of the rectangle, the height of the rectangle). In this case, the integrated rectangle of the label is converted as follows.

(Min (xa, xb), min (ya, yb), max (xa + wa, xb + wb)-min (xa, xb), max (ya + ha, yb + hb)-min (ya, yb) )
The labels are compared with each other from the labeling results stored in the initial labeling result holding list (srcArea), and if the rectangles overlap, the integration is performed in this way (one is deleted and the other is updated).

[S14: Chart Recognition Process]
The chart recognition unit 14 extracts a connected component indicating the chart using the labeling result integrated in S13, and divides it into a label representing the chart and other labels. In general, the extraction is performed by utilizing prior knowledge about the layout such as that the chart is larger than the character. In addition, since it is desirable to present the chart and the caption associated therewith to the user as a unit, the integrated processing of the connected components indicating the chart and the caption associated therewith is performed.

(1) Acquisition of a chart area Since a label representing a chart is much larger than a label representing a character, detection can be easily performed by providing a threshold value for the rectangular size. In the present embodiment, a label having a rectangular area size of 1% or more of the entire image area is recognized as a chart and stored in a chart area list (IR Layout :: imageArea) of the storage unit 18. A label that is not recognized as a chart is stored as a character candidate in a character candidate list (IR Layout :: charCandidate) in the storage unit 18.

(2) Integration of captions Charts and captions that describe them need to be integrated as a single chart area in order to maintain a list. Here, a process for integrating the detected chart and caption will be described. That is, when a label as a character candidate exists in the caption area candidates around the chart area, the label of the character candidate is determined as a caption, and the caption is integrated into the chart area. The caption integration process will be described below.

(A) Histogram Generation of Caption Area Candidates First, caption area candidates are set around the detected chart area. As the caption area candidate, a range of equidistant L from the four sides of the chart area 70 is set as shown in FIG. The upper end of the chart area is the Top area 71, the lower end is the Bottom area 72, the left end is the Left area 73, and the right end is the Right area 74. The distance L may be set to L = min (width, height) using, for example, a parameter representing a chart area. That is, it may be the smaller value of the width (width) and height (height) of the target chart area.

  In the present embodiment, a foreground histogram is calculated using the set four caption area candidates. In the Top area 71 and Bottom area 72, a histogram in the X direction is calculated, and in the Left area 73 and Right area 74, a histogram in the Y direction is calculated.

  As shown in FIG. 9, histogram peaks 75 and 76 are formed in the Right region 74 and the Bottom region 72 with captions (foreground). In the Top area 71 and the Left area 73 where there is no caption, the histogram is flat.

  A histogram is created by calculating a value that is an index of the foreground (character, character rectangle, etc.) existing on each coordinate in the X direction or Y direction of the image. In the present embodiment, as shown in FIG. 10, the histogram is calculated by using the character candidate list (IR Layout :: charCandidate) in the storage unit 18 to calculate the number of labels recognized as character candidates in the X direction or Count in the Y direction and calculate simply. As shown in FIG. 10, when three character rectangles exist in the character area, the number of labels present on each coordinate in the X direction or the Y direction is acquired.

Specifically, for the area r of each label in the character area,
When creating a histogram in the X direction, 1 is added to hx (i) for all i satisfying the left end ≦ i ≦ right end of the region r of the label,
When creating a histogram in the Y direction, 1 is added to hx (j) for all j satisfying the upper end ≦ i ≦ lower end of the region r of the label.

  Thereby, in this embodiment, high-speed calculation is realized without reducing accuracy.

  Note that the histogram may be created by scanning the pixels of the image of the character area as shown in FIG.

(B) Setting of caption area and update of area After the calculation of the histogram is completed, the caption area is set. As shown in the Bottom area of FIG. 12, the run length of the histogram is calculated, and the run end 82 of the outermost histogram 81 is expanded as a caption area 83.

  At this time, although the histogram 85 exists as in the Top area 84 of FIG. 12, if the run of the histogram 85 exists beyond the caption area candidate 84 (when the run does not converge within the caption area candidate), the caption is displayed. Therefore, the chart area is not expanded.

  In the example illustrated in FIG. 13, a label (character candidate) of a text that is not a caption exists in the right area of the caption area candidate. In this case, unlike the Top region 84 in FIG. 12, the label gap (line spacing) of the text may coincide with (overlap) the boundary 91 of the caption area candidate. At this time, a histogram peak is created, and it is determined that the histogram run does not exist beyond the caption area candidate (the run has converged within the caption area candidate), and the label of the text is erroneously detected as a caption. There is. For this reason, in the present embodiment, it is checked for each run of the histogram whether there is a label overlapping the boundary 92 of the caption area candidate. If there is a label, it is determined that the area belonging to the run is not a caption. In the case of FIG. 13, since the label 93 overlaps the boundary 92, it is determined that the region belonging to the run 95 is not a caption. In this process, the gap between characters may coincide with the boundary 92. Therefore, it is also checked whether or not there is a label recognized as a character candidate in a certain range 94 outside the boundary 92. If there is a label, it is determined that the area belonging to the run is not a caption. In FIG. 13, the character line of the line corresponding to the run 96 matches the boundary 92, but since the label is included in the range 94, it is determined that the area belonging to the run 96 is not a caption. If neither a label that overlaps the boundary nor a label in a certain range outside the boundary exists, it is determined as a caption.

  After setting the caption area as described above, the area existing in the caption area in the character candidate area stored in the character candidate list (IR Layout :: charCandidate) in the storage unit 18 is deleted. The chart area is enlarged and updated by the caption area. Also, the sum of the areas of the obtained chart regions is calculated and stored in the area of the chart region (IR Layout :: FIGURE AREA) (used for later layout conversion).

  As described above, in S14, the labeling result in S13 is divided into a chart area list (IR Layout :: imageArea) and a character candidate list (IR Layout :: charCandidate).

[S15: Cutting out character information]
The character cutout unit 15 uses the character candidate list (IR Layout :: charCandidate) in the storage unit 18 to perform a process of cutting out one character and one character rectangle.

  That is, it is thought that the label of the character candidate which was not recognized as a chart and a caption constitutes a character. First, the character area is defined as a rectangle from the distribution in the image of the labels constituting these characters. Next, it is inferred from the distribution of labels in the character area whether the character is written vertically or horizontally in the defined character area. After vertical / horizontal writing can be discriminated within each defined character area, columns are recognized for vertical writing and rows for horizontal writing, and one character and one character are cut out at the end. Hereinafter, these processes will be described.

(1) Acquisition of character area First, a character area is acquired using data of a character candidate list (IR Layout :: charCandidate). Assuming that a general book is a target, it is possible to divide the document area in the vertical direction. For example, it can be divided into a column area (an area where chapter titles are written), a body area, and a noble area (area where page numbers are written) from the top of the document and considered as a separate area. Therefore, a foreground histogram in the Y direction is created using a character candidate list (IR Layout :: charCandidate), and the area is divided vertically by calculating this run length.

  FIG. 14 shows a vertical division method. It can be seen that in the image 101, the column area 102, the body area 103, and the folio area 104 can be recognized as different areas. Even in the case of a page containing a chart, since the histogram is created using the character candidate list (IR Layout :: charCandidate), the chart portion 105 can be ignored and this process can be performed.

  In addition, even though the area is the same, it may be divided into a plurality of areas due to coincidence of character breaks. In order to prevent this, if adjacent areas are sufficiently close (for example, within 30 pixels), the areas should be integrated.

The character area is determined by obtaining the minimum value and maximum value in the X direction of the rectangular area of the character candidate list (IR Layout :: charCandidate) included in the finally divided area. In particular,
Upper end of character candidate ≧ upper end of divided character area,
And the lower end of the character candidate ≦ the lower end of the divided character area,
Among all the character candidates satisfying the above, the minimum left end and the maximum right end are set as the left end and the right end of the character area, respectively. The acquired character area is stored in a character area list (IR Layout :: charArea) in the storage unit 18.

(2) Estimating character direction In the acquired character area, it is determined whether characters are written in vertical writing or horizontal writing. First, as shown in FIG. 15, histograms in the X direction and the Y direction are created in the acquired character area. After creating the histogram, calculate the average of the white run (the length of the continuous foreground histogram) and the black run (the length of the continuous background histogram) in both the X and Y directions. However, the foreground histogram and the background histogram are a portion where the histogram value is greater than 0 and a portion where the histogram value is 0, respectively.

  Here, the average of white runs in the X direction is xwrun, the average of black runs is xbrun, the average of white runs in the Y direction is ywrun, and the average of black runs is ybrun.

  For example, as shown in FIG. 15, when the character area is vertically written, ybrun is almost eliminated. This is because when vertical writing is projected in the Y direction, almost no parts appear. Conversely, xbrun has a constant value. This is because spaces without characters are regularly generated between lines. That is, when xbrun> ybrun, it is basically determined as vertical writing, otherwise it is determined as horizontal writing.

  However, there are exceptions to this condition. That is the case of horizontal writing with only one line. In the case of a horizontal writing area with multiple lines, there is no problem with the above conditions. However, if there is only one line, ybrun is almost zero and xbrun has a constant value, so it is determined as vertical writing. For example, a column area or a noble area is still determined as a vertical writing area.

  Therefore, in order to avoid this, when the following condition 1 and condition 2 are satisfied, it is recognized that the line is horizontal writing, and horizontal writing is determined even if xbrun> ybrun.

Condition 1: The height heigt of the character area and ybrun are about the same size. For example, when the following expression is satisfied, it is determined that they are approximately the same.

Condition 2: xwrun and ywrun are not so far apart. For example, when the following expression is satisfied, it is determined that the distance is not so far.

  By adding the exception conditions of condition 1 and condition 2, it is possible to correctly recognize vertical writing and horizontal writing of the character area. The determined vertical writing / horizontal writing results are stored in the IR Area area type of the storage unit 18.

(3) Inclination estimation Once vertical / horizontal writing of a character area has been determined, the column is acquired using the histogram in the X direction for vertical writing, and the row is acquired using the histogram in the Y direction for horizontal writing. I do. Hereinafter, for the sake of simplicity, a case where the character area is vertically written will be described.

  The largest character area among the acquired character areas is specified, and the character area is divided into n. If it is a vertically written character area, it is divided horizontally (divided by lines in the X direction). If it is a horizontally written character area, it is divided vertically (divided by lines in the Y direction). It should be noted that the character area is preferably divided equally, but if it is not equally divided, a division ratio or the like may be used.

FIG. 16 is a diagram for explaining inclination estimation. Here, a case where the character region 121 is divided into two equal parts in the vertical direction will be described as an example. First, a foreground histogram in the X direction is created in each of the divided regions 122 and 123. That is, the histogram hu (x) of the upper region 122 and the histogram hd (x) of the lower region 123 are created. The histogram is created by the method shown in FIG. 10 or FIG. When the image read in S11 is tilted, the positions of the histogram runs (mountains) of the divided regions 122 and 123 are shifted according to the tilt. Therefore, x (the amount to be shifted to correct the inclination) that maximizes the following c (x) is calculated.

  Note that bu (i) binarizes the histogram hu (x) of the upper region 122 and the histogram hu (i) is greater than 0 (when at least one character candidate label exists). Sets “+1”, otherwise “-1” is set (if there is no character candidate label). Note that i is an x coordinate. bd (i) binarizes the histogram hd (i) of the lower region 123 and is the same as bu (i).

  Then, using the calculated x, the inclination s of the image is estimated by the following equation.

Slope s = x ÷ (height of character area ÷ 2)
In the example shown in FIG. 16, since it is an example of dividing into two equal parts, in the above formula, the height of the character area is divided by 2, but the distance between the divided areas is set according to the number of divisions. That is, when two of the divided areas are selected,
Inclination s = x ÷ (| y coordinate of the center of one area -y coordinate of the center of the other area | ÷ 2)
And In addition, when the number of divisions is 3 or more, there are a plurality of selection methods for the divided regions. For example, two of the most distant y coordinates may be selected, or two of the y coordinates may be selected from the smaller one. Also good. In general, the former is used because the accuracy is higher when the distance between the divided areas is larger. However, the latter is preferred when there are many lines in which characters do not continue below the character area, such as character areas with many lines in novels. May be good. Further, a plurality of combinations of divided areas may be taken to calculate s for each, and an average value or median value thereof may be taken. The plurality of combinations may be all combinations, combinations of adjacent divided areas, or other combinations.

  In this embodiment, the inclination of the entire page of the read image is estimated using the character area having the largest area in the page, but the inclination is estimated by the above method for each character area in the page. The inclination for each character area may be held. When estimating the inclination of the entire page, by using the largest character area, it is possible to avoid the inclination estimation failure in other small character areas and to shorten the processing time. On the other hand, when estimating the inclination for each character area, when the image has a geometric distortion (for example, when the scan is bent), the inclination is estimated with high accuracy for each character area. Can do.

(4) Row / column acquisition After vertical / horizontal writing of the character area is determined and the inclination is estimated, the column is acquired using the histogram in the X direction for vertical writing, and the Y direction for horizontal writing. Use the histogram to get the rows. Here, a case where the character area is vertically written will be described.

  First, the column width is obtained by detecting the run length of the foreground of the histogram. At this time, if it is extremely small (3 pixels or less), it is ignored as a false detection.

  Here, a histogram in consideration of the estimated slope s is recreated. FIG. 17A shows an example of a histogram recreated for a tilted character area. In the illustrated example, the histogram is created by shifting the addition position of the character candidate labels by the estimated inclination s. Specifically, for each label region r, 1 is added to h (i + s * (upper end of region r)) for all i satisfying the left end ≦ i ≦ right end of the region r. As a result, a histogram 131 is created by shifting the addition position of the region r by the inclination s.

Then, as shown in FIG. 17B, the region r of each label included in the peak range of the created histogram is included in the column, and the width of the region r of each label included in the column is calculated as the histogram run. Update to match the length. Specifically, for the area r of each label,
When the left end of the region r ≦ the left end of the histogram + s * the upper end of the region r and the right end of the region r ≦ the right end of the histogram + s * the upper end of the region r, the following processing is performed.

  The region r is included in the column, the left end of the region r is set to “left edge of the histogram−s * the upper end of the region r”, and the width of the region r is the same as the run length of the histogram. Thereby, the width of the region r of each label becomes equal to the column width, and the character candidates stored in the same column can be easily arranged at the time of rendering.

  Next, the column height is obtained using the minimum and maximum values in the Y direction of the data of the character candidate list (IR Layout :: charCandidate) included in the acquired column width range, and the column area list ( (CharArea.column), the rectangular information representing the column and the data of the character candidate list (IR Layout :: charCandidate) included in the column are stored in the storage unit 18.

  In the present embodiment, the inclination is estimated in this way, and a histogram is generated by shifting by the estimated inclination. Thereby, the column area or the row area of the original image can be correctly cut out without including an extra area in the row area cut out as a column or the row area cut out as a row. If the document image is tilted and processed as if there were columns or rows in the vertical or horizontal direction of the document image, the run length of the created histogram is larger than the original column width or row height. As a result, an extra area is included in the extracted column area or row area. In the present embodiment, by estimating the inclination, it can be avoided that an extra region is included in the extracted column region or row region.

  Next, the character candidates, row / column regions, and character regions are rearranged in the reading order. In preparation for the last data output or character segmentation, it is necessary to rearrange the list acquired so far in the reading order. FIG. 18 shows the rearrangement of the reading order in the case of vertical writing and in the case of horizontal writing. In FIG. 18, the numbers indicate the reading order. As shown in the figure, the candidate characters included in the row / column are rearranged in the order of the parameter y decreasing in the case of a column and in the order of decreasing parameter x in the case of a row. The list in the row / column area is sorted in the order of increasing parameter x in the case of columns, and in decreasing order of parameter y in the case of rows.

  The character area list first obtains the column area (IR Layout :: CHARV AREA) and line area (IR Layout :: CHARH AREA) of the entire screen, and determines whether the entire screen is in a vertical writing configuration or a horizontal writing configuration. That is, it is determined that IR Layout :: CHARV AREA> IR Layout :: CHARH AREA is vertical writing and horizontal writing is otherwise.

  If it is determined that the vertical writing is made based on the determination result, if the Y directions of the regions overlap, the parameter x is sorted in the descending order, and in other cases, the parameter y is sorted in the ascending order. If it is determined that the writing is horizontal writing, if the Y direction of the areas overlap, the parameter x is sorted in ascending order, otherwise the parameter y is sorted in ascending order.

(5) Integration of ruby Some of the rows and columns acquired so far include ruby. If the ruby is reflowed and displayed alone, the meaning will not be understood, so ruby will be integrated with the target character rectangle.

  As a feature of ruby, the distance from the character rectangle to which ruby is attached is short, and ruby has a smaller column width or row height than the target character rectangle. Ruby is integrated using these two features. FIG. 19 shows ruby integration processing.

  Here, for the sake of simplicity, the case of vertical writing will be described. In this case, ruby is attached to the right side of the character. If the right column is Cr and the left column is Cl 2, the distance between columns is l = Cr. Calculate x-(Cl.x + Cl.width). This distance l is the column width Cl. If it is less than half of width, it is determined that the distance is sufficiently close. Further, when the column width ratio (Cr.Width / Cl.width) is 0.5 or less, it is determined that the column width is smaller than the adjacent column width. It should be noted that the threshold values used in the determination (the column width is “half” or less and the column width ratio is “0.5” or less) are merely examples, and other values may be used.

  If it is determined as a ruby string by these two determinations, the character rectangle that overlaps the ruby string in the Y direction is extended to the right end of the ruby string. Also, in order to facilitate character centering during rendering, the left end is expanded as much as the portion extended to the right end here.

(6) Character segmentation Characters that make up Japanese (Kanji, Hiragana) are not connected to one character, so the rectangle of candidate characters obtained by the previous processing is one It often happens that characters are divided into multiple rectangles. Therefore, the rectangles of character candidates that can be estimated to be one character are integrated, and line breaks are recognized and character spacing is shaped to make it easier to see the reflow arrangement during rendering.

(a) Integration of character rectangles By the processing so far, character rectangles belong to rows and columns, and in the case of rows, the order of arrangement is the smallest in the X direction, the height of the rectangles is equal to the height of the rows, and the columns In the case of, the order of arrangement is the smallest in the Y direction, and the width of the rectangle is equal to the column width (except for the rectangle integrated with ruby). That is, there is no need to consider integration in the Y direction for rows and X direction for columns, and it is only necessary to determine whether or not the arranged rectangles should be integrated in order. We used the characteristics of Japanese characters and the fact that the width and height of the characters are the same as the rectangle integration rule.

  Hereinafter, for the sake of simplicity, the character rectangle integration (vertical writing) in the column will be described. This is realized by determining the integration of the rectangles before and after the arrangement order of the character rectangles for each column. First, if the front and rear rectangles partially overlap, unification is performed unconditionally. The overlapping of the rectangles has already been performed in the labeling process of S13. For example, the character “I” does not overlap the left part and the right part in the initial labeling result. Since the column width is updated, new rectangles may overlap.

Next, when the front and rear rectangles do not overlap, the rectangle height heightn when the rectangles are merged is calculated from the following equation.

Now, when the column width is column_width, integration is performed when the following formula is satisfied. Note that “1.0” on the right side of the following expression is an example, and other values may be used.

  Also, if the preceding and following character rectangles are rectangles in which ruby is integrated, integration is performed. This is forbidden on the viewer side, and even if a character with a series of ruby characters consists of multiple characters, it is cut out as a single rectangle. Note that it is possible to determine whether or not the rectangle is a combination of ruby because the column width is different from the rectangle width.

(b) Recognizing line breaks When the rendering unit 31 of the reflow display unit 3 recognizes a line break when performing reflow, it can properly insert a line break and contribute to readability. Recognize.

When integrating characters for each row / column, it recognizes whether a line break exists when the last character rectangle contained in the row / column is reached. For the sake of simplicity, vertical writing will be described. If there is a distance between the lower end in the Y direction of the last character rectangle, that is, the lower end in the Y direction of the column, it is recognized that there is a line break. Specifically, a line break is considered when there is a distance of n _d times or more (eg, 0.7 times or more) of the column width.

Furthermore, if there is a large margin between the next character string after a line break (for example, between the heading and the body), it is assumed that there is another line break. In this case, line feed data is added twice. Defining a large margin, the distance between the next column is greater margin than n _s times the column width (e.g. 1-fold).

(c) Formatting of character spacing Since the rectangle obtained by the processing so far is a rectangle circumscribing the character, the margin between characters has been cut off. If the reflow is performed in the rendering unit 31 as it is, characters are clogged. Therefore, the white space between characters is distributed to consecutive character rectangles. The margins do not exist evenly, but change according to the size of the front and rear character rectangles, so they are allocated in consideration.

  FIG. 20 is a diagram illustrating an example of a character spacing shaping method. For the sake of simplicity, the description will be focused on the vertical writing area. Here, let us consider a character rectangle that moves back and forth in the arrangement order. Assume that the ratio of height heighti and heighti + 1 of consecutive rectangles is heighti: heighti + 1 = A: B. At this time, the margins of the character rectangle i and the character rectangle i + 1 are allocated with an inverse ratio B: A. In the case of horizontal writing, the character width width may be used as well.

  It should be noted that the above margin distribution method cannot be used for the character rectangle at the end of the column (first and last). Therefore, as shown in FIG. 21, when the height of the character at the end of the column is small, the white space is not allocated and the height of the character rectangle remains small, and the character spacing of this portion becomes extremely large when performing reflow. Clogged. In order to prevent this, when the height of the character rectangle at the end of the column is a predetermined value (for example, column width × 0.8 to 1.2 or less), the height of the character rectangle is expanded to a predetermined value. When the height of the character rectangle is expanded to a predetermined value, the upper end of the character rectangle should not protrude from the character area for the first character rectangle, and the lower end of the character rectangle for the last character rectangle. Do not protrude from the character area.

  FIG. 22 displays an example of final cutout results such as a chart area and a character rectangle. A blank rectangle below the character string indicates a line feed.

[S16: Layout conversion]
Reflow data can be created based on the result of cutting out information on characters and charts so far, but it may not function properly depending on the layout of the target image. Therefore, the layout conversion unit 16 considers the readability of the reader, and if the analyzed result is under a certain condition, the layout conversion unit 16 displays the analyzed layout result such as making the entire page of the read image a chart area. Perform the change process.

(1) Conversion to image page In the case of a page that needs to be listed, or in a situation where the obtained layout analysis result has clearly failed, the page is a single image rather than forcibly generating reflow data. Is preferable from the viewpoint of the reader. Specifically, in the following case, conversion is performed using the read image as an image page and the entire page as a chart area.

(a) Pages with a chart as the main subject Since a document whose subject is a text is assumed as the processing target, if the area of the chart area is large compared to the area of the text area on the page (such as a cover page), a list of charts In order to ensure the property, conversion is performed as an image page.

  Specifically, the area of each area (vertical writing area (CHARH AREA), horizontal writing area (CHARV AREA), chart area (FIGURE AREA)) obtained in the process of layout analysis is utilized. If the area is larger than n times (for example, 1.5 times) the area of the character area, the chart is recognized as being the main subject and converted to an image page.

  In the present embodiment, CHARH AREA and CHARV AREA indicating the area of the character area are not the area of the so-called character area, but are the sum of the areas of the column portions in the character area, for example, in the case of vertical writing. (The blank area is not considered as the area of the area). Of course, the area of the character area itself including the blank area may be used.

(b) Table of contents page It is difficult to accurately recognize and cut out the table of contents page because the listability is important and the characters are often decorated. Therefore, if it can be recognized as a table of contents, it is converted as an image page.

When determining the table of contents, the table of contents page appears in the first half of the book, and the headings are continuous, so the line spacing is wide (the line breaks twice). Therefore, when the page number i is i <50 and the number of rows / columns COL_NUM existing on the page and the number of line breaks BR_NUM are measured, and the following formula is satisfied, it is recognized as a table of contents page and converted to an image page. . Note that the thresholds “50” and “0.7” used for the determination of this page are merely examples, and other values may be used.

(c) Pages with an extremely small detection area Pages with very few areas detected as character areas or chart areas (for example, pages where only chapter titles are written) are often important for listing. Convert to. Specifically, the sum of the areas of each region (SUM_AREA = CHARH_AREA + CHARV_AREA + FIGURE_AREA) obtained by layout analysis is n% (for example, 5 %) In the following cases, convert to an image page.

(2) Deletion of nobles and pillars Since the concept of “pages” in paper books disappears when reflow data is used, the noble area where page numbers are written becomes unnecessary data, and chapters called pillars, etc. Since the area written in is also information given to each page, it is not used for generating reflow data.

  Therefore, after the layout analysis, the data corresponding to the noble / column is deleted. The recognition of the noble and the pillar is performed using the position of the character area. That is, a character area existing within a predetermined range (for example, from the top and bottom edges of the image to n% (for example, 20%) of the height of the image) is recognized as a bumble or a pillar. In addition, when the character area exists in the predetermined range and the row or column included in the character area is a predetermined value (for example, 1 to 3) or less, it is recognized as a noble or a pillar. Also good.

  Further, when the character area exists in the predetermined range and the width of the character area / the width of the image is equal to or less than a predetermined value, the character area may be recognized as a noble or a pillar.

[S17: Output temporary result]
After the processing of S <b> 16, the layout conversion unit 16 outputs the processing result for one page of the processing target image to the storage unit 18. Note that the reflow data finally output in S20 has a one-to-one correspondence with the images of all pages read in S11, but by outputting temporary results in S17, Since it is not necessary to perform sequential processing from the image file of the first page to the image file of the last page, it is possible to improve the processing speed by parallel calculation.

  The processes from S12 to S17 described above are performed for each page of the read image. For this reason, if there is an unprocessed page (S18: NO), the process returns to S12 and the subsequent processes are performed. If all pages have been processed (S18: YES), the process proceeds to S19.

[S19: Barcode analysis]
The back cover of the book has a barcode, and the ISBN code, which is the book identifier, is acquired by recognizing the barcode. If the ISBN code can be acquired, the metadata of the book can be acquired in the same manner as the metadata can be acquired from the CDDB of the music CD, which can be used for managing the content of the image read in S11. Therefore, the barcode analysis unit 17 detects a barcode in an image of a predetermined page, determines the code type from the detected barcode, and acquires only the information of ISBN-13 or ISBN-10. In this embodiment, it is assumed that the spine is basically present on the last page, and this processing is performed only on the last page. However, not only the last page but also several pages from the last page (about 10 pages or less), for example. Alternatively, this process may be performed.

  In this embodiment, the ZBar library is used for bar code analysis. The ZBar library provides image input and output using ImageMagick, but it can be operated simply by passing the memory of a grayscale image, and can be replaced by another image reading library, so layout analysis has already been performed. Process by passing the loaded image format instead.

[S20: Output final result]
After the character information and charts are cut out for all pages of the image read in S11, the temporary result files output in S17 are integrated and output to the storage unit 18 as one reflow data. In this embodiment, the reflow data is output in the JSON format so that it can be used by a rendering engine such as the rendering unit 31 of the reflow display unit 3. JSON data is defined as one JSON file for one content, including the analysis results of all pages. In this embodiment, parallel processing is performed for speeding up, and processing is not performed sequentially from the top to the bottom. Therefore, the processing result for each page is output as a temporary result in S17, and is integrated at the end. Create the final JSON file.

  FIG. 23 shows an example of reflow data.

  The reflow data is set for each character area and object area of the chart. Specifically, it includes the type of the object area, the page ID of the read image, the upper left coordinates (x, y) of the object, the width of the object area, the height of the object, and the like. As the type of the object area, for example, 0: character, 1: chart, 2: line feed can be considered. Other types may include page breaks, chapter breaks, vertical writing characters, horizontal writing characters, and the like. When the object area type is a line feed, the subsequent elements are omitted and output. Note that the page ID of the image and the coordinates (x, y) are used to specify in which position of which page of the read image the object is located.

  For each page of the read image, characters and line feeds are output in the reading order, finally the chart area is output, and finally all pages are integrated to generate one reflow data, which is stored in the storage unit 18. .

  Next, a reflow display process using the read image (hereinafter referred to as an original image) and the generated reflow data will be described.

<Reflow display processing>
FIG. 24 is a flowchart of the reflow display process performed by the reflow display unit 3 mounted on a smartphone, a tablet terminal, or the like.

  In S21, the rendering unit 31 reads the reflow data generated by the reflow data generation process and the original image. The reflow display unit 3 acquires the original image and the reflow data stored in the storage unit 18 of the layout analysis unit 1 via, for example, a network and stores them in the data storage unit 2.

  In S <b> 22, the rendering unit 31 generates a separate volume obtained by dividing a set of read reflow data objects at a predetermined position. That is, a set of objects is divided based on a predetermined number of objects. As shown in FIG. 25, two methods can be considered for creating a separate volume. The first method creates a separate volume by dividing the reflow data by a predetermined number of objects. In the example shown in FIG. 25A, the reflow data is divided every 1000 objects. That is, the objects from 1 to 1000 are referred to as volume 1 and the objects from 1001 to 2000 are as volume 2.

  In the second method, a separate volume is created by dividing a predetermined number of objects or more at a location where an object with good separation is detected. Line breaks, charts, etc. can be considered as objects with good separation. In the example shown in FIG. 25 (b), the first separated object is detected after 1000 objects, and the first part is a volume 1 where the first separated object is detected. Then, after the separation of the volume 1, the first separated object after the 1000 objects (the diagram in the example shown) is separated at the location where the object is detected, and the volume up to the object is referred to as volume 2.

  In S <b> 23, the rendering unit 31 extracts each object included in the user's operation target volume from the original image using the reflow data, and creates the arrangement area generated according to the size of the display area (screen) of the display 32. , Rearrange objects sequentially. Then, a reflow page is generated by dividing (dividing) the rearrangement area into a size equal to the display area of the display. FIG. 26 shows an arrangement / arrangement image of characters and charts. In the case of vertical writing, the size of the area in which the object is arranged is, for example, the height equal to the height of the display area, and the width is unlimited. After all the characters of the separate volume are arranged in this arrangement area, the arrangement area is divided every length equal to the width of the display area, and one division is defined as one reflow page.

  In this embodiment, CSS3 (http://www.w3.org/TR/jlreq/ja/) multilingual support (CSS3 Writing Modes Module) is used to convert each character object into Japanese at a predetermined magnification. It shall be arranged vertically according to the typesetting requirements. Also, by specifying display: inline-block in CSS2, the object area cut out from the image can be handled as a character.

  As shown in FIG. 24, when the type of the object is a line feed, the line is unconditionally broken, and the subsequent objects are broken into the next line. In addition, the CSS3 multi-column layout module is used to divide the text into reflow pages. In addition, the object of the chart is one row and column, and the enlargement ratio is fixed in principle. However, if it does not fit within the screen, the enlargement ratio is adjusted to a size that fits.

  In S <b> 24, the rendering unit 31 displays on the display 32 the reflow page of the volume arranged in S <b> 23 that is noticed by the user.

  In S25, the operation reception unit 33 receives a user operation (such as touching the touch panel). If the user operation is a page movement operation such as page turning or jump (S26: YES), and the reflow page to be moved is in the same volume as the volume processed in S23 (S27: NO), S23 The reflow page of the movement destination in the reflow page generated in (1) is displayed (S24). If the destination reflow page is in another volume (S27: YES), the process proceeds to S23, where all the reflow pages in which the objects included in the destination volume are arranged are generated, and then specified by the user operation. The reflow page of the moved destination is displayed (S24).

  In the case of page turning, in the same volume, a reflow page that is simply one before or after the currently displayed reflow page is displayed. When the booklet moves beyond the end of the booklet and moves forward or backward, the destination reflow page is displayed after all the objects of the booklet to be moved are placed on the reflow page in S23.

  In the jump operation, the destination page is designated by the page number of the original image. As the movement destination, the reflow page including the first object of the designated original image page is displayed as the movement destination. If the destination is a different volume, the destination reflow page is displayed after all the objects in the destination volume are placed on the reflow page in S23.

  When the user operation is an enlargement / reduction operation (S28), the operation is changed to the operated enlargement ratio (S29), and the process proceeds to S23. Then, the reflow page after relocation including the first object of the reflow page before operation is displayed (S24).

  When the user operation is a change of the display area size (S30), the display area size is changed to the changed display area (S31), and the process proceeds to S23 to change the reflow page of the display area size after the change of the currently displayed volume object. After that, the reflow page after the rearrangement including the first object of the reflow page before the operation is displayed (S24). For example, the display area size can be changed by changing the orientation of the display 32 from portrait to landscape, expanding the display area size of the display 32, or reducing the display area size.

  Note that the user operation may include a switching operation between the reflow page display and the original image page display. For example, the original image page including the first object of the currently displayed reflow page is displayed in accordance with the display area of the display 32, or the first object included in the currently displayed original image page is displayed. It is conceivable to display a reflow page including the page.

  In the present embodiment described above, reflow data composed of rectangular coordinates enclosing each character and chart constituting the document image is generated, and the character image is reflowed using the reflow data according to the display size on the display side. Thus, comfortable reading can be provided to the user with various display sizes.

  In this embodiment, the inclination of the read image is estimated, and a histogram is generated by shifting the estimated inclination. Thereby, the column area or the row area of the original image can be correctly cut out without including an extra area in the row area cut out as a column or the row area cut out as a row. That is, when a character cut out from a column area or a line area is reflowed and displayed as an image, the rattling of the character is avoided, and a good-looking and comfortable reading can be provided to the user.

  Further, in the present embodiment, since the chart and its caption are integrated and handled as one chart area, the problem that the chart and the caption are displayed separately during reflow display can be solved.

  In the present embodiment, by creating a histogram in the X direction and Y direction of the character area, it is possible to correctly determine whether the character area is vertically or horizontally written.

  Moreover, in this embodiment, a character space | interval can be shape | molded by allocating the margin of a continuous character rectangle to the said continuous character rectangle.

  In the present embodiment, ruby and the target character are integrated and handled as one character rectangle, so that the ruby and the target character are displayed separately during reflow display. Can be resolved.

  Further, in this embodiment, by recognizing a line break in the column area or the line area of the read image, the line break portion in the read image can be maintained even in the reflow display. As a result, it is possible to avoid the occurrence of rattling when the line break portion in the read image is no longer a line break in the reflow display.

  In the present embodiment, the reflow of objects is performed in units of separate volumes obtained by dividing the generated reflow data by a predetermined number of objects. This eliminates the need to process many objects at the time of reflow display, thereby reducing resources such as processing time and the amount of memory used. In addition, it is possible to prevent degradation of the operation response and improve user convenience.

  In addition, this invention is not limited to the said embodiment, Many deformation | transformation are possible within the range of the summary.

1: layout analysis unit 11: image reading unit 12: binarization processing unit 13: labeling unit 14: chart recognition unit 15: character segmentation unit 16: layout conversion unit 17: barcode analysis unit 18: storage unit 2: data storage Unit 3: Reflow display unit 31: Rendering unit 32: Display 33: Operation receiving unit

Claims (8)

An image rearrangement method performed by a computer,
An area extraction step for extracting a character area including a plurality of character rectangles from the read document image;
An inclination estimation step of dividing the character area into a predetermined number, generating a foreground histogram for each divided area, and estimating an inclination of the image using a shift in the position of the histogram created in each divided area;
The position of each character rectangle is shifted by the inclination, a histogram of the character area is generated, a row area or a column area having a width of the histogram is extracted, and the height of the character rectangle included in the line area or the column area is extracted. A row / column acquisition step for adjusting the height or width to the height of the row region or the width of the column region;
A character cutout step of cutting out each character rectangle in accordance with the height of the row region or the width of the column region from the row region or the column region . The image rearrangement method according to claim 1,
The area extraction step extracts a chart area including a chart from the document image,
When a character rectangle exists in a caption candidate area around the chart area, the image rearrangement further includes a caption integration step of determining the character rectangle as a caption and integrating the caption into the chart area. Method. The image rearrangement method according to claim 1 or 2,
When the histogram of the X direction and the Y direction of the character area is generated and the average of the continuous length of the background histogram of the X direction is larger than the average of the continuous length of the background histogram of the Y direction, An image rearrangement method, further comprising: a character direction estimation step that determines that there is a character and that the other character is written horizontally. The image rearrangement method according to any one of claims 1 to 3,
The spacing between adjacent rows or columns is less than or equal to a specified percentage of the larger row height or column width, and the ratio of adjacent row height or column width is less than or equal to a specified percentage The image rearrangement further comprises a ruby integration step of determining that a row having a smaller height or a column having a smaller width is ruby and integrating the ruby into a character rectangle. Method. An image rearrangement method according to any one of claims 1 to 4, comprising:
An image rearrangement method characterized by allocating a space of continuous character rectangles to the character rectangles. An image rearrangement method according to any one of claims 1 to 5,
A division step of dividing a set of objects of a character rectangle or a chart area cut out from the document image on the basis of a predetermined number of objects;
A rearrangement step of rearranging the object in a rearrangement area generated according to the size of the display area of the display in divided units;
A display step of displaying on the display a reflow page obtained by dividing the rearrangement area according to the display area of the display. An image rearrangement system,
Area extraction means for extracting a character area including a plurality of character rectangles from the read document image;
Inclination estimating means for dividing the character region into a predetermined number, generating a foreground histogram for each divided region, and estimating the inclination of the image using a shift in the position of the histogram created in each divided region;
The position of each character rectangle is shifted by the inclination, a histogram of the character area is generated, a row area or a column area having a width of the histogram is extracted, and the height of the character rectangle included in the line area or the column area is extracted. Row / column acquisition means for adjusting the height or width to the height of the row region or the width of the column region;
Character re-arranging means for cutting out each character rectangle in accordance with the height of the row region or the width of the column region from the row region or the column region . An image rearrangement program executed by a computer,
In the computer,
An area extraction step for extracting a character area including a plurality of character rectangles from the read document image;
An inclination estimation step of dividing the character area into a predetermined number, generating a foreground histogram for each divided area, and estimating an inclination of the image using a shift in the position of the histogram created in each divided area;
The position of each character rectangle is shifted by the inclination, a histogram of the character area is generated, a row area or a column area having a width of the histogram is extracted, and the height of the character rectangle included in the line area or the column area is extracted. A row / column acquisition step for adjusting the height or width to the height of the row region or the width of the column region;
A character cutout step of cutting out each character rectangle in accordance with the height of the row region or the width of the column region from the row region or the column region;
Image rearrangement program to execute.