JP6559415B2 - Document image processing apparatus, information processing apparatus including the same, program, and recording medium - Google Patents

Description

  The present invention relates to a document image processing apparatus that reconstructs a document image, an information processing apparatus including the same, a computer program, and a recording medium.

  2. Description of the Related Art In recent years, image forming apparatuses such as copiers and multifunction peripherals have advanced functions, and a function of storing a document read by a scanner as document image data (hereinafter referred to as a document image) and managing the stored document image, etc. Is required. A document read by a scanner is stored as image data. The format of the image data is a file format having a fixed width and height called a fixed type.

  Typical file formats include, for example, PDF (Portable Document Format) files and TIFF (Tagged Image File Format) files. When document images in these file formats are displayed on an image display device with a small display area such as a mobile phone, smartphone, tablet, etc., they will not fit in the display area and both vertical and horizontal scroll operations are required. There is a case. In this case, the operation becomes very complicated.

  Therefore, for example, by adjusting the display magnification according to the number of pixels in the display area and reducing the width of the document image according to the width of the display area, the scroll operation in the row direction can be omitted. However, the readability of characters in the document image is reduced by performing the reduction process. Therefore, it is desirable to provide the read document image as a reflow type file that can be displayed in a folded manner in accordance with the display area, not a fixed type file.

  A reflow type file does not have a fixed width and height, and by automatically folding lines within the range of the display area of the image display device, a single line of text can be displayed without protruding from the display area. Therefore, it is possible to read the document only by scrolling in the direction orthogonal to the line direction without scrolling in the line direction. A file format provided by HTML (Hyper Text Markup Language) and an electronic book function developed for smartphones and tablets is an example of a file format that can be displayed in a reflow type. The readability problem is solved by converting the read document image into a reflow type file format.

  As an apparatus for converting the file format, for example, there is a document file display apparatus disclosed in Patent Document 1. The display device of Patent Document 1 converts a structured document file (doc, txt, odf, xls, etc.) into a document image file (jpeg, tiff, bmp, etc.), and constructs a document from the converted document image Layout information including the existence area of each element to be processed and the arrangement direction of the elements is detected. Then, based on the detected layout information, an element image (partial image) corresponding to the existence area of each element is extracted from the document image, and line information is extracted based on the size along the arrangement direction of the elements of each extracted element image. Create Next, paragraph information is created by determining a scroll direction of the created line information and arranging a plurality of pieces of line information along the decided scroll direction. Then, the created paragraph information is scroll-displayed within the display range of the display unit. Therefore, since the document file display device of Patent Document 1 displays a document image having the same width as that of the display area, the document image can be browsed only by a one-way scroll operation.

JP 2012-230623 A (published on November 22, 2012)

  However, the document file display device described in Patent Document 1 converts a document file specified by the user into a reflow type document regardless of a document unsuitable for the reflow type. For this reason, a document with a layout that originally had meaning, such as a document that uses a lot of tables or indents, has a layout that is corrupted and cannot be understood because it has been converted to a reflow document. Can happen.

  Accordingly, the present invention has been made in view of the above-described problems, and determines whether or not to convert a document image to a reflow type and processes a document image so that it can always be displayed in an optimal format. An object is to provide an image processing apparatus and the like.

  In order to solve the above problems, a document image processing apparatus according to an aspect of the present invention provides a structure analysis that performs a structure analysis of a document image in a document image processing apparatus that reconstructs a document image obtained by digitizing a document. Or each element included in the document image is reconstructed based on the character string extracted from the document image by the structural analysis and the feature amount of the figure or table If the conversion determination unit that determines whether or not the conversion determination unit determines to reconstruct each element of the document image, the document image is reconstructed based on the analysis result by the structure analysis unit. And a reference list generation unit that generates a reference list describing the order of each element.

  According to the above configuration, it is possible to determine whether or not to convert the document image to the reflow type, and to process the document image so that it can always be displayed in the optimum format.

1 is a block diagram illustrating a functional configuration of an image forming apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of a conversion processing unit included in an image processing apparatus included in the image forming apparatus. It is a figure which shows the example of the character of a line head prohibition character, and the example of the character of a line end prohibition character. It is a figure which shows the example of the character string of horizontal writing. It is a figure which shows the example of a document image. It is a figure which shows the example of the said document image classified into the line. (A) is a diagram showing an example of a two-column horizontal writing document, and (b) is a diagram showing an example of a two-column vertical writing document. (A)-(f) is a figure explaining the distance between two lines. (A) is an example of a document image classified into rows, and (b) is a diagram illustrating an example of a document image into which (a) is further classified into row blocks. It is a figure which shows the example of the document image classified into the line and the line block. It is a block diagram which shows the structure of the layout analysis process part which the said conversion process part has. (A) is an example of a document image classified into row blocks, (b) is an example of a document image into which (a) is further classified into columns, and (c) is a document in which (b) is further classified into columns. It is a figure which shows the example of an image. It is a figure which shows the example of the document image of a two-column structure. It is a figure which shows the example of the document image classified into the line, the line block, the stage (column), and the column group. It is a figure which shows the example of a line order list. It is a figure which shows the information about a row block, a column, and a column. It is a figure which shows the example of the structure of a document structure tree. It is an image figure which shows the outline | summary of the line feed determination process in the paragraph analysis process part which the said layout analysis process part has. It is a flowchart which shows the process sequence of the said paragraph analysis process part. It is an image figure which shows the outline | summary of the update process of a row ID buffer. It is a figure which shows the example of the initialized document structure tree. It is a figure which shows the example of the updated line order list. It is a figure which shows the example of the structure of the document structure tree produced | generated according to the line order list. It is a figure which shows the example of the document image classified into the paragraph. It is a flowchart which shows the process sequence of the rearrangement process part which the said conversion process part has. It is a figure which shows the example of the file described by the HTML language. It is a flowchart which shows the process sequence of a file description process. It is a figure which shows the example of the external file of the style sheet described in CSS format. (A) is a figure which shows the example of the external file of the style sheet described in CSS format, (b) is a figure which shows the example of the reference list described by the HTML language. It is a block diagram which shows the functional structure of the image reading apparatus which concerns on another embodiment of this invention. It is a figure which shows the example of the document image in which the block position is not aligned. (A)-(c) is a figure explaining the process which extracts a line-based graph from a document image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Embodiment 1: Image forming apparatus]
In the following description, a document image processing apparatus according to the present invention forms a part of an image processing apparatus as a conversion processing unit, and the image processing apparatus forms a part of an image forming apparatus.

[1. Image forming apparatus]
FIG. 1 is a block diagram illustrating a functional configuration of an image forming apparatus (information processing apparatus) 100 according to the first embodiment. The image forming apparatus 100 is a digital multifunction machine having a copy function, a scanner function, and the like. The image forming apparatus 100 includes an image processing apparatus 1, an image input apparatus 2, an image output apparatus 3, and a transmission apparatus 4.

  An operation panel 6 is connected to the image input device 2, the image processing device 1, the image output device 3, and the transmission device 4. The operation panel 6 includes an operation unit (not shown) such as a setting button and a numeric keypad for the user to set the operation mode of the image forming apparatus 100, and a display unit (not shown) configured by a liquid crystal display or the like. Prepare.

  Various processes executed by the image forming apparatus 100 are controlled by a control unit (a computer including a processor such as a CPU (Central Processing Unit) or DSP (Digital Signal Processor)) (not shown). The control unit of the image forming apparatus 100 performs data communication with a computer and other digital multifunction peripherals connected to the network via a network card and a LAN cable (not shown).

  Hereinafter, each part of the image forming apparatus 100 will be described in detail.

  The image input device 2 optically reads an image from a document. The image input device 2 is composed of, for example, a color scanner having a CCD (Charge Coupled Device), and a reflected light image from an original is converted into an RGB (R: red, G: green, B: blue) analog signal using the CCD. Read and output to the image processing apparatus 1. The image input device 2 may not be a scanner, but may be a digital camera, for example.

  The image processing device 1 performs processing on the image data read by the image input device 2, and generates a compressed file for storing or transmitting the processed image data.

  The image processing apparatus 1 performs an A / D conversion unit 11, a shading correction unit 12, a document type determination unit 13, an input tone correction unit 14, and a region separation for RGB analog signals input from the image input device 2. Image data composed of RGB digital signals (hereinafter referred to as RGB signals) is generated by executing image processing to be described later in the processing unit 15.

  In addition, the image processing apparatus 1 performs a color correction unit 16, a black generation and under color removal unit 17, a spatial filter processing unit 18, an output gradation correction unit 19, and a gradation for the RGB signal output from the region separation processing unit 15. The reproduction processing unit 20 executes image processing to be described later, thereby generating image data composed of digital signals of CMYK (C: cyan, M: magenta, Y: yellow, K: black), and generating an image as a stream. Output to the output device 3. Note that the image data may be temporarily stored in the storage unit 5 before being output to the image output device 3. The storage unit 5 is a non-volatile storage device (for example, a hard disk).

  The image output device 3 outputs an image based on the image data generated by the image processing device 1. The image output device 3 forms (prints) a color image on a recording sheet (for example, recording paper) based on the image data input from the image processing device 1 by a method such as thermal transfer, electrophotography, or inkjet. Output.

  In the present embodiment, the image output device 3 is configured to output a color image, but may be configured to form and output a monochrome (monochrome) image on a recording sheet. In this case, the image data of the color image is converted into the image data of the monochrome image by the image processing apparatus 1 and then output to the image output apparatus 3.

  Furthermore, the image processing apparatus 1 executes an image compression process on the RGB signal output from the region separation processing unit 15 by the compression processing unit 21, thereby generating a compressed file having image data of a compressed color image. Generate and output to the transmitter 4. Note that the compressed file may be temporarily stored in the storage unit 5 before being output to the transmission device 4.

  In the image processing apparatus 1, when the format conversion mode is selected on the operation panel 6, the conversion processing unit 22 performs format conversion processing on the RGB signals output from the region separation processing unit 15. As will be described later, a document structure tree is generated by analyzing the document layout of the image by the processing of the conversion processing unit 22.

  The conversion processing unit 22 functions as a document image processing apparatus according to the present invention. In addition, the converted file may be temporarily stored in the storage unit 5 before being output to the transmission device 4. Further, when the document image input from the image input device 2 covers a plurality of pages, only the page designated on the operation panel 6 can be analyzed to reconstruct the document as will be described later. . For example, the cover page is not subject to reconstruction, and the entire page can be directly output as an image.

  The transmission device 4 transmits the compressed file generated by the image processing device to the outside. The transmission device 4 can be connected to a communication network such as a public network (not shown), a LAN (Local Area Network), or the Internet, and transmits a compressed file to the outside via the communication network by a communication method such as facsimile or e-mail. To do. For example, when the scan to e-mail mode is selected on the operation panel 6, the transmission device 4 using a network card, a modem or the like attaches the compressed file to the e-mail and transmits it to the set transmission destination. To do.

  When facsimile transmission is performed, the control unit of the image forming apparatus 100 performs a communication procedure with the other party in the transmission apparatus 4 using a modem, and compression is performed when a transmission possible state is secured. After performing necessary processing such as changing the compression format on the file, the file is sequentially transmitted to the other party via a communication line.

  When receiving a facsimile, the control unit of the image forming apparatus 100 receives a compressed file transmitted from the other party and inputs it to the image processing apparatus 1 while performing a communication procedure in the transmission apparatus 4. In the image processing apparatus 1, the received compressed file is decompressed by a compression / decompression processing unit (not shown). The image data obtained by decompressing the compressed file is subjected to rotation processing and / or resolution conversion processing by a processing unit (not shown) as necessary, and the output gradation correction unit 19 outputs the output level. Tone correction is performed, and the gradation reproduction processing unit 20 performs gradation reproduction processing. The image data that has undergone various types of image processing is output to the image output device 3, and an image is formed on the recording sheet by the image output device 3.

[2. Image processing apparatus]
Hereinafter, the configuration of the image processing apparatus 1 will be described in detail while describing image processing and format conversion processing in the image processing apparatus 1.

  The A / D converter 11 converts RGB analog signals input from the image input device 2 to the image processing device 1 into RGB digital signals (that is, RGB signals).

  The shading correction unit 12 removes various distortions generated in the illumination system, the imaging system, and the imaging system of the image input device 2 from the RGB signal input from the A / D conversion unit 11.

  The document type discrimination unit 13 converts the RGB signal input from the shading correction unit 12 into a density signal indicating the density of each RGB color, and performs a document type discrimination process for discriminating the mode of a document such as characters, photographs, or photographic paper. Run. When the user manually sets the document type using the operation panel 6, the document type determination unit 13 outputs the RGB signal input from the shading correction unit 12 as it is to the input tone correction unit 14 at the subsequent stage. The processing result (document type) of the document type determination process is reflected in the subsequent image processing.

  The input gradation correction unit 14 performs image quality adjustment processing such as color balance adjustment, background density removal, and contrast adjustment on the RGB signals input from the document type determination unit 13.

  The region separation processing unit 15 separates each pixel in the image represented by the RGB signal input from the input tone correction unit 14 into one of a character region, a dot region, and a photo region. Further, the region separation processing unit 15 generates a region identification signal indicating which region each pixel belongs to based on the separation result, and generates a black generation and under color removal unit 17, a spatial filter processing unit 18, and a gradation reproduction processing unit. 20 and the compression processing unit 21. Further, the region separation processing unit 15 outputs the RGB signal input from the input tone correction unit 14 to the subsequent color correction unit 16, compression processing unit 21, and conversion processing unit 22 as they are.

  The color correction unit 16 converts the RGB signal input from the region separation processing unit 15 into a CMY digital signal (hereinafter, referred to as a CMY signal), and CMY colors including unnecessary absorption components for realizing faithful color reproduction. Color turbidity based on the spectral characteristics of the material is removed from the CMY signal.

  Based on the CMY signal input from the color correction unit 16, the black generation and under color removal unit 17 generates a black (K) signal from the CMY signal and a K signal obtained from the CMY signal by the black generation process. Subtraction is performed to generate a new CMY signal. As a result, the CMY3 color digital signals are converted into CMYK 4 color digital signals (hereinafter referred to as CMYK signals).

As an example of the black color generation process, a method of generating black color using skeleton black is generally used. In this method, the input / output characteristic of the skeleton curve is y = f (x), the input data is C, M, Y, the output data is C ′, M ′, Y ′, K ′, UCR (Under When the color removal rate is α (0 <α <1), the black color generation and under color removal processing is expressed by the following equations (1) to (4).
K ′ = f (min (C, M, Y)) (1)
C ′ = C−αK ′ (2)
M ′ = M−αK ′ (3)
Y ′ = Y−αK ′ (4)
Here, the UCR rate α (0 <α <1) indicates how much CMY is reduced by replacing the portion where CMY overlaps with K. Equation (1) indicates that the K signal is generated in accordance with the smallest signal strength among the signal strengths of CMY.

  The spatial filter processing unit 18 performs spatial filter processing using a digital filter on the image data of the CMYK signal input from the black generation and under color removal unit 17 based on the region identification signal input from the region separation processing unit 15. By correcting the spatial frequency characteristics, image blurring or graininess degradation is improved. For example, for the region separated into characters by the region separation processing unit 15, the spatial filter processing unit 18 uses a filter with a high enhancement amount of the high frequency component in order to improve the character reproducibility. I do. Further, the spatial filter processing unit 18 performs low-pass filter processing for removing the input halftone dot component on the region separated into halftone dots by the region separation processing unit 15.

  The output tone correction unit 19 performs an output tone correction process on the CMYK signal input from the spatial filter processing unit 18 based on the dot area ratio that is a characteristic of the image output device 3.

  The gradation reproduction processing unit 20 performs halftone processing corresponding to the region on the CMYK signal input from the output gradation correction unit 19 based on the region identification signal input from the region separation processing unit 15. For example, for a region separated into characters by the region separation processing unit 15, the gradation reproduction processing unit 20 performs binarization or multi-value conversion using a high-resolution screen suitable for reproducing high-frequency components. Process. In addition, for a region separated into halftone dots by the region separation processing unit 15, the gradation reproduction processing unit 20 performs binarization or multi-value processing on the screen with an emphasis on gradation reproducibility. . Next, the gradation reproduction processing unit 20 outputs the processed image data to the image output device 3.

  The compression processing unit 21 generates a compressed file based on the region identification signal input from the region separation processing unit 15 and the image data composed of RGB signals. The image data input to the compression processing unit 21 is composed of a plurality of pixels arranged in a matrix. The compression processing unit 21 separates this image data into a foreground layer and a background layer. Then, the foreground layer is further converted into a binary image, and each binary image is reversibly compressed by, for example, MMR (Modified Modified READ). On the other hand, the background layer is irreversibly compressed using, for example, JPEG. Finally, the reversible compressed binary image and the irreversible compressed background layer and the decompression information for decompressing them to form color image data are combined into one file. This file is a compressed file. As the decompression information, information indicating a compression format, an index color table (hereinafter referred to as an IC table), and the like are used. The compression of the region identification signal generated for each pixel is performed based on, for example, the MMR method and MR (Modified READ) method which are lossless compression methods.

  The conversion processing unit 22 performs format conversion processing on the input document image. Details of the conversion processing unit 22 will be described below.

  The above processing is controlled by a control unit (not shown) provided in the image forming apparatus 100.

[3. Conversion processing unit]
FIG. 2 is a block diagram illustrating a configuration of the conversion processing unit (document image processing apparatus) 22. The conversion processing unit 22 includes a row analysis processing unit (structure analysis unit) 31, a row block analysis processing unit (structure analysis unit) 32, a layout analysis processing unit (structure analysis unit) 33, and a conversion availability determination processing unit (conversion). A determination unit) 34, and a rearrangement processing unit (reference list generation unit) 35. Hereinafter, individual characters, diagrams, and tables constituting a document are referred to as elements (element images). The figure shall include a graph.

  The line analysis processing unit 31 extracts each element from the document image, and includes a character string line (character string line) composed of characters (character images) and a figure (graphic image) or a table (table image). Sort into rows (chart rows). Further, the first document direction indicating the writing direction such as horizontal writing or vertical writing of the document is analyzed.

  The row block analysis processing unit 32 performs processing for integrating the rows extracted by the row analysis processing unit 31 into row blocks having at least one or more rows.

  The layout analysis processing unit 33 analyzes the column structure from the positional relationship between the row blocks classified by the line block analysis processing unit 32, orders the lines of the entire document, and detects the line feed position from the line context. As a result, the document is classified into one or more paragraphs, and a document structure tree storing line information for each paragraph is generated. Details will be described later. Each paragraph of the document structure tree stores the information on the order of the character string lines and the information on the order of the chart separately, so that the arrangement of the chart can be corrected in the paragraph.

  The conversion possibility determination processing unit 34 determines whether or not to convert the document image to the reflow type from information obtained by the processing from the line analysis processing unit 31 to the layout analysis processing unit 33. The conversion possibility determination processing unit 34 outputs a determination signal to the compression processing unit 21 when determining that the conversion to the reflow type is not performed. Upon receiving the determination signal, the compression processing unit 21 converts the RGB image data into, for example, a JPEG file format and outputs it. In other words, it is converted into a format suitable for display on the display device of the transmission destination of the transmission device 4 so as to be displayed on the display device of the transmission destination of the transmission device 4 and output. Note that the compression processing unit 21 may convert, for example, to the PNG file format or the GIF file format in addition to the JPEG file format.

  The rearrangement processing unit 35 applies the characters, diagrams, and characters of the document image according to the document structure tree generated by the layout analysis processing unit 33 to the document image determined to be converted to the reflow type by the conversion possibility determination processing unit 34. An instruction for referring to each element of the table in order and a reference list listing instructions for declaring the start and end of a paragraph are generated. The format of the reference list is not particularly fixed. For example, a file generated as a document format described in a markup language such as HTML may be output as a file.

  In the present embodiment, the processing of the conversion processing unit 22 will be described using a case where image data (document image) read by the image input device 2 is processed. However, it is received via a network or a USB memory. For data stored in a memory such as a PDF file or a structured document file (doc, txt, odf, xls, etc.), processing is performed as follows. Data received or stored in the memory is converted into a document image file (jpeg, tiff, bmp, etc.) by a software processing unit (not shown), and the data converted into the document image file is converted into a conversion processing unit 22. To enter. If the data received or stored in the memory is a document image file, the software processing unit does not perform any processing.

  For the data converted into the document image file, the conversion possibility determination processing unit 34 determines whether or not to convert the data into a reflow type. If it is determined not to convert to the reflow type, the data converted into the document image file is output to the compression processing unit 21. When the data converted into the document image file is in the JPEG file format, the compression processing unit 21 outputs the data as it is without performing any processing. When the data converted into the document image file is tiff or bmp data, the data converted into the document image file is converted into, for example, a JPEG file format and output. In other words, the data is converted into a format suitable for display on the transmission destination display device of the transmission device 4 and output.

  Hereinafter, each processing unit of the conversion processing unit 22 will be described in detail.

[4. Line analysis processing section]
<4-1. Configuration of line analysis processing section>
The line analysis processing unit 31 extracts each element image from the document image and classifies it into a character string line and a chart line. Note that the graph is included in the chart row. The line analysis processing unit 31 includes a character string extraction processing unit 31a and a chart extraction processing unit 31b. The line analysis processing unit 31 further analyzes a document first direction indicating a description direction such as horizontal writing or vertical writing of the document.

<4-2. Character string extraction processing section>
The character string extraction processing unit 31a extracts (detects and cuts out) individual characters from the document image, and extracts a character string formed by arranging a plurality of characters. Characters and character strings are extracted by the following method. A pixel of a character area is extracted from a document image, and a minimum circumscribed rectangle surrounding a set of pixels considered to constitute one character is extracted from the document image as a character component. Further, it is determined whether there is a relationship between consecutive character components as a character string from the distance between rectangles of adjacent character components in each of the upper, lower, left, and right directions, and the character string region is determined from the number of consecutive rectangles. Identify. At this time, if the continuous number in the left-right direction exceeds the continuous number in the vertical direction, the character string area is written horizontally.If the continuous number in the vertical direction exceeds the continuous number in the left-right direction, the character string is written as the vertical character string area. The direction (character string direction, description direction) of is simultaneously acquired.

  The method for extracting characters and character strings is not limited to the method described above, and other methods can be used. For example, individual characters and character strings may be extracted by an optical character reader (hereinafter referred to as OCR).

<4-3. Chart Extraction Processing Unit>
The chart extraction processing unit 31b extracts a figure (figure area) and a table (table area) from the document image.

  The drawing area is extracted by the following method. Utilizing the fact that when a pixel value appearance frequency (that is, a histogram) is obtained for each predetermined area of a document image, a histogram with a wide range of density changes is obtained for each pixel on the photographic area, which is one of the figure areas. Then, the entropy (average information amount) of the histogram is calculated. Thus, by extracting a region with high entropy, it is possible to accurately extract a photographic region. The method for extracting the figure region is not limited to the method described above, and other methods may be used.

  The table area is extracted by the following method. Extracting candidate pixels that may be lines from the document image, and extracting the set of consecutive candidate pixels as a line when the candidate pixels continue for a predetermined number of pixels in the horizontal or vertical direction, the extracted From the positional relationship between the horizontal and vertical lines, it is determined whether each line is a ruled line or a single line constituting the table, and all of the sets of lines constituting the same table are enclosed. The minimum circumscribed rectangle is extracted as a table area. By extracting in this way, it is possible to detect the table area with high accuracy. Note that the table region extraction method is not limited to the method described above, and other methods may be used.

  Although the graph is a kind of diagram, the graph region can be extracted as follows. In the case of a graph that is composed only of its shape, such as a pie graph, or a graph that is connected to the axis line of a graph, such as a bar graph, etc. It is possible to extract a graph area by using it. Further, in the case of a graph that is separated from the graph axis by a line-based graph such as a line graph, it can be extracted as follows. As shown in (a) to (c) of FIG. 32, it is not as horizontal or vertical as the ruled lines constituting the table in parallel with the entropy value, but is different from a single straight line in a rectangle or L-shape. For areas with horizontal or vertical line connections such as U-shaped or U-shaped, the outermost outline of the area is rectangularized, and the size of the rectangle is a rectangle having an area equal to or greater than a predetermined threshold. It is possible to extract a graph region by determining whether there is a region having a certain degree of entropy value in the rectangular region. Note that the graph region extraction method is not limited to the method described above, and other methods may be used.

  If the character extracted by the character string extraction processing unit 31a overlaps the range extracted as a diagram or table extracted by the chart extraction processing unit 31b, the extracted character is canceled. In particular, there is a high possibility that characters are included in the extracted table. However, in order to match the size of the table to the width of the display area, it is necessary to adjust the width of each column constituting the table. As a result, the character string in the table is displayed in a folded manner in accordance with the adjusted column width, which causes a decrease in readability. For this reason, in the present embodiment, the area extracted as a table is extracted and displayed as a chart including characters.

<4-4. Setting of row ID>
The row analysis processing unit 31 sets a row ID (Identification) as a non-overlapping identification symbol for identifying each row for the character string row and the chart row extracted as described above. If two or more rows having the row ID do not exist for one row ID, it is not always necessary to assign the row ID according to the document order. As a method for setting the row ID, a case where the row ID is “0” is set as an invalid row which does not exist. For example, in the document image of each page, the upper left corner of the read original is set as the origin (0, 0). On the other hand, there is a method of adopting a coordinate system in which the right direction is the X coordinate and the lower direction is the Y coordinate, and the row IDs are assigned sequentially in ascending order of the top left Y coordinate representing the row range. It should be noted that the rows having the same Y coordinate are given priority in allocating the row having the smaller X coordinate. When this method is used, line IDs are not necessarily assigned according to the reading order of the document due to the column structure. However, in the case of a horizontally written document, there is a high possibility that the order is higher as the line is higher. It can be said that this is a way of allocating row IDs reflecting the order of the target documents. The setting method of the row ID is not limited to this, and can be freely selected.

<4-5. Determination of document first direction and document second direction>
Further, when the line analysis processing unit 31 classifies all lines in one page of the document image as a character string line or a chart line, the line analysis processing unit 31 determines a document first direction indicating the direction of the entire document from the direction of the character string. . The first direction of the document is horizontal for horizontal writing and vertical for vertical writing. The first direction of the document is determined by classifying the directions of all the acquired character strings and by the ratio. As a simple example of the ratio calculation method for determining the document first direction, there is a method of simply counting the number of horizontally or vertically written character strings and calculating the ratio of the number. In the case of this method, for example, the number of horizontal writing lines is compared with the number of vertical writing lines, and the larger direction is determined (set) as the first document direction. The determination of the document first direction is not limited to the above method, and various methods can be adopted.

  Here, when the calculated ratio is equal to or less than a predetermined threshold (for example, 0.7), the vertical writing line and the horizontal writing line are mixed in a ratio that cannot be ignored in the document, and the direction of the entire document is uniquely determined. If it is not possible, the line block analysis processing unit 32 and the layout analysis processing unit 33 do not perform processing, and the conversion possibility determination processing unit 34 determines that the input document image is not converted to the reflow type.

  Further, when the document first direction is determined by the above method, the document second direction is determined (set) as a direction orthogonal to the document first direction. That is, when the document first direction is horizontal (horizontal writing), the document second direction is vertical, and when the document first direction is vertical (vertical writing), the document second direction is horizontal.

<4-6. Integrated processing of symbols (about items)>
In the extraction of individual characters, a symbol (about object) integration process as shown below can be added. There are, for example, symbols that are prohibited from coming to the beginning of a line (prohibition at the beginning of a line) and symbols that are prohibited from coming to the end of a line (prohibition at the end of line), and characters as shown in FIG. Can be mentioned. When individual cut out characters are displayed, these rules may not be observed due to line wrapping, and readability may deteriorate. Therefore, by combining the preceding character with the preceding character for the prohibition symbol and integrating with the succeeding character for the prohibition symbol, the character at the beginning or the end of the line can be prevented.

  A known method can be used as a method for determining whether each character is a symbol having the above-described rule. For example, the character type may be collated using OCR processing, or may be determined from the size of the character and the characteristics of the pixels constituting the character. For example, in the case of a punctuation mark, when it is horizontal writing as shown in FIG. 4, it is composed of only the lower half of the line and has a width about half the height of the line (the range indicated by the arrow). Assuming that the character is likely to be a punctuation mark, it may be integrated with the previous character.

  Although an example of horizontal writing is shown in FIG. 4, integration can be similarly performed in the case of vertical writing. In order to distinguish it from single-byte lower-case letters, processing such as adding Japanese and English sentences based on the tendency of the height and width of other characters that make up the line, and determining punctuation marks only for Japanese sentences may be added. . For example, in Japanese, full-width characters such as hiragana, katakana, and kanji occupy most of the sentence, and there is a tendency that characters composed of only the upper half or the lower half of a line are reduced. In addition, compared to half-width characters, there are many types of characters with full-width characters whose width is greater than half of the line height. Therefore, (1) straddling a horizontal straight line that divides the line up and down, and (2) counting the number of characters whose horizontal width is equal to or greater than the value obtained by multiplying the line height by a predetermined coefficient (for example, 0.6). When the ratio of the number of characters satisfying the above (1) and (2) with respect to the number of characters constituting the line is equal to or greater than a predetermined threshold (for example, 0.5), a process for determining that the line is a Japanese sentence is applied. be able to. The determination method of Japanese and English is not limited to this method, and may be determined by other methods. If the document is vertically written, it may be automatically regarded as a Japanese sentence.

  Here, there is a possibility that punctuation marks cannot be distinguished from small Japanese characters such as “a”. However, since these small Japanese characters are also prohibited from beginning of line, there is no problem even if they are integrated in the same manner as punctuation marks. Therefore, it is not necessary to strictly process punctuation.

<4-7. Processing example>
As a specific example of the process executed by the line analysis processing unit 31, a case where the line analysis process is applied to a document image (one page) having the structure shown in FIG. 5 will be described. As shown in FIG. 6, the row analysis processing unit 31 separates a range of a circumscribed rectangle having a minimum size including all the elements of the row as a region of each row, and assigns a row ID to each row and an upper left corner of the circumscribed rectangle. Are assigned in the order of their vertical coordinates (Y coordinates). In the document image shown in FIG. 6, the remaining lines are horizontal character string lines except that the line ID 105 is a chart line, and the vertical character string line is 1. No one is included. Therefore, the line analysis processing unit 31 determines that the first document direction in the document image is the horizontal direction.

[5. Row block analysis processing unit]
<5-1. Processing of line block analysis processing section>
The row block analysis processing unit 32 integrates the character string rows classified by the row analysis processing unit 31 into a character string row block including at least one character string row, and creates a new row having a non-overlapping row block ID. Store (register) in the storage unit 5 as a block. The row block analysis processing unit 32 registers one row block with a single row as a new row block having a row block ID that does not overlap each other.

  The character string row block integration processing by the row block analysis processing unit 32 will be described in detail below. First, the target line L1 is selected from the group of character string lines among the lines classified by the line analysis processing unit 31. Subsequently, with respect to the target line L1, a character string line that continues forward and backward (for details, see <5-5> section below) is searched. Specifically, at most one character string row that is continuous forward or backward of the target line L1 is selected from a character string line that is different from the target line L1 in the group of character string lines. The character string rows that are candidates for the continuous line of the target line L1 are all the character string lines except the target line L1, and the conditions that are candidates for the continuous line will be described later.

  For all the character string lines, if a character string line that is continuous forward and backward is selected, it is classified as one line block from the connection of consecutive character string lines until there is no continuous character string line in both the forward and backward directions. Then, they are integrated and registered in the storage unit 5 as a new row block having an unassigned row block ID. The process is repeated until all the character string lines are registered in any one of the line blocks. When the registration for all the character string lines is completed, the line block analysis processing unit 32 ends the process.

<5-2. Judgment of candidates for continuous lines>
Here, a method for determining whether or not the character string row L2 selected separately from the attention row L1 is a candidate for a continuous row of the attention row L1 will be described. It is assumed that at least the following two conditions are satisfied as a condition that the row L2 is a candidate for a continuous row of the row L1.
Condition 1: In the range from the beginning to the end of one line, at least one of the beginning or the end of the other line exists.
Condition 2: The inter-line displacement amount linespace (L1, L2) of the two rows satisfies the following expression (5).
TH_MIN_LS ≦ linespace (L1, L2) ≦ TH_MAX_LS (5)
(TH_MIN_LS and TH_MAX_LS are preset threshold values)
Condition 1 is used so that lines belonging to different stages are not determined as continuous line candidates. Condition 2 is used in order not to determine a line whose line spacing is too wide or too narrow as a candidate for a continuous line.

  FIG. 7A shows an example of a horizontally written document with two columns, and FIG. 7B shows an example of a vertically written document with two columns. In order to satisfy the condition 1, part or all of the lines L1 and L2 need to overlap in the first direction of the document. For example, in the example of FIG. 7, row a and row b, row c and row d, row e and row f, row g and row h satisfy condition 1, and therefore these combinations are mutually consecutive row candidates. Become. However, since the condition 1 is not satisfied in the combination of the row a and the row d, the row e and the row h, these combinations are not candidates for continuous rows.

  As shown in FIG. 8, the interline displacement amount linespace (L1, L2) between the rows L1 and L2 shown in the condition 2 is the upper end of the lower row of the rows L1 and L2 in the case of horizontal writing. The difference value between the coordinates and the lower end coordinates of the other line (see (a) to (c) of FIG. 8). In the case of vertical writing, the right end coordinates of the left line of the lines L1 and L2 And the difference value between the left end coordinates of the other line (see (d) to (f) of FIG. 8). When the line L1 and the line L2 do not overlap, the interline displacement amount linespace (L1, L2) indicates the interline distance between the two lines. Moreover, TH_MIN_LS and TH_MAX_LS are predetermined coefficients indicating the minimum value and the maximum value of the difference value allowed as the inter-row displacement amount between consecutive rows. For example, a value obtained by multiplying the character size of the line L1 by a predetermined coefficient r1 (for example, r1 = 0.1) is set as a threshold TH_MIN_LS, and the predetermined coefficient r2 (r2 is a positive number, for example, r2 = 1.5) is set. The multiplied value is set as the threshold value TH_MAX_LS.

  The threshold value TH_MIN_LS and the threshold value TH_MAX_LS may be set by other methods. For example, the average value of the character sizes of the lines L1 and L2 may be multiplied by a predetermined coefficient. Moreover, by setting the threshold value TH_MIN_LS to a positive value, it is possible to prevent two overlapping rows from being recognized as continuous rows. Conversely, by setting the threshold value TH_MIN_LS to a negative value, it is also possible to allow a case where the row L1 and the row L2 slightly overlap as shown in (c) and (f) of FIG.

<5-3. Strengthening of conditions: Specifying the indent range>
In addition to the above conditions 1 and 2, another condition may be set in order to reinforce the condition for determining candidates for continuous lines. For example, it can be added as a condition that the following expression (6) is satisfied.
indent (L1, L2) ≦ TH_INDENT (6)
Here, indent (L1, L2) is the magnitude of the difference between the document first direction component at the start position of line L1 and the document first direction component at the start position of line L2, that is, the magnitude of indentation. Further, the threshold value TH_INDENT is a predetermined coefficient indicating a distance allowed as an indent at the beginning of a line. The threshold value TH_INDENT, for example, gives a value obtained by multiplying the character size of the line L1 by a predetermined coefficient α (α is a positive number, for example α = 1.5), and allows indentation within α characters. The threshold value TH_INDEX may be set by other methods, for example, an average value of the character sizes of the lines L1 and L2 may be multiplied by a predetermined coefficient α.

<5-4. Strengthening of conditions: Specifying tolerance range for line end position differences>
As another condition for strengthening the condition for determining the candidate for the continuous line, for example, by adding the condition that the following expression (7) is satisfied as a condition, the lines where the end positions of the lines are close to a certain extent are made candidates for the continuous line. Can do.
| L1MAX1-L2MAX1 | ≦ TH_DIFF_ENDPOS (7)
Here, L1MAX1 is the maximum value of the document first direction component of the row L1, and L2MAX1 is the maximum value of the document first direction component of the row L2. For example, when the first direction of the document is the horizontal direction (horizontal writing), L1MAX1 and L2MAX1 indicate the X coordinates of the right ends of the rows L1 and L2. The threshold value TH_DIFF_ENDPOS is a predetermined coefficient indicating a distance allowed as a difference between the end positions of rows. For example, the threshold value TH_DIFF_ENDPOS can be used to allow indentation within β characters by multiplying the average value of the character size of the line L1 by a predetermined coefficient β (β is a positive number, for example, β = 0.5). become.

<5-5. Select forward or backward continuous lines>
From the rows extracted as candidates for the continuous row of the row L1, the row closest to the front of the row L1 and the row closest to the rear are selected one at a time. When the document first direction is the horizontal direction (horizontal writing), the line above the line L1 is the front line, the line below the line L1 is the back line, and the document first direction is the vertical direction (vertical). In the case of writing), a line on the right side of the line L1 is a front line, and a line on the left side of the line L1 is a rear line. Further, for example, the above-described interline displacement amount linespace (L1, L2) is used as a value representing the closeness of the rows, and the smaller the linespace (L1, L2), the closer the rows can be considered. Note that there is a maximum of one continuous line for both the front and rear, and it is not always necessary to have a continuous line.

<5-6. Classification and integration into row blocks>
As described above, when the line block analysis processing unit 32 selects the front and rear continuous lines for all the character string lines, the line block analysis processing unit 32 performs classification and integration into the line blocks. However, since there may be a case where a plurality of rows are selected as continuous rows, the continuous relationship between the pairs of rows that are not considered to be continuous rows is canceled in advance. For example, when the document image is (a) in FIG. 9, there are two rows L4 and L5 for selecting the row L3 as the front continuous rows, but the row L5 is the continuous back of the row L3. Not selected as a row. Therefore, the continuous relationship between row L3 and row L5 is canceled. Similarly, the continuous relationship between the rows L4 and L6 is also eliminated. Accordingly, in the example as shown in FIG. 9A, the continuous relationship from the target row L1 is broken in the row L3 and the row L4, so that it may not be classified as the same row block as the target row L1. Can happen. However, as shown in FIG. 9B, it can be classified as a plurality of row blocks, and the column analysis processing unit 37 in the subsequent layout analysis processing unit 33 configures the same column and its column configuration. Therefore, even if the blocks are separated at this point, there is no problem.

  The classification and integration processing into row blocks is performed as follows. First. Regarding an arbitrary line L1 (target line L1) among character string lines not classified as a line block, first, the line L1 is set as a new line block. Subsequently, the continuous line before and after the line L1 is traced to expand the range of the line block. When there are no consecutive rows in both the front and rear, the expansion of the row block is terminated, and row IDs are acquired in order from the first row included in the row block. Further, as the row block information, the upper left coordinates, the width and the height of the smallest rectangle circumscribing all the rows included in the row block, and the number of included rows are acquired. The row block having the row order and various information obtained as described above is registered as a new row block having an ID that does not overlap with an already registered row block, and each row included in the row block is registered. Update the belonging row block ID. The row block analysis processing unit 32 repeats the classification and integration processing into the row blocks performed in this way until all the character string rows are classified into any row block.

<5-7. Conditions for rows that can be classified into the same row block (can be candidates for continuous rows)>
A character string line in the first direction of the document constitutes a line block only with a character string line in the first direction of the document, and a character string line in the second direction of the document constitutes a character string line in the second direction of the document. That is, the character string line in the document first direction and the character string line in the document second direction are not mixed in one line block. Therefore, when searching for a candidate for a continuous line of the target line L1, a character string line in a direction different from the character string direction of the target line L1 is not a candidate for a continuous line.

<5-8. Processing example>
The row block analysis processing executed by the row block analysis processing unit 32 will be described using a specific example. When the row block analysis process is applied to a plurality of rows detected from the document image as shown in FIG. 6, the document image is classified as a row block as shown in FIG. In the example shown in FIG. 10, the row block B3 is a chapter heading row, and has a larger character than the row block B4. It is also an effective means to add a condition for determining candidates for continuous lines so that two lines having greatly different character sizes are not selected as candidates for continuous lines.

[6. Layout analysis processing unit]
<6-1. Configuration of layout analysis processing unit>
FIG. 11 is a block diagram showing a detailed configuration of the layout analysis processing unit 33. The layout analysis processing unit 33 analyzes the row block configuration from the vertical and horizontal positional relationships between the row blocks classified by the previous row block analysis processing unit 32, and estimates the reading order of the sentences (text) in the document image. The process which performs is performed. The layout analysis processing unit 33 includes a column analysis processing unit 37, a line ordering processing unit 38, and a paragraph analysis processing unit 39.

<6-2. Multi-column analysis processing section>
The column analysis processing unit 37 executes column analysis processing for classifying the columns and the respective columns (columns) constituting the columns from the vertical and horizontal positional relationships of the plurality of row blocks. Assuming that the document is arranged in columns in the second direction of the document, and columns are arranged in the first direction of the document in each column structure, the set of row blocks in the page is appropriately set with a boundary line. Divide without crossing the row block to the initial column. Then, a set of row blocks included in the same initial column is divided without setting a boundary line so as to cross the row block, and used as an initial column constituting the initial column.

  The method of setting the boundary line is not particularly specified. As the simplest example, a straight line parallel to the second direction of the document is used for classification of the initial column, and a straight line parallel to the first direction of the document is used for classification of the initial column. Can be used. For example, when the structure of a row block is analyzed from a horizontally written document image as shown in FIG. 12A, the row block is classified into an initial column as shown in FIG. Each is classified as an initial column as shown in FIG. In FIGS. 12B and 12C, the boundary lines between the columns are indicated by solid lines, and the boundary lines between the columns are indicated by alternate long and short dash lines.

<6-3. Prohibition of column division (calculation of distance between row blocks allowed for the same column)>
Although it is originally the same column, there is a rare case where a row block can be divided by chance and thus divided into a plurality of columns. In order to cope with such a case, for example, a distance between two consecutive row blocks is calculated, and a boundary line is drawn between two blocks whose distance is equal to or less than a predetermined value (for example, twice the average inter-row distance of the row blocks). You can add a condition that prohibits this.

  FIG. 13 shows an example of a two-column configuration in which the row blocks B10, B11, and B12 are on the left column, and the row blocks B20 and B21 are on the right column. Since the space between the row block B10 and the row block B11 and the space between the row block B20 and the row block B21 are vacant, the two-stage configuration including the row block B10 and the row block B20, and the row blocks B11 and B12 and the row block There is also a possibility that it is divided as a two-stage configuration composed of the block B21. However, assuming that the inter-block distance (30) between the row block B20 and the row block B21 is equal to or less than a predetermined value (20 × 2 = 40) with respect to the average inter-row distance (20) of the row block B20, By prohibiting the boundary line from being drawn with the row block B21, it is possible to prevent these row blocks from being divided into two different columns.

<6-4. Utilization of information other than positional relationships>
Further, in addition to the positional relationship of the row blocks, it is possible to classify columns and columns (that is, set boundary lines) by using various information held by the row blocks. Examples of various information held by the line block include the length of the line and the size of main characters. If these pieces of information differ greatly between adjacent row blocks, it can be avoided to classify them into the same column set. On the other hand, even when the row blocks are far apart from each other, for example, the same column boundary line can be shared and similar information may be classified as the same column.

<6-5. When there are only 0 or 1 row block>
If the page image input to the column analysis processing unit 37 has only one row block, the page image is a one-stage document, and no boundary line is set. Even when the page image has no row block (that is, a blank page), no border is set.

  Further, when the document image includes character string rows in the first direction of the document and character string rows in the second direction of the document, the row block of the character string rows in the second direction of the document is replaced with the row block of the chart row. Thus, it is possible to prevent a sentence described in the second direction of the document from being mixed with a sentence described in the first direction of the document.

<6-6. Processing example>
As a specific example of the processing executed by the column analysis processing unit 37, for example, a case where the column analysis processing is applied to a plurality of row blocks detected from the document image shown in FIG. The column analysis processing unit 37 classifies the document image shown in FIG. 10 into columns and columns (light gray background) shown in FIG. The column C1 and the column C2 respectively constitute a column set G1 and a column set G2 having a one-stage configuration, and the column C3 and the column C4 constitute left and right columns in the column set G3 having a two-stage configuration. In FIG. 14, lines (character string lines and chart lines) are surrounded by straight lines, line blocks are surrounded by dotted lines, and columns are surrounded by alternate long and short dash lines.

<6-7. Line ordering processing section>
The line ordering processing unit 38 analyzes the order of lines in the entire document from the column, column, line block, and line positional relationship, and generates a line order list according to the following rules (1) to (9). Follow the instructions.
(1) For the rows belonging to the same row block, priorities are set in the order from top to bottom for horizontal writing documents and from right to left for vertical writing documents. Here, as described above, since the information on the order of the rows included in the row block is already acquired when the data is classified into the row blocks, this information is used.
(2) For row blocks belonging to the same column, priorities are set in order from top to bottom for horizontal writing documents and from right to left for vertical writing documents.
(3) Between two consecutive row blocks, the first row of the row block with the lower priority is set to be given priority after the last row of the row block with the higher priority.
(4) For columns belonging to the same column, priorities are set in order from left to right for horizontally written documents and from top to bottom for vertically written documents.
(5) Between two consecutive columns, the first row block of the lower priority column is set to be given priority after the last row block of the higher priority column.
(6) For columns belonging to the same page, priorities are set in order from top to bottom for horizontal writing documents and from right to left for vertical writing documents.
(7) Between two consecutive columns, setting is performed so that the first column in the column with the lower priority is given priority after the column at the end of the column with the higher priority.
(8) For pages belonging to the same document image file, settings are made so that the priorities are higher in order of increasing page numbers.
(9) Between two consecutive pages, setting is made so that the top column of the page with the lower priority is given priority after the last column of the page with the higher priority.

The row ordering processing unit 38 determines the order of pages, the order of columns, the order of columns, and the order of row blocks in accordance with the above rules (1) to (9), and performs ordering of rows by them. The ordered rows hold numbers indicating the order of row blocks, columns, columns and pages to which each row belongs, and store the row ID of each row in the row order list in order from the top.
The structure of the line order list is not particularly limited as long as it conforms to the following rules (a) to (c).
(A) Rows can be called in the order determined according to the ordering rules.
(B) With respect to the called line, various information such as coordinate information and type (whether it is a character string line or a chart line) can be referred to.
(C) Paragraph information (described later) can be stored for the called line.

<6-8. Processing example>
FIG. 15 shows an example in which the line ordering processing unit 38 generates a line order list for the document image having the configuration shown in FIG. 14 in accordance with the rules (a) to (c). The line order list includes, from the head of the determined order, the line ID, the line block ID to which the line belongs, the line type (whether it is a character string line or a chart line), and the line information The information of the upper left coordinate and the lower right coordinate of the circumscribed rectangle indicating the range is stored, and the paragraph information is stored for each line. In the example of FIG. 15, when a line break occurs, a new paragraph starts from that line, and the paragraph information is shown in two ways, whether there is a line break (Yes) or not (No), It is initialized with “No” in advance. By creating separate row block information, column information, and column information as shown in FIG. 16 and making it possible to refer to the columns, columns, and pages to which each row and each row block belong by cross-reference. It is possible to construct a line order list with few. Of course, all the information regarding each line may be extracted by the line order list alone.

<6-9. Paragraph analysis processing section>
The paragraph analysis processing unit 39 determines whether a line break has occurred at the position of the line from information such as the positional relationship before and after each line, and classifies each line in the document image into one or more paragraphs. Do. Specifically, a plurality of lines are referenced from the line order list to determine paragraph breaks, that is, line break positions (line break determination processing), and generate a document structure tree describing the line order for each paragraph (document structure) Tree generation processing).

  By the way, charts in a document are not necessarily arranged at paragraph breaks, and are often inserted, for example, at the end of a page, so that a sentence may be divided before and after the chart. If the lines are called in this order and the elements (characters, charts) constituting the lines are inserted, sentences that are unnaturally interrupted by the insertion of the charts are output. Therefore, in the present embodiment, for each paragraph, a document structure tree is generated in which the order of character string lines and the order of chart lines are held separately, not the order in which character string lines and chart lines are mixed. FIG. 17 is a diagram showing the structure of a document structure tree. Thereby, while grasping the order of only the character strings constituting the document image, the chart relating to the paragraph can be arranged collectively at the beginning or end of the paragraph.

<6-10. Line feed judgment processing>
FIG. 18 is an image diagram showing an outline of a line feed determination process in the paragraph analysis processing unit 39. The line break determination process is determined by M + N + 1 lines of a target line to be determined, M lines ordered before the target line, and N lines ordered after the target line. In the example shown in FIG. 18, M = N = 2. Note that the line that is subject to line feed determination and the lines before and after it are both character string lines. In the present embodiment, M + N + 1 row ID buffers L [0], L [1]..., L [M + N] are provided in the storage unit 5, and M + N + 1 row IDs referenced in the row order list are respectively provided. By storing, the attention line is compared with the lines before and after the attention line.

Hereinafter, the processing content of the paragraph analysis processing unit 39 will be described in detail. FIG. 19 is a flowchart showing the processing procedure of the paragraph analysis processing unit 39. Before starting the paragraph analysis process, initialize it in advance. Specifically, the list reference number is LNOW = 1, and all invalid rows (0) are stored in the row ID buffer. When the initialization is completed, the row ID is selected and stored in the row ID buffer L [M] corresponding to the target row (step S1, hereinafter abbreviated as S1). A method of selecting an arbitrary row ID buffer L [k] (k = 0, 1,..., M + N) is performed as follows (1) to (3).
(1) In the line order list, the character string lines are searched in order from the LNOWth.
(2) The row ID of the first found character string row is stored in the row ID buffer L [k], and a number obtained by adding 1 to the position (list number) of the row order list at that time is a new list reference number LNOW. Update as.
(3) When the search is completed up to the end of the line order list without finding the character string line, the invalid line (0) is stored in the line ID buffer L [k].

  After updating the row L [M], it is determined whether or not the row L [M] is a valid row (a row having a non-zero row ID) (S2). If the row L [M] is a valid row (YES in S2), Move on to S3. On the other hand, when L [M] is an invalid line (NO in S2), the document structure tree generation process (described later) is performed assuming that the input document image is a document image of only a chart row in which no character string row exists. ) Is executed (S7).

  Next, the row ID is selected and stored in the row ID buffers L [M + 1],..., L [M + N] corresponding to the row after the target row (S3). Since the selection method of the row ID in each buffer is the same as described above, the description is omitted. Subsequently, line feed determination for the target line L [M] is executed (S4). Line break determination is performed by the attention line L [M] to be the line break determination target, the lines L [0],..., L [M−1] positioned before the target line, and the line L [ M + 1],..., L [M + N] can be performed by a known method using a plurality of rows. As a simple example, there is a method of confirming whether or not there is an indent to be detected as a line feed. When the start position of the line L [M] has a positive shift in the first direction of the document compared to the other lines, the line L [M] has an indent, and the position of the line L [M] It can be considered that a line feed has been made. In addition, when the line L [M−1] immediately before the line feed determination target line L [M] is shorter than the other lines, it may be considered that a line break is made at the position of the line L [M]. it can. For example, when there is a horizontally written character string line as shown in FIG. 18 (M = N = 2), the start position of the target line L [2] is on the right side (the correct line in the case of horizontal writing) as compared to the other lines. And the previous line L [1] is shorter than the other lines. From these results, the target line L [2] It becomes easy to determine that there is. Note that indentation may not be added to the first line of a paragraph depending on the preference of the document editor, and the last line of the previous paragraph is not necessarily shortened.

  Another example of determining whether a line feed position is present is as follows. By determining whether the character size of the line of interest is significantly different from the surrounding lines, the character size is reduced, such as a line with a large character size, such as a heading line, or a supplementary comment. It can be determined that the line is at a line break position. In addition, when the distance between the target line and the previous line is larger than the distance between two adjacent consecutive lines, the paragraph may end at the previous line. Increases nature. Taking the above-mentioned conditions as an example, it is preferable to determine whether or not there is a line break in the line of interest by determining various conditions in combination.

  The result of line feed determination for line L [M] is reflected as paragraph information in the line information corresponding to L [M] in the line order list. The paragraph information may be anything as long as the break of the paragraph is known. As the simplest example, the presence / absence of a line feed may be indicated in two ways, Yes or No.

  When the line feed determination of the line feed determination target line L [M] is completed, it is determined whether or not the line L [M + 1] next to the target line is a valid line (S5), and if L [M + 1] is a valid line ( If the determination in S5 is YES), the line ID buffer is updated, and preparations are made for line feed determination for the next line (S6). Specifically, as shown in FIG. 20, the row ID buffer is updated with L [0] = L [1],..., L [M + N−1] = L [M + N] one by one. At the same time, the buffer L [M + N] is newly selected from the row order list. Since the method for selecting the buffer L [M + N] is the same as that described above, a description thereof will be omitted. After updating the buffer, the process returns to S4, and a line feed is determined for the updated attention line L [M]. This is repeated until the determination is NO in S5. If the determination in S5 is NO, the line feed determination has been completed for all the character string rows, and the document structure tree generation process is executed (S7).

<6-11. Document structure tree generation processing>
The document structure tree generation process by the paragraph analysis processing unit 39 is executed according to the paragraph information stored in the line order list considering the paragraph information. However, it is assumed that the document structure tree is initialized in advance to have a single empty paragraph (0th paragraph) as shown in FIG. The initial paragraph number is set to PNOW = 0, the line information is referred to in order from the top of the line order list, and the paragraph is set to PNOW = PNOW + 1 only when the line is determined to be the paragraph start line from the paragraph information (the paragraph information is YES). The number is updated, and an empty paragraph (the PNOW paragraph) is newly added to the document structure tree. If the line is a character string line, the line ID of the line is added to the end of the character string tree held in the PNOW paragraph of the document structure tree. On the other hand, when the line is a chart line, the line ID of the line is added to the end of the chart tree of the PNOW paragraph of the document structure tree. This is repeated for each line in the line order list, and when the search is completed to the end, the document structure tree generation process is terminated. Note that a line from the line feed position of the character string line to the next line feed position is one paragraph, so that the chart line is not leaked from the paragraph classification. Therefore, when there is a chart row after a group of character strings and a new paragraph starts (a line break occurs), the chart row is included in the previous paragraph.

<6-12. Processing example>
As a specific example of the processing executed by the paragraph analysis processing unit 39, a case will be described in which the processing in the paragraph analysis processing unit 39 is applied to a document image having the configuration shown in FIG. 6 (having the initial line order list shown in FIG. 15). . Each line of the line ID: 101, line ID 104, and line ID: 129 corresponding to the headline is determined to be a line feed position based on conditions such as the character size of the line and the distance between the previous lines. In addition, each line of line ID: 106, line ID: 112, line ID: 119, and line ID: 131 has a start position of the line shifted in the positive direction in the first direction of the document as compared to the preceding and following lines. Therefore, it is determined that there is a line break position because there is an indent. Since the line with the line ID: 102 is also the line next to the line ID: 101 of the heading line, it can be determined as a line feed position. Therefore, each line of line ID: 101, 102, 104, 106, 112, 129, 131, 119 is set as a line feed position, and the paragraph information of the line order list is updated as shown in FIG.

  For example, when the line order list indicates whether or not there is a line break (Yes or No) as paragraph information as shown in FIG. 22, it can be considered that a new paragraph starts from a line with a line break, and therefore there is a line break (Yes). The line can be determined as the paragraph start line. Also, the line from the start of the paragraph until the next line feed position is found is regarded as the same paragraph range. If the document images in FIG. 6 are classified for each paragraph in accordance with the updated paragraph information in the line order list, they can be classified as shown in FIG. In FIG. 24, the paragraph R7 starts from the lower left three character string lines (line ID: 131, 133, 135), the upper right chart line (line ID: 105), and the four character string lines (line ID: 111). , 113, 115, 117) are the same paragraph range, and there is no line break position between the character string lines of line ID: 135 and line ID: 111, so that a series of sentences can be formed. It becomes possible. Furthermore, when a document structure tree is generated according to this line order list, a document structure tree as shown in FIG. 23 can be obtained.

[7. Conversion enable / disable determination processing unit]
<7-1. Processing of conversion possibility determination processing unit>
The conversion possibility determination processing unit 34 next converts the document image from the information (character strings extracted from the document image or feature values of the figure or table) obtained by the processing from the line analysis processing unit 31 to the layout analysis processing unit 33. Whether to convert to the reflow type, in other words, whether to reconstruct each element included in the document image is determined.
First condition: The line analysis processing unit 31 determines that the vertical writing line and the horizontal writing line are mixed in a ratio that cannot be ignored in the document, and the direction of the entire document cannot be uniquely determined.
Second condition: In the processing of the row analysis processing unit 31, only the chart row was extracted.
Third condition: The number of lines of the character string extracted by the line analysis processing unit 31 is equal to or less than a predetermined threshold (THln) (first threshold).
Fourth condition: The height of the horizontal writing line (the width of the line in the case of vertical writing) for all the character string lines extracted by the line analysis processing unit 31 is a predetermined threshold (THcs) (second Threshold) or more.
Fifth condition: The positions of the horizontal writing or vertical writing row blocks ordered by the layout analysis processing unit 33 are not aligned within a certain range (THrg).

  The conversion possibility determination processing unit 34 determines that the input document image is not converted to the reflow type when any of the above five conditions is satisfied. If the conversion possibility determination processing unit 34 determines not to convert to the reflow type, it outputs a determination signal to the compression processing unit 21. Upon receiving the determination signal, the compression processing unit 21 converts the RGB image data into, for example, a JPEG file format and outputs it. That is, the entire image is output as it is so as to be displayed in a fixed type. As described above, in the present embodiment, the compression processing unit 21 functions as a format conversion processing unit that converts the format of a document image that is determined not to be converted into the reflow type so that a fixed type display can be performed.

  The conversion possibility determination processing unit 34 determines that the input document image is converted to the reflow type when any of the above five conditions is not satisfied. With respect to the document image determined to be converted to the reflow type, the next rearrangement processing unit 35 generates a reference list.

<7-2. Details of judgment conditions>
Each of the above five conditions will be described in detail.

  First, the first condition will be described. As described above, the line analysis processing unit 31 classifies the directions of all the acquired character strings, and determines the first direction of the document based on the ratios. Count the number. Then, by calculating the ratio of the counted number, when the calculated ratio is equal to or less than a predetermined threshold (for example, 0.7), the vertically written line and the horizontally written line are mixed in a ratio that cannot be ignored in the document image. It is determined that Therefore, the direction of the entire document cannot be uniquely determined, and it is difficult to determine how to connect vertical writing and horizontal writing lines when performing layout analysis. Therefore, it determines with not converting into a reflow type.

  Next, the second condition will be described. If only the diagram line is extracted by the line analysis processing unit 31, even if there are characters in the diagram or table, the contents may not be understood if the layout is not broken and displayed as it is. . Therefore, it determines with not converting into a reflow type.

  Next, the third condition will be described. If the number of lines in the character string extracted by the line analysis processing unit 31 is, for example, 5 lines or less (THln = 5), it is considered that the number of lines is not enough to be read in the reflow type. Therefore, it determines with not converting into a reflow type. The threshold (THln) can be determined by, for example, setting the threshold for the number of lines that can be displayed on one screen according to the size of the screen to be displayed and the character size of the lines.

  Next, the fourth condition will be described. If the height of a horizontal writing line (width in vertical writing) is, for example, 40 pixels or more (THcs = 40) with respect to all the character string lines extracted by the line analysis processing unit 31, Is sufficiently large, it is considered that it is sufficiently readable even if it is reduced in size. Therefore, it determines with not converting into a reflow type. The threshold value (THcs) is determined by, for example, 1 in the original image based on the size of the screen to be displayed, the reduction ratio based on the size of the entire image, and the number of pixels of the minimum font size that can be read when the image is reduced. It is conceivable that the number of pixels of the character size is calculated and used as a threshold value.

  Next, the fifth condition will be described. If the positions of the horizontal or vertical row blocks ordered by the layout analysis processing unit 33 are not within a certain range, there is a high possibility that the layout is not organized and arranged in a row. There is a high probability that the connection will fail and connect the wrong row blocks. Therefore, it determines with not converting into a reflow type. For example, consider a document image having a plurality of vertically written row blocks and a plurality of charts as shown in FIG. For this document image, the deviation of the position of the row blocks ordered by the layout analysis processing unit 33 is obtained from the block size and position information of the row blocks. Here, if THrg = 40 is set as a threshold value (THrg) as to whether or not they are aligned within a certain range, any row block exceeds the threshold value (THrg) and is not aligned within the certain range. In such a case, if rows are connected in order from the top in block units, they are connected in the order of vertical writing 1 → vertical writing 2 → vertical writing 4 → vertical writing 3 and the order is not correct. Further, when rows are connected in order from the right in block units, they are connected in the order of vertical writing 1 → vertical writing 3 → vertical writing 2 → vertical writing 4, and this is not the correct order. In this way, there is a high possibility that the row connection order will fail. Therefore, in the case of a document image in which the block positions are not aligned in this way, it is determined that the document image is not converted to the reflow type.

  The processing for displaying the document image (page) determined not to be converted into the reflow type by the conversion possibility determination processing unit 34 and the document image determined to be converted into the reflow type in a mixed manner will be described later in the section <9-3>. Will be explained.

[8. Relocation processing section]
<8-1. Processing of Relocation Processing Unit>
The rearrangement processing unit 35 generates a reference list for the document image determined to be converted into the reflow type by the conversion possibility determination processing unit 34 as follows.

  The rearrangement processing unit 35 calls the rows in the order defined by the layout analysis processing unit 33. If the called line is a character string line, characters are called in order from the beginning of the corresponding character string line. If the called line is a chart line, the corresponding figure or table is called, and the reference information of each element (Information for calling an element) is described in order. Further, if the element is the last element in the line including the line feed position, a line feed command is inserted to generate a reference list.

  That is, in the reference list, the order of each element included in the document image is described along the first direction of the document in the character string according to the ordering of the character string line and the chart line. Here, the reference list can be described in a markup language format represented by HTML. Below, the case where it describes as an HTML file is taken as an example, and the detail of the rearrangement process part 35 is demonstrated.

  FIG. 25 is a flowchart illustrating a processing procedure of the rearrangement processing unit 35. The processing contents of the rearrangement processing unit 35 will be described below with reference to FIG.

  As shown in FIG. 25, first, a file header is described (S11). The file header describes various information of the file. For example, in the case of an HTML file, as shown in FIG. 26, a declaration that the file is described in the HTML language (<HTML> tag), information on a file that is not described in the text, such as a style definition and a page title. Etc., a comment, a declaration that the body description starts (<BODY> tag), and the like.

  Next, the body text is described. Here, using the 0th paragraph of the document structure tree generated by the layout analysis processing unit 33 as an initial call position, it is determined whether or not paragraph information can be called from the document structure tree at the call position (S12). If the call is possible (YES at S12), a paragraph is called from the document structure tree, and it is determined whether the paragraph has at least one line (S13). When the paragraph has at least one line (Yes in S13), information on all lines constituting the paragraph is sequentially called, and elements included in the line are referred to and displayed in a file. A file description process for performing the description is executed (S14). On the other hand, if the paragraph has no lines (NO in S13), the calling position is moved to the next paragraph, and the process returns to S12.

  FIG. 27 is a flowchart showing the procedure of the file description process in S14. For the called paragraph, first, a paragraph start declaration command (see the section <8-2> described later) for declaring that the paragraph starts is executed (S21). Thereafter, the character string lines of the paragraph are sequentially called from the top of the document structure tree, and element reference processing for referring to the character elements included in the line is executed (S22). That is, the elements (characters) are called in order from the top of the line, and the reference instruction for the element is executed. Specifically, the elements are called in order from the top of the line, and the same processing is repeated for all the elements.

  When the reference command is completed for all the elements, the reference process for the line is terminated, the next character string line is called, and the same process is repeated. The above processing is executed for all character string rows (all character string rows included in the character string row tree of the paragraph) of the paragraph in the document structure tree. When the element reference processing for all the character string rows included in the character string row tree of the paragraph is completed, the element reference processing for the chart row is executed (S23). Since a chart row has only one element, if a reference instruction is performed once for each chart row, the next chart row (the next chart row included in the paragraph's chart row tree) of the paragraph is called, A similar reference process is executed. When the reference instruction is executed for all the chart rows included in the chart row tree of the paragraph, finally, a paragraph end declaration (chapter <8-2> described later) is declared to end the file description processing for the paragraph. ) Is executed (S24), and the file description process (S14) is terminated.

  Returning to FIG. 25, the description will be continued. After S14, the call position is updated to the next paragraph, and the process returns to S12.

  The above processing is repeated until the determination in S12 is NO, that is, the file description processing for all the paragraphs in the document structure tree is completed. If the determination in S12 is NO, the footer is described (S15). As shown in FIG. 26, the footer includes a declaration of the end of matters (for example, a description of the body text, a description of the start of HTML language description, etc.) declared to be the start in the header description. When the description of the footer part is finished, the file is saved (S16) and the process ends. The file is output from the rearrangement processing unit 35.

<8-2. Example of each command>
As an example of a paragraph start declaration command, for example, a method of inserting a paragraph tag <p> can be mentioned. In this case, a paragraph end declaration instruction is always inserted into the paragraph tag </ p>. As another method, there is a method of inserting a line feed tag <br> as a paragraph end declaration command without performing any particular paragraph start declaration command. The element reference command includes, for example, a method in which an image display tag <img> is inserted into an output file, and the path of the cut image file of the element is designated and displayed. In addition, the reference instruction for the chart element may be the same as the reference instruction for the character element, and only in the case of the chart element, it can be combined with insertion processing such as a line feed tag <br>, table tags <table>, and </ table>. A layout with higher visibility can be configured.

<8-3. If you want to modify the chart at the beginning of the paragraph>
In the above, the processing procedure in the case of arranging the chart after the text in each paragraph has been described. On the other hand, if it is desired to place the chart in front of the text, it can be easily realized by replacing S22 and S23 in FIG. In addition, the user may be able to specify the chart through the operation panel 6 from a plurality of modes such as placing the chart at the end of the paragraph, placing it at the beginning of the paragraph, or not displaying the chart (displaying only the character string).

<8-4. Definition of document style>
In addition to the above command, the file format of the document may be changed by incorporating a style sheet. A style sheet is a known means for efficiently controlling the appearance and display format of a document structured in an HTML language or the like. Such information such as appearance is called “style”. The style sheet is realized by a dedicated computer language, and examples thereof include CSS (Cascading Style Sheet). There are three main locations for defining style sheets. It includes a method of describing a style by adding a style element (<style> to </ style>) in a header element (between <head> to </ head>), and an external file describing the style. A method of calling the external file in the header element of the reference list, and a method of describing a style that is valid only for the tag in various tags in the body part (between <body> to </ body>). These methods are not just selected, and a style sheet can be defined by combining a plurality of methods. In this embodiment, it is assumed that a known method is used and will not be described in detail below, but only a simple description example will be introduced.

  FIG. 28 is an example of an external file of a style sheet described in the CSS format. In the range A in the figure, the style of the tag <p> that defines a paragraph is defined. In this example, it is defined that an indent for one character is added to the first line of each paragraph. Thereby, when the paragraph start declaration tag is called, an indent for one character can always be added. In the range B and the range C, the style of the tag <img> that refers to the image is defined, but in the range B, the style is limited to the case belonging to the class gajii, and in the range C, the style belongs to the class ig This is a limited style.

  In this way, a plurality of classes having unique styles are defined for the same tag and saved as an external file (here, the file name is style.css), and an example shown in FIG. Read the external file (same as in FIG. 28) as shown in FIG. 29 (a) in the header as described in the HTML language (partially excerpted) (3 lines in FIG. 29 (b)). Eye) By specifying a class having a desired style in each tag, it is possible to specify a local style. In the example of FIG. 28 and FIG. 29, when referring to an image, if it is an image representing a character, by describing class = “gaiji” as shown in FIG. 29B in the img tag, A gaiji (external character) class is specified, and in the case of an image representing a chart, a class (“fig”) is specified by describing class = “fig”, and thus an appropriate style is specified for each element. By defining it, it becomes possible to display a document with better appearance.

  Further, these styles may be selected and designated on the operation panel 6. For example, apart from the first document direction determined from the document image data, the user may be able to specify the direction of the document at the time of rearrangement through the operation panel 6. Specifically, by defining the style sheet as html {writing-mode: tb-rl;} in the CSS format in the HTML language format, the line direction of the document of the entire body is changed from top to bottom, and the line is further to the right Starting from, it goes to the left, that is, vertical writing is possible.

  Therefore, when the reflow type display in the vertical writing mode is selected, the vertical writing display can be realized by adding the style sheet definition as described above. Note that a method other than the above-described method can be used as a method for realizing vertical writing. In addition, “automatic mode” may be prepared as the document direction at the time of output, and the document may be automatically selected to be output in the same direction as the first direction of the document obtained by the line analysis processing unit 31.

<8-5. Other file formats>
In the rearrangement processing unit 35 of the present embodiment, the case where an HTML file is output from an input document image has been described. However, the output file is not limited to an HTML file, and any file format that realizes reflow display. (For example, an XML file, an XMDF file, etc.) can be selected.

  In the present embodiment, as described above, the structure is analyzed from the document image, the line order is understood by referring to the line order, and the characters and diagrams constituting the line are further referred from the top in order. By doing so, even if the document image is generated as a fixed image file, information (reference file) for conversion as a reflow file can be obtained. In addition, by determining the presence or absence of line breaks and defining the range of paragraphs, in addition to preventing line breaks within a single paragraph even if the document image has line wrapping, a figure or table subordinate to each paragraph If the figure is placed within the beginning or end of the paragraph and the figure or table is sandwiched between the strings, it can be considered to be within the same paragraph. The continuity and readability of the text can be improved by modifying the layout so that the figure or table is not sandwiched as it is.

  As described above, the reference list is information for conversion as a reflow type file according to the ordering of character string lines and chart lines and the document description direction. As a reference list, a document image converted into a file format that can be displayed in reflow format can be generated, reducing the amount of processing and enabling a layout arrangement suitable for each display area (display screen) of any display device. Become. By using the reference list, the display device can realize display only by scroll operation in a direction perpendicular to the document description direction of the document image.

  Also, by generating the reference list as text data listing the document structure, the reference list can be used for analyzing the document structure instead of for browsing. In addition, by converting the output text format reference list into a desired file format, it is not necessary to perform the process from the beginning when it is desired to output in another file format.

[9. Display device]
<9-1. Use of reference lists on display devices>
The generated reference list and the image data of each extracted element are transmitted from the transmission device 4 included in the image forming apparatus 100 of the present embodiment, and are a display device (for example, a smartphone or a tablet) that is a reception-side device (not shown). Can be viewed through a viewing program (viewer) which is an application provided in the device. The optimal browsing program differs depending on the file format taken by the reference list. For example, when the HTML file format is used as the reference list, it is opened using a well-known HTML browser such as Internet Explorer (registered trademark). Thus, it is possible to easily realize a reflow type display.

  It should be noted that the document structure obtained by the relocation processing unit 35, such as the order of referring to each element (character, figure, table) and the start declaration and end declaration of paragraphs, without being converted into a specific markup language format file, It is also possible to generate a reference list as listed text data. This reference list can be used, for example, as an analysis result of the document structure, or can be converted into a desired file format by a computer program (conversion program) provided in the receiving apparatus. In addition, using two stages of file format conversion in this way increases the number of processing steps, but uses a common reference list when you want to convert the same document image data to multiple file formats. It becomes possible. Furthermore, the layout analysis result can be manually corrected in the conversion program.

  When the display apparatus on the receiving side receives a reference list transmitted from the image forming apparatus 100 that has been converted into a file format (such as HTML) suitable for a viewer application provided in the display apparatus, the application simply reads the file in the application. There is no special processing because it is possible to display the converted document. However, when the reference list is not associated with a specific viewer, for example, when the coordinate information is listed as the simplest format, the reflow display cannot be performed as it is. When such a reference list is received, a process for rearrangement is necessary, but a known process can be used.

  In addition, the folded portion of the character referred to by the reference list is changed according to the display width of the display area of the display device, and the figure or table is reduced or enlarged according to the display width. For example, as described in <8-4> above, when assigning a fig class to a figure or table and outputting it in the HTML file format, the width (or height) of the image is set to the display width (or height) of the display device. Thus, even when the display width or display magnification of the display device is changed, it is possible to automatically adjust according to the display width (or height) at that time.

As an example of the method for setting the image width as a ratio to the display width (in the case of horizontal writing) of the display device as described above, for example, only those belonging to the fig class when the img tag is called are as the size attribute of the img tag as follows: Add a percentage to the
<Img class = “fig” src = ”˜˜“ width = “90%” /> (8)
Alternatively, add the following as a style sheet for the fig class in the CSS format.
.fig {width: 90%;} (9)
There are methods. In any of the methods (8) and (9), 90% of the display width is set as the width of the figure or table, and the display width after the change is automatically changed when the display area is changed. Reset 90% as the width of the figure or table. (8) applies only to the img tag to which the setting of the size attribute described above is added, that is, it is applied individually, whereas (9) is a point that sets the diagram to which the fig class is assigned collectively. Different. Of course, it is also possible to use known methods other than those described above.

<9-2. Another example of chart display>
In the above <8-1> and <8-3>, the method of displaying a chart at the end or the beginning of a paragraph has been disclosed, but separately from that, the chart is not displayed in the same file as the main text. Alternatively, a list of links to images (arrange charts in order from the first paragraph) may be created separately so that images selected by the user when desired from the list of links can be displayed individually.

  For example, in a display device that receives a reference list generated by the image forming apparatus 100 and an image of each extracted (cut out) element, a content display area for displaying content and an operation area for receiving user operations are individually provided. The operation area includes means for executing an operation function for switching between the display of the text and the image link list. With such a configuration, the contents displayed in the content display area can be switched between the text and the chart when the user desires.

  Further, the display device may be configured such that a content display area and an operation area are individually provided, and links to each image are displayed in the operation area in order from the top of the link list. As a method for displaying the link, for example, a code (preferably capable of recognizing the order such as FIG. 1, FIG. 2,...), An image thumbnail, or the like can be used. When a link is selected, a chart corresponding to the selected link is displayed. For example, in addition to the method of displaying in the content display area, the chart display method adopts a so-called pop-up format in which a chart display area is generated by being superimposed on the content display area and the chart corresponding to the chart display area is displayed. May be.

  When the display device includes the content display area and the operation area as described above, the operation area does not necessarily have to be separated from the content display area, and may be displayed so as to be superimposed on the content display area. The operation area may not be always displayed but may be displayed on the screen only when a display command is input. For example, in the display command input method, in the range corresponding to the display area of the touch panel included in the display device, the touch panel is kept touched without leaving the touch coordinates at a certain distance or longer for a certain time or longer. (So-called long press).

  Note that the above method itself does not improve the accuracy of the order of the chart and the association with the corresponding paragraph, but does not fix the display position of the chart, and "(almost) of the chart arranged in the order of the text" When the user himself / herself selects and browses from the “list”, it is possible to eliminate a sense of incongruity when reading a sentence having a complicated layout that is difficult to associate with the corresponding paragraph.

<9-3. Display of reflow type and fixed type documents>
Processing for displaying the page that the conversion possibility determination processing unit 34 determines to be unconvertible to the reflow type and the page determined to be convertible to the reflow type will be described below.

  In the case of the fixed type, the entire page can be handled as one image, and can be displayed in a mixed manner with the reflow type. However, if a fixed page treated as an image is mixed with the reflow type and displayed, the image is displayed in full screen. For this reason, there are pages where it is difficult to read characters. In that case, if the display device is configured to be able to switch between reflow type and fixed type display, it is possible to display in a format that is easy to read according to each page.

  For example, when displaying on a web browser using HTML5 or the like, a switching button (switching unit) between a reflow type display and a fix type display is displayed in the display area of the browser, and the reflow is performed in the display area according to the pressed button. It should be possible to switch between the display in the type and the display in the fixed type. For example, as the switching button, a button for selecting each type may be provided, or a toggle-like switch may be provided so that any type can be selected.

[Embodiment 2: Image reading apparatus]
In Embodiment 1, the configuration in which the document image processing apparatus according to the present invention is applied to the image processing apparatus included in the image forming apparatus has been described. However, the present invention is not limited to this. Therefore, in the present embodiment, an example in which the document image processing apparatus according to the present invention is applied to an image processing apparatus included in an image reading apparatus such as a flatbed scanner as a conversion processing unit will be described.

  In addition, about the member which has the same function as the member described in drawing used for description of Embodiment 1, the same code | symbol is attached | subjected also in the following description. The detailed description of each member will not be repeated here.

  FIG. 30 is a block diagram illustrating a configuration of an image reading apparatus (information processing apparatus) 200 including the image processing apparatus 1a according to the second embodiment. As shown in FIG. 30, the image reading apparatus 200 includes an image processing apparatus 1 a, an image input apparatus 2, a transmission apparatus 4, a storage unit 5, and an operation panel 6. The image processing apparatus 1a includes an A / D conversion unit 11, a shading correction unit 12, a document type determination unit 13, an input tone correction unit 14, a region separation processing unit 15, a compression processing unit 21, and a conversion processing unit (document image processing Device) 22. In the conversion processing unit 22, a reference list is generated in the same manner as described in the first embodiment.

  Various processes executed by the image reading apparatus 200 are controlled by a control unit (a computer including a processor such as a CPU or DSP) provided in the image reading apparatus 200.

  In the present embodiment, the image reading apparatus 200 is not limited to a scanner. For example, a digital still camera, a document camera, or an electronic device equipped with a camera (for example, a mobile phone, a smartphone, a tablet terminal, etc.) ). Even if these cameras or electronic devices equipped with cameras are configured to be able to perform structural analysis of document images on their own devices and display them in a reflow type or a fixed type on their display units Good.

[Embodiment 3: Document image processing apparatus via network]
In the above description, the document image processing apparatus according to the present invention is applied to the image processing apparatus 1 included in the image forming apparatus 100 or the image processing apparatus 1a included in the image reading apparatus 200. However, the present invention is not limited to this. The document image processing apparatus according to the present invention may be applied to, for example, a server apparatus. An example of the configuration of the server apparatus in this case is described in the first embodiment with the receiving apparatus that receives the document image that has been subjected to image reading and various image processing by the image forming apparatus or the image reading apparatus, via the network. A server apparatus comprising: a document image processing apparatus that executes processing in the conversion processing unit 22; and a transmission apparatus that transmits files (document images and reference lists) output from the document image processing apparatus via a network. Information processing apparatus).

  By configuring the server device in this manner, a document image that has been subjected to image reading and various image processing by the image forming device or the image reading device is received via the network, and the conversion processing unit 22 performs the above processing. It is possible to use such a method that a reference list is generated by a document image processing apparatus that executes the above processing (format conversion is applied), and an output file is transmitted to a user terminal device (for example, a smartphone or a tablet terminal). Become. In addition, the server apparatus can use the format conversion function without replacing an already installed image forming apparatus or image reading apparatus. Further, by storing the file after format conversion in the server device, it is possible to receive and browse the converted file when the user desires.

  Or you may apply the document image processing apparatus which concerns on this invention to communication terminal apparatuses, such as a mobile telephone, a smart phone, a tablet terminal, an electronic book exclusive terminal, for example. An example of the configuration of the communication terminal device in this case will be described in the first embodiment, a receiving unit that receives a document image that has been subjected to image reading and various image processing by the image forming device or the image reading device, and a network. The communication terminal device (information processing device) includes the conversion processing unit 22 and the display device described in the first embodiment. The communication terminal device can receive the digitized document image, analyze the structure of the received document image, and display it in a reflow type or a fixed type.

  The document image received by the receiving device of the server device or the communication terminal device need not be a document image generated by an image forming device, an image reading device, or the like. For example, an electronic version of a structured document file such as a Word file or a PDF file may be received as a document image.

[Embodiment 4: Recording Medium / Program]
The image processing devices 1 and 1a described above (particularly, the conversion processing units 22 and 22a), the server device (particularly the document image processing device), and the communication terminal device (particularly the conversion processing unit) are integrated circuits (IC chips). It may be realized by a logic circuit (hardware) formed in the above, or may be realized by software using a CPU.

  In the latter case, the image processing apparatuses 1 and 1a, the server apparatus, and the communication terminal apparatus can read a program command that is software that realizes each function, and the program and various data can be read by a computer (or CPU). A ROM (Read Only Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like are provided. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

  The document image processing apparatus and the document image processing method described above are configured to handle color image data, but are not limited thereto, and may be configured to handle monochrome image data.

  The present invention is not limited to the above-described embodiments, and various modifications can be made. That is, embodiments obtained by combining technical means appropriately changed within the scope not departing from the gist of the present invention are also included in the technical scope of the present invention.

[Summary]
A document image processing apparatus (conversion processing unit 22) according to an aspect 1 of the present invention is a document image processing apparatus that performs reconstruction of a document image obtained by digitizing a document. Analysis processing unit 31, line block analysis processing unit 32, layout analysis processing unit 33) and each character included in the document image based on the character string or the feature quantity of the figure or table extracted from the document image by the structural analysis. A conversion determination unit (conversion possibility determination processing unit 34) that determines whether to reconstruct each element that is a character, figure, and / or table, and the conversion determination unit reconstructs each element of the document image Then, based on the analysis result by the structure analysis unit, a reference list generation unit (relocation processing unit 35) that generates a reference list describing the order of the elements when the document image is reconstructed, The Obtain.

  According to the above configuration, it is possible to determine whether or not each element included in the document image is to be reconfigured based on the character string extracted from the document image by structural analysis, or the feature quantity of the figure or table. In this way, each element included in the document image is reconfigured, that is, the document image is made into a reflow type file format, or each element included in the document image is not reconfigured, that is, a fixed type (fixed type) It is possible to always display in the optimum file format by determining in the document image processing apparatus whether the file format is layout). Also, based on the analysis result by the structure analysis unit, generating information for converting the document image into a reflow file format by generating a reference list describing the order of each element when the document image is reconstructed Can do.

  In the document image processing apparatus according to aspect 2 of the present invention, in the aspect 1, when the document description direction in the document image cannot be determined from the character string extracted by the structural analysis, the conversion determination unit includes: It is determined that the document image is not reconstructed.

  If the document image contains a mixture of vertical and horizontal lines in a ratio that cannot be ignored, the direction of the entire document image cannot be uniquely determined. It becomes difficult to determine how to connect. Therefore, it becomes difficult to correctly convert to the reflow type. Therefore, it is possible to prevent the mistake that the conversion determination unit cannot correctly convert to the reflow type by performing the above determination with the above configuration.

  In the document image processing apparatus according to aspect 3 of the present invention, in the aspect 1, when the structure analysis determines that the document image is composed only of a figure and / or a table, the conversion determination unit includes: It is determined that the document image is not reconstructed.

  Even if there are characters in the figure and / or table, the contents often cannot be understood unless they are displayed as they are without breaking the layout. Therefore, the conversion determination unit makes the above determination according to the above configuration, thereby making it possible to prevent a document image made up of only figures and / or tables from being erroneously converted into a reflow type and unclear. Become.

  In the document image processing apparatus according to aspect 4 of the present invention, in the above aspect 1, the conversion determination is performed when the number of character strings extracted from the document image by the structural analysis is equal to or less than a predetermined first threshold value. The unit determines that the document image is not reconstructed.

  When there are few character strings, it is thought that it is not the length of the sentence which is reflow type and read. Therefore, it is possible to prevent unnecessary conversion into the reflow type when the conversion determination unit performs the above determination with the above configuration.

  The document image processing apparatus according to aspect 5 of the present invention is the above-described aspect 1, wherein the height or width of the character string extracted from the document image by the structural analysis is equal to or greater than a predetermined second threshold value. The conversion determination unit determines not to reconstruct the document image.

  When the characters are sufficiently large, it is considered that the characters are sufficiently readable even if they are displayed in a reduced size, so that it is not necessary to use the reflow type. Therefore, it is possible to prevent unnecessary conversion into the reflow type when the conversion determination unit performs the above determination with the above configuration.

  In the document image processing apparatus according to aspect 6 of the present invention, the position of each of a plurality of character strings or blocks made up of figures or tables extracted from the document image by the structural analysis is irregular. In this case, the conversion determination unit determines not to reconstruct the document image.

  If the positions of blocks consisting of character strings or figures or tables are not aligned within a certain range, there is a high possibility that the document images are not arranged in an organized and arranged layout. Therefore, there is a high possibility that connection of row blocks will fail and incorrect row blocks will be connected. Therefore, it is possible to prevent a mistake that conversion to the reflow type cannot be performed correctly when the conversion determination unit performs the above determination with the above configuration.

  An information processing apparatus according to an aspect 7 of the present invention fixes a document image processing apparatus according to any one of the above aspects 1 to 6 and a document image that is determined not to be reconstructed by the conversion determination unit. A format conversion processing unit that performs format conversion so that the type can be displayed, and a document image determined to be reconstructed by the conversion determination unit, the reference list of the document image and each element included in the document image And a transmission device that transmits the format-converted document image for a document image that is determined to be not reconstructed by the conversion determination unit.

  The information processing apparatus may be, for example, an image forming apparatus, an image reading apparatus, a server apparatus, or the like. In addition, the image reading apparatus may be a scanner, a digital still camera, a document camera, or an electronic device (for example, a mobile phone, a smartphone, a tablet terminal, or the like) equipped with a camera. When the information processing apparatus is, for example, an image forming apparatus, the read image data is subjected to conversion processing to reconstruct image data, and the reconfigured image data is designated by an address. By transmitting to the tablet terminal or the like, the image can be viewed only by scrolling in one direction (direction orthogonal to the document description direction) on the receiving side device.

  A display device according to an aspect 8 of the present invention is a display device that displays a document image received from the information processing device according to the aspect 7, and the document image determined to be reconstructed by the conversion determination unit. Based on the reference list and the image data of each element included in the document image, the document image is displayed in a reflow type, and the format-converted document image is displayed in a fixed type.

  According to the above configuration, the document image determined to be reconstructed by the conversion determination unit can be displayed in a reflow type, and the format-converted document image can be displayed in a fixed type. The original document in which the fixed page and the fixed page are mixed can be displayed.

  The display device according to aspect 9 of the present invention is the display device according to aspect 8, in which the document image determined to be reconstructed by the conversion determination unit from the information processing device according to aspect 7 is displayed in a fixed type. A switching unit that receives a document image that has been subjected to format conversion and that switches the display of the document image determined to be reconstructed by the conversion determination unit between a reflow type and a fix type.

  According to the above configuration, the user can switch the display of the document image determined to be reconstructed by the conversion determination unit between the reflow type and the fix type.

  The document image processing device, the information processing device, or the display device may be realized by a computer. In this case, there is also provided a program for realizing the document image processing device, the information processing device, or the display device on a computer by causing the computer to operate as each unit, and a computer-readable recording medium storing the program. Falls within the scope of the present invention.

  The present invention can be used in a document image processing apparatus that reconstructs a document image.

1, 1a Image processing device 4 Transmission device 22, 22a Conversion processing unit (document image processing device)
31 line analysis processing part (structure analysis part)
31a Character string extraction processing unit 31b Chart extraction processing unit 32 Line block analysis processing unit (structure analysis unit)
33 Layout Analysis Processing Unit (Structural Analysis Unit)
34 Conversion enable / disable determination processing unit (conversion determination unit)
35 Relocation processing unit (reference list generation unit)
37 Column analysis processing unit 38 Ordering processing unit 39 Paragraph analysis processing unit (paragraph analysis unit)
100 Image forming apparatus (information processing apparatus)
200 Image reading device (information processing device)

Claims (8)

In the document image processing apparatus,
A structure analysis unit for analyzing the structure of a document image obtained by digitizing a document;
It is determined whether or not the character string feature amount including at least one of the description direction, number, size per character, and position of the character string extracted from the document image by the structural analysis satisfies a predetermined condition. If it is determined that the character image is satisfied, each character, figure, and / or table included in the document image is displayed in a display area so that the character string in the document image can be displayed in a folded manner. A conversion determination unit that determines to reconfigure
A reference list generation unit that generates a reference list for reconstructing, describing the order of each element of the document image based on the analysis result by the structure analysis unit, and
When the conversion determination unit determines to reconstruct each element of the document image, the reference list generation unit is used to generate the reference list ,
The feature amount is a description direction of the character string, and the conversion determination unit determines that the predetermined condition is not satisfied when a ratio of vertical writing lines and horizontal writing lines is larger than a predetermined threshold, and the document image Is determined not to be reconfigured,
For a document image determined to be reconstructed by the conversion determination unit, the reference list of the document image and image data of each element included in the document image are output to a transmission device. Image processing device. In the document image processing apparatus,
A structure analysis unit for analyzing the structure of a document image obtained by digitizing a document;
It is determined whether or not the character string feature amount including at least one of the description direction, number, size per character, and position of the character string extracted from the document image by the structural analysis satisfies a predetermined condition. If it is determined that the character image is satisfied, each character, figure, and / or table included in the document image is displayed in a display area so that the character string in the document image can be displayed in a folded manner. A conversion determination unit that determines to reconfigure
A reference list generation unit that generates a reference list for reconstructing, describing the order of each element of the document image based on the analysis result by the structure analysis unit, and
When the conversion determination unit determines to reconstruct each element of the document image, the reference list generation unit is used to generate the reference list ,
The feature amount is the height per character of a horizontal character string line or the vertical character string line.
It is a width per character, and the conversion determination unit determines the height or the width in advance.
When it is greater than or equal to a second threshold, it is determined that the predetermined condition is not satisfied, and the document image is reconstructed
And decide not to
For a document image determined to be reconstructed by the conversion determination unit, the reference list of the document image and image data of each element included in the document image are output to a transmission device. Image processing device. In the document image processing apparatus,
A structure analysis unit for analyzing the structure of a document image obtained by digitizing a document;
It is determined whether or not the character string feature amount including at least one of the description direction, number, size per character, and position of the character string extracted from the document image by the structural analysis satisfies a predetermined condition. If it is determined that the character image is satisfied, each character, figure, and / or table included in the document image is displayed in a display area so that the character string in the document image can be displayed in a folded manner. A conversion determination unit that determines to reconfigure
A reference list generation unit that generates a reference list for reconstructing, describing the order of each element of the document image based on the analysis result by the structure analysis unit, and
When the conversion determination unit determines to reconstruct each element of the document image, the reference list generation unit is used to generate the reference list ,
The feature amount is a size of a position shift between a plurality of blocks made up of a plurality of character strings or figures or tables, and the conversion determination unit determines that the size of the shift is larger than a predetermined third threshold value. The document image is determined not to be reconstructed, the document image is determined not to be reconstructed, and the document image determined to be reconstructed by the conversion determining unit is the reference list of the document image and A document image processing apparatus that outputs image data of each element included in the document image to a transmission apparatus. A document image processing apparatus according to any one of claims 1 to 3 ,
A format conversion processing unit that converts the format of a document image that is determined not to be reconstructed by the conversion determination unit so that a fixed-type display is possible;
For the document image determined to reconstruct each element by the conversion determination unit, the reference list of the document image and the image data of each element included in the document image are transmitted, and the conversion determination unit An information processing apparatus comprising: a transmission device that transmits the format-converted document image for a document image determined not to be reconstructed. A display device for displaying a document image received from the information processing device according to claim 4 ,
Based on the reference list of document images determined to reconstruct each element by the conversion determination unit and the image data of each element included in the document image, the document image is displayed in a reflow type, and the format A display device that displays a converted document image in a fixed format. A document image obtained by converting the document image determined to be reconstructed by the conversion determination unit from the information processing device according to claim 4 in a format that enables fixed display is received.
The display device according to claim 5 , further comprising a switching unit that switches a display of the document image determined to be reconstructed by the conversion determination unit between a reflow type and a fixed type. A program for operating the document image processing apparatus according to any one of claims 1 to 3 , wherein the program causes a computer to function as each unit. A computer-readable recording medium on which the program according to claim 7 is recorded.

Images