JP6607490B2 - CONVERSION PROCESSING DEVICE, INFORMATION PROCESSING DEVICE EQUIPPED WITH THE SAME, PROGRAM, AND RECORDING MEDIUM - Google Patents

Description

  The present invention relates to a conversion processing apparatus, an information processing apparatus including the conversion processing apparatus, a program, and a recording medium.

  2. Description of the Related Art Conventionally, image forming apparatuses such as copiers or multi-function machines that perform image formation by an electrophotographic method or an ink jet method have been widely used. In recent years, advanced functions have been advanced in multifunction peripherals, and a function of storing a document read by a scanner as document image data (hereinafter also referred to as document image information or document image) and managing the stored document image. Etc. are required. In general, since a document read by a scanner is stored as image data, the read document cannot be re-edited as it is. On the other hand, there is a demand to acquire and re-edit information of characters and diagrams included in the read document image, and further, the document creation software recognizes the structure of characters, diagrams and tables included in the document image. In addition, there is an increasing demand for a function for reconstructing an office document file created and used by spreadsheet software or presentation software.

  As a technology that supports such an office document file conversion processing function, a technology for extracting and analyzing elements such as characters, diagrams, photographs, and tables included in a document image is required. Many scanners and the like are already equipped with a function of analyzing a character image included in a document image and converting it into information such as a character code by using an optical character recognition (OCR) technique. As a result, a collection of character images in a document image is generally acquired as editable text data. Various functions have also been proposed for separating and extracting diagrams, photographs, and table areas included in document images. Also, considering the use cases such as recreating forms that are stored only on paper media and inputting a large amount of experimental data into spreadsheet software, it is considered that the frequency of table reconstruction is particularly high. However, when a document having the same layout is manually created from a document image, the work of reconfiguring the table is particularly troublesome work, and automation of the reorganization of the table is becoming increasingly important.

  A technique for accurately reconstructing a table from such a table image has been developed. For example, the technique described in Patent Document 1 proposes an image extraction device that can accurately extract a frame from an image including characters that touch the frame and restore the characters and the like to high quality. In this image extraction device, a partial pattern in which pixels are connected to each other is extracted from a pattern constituting an image, a frame is extracted based on each extracted partial pattern, and characters and frames are extracted based on the extracted partial pattern and frame. The intersection point with is calculated. Adaptively change the distance and inclination continuity criteria between character lines that sandwich the frame according to the line width of the frame, and associate the intersections calculated based on the distance and inclination continuity. The character line segment in the frame is extracted based on the intersection. As a result, even if there are multiple character frames separated by rectangles whose size and position are not known, and handwritten characters are written in contact with or protruding from the character frame It is possible to cut out only one character or one character from a character and character frame pattern.

Japanese Patent Laid-Open No. 11-353415

  However, in the case of the technique disclosed in Patent Document 1, when a table area is extracted from image data of an electronic document and converted into reusable table data, image objects such as photographs and illustrations exist in the table. In some cases, there is no means of recognizing it as an image object rather than a character. For this reason, there is a problem that the image object may be erroneously recognized as a character and the image object may be reconfigured as an incorrect character.

  Therefore, in the present invention, even when an object other than a character such as an image exists in a table cell, the object is correctly extracted and converted without erroneously extracting the object as a character, and the object is converted into the table. It is an object of the present invention to provide a conversion processing device capable of correctly arranging objects, an information processing device including the same, a program, and a storage medium.

  In order to solve the above problem, one aspect of the present invention provides a character extraction processing unit that extracts a character region existing in document image information, a line extraction processing unit that extracts a line segment existing in the document image information, Using the line segment information extracted from the line extraction processing unit, a table region extraction processing unit for extracting a table region and a predetermined local region for the document image information are set, and the local region A luminance histogram is created to determine luminance change information of the local region, the luminance change information, information on the character region extracted by the character extraction processing unit, and a line segment extracted by the line extraction processing unit Using the information and the table area information extracted by the table area extraction processing unit, an image object area including a figure or a photograph outside the table area or in the table area is extracted. Figure The table structure is analyzed based on the area extraction processing unit, the information on the character area in the table area, the information on the line segment, and the information on the image object area, and the table structure information for reconfiguring the table is acquired. A conversion processing apparatus comprising: a table structuring processing unit.

  Further, according to an aspect of the present invention, in the invention described above, the character region extracted by the character extraction processing unit from the document image information includes a character string region including a character string, The figure region extraction processing unit sets a predetermined local region for the document image information, creates a luminance histogram of the local region to obtain luminance change information of the local region, and the luminance change information; Using the character string region information extracted from the character extraction processing unit, the line segment information extracted from the line extraction processing unit, and the table region information extracted from the table region extraction processing unit The image object area including a figure or a photograph existing outside the table area or in the table area is extracted, and the table structuring processing unit includes information on the character string area in the table area, Line segment Distribution, and the analyzes based on the table structure information of the image object area may also be characterized by acquiring the table structure information to reconstruct the table.

  In addition, according to one aspect of the present invention, in the above-described invention, the character string area object, the table area object, the line segment object, the image object order, or the character string area object And a file description section described in a file format designated to arrange the objects in the order of the table area object, the image object, and the line segment object.

  According to another aspect of the present invention, in the above-described invention, the figure region extraction processing unit performs edge detection on the document image information, and extracts from the character extraction processing unit based on the result of the edge detection. A non-character string map generation processing unit that generates a non-character string map that is a candidate for the image object region by excluding a character string region that is one of the document components to be processed, A non-character string area addition processing unit that calculates the entropy of a histogram in a local region of the document image information, and adds a region having a high value of the calculated entropy as a candidate for the image object region to the non-character string map; A line segment of the table area extracted by the table area extraction processing unit for the non-character string map to which the candidate for the image object area is added. The object is obtained by labeling the image object area by performing a labeling process on the non-character string map from which the line segment of the table area is deleted, and obtaining a rectangular area of the labeled image object area. And an object map generation processing unit that generates a map.

  According to another aspect of the present invention, in the above-described invention, the figure region extraction processing unit performs the integration processing of the rectangular regions or the division processing of the rectangular regions for each of the rectangular regions of the image object region. An effective object area determination processing unit may be provided.

  Further, according to one aspect of the present invention, in the above-described invention, the effective object area determination processing unit may be configured such that when the rectangular areas of the plurality of image object areas overlap, the rectangular areas of the plurality of image objects. A maximum value and a minimum value of coordinates may be calculated, and overlapping image object regions may be integrated into one rectangular region.

  According to another aspect of the present invention, in the above-described invention, the valid object area determination processing unit is configured to display the image when the character string area overlaps the rectangular area of the image object area. It is also possible to calculate the maximum value and the minimum value of the coordinates of the object rectangular area and the character string area, and to combine the character string area overlapping the image object area into one rectangular area. Good.

  Further, according to one aspect of the present invention, in the invention described in the above, the valid object area determination processing unit is configured to display the image object when the table area overlaps a rectangular area of the image object area. The area where the table area overlaps from the rectangular area of the area is excluded, and the image object area along a horizontal frame line or an extension of the vertical frame line of the overlap table area The rectangular area may be divided.

  Another embodiment of the present invention is an information processing device including the conversion processing device described above.

  According to another aspect of the present invention, there is provided a computer, a character extraction processing unit that extracts a character region existing in document image information, a line extraction processing unit that extracts a line segment existing in the document image information, and the line extraction processing unit. Table area extraction processing means for extracting a table area using the extracted line segment information, setting a predetermined local area for the document image information, and creating a luminance histogram of the local area The luminance change information of the local region is obtained, the luminance change information, the information of the character region extracted by the character extraction processing unit, the line segment information extracted by the line extraction processing unit, and the table region The table object information extracted by the extraction processing means is used to extract an image object area including a figure or a photograph outside or within the table area. The table structure is analyzed based on the figure region extraction processing means, the information on the character region in the table region, the information on the line segment, and the information on the image object region, and the table structure information for reconstructing the table is obtained It is a program for functioning as a table structuring processing means.

  One embodiment of the present invention is a computer-readable recording medium on which the program described above is recorded.

  According to the present invention, even when an object other than a character such as an image exists in a table cell, the object is correctly extracted and converted without erroneously extracting the object as a character, and the object is converted into the table. It becomes possible to arrange objects correctly.

1 is a block diagram illustrating a configuration of an image forming apparatus according to Embodiment 1 of the present invention. It is a block diagram which shows the structure of the conversion process part by the embodiment. It is a figure (the 1) which shows an example of the table structure information by the embodiment. It is a figure (the 2) which shows an example of the table structure information by the embodiment. It is a figure which shows an example of the file structure of a Word file. It is a figure which shows an example of the table described using the markup language. It is a block diagram which shows the structure of the figure area | region extraction process part by the embodiment. It is a figure which shows an example of the table | surface image by the embodiment. It is a flowchart which shows the flow of a process by the non-character string map production | generation process part by the embodiment. It is a figure which shows an example of the Laplacian filter applied to the embodiment, and an application result. It is a figure which shows an example of the table image to which the process by the non-character string map production | generation process part by the embodiment was applied. It is a flowchart which shows the flow of a process by the non-character string area addition process part by the embodiment. It is FIG. (1) which shows an example of the table image to which the process by the non-character string area addition process part by the same embodiment was applied. It is FIG. (2) which shows an example of the table image to which the process by the non-character string area addition process part by the same embodiment was applied. It is a flowchart which shows the flow of a process by the object map production | generation process part by the embodiment. FIG. 6B is a diagram illustrating an example of a table image to which the process by the object map generation processing unit according to the embodiment is applied (part 1); It is FIG. (2) which shows an example of the table image to which the process by the object map production | generation process part by the embodiment was applied. It is FIG. (3) which shows an example of the table image to which the process by the object map production | generation process part by the embodiment was applied. It is a flowchart which shows the flow of a process by the effective object area determination process part by the embodiment. It is FIG. (1) which shows an example of the image object to which the process by the effective object area determination process part by the same embodiment was applied. It is FIG. (2) which shows an example of the image object to which the process by the effective object area determination process part by the same embodiment was applied. It is FIG. (3) which shows an example of the image object to which the process by the effective object area determination process part by the same embodiment was applied. It is a block diagram which shows the structure of the image reading apparatus by Embodiment 2 of this invention. It is a figure for demonstrating the modification of brightness | luminance change information.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In an embodiment according to the present invention, when a table area is extracted from image data of an electronic document and converted into reusable table data, a table cell contains characters other than characters such as photographs, diagrams, graphs, and illustrations. Even when an image object exists, the image object is correctly extracted and converted without erroneously extracting the image object as characters, and the image object is correctly arranged in the table. Further, even when the image object areas, the image object area and the character string area, or the image object area and the table area overlap each other, the image object area and the table area are converted with a good appearance when converted into a predetermined file format.

[Embodiment 1]
In the following description, the conversion processing apparatus according to the present invention forms a part of the image processing apparatus 1 as the conversion processing unit 30, and the image processing apparatus 1 forms a part of the image forming apparatus 100. . FIG. 1 is a block diagram illustrating a functional configuration of an image forming apparatus 100 (information processing apparatus) according to the first embodiment. The image forming apparatus 100 is, for example, a digital multifunction machine having a copy function, a scanner function, and the like. The image forming apparatus 100 includes an image processing apparatus 1, an image input apparatus 2, an image output apparatus 3, a transmission apparatus 4, an operation panel 5, and a storage unit 6.

  The operation panel 5 is connected to the image input device 2, the image processing device 1, the image output device 3, and the transmission device 4. The operation panel 5 includes an operation unit (not shown) such as a setting button and a numeric keypad for the user to set the operation mode of the image forming apparatus 100, and a display unit (not shown) configured by a liquid crystal display or the like.

  Various processes executed by the image forming apparatus 100 are controlled by a control unit (a computer including a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor)) (not shown). The control unit of the image forming apparatus 100 performs data communication with a computer and other digital multifunction peripherals connected to the network via a network card (not shown) and a LAN cable.

  Hereinafter, each part of the image forming apparatus 100 will be described in detail. The image input device 2 optically reads an image from a document. The image input device 2 is composed of, for example, a color scanner having a CCD (Charge Coupled Device), and an RGB (R: red, G: green, B: blue) analog signal is obtained from the reflected light image from the original using the CCD. And output to the image processing apparatus 1. The image input apparatus 2 may not be a scanner, and may be a digital camera, for example.

  The image processing device 1 performs processing on the image data read by the image input device 2, and generates a compressed file for storing or transmitting the processed image data. The image processing apparatus 1 performs an A / D conversion unit 10, a shading correction unit 11, a document type determination unit 12, an input tone correction unit 13, and a region separation for RGB analog signals input from the image input device 2. Image data composed of RGB digital signals (hereinafter referred to as RGB signals) is generated by executing image processing to be described later by the processing unit 14.

  The image processing apparatus 1 also performs color correction unit 15, black generation background removal unit 16, spatial filter processing unit 17, output tone correction unit 18, and tone reproduction for the RGB signal output from region separation processing unit 14. The processing unit 19 executes image processing to be described later to generate image data composed of digital signals of CMYK (C: cyan, M: magenta, Y: yellow, K: black), and output the image as a stream. Output to device 3. Note that the image data may be temporarily stored in the storage unit 6 before being output to the image output device 3. The storage unit 6 is, for example, a nonvolatile storage device (for example, a hard disk).

  The image output device 3 outputs an image based on the image data generated by the image processing device 1. The image output device 3 forms (prints) a color image on a recording sheet (for example, recording paper) by a method such as thermal transfer, electrophotography, or ink jet based on the image data input from the image processing device 1. Output. Note that the image output device 3 is not limited to a configuration that outputs a color image, and may be a configuration that forms and outputs a monochrome image (monochrome image) on a recording sheet, for example. In this case, the image processing apparatus 1 converts the color image image data into monochrome image data, and then outputs the image data to the image output apparatus 3.

  Furthermore, in the image processing apparatus 1, the compression processing unit 20 executes image compression processing on the RGB signals output from the region separation processing unit 14, thereby generating a compressed file having image data of a compressed color image. Generate and output to the transmitter 4. Note that the compressed file may be temporarily stored in the storage unit 6 before being output to the transmission device 4.

  When the format conversion mode is selected on the operation panel 5, the conversion processing unit 30 of the image processing apparatus 1 executes format conversion processing on the RGB signals output from the region separation processing unit 14. By this format conversion processing, the conversion processing unit 30 analyzes the document layout of the color image to generate a document structure tree as will be described later, and converts this document structure tree into the format selected by the user on the operation panel 5. And output to the transmission device 4. The conversion processing unit 30 is also a functional unit that functions as a conversion processing device according to the present invention. In addition, the converted file may be temporarily stored in the storage unit 6 before being output to the transmission device 4.

  The transmission device 4 transmits the compressed file generated by the image processing device 1 to the outside. The transmission device 4 can be connected to a communication network such as a public network (not shown), a LAN (Local Area Network), or the Internet, and transmits a compressed file to the outside via the communication network by a communication method such as facsimile or e-mail. To do. For example, when the “scan to e-mail” mode is selected on the operation panel 5, the transmission device 4 using a network card, a modem, etc. attaches the compressed file to the e-mail and sends it to the set destination. Send.

When facsimile transmission is performed, the control unit of the image forming apparatus 100 performs a communication procedure with the other party by the transmission apparatus 4 using a modem or the like, and compression is performed when a transmission possible state is ensured. After performing necessary processing such as changing the compression format on the file, the file is sequentially transmitted to the other party via a communication line.
When receiving a facsimile, the control unit of the image forming apparatus 100 receives a compressed file transmitted from the other party and inputs it to the image processing apparatus while performing a communication procedure by the transmission apparatus 4.

  In the image processing apparatus 1, the received compressed file is decompressed by a compression / decompression processing unit (not shown). The image data obtained by decompressing the compressed file is subjected to rotation processing and / or resolution conversion processing by a processing unit (not shown) as necessary, and the output gradation correction unit 18 outputs the output level. Tone correction is performed, and the gradation reproduction processing unit 19 performs gradation reproduction processing. The image data that has been subjected to various types of image processing is output to the image output device 3, and an image is formed on the recording sheet by the image output device 3.

  Hereinafter, the configuration of the image processing apparatus 1 will be described in detail with respect to image processing and format conversion processing in the image processing apparatus. The A / D conversion unit 10 receives RGB analog signals input from the image input device 2 to the image processing device 1, converts the RGB analog signals into RGB digital signals (that is, RGB signals), and converts the converted RGB signals. Is output to the shading correction unit 11.

  The shading correction unit 11 performs processing for removing various distortions generated in the illumination system, the imaging system, and the imaging system of the image input device 2 on the RGB signal input from the A / D conversion unit 10. Next, the shading correction unit 11 outputs the RGB signal from which distortion has been removed to the document type determination unit 12. The document type discrimination unit 12 converts the RGB reflectance signal input from the shading correction unit 11 into a density signal indicating the density of each of the RGB colors, and converts a document such as a character, a printed photograph, or a photograph (continuous tone photograph). Document type determination processing for determining the mode is executed. When the user manually sets the document type determination process using the operation panel 5, the document type determination unit 12 outputs the RGB signal input from the shading correction unit 11 to the input tone correction unit 13 at the subsequent stage as it is. The processing result of the document type determination processing is reflected in the subsequent image processing.

  The input tone correction unit 13 performs image quality adjustment processing such as color balance adjustment, background density removal, and contrast adjustment on the RGB signals. Next, the input tone correction unit 13 outputs the processed RGB signal to the region separation processing unit 14. The region separation processing unit 14 separates each pixel in the image represented by the RGB signal input from the input tone correction unit 13 into one of a character region, a halftone region, and a photo region. Further, the region separation processing unit 14 generates a region identification signal indicating which region each pixel belongs to based on the separation result, and generates a black generation background removal unit 16, a spatial filter processing unit 17, and a gradation reproduction processing unit 19. And to the compression processing unit 20. Further, the region separation processing unit 14 outputs the RGB signal input from the input tone correction unit 13 to the subsequent color correction unit 15 and compression processing unit 20 as they are.

  The color correction unit 15 converts the RGB signal input from the region separation processing unit 14 into a CMY digital signal (hereinafter referred to as “CMY signal”), and CMY colors including unnecessary absorption components for realizing faithful color reproduction. Processing for removing color turbidity based on the spectral characteristics of the material from the CMY signal is performed. Next, the color correction unit 15 outputs the color-corrected CMY signal to the black generation background removal unit 16. Based on the CMY signal input from the color correction unit 15, the black generation background removal unit 16 subtracts the black generation process for generating a black (K) signal from the CMY signal and the K signal obtained by the black generation process from the CMY signal. To generate a new CMY signal. As a result, the CMY3 color digital signals are converted into CMYK 4 color digital signals (hereinafter referred to as CMYK signals). Next, the black generation background removal unit 16 outputs the CMYK signal obtained by converting the CMY signal to the spatial filter processing unit 17.

  As an example of the black generation process, a method of generating black by skeleton black is generally used. In this method, the input / output characteristic of the skeleton curve is y = f (x), the input data is C, M, Y, the output data is C ′, M ′, Y ′, K ′, UCR (Under When the color removal rate is α (0 <α <1), the black generation background removal processing is expressed by the following equations (1) to (4).

  Here, the UCR rate α (0 <α <1) indicates how much CMY is reduced by replacing the portion where CMY overlaps with K. Equation (1) indicates that the K signal is generated in accordance with the smallest signal strength among the signal strengths of CMY.

  The spatial filter processing unit 17 performs spatial filter processing using a digital filter on the image data of the CMYK signal input from the black generation background removal unit 16 based on the region identification signal input from the region separation processing unit 14, By correcting the spatial frequency characteristics, blurring or graininess degradation of the image is improved. For example, for a region separated into characters by the region separation processing unit 14, the spatial filter processing unit 17 performs spatial filter processing using a filter having a high enhancement amount of high-frequency components in order to improve character reproducibility. Do. In addition, for the region separated into halftone dots by the region separation processing unit 14, the spatial filter processing unit performs low-pass filter processing for removing the input halftone component.

  Next, the spatial filter processing unit 17 outputs the processed CMYK signal to the output tone correction unit 18. The output tone correction unit 18 performs output tone correction processing based on the characteristics of the image output device 3 on the CMYK signal input from the spatial filter processing unit 17, and outputs the CMYK signal after the output tone correction processing. Output to the tone reproduction processing unit 19. The gradation reproduction processing unit 19 performs halftone processing on the CMYK signal input from the output gradation correction unit 18 based on the region identification signal input from the region separation processing unit 14. For example, for a region separated into characters by the region separation processing unit 14, the gradation reproduction processing unit 19 performs binarization or multi-value processing using a high-resolution screen suitable for reproducing high-frequency components. I do. In addition, for a region separated into halftone dots by the region separation processing unit 14, the gradation reproduction processing unit 19 performs binarization or multi-value processing on the screen with an emphasis on gradation reproducibility. . Next, the gradation reproduction processing unit 19 outputs the processed image data to the image output device 3.

  The compression processing unit 20 generates a compressed file based on the region identification signal input from the region separation processing unit 14 and the image data composed of RGB signals. The image data input to the compression processing unit 20 is composed of a plurality of pixels arranged in a matrix. This image data is separated into a foreground layer and a background layer, the foreground layer is further converted into a binary image, each binary image is reversibly compressed by, for example, MMR (Modified Modified READ), and the background layer is converted to, for example, JPEG (Joint It is lossy compressed by Photographic Experts Group). Finally, the reversible compressed binary image and the irreversibly compressed background layer, and decompression information for decompressing them into color image data are collected into one file. This file becomes a compressed file. As the decompression information, information indicating a compression format, an index color table, and the like are used. The compression of the region identification signal generated for each pixel is performed based on, for example, the MMR method and the MR (Modified READ) method, which are lossless compression methods. The compressed image data (compressed image) is temporarily stored in the storage unit 6. For example, when the “scan to e-mail” mode is selected on the operation panel 5, the image data is attached to the e-mail from the transmission device 4. And sent to the set destination.

(Configuration of conversion processing unit)
The conversion processing unit 30 performs format conversion processing on the input document image information (hereinafter also referred to as input image information). Details of the conversion processing unit 30 will be described below. FIG. 2 is a block diagram illustrating a configuration of the conversion processing unit 30. The conversion processing unit 30 includes a character extraction processing unit 31, a line extraction processing unit 32, a table region extraction processing unit 33, a diagram region extraction processing unit 34, a table structuring processing unit 35, and a file description unit 36.

  The character extraction processing unit 31 extracts a character image included in the input image, and analyzes character information indicated by the character image by OCR or the like. Further, the character extraction processing unit 31 defines a character string composed of one or more characters from the arrangement of each character (here, one character is included in the character string). Furthermore, the character extraction processing unit 31 acquires attributes such as the extracted character size and character color.

  The line extraction processing unit 32 extracts lines (line segments) included in the input image and acquires information on each line. The line information includes at least the extracted position (coordinates), direction, length, width, and color of the line. The table area extraction processing unit 33 extracts, as a table area, a set of horizontal lines and vertical lines that intersect each other from the line information extracted by the line extraction processing unit 32. The figure region extraction processing unit 34 extracts regions such as diagrams and photographs included in the input image, and acquires information on each region. The figure area information includes at least the upper left position and size (width, height) of a rectangle surrounding each figure area. In the extracted table area, the table structuring processing unit 35 analyzes the table structure using the elements included in the table area, such as lines included in the table area and characters and diagrams extracted at positions overlapping with the table area. Information for structuring (table structure information) is acquired. The file description unit 36 uses a specified file format so that the document structure has a layout similar to that of the input image from information such as table structure information, elements such as characters and diagrams, and information such as lines not included in the table. Describe the file according to the description method.

  The character extraction processing unit 31, the line extraction processing unit 32, the table area extraction processing unit 33, and the table structuring processing unit 35 may use known techniques. The character extraction processing unit 31 can extract characters and character strings by a known OCR technique. As a method for extracting a character string, for example, a method for extracting a character string from a table (the technique described in Japanese Patent Application No. 2014-174348) can be used. In this method, first, the character extraction processing unit 31 obtains a character rectangle (circumscribed rectangle) of a character extracted by OCR. The character extraction processing unit 31 calculates the distance between adjacent rectangles for each obtained rectangle, and integrates the close rectangles into the same character string group. At this time, a rectangle whose size is too different from the neighboring rectangle size and a rectangle similar to the neighborhood does not exist is removed as a non-character rectangle. The character extraction processing unit 31 detects the horizontal size and the vertical size of each character string group, and sets the longer one as the character string direction. If the size in the horizontal direction is the same as the size in the vertical direction, the direction cannot be determined, and the character string direction is set to be indefinite. The character extraction processing unit 31 performs grouping on adjacent character strings in the same direction that are close in size, or those that are close in the start position and end position of the character string to form character string areas.

The character extraction and character string extraction by the character extraction processing unit 31 may be performed before or after the processing of the line extraction processing unit 32. However, since the processing by the figure region extraction processing unit 34 described later uses information on characters, lines, and tables, the processing by the character extraction processing unit 31, the line extraction processing unit 32, and the table region extraction processing unit 33 is performed in the figure region. It needs to be performed before the processing by the extraction processing unit 34.
Further, when the character extraction processing unit 31 obtains the circumscribed rectangle, the size of each cell in the table including the character or the character string is known, and the circumscribed rectangle does not exceed the size of the cell. When a circumscribed rectangle exceeding the range is obtained, the size of the circumscribed rectangle may be corrected so as to be within the range of the cell.

  The line extraction processing unit 32 and the table region extraction processing unit 33 can extract lines and table regions by the method described in Reference Document 1 (Japanese Patent No. 5153857), for example. In the method described in Reference 1, candidate pixels that are likely to be lines are extracted from document image data, and when the candidate pixels are continuous in a horizontal direction or a vertical direction by a predetermined number of pixels, the continuous candidate pixels are extracted. Is extracted as a line. From the positional relationship between the extracted horizontal and vertical lines, it is determined whether each line is a ruled line constituting a table or a single line, and for a set of lines constituting the same table, The minimum bounding rectangle that surrounds all is extracted as a table area. In the method described in Reference 2, since pixels (line pixels) constituting a line are extracted from a document image, it is possible to calculate an average value of pixel values of the line pixels as a line color. In addition, the line thickness can be calculated from the number of pixels arranged in a direction orthogonal to the direction of the line pixels (or a vertical direction in the case of a horizontal line). For example, an average value of line pixels continuous in the vertical direction can be calculated. It is also possible to calculate the thickness of the line.

  Moreover, it is possible to extract a dotted line or a broken line by a well-known method, and it is also possible to extract a line type line other than a solid line by combining the method with the method of Reference Document 1. . For example, in the method of Reference 2 (Japanese Patent Application Laid-Open No. 7-230525), black pixels connected in the noted ruled line direction (horizontal or vertical direction) are extracted from a binarized document image, and each connected pixel is extracted. Is extracted as a dotted line element when its size is equal to or smaller than a predetermined threshold, and when the distance between the dotted line elements is within the predetermined threshold, a rectangle that integrates the dotted line elements is extracted. , And can be extracted as a dotted ruled line.

  In Reference 3 (Japanese Patent Laid-Open No. 4-68477), a contour vector is extracted by tracking the connection of black pixels from a binarized document image. The area as the candidate for the broken line is extracted from the relationship between the length of the major axis and the minor axis of the area surrounded by the contour vector excluding the solid line or curved line, and the correlation between the obtained broken line candidates is calculated. By examining, a broken line area constituting the same broken line is extracted.

  The figure region extraction processing unit 34 will be described later. The table structuring processing unit 35 determines, for example, a horizontal reference line and a vertical reference line from the input document image and lines extracted from the input image, and two adjacent horizontal reference lines. A quadrangle composed of two adjacent vertical reference lines is defined as a temporary cell. Based on the element information extracted from the input image, the defined temporary cell is first classified as a combined cell candidate that is a set of quadrangles including at least one temporary cell. For each combined cell candidate subjected to the first classification, the temporary cell included in the combined cell candidate is determined to be the most appropriate cell block (in accordance with the criterion) based on the element information extracted from the input image ( The second classification is performed as a set of at least one temporary cell). The table structure is analyzed from the attribute of the cell block subjected to the second classification and the element information extracted in advance, and information to be referred to in order to reconstruct the table is described as the table structure information.

  The specific description method of the table structure information is not particularly determined. For example, if it is described so that it can be referred to in a tree shape as shown in FIG. 3, an XML (Extensible Markup Language) format file format described later is used. Conversion to etc becomes easy. In addition, a list or the like is separately described for each attribute type, and information is acquired by referring to an ID (Identification) specified in the list, so that the same attribute is repeatedly described. You can avoid that. For example, the extracted line information is described as a list as shown in FIG. 4, IDs (line IDs) are assigned to the respective lines, and each cell block in the tree-shaped table structure information described as shown in FIG. By describing the line ID as the ruled line information, it is not necessary to repeatedly describe the same information among a plurality of cell blocks sharing the same line. Although the order of description is not particularly determined, it is desirable that the order of cell blocks to be described is in descending order of parent ID, and information is described in order from the upper left of the table. Of course, information to be described may be increased in addition to the items shown in FIGS.

  Note that the file description method (conversion method) executed by the file description unit 36 described later differs depending on the type of office document file to be converted. For example, in an office suite (a set of office business software) compliant with the OfficeOpenXML (OOXML) standard, a document group described in XML and binary data such as photographs, illustrations, and figures are integrated into one file by compression. Use as an office document file. On the other hand, the file description method differs between different types of software (for example, document creation software and presentation software), and the same type of software between OOXML and different document formats (for example, OpenDocument Format). Even so, the file description method is different.

  Therefore, in order to be able to easily cope with any file format, the table structuring processing unit 35 acquires table structure information not limited to a specific file format, and the file description unit 36 described later uses the table structure information. Use information to structure the table for the specified file format. In this way, when a user uses a conversion processing device having the function of the conversion processing unit 30, the user can see the result of conversion in a certain file format and then convert the file into a different file format. Even if the conversion processing unit 35 is not executed from the beginning, the table structure information already obtained can be reused and easily converted.

The file description unit 36 performs conversion into a designated file format using information of elements such as characters and figures, line information, and table structure information extracted up to the previous stage. The file description unit 36 has a different description method depending on the designated file format. However, the file format whose file structure is publicly disclosed can be obtained by a known method using already obtained table structure information, element information, or the like. Can be described. For example, a document creation software provided by Microsoft (registered trademark) “Microsoft Word 2010” (in the left column, Microsoft is a registered trademark), or a succeeding version (hereinafter simply referred to as Word) file format (docx) OOXML, which is a file format adopted by) is standardized as ECMA-376 and ISO / IEC 29500, and can be structured as a Word file by describing according to the format described in those specifications. . The format standardized as ECMA-376 is disclosed as the following Reference 4 and is described according to the format described in Reference 4. In the following, a processing example of a part related to the structuring of the table in the document image will be given, and a detailed description method will be omitted.
“Reference 4: ECMA-376, 4th Edition Office Open XML File Formats, [searched on September 10, 2015], Internet (URL http://www.ecma-international.org/publications/standards/Ecma-376 .htm) "

  FIG. 5 is a tree diagram illustrating an example of a file structure (part) of a Word file. In the Word file, a series of folders and files as shown in FIG. 5 are ZIP-compressed and the file extension is replaced with docx to form one file. A file describing data for structuring a document, an image file storing graphics extracted from a document image, and the like are stored in the / word / folder on the first line of FIG. For example, the document.xml file describes objects such as characters (columns), graphics, and tables that make up the document body according to the information using a markup language called WordProcessingML. For example, as shown in FIG. 6, the table is based on information described between tags having <w: tbl> as a start declaration and </ w: tbl> as an end declaration (hereinafter, this information is referred to as description 1). Structuring the table. The description 1 is divided into information such as properties over the entire table (hereinafter this information is referred to as description 2) and information for structuring each row (hereinafter this information is referred to as description 3).

  Description 3 is described between <w: tr> and </ w: tr>, and repeats for the number of lines in order from the first line. The description 3 is further divided into information such as properties over the entire row (hereinafter this information is referred to as description 4) and information for structuring each cell (hereinafter this information is referred to as description 5). Description 5 is described between <w: tc> and </ w: tc>, and is repeated for the number of columns in order from the top (left end). The description 5 further includes information such as properties related to cells (hereinafter, this information is referred to as description 6) and information for structuring a paragraph that is a set of elements stored in each cell (hereinafter, this information is described as 7). Divided). The content described between <w: p> and </ w: p> in description 7 indicates one paragraph. The description 7 is further divided into information such as properties relating to each paragraph (hereinafter this information is referred to as description 8) and information for structuring the stored elements (hereinafter this information is referred to as description 9).

  The content described between <w: r> and </ w: r> of description 9 indicates one run. For each run, information such as properties (hereinafter referred to as “description 10”) and data information indicating elements (hereinafter referred to as “description 11”) are described. Classify into runs. That is, even in the case of a character string, if properties such as character color and size are different, the runs are divided. In FIG. 6, information between <w: t> and </ w: t> corresponding to description 11 describes character (string) data. When storing an image, from <w: drawing> </ w: drawing> describes information about the image to be stored. The image file is stored in the / word / media folder in FIG. 5, and the corresponding graphic is obtained by referring to the ID associated with the file name in /word/rels/document.xml.rels in the document.xml file. Can be pasted into a document file. For details of each file in FIG. 5 and a specific description method of each file, refer to the format specifications such as Reference 4.

  If appropriate values are not set for various attributes, such as omitting the calculation process, description of properties related to the attributes may be omitted, and initial values set by various applications may be used. . For example, when the calculation of the character color is omitted, a predetermined color such as black can be used as the initial value. When the ruled lines on the four sides of the cell block are given as “undefined” as described above, the ruled line information on the sides of the cell is not set. Also, in each row, when the ruled lines on the upper side or the lower side of all the temporary cells constituting the row are constituted by the same line, the upper side or the lower side is not a cell unit but a row unit (corresponding to description 4 in FIG. 6). The ruled line information may be set collectively.

  In addition, in FIG. 3 described above, as storage elements (first element, second element,...) Associated with cell blocks, in addition to characters and character strings formed by arranging a plurality of characters, FIG. And photos. How to use these pieces of information depends on the description method in the file description unit 36 and the file format to be converted. For example, Microsoft (registered trademark) Excel and the like cannot simply describe an image in a cell like a character. On the other hand, in the case of Word, an image object can be inserted into a table cell. Therefore, in this embodiment, conversion is performed according to the file format of the conversion destination.

  FIG. 7 is a block diagram showing the configuration of the figure area extraction processing unit 34 (figure area extraction apparatus). The figure region extraction processing unit 34 includes a non-character string map generation processing unit 341, a non-character string area addition processing unit 342, an object map generation processing unit 343, and an effective object area determination processing unit 344.

  The non-character string map generation processing unit 341 performs edge detection processing on the input image, and excludes a character string region composed of characters extracted by the character extraction processing unit 31 from the obtained edge detection result. As a result, the remaining edge region is generated as a non-character string map. The non-character string area addition processing unit 342 calculates a histogram entropy value of the image with respect to a region excluding the character string region composed of the characters extracted by the character extraction processing unit 31, and removes a region having a high entropy value. Add to non-string map as string area. The object map generation processing unit 343 performs a line area removal process, a labeling process, and a rectangular process on the non-character string map to which the non-character string area is added, and generates an object map. The valid object area determination processing unit 344 performs rectangular area integration / division processing on each object on the object map generated by the object map generation processing unit 343 as necessary, and finally converts it during the format conversion processing. It is determined whether the image object is to be converted. If it is determined that the image object is to be converted, the object area is left on the map. If it is determined that the image object is not to be converted, the object area is deleted from the map.

  Hereinafter, each processing unit of the figure region extraction processing unit 34 will be described in detail with reference to the table image 400 of FIG. 8 and the flowcharts of FIGS. In the table in the table image 400 of FIG. 8, the square and the heart are shaded for convenience, but actually there is an appropriate density change as in the photograph, and the edge strength at the outer periphery of the figure is weak. To do. As shown in the flowchart of FIG. 9, the non-character string map generation processing unit 341 first performs edge detection processing on the input image (step Sa1). As an edge detection method, detection is performed using, for example, a primary differential filter such as a Sobel filter or a Prewitt filter, or a secondary differential filter such as a Laplacian filter. Here, as an example, a method for performing edge detection on G values of RGB values using a Laplacian filter will be described. FIG. 10A shows a 3 × 3 Laplacian filter. When the horizontal coordinate position of the pixel of interest to be filtered is x, the vertical coordinate position is y, and the G value of the pixel of interest is represented by p_g (x, y), the Laplacian filter processing result p′_g (x, y) is (5)

  The result of performing the edge intensity detection processing by the Laplacian filter of FIG. 10A on the G value of each pixel in the region surrounded by the thick line of FIG. c). In the calculation of the Laplacian filter, a value of ± appears, but here the absolute value is taken so that the processing result becomes only a positive value.

  Further, for this edge strength detection result, the non-character string map generation processing unit 341 detects only pixels having edge strength larger than the threshold value as edge pixels using a predetermined threshold value (step Sa2). For example, FIG. 10D shows a result of setting the threshold value to 50, setting 1 as an edge pixel for a pixel having an edge strength greater than 50, and setting the other pixels to 0. In response to this edge detection result, the non-character string map generation processing unit 341 eliminates the edge detection result existing in the character string region defined by the character extraction processing unit 31 (step Sa3). For example, the edge detection result for the table image 400 in FIG. 8 is the table image 401 in FIG. 11A, and the character string region defined by the character extraction processing unit 31 is black in the table image 402 in FIG. If it is a painted part, the result of eliminating the edge detection result existing in the character string area defined by the character extraction processing unit 31 with respect to the edge detection result is as shown in FIG. The non-character string map 403 is the result of performing such processing on the entire table image.

  Next, the non-character string area addition processing unit 342 extracts objects such as photographs that cannot be extracted based on the edge detection processing by the non-character string map generation processing unit 341 as a non-character string area according to the flowchart shown in FIG. And add it to the non-string map. When the frequency of appearance of pixel values (ie, a histogram) is determined for each predetermined local area of a document image, a histogram with a wide range of luminance changes can be obtained for each pixel on the photographic area, which is one of the figure areas. The histogram entropy (average information amount) is calculated as luminance change information. For example, the non-character string area addition processing unit 342 regards an area of 11 × 11 pixels as one local region, and calculates a histogram in this region (step Sb1). Next, the non-character string area addition processing unit 342 calculates an entropy value of this histogram (step Sb2). When obtaining the histogram, it is possible to suppress the calculation of the entropy value of the character as much as possible by obtaining the histogram with only the pixels excluding the character string area composed of the character extracted by the character extraction processing unit 31. be able to. The entropy of the histogram is obtained by the following equation (6).

  In Equation (6), L represents the number of gradations in the histogram (L = 256 if 8-bit), h (i) is the frequency of gradation i, N is the number of pixels to be counted, and p (i) is This is a value obtained by normalizing h (i) by N. By extracting a region having a high entropy value obtained as described above, it is possible to accurately extract a photographic region. For this purpose, the non-character string area addition processing unit 342 sets only pixel regions having an entropy value exceeding a predetermined threshold (for example, a value of about 35) to 1 and sets the others to 0 (step Sb3). FIG. 13 is an example showing a non-character string area 404 that is a result of the non-character string area addition processing unit 342 performing a non-character string area detection process based on histogram entropy on the table image 400 of FIG. When the non-character string area 404 is detected, the non-character string area addition processing unit 342 adds the non-character string area 404 to the non-character string map 403 (step Sb4). The result of adding the non-character string area 404 of FIG. 13 to the non-character string map 403 of FIG. 11C is a non-character string map 405 shown in FIG.

  Next, according to the flowchart shown in FIG. 15, the object map generation processing unit 343 first performs a process of removing the line in the table area determined by the line extraction processing unit 32 (step Sc1). Assume that the table image 400 of FIG. 8 is determined as a table region, and each ruled line can be extracted as a line by line extraction. As illustrated in FIG. 16, the object map generation processing unit 343 deletes the table area line from the updated non-character string map 405 illustrated in FIG. 14. In the non-character string map 406 after line deletion shown in FIG. 16, the deleted line is indicated by a dotted line so that the deleted trace can be easily understood. However, the dotted line does not actually exist. By performing the process of removing the ruled line of the table on the non-character string map 405 in this manner, even when the image object is in contact with or superimposed on the ruled line of the table, the image object is separated from the ruled line of the table. Only the area can be extracted.

  Next, the object map generation processing unit 343 performs a labeling process (step Sc2). The labeling process is performed on condition that the pixels connected diagonally up, down, left, and right have the same label. A general technique may be used as the labeling technique. FIG. 17 shows an example of a result of labeling processing performed on the non-character string map 406 of FIG. The object map generation processing unit 343 causes the square to be the label 201, the outer circle line to the label 202, the inner circle line to the label 203, the triangle to the label 204, the heart to the label 205, and the star outer line. Are labeled 206 and the inner line of the star is labeled 207.

  The object map generation processing unit 343 compares the coordinate information of each pixel with respect to each labeled object, and obtains the minimum value and the maximum value in the horizontal and vertical directions of the coordinates for each label. The object map generation processing unit 343 sets the value of the rectangular area having the obtained maximum value and minimum value as vertices (the minimum value represents the upper left corner of the rectangular area and the maximum value represents the lower right vertex of the rectangular area) to 1. As shown in FIG. 18, the labeled object areas are rectangularized (step Sc3). At that time, the object map generation processing unit 343 compares the maximum value coordinate and the minimum value coordinate of each label. As a result, since the rectangular area of the label 203 is found to be included in the rectangular area of the label 202, the object map generation processing unit 343 integrates the label 203 into the label 202. Similarly, since it can be seen that the rectangular area of the label 207 is included in the rectangular area of the label 206, the object map generation processing unit 343 integrates the label 207 into the label 206. In this way, the object map generation processing unit 343 creates five rectangular areas of a label 201, a label 202, a label 204, a label 205, and a label 206.

  When a ruled line other than the table extracted by the line extraction process is handled as an image object, the ruled line may be added as an image object to the object map including the image object subjected to the rectangularization process. Further, if ruled lines other than the table extracted by the line extraction processing are handled as vector information instead of image objects, the object map is only the rectangular image processing object.

  As described above, the non-character string map generation processing unit 341 extracts line drawings such as illustrations and graphs with high luminance changes, and diagrams such as graphs on an edge basis. The non-character string area addition processing unit 342 extracts an image region such as a photograph that has a change in luminance, although the edge strength is not so strong, based on the histogram entropy. As a result, the object map generation processing unit 343 can widely extract various types of image objects. In addition, even if a part of the figure or picture is not extracted, the figure or photograph is often a rectangle. Therefore, by performing labeling and rectangle processing, it is possible to prevent a part of the figure or picture from being extracted. .

  Next, as shown in the flowchart of FIG. 19, the effective object area determination processing unit 344 first integrates rectangular areas for each object on the object map generated by the object map generation processing unit 343 as necessary. Division processing is performed (step Sd1). In the case of the example of FIG. 18, since the rectangle is divided only by the image objects in the table, the integration / division processing is not necessary. Actually, image objects may exist outside the table regardless of only the image objects in the table. Depending on the shape and arrangement of the image object in the input image, some of the rectangular areas of the image object It can happen that they overlap, the rectangular area of the image object overlaps the character string, or a part of the rectangular area of the image object overlaps the table. In such a case, integration processing and division processing of rectangular areas are performed.

  For example, when the rectangular areas partially overlap each other like the image objects 501 and 502 shown in FIG. 20, the valid object area determination processing unit 344 performs the labeling process again, and determines the maximum and minimum coordinate values. Calculate and perform the rectangle processing. As a result, the rectangular areas of the two overlapping image objects 501 and 502 are integrated into one rectangular area 504. As a result, it is possible to prevent the appearance of the image object from being overlaid on the image object at the time of the file format conversion process, or the file size from being increased due to the overlap.

  In addition, as shown in FIG. 21, when the character string 301 overlaps the rectangular area of the image object 501, if only a part of the character string 301 becomes an image object, a partial font and a partial image are used in the layout after format conversion. And so on. In other words, in the example of FIG. 21, the character object, the table object, the line object, and the image object overlap in order from the lowest layer, and the character string 301 that overlaps the image object 501 Assume that information is stored by dividing a portion of the character string 301 that does not overlap the image object 501. At this time, the character string 301 of the portion overlapping the image object 501 is stored as a character image as a simple image included in the image object 501. When the information stored in this way is reconstructed, the portion of the character string 301 that does not overlap the image object 501 may be replaced with another font. In that case, as a result of overlapping with the character image displayed on the image object 501, the font and size may be different, and the appearance will be deteriorated.

  In the case shown in FIG. 21, the valid object area determination processing unit 344 compares the rectangular coordinates of the character string 301 with the rectangular coordinates of the image object 501, and the maximum coordinate value is the same as when the image objects 501 and 502 overlap. And the minimum value is calculated, and rectangularization processing is performed (step Sd2). As a result of this processing, the portion of the character string 301 that does not overlap the image object 501 is integrated into the rectangular area of the image object 503 that includes the character string 301. This integration does not eliminate the character string 301 itself, but the character string 301 itself is arranged under the image object 501 at the time of format conversion. Therefore, the character string integrated into the image object 504 is also a text It can be used as text data such as search. As a result, during the file format conversion process, the character image in another image object 501 is overlaid on the character string 301 converted into the font, and the font type and size of the character change to deteriorate the appearance. It becomes possible to prevent.

  Further, as shown in FIG. 22, the rectangular area of the image object 505 may overlap the table area 410. In this case, the effective object area determination processing unit 344 compares the coordinates of the rectangular area of the image object 505 with the coordinates of the table area 410 and extracts an overlapping range, and extracts the overlapping table from the rectangular area of the image object 505. The area 410 is deleted. Further, the effective object area determination processing unit 344 performs division processing on the remaining area of the image object 505 along the extension of the outer frame line of the table area 410 (step Sd3). As a result, the valid object area determination processing unit 344 eliminates the overlapping area of the image object 505 by dividing the image object 505 even when the area of the rectangular image object 505 overlaps the table area 410. It can be avoided that a part of the table area 410 becomes invisible by overlapping with the area 410. That is, it is possible to prevent the image object 505 from being overlaid on the table area 410 and partially obscuring the table area 410 during the file format conversion process.

  In the example of FIG. 22, the remaining area of the image object 505 is divided into image objects 506 and 507 along the extended line of the outer frame line in the horizontal direction of the table area 410. Note that the remaining area of the image object may be divided not along the horizontal direction of the outer frame line of the table area 410 but along the extension line of the vertical outer frame line, or both horizontal and vertical may be divided. It doesn't matter. Further, when dividing, for example, the image is divided into areas of two pixels, and when the image object 505 is cut out later using the object area determination result, the area expanded by one pixel from the rectangular area is cut out. As a result, the divided image objects 506 and 507 can be format-converted without seemingly being divided into two.

  For the rectangular regions of the image objects 504, 503, 506, and 507 obtained in this way, the valid object area determination processing unit 344 performs re-labeling to perform a rectangular process (step Sd4). At this time, the valid object area determination processing unit 344 may perform the canceling process so that a rectangular area having a size smaller than a predetermined area is not suitable for an image object. For example, when a resolution conversion process is performed on an input image of 300 dpi to 75 dpi and a rectangular process is performed with an image size of 75 dpi, the horizontal size of the rectangular area is less than 30 pixels, or the vertical size of the rectangular area is less than 30 pixels. Alternatively, a rectangular area having an area of less than 900 pixels may not be an image object. Further, a rectangular area in which the number of pixels of an object area that is actually labeled is less than 25% of the area of the rectangular area may not be an image object. However, it is determined whether or not an image object subjected to the division process is to be an image object in consideration of the size before the division.

  By performing the above processing, the effective object area determination processing unit 344 determines the finally remaining image object area as the effective object area (step Sd5). The effective object area determination processing unit 344 cuts out the image object from the input image according to the information determined as the valid object area, so that not only the image object in the table but also other image objects can be cut out with high accuracy. Even when the format is converted, it is possible to obtain a good-looking result.

  By using the character string region information, the line segment information, the table region information, and the luminance change information based on the histogram entropy, the configuration of the first embodiment described above allows not only the image object region outside the table but also the inside of the table. It is also possible to extract the image object area existing in the. Further, even when the rectangular image object area and the table area overlap, it is possible to eliminate the area where the image object overlaps and prevent the table from overlapping and becoming invisible. Therefore, even if an object other than a character such as an image exists in a table cell, the object is correctly extracted and converted without accidentally extracting the object as a character, and the object is correctly placed in the table. It becomes possible to do.

  Further, as described above, the order in which the objects are overlapped by the file description unit 36 is the order of the character area object, the table area object, the line segment area object, and the image object from the lowest layer. Since the line segment object may be handled as a part of the image object, the order of the object in the line segment area and the image object may be switched. In this way, by placing the object in the character area at the bottom, when it overlaps with the image object, the overlapped character area is displayed as a character image integrated with the image object, so that the appearance is improved. Then, it is possible to make a searchable state by using the OCR character information. Further, since the image object that overlaps the table area is divided, the table area is not hidden even if the table area is below the image object.

Note that in the non-character string map generation processing unit 341, the non-character string area addition processing unit 342, the object map generation processing unit 343, and the valid object area determination processing unit 344 of the figure region extraction processing unit 34 in the first embodiment described above. Although the character string extracted by the character extraction processing unit 31 is handled, the configuration of the present invention is not limited to the embodiment. Processing may be performed not only for character strings but also for characters extracted by the character extraction processing unit 31. Compared with the case of extracting as a character string, when extracting as a character, there is a possibility that a part of the character may not be extracted or punctuation marks may not be extracted depending on the extraction accuracy. Even if the process is processed as a character string or as a character, the same effect is obtained.
In the first embodiment, the character extraction processing unit 31 obtains a circumscribed rectangle as the character string area or the character area when extracting the character string area. Is not limited to this embodiment. The character string area or the character area is not limited to a circumscribed rectangle, but may be a shape including a curve, for example, a circumscribed circle or a circumscribed ellipse.

Further, in the non-character string area addition processing unit 342 in the first embodiment, histograms may be obtained in all regions and entropy may not be calculated, but may be performed only on unknown regions. For example, the entropy may be calculated by obtaining a histogram only for the remaining table areas excluding the table areas detected by edge detection.
In the first embodiment, when the non-character string area addition processing unit 342 obtains the histogram entropy, it may be regarded as one image object if the distance between adjacent image objects is short. In this case, since each image object is often included in one cell, the non-character string area addition processing unit 342 clearly identifies each object with reference to the information indicating the cell boundary. Histogram entropy may be obtained separately.

In the first embodiment, the luminance change information that the non-character string area addition processing unit 342 obtains is not limited to entropy, and may be another index.
Here, a modified example of the luminance change information will be described.
FIG. 24 is a diagram for explaining a modification of the luminance change information.
In the example shown in FIG. 24, the image P includes a character string and an image object. For example, the non-character string area addition processing unit 342 obtains a histogram of a local region (for example, a region of 11 [pixel] × 11 [pixel]) corresponding to the target pixel, and the frequency is a predetermined threshold T (for example, the frequency) 5) A gradation value indicating the frequency of the peak is extracted from the gradation values (bins) described above. Next, the non-character string area addition processing unit 342 extracts gradation values whose frequency is continuous at a threshold value T or more around the peak, and sets the width (gradation width) of the series of extracted gradation values. Used as luminance change information. If the gradation width is equal to or greater than a predetermined gradation number (for example, 48 gradations) (reference width SW), the non-character string area addition processing unit 342 determines that the image area is a non-character string area. May be added.

  Specifically, for example, in FIG. 24, in the case of a local region A1 including a photograph in the image P, the gradation width WB related to the peak B is larger than the reference width SW, and thus the local region A1 is a region of a photographic object. It is determined that there is. On the other hand, in the case of the local region A2 that does not include a photograph or a character string or the local region A3 that includes a character string in the image P, the gradation widths WC and WD related to the peaks C, D, and E appearing in each local region. , WE are less than the reference width SW, and it is determined that the local areas A2 and A3 are not photographic object areas. In this way, not only entropy but also the gradation width related to each peak may be used as the luminance change information.

In Embodiment 1 described above, the object map generation processing unit 343 processes two rectangular areas when there are non-overlapping rectangular areas in one cell. However, the present invention is not limited to this embodiment, and rectangular areas in one cell may be integrated into one.
In the first embodiment, the valid object area determination processing unit 344 performs the same processing for the character string area when the character string area overlaps the table area and the image object overlaps the table area. You may make it apply.

  In the first embodiment and the second and third embodiments described below, the determination as to whether or not the threshold value or the like is greater than or equal to a predetermined numerical value is an example, and depending on the size of the predetermined numerical value, a predetermined numerical value may be used. It may also be a determination as to whether or not the value exceeds a predetermined value, and the determination as to whether or not the value is less than or equal to a predetermined value may also be a determination as to whether or not the value is less than a predetermined value depending on the size of the predetermined value. .

[Embodiment 2]
In the first embodiment described above, the configuration in which the conversion processing apparatus according to the present invention is applied to the image processing apparatus 1 included in the image forming apparatus 100 as the conversion processing unit 30 has been described. However, the configuration of the present invention is not limited thereto. It is not a thing. In the second embodiment, an example in which the conversion processing apparatus according to the present invention is applied as the conversion processing unit 30 to the image processing apparatus 1a included in the image reading apparatus 100a such as a flatbed scanner will be described.
In addition, about the member which has the same function as the member described in drawing used for description of Embodiment 1, the same code | symbol is attached | subjected also in the following description. The detailed description of each member will not be repeated here.

  FIG. 23 is a block diagram illustrating a configuration of an image reading apparatus 100a (information processing apparatus) including the image processing apparatus 1a according to the second embodiment. As shown in FIG. 23, the image reading apparatus 100a includes an image processing apparatus 1a, an image input apparatus 2, a transmission apparatus 4, a storage unit 6, and an operation panel 5. The image processing apparatus 1a includes an A / D conversion unit 10, a shading correction unit 11, a document type determination unit 12, an input tone correction unit 13, a region separation processing unit 14, a compression processing unit 20, and a conversion processing unit 30 (conversion processing unit 30). Device). The conversion processing unit 30 outputs a file converted into the designated file format, as described in the first embodiment. Various processes executed by the image reading apparatus 100a are controlled by a control unit (a computer including a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor)) provided in the image reading apparatus 100a. In the second embodiment, the image reading apparatus 100a is not limited to a scanner. For example, a digital still camera, a document camera, or an electronic device equipped with a camera (for example, a mobile phone, a smartphone, a tablet terminal, etc.) ).

[Embodiment 3]
In Embodiments 1 and 2, the conversion processing apparatus according to the present invention is applied as the conversion processing unit 30 to the image processing apparatuses 1 and 1a included in the image forming apparatus 100 or the image reading apparatus 100a. The configuration of the invention is not limited to this. The conversion processing device according to the present invention may be applied to, for example, a server device. An example of the configuration of the server device in this case is a receiving device that receives a document image subjected to image reading and various image processing by the image forming device 100 or the image reading device 100a via a network, and the first embodiment. A server apparatus (including a conversion processing device that executes processing in the conversion processing unit described above, and a transmission device that transmits a document file output from the file description unit included in the conversion processing device via a network ( Information processing apparatus). By configuring the server device in this way, a document image that has been subjected to image reading and various types of image processing by the image forming apparatus 100 or the image reading apparatus 100a is received via the network, and the conversion processing unit described above is received. It is possible to use such a method that a document file is created by a conversion processing device that executes the processing in (1), and the output file is transmitted to a user terminal device (for example, a personal computer or a tablet terminal). In addition, the server apparatus can use the format conversion function without replacing an already installed image forming apparatus or image reading apparatus.

  Further, the conversion processing device may be configured without the file description unit 36, and various information for structuring the document may be created and transmitted via a network. In this case, the processing means executed by the file description unit 36 is executed by the terminal device side that has received the various information, so that the desired information can be obtained without repeating acquisition of various information necessary for structuring the document. Conversion to a file can be performed smoothly, so if you have specified an incorrect file format due to an operation error, or if you are not satisfied with the conversion result of the specified file format, you can convert it to a different file format. It becomes easy to fix.

  The image processing apparatuses 1, 1a (particularly the conversion processing unit 30) and the server apparatus (particularly the conversion processing apparatus) in the first, second, and third embodiments are logical circuits (hardware) formed in an integrated circuit (IC chip) or the like. Hardware), or software using a CPU (Central Processing Unit). In the latter case, the image processing apparatuses 1 and 1a and the server apparatus include a CPU that executes instructions of a program that is software for realizing each function, and a ROM in which the program and various data are recorded so as to be readable by a computer (or CPU). (Read Only Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission. The image processing apparatuses 1 and 1a and the image processing method described above are configured to handle color image data, but are not limited thereto, and may be configured to handle monochrome image data.

  The present invention is not limited to the above-described embodiments, and various modifications can be made. That is, embodiments obtained by combining technical means appropriately changed within the scope not departing from the gist of the present invention are also included in the technical scope of the present invention.

30 conversion processing unit 31 character extraction processing unit 32 line extraction processing unit 33 table region extraction processing unit 34 diagram region extraction processing unit 35 table structuring processing unit 36 file description unit 341 non-character string map generation processing unit 342 non-character string area addition Processing unit 343 Object map generation processing unit 344 Effective object area determination processing unit

Claims (11)

A character extraction processing unit for extracting a character region existing in the document image information;
A line extraction processing unit for extracting a line segment existing in the document image information;
A table region extraction processing unit for extracting a table region using the line segment information extracted from the line extraction processing unit;
A predetermined local region is set for the document image information, a luminance histogram of the local region is created to determine luminance change information of the local region, and the luminance change information and the character extraction processing unit extract The outside of the table area using the information on the character area, the line segment information extracted from the line extraction processing section, and the table area information extracted from the table area extraction processing section, or A diagram area extraction processing unit for extracting an image object area including a picture or a photograph existing in the table area;
A table structuring processing unit that analyzes the table structure based on the information on the character area in the table area, the information on the line segment, and the information on the image object area, and acquires table structure information for reconfiguring the table; ,
A conversion processing apparatus comprising: The character region extracted from the document image information by the character extraction processing unit includes a character string region including a character string,
The figure region extraction processing unit
A predetermined local region is set for the document image information, a luminance histogram of the local region is created to determine luminance change information of the local region, and the luminance change information and the character extraction processing unit extract Using the information on the character string region, the line segment information extracted from the line extraction processing unit, and the table region information extracted from the table region extraction processing unit. Or, extract the image object area including the figure or photo that exists in the table area,
The table structuring processing unit
Analyzing the table structure based on the information on the character string area in the table area, the information on the line segment, and the information on the image object area, and obtaining table structure information for reconfiguring the table. The conversion processing apparatus according to claim 1. Order of the character string area object, the table area object, the line segment object, the image object, or the character string area object, the table area object, the image object, the line object The conversion processing apparatus according to claim 2, further comprising: a file description unit described in a file format designated to arrange objects in order. The figure region extraction processing unit
Edge detection is performed on the document image information, and a character string region that is one of the document components extracted by the character extraction processing unit is excluded from the result of the edge detection. A non-character string map generation processing unit for generating a candidate non-character string map;
As the brightness change information, the entropy of the histogram in the local region of the document image information is calculated, and a region having a high value of the calculated entropy is added to the non-character string map as the image object region candidate. A column area addition processing unit;
For the non-character string map to which the candidate for the image object area is added, the line segment of the table area extracted by the table area extraction processing unit is deleted, and the non-character string from which the line segment of the table area is deleted Labeling the image object region by performing a labeling process on the map, and generating an object map by obtaining a rectangular region of the labeled image object region;
The conversion processing apparatus according to claim 2, further comprising: The figure region extraction processing unit
The conversion processing apparatus according to claim 4, further comprising: an effective object area determination processing unit that performs integration processing of the rectangular regions or division processing of the rectangular regions for each of the rectangular regions of the image object region. The effective object area determination processing unit
When the rectangular areas of the plurality of image object areas overlap, the maximum and minimum values of the rectangular area coordinates of the plurality of image objects are calculated, and the overlapping image object areas are integrated into one rectangular area. The conversion processing device according to claim 5, wherein: The effective object area determination processing unit
When the character string area overlaps the rectangular area of the image object area, the maximum and minimum coordinates of the rectangular area of the image object and the area of the character string area are calculated, The conversion processing apparatus according to claim 5, wherein the character string area overlapping with the image object area is integrated into one rectangular area. The effective object area determination processing unit
When the table area overlaps the rectangular area of the image object area, the overlapping area of the table area is excluded from the rectangular area of the image object area and the overlapping table is excluded. 8. The conversion according to claim 5, wherein the rectangular area of the image object area is divided along a horizontal frame line of the area or an extension line of the vertical frame line. 9. Processing equipment. An information processing apparatus comprising the conversion processing apparatus according to claim 1. Computer
Character extraction processing means for extracting a character region existing in the document image information;
Line extraction processing means for extracting a line segment existing in the document image information;
Table area extraction processing means for extracting a table area using the line segment information extracted by the line extraction processing means,
A predetermined local region is set for the document image information, a luminance histogram of the local region is created to determine luminance change information of the local region, and the luminance change information and the character extraction processing unit extract The outside of the table area using the information on the character area, the line segment information extracted by the line extraction processing means, and the table area information extracted by the table area extraction processing means, or Figure area extraction processing means for extracting a picture existing in the table area or an image object area including a photograph;
Table structuring processing means for analyzing the table structure based on the information on the character area in the table area, the information on the line segment, and the information on the image object area, and acquiring table structure information for reconfiguring the table;
Program to function as.   The computer-readable recording medium which recorded the program of Claim 10.

Images