JP2007043662A - Image forming apparatus and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To produce a structured document from an original image with a little user burden and high precision while maintaining high productivity of a compound machine. <P>SOLUTION: An image of an original read by a compound machine is separated into image regions by an image region separation circuit 28, position/shape information 102 and an image region class 104 are determined for each image region, and a spatial frequency and other image feature information are determined for images in each of the image regions. A structuring section 40 refers to schemer information of structured documents registered on a schemer DB 42 and collates the position/shape, image region class and image features of each of elements with the position/shape, image region class and image features of each of the regions of the original image, thereby determining whether the original is suited to the schemer information. If suited, the original image is converted into structured document in accordance with the schemer information. If not suited, schemer information with respective image regions as temporary components is created and presented to a user and after correction from the user, a result of the correction is registered on the schemer DB 42 as new schemer information. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image forming apparatus or an image processing apparatus, and more particularly to a technique for generating structured document data from a read original image.

  As a technique for generating a structured document described in a structured description language such as SGML (Standard Generalized Markup Language), HTML (Hypertext Markup Language), or XML (eXtensible Markup Language) from a document image read by a scanner, Patent Document 1 There are those shown in ~ 3.

  The apparatus of Patent Document 1 extracts a character area and an image area from an optically read document image, obtains a layout of each area, performs character recognition on the character area, and uses the layout and the character recognition result. To create a structured document. In this apparatus, this series of processing is performed automatically.

  The apparatus of Patent Document 2 separates a read document image into a character area, a table area, and an image area, and displays a rectangle of each area. When the user moves the rectangle, layout editing is performed, and hypertext is created according to the edited layout.

  The apparatus of Patent Document 3 receives an input of an area designation and an attribute of each area (whether the area is a character string or a chart, a dictionary type for character recognition, etc.) from the user for the read document. When character recognition of an area is performed, a dictionary selected based on the attribute of the area is used. Based on the attributes, a markup process is performed on both the character area and the chart area to generate a structured document.

Japanese Patent Application Laid-Open No. 11-066196 JP 11-312231 A Japanese Patent Laid-Open No. 10-162098

  Although the techniques of Patent Documents 1 and 2 reflect the result of automatic region separation of a document image in a structured document, character regions and image regions or table regions can be separated by automatic region separation. This is a relatively rough classification. For example, since the title and the text are only classified as the same character area, even if it is desired to create a structured document in which the title element and the text element are separated, it cannot be automatically realized.

  Since the technique of Patent Document 3 requires the user to specify the area, it may be possible to classify the area finely. However, if the user specifies the area for every single document, the user burden is reduced. Too big.

  According to the present invention, a process for generating a structured document from an optically read original image can be realized with high accuracy and high productivity of a digital multi-function peripheral in a form that reduces the burden on the user. To.

  The present invention provides a reading unit that reads an original, an image area separating unit that separates an image of an original read by the reading unit, and an image in the image area for each image area separated by the image area separating unit. An image forming apparatus comprising: an image processing unit that performs image processing according to a type; and an output unit that outputs an image of a document image-processed by the image processing unit. A feature calculation unit for obtaining an image feature for each image area, a structure information registration unit for registering the position and image feature of each element of the structured document as structure information of the structured document, and the image area separation unit Each image area is associated with each element of the structured document based on the position of each image area, the image feature of each image area obtained by the feature calculation unit, and the information registered in the structure information registration unit. , Corresponding to the original image based on the result of this association Comprising a structural unit which generates a structured document that, a.

  In a preferred aspect of the present invention, the image area separation unit obtains an image feature of each part of the image, performs image area separation of the image of the document using the obtained image feature, and performs the feature calculation. The unit uses the result of the calculation performed by the image region separation unit to obtain the image feature for image region separation when obtaining the image feature for each image region.

  In another preferred aspect, the structuring unit corresponds to each image area in the layout, and a layout presentation unit that presents a layout of each image area obtained by the processing of the image area separation unit to the user. The specification of the element of the new structured document is received from the user, the position and image feature of each image area are associated with the element specified by the user for the image area, and the structure information of the new structured document is A registration processing unit registered in the structure information registration unit.

  The best mode for carrying out the present invention (hereinafter referred to as “embodiment”) will be described below with reference to the drawings.

  FIG. 1 is a diagram illustrating a system configuration example of the present embodiment. As shown in FIG. 1, this system comprises a digital multi-function peripheral 1, a client machine 2, and a document DB (database) server 3 connected to each other via a network 4 such as a LAN (local area network). ing.

  The digital multi-function peripheral 1 is a multi-function device having functions of a network printer, a network scanner, a copying machine, a facsimile machine, and the like. In the present embodiment, the digital multi-function peripheral 1 is provided with a function of converting a document image read by the scanner function into structured document data in a structured description language such as XML. This conversion function aims to automate routine tasks such as generating structured document data from a paper document and registering it in the document DB server 3.

  For the function of converting a document image into a structured document, an image area separation function provided in the digital multi-function peripheral 1 is used. That is, an image forming apparatus such as a copying machine or a multi-function machine is generally provided with an image area separation function for separating a character area and an image area in a read image in order to improve print image quality. . For high-speed printing processing, many parts of the image area separation function are implemented as hardware circuits such as an ASIC (Application Specific Integrated Circuit) and a DSP (Digital Signal Processor). In the present embodiment, the conversion into a structured document is made efficient by using such an image area separation circuit that has been expanded as necessary.

  However, the image area separation process can identify the areas such as character areas and image areas in the image, but structural information such as which element of each type of structured document each area can be obtained only by image area separation. I can't. Therefore, in the present embodiment, the user modifies and edits the provisional automatic conversion result of the digital multi-function peripheral 1 to determine the structure information of the structured document, and the structure information is fed back to the digital multi-function peripheral 1. By doing so, we take the approach of improving the accuracy of automatic conversion. In the example of FIG. 1, the provisional conversion result of the digital multi-function peripheral 1 is passed to the structured document editor 2a installed in the client machine 2, and the user edits the structural information on the editor 2a.

  The main part of the control mechanism of the digital multi-function peripheral 1 is shown in FIG. In FIG. 2, a ROM (Read Only Memory) 12 stores digital information such as a control program for controlling the operation of the digital multi-function peripheral 1. A CPU (Central Processing Unit) 10 executes a control program in the ROM 12 to realize control of each part of the digital multi-function peripheral 1.

  A RAM (Random Access Memory) 14 is a main storage device of the digital multi-function peripheral 1 and is also used as a work memory when executing a control program. The RAM 14 can also be used as a page buffer that stores image data for one page to be supplied to the print engine 24, for example.

  The mass storage device 16 is an auxiliary storage device for storing various data, and is a non-volatile storage device such as a hard disk or an EEPROM (Electrically Erasable Programmable Read-Only Memory).

  The operation panel 18 is a user interface means for displaying the user interface of the image forming apparatus and receiving input of various instructions from the user. The operation panel 18 includes, for example, mechanical operation buttons such as a start button, and a liquid crystal touch panel for GUI (graphical user interface). The liquid crystal touch panel displays a GUI screen generated by the control program executed by the CPU 10, detects the touch position of the user on the display, and passes it to the control program. The control program interprets the input content of the user from the information on the touch position.

  The communication interface 20 is a device that performs control for data communication with other devices on the network 4. A print instruction or the like from the remote host is input into the image forming apparatus via the communication interface 20.

  The scan engine 22 is a device that provides a scanner function for optically reading a document and generating electronic image data. Documents set on an automatic document feeder (ADF) (not shown) are fed one by one to the scan engine by the ADF function and optically read.

  The print engine 24 is a device that provides a printer function for forming (printing) image data supplied on the paper under the control of the CPU 10.

  The facsimile module 26 is a module that transmits and receives facsimile data.

  The image area separation circuit 28 is a circuit that performs image area separation of a document image read by the scan engine 22. As is well known, in image area separation, image areas such as characters and images are discriminated using various image features such as edge strength and spatial frequency of the image. Although the conventional image area separation circuit outputs the result of such image area separation, the image area separation circuit 28 according to the present embodiment also provides image characteristic data of each image area in addition to the image area separation result. Also output (details will be described later). Conventionally, the image area separation circuit is exclusively used for image output such as printing, but the image area separation circuit 28 of the present embodiment is not only for image output, but also from a document image to a structured document. It is also used for preprocessing when generating.

  The image processing circuit 30 is a circuit that performs image processing on a document image according to applications such as printing and facsimile transmission. For example, when printing, character image processing (such as edge enhancement) and image processing (such as gradation correction) are performed on the character and image image regions separated by the image region separation circuit 28, respectively. .

  A mechanism for realizing the function of converting a document image into a structured document in the digital multi-function peripheral 1 will be described with reference to FIG.

  Among the functional modules shown in FIG. 3, the image area separation circuit 28 is a hardware circuit such as an ASIC or a DSP, and the structuring unit 40 and the character recognition unit 44 are realized by software by the CPU 10 executing a program. . Further, the schema DB 42 is constructed using, for example, the mass storage device 16. However, this is only an example, and part or all of the structuring unit 40 and the character recognition unit 44 may be realized as a hardware circuit, and part of the image area separation processing is realized by software. It is also possible to do. As described above, in this embodiment, the conventional processing for generating a structured document is performed by using a conventional image area separation circuit as an internal processing mechanism of a multifunction peripheral.

  In the process of converting a document image into a structured document, a document image read by the scan engine 22 is first supplied to the image area separation circuit 28. The image area separation circuit 28 performs image area separation on the original image and calculates image feature data of each separated image area. In this image area separation processing, one or a plurality of image features such as edge intensity and spatial frequency (for example, DCT (discrete cosine transform frequency component)) of each part of the original image are obtained, and these image features are comprehensively evaluated to obtain an image. Are divided into one or a plurality of image areas, and the type of each image area (character, photograph, graphics, etc.) is determined. Even in the conventional image forming apparatus, the calculation of the image feature and the image area separation based on the calculation are all or most realized by the hardware circuit, but the image area separation circuit 28 of the present embodiment is also the same as the conventional image forming apparatus. A hardware circuit for image feature calculation and image area separation is provided.

  The image area separation circuit 28 of the present embodiment further has a function of obtaining one or more image features for each separated image area. The image characteristics of each image area obtained here include, for example, spatial frequency information of an image in the image area, a histogram of pixel values in the image area, pixel on in a binary image obtained by binarizing the image in the image area. An off ratio (for example, a ratio of on pixels), a run length histogram in the binary image, and the like can be exemplified. The spatial frequency information is, for example, an index value such as an average frequency or a peak frequency indicating the characteristics of the spatial frequency distribution in the image area, a combination thereof, or the like. As shown in FIG. 4, the histogram of pixel values is a histogram in which the number of pixels having the pixel value for each pixel value and the frequency of the pixels are counted. FIG. 4 schematically shows a pixel value histogram of the character image area. In the character image area, the contrast between the character part and the non-character part is very large. A peak appears. In the case of an image image area, the histogram does not show a prominent peak pattern as in a character image area, so that this histogram can be used to determine the image area of a character / image. Even if the number of characters, character size, font, padding, line padding, etc. are different even in the same character image area, the peak height and position of the histogram change. This histogram is used to identify individual character image areas. One clue. The same applies to the image (photo) image area, and the shape of the histogram changes variously depending on the content of the image. Therefore, this histogram can also be used as a judgment material for identifying the image image area itself. The histogram itself may be used as the image feature. In addition, an index value indicating the feature of the histogram, for example, a position (pixel value) and height (frequency) pair for each peak may be used as the image feature. In the case of a color image, a combination of histograms (or index values thereof) for each primary color such as R, G, B, etc. can be used as an image feature.

  Similarly, the run length histogram of the binarization result of the image area is a graph obtained by calculating the frequency at which the run length value appears in the binarization result of the image area for each run length value. The run length may be obtained for the on pixel, for example. In the case of a run-length histogram, the histogram itself may be used as an image feature, or an index value indicating the feature of the histogram may be used as an image feature.

  In calculating the image features for each image area, it is possible to use the image feature data of each part of the original image obtained as basic data for image area separation and the various calculation results obtained in the process of obtaining the basic data. . For example, some image features used for image area separation use image binarization results, so the binarized image obtained for image area separation calculates the image features for each separated image area. Available to: Some image area separation methods determine the spatial frequency in each part of the original image for image area separation. In such a system, the information on the obtained spatial frequency is used as the spatial frequency of each image area. This can be used to determine distribution characteristics. As described above, the calculation of the image feature of each image area according to the present embodiment can use the processing result of the existing image area separation circuit, thereby suppressing an increase in the circuit scale of the entire image area separation and image feature calculation process. be able to.

  The calculation of the image feature of the image area exemplified above can be realized as a hardware circuit of wired logic, DSP, or a combination thereof. Therefore, the image area separation circuit 28 of the present embodiment can be realized by adding a circuit for calculating the image characteristics of the image area to the conventional image area separation circuit.

  Although several image features of each image area required by the image area separation circuit 28 have been described above, these are merely illustrative. An appropriate image feature may be selected and used according to the purpose and application. It is not necessary to use all of the exemplified ones, and other than those exemplified may of course be used. The image characteristics of the image area exemplified above can be calculated by a hardware circuit. However, the present invention is not limited to this, and some of the image characteristics of the image area may be obtained by software processing. As an example of the image characteristics required by software processing, for example, the number of characters in the image area, the number of lines, and the like can be exemplified.

  As described above, the image area separation circuit 28 outputs the image area attribute 100 and the image area image 110 as shown in FIG. 3, for example. The image area attribute 100 includes position / shape information 102 indicating the position and shape of the image area, an image area type 104 indicating the type of the image area, and each image feature (spatial frequency, binarized image) of the image area. Image feature information 106 indicating the on / off ratio of the image). If the shape of the image area is a rectangle, the position / shape information 102 may be coordinate information of two vertices on the diagonal line. In addition, the position and shape of the image area can be expressed by a method used in conventional image area separation. The image area image 110 is image data of the image area. The image area attribute 100 and the image area image 110 are passed to the structuring unit 40. Note that instead of the image area separation circuit 28 passing both the image area attribute 100 and the image area image 110 to the structuring unit 40, only the image area attribute 100 is passed. The image area image 110 may be extracted from the original image read by the scan engine 22 using the shape information 102 and used.

  The structuring unit 40 creates structured document data corresponding to the document image using the information of the image area attribute 100 and the image area image 110. In this process, the structuring unit 40 refers to the schema information registered in the schema DB 42 as necessary.

  As shown in FIG. 5, schema information including a document type name 200, structure data 202, and element data 204 is registered in the schema DB 42 as shown in FIG. The document type is appropriately determined by the user. For example, the user can appropriately set the type of various documents that the user wants to register and manage in the document DB server 3 such as a weekly report, a communication form, a request form, a technical commentary, and a thesis. The document type name 200 is an identification name set by the user for the document type.

  The structure data 202 is data indicating the document structure of the structured document corresponding to the document type. As is well known, a structured document represented by SGML, XML, etc. is defined as a tree structure formed by document elements (hereinafter simply referred to as “elements”) as shown in FIG. The structured document illustrated in FIG. 6 is a document having a tree structure in which elements B and F exist as children of the element A, and elements C, D, and E exist as children of the element B. As described above, the structure data 202 is data indicating a tree structure formed by the element group of the structured document of the document type.

  The element data 204 is individual information of each element constituting the tree structure, and includes an element name 210, a position / shape 212, an image area type 214, an image feature 216, and a style attribute 222 as shown in FIG. . The element name 210 is an identification name of the element. The position / shape 212 indicates the position / shape of the area (image area) occupied by the element in the document image, and the image area type 214 indicates the type of image area of the element (distinguishment between characters, photographs, etc.). The image feature 216 is information on one or more image features that the image in the image area of the element has. The image feature 216 includes data of predetermined feature items such as a DCT frequency 218 (or spatial frequency), an on / off ratio 220 of a binarized image, and histogram information of pixel values. The style attribute 222 is style information for characters / images in the image area of the element. For example, information such as the size, font, and padding of characters in the image area is an example of the style attribute 222. The style attribute 222 is used as auxiliary information when analyzing the inside of each image area in the document image (for example, information on character size or padding is used to cut out each character during character recognition). It can be used for the purpose of describing as an attribute of an element in creating a document.

  In the above, various image features have been exemplified, but it is not necessary to register all types of image features in the element data 204 of one element. For example, some image features are suitable as indices for distinguishing between character image areas, while others are suitable as indices for distinguishing between image (photo) image areas (different types). The image area can be identified by information on the image area type). Therefore, the image data selected corresponding to the image area type of the element may be registered in the element data 204.

  It should be noted that since the details of the same kind of documents are often slightly different, it is very rare that the image features of each image area are exactly the same among the documents of the same kind. Therefore, in order to be recognized by the structuring unit 40 as a structured document of the same document type, the value of each item of the position / shape 212 and the image feature 216 is not a single “point” but an appropriate value. It is preferable to set it as a “range” having a width.

  The registration data structure of the schema DB 42 described above is conceptual only, and the actual registration data can be expressed in any representation format (for example, DTD (document type definition), XML schema) as long as it can express the above-described structure. ).

  When the image area attribute 100 and the image area image 110 of each image area in the document image are given, the structuring unit 40 refers to the schema information of each document type in the schema DB 42 and conforms to the document image. Find the type. That is, one document type is extracted, and the position / shape 212, the image area type 214, and the image feature 216 of each element of the schema information of the document type are the position / shape information 102, image of each image area of the document image. It is determined whether or not the area type 104 and the image feature information 106 are matched. If they match, it is determined that the document image belongs to the document type. In this case, for example, the position / shape information 102 of one image area coincides with the position / shape 212 of an element (or is within an allowable range set in advance from the value of the position / shape 212), and The image area type 104 of the image area matches the image area type 214 of the element, and the value of each item of the image feature information 106 of the image area matches the value of the corresponding item of the image feature 216 of the element (or If it is within the tolerance range, it is determined that the image area matches the element. If all image areas in the document image correspond to all elements of the schema information in a one-to-one correspondence and match, it is determined that the document image is the document type of the schema information. With this determination, elements corresponding to each image area of the document image are also specified. Also, ideal values are set for the items of the position / shape 212 and the image feature 216 of each element of the document type, and the position / shape of each image area of the document image and the value of each item of the image feature are set as the features of the document image. The distance between the coordinates and the coordinates indicated by the ideal value pair is obtained as an index value of the similarity between the document image and the document type, and the similarity index value is the best (in the case of distance) (Minimum) may be determined as the document type of the document image. However, in this case, if even the best similarity index value is lower than a preset threshold value, it is determined that there is no document type corresponding to the document image. If there is no corresponding document type, it is processed as a new document type (details will be described later).

  When the document type to which the document image belongs can be determined, the structuring unit 40 converts the document image into a structured document using the structure data 202 and element data 204 of the document type. That is, for each image area of the original image, if the image area is a character image area, the character recognition unit 44 that performs a known character recognition process passes the image area image to perform the character recognition process. Get text data of text in the image area. At the time of character recognition, if the style attribute 222 is passed to the character recognition unit 44, the recognition accuracy can be improved. Then, by modifying the text data with a tag of the element name 210 of the element corresponding to the image area, a description of the element corresponding to the image area is configured. In the case of an image (photo) image area, for example, an image file of the image area is created, and a reference to the image file is described in the tag of the element name 210 of the element corresponding to the image area. Construct a description of the element corresponding to the image area. Then, a description of each element is arranged in accordance with a tree structure between elements indicated by the structure data 202, and a description of an element that does not correspond to an image area is added as necessary, thereby generating a structured document.

  Next, the processing procedure of the digital multi-function peripheral 1 will be described with reference to FIG. For example, when the user selects structured document creation from the operation menu displayed on the operation panel 18 of the digital multi-function peripheral 1, sets the original on the platen or ADF, and instructs the execution of the processing, the processing procedure of FIG. 8 starts. Is done.

  In this procedure, first, the scan engine 22 reads the original (S 1), and passes the read original image to the image area separation circuit 28. The image area separation circuit 28 performs a known image area separation process on the original image, calculates the above-described image features for each image area separated as a result, and processes the results (that is, the image area attribute 100). To the structuring unit 40. The structuring unit 40 collates the image area attribute 100 of each image area with the information of each document type registered in the schema DB 42 (S3), and searches for the document type to which the document image belongs. If such a document type is found (the determination result in S4 is affirmative (Y)), the structuring unit 40, as described above, according to the structure data 202 and element data 204 of the document type most suitable for the document image. The document image is converted into structured document data (S5), and the obtained structured document data is registered in the document DB server 3 (S6).

  When the document type to which the document image belongs is not found from the schema DB 42 (the determination result in S4 is negative (N)), the structuring unit 40 temporarily determines from the image area attribute 100 and the image area image 110 of each image area. A structured document is created (S7), which is provided to the structured document editor 2a of the client machine 2 (S8), and is subjected to user modification / editing (S9).

  If the document read by the digital multi-function peripheral 1 is a document of a type that has never been input, the processes after step S7 are performed. For example, when the original image 300 shown in FIG. 9 is a document of an unprecedented type, an image area separation result 310 obtained by dividing the original image 300 into image areas R1 to R5 is obtained by the image area separation circuit 28. If it is found in step S3, 4 that the original image 300 does not correspond to any existing document type, in step S7, the structuring unit 40 provides a temporary element name (in this case, the image area for convenience). R1, R2,..., R5) are assigned according to the names of, and for example, a temporary document structure as shown in FIG. 10 is constructed in which each of the elements R1 to R5 is a child of the root element in the same row. To do. Further, as shown in FIG. 11, the structuring unit 40 determines the position / shape, image area type, and image characteristics (these have already been obtained by the image area separation circuit 28) of the image areas R1 to R5. Created as attribute information of elements R1 to R2 corresponding to each image area. Then, the structuring unit 40 uses a method similar to the method for creating the structured document using the schema information of the schema DB 42 described above, and determines the provisional structure from the document structure, the attribute information of each element, and the image of each image area. Create a document. In step S8, the structuring unit 40 transmits the data of the provisional structured document and the schema (data describing the information structure of the attribute structure of the document structure in FIG. 10 and each element in FIG. 11) to the client. This is provided to the structured document editor 2a of the machine 2. This provision may be realized, for example, by registering the address of the client machine 2 (such as the user's mail address) in advance in the digital copying machine 1 and transmitting the data to that address, or the data. May be stored in a user's confidential box designated in advance, and the user may access the confidential box from the client machine 2 and download the data. If the user has instructed the digital multifunction device 1 to create a structured document after performing user authentication by inputting a password, the structured document and schema information are stored in advance by the user in the digital multifunction device 1. (Alternatively, it can be sent to an e-mail address registered in the user management server from which the multifunction device 1 requests user authentication) or can be stored in the confidential box of the user.

  The client machine 2 that has received the provisional structured document and its schema information in this way activates the structured document editor 2a and edits the data of the provisional structured document (and schema). The structured document editor 2a displays a provisional structured document image 320 on the display screen of the client machine 2 as shown in FIG. In FIG. 12, in order to avoid complication, the outline of the image area corresponding to each of the elements R1 to R5 is shown as the provisional structured document image 320. Display the contents of the image area (text, images, etc.). Further, an image indicating the range of each image area (for example, an outline of the image area) may be superimposed on the display. For the temporary structured document displayed in this way, the structured document editor 2a accepts an input of a document type name and an element name of each element (image area) from the user. For the input of the element name, for example, when an element of the displayed structured document is selected by clicking the mouse, etc., a dialog box for inputting the element name is displayed, and the user interface such as the user inputting the element name is displayed. Use it. Similarly, a user can input a style attribute for a selected element. The style attributes, such as character size and font, may be selected from options prepared in advance, for example, in the form of a pull-down menu. By such user editing work, a structured document 330 in which the document type name, the element name of each element, and the like are determined can be created. In this example, the tree structure does not change just by specifying the element name of each element. Therefore, the temporary element name tree shown in FIG. 10 is the same tree structure as shown in FIG. It just changes from the true to the real. Here, for example, the input document type name is set as the element name of the root element.

  The information such as the document type name, each element name, and the style attribute input in this way is registered in the schema of the structured document (for example, FIG. 11), thereby completing the editing of the schema of the structured document. . The other information of the schema, that is, the position / shape, the image area type, the image feature, and the like, the information content from when it was provided from the digital multifunction peripheral 1 is maintained.

  As described above, it is possible to perform editing at the structure level, such as inserting an element that bundles existing elements as well as simply inputting an element name. This is realized by a user interface mechanism such as accepting the range specification of the range containing the elements to be bundled on the screen displaying the temporary structured document, and accepting the input of the element name for the range. it can. Thereby, as illustrated in FIG. 14, for example, an element “representative item” that bundles the elements “title”, “author name”, and “summary” can be inserted as a child of the root element “report”. The editing result on the tree structure is also reflected in the schema information (the structure information portion).

  The user sends the schema information of the structured document edited in this way to the digital multi-function peripheral 1 and registers it in the schema DB 42.

  Once the schema information of a certain document type is once registered in the schema DB 42 of the digital multi-function peripheral 1 in this way, if an original of the same document type is subsequently input to the digital multi-function peripheral 1, each image area of the original image is In order to match the position / shape and image feature of each element indicated by the schema information, the schema information is used to convert it into a structured document. As a result, for example, as shown in FIG. 15, when a report entitled “Investigation of XXX” is input to the digital multi-function peripheral 1 and conversion to a structured document is instructed, By processing according to the schema information, the text and images in each image area of the document “Investigating XXX” are associated with each element of the “report”, and a structured document is created based on this. .

  As described above, according to the present embodiment, once schema information created by the user is registered in the schema DB 42 of the digital multi-function peripheral 1, an original of the same document type is input to the digital multi-function peripheral 1 thereafter. Structured document data is automatically generated from the image according to the schema information. In this automatic processing, the image features of each image area of the original image are compared with the image features of each element of the structured document (this is included in the schema information) to determine which element each image area corresponds to. Since the determination is performed, it is possible to perform a determination with higher accuracy than a method in which the determination is made only by comparing the image area and the position / shape of the element. Further, since the image feature used at this time is obtained by hardware processing of the image area separation circuit 28, high-speed processing and real-time processing can be realized.

  In the present embodiment, when a document of a new document type is input, the digital multi-function peripheral 1 generates a temporary structured document and its schema information by the image area separation circuit 28 and the structuring unit 40. Since this is provided to the user, the user can complete the schema information with a relatively simple editing operation.

  Next, a modification of the above embodiment will be described with reference to FIG. In FIG. 16, the same steps as those shown in FIG.

  In the above embodiment, the schema information registered in the schema DB 42 registers the position / shape of each image area and various image feature values based on the schema information. The document of the same document type is analyzed. However, even in the case of documents of the same document type, the position / shape of each element and the value of each image feature vary to some extent, so when determining the document type of a document image, the value of each image area of the document The document type cannot be determined if the strict match between the value of each element of the schema information is strictly requested. However, if the allowable range for determining coincidence is too wide, misjudgments increase. Appropriate tolerances are considered to be different for each document type and image area / element. Therefore, in this modification, a process for appropriately determining the allowable ranges of the values of the items of the position / shape 212 and the image feature 216 in the schema information using the reading results of many originals belonging to the same document type will be described.

  In this process, when the user knows the document type of a document to be input as a structured document creation target, the user inputs the type to the digital multifunction peripheral 1. That is, for example, when structured document creation is instructed on the menu, a list of document type type names registered in the schema DB 42 is displayed on the display of the operation panel 18, and the document type of the document to be input from among them is displayed. The user may select it. If the user knows the document type, the user may instruct the start of processing after selecting the type, and if not known, the user may instruct the start of processing without selecting it.

  When the start of processing is instructed, the digital multi-function peripheral 1 reads the document (S1), and performs image area separation and image feature calculation (S2). If no document type has been selected, the process proceeds to step S3 and subsequent steps in FIG. On the other hand, when the document type is selected, the structuring unit 40 obtains the schema information of the selected document type from the schema DB 42, and the position / shape 212 of each element indicated by the schema information, the image area The type 214 and the image feature 216 are collated with the position / shape information of each image area of the original image, the image area type, and the image feature information (S12). It is determined whether or not to perform (S13). Since this collation / determination is for the purpose of detecting an error in the user's document type selection, it is a judgment criterion that is sweeter than the determination in steps S3 and S4 in FIG. 8 (for example, the allowable range is larger than that in the case of S3 and S4. Etc.). If it is determined that they do not match, the structuring unit 40 performs error processing such as displaying a message on the operation panel 18 that suggests the possibility of an error in selecting the document type (S14). On the other hand, if it is determined that they match, the structuring unit 40 converts the document image into a structured document according to the schema information of the selected document type (S15), and registers the conversion result in the document DB server 16. (S16). Further, the structuring unit 40 uses the values of the items of the position / shape information 102 and the image feature information 106 of each image area of the document image to determine the position / shape 212 of the schema information and the image feature 216 in the schema DB 42. It is reflected in the information of each item (S17). Here, for example, when the position / shape of the image area of the document image is out of the allowable range of the position / shape of the corresponding element registered in the schema information, the value of the position / shape of the image area is included. Correct the tolerance. Further, when the position / shape and the like in the schema information are defined by statistical data such as an average value and variance, the statistical data is corrected using information on each image area of the original document image that has been processed. When each item of schema information is represented by statistical data, whether or not each image area of the document corresponds to each element is statistically determined that the position / shape of each image area is an element of the document. The certainty may be obtained from the statistical data of the element, and the determination may be made based on the certainty.

  The preferred embodiments and modifications of the present invention have been described above. However, what has been described above is merely an example, and various modifications can be made within the scope of the present invention. For example, in the above example, the schema information of the provisional structured document created by the digital multifunction device 1 is corrected by the structured document editor 2a of the client machine 2. However, the present invention is not limited to this. The above-described schema information editing function may be provided.

  In the above example, the original image read by the digital multi-function peripheral 1 is subjected to image area separation by the digital multi-function apparatus 1 and converted into a structured document. Each of the functions described above may be executed by a device other than the device that reads the document image. This is also true for the following modifications. That is, in the following, a case where a further modification is realized by the above-described digital multi-function peripheral 1 will be described as an example, but the functions of the modification described below are executed by a device different from the device that read the document image. May be.

  In this case, another device may be realized, for example, by executing a program describing the function or processing content of each unit described above on a general-purpose computer. The computer has, for example, a circuit configuration in which a CPU (Central Processing Unit), a memory (primary storage), various I / O (input / output) interfaces and the like are connected via a bus as hardware. Also, a disk drive for reading various types of portable non-volatile recording media such as an HDD (hard disk drive), a CD, a DVD, and a flash memory is connected to the bus via, for example, an I / O interface. Such a drive functions as an external storage device for the memory. A program in which the processing content of the embodiment is described is stored in a fixed storage device such as an HDD via a recording medium such as a CD or DVD, or via a network, and is installed in a computer. The program stored in the fixed storage device is read into the memory and executed by the CPU, whereby the processing of the embodiment is realized.

  The processing procedure of another modification will be described with reference to FIG. In the procedure of FIG. 17, the image area separation circuit 28 of the digital multi-function peripheral 1 separates the image area of the original image to be structured, and the structuring unit 40 uses the result of the image area separation to make a provisional structure. A document and schema information (for example, a flat structure as illustrated in FIG. 10) is created (S21). Further, the separation accuracy (referred to as image region separation accuracy) in the image region separation result is calculated (S22) (S22 is shown after step S21 for convenience, but the calculation of the image region separation accuracy is the same as the image region separation process. May be run in parallel). The image area separation accuracy can be calculated for each individual image area obtained by the image area separation process.

  In other words, in image area separation, image features are obtained for image units such as pixels or a block made up of a plurality of pixels, and for each image unit, the image units are converted to character, continuous tone ( It is determined whether the image belongs to an image area type such as a photograph) image or an error diffusion image. For this determination, for example, a range of image feature values (or a set of a plurality of image feature values; hereinafter collectively referred to as “image feature values”) is defined for each image area type. If the image feature value of the image unit is included in a range corresponding to a certain image area type, it is determined that the image unit corresponds to the image area type. However, when the automatic determination is made according to whether or not the image is included in the range, the image feature value of the image unit is different from the range as long as the image feature value is included in the range corresponding to the image area type. If the image unit is located at the boundary with the range of the image area type, the image unit is equal to the range of the other image area type, and the position of the image area type is high. It will be determined as the image area type. However, originally, the former (when located at the boundary) is less likely to be the image area type of the determination result than the latter. As described above, even when it is determined that the image unit is the same image area type, the determination result is more likely in the latter than in the former. This certainty is the aforementioned image area separation accuracy. For example, an evaluation formula for determining the probability that the image unit belongs to the image area type, that is, the image area separation accuracy, using the image feature value of the image unit and the range of the image feature value corresponding to the image area type. Alternatively, an evaluation rule or the like can be defined. For example, in a simple example, if the image feature value is included in the range from a to b, the image unit belongs to the image area type A with an image area separation accuracy of 0.1, and if included in the range from b to c, Rules such as belonging to image area type A with an image area separation accuracy of 0.9 can be defined.

Although an example has been shown above, the image feature value for each image region is set to a typical value of the image feature in each image region type regardless of the method used to determine the image region type, not limited to the above-described method. Depending on how close or far it is, the probability that the image unit belongs to each image area type can be quantified. The function for obtaining the image area separation accuracy can be provided in the image area separation circuit 28.

  By combining the image area separation accuracy of each image unit obtained by such an evaluation formula or evaluation rule, the image area separation accuracy for the entire image area separation result of the original image can be obtained. Various calculations can be considered for obtaining the total image area separation accuracy of each image unit. For example, one of the methods is to use the lowest value of the image area separation accuracy of each image unit as the image area separation accuracy of the entire image area separation result. Further, the entire image area separation accuracy may be obtained using a predetermined calculation formula (for example, an expression for obtaining an average or an expression for obtaining a mean square) from the image area separation accuracy of each image unit. Which calculation formula is appropriate depends on the type of image feature used and on the content of the image area separation process, but in any case, the entire image area separation result is determined from the image area separation accuracy of each image unit. It is possible to define the image area separation accuracy.

  When the image area separation accuracy of the entire image region separation result is thus obtained, it is determined whether or not the accuracy is equal to or greater than a predetermined threshold value (S23). If it is equal to or greater than the threshold value, the image area separation accuracy is sufficiently high. In this case, the digital multi-function peripheral 1 uses the provisional structured document and schema information obtained in step S21 as the formal structured document and the schema information. The schema information is registered in the document DB server 3 and the schema DB 42 (S24).

  On the other hand, if the image area separation accuracy is less than the threshold, provisional structured document and schema information are provided to the client (S25), and editing work on the structured document and schema information is received from the user (S26). The structured document and schema information reflecting the editing result are registered in the document DB server 3 and the schema DB 42 (S27). The editing in step S26 is generally the same as the editing in step S9 in the procedure of FIG. However, in the case of step S26, since the image area separation accuracy is low, there is a high possibility that erroneous image area separation is performed. Accordingly, in the editing in step S26, the position, shape, size, etc. of the image area itself indicated in the provisional schema information may be changed by the user. Such a change in the image area is reflected in schema information and a structured document conforming to the schema information. As described above, when the image area separation accuracy is lower than the threshold value, the reliability of the image area separation result itself is low, and thus user intervention is required.

  It should be noted that an appropriate value for the threshold used in step S23 may be determined in advance through experiments or the like and registered in the digital multi-function peripheral.

  As described above, in the procedure of FIG. 17, when the image area separation accuracy is equal to or higher than the threshold value, the structured document and schema information are automatically registered without bothering the user. In such structured documents and schema information that are automatically registered, there is a limited logical structure (for example, a flat structure), or the meaning of elements as illustrated in FIG. 12 cannot be given (or It is not impossible to give meaning to some extent by language analysis or other advanced analysis processing, but high-precision meaning cannot be expected with such automatic processing). However, even if these points are taken into account, if it is more important for the user to save labor, it is meaningful to use the procedure shown in FIG.

  Next, a modification of the procedure in FIG. 17 will be described with reference to FIG. For the processing of FIG. 18, image area separation assumed to be obtained when the image of the document corresponding to the document type is associated with the schema information of each document type in the schema DB 42 and the image area is temporarily separated. (In other words, the determination result of the image area type for each image unit) and the image area separation accuracy for each image unit are registered as reference values. Such a reference value can be obtained, for example, by actually performing image area separation on an image of a document corresponding to the document type. The image area separation result and image area separation accuracy required when a document belonging to a new document type is read into the system can be used as a reference value for the document type. The average image area separation result and image area separation accuracy of a plurality of documents belonging to the document type may be used as the reference value.

  In the procedure of FIG. 18, the digital multi-function peripheral 1 performs image area separation on the document image (S31), and obtains the result of the image area separation and the image area separation accuracy for each image unit (S32). Then, the determination result of the image area type of each image unit, the image area separation accuracy of each image unit, and the image area separation result and image area separation accuracy associated with the schema information of each document type in the schema DB 42 From the comparison with the value, a document type that matches the document image is searched (S33). In step S33, for example, the distance between the image area separation result and image area separation accuracy of the document image and the reference value corresponding to the document type is obtained. If the distance is equal to or less than a predetermined threshold, the document type is the document image. (That is, the possibility that the document image belongs to the document type can be said with a certainty or more with a certain threshold or more) and can be extracted. Then, the document image is converted into a structured document in accordance with the schema information corresponding to the document type most suitable for the document image among the retrieved document types (for example, the one having the smallest distance as described above) (S35). ). The conversion process in step S35 may be the same process as step S5 in the procedure of FIG. Then, the structured document generated as a result is registered in the document DB server 3.

  If it is determined in step S34 that there is no document type that matches the document image, for example, the process may proceed to step S22 in the procedure of FIG. In this case, the digital multi-function peripheral 1 obtains the image area separation accuracy of the entire original image, and if this is equal to or greater than a predetermined threshold, the original image is a new document type (ie, a document that has not been registered in the schema DB 42 until now). The provisional structured document and schema information generated from the image area separation result of the original image are registered in the document DB server 3 and the schema DB 42 (S24). If the image area separation accuracy of the entire original image is less than the threshold value, provisional structured document and schema information are provided to the user, and confirmation or editing is performed on the document, and the resulting structured document and schema are obtained. Information is registered in the document DB server 3 and the schema DB 42 (S25 to S27).

  If it is determined in step S34 that there is no document type that matches the document image, the processing from step S3 onward in the procedure of FIG. 8 may be performed instead of the above processing.

  Next, a further modification will be described with reference to FIG. For the processing of FIG. 19, a representative image of a document corresponding to the document type is registered in the system in association with the schema information of each document type in the schema DB. For example, the representative image may be one of specific document images corresponding to the document type, or may be an average image of images of a plurality of documents corresponding to the document image. The representative image of each document type may be stored anywhere as long as it can be accessed from the digital multi-function peripheral 1. For example, the representative image may be stored in the schema DB 42 or may be stored in the document DB server 3.

  In the procedure of FIG. 19, when the digital multifunction peripheral 1 acquires a document image to be structured, the similarity to the document image is greater than or equal to a predetermined value from the representative images of each document type registered in the system. A certain thing is searched (S41). The similarity of the representative image to the document image can be obtained by an existing method for obtaining the similarity between images. For example, the difference between the values of the corresponding pixels of the original image and the representative image is obtained by calculating the root mean square over all the pixels as the distance between the original image and the representative image, and the similarity is higher as the distance is shorter. Calculating similarity, etc. Such similarity between images can be obtained by hardware circuit or software processing. The system may be equipped with hardware or software for obtaining such similarity.

  Then, it is determined whether or not there is a document type having a similarity greater than or equal to a predetermined value (S42). If such a document type is found, the document is determined according to the schema information of the document type having the highest similarity among them. The image is converted into a structured document (S43). This conversion process may have the same processing contents as step S5 in the procedure of FIG. The obtained structured document is registered in the document DB server 3 (S44).

  Further, instead of such processing, when a plurality of document types having a similarity equal to or higher than the predetermined value in step S42 are found, the processing after S2 in FIG. 8 may be performed. In this case, instead of searching for all document types registered in the schema DB 42 for document types that match the document image, it is only necessary to search for document types having a similarity equal to or higher than the predetermined value.

  If it is determined in step S42 that there is no document type having a similarity greater than or equal to a predetermined value, for example, the process of FIG. 17 may be performed. Alternatively, the process of FIG. 8 may be performed, or the process of FIG. 18 may be performed.

  As described above, in the processing procedure of FIG. 19, before performing the image area separation and the structuring process based thereon, the document types to which the document image is likely to belong are narrowed down by the similarity at the image level. Processing for structuring based on image area separation is performed on the narrowed-down result. Therefore, the time required for processing for structuring based on image area separation is shortened.

  In each of the examples described with reference to FIGS. 17 to 19, threshold values are used for various determinations. A user interface for setting or changing such a threshold value is a digital composite. It may be provided in the machine 1. Further, a user interface for setting whether or not to execute each process described with reference to FIGS. 17 to 19 may be provided in the digital multi-function peripheral 1.

It is a figure which shows the system configuration example of embodiment. It is a figure which shows the hardware constitutions of the control mechanism of the digital multi-functional peripheral of embodiment. FIG. 3 is a functional block diagram illustrating details of a function of converting a document image into a structured document in the digital multi-function peripheral according to the embodiment. It is a figure which shows the example of the histogram of a pixel value. It is a figure which shows the schematic data content of the schema information registered into schema DB. It is a figure which shows the example of the tree structure which structure data shows. It is a figure which shows the example of the structure of element data. 10 is a flowchart illustrating an example of a procedure of structured document creation processing executed by the digital multifunction peripheral. It is a figure for demonstrating provisional structuring processing when there is no document type corresponding to a manuscript image. It is a figure which shows an example of the tree structure of a temporary structure result. It is a figure which shows an example of the element data of a temporary structure result. It is a figure for demonstrating editing operation on a structured document editor. It is a figure which shows an example of the tree structure of the edited structured document. It is a figure which shows another example of the tree structure of the edited structured document. It is a figure for demonstrating the process of the structure part when the original of the document classification to which schema information was registered is read. It is a flowchart which shows the process sequence of a modification. It is a flowchart which shows the process sequence of another example. It is a flowchart which shows the process sequence of another example. It is a flowchart which shows the process sequence of another example.

Explanation of symbols

  1 Digital MFP, 2 Client machine, 2a Structured document editor, 3 Document DB server, 4 Network, 28 Image area separation circuit, 40 Structured part, 42 Schema DB, 44 Character recognition part, 100 Image area attribute, 102 Position Shape information, 104 image area types, 106 image feature information, 110 image area images.

Claims (18)

A reading section for reading a document;
An image area separating unit that separates an image area of an image of a document read by the reading unit;
An image processing unit that performs image processing according to the image type of each image area for each image area separated by the image area separation unit;
An output unit that outputs an image of a document image-processed by the image processing unit;
An image forming apparatus comprising:
A feature calculator for obtaining image features for each image area separated by the image area separator;
A structure information registration unit in which the position and image feature of each element of the structured document are registered as structure information of the structured document;
Each image area is structured based on the position of each image area obtained by the image area separation unit, the image feature of each image area obtained by the feature calculation unit, and the information registered in the structure information registration unit. A structuring unit that associates each element of the document and generates a structured document corresponding to the document image based on the correspondence result;
An image forming apparatus comprising: The image area separation unit obtains image features of each part of the image, and performs image area separation of the image of the document using the obtained image features.
The feature calculation unit uses an operation result performed by the image area separation unit to obtain an image feature for image area separation when obtaining an image feature for each image area.
The image forming apparatus according to claim 1.   3. The image forming apparatus according to claim 2, wherein the image forming apparatus is a digital multi-function peripheral, and the image area separation unit and the feature calculation unit are configured as an internal processing mechanism of the digital multi-function peripheral.   The image forming apparatus according to claim 1, wherein the image feature of each image area obtained by the feature calculation unit is a distribution of spatial frequency components of the image in the image area.   The image forming apparatus according to claim 1, wherein the image feature of each image area obtained by the feature calculation unit is a ratio of black and white pixels in a binarization result of an image of the image area.   The image forming apparatus according to claim 1, wherein the image feature of each image area obtained by the feature calculation unit is an appearance frequency distribution of each pixel value in the image area.   The image forming apparatus according to claim 1, wherein the image feature of each image area obtained by the feature calculation unit is an appearance frequency distribution of each run-length value in the binarization result of the image of the image area.   2. The image forming apparatus according to claim 1, wherein an image feature corresponding to an image area type of an element of the structured document is registered in the structure information registration unit. The structured part is
A layout presentation unit that presents a layout of each image area obtained by the processing of the image area separation unit to a user;
The specification of a new structured document element corresponding to each image area in the layout is received from the user, the position and image feature of each image area are associated with the element specified by the user for the image area, and A registration processing unit for registering in the structure information registration unit as structure information of a new structured document;
The image forming apparatus according to claim 1, further comprising: The layout presentation unit provides the layout to a predetermined user terminal,
The registration processing unit registers the position and image feature information of each element of the structured document input from the user terminal with respect to the provided layout in the structure information registration unit as new structured document information. To
The image forming apparatus according to claim 9.   The structuring unit, when the position and image feature of each image area separated by the image region separation unit does not match the position and image feature of each element of the structured document registered in the structure information registration unit, The image forming apparatus according to claim 9, wherein the layout presenting unit and the registration processing unit perform processing for registering a new structured document. An image area separating unit that separates an image area of a document image obtained by reading an original, and outputs each separated image area in association with the position and image characteristics of the image area;
Based on the position and image feature associated with each image area separated by the image area separation unit, each image area is associated with each element of the structured document, and based on this association result, the original image is associated. A structured part for generating a structured document;
An image processing apparatus comprising: An evaluation unit for evaluating the accuracy of image area separation by the image area separation unit;
When the accuracy evaluated by the evaluation unit is higher than a predetermined threshold, structure information having each image area as a result of image area separation by the image area separation unit is created and registered in the structure information registration unit A registration department to
The image forming apparatus according to claim 1, further comprising: An evaluation unit for evaluating the accuracy of image area separation by the image area separation unit;
When the accuracy evaluated by the evaluation unit is higher than a predetermined threshold, structure information of a structured document having each image area as a result of image area separation by the image area separation unit is created and registered as structure information A registration processing unit to register with the
Further comprising
The structuring unit refers to the structure information registered in the structure information registration unit and generates a structured document corresponding to the document image;
The image processing apparatus according to claim 12. An evaluation unit for evaluating the accuracy of the image area separation by the image area separation unit;
The structure information registration unit
In the structure information registration unit, the structure information including the image area separation accuracy obtained by the evaluation unit as at least one of the image features is registered,
The structured part is
By searching the structure information registration unit using the image area separation accuracy obtained by the evaluation unit for the document image as a key, one or more structure information corresponding to the document image is obtained, and the structure information is obtained based on the obtained structure information. Generate a structured document corresponding to the original image,
The image forming apparatus according to claim 1. An evaluation unit for evaluating the accuracy of the image area separation by the image area separation unit;
The structure information registration unit
In the structure information registration unit, the structure information including the image area separation accuracy obtained by the evaluation unit as at least one of the image features is registered,
The structured part is
By searching the structure information registration unit using the image area separation accuracy obtained by the evaluation unit for the document image as a key, one or more structure information corresponding to the document image is obtained, and the one or more structure information is included in the one or more structure information. Generating a structured document corresponding to the original image,
The image processing apparatus according to claim 12. A document image storage unit that stores an image of a document corresponding to the structure information corresponding to each structure information registered in the structure information registration unit;
A search unit that searches the document image storage unit for an image having a similarity equal to or higher than a predetermined threshold with respect to the document image read by the reading unit;
Further comprising
The structuring unit generates a structured document corresponding to the document image based on the structure information corresponding to the image searched by the search unit;
The image forming apparatus according to claim 1. A document image storage unit for storing a representative image representing each document type;
A search unit that searches the document image storage unit for a representative image having a similarity equal to or higher than a predetermined threshold with respect to the document image;
Further comprising
The structuring unit associates between each image area of the original image output from the image area separation unit and each element of the structured document of the document type corresponding to the representative image searched by the search unit. Generate a structured document by doing
The image processing apparatus according to claim 12.

Images