JP2006293970A - Image processor, image processing method and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To execute processing based on writing content of a paper document written by a user. <P>SOLUTION: This image processor has a paper document reading means having a content area in which content is drawn and for reading the paper document written by the user and an extraction means for extracting the content in the content area in considerable positional relation with a writing area including a written image written in the read paper document or the written image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for processing a written document.

  In recent years, many electronic documents have been created using information devices such as computers, and the amount and the frequency of creation tend to increase more and more. In addition, an existing electronic document is used to create a new electronic document, or a part of the information is used later.

  In this case, start the application that edits the electronic document on the computer, read the electronic document, specify the area, copy the data in the electronic document to the memory, paste it into another window of the application, etc. The content of electronic documents is reused.

  Also, from the viewpoint of visibility, listability, transfer to people, etc., even if an electronic document is distributed and at hand, you can think by printing it once on paper and reading it, I often write on paper documents.

To re-digitize the written contents, a paper document is read with a scanner and stored as an image file. It is distinguished from the original paper document and the written paper document by the difference between the printed characters and the written characters.
JP 2002-318799 A Japanese Patent Laid-Open No. 9-305701

  As described above, it is not easy in the conventional system to reflect the written contents of the paper document written by the user in the electronic document.

  Patent Document 1 relates to a calibration work using paper and electronics. A technique for realizing an environment in which calibration on a screen and calibration on a conventional paper is mixed is disclosed. In this patent publication, a proofreading instruction is written with special ink on a proofreading document printed on paper, only the special ink is extracted by reading only the special ink, and the electronic document is displayed in a superimposed manner. Seamless calibration work on paper and screen has been realized. However, this patent public information improves the visibility of the display of writing as an image, and does not extend to the use of electronic document content by writing.

  Patent Document 2 discloses a technique for extracting an underline from a paper form image. This patent document 2 extracts lateral run-length data that does not calibrate a frame in a form image, creates a histogram, and determines an underline. Also, characters in the image are cut out and identified as character data, which is used as judgment information for form recognition. However, in Patent Document 2, the underline is printed, and a content having a special meaning for identifying a form is arranged as content, and the user writes it at will. They are used for different purposes and are identified from images, and there is a risk of recognition errors.

  In view of such problems, an object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program that execute processing based on the written contents of a paper document written by a user.

  In order to solve the above problems, the present invention has a content area in which content is drawn, a paper document reading means for reading a paper document written by a user, and a written image written on the read paper document And an extraction means for extracting the content of the content area or the written image that is in a considerable positional relationship with the writing area.

  Moreover, in order to solve the said subject, this invention is characterized by the said writing area | region being a rectangle.

  In order to solve the above problem, the present invention is characterized in that the substantial positional relationship is a relationship in which the writing area and the content area have a common area.

  In order to solve the above problem, the present invention is characterized in that the substantial positional relationship is a relationship in which the content area is positioned in the vicinity of the writing area.

  In order to solve the above problem, the present invention is characterized in that the neighborhood is a neighborhood of a long side of the rectangle.

  In order to solve the above-described problem, the present invention provides the extraction unit, in a case where the content is a text composed of one or more characters, a position corresponding to the writing area among the characters included in the text. Only relevant characters are extracted as the content.

  In order to solve the above-described problem, the present invention is characterized in that, when the writing image included in the writing area is not a horizontal line segment, the extracting unit extracts the writing image without extracting the content. And

  In order to solve the above problem, the present invention is characterized in that it is determined whether or not the written image is a horizontal line segment based on an aspect ratio of a side of the writing area that is a rectangle.

  In order to solve the above-mentioned problem, the present invention relates to the height of the line of characters written horizontally in the paper document and the direction in which the character is written among the sides of the writing area that is rectangular. Whether the written image is a horizontal line segment is determined based on the length of a side that is vertical.

  In order to solve the above problem, the present invention is characterized in that, when the written image is a character, the written image is determined not to be a horizontal line segment.

  In order to solve the above problem, the present invention is characterized in that a process associated with the content extracted by the extraction means is executed.

  In order to solve the above problem, according to the present invention, the process associated with the content is at least one of a moving image display process, an audio transmission process, a spreadsheet data display process, and a text data display process. It is characterized by.

  In order to solve the above-described problem, the present invention is characterized in that the process associated with the content is a process of transmitting a file associated with the content to a predetermined destination.

  In order to solve the above-mentioned problem, the present invention is characterized in that it has content acquisition means for acquiring the content from an electronic document corresponding to the paper document.

  In order to solve the above problem, the present invention is characterized in that it has a print processing means for printing the paper document together with specific information for specifying an electronic document corresponding to the paper document.

  In order to solve the above problem, the present invention has a content area in which content is drawn, a paper document reading stage for reading a paper document written by a user, and writing on the read paper document. And an extraction stage for extracting the content of the content area having a considerable positional relationship with the writing area as the area.

  Moreover, in order to solve the said subject, this invention is characterized by the said writing area | region being a rectangle.

  In order to solve the above problem, the present invention is characterized in that the substantial positional relationship is a relationship in which the writing area and the content area have a common area.

  In order to solve the above problem, the present invention is characterized in that the substantial positional relationship is a relationship in which the content area is positioned in the vicinity of the writing area.

  In order to solve the above problem, the present invention is characterized in that the neighborhood is a neighborhood of a long side of the rectangle.

  In order to solve the above-described problem, in the present invention, in the extraction step, when the content is a text composed of one or more characters, a position corresponding to the writing area among the characters included in the text. Only relevant characters are extracted as the content.

  In order to solve the above problem, the present invention is characterized in that, in the extraction step, when the writing image included in the writing area is not a horizontal line segment, the writing image is extracted without extracting the content. And

  In order to solve the above problem, the present invention is characterized in that it is determined whether or not the written image is a horizontal line segment based on an aspect ratio of a side of the writing area that is a rectangle.

  In order to solve the above-mentioned problem, the present invention relates to the height of the line of characters written horizontally in the paper document and the direction in which the character is written among the sides of the writing area that is rectangular. Whether the written image is a horizontal line segment is determined based on the length of a side that is vertical.

  In order to solve the above problem, the present invention is characterized in that, when the written image is a character, the written image is determined not to be a horizontal line segment.

  In order to solve the above problem, the present invention is characterized in that a process associated with the content extracted in the extraction step is executed.

  In order to solve the above problem, according to the present invention, the process associated with the content is at least one of a moving image display process, an audio transmission process, a spreadsheet data display process, and a text data display process. It is characterized by.

  In order to solve the above-described problem, the present invention is characterized in that the process associated with the content is a process of transmitting a file associated with the content to a predetermined destination.

  In order to solve the above problem, the present invention includes a content acquisition step of acquiring the content from an electronic document corresponding to the paper document.

  In order to solve the above problems, the present invention is characterized in that it includes a print processing stage for printing the paper document together with specifying information for specifying an electronic document corresponding to the paper document.

  In order to solve the above-mentioned problem, the present invention is an image processing program for causing a computer to execute the image processing method according to any one of claims 15 to 28.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus, the image processing method, and image processing program which perform a process based on the writing content of the paper document written by the user can be provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, a configuration example of the image processing apparatus according to the first embodiment will be described. There are two configuration examples of the image processing apparatus, which will be described with reference to FIGS. 1 and 2.

  FIG. 1 shows an image processing unit 101, an input / output device 112, a paper input device 113, a paper output device 114, and a disk device 115.

  The image processing unit 101 includes a user I / F control unit 102, an operation interpretation unit 103, an application unit 104, a scanner I / F control unit 105, a paper processing unit 106, a content processing unit 107, a printer I / F. The F control unit 108, the print processing unit 109, the ID management unit 110, and the disk I / F control unit 111 are configured.

  The image processing unit 101 communicates with a user operation command (document registration, printing) from an input / output device 112 such as a display or a keyboard or content from a paper document via a user I / F control unit 102. Information is exchanged. The operation interpretation unit 103 receives an operation command from the user I / F control unit 102, and interprets and executes the operation.

  In addition, the image processing unit 101 is connected to a paper input device 113 such as a scanner, and includes a scanner I / F control unit 105 that controls the paper input device 113 to acquire an image of a paper document. Further, the paper processing unit 106 identifies the barcode that is printed from the image of the paper document from the scanner I / F control unit 105.

  The content processing unit 107 extracts content data in the electronic document that has a considerable positional relationship with the written area from the scanned paper image and the imaged electronic document. In addition, a paper output device 114 such as a printer is connected, and a printer I / F control unit 108 that controls the paper output device 114 to operate and acquire an image of the electronic document is provided. The ID management unit 110 manages the ID table so that the relationship between the electronic document and ID (identifier), which is specific information for specifying the electronic document, corresponds one-to-one, and performs ID registration, deletion, reference, and the like. The paper processing unit 106 assigns an ID code to an electronic document stored in the disk device 115 or the like, and creates an image to be printed by a printer.

  The image processing unit 101 described above and the paper input device 113 and the disk device 115 connected thereto are the minimum configuration of the image processing device. In the case of printing a paper document, a paper output device 114 is further added to the configuration. The input / output device 112 is not particularly necessary when the specification is such that the image processing unit 101 automatically performs extraction processing on what is read by the paper input device 113.

  Next, a second configuration example will be described with reference to FIG. The description of the same reference numerals as those in FIG. 1 is omitted.

  The image processing unit 101 in FIG. 1 is configured to implement all the functions of paper scanning, electronic document printing, and electronic document storage with hardware directly connected to the image processing unit 101. On the other hand, the image processing unit 201 shown in FIG. 2 has a configuration in which all terminal hardware is arranged outside the network and used remotely.

  The image processing unit 201 is connected to an external network 203 such as a LAN (Local Area Network) via a network I / F control unit 202. On the external network 203, there are a network scanner device 204, a network printer device 205, a computer device 206 that operates as a network file server, and the like that can communicate with the image processing unit 201. Is executed remotely with the image processing unit 201. Of course, a part of each function of scanning a paper document, printing an electronic document, and storing an electronic document is internally provided like the image processing unit 101 in FIG. 1, and the other functions are provided like the image processing unit 201 in FIG. Another embodiment realized by using the outside may be used.

  The paper input device 113 and the network scanner device 204 described above correspond to a paper document reading unit. The content processing unit 107 corresponds to extraction means. The print processing unit 109 corresponds to a print processing unit.

  The image processing unit 201 described above, and the network scanner device 204 and the computer device 206 connected thereto are the minimum configuration of the image processing device. In the case of printing a paper document, a network printer device 205 is further added to the configuration. The input / output device 112 is not particularly necessary when the specification is such that the image processing unit 201 automatically performs extraction processing on what is read by the network scanner device 204.

  Next, specific processing of the image processing apparatus will be described. The processes described below are two processes: a process of printing a paper document and a process of reading a written paper document and extracting content. First, a process for printing a paper document will be described with reference to FIG. FIG. 3 shows a control procedure in the operation interpretation unit 103 when an input is received from the user I / F control unit 102.

  The operation interpretation unit 103 first receives the input from the user I / F control unit 102 (S101). It is determined whether the operation is for an application or for printing a paper document (S102, S104). If the operation is an input operation to the application, the input is passed to the application and the process ends (S103). When the input is printing of a paper document, if an electronic document corresponding to the paper document is input, it is stored in the disk device 115 via the disk I / F control unit 111 and input to the paper processing unit 106. The contents (including the stored electronic document name) are passed and the process ends (S105).

  Next, the flowchart of FIG. 4 will be described. 4 is a control procedure of the paper processing unit 106 when the operation from the user I / F control unit 102 is printing of paper in the operation interpretation unit 103 and a request (S105) is received from the operation interpretation unit 103. is there.

  First, a print request including an electronic document name is received from the operation interpretation unit 103 (S201). An ID registration request is sent to the ID management unit 110 for the electronic document name to obtain an ID (S202). The acquired ID, electronic document name, and page number to be printed are passed to the print processing unit 109 (S203), and the process ends.

  FIG. 5 illustrates an example of information stored in the ID table managed by the ID management unit 110. In the ID table 801 in FIG. 5, each row (for example, the range of the row 805) is a unit of ID registration, and each column is a registration element. Column 802 stores a unique ID in the ID table. A column 803 stores information for uniquely determining an electronic document, in this figure, a path name indicating a position in the mass storage unit 115 in the document processing apparatus 101. A column 804 stores printed page numbers.

  Next, the flowchart of FIG. 6 will be described. FIG. 6 shows a control procedure when the print processing unit 109 receives a print request from the paper processing unit 106 (S203).

  The print processing unit 109 acquires an ID, an electronic document name, and a print page from the paper processing unit 106 together with the print request (S301). The electronic document delivered from the disk device 115 via the disk I / F control unit 111 is taken out (S302). Then, the ID is imaged into a barcode or the like (S303).

  Also, a print image is created by superimposing it on a page to be printed of the electronic document together with a timing mark for correcting skew of the scanned image (S304), and the print image is printed via the printer I / F control unit 108 via the printer 114. (S305), printing is completed.

  FIG. 7 shows an example of a paper document 901 created by the print processing unit 109. In the paper document 901, a content portion 907 held in the electronic document is drawn. Further, black square timing marks 902, 903, 904, and 905 are provided at the four corners, and their positions are defined. Furthermore, the code 906 has an ID.

  Even if an image read by the scanner device 113 or the network scanner device 204 is distorted by this timing mark, correction can be made. Further, there is a code 906 in which the ID acquired by the paper processing unit 106 from the ID management unit 110 is pasted as a code image.

  This completes the description of the process for printing a paper document. Next, a process for reading a written paper document and extracting contents will be described.

  FIG. 8 is a control procedure when a scanned image is received from the scanner I / F control unit 105 in the paper processing unit 106. The paper processing unit 106 receives an image obtained by scanning a paper document from the scanner I / F control unit 105 (S401). This step S401 corresponds to the paper document reading stage. Further, it is assumed that the image here includes handwritten images 1007, 1008, 1009, and 1010 in which the paper document is rewritten, like a paper document 901 shown in FIG. 9. Further, FIG. 9 shows content areas 1001, 1002, 1003, 1004, 1005, and 1006.

  Next, skew correction is performed on the image 901 (S402). Then, the code 906 is searched, decoded, and an ID is acquired (S403). The ID is passed to the ID management unit 110, and information (such as a path name stored in the disk device 115) of the electronic document corresponding to the paper document is received (S404). Then, the scanned image and the electronic document information are transferred to the content processing unit 107 (S405), and the process ends.

  FIG. 10 shows a control procedure when information by scanning is passed from the paper processing unit 106 in the content processing unit 107 (S405). The content processing unit 107 acquires the scanned image and information (electronic document name or the like) for acquiring at least an electronic document from the paper processing unit 106 (S501).

  Then, an electronic document is acquired from the disk device 115 from the information via the disk I / F control unit 111 (S502). The acquired electronic document is converted into an image and a difference from the image from the paper processing unit 106 is taken (S503), and a written image is extracted. Then, clustering is performed on the written image on the entire paper in consideration of the group of regions (S504). As a result, four handwritten images 1007, 1008, 1009, and 1010 as shown in FIG. 9 are taken, and the coordinates of the vertices of a rectangular area surrounding these handwritten images are obtained. FIG. 11 shows an example of the rectangle, and a rectangular writing area 1401, a writing image 1008, and a content area 1003 are shown. A rectangular area including such a written image is a writing area.

  Next, the electronic document is converted into PDF (Portable Document Format) (S505). Here, the file format is such that the position of the text area and the image area can be held and the contents can be extracted, and if a driver for converting to the format image is provided, the file format is not limited to PDF. Absent.

  In S506 to S509 corresponding to the extraction stage, processing is performed for the number of acquired writing areas (n = 4). Therefore, when there is no writing area that has not been processed, the process proceeds to S511. The process proceeds to S507 and subsequent steps as processing of one writing area (S506). It is determined whether the target writing area is a text area (S507). This text area is one of the content areas and has characters written therein. There are other image areas in the content area. Note that the writing area is a text area means that the written area is a text area.

  In S507, the determination can be made by imaging the electronic document and identifying the area. Regarding the method of area identification, pixels are classified into white pixels and black pixels using one disclosed fixed threshold value or two fixed threshold values, and both the connected black pixel group and the connected white pixel group are local regions. For example, a method of detecting pixels existing in a character pixel is used.

  Here, as shown in FIG. 9, it is assumed that the content areas 1001, 1002, 1004, and 1006 are determined as text areas, and the content areas 1003 and 1005 are determined as image areas.

  For example, since the writing area 1401 and the content area 1003 have a common area, the content in the content area 1003 is extracted from the PDF file in step S510. On the other hand, content in the content area 1005 that is not so is not extracted in step S510.

  In the case of the content areas 1002, 1004, and 1010 determined as text areas, the process proceeds to S508, and it is determined whether the writing area has a common area with the content area. Text having a common area is extracted (S510).

  FIG. 12 shows an example in which the writing area 1102 of the writing image 1007 has a common area with the content area 1101 composed of a plurality of characters. The content area 1101 is made up of a plurality of content areas in units of one character, and it is determined whether or not each area has a common area. As a result, “Sashisuseso” is extracted from the content area 1101 as the character content.

  If it is determined in S508 that there is no common area, a content area in the vicinity of the writing area is searched, and contents in the content area are extracted (S509). FIG. 13 shows an example in which the writing area 1302 has no common area with the content area 1301. In this case, the content “ABC” in the vicinity of the long side of the rectangle of the writing area 1302 is extracted. FIG. 14 also shows an example in which the writing area 1202 has no common area with the content area 1201. Also in this case, the content “he” in the vicinity of the long side of the rectangle is extracted. When the processing is completed for all writing areas in S506 to S509, the extracted content is sent to the application unit (S511), and the processing is terminated.

  By doing so, the application can perform various processes using information from the content processing unit. For example, it is possible to easily create an electronic scrapbook from paper that allows a WWW service to search for extracted character content, or to store extracted image content and characters in a separate file. As described above, according to the present embodiment, a paper document printed including an ID associated with the content of the electronic document corresponds to what the user wrote based on the positional relationship between the writing area and the content. Content can be acquired and used by applications.

  In addition, by treating the writing area as a rectangle, the positional relationship with the content becomes clear, and it becomes easier to determine the content to be extracted.

  Furthermore, when the writing area has a common area with the content area, the content is targeted, so that the content targeted for writing can be extracted without any shortage and used by the application.

  In addition, even if there is no common area between the writing area and the content area, the content in the vicinity is extracted, so text content is definitely extracted even when the underline is drawn under the line like text. In addition, applications can be used.

  In addition, when the content written by the user is text content, it is possible to determine the area of the text in character units, extract the text as content, and use the application. As a result, it is possible to correctly extract not only the whole text but also a part of the text specified by the user.

  If the content written by the user is text content, the desired character content specified by the user is extracted regardless of the direction of the text content line by extracting characters near the long side of the writing area. Application can be used.

  Further, by programming an image processing method according to an OS that can be controlled by a computer, any computer equipped with the OS can be controlled by the same processing method.

  In the second embodiment, the configuration and processing shown in FIGS. 1 to 8 described in the first embodiment are not changed. Although the paper document handled in the second embodiment is the same as that shown in FIG. 7, it is a horizontal writing document as shown in FIG. 7, and it is assumed that the reference instruction to the content is written with a line segment. Also, the written paper document is shown in FIG. Accordingly, “FIG. 9” in the description of FIG. 8 is replaced with FIG.

  In the paper document shown in FIG. 15, horizontal writing is written in the direction indicated by the arrow 6001 without changing from the generally used meaning. Therefore, when the text shown in FIG. 15 is read as written horizontally, “Aiue Okeki Keko…” is obtained. The side perpendicular to the direction in which the characters are written is a side parallel to the arrow 6002. Further, the height of the row is the same as the generally used meaning, and is the length of the arrow 6002.

  FIG. 16 is a control procedure when an input by scanning is passed from the paper processing unit 106 in the content processing unit 107 (see S405 in FIG. 8). The content processing unit 107 receives information for acquiring an image and at least an electronic document from the paper processing unit 106 (S601).

  Then, the content processing unit 107 acquires an electronic document from the disc device 115 via the disc I / F control unit 111 from the information (S602). Next, the content processing unit 107 converts the acquired electronic document into an image, takes a difference from the image from the paper processing unit 106 (S603), and extracts a written image.

  FIG. 17 shows the resulting written image. From the handwritten image shown in FIG. 17, the handwritten image of the entire paper is clustered in consideration of the group of areas (S604), and five handwritten areas 7004, 7007, 7008, 7009, and 1010 as shown in FIG. Then, the coordinates of the writing area (each wavy line part) are acquired.

  Next, the electronic document is converted into PDF (S605). Here, the file format is such that the position of the text or image can be held and the content can be extracted, and it is not limited to PDF as long as a driver for converting to the format image is provided.

  In S606 to S611, since the number of acquired writing areas is processed n times (n = 5 in the example of FIG. 17), when there is no writing area that has not been processed, the process proceeds to S612, and there is a writing area to be processed. Advances to S607 and later as processing of one writing area (S606). In that case, first, the type of writing image to be processed is determined (S607). There are three means for interpreting the written image type, which will be described later.

  As a result of the interpretation of the written image type in S607, it is checked whether or not it is a horizontal line segment (S609). Here, a handwritten image of a horizontal line means a horizontally long line segment. For example, a written image that is an underline is a horizontal line segment. If the written image is a horizontal line segment, it is further checked whether there is content in the vicinity of the writing area (S610). If the content area exists, it is determined that the written image is an instruction for the content, the content at that position or a part thereof is extracted from the PDF, and the process returns to S606.

  When the handwritten image 1007 in FIG. 17 is interpreted as a horizontal line segment, “Iueoka” is extracted as a character string. When the writing area is not a horizontal component in S609 or there is no content area in the vicinity in S610, the writing area is taken out as it is as data of a comment or a picture intended by the user and held as writing data (S608). Return to S606.

  If it is interpreted in FIG. 17 that the writing area 7010 is not a horizontal line segment, the image of the writing area itself is stored, or the character recognition processing result is stored. When the processing is completed for all writing areas in S606 to S610, the extracted content and writing data are sent to the application unit (S612), and the process ends. Thus, by interpreting the type of writing cluster, the data intended by the user can be used in subsequent applications.

  Next, three processing examples will be described as processing for interpreting the type of the writing area in step S607 described above. First, the first will be described with reference to FIG. The aspect ratio (ratio = vertical length / horizontal length) of the shape of the target writing area (the circumscribed rectangle shown by the dotted line in FIG. 17) is obtained (S701). This is compared with a predetermined default value (S702). If the aspect ratio is smaller than the predetermined value, it is determined that the writing area is a horizontal line segment (S703), otherwise it is determined that it is not a horizontal line segment (S704), and the process is terminated. Thereby, it is possible to determine a thin long writing area as a line segment and a short writing with a height as a simple writing image.

  In this way, it is determined whether or not the written image is a horizontal line segment based on the aspect ratio of the sides of the rectangular writing area.

  Next, the second process will be described with reference to FIG. The vertical length of the writing area is obtained (S801). Further, the text content in the vicinity of the writing area is searched for and the line height (cheight) of the text content is obtained (S802). In the case of the writing area 1009 in FIG. 15, the text line height can be obtained by searching for text in the content area 1002 and extracting the font height attribute from the created PDF.

  Then, the height and cheight are compared (S803), and if the vertical length of the writing area is larger, it is determined that the writing area is a non-horizontal line segment (S804), otherwise it is determined as a horizontal line segment (S805). ), The process is terminated.

  The comparison in S803 may be performed with a value multiplied by a specified value in cheight. As a result, the writing area can be determined as a horizontal line segment or a writing image based on comparison with the size of surrounding characters, so that even a short writing area or a short horizontal line segment can be accurately determined.

  In this way, the determination based on the height of the line of characters written horizontally in the paper document and the length of the side of the writing area that is rectangular is perpendicular to the direction in which the character is written. Done.

  Next, the third process will be described with reference to FIG. A handwritten character recognition process is performed on the writing area (S901). It is determined whether or not the recognition result is successful (S902). If successful, it is determined to be text data (non-horizontal line segment) (S903), otherwise it is determined to be non-text data (possible horizontal line segment) (S904), and the process is terminated. Regardless of the size of the writing, an accurate judgment is made as to whether or not it is a letter. Thus, when the written image is a character, it is determined that the written image is not a horizontal line segment.

  It is also possible to combine the above-described processing for determining the meaning of the three writing areas. In this case, it is possible to more accurately determine the horizontal line segment and the non-horizontal line segment.

As described above, the application can perform various processes using information from the content processing unit. For example, it is possible to easily create an electronic scrapbook from paper that allows the WWW service to search for extracted character content and written characters, or to store the extracted image content, characters, and written comments in a separate file.
Further, according to the second embodiment, whether or not the writing itself is acquired as data by judging from the meaning of the writing on the paper printed by the user including the ID associated with the content of the electronic document. Since processing is performed by determining whether to acquire content data corresponding to the writing position, it is possible to use data instructed by the writer to use in the application.

Furthermore, according to the second embodiment, the writing shape is handled as a rectangle, and it is determined whether to acquire the writing itself as data or to acquire the data of the content corresponding to the writing position by judging from the aspect ratio. Since the process is performed, it is possible to more accurately determine a long writing as a line segment (reference to content) and a short writing with a height as writing data.
According to the second embodiment, the writing area is handled as a rectangle, and the writing area itself is acquired as data by judging from the line height of the text content related to the vertical length, or the content corresponding to the writing area Since processing is performed by determining whether to acquire an area, it is possible to accurately determine even a short writing or a short line segment.

  In addition, according to the second embodiment, whether the contents of the writing are read, the handwritten character recognition process is performed and the success or failure is determined and the writing itself is acquired as data, or the content data corresponding to the writing position is acquired. Since the process is performed, it is accurately determined whether or not the character is written by the user regardless of the size of the writing.

  As described above, according to the second embodiment, a paper document linked to an electronic document is read, and it is automatically determined whether the writing is a reference to the content or the writing data itself is meaningful. Appropriate data can be retrieved, for example, an electronic scrapbook that allows the WWW service to search for extracted text content and written characters, or to store the extracted image content, text, and written comments in a separate file Can be made easily from

  A configuration example of the image processing apparatus according to the third embodiment will be described. There are two configuration examples of the image processing apparatus, which will be described with reference to FIGS. 21 and 22. In addition, in the following description, description of the code | symbol already demonstrated is omitted.

  First, FIG. 21 will be described. In FIG. 21, the application 104 and the operation interpretation unit 103 described in FIG. 1 are eliminated, and a speaker 3104, a touch panel 3103, and a keyboard 3112 are added.

  The image processing apparatus 101 receives a paper operation instruction (document print instruction or content processing instruction) from an external input device such as the keyboard 3112 or the touch panel 3103 via the user I / F control unit 102 and passes it to the paper processing unit 106. In addition, an output request received from the paper processing unit 106 or the content processing unit 107 is sent to an external output device such as the touch panel 3103 or the speaker 3104.

  FIG. 22 is a diagram illustrating a second configuration example according to the third embodiment. Similarly in FIG. 22, the application 104 and the operation interpretation unit 103 illustrated in FIG. 2 are eliminated, and a speaker 3104, a touch panel 3103, a keyboard 3112, and a computer device 207 are added.

  The computer apparatus 207 can perform FTP (File Transfer Protocol), remote file storage, and mail transmission / reception via the image processing apparatus 201 and the network 203, and it is assumed that one or more such computer apparatuses exist.

  Hereinafter, control procedures and management forms of the paper processing unit 106, the content processing unit 107, the ID management unit 110, and the print processing unit 109 according to the third embodiment will be described in detail with reference to the drawings. First, a paper document read in the third embodiment will be described with reference to FIGS. FIG. 23 shows an original electronic document 601 to be printed. FIG. 23 shows an electronic document 601 having text portions 602, 603, and 605, a moving image portion 604, and an audio portion 605 as contents.

  A paper document obtained by printing the electronic document 601 by the processing of the print processing unit 109 is shown in FIG. This paper document has timing marks 902, 903, 904, and 905, and the positions thereof are defined. Therefore, even if an image read by the scanner devices 113 and 204 is distorted, correction is possible. In addition, there is a code 906 pasted with an ID acquired by the paper processing unit 106 from the ID management unit 110.

  A paper document 5001 shown in FIG. 25 indicates a paper document on which the paper document 901 is written. In FIG. 25, written images 5002, 5003, and 5004 are shown. The processing described below is performed on each of these documents.

  FIG. 26 is a flowchart showing a control procedure in the paper processing unit 106 when an input is received from the user I / F control unit 102. First, the paper processing unit 106 determines which instruction is input. First, it is determined whether the received input is a print request (S1001).

  If the request is a print request, parameter information (electronic document name, page number) of the electronic document is acquired from the user I / F control unit 102 (S1002). An ID is acquired by requesting the ID management unit 110 to register the parameter information of the acquired electronic document (S1003). The acquired ID, electronic document name, and page number are passed to the print processing unit 109 (S1004), and the process ends. An ID table for managing IDs is shown in FIG.

  If the input received in S1001 is not a print request, it is checked whether the input is a content processing request (S1005). If it is not a content request, the process ends. If it is a content request, the user I / F control unit 102 is requested to input process parameter information, and a transfer method and a transfer destination are acquired as the process parameter information (S1006). This input is performed on the touch panel by the user, and an input screen displayed on the touch panel will be described later.

  Next, the scanner I / F control unit 105 (or the network I / F control unit 202) is instructed to scan a paper document, and a scanned image is received (S1007). Next, skew correction is performed (S1008). Then, the code 906 (see FIG. 25) is searched, decoded, and an ID is acquired (S1009). The ID is passed to the ID management unit 110, and information on the electronic document that is the original of the paper document (path name and page number stored in the disk device 115) is received (S1010). Then, the image and electronic document information are transferred to the content processing unit 107 (S1011), and the process is terminated.

  Next, the above input screen will be described with reference to FIG. FIG. 27 shows an example of an input screen displayed on the touch panel. In this input screen, “transfer method” and “transfer destination” are input. In this case, “content confirmation”, “FTP”, “network folder”, and “E-Mail” can be designated as the transfer method. The designation method is designated by a check box in front of it.

  In the case of “content confirmation”, the content is expressed by executing processing associated with the content on the image processing apparatus 101 or 201. Specifically, to express content is to output music to a speaker if the content means music data, or to display the content using an editor or the like if the content means text. It is an example of content expression. That is, the process associated with the content is executed.

  In the case of “FTP”, the content file is transmitted to the FTP server designated as the transfer destination. In the case of “network folder”, the contents are saved as a file in the network folder of the host designated as the transfer destination. In the case of “E-Mail”, an e-mail attached with content is transmitted to the address designated as “Forward”.

  Next, the control procedure when the print processing unit 109 receives a print request from the paper processing unit 106 (S1004) will be described with reference to the flowchart of FIG. The print processing unit 109 acquires an ID, an electronic document name, and a print page from the paper processing unit 106 (S1101). The electronic document file transferred from the disk device 115 via the disk I / F control unit 111 (or the network I / F control unit 202) is taken out (S1102).

  Then, the ID is encoded into a barcode or the like (S1103), and a print image is created by superimposing it on the page to be printed with the timing mark for correcting the skew of the scanned image (S1104). At this time, data such as non-printable multimedia data is indicated by a frame such as a rectangle indicating its position, and it is easy for the user to create a print image so as to indicate the data type there.

  Printing is completed by making a print request to the printing apparatus 114 or 205 via the printer I / F control unit 108 or the network I / F control unit 202 (S1105).

  In this way, the paper document of FIG. 24 described above is printed. An example of another paper document is shown in FIG. The paper document 5200 shows content areas 5201, 5202, 5203, 5204, 5205. Each content area is described as Movie, Voice, DataSheet, Text.

  The contents of the moving image, sound, database, and text data shown in the electronic document corresponding to the paper document 5200 are moved to the pasted positions of Movie (1201, 1204, 1205), Voice (1202), Data Sheet (1203), Text. The processing of (1205) is associated with each other and is indicated by a rectangular area together with those characters.

  By writing in a desired content area, the user can browse and view a movie (Movie), voice (Voice), spreadsheet data (DataSheet), and text data (Text).

  By using this paper document, the user can determine the type of content to be extracted, and can easily grasp the position to fill in. In addition, a mark such as an icon may be used instead of a character to display the type of content.

  Next, the processing of the content processing unit 107 will be described with reference to FIG. The content processing unit 107 receives, from the paper processing unit 106, electronic document information for acquiring an image and an electronic document, a content processing method and a transfer destination as processing parameters (S1201). Then, an electronic document is acquired from the disk device 115 from the electronic document information via the disk I / F control unit 111 (network I / F control unit 202) (S1202).

  The acquired electronic document is converted into an image by the print processing unit 109, a difference from the image from the paper processing unit 106 is taken, and a written image is extracted (S1203). Then, clustering is performed on the written image on the entire paper in consideration of the group of regions (S1204).

  FIG. 31 is a diagram illustrating an image obtained by clustering the difference. In the case of FIG. 31, clustering is performed in three. They are shown as writing areas 5101, 1102, 1103. Then, the content processing unit 107 obtains the writing position by obtaining the coordinates of the vertices of these rectangular writing areas.

  Next, the electronic document is converted into PDF (S1205). Here, the file format is such that the position of the text or image can be retained and the contents can be extracted, and the file format is not limited to PDF as long as it can be converted into the format image.

  In S1206 to S1209, processing is performed for the number of acquired writing areas (n = 3). Therefore, when there is no writing area that has not been processed, the process is terminated. It progresses after S1207 (S1206).

  Next, it is checked whether the target writing area has a common area with the content area (S1207). If there is a content area having a common area with the writing area, the content object is extracted from the PDF file (S1208). Otherwise, the process returns to S1206.

  In the case of the image processing apparatus 201 having a network communication function with the outside, a usage method is extracted from the processing parameters acquired in S1201, and it is checked whether it is a content expression (S1209).

  If it is a content expression, an output request is made to the user I / F control unit 102 so that the extracted content type is expressed by an external device corresponding to the type, and the process returns to S1206 (S1210).

  In the case of the image processing apparatus 101 having no network communication function with the outside, the processing of S1209 does not exist and only the processing of S1210 is performed. For example, the writing area 5002 has a common area with the content area 5002. Since the content area 5002 indicates a moving image, the content processing unit 107 requests the user I / F control unit 102 to reproduce an image on the touch panel 3103, and to reproduce an audio on the speaker 3104 if there is audio in the moving image.

  The writing area 1102 has a common area with the content area 1003. Since the content area 1003 indicates audio, the content processing unit 107 requests the user I / F control unit 102 to perform audio reproduction through the speaker 3104. The writing area 1103 has a common area with the content area 5004. Since the content area 5004 indicates spreadsheet data, the spreadsheet application is activated and requests that the image be displayed on the touch panel 3103.

  On the other hand, if the usage method is not content expression in S1209, the protocol processing is performed for the method designated either FTP, network file, or E-Mail and the transfer destination, the content is transmitted, and the process returns to S1206 (S1211). . Alternatively, the processing of S1207 to S128 may be performed for all clusters, the extracted content may be held, and finally all the content may be collected and the processing of S1209, S1210, and S1211 may be performed.

  The image processing apparatuses 101 and 201 according to the third embodiment described above can be configured by a computer such as a personal computer. In this case, the paper processing unit 106, the content processing unit 107, and the ID management unit 110 have their respective functions. It is configured as a program to be realized.

  According to the third embodiment, a printed image of a paper document including an ID associated with the content of the electronic document is read with respect to what the user wrote, the written image is extracted, and the group is collected. Since it is possible to express the data after extracting and extracting the data on the corresponding electronic document from the position, the user can simply write and scan the paper document to scan the multimedia or original data of the electronic document. It becomes possible to confirm easily on the spot.

  Further, according to the third embodiment, after extracting data on the electronic document corresponding to the position of the handwritten image, it is possible to transfer the extracted content data to a desired destination. By simply writing and scanning, it is possible to easily save the multimedia and original data of the electronic document in an environment for later use.

  Furthermore, according to the third embodiment, when printing is performed including an ID associated with the content of the electronic document, the user can extract by printing after putting a frame in an area where the content in the electronic document exists. It becomes easy to confirm where to write the content you want to write.

  Also, according to the third embodiment, when printing is performed including an ID associated with the content of the electronic document, if the content in the electronic document exists, the data type of the content is printed, so that the user can It becomes possible to select the type of content actually required.

1 illustrates a configuration example of an image processing apparatus according to a first exemplary embodiment (part 1). 2 is a configuration example of an image processing apparatus according to the first embodiment (No. 2). It is a flowchart which shows the process in the operation interpretation part when receiving an input from a user I / F control part. It is a flowchart which shows the process of a paper process part. It is a figure which shows an example of the information which ID table stores. 6 is a flowchart illustrating processing when a print processing unit receives a print request. It is a figure which shows the example of a paper document. It is a flowchart which shows a process when a paper processing part is given a scanned image. It is a figure which shows the image of the paper document written. It is a flowchart which shows the process of a content process part. It is a figure which shows the common area | region in the case of an image area | region. It is a figure which shows the example in which a writing area | region has a common area | region with a some character. It is a figure which shows the example in which a writing area | region does not have an area | region common with a some character (the 1). It is a figure which shows the example in which a writing area | region does not have a common area | region with a some character (the 2). It is a figure which shows the example of a paper document. It is a sequence diagram which shows the process of a content process part. It is a figure which shows a written image. It is a flowchart which shows the process which interprets the kind of writing area | region (the 1). It is a flowchart which shows the process which interprets the kind of writing area (the 2). It is a flowchart which shows the process which interprets the kind of writing area (the 3). 7 illustrates a configuration example of an image processing apparatus according to a third exemplary embodiment (part 1). 12 illustrates a configuration example of an image processing apparatus according to a third exemplary embodiment (part 2). It is a figure which shows an electronic document. It is a figure which shows a paper document. It is a figure which shows the paper document written. It is a flowchart which shows the process in a paper process part. It is a figure which shows the example of an input screen displayed on a touch panel. 6 is a flowchart illustrating processing in a print processing unit. It is a figure which shows a paper document. It is a flowchart which shows the process of a content process part. It is a figure which shows the image which took the difference and was clustered.

Explanation of symbols

101, 201 Image processing unit 102 User I / F control unit 103 Operation interpretation unit 104 Application unit 105 Scanner I / F control unit 106 Paper processing unit 107 Content processing unit 108 Printer I / F control unit 109 Print processing unit 110 ID management unit 111 Disk I / F Control Unit 112 Input / Output Device 113 Paper Input Device 114 Paper Output Device 115 Disk Device 203 External Network 204 Network Scanner Device 205 Network Printer Device 206 Computer Device 801 ID Table 802, 803, 804 Column 805 Row 901, 5200 Paper document 902, 903, 904, 905 Timing mark 906 Code 907 Content part 1001, 1002, 1003, 1004, 1005, 1006, 1101, 1201, 1301, 5201, 5202, 5203, 5204, 5205, 5206 Content area 1007, 1008, 1009, 1010, 2002, 2004, 2005, 5002, 5003, 5004, 7004, 7007, 7008, 7009, 7010 Written image 1102, 1103, 1302, 1401, 2001, 2003, 5101 Writing area

Claims (31)

A paper document reading unit that has a content area in which content is drawn and reads a paper document written by a user;
Image processing comprising: extraction means for extracting the content of the content area or the written image that has a considerable positional relationship with a writing area that is an area including a written image written on the read paper document. apparatus. The image processing apparatus according to claim 1, wherein the writing area is rectangular. The image processing apparatus according to claim 1, wherein the substantial positional relationship is a relationship in which the writing area and the content area have a common area. The image processing apparatus according to claim 2, wherein the substantial positional relationship is a relationship in which the content area is positioned in the vicinity of the writing area. The image processing apparatus according to claim 4, wherein the neighborhood is a neighborhood of a long side of the rectangle. In the case where the content is text composed of one or more characters, the extraction means extracts only characters having a considerable positional relationship with the writing area among the characters included in the text as the content. The image processing apparatus according to any one of claims 1 to 5. The image processing apparatus according to claim 6, wherein when the writing image included in the writing area is not a horizontal line segment, the extracting unit extracts the writing image without extracting the content. 8. The image processing apparatus according to claim 7, wherein it is determined whether or not the written image is a horizontal line segment based on an aspect ratio of a side of the writing area that is a rectangle. Based on the height of the line of characters written in horizontal writing on the paper document and the length of the side of the writing area that is rectangular, the side of the side that is perpendicular to the direction in which the character is written, The image processing apparatus according to claim 7, wherein it is determined whether the line segment is a horizontal line segment. The image processing apparatus according to claim 7, wherein when the writing image is a character, the writing image is determined not to be a horizontal line segment. The image processing apparatus according to claim 1, wherein a process associated with the content extracted by the extraction unit is executed. The image processing apparatus according to claim 11, wherein the process associated with the content is at least one of a moving image display process, an audio transmission process, a spreadsheet data display process, and a text data display process. The image processing apparatus according to claim 11, wherein the process associated with the content is a process of transmitting a file associated with the content to a predetermined destination. The image processing apparatus according to claim 1, further comprising: a content acquisition unit that acquires the content from an electronic document corresponding to the paper document. The image processing apparatus according to claim 1, further comprising a print processing unit that prints the paper document together with identification information for identifying an electronic document corresponding to the paper document. A paper document reading stage that has a content area in which content is drawn and reads a paper document written by a user;
An image processing method comprising: an extraction step of extracting content in the content area having a considerable positional relationship with a writing area that is an area written on the read paper document. The image processing method according to claim 16, wherein the writing area is rectangular. The image processing method according to claim 16 or 17, wherein the substantial positional relationship is a relationship in which the writing area and the content area have a common area. The image processing method according to claim 17, wherein the substantial positional relationship is a relationship in which the content area is positioned in the vicinity of the writing area. The image processing method according to claim 19, wherein the neighborhood is a neighborhood of a long side of the rectangle. In the extraction step, when the content is a text composed of one or more characters, only the characters having a considerable positional relationship with the writing area are extracted as the content among the characters included in the text. The image processing method according to any one of claims 16 to 20. The image processing method according to claim 21, wherein, in the extraction step, when the writing image included in the writing area is not a horizontal line segment, the writing image is extracted without extracting the content. 23. The image processing method according to claim 22, wherein it is determined whether or not the written image is a horizontal line segment based on an aspect ratio of a side of the writing area that is a rectangle. Based on the height of the line of characters written in horizontal writing on the paper document and the length of the side of the writing area that is rectangular, the side of the side that is perpendicular to the direction in which the character is written, 23. The image processing method according to claim 22, wherein it is determined whether or not the segment is a horizontal line segment. 23. The image processing method according to claim 22, wherein when the written image is a character, it is determined that the written image is not a horizontal line segment. The image processing method according to any one of claims 16 to 25, wherein a process associated with the content extracted in the extraction step is executed. 27. The image processing method according to claim 26, wherein the process associated with the content is at least one of a moving image display process, an audio transmission process, a spreadsheet data display process, and a text data display process. 27. The image processing method according to claim 26, wherein the process associated with the content is a process of transmitting a file associated with the content to a predetermined destination. 29. The image processing method according to claim 16, further comprising a content acquisition step of acquiring the content from an electronic document corresponding to the paper document. 30. The image processing method according to claim 16, further comprising a printing process step of printing the paper document together with specifying information for specifying an electronic document corresponding to the paper document. An image processing program for causing a computer to execute the image processing method according to any one of claims 16 to 30.

Images