JP6780380B2 - Image processing equipment and programs - Google Patents

Description

The present invention relates to an image processing apparatus and a program, and relates to a technique for converting image data obtained by reading a handwritten paper document into structured data.

Conventionally, a handwritten memo may be written with a pencil, a ballpoint pen, or the like on a paper document created and printed by a word processor or presentation material creation software. When a paper document with handwritten characters added is imaged with a scanner, etc., character recognition processing is performed, and it is converted into structured data such as an application file of Microsoft Office (registered trademark) (for example, PowerPoint (registered trademark) file). , I can't tell if the characters were originally printed on a paper document or added by hand. Therefore, it was not possible to know what kind of characters (information) were added only from the structured data.

In addition, when it is desired to share a document in a form that does not include handwritten information, it takes time and effort to erase the handwritten characters with an eraser or a correction fluid.

For example, Patent Document 1 discloses a technique of arranging a portion determined to have been added to an original document on a layer of an electronic document according to the characteristics of the addition and converting it into an integrated electronic document.

Japanese Unexamined Patent Publication No. 2012-48637

However, the technique described in Patent Document 1 requires the viewer to recognize that the electronic document has a layered structure. In addition, it is difficult for the viewer to determine which part has been added from the display of the integrated electronic document or the display for each layer.

From the above situation, it is an object of the present invention that the information added to the paper document can be easily visually recognized in the electronic image generated from the paper document.

The image processing apparatus according to one aspect of the present invention includes an area discriminating unit that analyzes image data obtained by reading a handwritten paper document and discriminates between characters and areas other than characters included in the image data. A character type determination unit that analyzes the character area determined by the area determination unit and determines whether the character existing in the character area is a printed character or a handwritten character, and a character that encodes the printed character and the handwritten character. A coding processing unit, a structured data conversion unit that converts an area other than the characters determined by the area determination unit, and a character area including a character-coded printed character into the main body attribute of the structured data, and character coding. The handwritten character exists in the supplementary attribute conversion unit that converts the handwritten character to the supplementary attribute of the structured data, and the handwritten character part in the main body area that corresponds to the main body attribute in which the handwritten character determined by the character type determination unit exists. It is provided with a handwritten character replacement unit that replaces information indicating that the data has been performed.

According to at least one aspect of the present invention, the information added to the paper document can be easily visually recognized in the electronic image generated from the paper document.
Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

It is the schematic which shows the whole structure of the system including the image processing apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the hardware structure of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the functional structure of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a flowchart which shows the operation of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the example of the input image including the handwritten character. It is a figure which shows the result of having performed the area discrimination processing on the input image which concerns on 1st Embodiment of this invention. It is a flowchart which shows the character type discrimination process which concerns on 1st Embodiment of this invention. It is a figure which shows the result of having performed the character type discrimination process which concerns on 1st Embodiment of this invention. This is an example of displaying an application screen by a PowerPoint file application according to the first embodiment of the present invention. It is a figure which shows the structure example of the file format of a general PowerPoint file including a note attribute. It is a figure which shows the display example of the application screen which arranged the handwritten character in the note area of the PowerPoint file which concerns on 1st example of 1st Embodiment of this invention. It is a figure which shows the display example of the application screen which arranged the handwritten character in the note area of the PowerPoint file which concerns on the 2nd example of 1st Embodiment of this invention. It is a figure which shows the display example of the application screen which arranged the handwritten character in the comment area of the PowerPoint file which concerns on 1st example of the 2nd Embodiment of this invention. It is a figure which shows the structure example of the file format of a general PowerPoint file including a comment attribute. It is a figure which shows the display example of the application screen which arranged the handwritten character in the comment area of the PowerPoint file which concerns on 2nd example of 2nd Embodiment of this invention. It is a figure which shows the display example of the application screen which arranged the handwritten character in the comment area of the PowerPoint file which concerns on 3rd example of the 2nd Embodiment of this invention. It is a figure which shows the display example of the application screen which arranged the handwritten character in the comment area of the PowerPoint file which concerns on 4th example of 2nd Embodiment of this invention. It is a figure which shows the display example of the application screen which deleted the content of the handwritten character information displayed in the comment area of the PowerPoint file which concerns on 5th example of 2nd Embodiment of this invention. It is a figure which shows the display example of the application screen which arranged the handwritten character in the annotation area of a PDF file which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the functional structure of the image processing apparatus which concerns on 4th Embodiment of this invention.

Hereinafter, examples of embodiments for carrying out the present invention will be described with reference to the accompanying drawings. The explanation will be given in the following order. In each figure, components having substantially the same function or configuration are designated by the same reference numerals, and duplicate description will be omitted.
1. 1. First embodiment (example of using a note area of a PowerPoint file)
2. Second embodiment (example of using a note area of a PowerPoint file)
3. 3. Third embodiment (example of using the annotation function of the PDF file)
4. Fourth embodiment (example of selecting an output mode)
5. Other

<1. First Embodiment>
[System configuration]
FIG. 1 is a schematic view showing an overall configuration of a system including an image processing apparatus according to the first embodiment.

In the system 10 of FIG. 1, the client terminal 1, the printer controller 2, the image forming apparatus 3, the scanner 4, and the camera 5 are connected to each other so as to be able to communicate with each other via the network N. The network N is a network such as a LAN conforming to a standard such as Ethernet (registered trademark). The client terminal 1, the printer controller 2, and the image forming apparatus 3 are examples of the image processing apparatus.

The client terminal 1 is a terminal device such as a personal computer (PC). The client terminal 1 transmits a print job for which print output is instructed by a user input operation to the printer controller 2 via the network N. This print job is, for example, data generated by the client terminal 1 according to PDL (Page Description Language), and includes output settings and input data. Further, the client terminal 1 receives an electronic document (image data) from another device and stores it.

The printer controller 2 causes the image forming apparatus 3 to print and output an image. The printer controller 2 receives a print job from the client terminal 1 via the network N. Then, the printer controller 2 executes a rasterization process (RIP process) on the input data taken out from the received print job to generate bitmap data (image forming data).

Although the printer controller 2 and the image forming apparatus 3 are connected via the network N, they may be directly connected. At this time, the printer controller 2 and the image forming apparatus 3 may be connected via a dedicated line such as a video interface line.

The image forming apparatus 3 forms an image on paper and outputs it based on the print job received from the printer controller 2. The image forming apparatus 3 may be a multifunction device (MFP: Multifunction Peripheral) having a plurality of types of functions (printing function, copying function, scanning function, etc.).

The scanner 4 reads the surface of the paper document P placed on the reading surface to generate image data (bitmap data), and outputs the image data (bitmap data) to the client terminal 1 or the like via the network N or wirelessly.

The camera 5 captures the paper document P, generates image data, and outputs the image data to the client terminal 1 or the like via the network N.

[Hardware configuration of each device]
FIG. 2 is a block diagram showing a hardware configuration of each device.

Here, the hardware configuration of the computer 20 constituting the client terminal 1, the printer controller 2, the image forming apparatus 3, the scanner 4, and the camera 5 shown in the system 10 described above will be described. Each part of the computer 20 is selected according to the function and purpose of use of each device.

The computer 20 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, and a RAM (Random Access Memory) 23, which are connected to the bus 24, respectively. Further, the computer 20 includes a display unit 25, an operation unit 26, a non-volatile storage 27, and a network interface 28.

The CPU 21 reads the program code of the software that realizes each function according to the present embodiment from the ROM 22 and executes it. The computer 20 may be provided with a processing device such as an MPU (Micro-Processing Unit) instead of the CPU 21.

Variables, parameters, etc. generated during the arithmetic processing are temporarily written in the RAM 23. The display unit 25 is, for example, a liquid crystal display monitor, and displays the result of processing performed by the computer 20 and the like. For example, a keyboard, a mouse, a touch panel, or the like is used for the operation unit 26, and the user can perform predetermined operation input and instruction.

Examples of the non-volatile storage 27 include HDD (Hard Disk Drive), SSD (Solid State Drive), flexible disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, non-volatile memory card and the like. Used. In the non-volatile storage 27, in addition to the OS (Operating System) and various parameters, a program for operating the computer 20 is recorded. For example, image data of an electronic document is stored in the non-volatile storage 27.

For the network interface 28, for example, a NIC (Network Interface Card) or the like is used, and various data can be transmitted and received between each device via a network N such as a LAN.

The image processing apparatus according to the present invention operates on the computer 20 as shown in FIG. Therefore, the image processing device includes a personal computer, a mobile terminal such as a smartphone or a tablet terminal, a server (for example, a printer controller 2) arranged on the network N, a multifunction device such as an MFP, a scanner 4, a camera 5, and other devices. It can also operate on a device.

[Functions of image processing device]
FIG. 3 shows the functional configuration of the computer 20 included in the image processing apparatus according to the first embodiment. Here, an example in which the image processing device is applied to the client terminal 1 will be described. In the following description, since the technique of converting the image data obtained by reading the paper document into the main body attribute of the structured data is well known (for example, Japanese Patent Application Laid-Open No. 2005-149907), detailed description thereof will be omitted.

As shown in FIG. 3, the client terminal 1 includes an input image receiving unit 31, an image data recording unit 32, an area discrimination unit 33, a character type discrimination unit 34, a character coding processing unit 35, a structured data conversion unit 36, and handwriting. It includes a character replacement unit 37, a structured data recording unit 38, and a structured data output unit 39. The functions of each part are realized by the CPU 21 of the computer 20 executing the program stored in the ROM 22.

The input image receiving unit 31 receives the image data (input image) obtained by reading the handwritten paper document P. The image data is obtained by, for example, a scanner 4 or a camera 5. Alternatively, it may be received from a server (not shown) or acquired from removable media via the network N.

The image data recording unit 32 records the image data received by the input image receiving unit 31. For the image data recording unit 32, for example, a non-volatile storage 27 is used.

The area determination unit 33 acquires image data from the input image receiving unit 31 or the image data recording unit 32 and analyzes the image data. Characters included in the image data and areas other than the characters are discriminated, and the discriminant result is output to the character type discriminating unit 34 and the structured data conversion unit 36. For example, the area determination unit 33 determines a character area, a graphic area, and a photographic area.

The character type determination unit 34 analyzes the character area determined by the area determination unit 33, and determines whether the character existing in the character area is a printed handwritten character or a printed character. The character type determination unit 34 includes a print character recognition processing unit 341, a handwriting character recognition processing unit 342, and a handwriting character determination unit 343.

The print character recognition processing unit 341 executes the print character recognition process on the image determined to be the character area. Further, the handwritten character recognition processing unit 342 executes the handwritten character recognition process on the image determined to be the character area. Further, the print character recognition processing unit 341 calculates the print character recognition degree after the print character recognition process. Further, the handwritten character recognition processing unit 342 calculates the handwritten character recognition degree after the handwritten character recognition process. The printed character recognition processing unit 341 and the handwritten character recognition processing unit 342 may be combined into one block.

The handwritten character determination unit 343 uses the printed character recognition degree calculated by the printed character recognition processing unit 341 and the handwritten character recognition degree calculated by the handwritten character recognition processing unit 342 as the printed character from the ratio of the recognition degrees of both characters. Determine handwritten characters.

The character coding processing unit 35 performs a process of character-coding the printed characters and the handwritten characters included in the character area received from the handwritten character determination unit 343.

The structured data conversion unit 36 vectorizes the input image data, converts it into the main body attribute of the structured data, and outputs it to the structured data recording unit 38 or the structured data output unit 39. The structured data conversion unit 36 includes a main body attribute conversion unit 361 and a supplementary attribute conversion unit 362.

The main body attribute conversion unit 361 converts the area other than the characters (figure, photograph) determined by the area determination unit 33 and the character area including the character-encoded printed character into the main body attribute of the structured data. The supplementary attribute conversion unit 362 converts the character-encoded handwritten characters into supplementary attributes of structured data and arranges them in the supplementary area (note area, comment area, pop-up window of annotation function, etc. described later) corresponding to the supplementary attribute. To do.

The supplementary area corresponding to the supplementary attribute is an area defined by the comment attribute of the structured data used in the document creation software, an area defined by the note attribute, or an area defined by the pop-up window of the annotation function. ..

The handwritten character replacement unit 37 replaces the handwritten character portion of the main body area corresponding to the main body attribute in which the handwritten character determined by the character type determination unit 34 exists with information indicating that the handwritten character exists. In other words, the handwritten character replacement unit 37 generates information (symbols, figures, character strings, images, etc.) indicating the relevance to the handwritten characters converted into supplementary attributes in the portion corresponding to the handwritten character unit in the main body area. ..

The structured data recording unit 38 records the structured data input from the structured data conversion unit 36. For the structured data recording unit 38, for example, a non-volatile storage 27 is used.

The structured data output unit 39 outputs the structured data output from the structured data conversion unit 36 or read from the structured data recording unit 38 to the display unit 25 or the like.

[Operation of image processing device]
FIG. 4 is a flowchart showing the operation of the client terminal 1 to which the image processing device according to the first embodiment is applied. The processing of the flowchart of FIG. 4 is realized by the CPU 21 of the computer 20 executing the program stored in the ROM 22.

First, the input image receiving unit 31 receives as an input image image data in which a paper document P in which a memo is added by hand to a printed document is electronically imaged by an electronic imaging means such as a scanner 4 or a camera 5 (S1). .. An example of the input image is shown in FIG.

FIG. 5 shows an example of an input image including handwritten characters.
In the case of the input image D of FIG. 5, the characters "too late!" Are added by hand below the line printed with "April 2017: Release" in the print area 40. The area including the arrow pointing diagonally upward to the left and the characters "too late!" Is defined as the handwriting area 41.

Next, the area discrimination unit 33 scans the input image and performs the area discrimination process to discriminate the character area, the graphic area, and the photographic area (S2). The discrimination result is shown in FIG.

FIG. 6 shows the result of performing the area discrimination processing on the input image.
The broken line frame is determined as the graphic area Ag, and the solid line frame is determined as the character area At. In the example of FIG. 6, there is no photographic area. In the handwriting area 41, the arrow is determined in the graphic area 42, and the handwritten character is determined in the character area 43.

Next, the area determination unit 33 determines whether or not the result of the area determination process is a character area (S3), and if it is an area other than the character area (NO in S3), proceeds to step S4 and proceeds to the character. If it is an area (YES in S3), the process proceeds to step S6.

Next, the main body attribute conversion unit 361 of the structured data conversion unit 36 performs recognition processing on the areas (graphic area, photographic area) other than the characters determined by the area determination unit 33 (S4), and is structured. Convert to the body attribute of the data (S5). After the process of step S5 is completed, the process proceeds to step S11.

Next, when the result of the area discrimination process is the character area (YES in S3), the character type discrimination process is performed by the character type discrimination unit 34 (S6). FIG. 7 shows a flowchart (subroutine) of the character type discrimination process.

[Character type discrimination processing]
FIG. 7 is a flowchart showing the character type discrimination process of step S5 of FIG.

First, the character area of the input image D is input to the character type determination unit 34 (S21). Next, the print character recognition processing unit 341 of the character type determination unit 34 performs the print character recognition process (S22) on the image determined to be the character area, and calculates the print character recognition degree.

Further, the handwritten character recognition processing unit 342 of the character type determination unit 34 performs the handwriting character recognition process (S23) on the image determined to be the character area, and calculates the handwriting character recognition degree.

Next, the handwritten character determination unit 343 compares the printed character recognition degree calculated by the printed character recognition processing unit 341 with the handwritten character recognition degree calculated by the handwritten character recognition processing unit 342 (S24). Here, the printed character recognition degree is OCRS, and the handwritten character recognition degree is ICRS.

When the printed character recognition degree (OCRS) is larger than the handwritten character recognition degree (ICRS) (YES in S24), the handwritten character determination unit 343 determines that the character included in the character area is a printed character (S25). ).

On the other hand, when the printed character recognition degree (OCRS) is equal to or lower than the handwritten character recognition degree (ICRS) (NO in S24), the handwritten character determination unit 343 determines that the character included in the character area is a handwritten character. (S26).

Next, the character encoding processing unit 35 performs character encoding processing on each of the characters determined to be printed characters or handwritten characters in steps S25 and S26 (S27). The processing of step S27 and the processing of steps S22 and S23 can be performed in no particular order. After the process of step S27 is completed, the process proceeds to step S7 of FIG.

The printed character recognition processing unit 341 and the handwritten character recognition processing unit 342 may have the functions of the character coding processing unit 35. That is, when the printed character recognition processing unit 341 and the handwritten character recognition processing unit 342 perform their respective character recognition, the character encoding process may be performed. Here, an example of the character type discrimination result is shown in FIG.

FIG. 8 shows the result of performing the character type discrimination processing on the character area of the input image D. The character area including handwritten characters is filled with dots.

In FIG. 8, it can be seen that the character area 43 of FIG. 6 is filled with dots with dots, and the character in the character area 43 is determined to be a handwritten character (handwritten character portion 43A). Based on this determination result, conversion to structured data is performed.

Structured data is data described in a format such as OOXML, ODF, or PDF (registered trademark).

OOXML (Office Open XML, OpenXML) is an XML-based file format for office suites. Microsoft Word (registered trademark), Microsoft Excel (registered trademark), and Power Point are OOXML compliant.
ODF (Open Document Format) is an XML-based open file format for storing document files handled by office software such as word processing documents and spreadsheets of spreadsheet software.
PDF (Portable Document Format) is a format for electronic documents developed by Adobe Systems. In the PDF format file, not only character information but also information such as font, character size, character decoration, embedded image, and their layout can be stored.

Returning to the description of the flowchart of FIG. The handwritten character determination unit 343 of the character type determination unit 34 determines whether or not the character area of the input image D is a handwritten character (S7). When it is not a handwritten character (NO in S7), the main body attribute conversion unit 361 of the structured data conversion unit 36 converts the character string determined to be the printed character into the main body attribute of the structured data (S8). After the process of step S8 is completed, the process proceeds to step S11.

On the other hand, when the character area of the input image D is a handwritten character (YES in S7), the supplementary attribute conversion unit 362 of the structured data conversion unit 36 sets the character string determined to be the handwritten character as the supplementary attribute of the structured data. Convert (S9).

The following describes the case of converting to the PowerPoint note attribute of Microsoft Office as an example of the supplementary attribute of structured data. FIG. 9 shows a display example in which a PowerPoint file is opened by an application.

(Application display example)
FIG. 9 is a display example when the PowerPoint file according to the first embodiment is opened by an application.

The application screen 50 shown in FIG. 9 is provided with a plurality of menus such as a home menu 51 and a review menu 52. A main body area 53 corresponding to the main body attribute is provided below the plurality of menus, and a note area 54 corresponding to the supplementary attribute is provided below the main body area 53. The main body area 53 occupies a large area in the center of the application screen 50. This file format is an XML format standardized by ISO29500.

(File format structure example)
FIG. 10 shows a structural example of a file format of a general PowerPoint file including note attributes.

The object of the main body area 53 shown in FIG. 9 is described in'slide1.xml'in the'slides' folder 61 (main body attribute directory), and the character string of the note area 54 is in the'notesSlides' folder 62 (note attribute directory). It is described in'notesSlide1.xml'.

Returning to the description of the flowchart of FIG. Next, the handwritten character replacement unit 37 replaces the handwritten character portion of the main body area in which the handwritten character portion of the structured data exists with information (structured data) indicating that the handwritten character exists (S10). The structured data conversion unit 36 records these structured data in the structured data recording unit 38.

Next, the structured data output unit 39 outputs the structured data recorded in the structured data recording unit 38 in response to the user's instruction. For example, the structured data output unit 39 displays the structured data on the display unit 25 by the application (S11). When the process of step S11 is completed, the process of this flowchart is completed.

[Example of information indicating that handwritten characters existed]
An example of information indicating that a handwritten character exists, which is created in the handwritten character portion of the main body area 53 where the handwritten character exists, will be described.

[First example]
FIG. 11 shows a display example of an application screen in which handwritten characters are arranged in a note area of a PowerPoint file according to the first example of the first embodiment.

In the application screen 50A of FIG. 11, the handwritten character string 541 written as "too late!" Is arranged in the note area 54. Further, information that can be compared with the handwritten character string 541 arranged in the note area 54 in the portion (handwritten character portion 531) where the handwritten character string of the document image 530A of the main body area 53 exists (in the example of FIG. 11). Place the ★ symbol). In FIG. 11, the same information (★) indicating that the handwritten characters arranged in the document image 530A of the main body area 53 exist is also arranged in the note area 54. As a result, the user can easily recognize the correspondence between the character string in the note area 54 and the handwritten character unit 531 in the main body area 53.

In addition, since the user may overlook the existence of the handwritten character string only with the symbol, a character string indicating that the handwritten character string existed may be added and arranged. For example, in FIG. 11, a character string of “with handwriting” is arranged in the handwritten character unit 531. In this way, the information indicating the relevance to the information indicating the existence of the handwritten characters may be arranged in at least one of the main body area 53 and the note area 54 (supplementary area).

[Second example]
In the second example, when a plurality of handwritten characters exist in the paper document, the relationship between the information indicating that the plurality of handwritten characters existed and the information indicating that the handwritten characters existed. Information indicating the correspondence is added to each of the information indicating the sex. For example, when there are many handwritten character strings added to a paper document, it may be'symbol + serial number'.

FIG. 12 shows a display example of an application screen in which handwritten characters are arranged in a note area of a PowerPoint file according to a second example of the first embodiment.

In the application screen 50B of FIG. 12, two handwritten character strings 541a and 541b exist in the note area 54, and handwritten character portions 531a and 531b also exist in two places in the document image 530B of the main body area 53. Then, the symbols ‘★ 1’ and ‘★ 2’ are arranged in front of the handwritten character portions 531a and 531b of the main body area 53 corresponding to the handwritten character strings 541a and 541b of the note area 54.

Further, the character string added to the main body area 53 is different in character color, character size, character thickness, italic type, etc. so as to be distinguished from other information (printed character string, etc.) originally existing in the main body area 53. It may be added as a decoration. In the example of FIG. 12, the character string is displayed in italics.

[Effect of the first embodiment]
According to the present invention having the above-described configuration, in the process of converting a paper document in which printed characters and handwritten characters are mixed into an electronic image and vectorizing the electronic image into a clean copy file format, the printed characters and the handwritten characters are converted. Can be easily identified.

<2. Second embodiment>
As an example of another supplementary attribute of the structured data according to the first embodiment, a case where handwritten characters are converted into a comment attribute of PowerPoint of Microsoft Office will be described.

[First example]
FIG. 13 shows a display example of an application screen in which handwritten characters are arranged in a comment area of a PowerPoint file according to the first example of the second embodiment.

In the application screen 70A of FIG. 12, the transparent object 731a is arranged in the portion (handwritten character portion 531) where the handwritten character of the document image 730A of the main body area 73 exists. Further, as a comment related to the arranged transparent object 731a, the handwritten character string "too late!" Is added to the comment field 751 of the comment area 75. By pressing the comment display button 521 with the mouse pointer or the like, the comment area 75 is displayed.

The transparent object 731a may be an invisible character code (blank, tab) or a transparent graphic object.

Also, the user name (KUROKI) of the client terminal 1 is displayed on the left side of "Too late!" In the comment field 751, but even if you operate the operation unit 26 and enter the user name with a handwritten comment added later. Good. Alternatively, the user name may not be displayed in the comment field 751.

When the comment function is used, the application automatically displays a mark 732a (symbol, figure, etc.) indicating that a comment exists in the comment area 75 in the corresponding place in the main body area 73, and is combined with the handwritten character string in the comment field 751. It is possible to show a relationship. The corresponding place of the main body area 73 is a position corresponding to the handwritten character portion 531.

In FIG. 13, the application indicates that the transparent object 731a and the mark 732a are arranged in the document image 730A of the main body area 73, and there is a comment there. Further, in FIG. 13, a handwritten character string is added to the comment field 751 of the comment area 75 as a comment related to the transparent object 731a and the mark 732a.

(File format structure example)
FIG. 14 shows a structural example of a file format of a general PowerPoint file including a comment attribute.

The transparent object 731a in the main body area 73 shown in FIG. 13 is described in'slide1.xml'in the'slides' folder 61 (main body attribute directory), and the comments in the comment area 75 are described in the'comments' folder 63 (comment attribute directory). It is described in'comment1.xml'in.

[Second example]
The transparent object arranged in the main body area 73 of FIG. 13 may be a visible character string indicating that there was a handwritten character string.

FIG. 15 shows a display example of an application screen in which handwritten characters are arranged in a comment area of a PowerPoint file according to a second example of the second embodiment.

In the application screen 70B of FIG. 15, an object 731b composed of a visible character string'handwritten'is arranged next to the mark 732a in the handwritten character portion 531 of the document image 730B in the main body area 73.

By arranging the object 731b composed of the visible character string in the main body area 73 in this way, the possibility that the user overlooks the existence of the handwritten character string is reduced.

[Third example]
FIG. 16 shows a display example of an application screen in which handwritten characters are arranged in a comment area of a PowerPoint file according to a third example of the second embodiment.

In the application screen 70C of FIG. 16, the mark 732c is arranged in the handwritten character portion 531 of the document image 730C in the main body area 73, at the upper right of the object 731b composed of the visible character string'handwritten available'.

[4th example]
FIG. 17 shows a display example of an application screen in which handwritten characters are arranged in a comment area of a PowerPoint file according to a fourth example of the second embodiment.

In the application screen 70D of FIG. 17, only the mark 732a is displayed on the handwritten character portion 531 of the document image 730D in the main body area 73, and no object is arranged or the transparent object 731a (FIG. 13) is arranged. Then, in the comment field 751 of the comment area 75, the handwritten comment "too late!" Is displayed together with the visible character string 751d "handwritten" indicating the existence of the handwritten character string.

As a result, even if only the mark 732a is displayed in the main body area 73 and the visible character string indicating the existence of the handwritten character string is not displayed, the user sees the visible character string 751d'handwritten yes'arranged in the comment area 75. This makes it possible to recognize the existence of handwritten characters.

[Example 5]
FIG. 18 shows a display example of the application screen in which the content of the handwritten character information displayed in the comment area of the PowerPoint file is deleted according to the fifth example of the second embodiment.

In the application screen 70E of FIG. 18, the same document image 730D as that of FIG. 17 is displayed in the main body area 73, but the comment in the comment field 751 of the comment area 75 is deleted.

As a function of an XML-based document creation application such as Microsoft Office, a batch deletion function for deleting character strings in a supplementary area (note area or comment area) at once is provided. When the handwritten character string is placed in the supplementary area, by using this batch deletion function, it is possible to easily respond to the request to distribute the document in a form that does not include the handwritten character string. This example is also applicable to the first embodiment.

<3. Third Embodiment>
The PDF and Microsoft Word functions also have a similar comment function (annotation function), and by converting to each format using this function, it is possible to have the same effect as in the first and second embodiments. It is possible.

FIG. 19 shows a display example of the application screen in which the handwritten characters are arranged in the annotation area of the PDF file according to the third embodiment.

In the application screen 70E of FIG. 19, an annotation icon 81 composed of figures, symbols, etc. is arranged in the handwritten character portion 531 of the document image 80 corresponding to the main body area, and a pop-up window 82 associated with the annotation icon 81 is displayed. To. The pop-up window 82 is a supplementary area corresponding to the supplementary attribute. In the pop-up window 82, the handwritten string'Too late! ’Is placed.

In the case of PDF, it can be realized by registering a handwritten character string (annotation data) in the Annots array of the page object for each page.

According to the third embodiment described above, even when the PDF file is used, the same effects as those of the first embodiment and the second embodiment are obtained.

Even in the case of PDF, an object composed of a visible character string (for example,'with handwritten characters') may be placed near the annotation icon 81.

<4. Fourth Embodiment>
An application (such as a client terminal 1 or an image forming apparatus 3) that implements this structured data conversion function is provided with a UI that can select an output form. And the user can also select options such as "Place handwritten character string in note area", "Place in comment area", "Place in body area", "Delete completely and do not place anywhere". is there.

FIG. 20 shows the functional configuration of the computer 20A included in the image processing apparatus according to the fourth embodiment. Here, an example in which the image processing device is applied to the client terminal 1 will be described.

The computer 20A of FIG. 20 is different from the computer 20 of FIG. 3 in that it includes an output form setting unit 90.

The output form setting unit 90 displays a format selection screen (not shown) that prompts selection of which format (data format) the structured data is converted from the electronic data obtained by the electronic imaging means such as the scanner 4. Display on 25. Next, the output form setting unit 90 displays an output form selection screen (not shown) for selecting the above output form when the selected format has a plurality of output forms (handwritten character string arrangement mode). To do. Then, the output form setting unit 90 outputs the handwritten character string to the application screen in the output form based on the user's selection.

When "Completely delete and do not place anywhere" is selected, the output form setting unit 90 includes information other than handwritten characters (for example, an arrow determined as a graphic area 42) during vectorizing processing. Does not reflect the added information in the structured data after conversion. By doing so, the document can be shared and distributed with other users in a form that does not include handwritten information.

<5. Others>
Even if the electronic imaging means such as the scanner 4 and the camera 5 is equipped with a computer as shown in FIG. 2, the image processing device can be applied.

Furthermore, the present invention is not limited to the above-described embodiments, and it goes without saying that various other application examples and modifications can be taken as long as the gist of the present invention described in the claims is not deviated. is there.

For example, the above-described embodiment describes the configurations of the apparatus and the system in detail and concretely in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those including all the described configurations. .. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

Further, in the present specification, the processing steps for describing the time-series processing are not necessarily the processing performed in the time-series according to the described order, but are parallel or individual. It also includes processing executed in (for example, parallel processing or processing by an object).

1 ... Client terminal, 4 ... Scanner, 5 ... Camera, 10 ... System, 21 ... CPU, 27 ... Non-volatile storage, 31 ... Input image receiver, 33 ... Area discriminator, 34 ... Character type discriminator, 35 ... Character Coding processing unit, 36 ... Structured data conversion unit, 37 ... Handwritten character replacement unit, 39 ... Structured data output unit, 40 ... Printed character area, 41 ... Handwritten area, 42 ... Graphic area, 43 ... Character area, 43A ... Handwritten character part, 53 ... Main body area, 54 ... Note area, 61 ... Main body attribute folder, 62 ... Note attribute folder, 63 ... Comment attribute folder, 73 ... Main body area, 74 ... Correction area, 75 ... Comment area, 361 ... Body attribute conversion unit, 362 ... Correction attribute conversion unit, 521 ... Comment display button, 531, 531a, 531b ... Handwritten character part, 541, 541a, 541b ... Handwritten character string, 751 ... Comment field, P ... Paper document, D ... image data

Claims (10)

An area determination unit that analyzes the image data obtained by reading a handwritten paper document and discriminates between the character area and the non-character area included in the image data.
A character type determination unit that analyzes the character area determined by the area determination unit and determines whether the character existing in the character area is a printed character or a handwritten character.
A character encoding processing unit that converts the printed characters and the handwritten characters into character codes,
A structured data conversion unit that converts an area other than the character determined by the area determination unit and a character area including the character-encoded printed character into the main body attribute of the structured data.
A supplementary attribute conversion unit that converts the character-encoded handwritten characters into supplementary attributes of the structured data, and
It is provided with a handwritten character replacement unit that replaces the handwritten character unit of the main body area corresponding to the main body attribute in which the handwritten character determined by the character type determination unit exists with information indicating that the handwritten character exists .
An image processing device that arranges the same information as the information indicating that the handwritten character arranged in the main body area exists in the supplementary area corresponding to the supplementary attribute . An area determination unit that analyzes the image data obtained by reading a handwritten paper document and discriminates between the character area and the non-character area included in the image data.
A character type determination unit that analyzes the character area determined by the area determination unit and determines whether the character existing in the character area is a printed character or a handwritten character.
A character encoding processing unit that converts the printed characters and the handwritten characters into character codes,
A structured data conversion unit that converts an area other than the character determined by the area determination unit and a character area including the character-encoded printed character into the main body attribute of the structured data.
A supplementary attribute conversion unit that converts the character-encoded handwritten characters into supplementary attributes of the structured data, and
It is provided with a handwritten character replacement unit that replaces the handwritten character unit of the main body area corresponding to the main body attribute in which the handwritten character determined by the character type determination unit exists with information indicating that the handwritten character exists .
When information indicating the relevance to the information indicating the existence of the handwritten character is arranged in at least one of the main body area and the supplementary area corresponding to the supplementary attribute, and a plurality of the handwritten characters are present. Is added with information indicating a correspondence relationship to each of the information indicating the existence of the plurality of handwritten characters and the information indicating the relevance of the information indicating the existence of the handwritten characters. Image processing device. The image processing apparatus according to claim 1, wherein the information indicating the presence of the handwritten character in the main body area is decorated so as to be distinguishable from other information existing in the main body area. The information indicating the existence of the handwritten character in the main body area and the information indicating the relevance to the information indicating the existence of the handwritten character can be distinguished from other information existing in the main body area. Decorated
The image processing apparatus according to claim 2. The image processing according to claim 1 or 2 , wherein the supplementary area corresponding to the supplementary attribute is an area defined by the comment attribute of the structured data or an area defined by the note attribute used in the document creation software. apparatus. The image processing apparatus according to claim 5 , wherein the structured data used in the document creation software is data described in an OOXML or ODF format. The image processing apparatus according to claim 1 or 2 , wherein when the structured data is data described in PDF (registered trademark), the annotation function defined by the Annots array is used for the supplementary attribute. The image processing apparatus according to any one of claims 1 to 7 , wherein the information indicating the existence of the handwritten character is at least one of a symbol, a character string, a figure, and an image. A procedure for analyzing image data obtained by reading a handwritten paper document and determining a character area and a non-character area included in the image data, and
A procedure for analyzing the determined character area and determining whether the character existing in the character area is a printed character or a handwritten character.
The procedure for character-coding the printed characters and the handwritten characters, and
A procedure for converting the identified area other than the character and the character area including the character-encoded printed character into the main body attribute of the structured data, and
The procedure for converting the character-encoded handwritten character into the supplementary attribute of the structured data, and
A procedure for replacing the handwritten character portion of the main body area corresponding to the main body attribute in which the handwritten character exists with information indicating that the handwritten character exists, and
A procedure for arranging the same information as the information indicating that the handwritten character arranged in the main body area exists in the supplementary area corresponding to the supplementary attribute, and
A program that lets your computer run. A procedure for analyzing image data obtained by reading a handwritten paper document and determining a character area and a non-character area included in the image data, and
A procedure for analyzing the determined character area and determining whether the character existing in the character area is a printed character or a handwritten character.
The procedure for character-coding the printed characters and the handwritten characters, and
A procedure for converting the identified area other than the character and the character area including the character-encoded printed character into the main body attribute of the structured data, and
The procedure for converting the character-encoded handwritten character into the supplementary attribute of the structured data, and
A procedure for replacing the handwritten character portion of the main body area corresponding to the main body attribute in which the handwritten character exists with information indicating that the handwritten character exists, and
When information indicating the relevance to the information indicating the existence of the handwritten character is arranged in at least one of the main body area and the supplementary area corresponding to the supplementary attribute, and a plurality of the handwritten characters are present. Is added information indicating a correspondence relationship to each of the information indicating the existence of the plurality of handwritten characters and the information indicating the relationship between the information indicating the existence of the handwritten characters. Procedure and
A program that lets your computer run.

Images