JP7019963B2 - Character string area / character rectangle extraction device, character string area / character rectangle extraction method, and program - Google Patents

Description

The present invention relates to a character string area / character rectangle extraction device, a character string area / character rectangle extraction method, and a program.

In Optical Character Recognition (OCR), character string area extraction that extracts a character string area that indicates the existence range of characters in a document and characters that indicate the range of each character included in the character string area are extracted. Character rectangle extraction is performed to extract the rectangle. In optical character recognition technology, a document read by a scanner or the like is converted into image data, and character string area extraction and character rectangle extraction are performed by analyzing the image data, and character recognition is performed for each of the extracted character rectangles. Is generally performed.
As a method for automatically extracting a character string area and a character rectangle by a computer or the like, there are various methods such as a method using projection (for example, Patent Document 1) and a method by integrating black pixels (for example, Patent Document 2). Method has been proposed. Further, a method (for example, Patent Document 3) in which a character string area is manually extracted based on a user's operation and a character rectangle is automatically extracted has also been proposed.

Japanese Unexamined Patent Publication No. 5-266250 Japanese Unexamined Patent Publication No. 5-274472 Japanese Unexamined Patent Publication No. 2009-70242 Japanese Unexamined Patent Publication No. 2001-43313 Japanese Unexamined Patent Publication No. 2007-102702 Japanese Unexamined Patent Publication No. 2010-225112 Japanese Patent No. 5699570

Y. LeCun et al. , "Gradient-Based Learning Applied to Document Recognition", November 1998, Proceedings. of the IEEE, volume 86, issu 11, pp2278-2324.

However, in the prior art in which the character string area is manually extracted based on the user's operation, for example, when a character string area in a range spanning several lines is specified, erroneous recognition in line extraction occurs. Sometimes. For example, when a character string area is specified for each range of one line, when a line break is made in the middle of the character string (word), the character string at the end of the character string area is divided in the middle. Therefore, there is a problem that a process for combining the divided character strings is required.

The present invention has been made in view of the above points, and the present invention is a character string area capable of improving the accuracy of extracting a character string area and extracting a character rectangle by using auxiliary information based on a user's operation. -Provides a character rectangle extraction device, a character string area / character rectangle extraction method, and a program.

(1) The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is an image data acquisition unit for acquiring image data indicating an image including a character string, and the acquired image data acquisition unit. A display unit that displays an image based on image data, an operation input unit that accepts operation input from the user, and a start point of a line consisting of the character string included in the image specified based on auxiliary information based on the operation input. A character string area extraction unit that extracts a character string area that encloses the entire character string contained in the line based on the above and the end point, the extracted character string area, and other character strings adjacent to the character string area. It is a character string area / character rectangular extraction device characterized by including a character string area connecting portion for connecting a region and a character string region.

(2) As one aspect of the present invention, the character string area extraction unit extracts the character string area based on the range where the connection line connecting the start point and the end point and the character intersect. The character string area / character rectangle extraction device according to (1).
( 3 ) In one aspect of the present invention, the operation input unit accepts the operation input based on the operation of the pointer, and the character string area joining unit receives the extracted character string area based on the operation of the pointer. The character string area / character rectangular extraction device according to (1) or (2) , wherein the character string area and another character string area adjacent to the character string area are combined.

( 4 ) One aspect of the present invention is characterized in that the character string area extraction unit modifies the extracted character string area based on auxiliary information based on the operation input (1) to (1). 3 ) The character string area / character rectangle extraction device described in any one of the above.

( 5 ) In one aspect of the present invention, the operation input unit accepts the operation input based on the operation of the pointer, and the character string area extraction unit extracts the character string area based on the operation of the pointer. The character string area / character rectangle extraction device according to ( 4 ), which is characterized by modifying.

( 6 ) In one aspect of the present invention, the character string area joining portion includes the character string area extracted based on statistical information in the appearance of the character string, and another character string adjacent to the character string area. The character string area / character rectangle extraction device according to (1), characterized in that the area and the area are combined.

( 7 ) One aspect of the present invention is characterized in that a ruby mapping unit for associating the character string with the ruby corresponding to the character string is provided based on the auxiliary information based on the operation input. The character string area / character rectangle extraction device according to any one of (1) to ( 6 ).

( 8 ) One aspect of the present invention is characterized in that the display unit displays a ruby correspondence area which is a range in which the associated character string and the ruby corresponding to the character string are enclosed. This is the character string area / character rectangle extraction device described in ( 7 ).

( 9 ) One aspect of the present invention is any of (1) to ( 8 ), characterized in that a character rectangle extraction unit for extracting a character rectangle representing a character rectangle of each character constituting the character string is provided. It is a character string area / character rectangle extraction device described in one of them.

( 10 ) As one aspect of the present invention, the character rectangular extraction unit is based on an evaluation value calculated from character shape information, recognition accuracy in character recognition, etc. from among a plurality of character cutout position candidates. The character string area / character rectangle extraction device according to ( 9 ), wherein the character rectangle is extracted by specifying an appropriate character cutout position.

( 11 ) One aspect of the present invention is the character described in ( 9 ), wherein the character rectangle extraction unit corrects the extracted character rectangle based on auxiliary information based on the operation input. It is a column area / character rectangle extractor.

( 12 ) In one aspect of the present invention, the operation input unit accepts the operation input based on the operation of the pointer, and the character rectangle extraction unit modifies the extracted character rectangle based on the operation of the pointer. The character string area / character rectangle extraction device according to ( 11 ).

( 13 ) In one aspect of the present invention, the display unit assigns a line number, which is a number assigned in the order in which a character string area indicating a display target range of a character string included in the line is extracted, to the character string. The character string area / character rectangular extraction device according to any one of (1) to ( 12 ), which is characterized by displaying in association with an area.

( 14 ) One aspect of the present invention is characterized in that the display unit displays the image based on the image data and a list display image which is an image in which the line numbers are listed ( 13 ). It is a character string area / character rectangle extraction device described in 1.

( 15 ) In one aspect of the present invention, the character string area extraction unit is based on the start point and the end point specified by the auxiliary information based on the operation input for the line consisting of the character string included in the image. Generates specific information that identifies the position of the start point and end point of another line consisting of the character string contained in the other line, and based on the start point and end point specified by the specific information, the character string contained in the other line. The character string area / character rectangular extraction device according to any one of (1) to ( 14 ), which is characterized by extracting the character string area.

(16) In one aspect of the present invention, the character string region extraction unit attaches to the image based on auxiliary information based on the operation input for at least one of the start point and the end point specified for the image. The character string area / character rectangular extraction device according to ( 15 ), which modifies the specific information generated in response to the above.

( 17 ) In one aspect of the present invention, the character string area extraction unit is extracted for the image based on auxiliary information based on the operation input for the character string area extracted for the image. The character string area / character rectangular extraction device according to ( 15 ) or ( 16 ), wherein the character string area is modified.

( 18 ) As one aspect of the present invention, the character string region extraction unit has auxiliary information based on the operation input performed on the first image and the specific information generated on the first image. The character string area according to any one of ( 15 ) to ( 17 ), characterized in that the specific information for a second image different from the first image is generated based on at least one of the two. It is a character rectangle extractor.

( 19 ) In one aspect of the present invention, the character string area extraction unit uses the character string for a second image different from the first image based on the information about the character string area extracted from the first image. The character string area / character rectangular extraction device according to any one of ( 15 ) to ( 18 ), which is characterized by extracting an area.

(20) One aspect of the present invention is a method for extracting a character string area / character rectangle by a computer, in which an image data acquisition unit acquires image data indicating an image including a character string, and displays an image data acquisition step. A display step for displaying an image based on the acquired image data, an operation input step for the operation input unit to receive an operation input from a user, and an auxiliary information based on the operation input for a character string area extraction unit. A character string area extraction step for extracting a character string area that encloses the entire character string included in the line based on the start point and end point of the line consisting of the character string included in the image specified based on the character. The character string area joining portion is characterized by having a character string area joining step for joining the extracted character string area and another character string area adjacent to the character string area. This is a character string extraction method.

( 21 ) As one aspect of the present invention, an image data acquisition step for acquiring image data indicating an image including a character string, a display step for displaying an image based on the acquired image data, and a user The entire character string included in the line based on the operation input step that accepts the operation input of and the start point and end point of the line consisting of the character string included in the image specified based on the auxiliary information based on the operation input. A character string area extraction step for extracting a character string area surrounding a square , and a character string area joining step for combining the extracted character string area with another character string area adjacent to the character string area. It is a program to be executed.

According to the present invention, it is possible to improve the accuracy of extracting the character string region by using the auxiliary information based on the user's operation.

It is a block diagram which shows the functional structure of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area extraction processing by the character string area extraction part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area extraction processing by the character string area extraction part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character rectangle extraction processing by the character rectangle extraction part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the ruby extraction processing by the ruby correspondence part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area management screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area management screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area management screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the character string area management screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a flowchart which shows the operation of the character string area / character rectangle extraction apparatus which concerns on embodiment of this invention. It is a block diagram which shows the functional structure of the character string area / character rectangle extraction apparatus which concerns on modification 1 of embodiment of this invention. It is a flowchart which shows the operation of the character string area / character rectangle extraction apparatus which concerns on modification 1 of embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on modification 2 of the Embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on modification 2 of the Embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on modification 2 of the Embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on modification 2 of the Embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on modification 2 of the Embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on modification 2 of the Embodiment of this invention. It is a figure which shows an example of the character string area extraction screen displayed by the display part of the character string area / character rectangle extraction apparatus which concerns on modification 2 of the Embodiment of this invention.

<Embodiment>
Hereinafter, embodiments of the present invention will be described.
In the apparatus according to the present embodiment, image data indicating an image including a character string (for example, image data generated by reading a document with a scanner or the like) is input, and the character string is input in an image based on the input image data. Extract (specify) the displayed range (hereinafter, also referred to as the character string area). Further, the apparatus according to the present embodiment extracts (specifies) a range (hereinafter, also referred to as a character rectangle) in which each character in the character string included in the extracted character string area is displayed. That is, the device according to the present embodiment is a character string area / character rectangle extraction device that extracts a character string area and a character rectangle.

Then, the character string area / character rectangle extraction device according to the present embodiment performs optical character recognition (OCR) on the image indicating the character included in each of the extracted character rectangles, and recognizes the character. In this embodiment, a well-known technique is used for optical character recognition. For example, as one of the typical character recognition methods, there is a method using a convolutional neural network described in Non-Patent Document 1.

In the prior art in which the character string area extraction and the character rectangle extraction are automatically performed, for example, when the layout of the character string area in the document is complicated, it is erroneously recognized as a layout that is significantly different from the actual layout. Sometimes. For example, misrecognition that the order of the character string area is different from the actual sentence, misrecognition in line extraction such as two lines being recognized as one line, and misrecognition that a picture or figure is recognized as a character occur. do. Further, in such a case, erroneous recognition may occur due to the failure of character rectangle extraction. For example, misrecognition that a character is overlooked, misrecognition that two characters are recognized as one character, misrecognition that one character is recognized as two characters, and ruby that is attached to a character cannot be recognized and includes ruby. Misrecognition that recognizes as one character occurs.

Further, in the conventional technique in which the character string area is manually extracted, for example, when a character string area in a range spanning several lines is specified, an erroneous recognition in the line extraction and an erroneous due to a failure in character rectangle extraction are performed. Recognition etc. may occur. Further, for example, even when a character string area is specified for each range of one line, erroneous recognition may occur due to a failure of character rectangle extraction. Furthermore, in such a case, when a line break is made in the middle of the character string (word), the character string at the end of the character string area is in a divided state in the middle, so the divided character strings are combined. Processing is required.

<Structure of character string area / character rectangle extractor>
Hereinafter, the configuration of the character string area / character rectangle extraction device 1 according to the embodiment will be described with reference to the drawings.
FIG. 1 is a block diagram showing a functional configuration of a character string area / character rectangle extraction device 1 according to an embodiment of the present invention.
As shown in the figure, the character string area / character rectangular extraction device 1 according to the present embodiment includes a control unit 10, an image data acquisition unit 11, an operation input unit 12, a display unit 13, a storage unit 14, and characters. It is composed of a column area extraction unit 15, a character string area combination unit 16, a character rectangle extraction unit 17, a ruby mapping unit 18, a management screen generation unit 19, and a character recognition unit 20.

The control unit 10 controls various processes in the character string area / character rectangle extraction device 1. The control unit 10 includes, for example, a CPU (Central Processing Unit).

The image data acquisition unit 11 acquires image data indicating an image including a character string. The image data acquisition unit 11 includes, for example, an input interface for acquiring image data generated by reading a document by an external device such as a scanner. Alternatively, the image data acquisition unit 11 includes, for example, a scanner.

The operation input unit 12 receives an operation input from the user. The operation input unit 12 generates an operation input signal (auxiliary information) based on the received operation input, and the display unit 13, the character string area extraction unit 15, the character string area connection unit 16, the character rectangle extraction unit 17, and the ruby, which will be described later. It is output to the correspondence unit 18 or the management screen generation unit 19. The operation input unit 12 includes a pointing device used for the user to operate a pointer (or a cursor) displayed on the display unit 13 described later, for example, a mouse, a touch pad, a touch panel, a stylus, a trackball, or the like. Consists of.

In the present embodiment, the character string area / character rectangle extraction device 1 is provided with the operation input unit 12, but the present invention is not limited to this. For example, the operation input unit 12 may be provided in an external device, and the character string area / character rectangle extraction device 1 may acquire a signal indicating a pointer operation by the user from the external device.

The display unit 13 displays an image based on the image data acquired by the image data acquisition unit 11. The display unit 13 includes a display, for example, a liquid crystal display (LCD), an organic EL (Organic Electroluminescence) display, a CRT (Cathode Ray Tube), or the like.

In the present embodiment, the character string area / character rectangle extraction device 1 is provided with the display unit 13, but the present invention is not limited to this. For example, the character string area / character rectangle extraction device 1 may transmit image data indicating an image to be displayed to an external device so that the image is displayed on a display unit included in the external device.

The storage unit 14 stores various computer programs, data, and the like used in the character string area / character rectangle extraction device 1. Further, the storage unit 14 also has a function as a temporary storage area used in various arithmetic processes and the like in the character string area / character rectangle extraction device 1. The storage unit 14 is a storage medium, for example, an HDD (Hard Disk Drive), a flash memory, or an EEPROM (Electrically).
Includes Erasable Program Read Only Memory, RAM (Random Access read / write Memory), ROM (Read Only Memory), or any combination thereof.

The character string area extraction unit 15 is the start point of a line consisting of a character string included in the image based on the image data acquired by the image data acquisition unit 11 based on the operation input signal (auxiliary information) input from the operation input unit 12. And the end point are specified, and the character string area indicating the display target range of the character string included in the line is extracted based on the start point and the end point of the specified line. Further, the character string area extraction unit 15 corrects the extracted character string area based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user.

The character string area joining unit 16 is extracted before the character string area extracted by the character string area extraction unit 15 and another character string area adjacent to the character string area (for example, the character string area is extracted). And other adjacent string areas) and. The character string area combining unit 16 causes the display unit 13 to display an image showing the combined character string area.

The character string area combining unit 16 is adjacent to the extracted character string area and the character string area based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user. Combine with other string areas.

The character rectangle extraction unit 17 extracts a character rectangle representing a rectangle of each character constituting the character string included in the character string area extracted by the character string area extraction unit 15. Further, the character rectangle extraction unit 17 corrects the extracted character rectangle based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user.

For example, as in the image processing device described in Patent Document 7, the character rectangle extraction unit 17 is calculated from a plurality of character cutout position candidates, such as character shape information and recognition accuracy in character recognition. The character rectangle may be extracted by specifying an appropriate character cutout position based on the evaluation value.

For example, when a character is cut out from the character string "Chemistry", a pattern in which "ka" and "gaku" are cut out, a pattern in which "i" and "higaku" are cut out, and "i" There are multiple candidate character cutout positions so that there is a pattern in which "hi" and "gaku" are cut out. From among the candidates for the character cutout position, an appropriate character cutout position is specified based on the evaluation value calculated from the character shape information accumulated in advance and the recognition accuracy in character recognition.

The ruby mapping unit 18 is a character string included in the character string area extracted by the character string area extraction unit 15 based on an operation input signal (auxiliary information) based on a pointer operation by the user, and a character string area extraction unit. The ruby extracted by 15 and corresponding to the character string is associated with the ruby. The ruby correspondence unit 18 causes the display unit 13 to display an image showing a ruby correspondence area which is a range in which the associated character string and the ruby corresponding to the character string are enclosed.

The management screen generation unit 19 generates a management screen for the user to modify the character string area and the character rectangle. The management screen generation unit 19 associates a line number, which is a number assigned in the order of extraction, with the character string area (character string area extracted in line units) extracted by the character string area extraction unit 15. Then, the management screen generation unit 19 includes an image in which a line number is assigned to each of the character string areas (character string areas extracted in line units) included in the image based on the image data acquired by the image data acquisition unit 11. , A management screen for displaying a list display image, which is an image in which the line number is displayed in a list, is displayed on the display unit 13.

The character recognition unit 20 performs optical character recognition (OCR) on an image showing characters included in each character rectangle extracted by the character rectangle extraction unit 17, and recognizes the characters. Specifically, the character recognition unit 20 analyzes image data indicating characters contained in a document (for example, read by a scanner) and converts it into a data format (for example, a character code string) that can be edited by a computer. do.

(Extraction of character string area)
Hereinafter, the extraction of the character string area by the character string area / character rectangle extraction device 1 will be described with reference to the drawings.
2 to 4 are diagrams showing an example of a character string area extraction screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment of the present invention.

As shown in FIG. 2, the user operates the pointer pt1 by the operation input unit 12 to specify the range of the character string included in the character string area extraction screen ds1 in line units. To specify the character string area for each line, for example, the user first moves the pointer pt1 to a position near the start point of the line, clicks the mouse button, and then near the end point of the line. This is done by moving the pointer pt1 to a position and clicking the mouse button. Alternatively, to specify the character string area for each line, for example, the user first moves the pointer pt1 to a position near the start point of the line, clicks the mouse button, and then clicks the mouse button as it is near the end point of the line. This is done by dragging the pointer pt1 to the position.

As shown in FIG. 2, when the operation of specifying the start point of the line is performed, the position specified as the start point (that is, the position where the mouse button is clicked for the first time or the position where the drag is started) is performed. For example, a white round icon is displayed at a certain starting point st1.
Next, as shown in FIG. 3, when the operation of specifying the end point of the line is performed, the position specified as the end point (that is, the position where the mouse button is clicked for the second time, or the drag is finished, for example). At the end point ed1 which is the position), for example, a black circle icon is displayed. Further, when the operation of designating the end point of the line is performed, the connection line cn1 connecting the start point st1 and the end point ed1 is displayed.

Then, as shown in FIG. 4, when the operation of designating the start point and the end point of the line is completed, a surrounding line is displayed so as to enclose the entire character string included in the line in which the start point and the end point are designated. The area surrounded by the enclosing line is the character string area extracted by the character string area extraction unit 15. FIG. 4 is a diagram showing a state in which three character string areas of the character string area sa1, the character string area sa2, and the character string area sa3 are extracted.

Hereinafter, an example of the character string area extraction process will be described with reference to the drawings.
FIG. 5 is a diagram showing an example of a character string area extraction process by the character string area extraction unit 15 of the character string area / character rectangle extraction device 1 according to the embodiment of the present invention.

As shown in FIG. 5A, when the connection line cn1 is set by designating the start point st1 and the end point ed1, the character string area extraction unit 15 is in a range where the connection line cn1 and the character intersect. To identify. That is, for example, in the example shown in FIG. 5A, the reading point and the connecting line cn1 do not intersect at the position of the reading point ",", but all the other characters are connected to the connecting line cn1. It intersects. The character string area extraction unit 15 measures the positions of the upper end portion and the lower end portion of the character at the positions where the character and the connection line cn1 intersect, respectively. Then, the character string area extraction unit 15 has an uppermost end portion up1 which is a portion located at the uppermost portion and a lowermost end portion lp1 which is a portion located at the lowermost portion in the upper end portion and the lower end portion of each measured character. And identify.

In the example shown in FIG. 5, as shown in FIG. 5 (B), the uppermost end portion up1 is the upper end portion of the character of "calligraphy". Further, in the example shown in FIG. 5, as shown in FIG. 5 (C), the lowermost end portion lp1 is the lower end portion of the character “Gaku”.

When the uppermost end portion up1 is specified, the character string area extraction unit 15 sets the uppermost end line ul1, which is a line extending horizontally from the uppermost end portion up1 to the left and right, as a line representing the upper end of the character string area sa2. do. Similarly, when the character string area extraction unit 15 specifies the lowermost end portion dl1, the lowermost end line dl1 which is a line extending horizontally from the lowermost end portion dl1 to the left and right represents the lower end of the character string area sa2. Set as a line.

Further, the character string area extraction unit 15 sets a line extending vertically from the start point st1 as a line representing the left end of the character string area sa2. Similarly, the character string area extraction unit 15 sets a line extending vertically from the end point ed1 as a line representing the right end of the character string area sa2.

As described above, since the range in the vertical direction and the horizontal direction of the character string area sa2 is determined, the range of the character string area sa2 is uniquely determined.
FIG. 5D shows the character string area sa2 extracted by the above-mentioned character string area extraction method.

The above method for extracting the character string area is just an example. In addition to this, for example, the character string area extraction unit 15 is located at a position separated from the uppermost end portion up1 by a predetermined length upward and at a position separated from the lowermost end portion dp1 by a predetermined length downward. Based on this, the uppermost end line ul1 and the lowermost end line dl1 may be set, respectively. That is, the character string area extraction unit 15 sets a range having a width slightly in the vertical direction from the uppermost end portion up1 and the lowermost end portion dp1 as the character string area sa2 (that is, widen by a predetermined length). The character string area sa2 may be set).

In addition to this, for example, the character string area extraction unit 15 is separated from the connection line cn1 by a predetermined length in the vertical direction without specifying the positions of the uppermost end portion up1 and the lowermost end portion dp1. The uppermost end line ul1 and the lowermost end line dl1 may be set for the line corresponding to the position, respectively.

The character string to be extracted by the character string area extraction unit 15 is not necessarily a character string in which characters are arranged in the horizontal direction (that is, horizontally written). For example, the character string to be extracted by the character string area extraction unit 15 is a character string in which characters are arranged in a vertical direction (that is, vertically written) or a character string in which characters are arranged in an oblique direction. In some cases. Therefore, it is preferable that the range of the character string area is set by an appropriate method suitable for the state in which the character string is described.

Further, as described above, the character string area extraction unit 15 modifies the extracted character string area based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user. can do.
FIG. 6 is a diagram showing an example of a character string area extraction process by the character string area extraction unit 15 of the character string area / character rectangle extraction device 1 according to the embodiment of the present invention.

In FIG. 6, the character string area extracted by the character string area extraction unit 15 is an area including a range described as “extracting a rectangle” and a range described as “kukei”. As shown in the figure, the character string area extracted by the character string area extraction unit 15 is an area including a range surrounded by a dotted line. That is, the character string area extraction unit 15 recognizes that it is a character string area of one line including the character string of "Kukei" representing the furigana of "rectangle", and extracts an erroneous character string area. ing.

According to the character string area / character rectangle extraction device 1 according to the present embodiment, the user can correct the character string area erroneously extracted by the character string area extraction unit 15.
The user moves the line at the upper end of the character string area sa110 extracted by mistake by operating the pointer pt1 by the operation input unit 12. Specifically, for example, the user operates the mouse to move the pointer pt1 to the position of the upper end line of the character string area sa100 shown by the dotted line in FIG. 6, and drags the upper end line to FIG. It is moved to the position of the line at the upper end of the character string area sa101 shown by the solid line. As a result, the character string area sa101 is correctly corrected so that only the character string of "extracting the rectangle" is included and the character string of the furigana "kukei" is not included.

(Combination of character string areas)
The above-mentioned extraction of the character string area was the extraction of the character string area on a line-by-line basis. However, in line units, the end of a line may be in the middle of a word, so there are cases where it is desired to collectively recognize a plurality of lines as one character string area.
Hereinafter, the combination of the character string area and the character string area by the character rectangle extraction device 1 will be described with reference to the drawings.
7 to 8 are diagrams showing an example of a character string area extraction screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment of the present invention.

As shown in FIG. 7, when the user performs an operation for setting the end point ed1, an operation indicating that the character string area to be set and another character string area adjacent to the character string area are combined is performed. conduct. In the example shown in FIG. 7, the operation indicating that the character string area to be set and another character string area adjacent to the character string area are combined is the operation of moving the pointer pt1 downward when setting the end point ed1. It is a drag operation. Or, for example, when the display unit 13 is a touch panel, the operation indicating that the character string area to be set and another character string area adjacent to the character string area are combined is an operation of flicking.
When these operations are performed, the character string area joining unit 16 sets the character string area and the character string area set immediately before the character string area (in the example shown in FIG. 7, " In optical character recognition (OCR), a process of combining a character string area including the character string "in a document") is performed.

When the operation of combining the character string areas is performed, the combined character string areas are extracted as shown in FIG. FIG. 8 is a diagram showing a state in which two character string areas, a character string area sa1 and a character string area sa4, are extracted on the character string area extraction screen ds1. Unlike the character string area extracted in the example shown in FIG. 4, in the example shown in FIG. 8, a character that is a character string area in which the character string area sa1 and the character string area sa2 shown in FIG. 4 are combined. The column area sa4 is extracted.

(Extraction of character rectangle)
Hereinafter, the extraction of the character rectangle by the character string area / character rectangle extraction device 1 will be described with reference to the drawings.
FIG. 9 is a diagram showing an example of a character rectangle extraction process by the character rectangle extraction unit 17 of the character string area / character rectangle extraction device 1 according to the embodiment of the present invention.

FIG. 9A shows a state in which the character rectangle extraction unit 17 has performed character rectangle extraction processing for each of the character string area sa5 and the character string area sa6 extracted by the character string area extraction unit 15.

As a method of extracting a character rectangle of each character from a character string including a plurality of characters, for example, a method using two threshold values described in Patent Document 4, a method using a histogram, and the like are known. .. In the extraction of the character rectangle in the present embodiment, an arbitrary character rectangle extraction method is used.

In the example shown in FIG. 9A, in the character string area sa5, character rectangles are extracted without error for all the characters included in the character string area sa5. Further, in the example shown in FIG. 9A, since the erroneous character rectangle is extracted at three places in the character string area sa6, the user performs an operation to correct the erroneous character rectangle. 9 (B), 9 (C), and 9 (D) illustrate the operation of the user to correct the character rectangle with respect to the above three places.

As shown in the upper part of FIG. 9B, the character rectangle extraction unit 17 collectively combines the character of the "area" and the biased part (that is, the hand bias) of the character of the "drawing" into one character. The rectangle rc01, which is an erroneous character rectangle, is extracted. Further, as shown in the upper part of FIG. 9B, the character rectangle extraction unit 17 recognizes only the part of making the character of "drawing" (that is, "Yu") as one character, and recognizes only the part of making the character "Yu", and is an erroneous character. The rectangle rc02, which is a rectangle, is extracted.

According to the character string area / character rectangle extraction device 1 according to the present embodiment, the user can correct the character rectangle erroneously extracted by the character rectangle extraction unit 17.
The user operates the pointer pt1 on the line (that is, the dividing line sl1) which is the right end of the erroneously extracted character rectangle rc01 and also the left end of the erroneously extracted character rectangle rc02 by the operation input unit 12. Move by. Specifically, for example, the user operates the mouse to move the pointer pt1 to the position of the dividing line sl1 shown in the upper part of FIG. 9B, and drags the dividing line sl1 to the position of the dividing line sl1 in FIG. 9B. Move to the position shown in the lower row.
As a result, the rectangle rc11 is modified so that only the characters of the "area" are included, and the rectangle rc12 correctly includes the characters of the "drawing".

As shown in the upper part of FIG. 9C, the character rectangle extraction unit 17 mistakenly divides the characters into two character rectangles and extracts them at the positions where a plurality of characters of ". (Middle black)" are lined up.
In this example, when multiple symbols such as "・ (middle black)" are lined up in succession, it is correct to extract them as one character rectangle. Suppose that

The user makes a correction to delete the dividing line sl2 which is the boundary line of the character rectangle extracted by mistakenly dividing it into two character rectangles. Specifically, for example, the user operates the mouse to move the pointer pt1 to the position of the dividing line sl2 shown in the upper part of FIG. 9C, and drags the dividing line sl1 to the position of the dividing line sl1 in FIG. 9C. Move the pointer pt1 to the position shown in the lower row (the position outside the rectangle). As a result, the dividing line sl2 is deleted, and it is corrected so that a plurality of consecutively arranged "・ (middle black)" symbols are collectively included in one character rectangle.

As shown in the upper part of FIG. 9D, the character rectangle extraction unit 17 recognizes the "-(hyphen)" symbol and the "2" character as one character, and recognizes the wrong character. The rectangle rc03, which is a rectangle, is extracted.

The user modifies the extracted character rectangle by erroneously recognizing two characters as one character. Specifically, for example, the user operates the mouse to move the pointer pt1 to the position of the upper end of the character string area sa6 shown in the upper part of FIG. 9D, and drags the pointer pt1 to FIG. 9 (C). ) To the position of the lower end of the character string area sa6 shown in the lower part.
By this operation, the dividing line sl3 is generated and displayed, and the "-(hyphen)" symbol and the character "2" are modified so as to be included in the character rectangle rc03a and the character rectangle rc03b, respectively.

(Mapping of ruby)
According to the character string area / character rectangle extraction device 1 according to the present embodiment, the user can associate a word included in a character string with ruby associated with the word.
The term "ruby" as used herein is a character that is written on the upper side of a word included in a character string with a character having a size smaller than that of the character string, and is a character representing the furigana of the word.
Hereinafter, the correspondence of ruby by the character string area / character rectangle extraction device 1 will be described with reference to the drawings.
FIG. 10 is a diagram showing an example of ruby extraction processing by the ruby mapping unit 18 of the character string region / character rectangle extraction device 1 according to the embodiment of the present invention.

FIG. 10A shows a character string area including the word “rectangle” and a ruby character string area including the furigana “Kukei”, which is the ruby corresponding to the word, by the character string area extraction unit 15. , Represents the extracted state. Further, FIG. 10A shows a state in which the character rectangle extraction unit 17 extracts the character rectangles for the characters included in the character string area and the characters included in the ruby character string area. show.

The user performs an operation of associating a word with a ruby character representing a furigana of the word by the operation input unit 12. Specifically, as illustrated in FIG. 10B, for example, the user operates the mouse to move the pointer pt1 to the vicinity of the word or ruby, and then drags the pointer pt1 to the word and ruby. Move to surround.

By performing this operation, as shown in FIG. 10C, a ruby correspondence area rb1 surrounding the word and the ruby representing the furigana of the word is generated and displayed, and the word and the furigana of the word are generated and displayed. Is associated with ruby that represents.
By associating the word with the ruby character representing the furigana of the word in this way, for example, when the character recognition unit 20 performs character recognition for the word and the ruby character, both the word and the ruby character are used. Since the information of can be utilized, the accuracy of character recognition can be improved.

(Management of character string area)
According to the character string area / character rectangle extraction device 1 according to the present embodiment, the user can use the character string area management screen generated by the management screen generation unit 19 to extract the character string area by the character string area extraction unit 15. It is possible to manage the combination and division of characters, and the order of the character string areas extracted by the character string area extraction unit 15.
11 to 14 are diagrams showing an example of a character string area management screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment of the present invention.

As shown in FIG. 11, the character string area management screen ms1 is a screen in which two screens are arranged side by side. Similar to the character string area extraction screen, the screen on the left side displays the image indicated by the image data acquired by the image data acquisition unit 11. Note that FIG. 11 shows a screen in which the start point and the end point are set for each line by the user's operation and the character string area is extracted for each line.

Character string area numbers are assigned to the extracted character string areas in the order of extraction, and are displayed on the character string area management screen ms1. As shown in FIG. 11, for example, the character string area sa11 is assigned the character string area number n1 representing the "1" number, and the character string area sa12 is assigned the character string area number n2 representing the "2" number. Is assigned.

Of the two screens that make up the character string area management screen ms1, the screen on the right side has a character string area number list that is a list of character string area numbers assigned to each character string area on the left screen. Is displayed. As shown in FIG. 11, for example, the list element ln1 representing the character string area number "1" assigned to the character string area sa11 on the left screen and the character string assigned to the character string area sa12 on the left screen. List elements such as ln2 representing the area number “2” are sequentially displayed in the character string area number list.

As described above, the character string area numbers are assigned to the character string areas in the order in which the character string areas are extracted. Further, the order of the list elements in the character string area number list represents the order recognized as the order of the content character strings (of the text) by the character string area / character rectangle extraction device 1.

That is, in the example shown in FIG. 11, in the character string area number list displayed on the screen on the right side, the list element ln1 representing the character string area number n1 and the list element ln2 representing the character string area number n2 are arranged in this order from the top. Since it is displayed, the character string area / character rectangle extraction device 1 is next to "3. Optical character recognition" which is a character string included in the character string area sa11 corresponding to the character string area number n1. In addition, it is recognized that "in the document in optical character recognition (OCR)", which is a character string included in the character string area sa12 corresponding to the character string area number n2, continues.

The character string area management screen ms1 shown in FIG. 11 represents a screen immediately after the start point and the end point are set for each line by the user's operation and the character string area is extracted for each line. Therefore, the order of the list elements displayed in the character string area number list is the same as the order of the character string area numbers. That is, in the character string area number list in the initial state (default state), the list elements are displayed in the order of the character string area number.

As described above, on the character string area management screen, the user can manage the combination and division of the character string area extracted by the character string area extraction unit 15 by the operation by the operation input unit 12.

The character string area management screen ms2 shown in FIG. 12 is an example of the state of the character string area management screen at a time point after the operation of combining the character string areas is performed by the user. As shown in the figure, the eight character string areas of the character string area sa12, the character string area sa13, the character string area sa14, ..., And the character string area sa19 in FIG. 11 are combined into one in FIG. It is the character string area sa21. Further, as shown in the figure, the two character string areas of the character string area sa20 and the character string area sa21 in FIG. 11 are combined to form one character string area sa23 in FIG.

Further, as shown in the figure, by combining the character string areas as described above, the character string area numbers n2 representing the "2" number in FIG. 11 and the character string area numbers n3 and "4" representing the "3" number are combined. In FIG. 12, the character string area numbers n4 and ... Representing the number "9" and the character string area number n9 representing the "9" are the character string area numbers n21 and "2" representing the "2-1" numbers, respectively. -The character string area number n22 representing "-2", the character string area number n23 representing "2-3", ..., And the character string area number n28 representing "2-8" have been replaced. .. Further, as shown in the figure, since the character string areas are combined as described above, the character string area number n10 representing the "10" number and the character string area number n11 representing the "11" number in FIG. 11 are replaced with each other. In FIG. 12, the character string area number n31 representing the "3-1" number and the character string area number n32 representing the "3-2" number are replaced, respectively.

Further, as shown in the figure, due to the replacement of the character string area numbers as described above, for example, the list element ln2 representing the "2" number in FIG. 11, the list element ln3 representing the "3" number, and " In FIG. 12, the list element ln4 representing the number 4 ”is a list element ln21 representing the number“ 2-1 ”, a list element ln22 representing the number“ 2-2 ”, and a list element representing the number“ 2-3 ”. Changes have been made to ln23. Further, as shown in the figure, due to the replacement of the character string area numbers as described above, the list element ln10 representing the "10" number and the list element ln11 representing the "11" number in FIG. 11 are shown in FIG. In, the list element ln31 representing the "3-1" number and the list element ln32 representing the "3-2" number have been changed.

As described above, on the character string area management screen, the user can manage the order change of the character string area extracted by the character string area extraction unit 15 by the operation by the operation input unit 12. Specifically, the user moves the pointer pt1 to the position of the character string area whose order is to be changed, and drags the pointer pt1 from that position to the desired destination position in the character string area number list to display the character. The order of the column areas is changed.

For example, as shown in FIG. 13, the user moves the pointer pt1 to the position of the character string area sa11 by the operation by the operation input unit 12, clicks the mouse button, and directly clicks the desired position in the character string area number list (FIG. 13). In the example shown in 13, the pointer pt1 is dragged to the position at the end of the character string area number list).

The character string area management screen ms3 shown in FIG. 14 is an example of the state of the character string area management screen at a time point after the operation of changing the order of the character string areas is performed by the user. As shown in the figure, the list element ln1 indicating the number "1" displayed at the beginning (at the top) of the character string area number list shown in FIG. 13 is at the end of the character string area number list shown in FIG. It is displayed (at the bottom, which is the position below the list element ln32), and all the other list elements are displayed one level up.

When the list elements in the character string area number list are rearranged as described above, the character string area / character rectangle extraction device 1 is included in the list element (corresponding to the character string area) after the rearrangement is performed. Recognize that the order of the character string) is the order of the content character string. That is, in the example shown in FIG. 14, the character string area / character rectangular extraction device 1 is, in terms of content, "・ a character string manually based on the user's operation" corresponding to the character string area number "3-2". Recognize that the character string "3. Optical character recognition" corresponding to the character string area number "1" follows the character string "Chapter 3-2". ..

(Operation of character string area / character rectangle extractor)
Hereinafter, the operation of the character string area / character rectangle extraction device 1 according to the embodiment will be described with reference to the drawings.
FIG. 15 is a flowchart showing the operation of the character string area / character rectangle extraction device 1 according to the embodiment of the present invention. This flowchart starts when image data indicating an image including a character string (for example, image data generated by reading a document with a scanner or the like) is input to the image data acquisition unit 11.

(Step st001) The image data acquisition unit 11 of the character string area / character rectangle extraction device 1 acquires image data (for example, image data generated by reading a document with a scanner or the like) indicating an image including a character string. .. After that, the process proceeds to step st002.

(Step st002) The control unit 10 of the character string area / character rectangle extraction device 1 stores the image data acquired by the image data acquisition unit 11 in the storage unit 14. Then, the control unit 10 performs various preprocessing on the image data stored in the storage unit 14. The pre-processing referred to here is a process of correcting the inclination of an image based on the image data acquired by the image data acquisition unit 11, a process of performing color separation of the image, and the like. That is, the preprocessing facilitates, for example, the processing of character string area extraction, character rectangle extraction, and character recognition, or enhances the accuracy of character string area extraction, character rectangle extraction, and character recognition. It is a process performed for the purpose. Then, the process proceeds to step st003.

As a processing method for correcting the inclination of characters, for example, a weighted histogram is created from a contour image for each of various shift angles as described in Patent Document 5, and an optimum path is searched for. A processing method for correcting the inclination of characters is known. Further, as a method of extracting a character string area by performing color separation, for example, a method of extracting a character string area using edge image data and color image data as described in Patent Document 6 is used. Are known.

(Step st003) The character string area extraction unit 15 of the character string area / character rectangle extraction device 1 is an image acquired by the image data acquisition unit 11 based on the operation input signal (auxiliary information) input from the operation input unit 12. The start point and end point of the line consisting of the character string included in the image based on the data are specified, and the character string area indicating the display target range of the character string included in the line is specified based on the start point and end point of the specified line. Extract. Then, the process proceeds to step st004.

(Step st004) The character string area combining unit 16 of the character string area / character rectangular extraction device 1 includes a character string area extracted by the character string area extraction unit 15 and another character string area adjacent to the character string area (step st004). For example, it is determined whether or not to combine with another adjacent character string area extracted before the character string area is extracted. Then, the process proceeds to step st005.

(Step st005) The character string area joining unit 16 extracts the character string area extracted by the character string area extraction unit 15 and another character string area adjacent to the character string area (for example, the character string area is extracted). If it is determined to combine (another adjacent character string area extracted before), the process proceeds to step st006. If not (that is, when it is determined not to combine), the process proceeds to step st007.

(Step st006) The character string area joining unit 16 extracts the character string area extracted by the character string area extraction unit 15 and another character string area adjacent to the character string area (for example, the character string area is extracted). (Other adjacent string areas extracted before), are combined. Then, the process proceeds to step st007.

(Step st007) The character string area joining unit 16 causes the display unit 13 to display an image showing the extracted character string area. Then, the user confirms the image displayed on the display unit 13 and confirms whether or not the character string area is extracted without error. If the character string area is extracted without error, the process proceeds to step st009. If this is not the case, that is, if there is a place where the character string area is erroneously extracted, the process proceeds to step st008.

(Step st008) The character string area extraction unit 15 modifies the extracted character string area based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user. The user performs an operation for correcting the character string area by the operation input unit 12 (for example, a mouse) while referring to the character string area management screen displayed on the display unit 13. Then, the process proceeds to step st009.

(Step st009) When the extraction of all the character string regions included in the image based on the image data acquired by the image data acquisition unit 11 is completed, the process proceeds to step st010.
If not, the process returns to step st003.

(Step st010) The character rectangle extraction unit 17 of the character string area / character rectangle extraction device 1 represents a character representing a rectangle of each character constituting the character string included in the character string area extracted by the character string area extraction unit 15. Extract the rectangle.
For example, the character rectangle extraction unit 17 determines an appropriate character cutout position from a plurality of character cutout position candidates based on an evaluation value calculated from character shape information, recognition accuracy in character recognition, and the like. By specifying, the character rectangle is extracted. After that, the process proceeds to step st011.

(Step st011) The character rectangle extraction unit 17 causes the display unit 13 to display an image showing the extracted character rectangle. Then, the user confirms the image displayed on the display unit 13 and confirms whether or not the character rectangle is extracted without error. If the character rectangle is extracted without error, the process proceeds to step st013. If this is not the case, that is, if there is a place where the character rectangle is erroneously extracted, the process proceeds to step st012.

(Step st012) The character rectangle extraction unit 17 corrects the extracted character rectangle based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user. The user performs an operation for correcting the character rectangle by the operation input unit 12 (for example, a mouse) while referring to the character rectangle management screen (not shown) displayed on the display unit 13. Then, the process proceeds to step st009.

(Step st013) The character recognition unit 20 of the character string area / character rectangle extraction device 1 performs optical character recognition (OCR) on an image showing characters included in each character rectangle extracted by the character rectangle extraction unit 17. Do and recognize characters. Specifically, the character recognition unit 20 analyzes image data indicating characters contained in a document (for example, read by a scanner) and converts it into a data format (for example, a character code string) that can be edited by a computer. do.
This completes the process shown in this flowchart.

As described above, the character string area / character rectangle extraction device 1 according to the present embodiment has an image data acquisition unit 11 for acquiring image data indicating an image including a character string, and an image based on the acquired image data. The start point and the end point of the line consisting of the character string included in the image are specified and specified based on the display unit 13 that displays the display unit 13, the operation input unit 12 that accepts the operation input from the user, and the auxiliary information based on the operation input. The character string area extraction unit 15 that extracts the character string area indicating the display target range of the character string included in the line based on the start point and the end point of the line, and the extracted character string area and the character string area are adjacent to each other. A character string area connecting portion 16 for connecting to another character string area is provided.

As described above, the character string area / character rectangle extraction device 1 according to the embodiment of the present invention can improve the accuracy of the character string area extraction and the character rectangle extraction by using the auxiliary information based on the user's operation.

<Modification 1 of the embodiment>
In the character string area / character rectangle extraction device 1 according to the above-described embodiment, the processing of combining the character string areas is performed according to the auxiliary information based on the user's operation. However, the configuration may be such that the processing of combining the character string regions is performed using the information indicating the result of character recognition by the character recognition unit 20.
In the character string area / character rectangle extraction device 2 according to the first modification of the embodiment described below, the processing of combining the character string areas is performed using the information indicating the result of character recognition by the character recognition unit 20.

(Structure of character string area / character rectangle extractor)
Hereinafter, the configuration of the character string area / character rectangle extraction device 2 according to the modification 1 of the embodiment will be described with reference to the drawings.
FIG. 16 is a block diagram showing a functional configuration of the character string area / character rectangle extraction device 2 according to the first modification of the embodiment of the present invention.
As shown in the figure, the character string area / character rectangular extraction device 2 according to the modified example 1 of the embodiment includes a control unit 10, an image data acquisition unit 11, an operation input unit 12, a display unit 13, and a storage unit 14. , A character string area extraction unit 15, a character string area combination unit 16, a character rectangle extraction unit 17, a ruby mapping unit 18, a management screen generation unit 19, and a character recognition unit 20. To.
Further, the character string area joining unit 16 includes a language analysis unit 26.

The control unit 10 controls various processes in the character string area / character rectangle extraction device 2. The control unit 10 includes, for example, a CPU.

The image data acquisition unit 11 acquires image data indicating an image including a character string. The image data acquisition unit 11 includes, for example, an input interface for acquiring image data generated by reading a document by an external device such as a scanner. Alternatively, the image data acquisition unit 11 includes, for example, a scanner.

The operation input unit 12 receives an operation input from the user. The operation input unit 12 generates an operation input signal (auxiliary information) based on the received operation input, and the display unit 13, the character string area extraction unit 15, the character string area connection unit 16, the character rectangle extraction unit 17, and the ruby, which will be described later. It is output to the correspondence unit 18 or the management screen generation unit 19. The operation input unit 12 includes a pointing device used for the user to operate a pointer (or a cursor) displayed on the display unit 13 described later, for example, a mouse, a touch pad, a touch panel, a stylus, a trackball, or the like. Consists of.

In the first modification of the present embodiment, the character string area / character rectangle extraction device 1 is provided with the operation input unit 12, but the present invention is not limited to this. For example, the operation input unit 12 may be provided in an external device, and the character string area / character rectangle extraction device 1 may acquire a signal indicating a pointer operation by the user from the external device.

The display unit 13 displays an image based on the image data acquired by the image data acquisition unit 11. The display unit 13 includes a display, for example, a liquid crystal display (LCD), an organic EL display, a CRT, or the like.

In the first modification of the present embodiment, the character string area / character rectangle extraction device 1 is provided with the display unit 13, but the present invention is not limited to this. For example, the display unit 13 may be provided in an external device, and the character string area / character rectangle extraction device 1 may transmit image data indicating an image to be displayed to the external device.

The storage unit 14 stores various computer programs and data used in the character string area / character rectangle extraction device 1. Further, the storage unit 14 also has a function as a temporary storage area used in various arithmetic processes and the like in the character string area / character rectangle extraction device 1. The storage unit 14 includes a storage medium, for example, an HDD, a flash memory, an EEPROM, a RAM, a ROM, or any combination thereof.

The character string area extraction unit 15 is the start point of a line consisting of a character string included in the image based on the image data acquired by the image data acquisition unit 11 based on the operation input signal (auxiliary information) input from the operation input unit 12. And the end point are specified, and the character string area indicating the display target range of the character string included in the line is extracted based on the start point and the end point of the specified line.

The character string area joining unit 16 is extracted before the character string area extracted by the character string area extraction unit 15 and another character string area adjacent to the character string area (for example, the character string area is extracted). And other adjacent string areas) and. The character string area combining unit 16 causes the display unit 13 to display an image showing the combined character string area.

Further, as described above, the character string region joining unit 16 includes a language analysis unit 26.
The language analysis unit 26 analyzes information indicating the result of character recognition by the character recognition unit 20. Then, the language analysis unit 26 determines whether or not to combine the character string regions based on the information indicating the analysis result. Specifically, for example, the language analysis unit 26 provides information indicating the result of character recognition by the character recognition unit 20 for characters contained in each of two character string regions, and statistical statistics on the occurrence probability of the characters. Whether to combine string regions based on information (for example, the N-gram model, which is a language model that investigates how many combinations of N strings or words appear in a string). Whether or not (that is, whether or not the two character string areas are the same character string area) is determined.

Alternatively, the character string area combining unit 16 has the extracted character string area and the character string area based on the operation input signal (auxiliary information) based on the operation of the pointer by the user, which is input from the operation input unit 12. Combines with other adjacent string areas.
That is, the character string area joining unit 16 may (manually) join the character string areas by the user's operation, or (automatically) join the character string areas based on the statistical information.

The character rectangle extraction unit 17 extracts a character rectangle representing a rectangle of each character constituting the character string included in the character string area extracted by the character string area extraction unit 15. Further, the character rectangle extraction unit 17 corrects the extracted character rectangle based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user.

The ruby mapping unit 18 corresponds to the character string included in the character string area extracted by the character string area extraction unit 15 based on the operation input signal (auxiliary information) based on the operation of the pointer by the user, and the character string. Correspond to the ruby. The ruby correspondence unit 18 causes the display unit 13 to display an image showing a ruby correspondence area which is a range in which the associated character string and the ruby corresponding to the character string are enclosed.

The management screen generation unit 19 generates a management screen for the user to modify the character string area and the character rectangle. The management screen generation unit 19 associates a line number, which is a number assigned in the order of extraction, with the character string area (character string area extracted in line units) extracted by the character string area extraction unit 15. Then, the management screen generation unit 19 includes an image in which a line number is assigned to each of the character string areas (character string areas extracted in line units) included in the image based on the image data acquired by the image data acquisition unit 11. , A management screen for displaying a list display image, which is an image in which the line number is displayed in a list, is displayed on the display unit 13.

The character recognition unit 20 performs optical character recognition (OCR) on an image showing characters included in each character rectangle extracted by the character rectangle extraction unit 17, and recognizes the characters. Specifically, the character recognition unit 20 analyzes image data indicating characters contained in a document (for example, read by a scanner) and converts it into a data format (for example, a character code string) that can be edited by a computer. do.

(Operation of character string area / character rectangle extractor)
Hereinafter, the operation of the character string area / character rectangle extraction device 2 according to the modification 1 of the embodiment will be described with reference to the drawings.
FIG. 17 is a flowchart showing the operation of the character string area / character rectangle extraction device 2 according to the first modification of the embodiment of the present invention. This flowchart starts when image data indicating an image including a character string (for example, image data generated by reading a document with a scanner or the like) is input to the image data acquisition unit 11.

(Step st101) The image data acquisition unit 11 of the character string area / character rectangle extraction device 2 acquires image data (for example, image data generated by reading a document with a scanner or the like) indicating an image including the character string. .. Then, the process proceeds to step st102.

(Step st102) The control unit 10 of the character string area / character rectangle extraction device 2 stores the image data acquired by the image data acquisition unit 11 in the storage unit 14. Then, the control unit 10 performs various preprocessing on the image data stored in the storage unit 14. The pre-processing referred to here is a process of correcting the inclination of the image indicated by the image data, a process of performing color separation, and the like. That is, the preprocessing facilitates, for example, the processing of character string area extraction, character rectangle extraction, and character recognition, or enhances the accuracy of character string area extraction, character rectangle extraction, and character recognition. It is a process performed for the purpose. Then, the process proceeds to step st003.

(Step st103) The character string area extraction unit 15 of the character string area / character rectangle extraction device 2 is an image acquired by the image data acquisition unit 11 based on the operation input signal (auxiliary information) input from the operation input unit 12. The start point and end point of the line consisting of the character string included in the image based on the data are specified, and the character string area indicating the display target range of the character string included in the line is specified based on the start point and end point of the specified line. Extract. Then, the process proceeds to step st104.

(Step st104) When the extraction of all the character string regions included in the image based on the image data acquired by the image data acquisition unit 11 is completed, the process proceeds to step st105.
If not, the process returns to step st103.

(Step st105) The character rectangle extraction unit 17 of the character string area / character rectangle extraction device 2 represents a character representing a rectangle of each character constituting the character string included in the character string area extracted by the character string area extraction unit 15. Extract the rectangle. Then, the process proceeds to step st106.

(Step st106) The character rectangle extraction unit 17 causes the display unit 13 to display an image showing the extracted character rectangle. Then, the user confirms the image displayed on the display unit 13 and confirms whether or not the character rectangle is extracted without error. If the character rectangle is extracted without error, the process proceeds to step st108. If this is not the case, that is, if there is a place where the character rectangle is erroneously extracted, the process proceeds to step st107.

(Step st107) The character rectangle extraction unit 17 corrects the extracted character rectangle based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user. The user performs an operation for correcting the character rectangle by the operation input unit 12 (for example, a mouse) while referring to the character rectangle management screen (not shown) displayed on the display unit 13. Then, the process proceeds to step st108.

(Step st108) The character recognition unit 20 of the character string area / character rectangle extraction device 2 performs optical character recognition (OCR) on an image showing characters included in each character rectangle extracted by the character rectangle extraction unit 17. Do and recognize characters. Specifically, the character recognition unit 20 analyzes image data indicating characters contained in a document (for example, read by a scanner) and converts it into a data format (for example, a character code string) that can be edited by a computer. Convert. Then, the process proceeds to step st109.

(Step st109) The language analysis unit 26 provided in the character string area joining unit 16 analyzes information indicating the result of character recognition by the character recognition unit 20. Then, the language analysis unit 26 determines whether or not to combine the character string regions based on the information indicating the analysis result. Specifically, for example, the language analysis unit 26 provides information indicating the result of character recognition by the character recognition unit 20 for characters contained in each of two character string regions, and statistical statistics on the occurrence probability of the characters. Whether to combine string regions based on information (for example, the N-gram model, which is a language model that investigates how many combinations of N strings or words appear in a string). Whether or not (that is, whether or not the two character string areas are the same character string area) is determined.

Then, the character string area joining unit 16 has the character string area extracted by the character string area extraction unit 15 based on the result of the above determination by the language analysis unit 26, and another character string adjacent to the character string area. The area is combined (eg, another adjacent string area extracted before the string area was extracted). Then, the process proceeds to step st110.

(Step st110) The character string area combining unit 16 causes the display unit 13 to display an image showing the combined character string area. Then, the user confirms the image displayed on the display unit 13 and confirms whether or not the character string areas are combined without error. If the character string areas are combined without error, the process shown in this flowchart ends. If this is not the case, that is, if there is a place where the character string areas are erroneously combined (or erroneously divided), the process proceeds to step st111.

(Step st111) The character string area extraction unit 15 modifies the extracted character string area based on the operation input signal (auxiliary information) input from the operation input unit 12 based on the operation of the pointer by the user. The user performs an operation for correcting the character string area by the operation input unit 12 (for example, a mouse) while referring to the character string area management screen displayed on the display unit 13.
This completes the process shown in this flowchart.

As described above, the character string area / character rectangle extraction device 1 according to the modification 1 of the present embodiment has an image data acquisition unit 11 for acquiring image data indicating an image including a character string, and an acquired image. A display unit 13 that displays an image based on data, an operation input unit 12 that receives an operation input from a user, and a start point and an end point of a line consisting of a character string included in the image based on auxiliary information based on the operation input. The character string area extraction unit 15 for extracting the character string area indicating the display target range of the character string included in the line based on the start point and the end point of the specified line, the extracted character string area, and the character string. A character string area connecting portion 16 for connecting an area and another adjacent character string area is provided.

Further, the character string area / character rectangular extraction device 1 according to the modification 1 of the present embodiment analyzes the information indicating the result of character recognition by the character recognition unit 20, and the character string is based on the information indicating the analyzed result. A language analysis unit 26 for determining whether or not to combine areas is provided, and the character string area combination unit includes the character string area extracted and the character string area based on the result determined by the language analysis unit 26. Combines with other adjacent string areas.

As described above, the character string area / character rectangle extraction device 2 according to the first modification of the embodiment of the present invention improves the accuracy of the character string area extraction and the character rectangle extraction by using the auxiliary information based on the user's operation. be able to. Further, since the character string area / character rectangular extraction device 2 according to the first modification of the embodiment of the present invention can determine the combination of the character string areas by using the information indicating the result of the character recognition, the character string can be determined. It is possible to improve the accuracy of area extraction and improve the efficiency of processing for extracting the character string area.

<Modification 2 of the embodiment>
In the character string area / character rectangle extraction device 1 according to the above-described embodiment, the character string area extraction unit 15 corresponds to auxiliary information based on the user's operation, and from the character string included in the image displayed by the display unit 13. The position of the start point and the end point of the line where the user's operation was performed was specified. However, the character string area extraction unit 15 identifies the positions of the start points and end points of other lines included in the image based on the line in which the start point and the end point are specified by the auxiliary information based on the user's operation on the image. May be good.
In the character string area / character rectangular extraction device 1 according to the second modification of the embodiment described below, the character string area extraction unit 15 designates the start point and the end point of the image displayed by the display unit 13 by the user's operation. Based on the row, the location of the start and end points of other rows contained in the image is determined. In the description of the second modification of the embodiment described below, the same reference numerals are given to the same configurations as those of the embodiment, and the description thereof will be omitted.

The character string area extraction unit 15 is derived from the character string included in the image based on the start point and the end point specified by the auxiliary information based on the operation input for the line consisting of the character string included in the image displayed by the display unit 13. Generates specific information that identifies the positions of the start and end points of other lines, and extracts the character string area of the character string contained in the other lines based on the start and end points specified by the specific information. For example, in the character string area extraction unit 15, when the start point and end point positions of any one of a plurality of lines included in the image are specified by a user input operation, the start point and end point positions of the remaining lines are specified. To identify.

Further, the character string area extraction unit 15 is generated for the image by the auxiliary information based on the operation input for at least one of the start point and the end point specified for the image displayed by the display unit 13. Correct specific information. For example, when the position of the start point and the end point of a plurality of lines included in the image is specified, the character string area extraction unit 15 determines the position of the start point of any one of the start points of the plurality of lines by a user input operation. Once modified, it modifies the position of the starting point of the remaining lines. Alternatively, when the positions of the start point and the end point of a plurality of lines included in the image are specified, the character string area extraction unit 15 determines the position of the end point of any one of the start points of the plurality of lines by a user input operation. Once modified, it modifies the position of the end points of the remaining lines.

Further, the character string area extraction unit 15 modifies the character string area extracted for the image by the auxiliary information based on the operation input for the character string area extracted for the image displayed by the display unit 13. .. For example, in the character string area extraction unit 15, when the positions of the character string areas of a plurality of character strings included in the image are extracted, the position of any one of the plurality of character string areas is the position of the user. When corrected by an input operation, the position of the remaining character string area is corrected.

Further, the character string area extraction unit 15 contains an operation input signal (auxiliary information) performed by the user for the first image displayed by the display unit 13, and specific information generated for the first image. Based on at least one of them, specific information for the second image displayed by the display unit 13 is generated.
Here, the first image and the second image are images including a character string to be extracted from the character string area. For example, the first image is the first page of a book, and the second image is 2 of the book. It is an image of a page or the like.
For example, the character string area extraction unit 15 determines the start point and end point of the character string included in another page of the book based on the positions of the start point and the end point specified by the user with respect to the image obtained by capturing the image of one page of the book. Identify the location. Alternatively, the character string area extraction unit 15 separates the books based on the positions of the start points and end points of other lines specified from the positions of the start points and end points specified by the user with respect to the image obtained by capturing a page of the book. Specify the position of the start point and end point of the character string contained in the page of. Alternatively, the character string area extraction unit 15 has the positions of the start point and the end point specified by the user with respect to the image obtained by capturing a page of the book, and the start point and the end point of another line specified by the character string area extraction unit 15. Determine the position of the start and end points of a string contained on another page of the book based on the position.

Further, the character string area extraction unit 15 extracts a character string area for a second image different from the first image based on the information regarding the character string area extracted for the first image displayed by the display unit 13. .. For example, the character string area extraction unit 15 extracts a character string area included in another page of a book based on the position of the character string area extracted with respect to an image obtained by capturing a page of the book.

(Generation of specific information)
Hereinafter, the character string area extraction unit 15 will use another line consisting of the character string included in the image based on the start point and the end point specified by the auxiliary information based on the operation input for the line consisting of the character string included in the image. A method of generating specific information for specifying the positions of the start point and the end point of the above will be described with reference to the drawings.
18 to 19 are diagrams showing an example of a character string area extraction screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment.

As shown in FIG. 18, the user operates the pointer pt1 by the operation input unit 12 to specify the range of the character string included in the character string area extraction screen ds2 in line units. Here, the user will be described as designating the start point and the end point of the character string described in the first line at the right end of the vertically written character strings included in the image displayed on the character string area extraction screen ds2. Further, since the operation of specifying the character string area for each line by the user is the same operation as that already described, detailed description thereof will be omitted here.

When the user performs an operation to specify the start point and the end point of the line, the display unit 13 displays a white circle icon at the start point st1 and a black circle icon at the end point ed1 on the character string area extraction screen ds2. Will be done. Further, when the user performs an operation of designating the end point of the line, the connection line cn1 connecting the start point st1 and the end point ed1 is displayed on the character string area extraction screen ds2.

Then, when the start point and the end point of the line are specified by the user, the character string area extraction unit 15 is based on the positional relationship of the line (in the example of FIG. 18, the first line) to which the start point and the end point are specified. , Generates specific information for specifying the positions of the start points and end points of other lines (lines after the second line in the example of FIG. 18) included in the image displayed on the character string area extraction screen ds2. The specific information is, for example, xy coordinate values of the start point and the end point of the second and subsequent rows. Hereinafter, it is assumed that the specific information is the xy coordinate values of the start point and the end point of the row. Further, in the following, in the process of generating specific information, a character string arranged in the y-axis direction (that is, a vertically written character string) in the xy coordinate system is moved in the negative direction of the x-axis (that is, from right to left). The operation of processing in the direction of) will be described as an example.

For example, assuming that the xy coordinate value of the start point st1 of the first row specified by the user's operation is (xst1, yst1), the character string area extraction unit 15 has the xy coordinates of the start point Est2 of the second row. The value is (xst1-d, yst1). That is, the start point Est2 of the second row is set to a position d away from the position of the start point of the first row in the negative direction of the x-axis and does not change in the y-axis direction (same y-coordinate value).
Further, if the xy coordinate value of the end point ed1 of the first line specified by the user's operation is (xed1, yed1), the character string area extraction unit 15 has the end point Eed2 of the second line. Let the xy coordinate value be (xed1-d, yesd1). That is, the end point Eed2 of the second row is a position where the distance between the rows is d in the x-axis direction from the position of the end point of the first row, and the y coordinate of the start point of the first row in the y-axis direction. The position is the same as the value.
Note that d may be a value stored in the storage unit 14 in advance, or may be a value specified by the user. Further, d may be a value set according to the type of the image displayed on the character string area extraction screen ds2. The type of the image is, for example, an image obtained by capturing a character string described based on the format of a paperback book, an image obtained by capturing a character string described based on the format of a new book, or the like. As a result, when the space between lines differs between the format of the paperback book and the format of the new book, the character string area extraction unit 15 can generate specific information using the space between lines corresponding to each.

Similarly, for the third and subsequent lines, the character string area extraction unit 15 sets the xy coordinate values of the start points Est3 to Est8 (xst1-k × d, yst1) and the xy coordinate values of the end points Eed3 to Eed8 (xst1-k × d, yst1). Let it be xed1-k × d, yed1). Here, k is a value obtained by subtracting 1 from the line number. The line number is a number set in the order of 3, the third line is 3, the fourth line is 4, and so on. In this way, the character string area extraction unit 15 extracts the character string area based on the xy coordinates of the start point of the first line included in the image displayed on the character string area extraction screen ds2 designated by the user's operation. The xy coordinate value of the start point of the second and subsequent rows of the image displayed on the screen ds2 is the position where the space between the rows is d in the x-axis direction with respect to the first row specified by the user. The position is the same as the y-coordinate value of the start point of the first row in the y-axis direction.

In the example of FIG. 18, the display unit 13 displays the character string area extraction screen ds2 on the character string area extraction screen ds2, the display unit 13 has a white square icon at the start point Est2 of the second line, and the end point Eed2 based on the specific information specified by the character string area extraction unit 15. A black square icon is displayed on each. Further, the display unit 13 displays a white square icon at the start points Est3 to Est8 of the third and subsequent lines and a black square icon at the end points Eed3 to Eed8, respectively, as in the second line.

Here, when the character string area extraction unit 15 specifies the positions of the start point and the end point of the second and subsequent lines, the start point and the start point and the end point regardless of whether or not the character string is shown at the position corresponding to the line. The position of the end point is specified. The character string area extraction unit 15 sets all points that can be displayed on the character string area extraction screen ds2 as starting points at points corresponding to points separated by an integral multiple of d in the x-axis direction from the xy coordinate value of the starting point st1. do. Further, the character string area extraction unit 15 has all the points that can be displayed on the character string area extraction screen ds2 at points corresponding to points separated by an integral multiple of d in the x-axis direction from the xy coordinate value of the end point ed1. Let it be the end point. In the example of FIG. 18, the character string area extraction unit 15 identifies the start point Est8 and the end point Eed8 even when the character string does not exist between the start point Est8 and the end point Eed8.

As shown in FIG. 19, the character string area extraction unit 15 identifies the positions of the start point and the end point of another line based on the xy coordinate values of the start point and the end point of the first line by the user, and then the specified start point. The rectangular character string areas sa1 to sa7 surrounding the character string included from to the end point are extracted. Since the process of extracting the character string area by the character string area extraction unit 15 is the same process as that already described, detailed description thereof is omitted here, but the character string area extraction unit 15 is the specified start point. The range where the connection line connecting the end points and the character intersects is defined as the character string area. Therefore, when the character string does not exist between the start point and the end point as in the start point Est8 and the end point Eed8 in FIG. 18, the character string area corresponding to the start point Est8 and the end point Eed8 is not extracted.

In the above, the character string area extraction unit 15 has described the case where a vertically written character string is shown in the image displayed on the character string area extraction screen ds2. However, even when the horizontal character string is shown in the image displayed on the character string area extraction screen ds2, the character string area extraction unit 15 is set at the start point and end point positions of the line specified by the user's operation. Based on this, the positions of the start point and the end point of other lines included in the image displayed on the character string area extraction screen ds2 can be specified. In this case, the horizontally written character string is a character string arranged in the x-axis direction in the xy coordinate system, and is a character string shown in the positive direction of the y-axis (that is, from top to bottom). From this, the character string area extraction unit 15 has the same x-coordinate value in the x-axis direction with respect to the xy-coordinate value of the start point of the line specified by the user's operation, and is in the positive direction of the y-axis. A position separated by a fixed interval (for example, d) is set as the position of the start point of another line. Further, the character string area extraction unit 15 has the same x-coordinate value in the x-axis direction with respect to the xy-coordinate value of the end point of the line specified by the user's operation, and is constant in the positive direction of the y-axis (the character string area extraction unit 15 has the same value in the x-axis direction. For example, the positions separated by the interval of d) are set as the positions of the end points of other rows.

The character string area extraction unit 15 is a vertically written character string, and is not in the negative direction of the x-axis (that is, the direction from right to left) but in the positive direction of the x-axis (that is, from left to right). Direction) or a horizontal string that is not in the positive direction of the y-axis (that is, from top to bottom) but in the negative direction of the y-axis (that is, from bottom to top). ), The same method as above is used to display the image displayed on the character string area extraction screen ds2 based on the positions of the start and end points of the line specified by the user's operation. The positions of the start and end points of other contained lines can be specified.

As described above, in the character string area / character rectangular extraction device 1 of the modification 2 of the present embodiment, the character string area extraction unit 15 displays the character string displayed by the display unit 13 based on the auxiliary information based on the operation input. Of the lines consisting of character strings displayed in the area extraction screen ds2 and consisting of the character strings displayed in the image based on the image data acquired by the image data acquisition unit 11 (an example of "image"), the start point st1 of the line specified by the auxiliary information. And the end point ed1 generates specific information that specifies the start point and end point of another line (for example, the second line to the eighth line) different from the line (for example, the first line), and is specified based on the specific information. Based on the start points Est2 to Est8 and the end points Eed2 to Eed8 of the other lines, the character string areas sa2 to sa5 of the character strings included in the other lines are extracted.

In the case of books and the like, character strings are often written in a regular arrangement at regular intervals. For this reason, in most cases, the character strings included in the captured image of a book or the like are arranged at regular intervals. That is, the character string area extraction unit 15 can specify the start points Est2 to Est8 and the end points Eed2 to Eed8 of other lines based on the start point st1, the end point ed1, and the connection line cn1 of the line specified by the user. .. Since the character string area extraction unit 15 can specify the positions of the start point and the end point of another line, the user does not need to specify the start point and the end point for each line of the character string in the image. That is, in the character string area / character rectangle extraction device 1, the accuracy of extracting the character string area can be improved by using the auxiliary information based on the user's operation, and the character string area extraction unit 15 is based on the auxiliary information. , The positions of the start and end points of other lines can be specified, and the user can save the trouble of specifying the start point and the end point for each line consisting of a character string.

The character string area extraction unit 15 may be able to select whether or not to specify the positions of the start points and end points of other lines when the start point and end point of a specific line are specified by the user. Then, when the position of the start point and the end point of another line is selected, the character string area extraction unit 15 specifies the positions of the start point and the end point of all the lines of the other line, or other It may be possible to select whether to specify the positions of the start and end points of a part of the line.
When the character string area extraction unit 15 specifies the positions of the start point and the end point of a part of another line, for example, the character string is described in seven lines in the image, and the second to fourth lines in the image. The case of specifying the positions of the start point and the end point of the line up to is described. First, the user moves the pointer pt1 to a position near the start point of the second row, clicks the mouse button, and drags the pointer pt1 to the position of the end point of the fourth row as it is. After that, the pointer pt1 is moved to a position near the start point of the second row and the mouse button is clicked, then the pointer pt1 is moved to the position of the end point of the second row and the mouse button is clicked. By performing such an input operation, the character string area extraction unit 15 performs three or four based on the positions of the start point and the end point specified for the second line of the seven lines of the character string. Specify the positions of the start and end points of each line in the line.
When the character string area extraction unit 15 identifies the positions of the start point and the end point of a part of a plurality of lines so that the character strings described in the plurality of lines have different formats from each other (for example, a table of contents or a heading). It is possible to specify the positions of the start point and the end point according to each format (character string indicating, character string indicating the text, etc.).

(Correction of specific information)
Hereinafter, a method of correcting the specific information generated by the character string area extraction unit 15 based on the auxiliary information based on the operation of the pointer by the user will be described with reference to the drawings.
20 to 21 are diagrams showing an example of a character string area extraction screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment.

As shown in FIG. 20, even if the character string area extraction unit 15 specifies the positions of the start points and end points of other lines included in the same image based on the positions of the start points and end points of the line specified by the user. , Does not always match the actual start and end points of the line. In the example of FIG. 20, the position of the start point specified by the character string area extraction unit 15 is shifted to the right from the position of the actual start point. The character string area extraction unit 15 sets a position on another line at a position separated by an integral multiple of d in the x-axis direction from the x-coordinate value of the start point specified by the user and in which the coordinate value in the y-axis direction does not change. The position of the starting point was used. However, in reality, the position of the start point of the row specified by the user and the position of the start point of another row are separated by d1 in the x-axis direction, so that the above-mentioned deviation in the right direction occurs.

When the positions of the start point and the end point of the other lines specified by the character string area extraction unit 15 are different from the actual start point and end point positions, the start point and the end point specified by the character string area extraction unit 15 are connected. The connecting line may not intersect the string. If the connecting line does not intersect the character string, the character string area extraction unit 15 cannot extract the character string area. Therefore, it is desirable that the character string area extraction unit 15 can correct the start point and the end point specified for the image.

According to the character string area / character rectangular extraction device 1 according to the modification 2 of the present embodiment, the character string area extraction unit 15 is generated for the character string included in the screen based on the input operation by the user. Specific information can be modified.
As shown in FIG. 21, the user operates the pointer pt1 by the operation input unit 12 at the position of the start point Est2 specified with respect to the character string included in the screen displayed on the character string area extraction screen ds2. Move it. Specifically, for example, the user operates the mouse to move the pointer pt1 to the position where the start point Est2 shown by the dotted line in FIG. 21 is displayed, drags the start point Est2, and is shown by the solid line in FIG. Move the start point Est12 to the displayed position. As a result, the character string area extraction unit 15 corrects the position of the start point of the second line from the start point Est2 to the place where the start point Est12 is displayed.

When the position of the start point of the second line is corrected by the pointer input operation of the user, the character string area extraction unit 15 corrects the position of the start point of the third and subsequent lines based on the corrected content. Specifically, the character string area extraction unit 15 calculates the difference between the x-coordinate value of the start point Est1 of the first row and the x-coordinate value of the start point Est12 of the modified second row. The difference between the x-coordinate values corresponds to the distance between the first row and the second row in the x-axis direction, and in the example of FIG. 21, the difference in the x-axis direction is d1. Further, the character string area extraction unit 15 calculates the difference in y-coordinates between the start point Est2 of the second row before modification and the start point Est12 of the second row after modification. The difference in the y-coordinate corresponds to the correction amount in the y-axis direction.

In the character string area extraction unit 15, the xy coordinate value of the start point st1 of the first line specified by the user's operation is (xst1, yst1), and the start point Est12 of the second line is corrected by the user's pointer operation. It is assumed that the xy coordinate value is (xst1-d1, yst1 + y1). That is, it is assumed that the user has made a correction to change the space between rows from d to d1 in the x-axis direction and to move y1 in the positive direction of the y-axis. In this case, the character string area extraction unit 15 sets the xy coordinate value of the start point Est3 of the third line to (xst1-2 × d1, yst1 + y1). That is, the character string area extraction unit 15 sets the position of the start point Est3 of the third row to a position d1 away from the position of the start point of the second row in the x-axis direction and does not change in the y-axis direction. Similarly, the character string area extraction unit 15 sets the xy coordinate values of the start points Est4 to Est8 to (xst1-k × d1, yst1 + y1) for the fourth and subsequent rows. Here, k is a value obtained by subtracting 1 from the line number.

The display unit 13 erases the white square icon displayed at the start points Est11 to Est18 on the character string area extraction screen ds3 based on the specific information corrected by the character string area extraction unit 15, and at the start points Est21 to Est28. Display each of the white square icons.

In the above, an example in which the character string area extraction unit 15 corrects the position of the start point has been described, but the same applies to the end point. For example, when the position of the end point of the second line is corrected by the pointer input operation of the user, the character string area extraction unit 15 corrects the position of the end point of the third and subsequent lines based on the correction content. ..

As described above, in the character string area / character rectangular extraction device 1, the character string area extraction unit 15 corrects the specific information based on the operation input signal (auxiliary information) input from the operation input unit 12. If, for example, the user corrects the position of one of the plurality of start points or end points specified by the character string area extraction unit 15, the character string area extraction unit 15 remains according to the correction content. Correct the position of the start point or end point of. Therefore, the user does not need to correct the position of the start point or the end point for each line. Therefore, in the character string area / character rectangle extraction device 1, the accuracy of extracting the character string area can be improved by using the auxiliary information based on the user's operation, and the character string area extraction unit 15 can be used for other lines. Even if the start point or end point is erroneously specified, it is possible to save the trouble of correcting the erroneously specified start point or end point line by line.

Further, the character string area extraction unit 15 may be able to select whether or not to correct the start point or end point of another line when the position of the start point or end point of a specific line is corrected by the user. .. Then, when it is selected to modify the start point or end point of another line, the character string area extraction unit 15 corrects the start point or end point of all the lines of the other line, or one of the other lines. It may be possible to select whether to modify the start point or the end point of the line of the part.

When the character string area extraction unit 15 modifies the start point or end point of a part of another line, for example, the positions of the start point and end point of seven lines are specified in the image, and the second to fourth lines in the image are specified. The case of correcting the start point of the line up to is described. First, the user moves the pointer pt1 in order to a position near the start point of the lines from the second line to the fourth line while pressing the shift key, and clicks the mouse button each time the pointer is moved. Next, the user stops pressing the shift key, moves the pointer pt1 to a position near the start point of the second line, clicks the mouse button, and then clicks the pointer to the position where the start point of the second line is desired to be moved. Move pt1 and drag. By performing such an input operation, the character string area extraction unit 15 performs the third and fourth lines based on the position of the starting point corrected with respect to the second line of the seven lines of the character string. Correct the position of the start point of each line of.

(Correction of character string area)
Hereinafter, a method of modifying the character string area extracted by the character string area extraction unit 15 based on the auxiliary information will be described with reference to the drawings.
FIG. 22 is a diagram showing an example of a character string area extraction screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment.

As shown in FIG. 22, even if the character string area extraction unit 15 specifies the character string area of the character string included in the image, it does not always match the actual character string area. In the example of FIG. 22, in each of the character string areas sa1 to sa7 specified by the character string area extraction unit 15, the area in which the character string is described is the area in which the spelling pseudonym attached to the character string is described. Has been included.
As shown in FIG. 22, when a character other than the character included in the character string is included in the character string area, there is a possibility that the character rectangle extraction unit 17 described later cannot extract the character rectangle. be.

According to the character string area / character rectangle extraction device 1 according to the second modification of the present embodiment, the user can modify the character string area specified by the character string area extraction unit 15.
As shown in FIG. 22, the user moves the rightmost line of the character string area sa1 by operating the pointer pt1 by the operation input unit 12. Specifically, for example, the user operates the mouse to move the pointer pt1 to a position near the corner of the upper right upper end of the character string region sa1 shown by the dotted line in FIG. 22, and drags the pointer pt1 from the position near the corner. Then, it is moved to a position near the corner of the upper right upper end of the character string area sa11 shown by the solid line in FIG. As a result, the area of the character string area sa1 is modified to the area of the character string area sa11, and the character string area sa11 contains only the character string and does not include the area in which the pseudonym of the character string is described. It will be fixed.

When the character string area sa1 is modified by the input operation of the user, the character string area extraction unit 15 modifies each of the other character string areas sa2 to sa7 based on the modified content. Specifically, the character string area extraction unit 15 has xy coordinate values of the representative points of the character string area sa1 before modification (for example, the upper right corner of the area) and the representative points of the character string area sa11 after modification. The movement amount is calculated by comparing the xy coordinate values. The character string area extraction unit 15 modifies the character string areas sa3 to sa7 by moving their respective representative points by the calculated movement amount with respect to each of the other character string areas sa2 to sa7, and after each correction. The character string areas are sa13 to sa17.

The character string area extraction unit 15 enables the user to select whether or not to modify the character string area corresponding to another line when the character string area corresponding to a specific line is modified by the user. May be good. Then, when it is selected to modify the character string area corresponding to the other line, the character string area extraction unit 15 corrects all of the character string area corresponding to the other line, or the other line. It may be possible to select whether to modify a part of the character string area corresponding to.

When the character string area extraction unit 15 modifies a part of the character string area of another character string area, for example, a character string area for seven lines is specified in the image, and the second to fourth lines in the character string area are specified. The case of modifying the character string area up to is described. The user first moves the pointer pt1 to a position inside the character string area of the second to fourth lines while pressing the shift key, and clicks the mouse button each time the pointer pt1 is moved. The user then stops pressing the shift key, moves the pointer pt1 to a predetermined position in the character string area of the second line (for example, the upper right corner), and clicks the mouse button. Then, the user moves the pointer pt1 to the position where he / she wants to move the corner of the character string area as it is, and drags it. By performing such an input operation, the character string area extraction unit 15 performs the third and fourth lines based on the contents corrected for the character string area of the second line among the character strings of the seven lines. Modify each string area of.
By modifying a part of the character string area of the plurality of character string areas, for example, the character string area of the character string in which the choreography is not assigned in the image and the choreography kana can be obtained by the character string area extraction unit 15. When the character string area of the assigned character string is mixed, only the character string area of the character string to which the assigned pseudonym is assigned can be modified.

As described above, in the character string area / character rectangle extraction device 1, the character string area extraction unit 15 displays the character string area extraction screen ds2 based on the operation input signal (auxiliary information) input from the operation input unit 12. The character string areas sa1 to sa5 extracted from the image are modified. For example, even if the character string area extracted by the character string area extraction unit 15 includes an area to which a pseudonym is assigned, one character string area (for example, the character string area sa1) can be used by the user. If is corrected, the character string area extraction unit 15 corrects another character string area (for example, the character string areas sa2 to sa7) according to the correction content. Therefore, the user does not need to make corrections for each character string area. Therefore, in the character string area / character rectangle extraction device 1, in addition to the above-mentioned effects, when the character string area is modified, it is possible to save the user the trouble of performing an operation of modifying each of the plurality of character string areas.

(Generation of specific information in the second image)
Hereinafter, a method in which the character string region extraction unit 15 generates specific information for a second image different from the first image will be described with reference to the drawings.
FIG. 23 is a diagram showing an example of a character string area extraction screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment.

As shown in FIG. 23, the character string area extraction unit 15 sets the xy coordinate values of the start point and the end point specified for the image displayed on the character string area extraction screen ds2 to different characters from the character string area extraction screen ds2. Specific information may be generated as xy coordinate values of the start point and the end point of the line consisting of the character string included in the image displayed on the column area extraction screen ds3.
The character string area extraction unit 15 stores, for example, specific information indicating the xy coordinate values of the start point and the end point of the line consisting of the character string included in the image displayed on the character string area extraction screen ds2 in the storage unit 14. When the character string area extraction unit 15 specifies the positions of the start point and the end point of the line consisting of the character string included in the image displayed on the character string area extraction screen ds3, the character string area extraction unit 15 stores the character string area extraction screen in the storage unit 14. Refer to the xy coordinate values of the start point and the end point of the line consisting of the character string included in the image displayed in ds2, respectively. Then, each of the referenced xy coordinate values is set as the xy coordinate value of the start point and the end point of the line consisting of the character string included in the image displayed on the character string area extraction screen ds3.

Alternatively, the character string area extraction unit 15 may use auxiliary information instead of the specific information. For example, the character string area extraction unit 15 stores the xy coordinate values of the start point and the end point designated by the user with respect to the screen displayed on the character string area extraction screen ds2 in the storage unit 14. When the character string area extraction unit 15 specifies the positions of the start point and the end point of the line consisting of the character string included in the image displayed on the character string area extraction screen ds3, the character string area extraction unit 15 is based on the auxiliary information stored in the storage unit 14. Refers to the xy coordinate values of the start point and the end point specified in the above. Then, the referenced xy coordinate value is set as the xy coordinate value of the start point and the end point with respect to the character string included in the image displayed on the character string area extraction screen ds3.
Then, the character string area extraction unit 15 is in the second and subsequent lines based on the xy coordinate values of the start point and the end point of the first line specified by the user in the image displayed on the character string area extraction screen ds2. In the image displayed on the character string area extraction screen ds3 in the same manner as when specifying the positions of the start point and the end point, in the second and subsequent lines based on the xy coordinate values of the start point and the end point of the first line. Identify the location of the start and end points.

In the example of FIG. 23, the display unit 13 has a white square icon at the start points Est11 to Est18 and black at the end points Eed11 to Eed18 on the character string area extraction screen ds3 based on the specific information specified by the character string area extraction unit 15. Display each square icon.

Alternatively, the character string area extraction unit 15 may use auxiliary information instead of the specific information. For example, the character string area extraction unit 15 stores the auxiliary information specified by the user in the image displayed on the character string area extraction screen ds2 in the storage unit 14. When the character string area extraction unit 15 specifies the positions of the start point and the end point of the line consisting of the character string included in the image displayed on the character string area extraction screen ds3, the character string area extraction unit 15 stores the character string area extraction screen in the storage unit 14. Refer to the auxiliary information of the image displayed on ds2. Then, the character string area extraction unit 15 has a specific line consisting of a character string included in the image displayed on the character string area extraction screen ds3, based on the auxiliary information of the image displayed on the character string area extraction screen ds2. For example, the positions of the start point and the end point of the first line) are specified. Then, the character string area extraction unit 15 may specify the positions of the start points and end points of other lines in the image displayed on the character string area extraction screen ds3 based on the positional relationship between the specified start point and end point. good. Alternatively, the character string area extraction unit 15 specifies the positions of the start point and the end point in the image displayed on the character string area extraction screen ds3 based on the auxiliary information and the specific information in the image displayed on the character string area extraction screen ds2. You may try to do it.

As described above, in the character string area / character rectangular extraction device 1 of the modification 2 of the present embodiment, the character string area extraction unit 15 refers to the first image (the image displayed on the character string area extraction screen ds2). The second image (the image displayed on the character string area extraction screen ds3) based on at least one of the auxiliary information based on the operation input performed and the specific information generated for the character string area extraction screen ds2. Generates specific information about a line consisting of the strings contained in. As a result, in the character string area / character rectangular extraction device 1 of the modification 2 of the present embodiment, in addition to the above-mentioned effects, the auxiliary information or specific information acquired by the character string area extraction unit 15 for the first image is obtained. Based on the above, the positions of the start point and the end point of the line consisting of the character string included in the second image can be specified, and the user can omit the operation of designating the start point and the end point for each image.

Since the character string area extraction unit 15 uses the xy coordinate values of the start point and the end point of the first image as the xy coordinate values of the start point and the end point of the second image, the positions of the start point and the end point of the second image are set. It does not always match the actual start and end points of the line. When the first image or the second image is an image obtained by capturing a page of a book, it is considered that the position where the character string is started to be described is often deviated between the first image and the second image. .. In such a case, the user can have the character string area extraction unit 15 modify the specific information described above. The user can correct the positions of the start point and the end point of the second and subsequent lines by, for example, correcting the positions of the start point and the end point of the first line by operating the pointer pt1.

(Extraction of character string area in the second image)
Hereinafter, a method of extracting the character string area in the character string included in the second image based on the information regarding the character string area extracted from the first image by the character string area extraction unit 15 will be described with reference to the drawings.
FIG. 24 is a diagram showing an example of a character string area extraction screen displayed by the display unit 13 of the character string area / character rectangle extraction device 1 according to the embodiment.

As shown in FIG. 24, the character string area extraction unit 15 sets the character string area specified for the image displayed on the character string area extraction screen ds2 to another character string area extraction screen different from the character string area extraction screen ds2. It may be a character string area of the character string included in the image displayed on ds3.
The character string area extraction unit 15 stores, for example, information about each of the character string areas extracted in the image displayed on the character string area extraction screen ds2 in the storage unit 14. The information regarding the character string area includes, for example, information that can specify the character string area, for example, xy coordinate values of the four corners in the area.
When the character string area extraction unit 15 extracts the character string area of the character string included in the image displayed on the character string area extraction screen ds3, the character extracted on the character string area extraction screen ds2 stored in the storage unit 14 Refer to the xy coordinate values of the four corners of each of the column regions (for example, the character string regions sa1 to sa7 in FIG. 20). Then, the character string area extraction unit 15 extracts the area surrounded by each xy coordinate value as the character string areas Esa1 to Esa7 for the character string included in the image displayed on the character string area extraction screen ds3.

As described above, in the character string area / character rectangular extraction device 1 of the modification 2 of the present embodiment, the character string area extraction unit 15 refers to the first image (the image displayed on the character string area extraction screen ds2). Based on the information about the extracted character string area, the character string area of the character string included in the second image (the image displayed on the character string area extraction screen ds3) is extracted. As a result, in the character string area / character rectangular extraction device 1, in addition to the above-mentioned effects, the characters included in the second image based on the character string area extracted by the character string area extraction unit 15 with respect to the first image. The character string area of the column can be extracted, and the time and effort of the user for extracting the character string area for each image can be saved.

Since the character string area extraction unit 15 sets the position of the character string area of the first image as the position of the character string area of the second image, the position of the character string area of the second image is not necessarily the actual character. It does not always match the position of the column area. When the first image or the second image is an image obtained by capturing a page of a book, it is considered that the position of the character string region is often deviated between the first image and the second image. In such a case, the user can have the character string area extraction unit 15 modify the character string area as described above. The user can, for example, correct the position of the character string area on the first line by operating the pointer pt1, and correct the position of the character string area on the second and subsequent lines.

Although the embodiments of the present invention have been described in detail above, the specific configuration is not limited to the above-mentioned ones, and various design changes and the like can be made without departing from the gist of the present invention. be.

In addition, a part or all of the character string area / character rectangle extraction device 2 in the character string area / character rectangle extraction device 1 in the above-described embodiment, the modification 1 of the above-described embodiment, and the modification 2 of the above-mentioned embodiment. May be realized by a computer. In that case, the program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed.

The "computer system" here is a computer system built in the character string area / character rectangle extraction device 1 and the character string area / character rectangle extraction device 2, and includes hardware such as an OS and peripheral devices. It shall include. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, and a storage device such as a hard disk built in a computer system.

Further, a "computer-readable recording medium" is a medium that dynamically holds a program for a short time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In that case, a program may be held for a certain period of time, such as a volatile memory inside a computer system serving as a server or a client. Further, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Further, the character string area / character rectangle extraction device 1 in the above-described embodiment and the character string area / character rectangle extraction device 2 in the modification 1 of the above-described embodiment are realized as an integrated circuit such as an LSI (Large Scale Integration). You may. Each functional block of the character string area / character rectangle extraction device 1 and the character string area / character rectangle extraction device 2 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of making an integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, when an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

1 ... Character string area / character rectangle extractor, 2 ... Character string area / character rectangle extractor, 1
0 ... Control unit, 11 ... Image data acquisition unit, 12 ... Operation input unit, 13 ... Display unit, 14 ... Storage unit, 15 ... Character string area extraction unit, 16 ...・ ・ Character string area joint, 1
7 ... Character rectangle extraction unit, 18 ... Ruby mapping unit, 19 ... Management screen generation unit, 26
・ ・ ・ Language analysis department

Claims (21)

An image data acquisition unit that acquires image data indicating an image containing a character string,
A display unit that displays an image based on the acquired image data,
An operation input unit that accepts operation input from the user,
A character that extracts a character string area that encloses the entire character string included in the line based on the start point and end point of the line consisting of the character string included in the image specified based on the auxiliary information based on the operation input. Column area extractor and
A character string area joining portion that connects the extracted character string area and another character string area adjacent to the character string area,
A character string area / character rectangle extraction device characterized by being equipped with. The character string area extraction unit extracts the character string area based on the range where the connection line connecting the start point and the end point and the character intersect.
The character string area / character rectangle extraction device according to claim 1. The operation input unit receives the operation input based on the operation of the pointer, and receives the operation input.
The character string area joining portion is characterized in that the extracted character string area and another character string area adjacent to the character string area are combined based on the operation of the pointer. Alternatively, the character string area / character rectangle extraction device according to claim 2 . The item according to any one of claims 1 to 3 , wherein the character string area extraction unit modifies the extracted character string area based on auxiliary information based on the operation input. Character string area / character rectangle extractor. The operation input unit receives the operation input based on the operation of the pointer, and receives the operation input.
The character string area / character rectangular extraction device according to claim 4 , wherein the character string area extraction unit corrects the extracted character string area based on the operation of the pointer. The character string area joining portion is characterized in that the extracted character string area and another character string area adjacent to the character string area are combined based on statistical information in the appearance of the character string. The character string area / character rectangle extraction device according to claim 1. A ruby mapping unit that associates the character string with the ruby corresponding to the character string based on the auxiliary information based on the operation input.
The character string area / character rectangle extraction device according to any one of claims 1 to 6 , wherein the character string area / character rectangle extraction device is provided. The character string area according to claim 7 , wherein the display unit displays a ruby correspondence area which is a range in which the associated character string and the ruby corresponding to the character string are enclosed. Character rectangle extractor. The character string area / character according to any one of claims 1 to 8 , further comprising a character rectangle extraction unit for extracting a character rectangle representing a character rectangle of each character constituting the character string. Rectangular extractor. The character rectangle extraction unit identifies an appropriate character cutout position from a plurality of character cutout position candidates based on an evaluation value calculated from character shape information, recognition accuracy in character recognition, and the like. The character string area / character rectangle extracting device according to claim 9 , wherein the character rectangle is extracted. The character string area / character rectangle extraction device according to claim 9 , wherein the character rectangle extraction unit corrects the extracted character rectangle based on auxiliary information based on the operation input. The operation input unit receives the operation input based on the operation of the pointer, and receives the operation input.
The character string area / character rectangle extraction device according to claim 11 , wherein the character rectangle extraction unit corrects the extracted character rectangle based on the operation of the pointer. The display unit is characterized in that line numbers, which are numbers assigned in the order in which character string areas indicating a display target range of character strings included in the line are extracted, are displayed in association with the character string area. The character string area / character rectangle extraction device according to any one of claims 1 to 12 . The character string area / character rectangle extracting device according to claim 13 , wherein the display unit displays the image based on the image data and a list display image which is an image in which the line numbers are listed. .. The character string area extraction unit is based on the start point and the end point specified by the auxiliary information based on the operation input for the line consisting of the character string included in the image, and the other line consisting of the character string included in the image. Generates specific information that identifies the location of the start and end points,
One of claims 1 to 14 , characterized in that the character string area of the character string included in the other line is extracted based on the start point and the end point specified by the specific information. The character string area / character rectangle extraction device described in. The character string area extraction unit corrects the specific information generated for the image based on the auxiliary information based on the operation input for at least one of the start point and the end point specified for the image. The character string area / character rectangle extracting device according to claim 15 , wherein the character string area / character rectangle is extracted. The character string area extraction unit is characterized in that the character string area extracted for the image is modified based on auxiliary information based on the operation input for the character string area extracted for the image. The character string area / character rectangle extracting device according to claim 15 or 16 . The character string region extraction unit is based on at least one of the auxiliary information based on the operation input performed on the first image and the specific information generated on the first image. The character string area / character rectangular extraction device according to any one of claims 15 to 17 , wherein the specific information for a second image different from the first image is generated. The character string area extraction unit is characterized in that it extracts the character string area for a second image different from the first image based on the information about the character string area extracted from the first image. The character string area / character rectangle extracting device according to any one of claims 15 to 18 . It is a method of extracting character string areas and character rectangles by a computer.
The image data acquisition unit acquires image data indicating an image including a character string, and an image data acquisition step.
A display step in which the display unit displays an image based on the acquired image data, and
An operation input step in which the operation input unit receives an operation input from the user,
The character string area extraction unit encloses the entire character string included in the line based on the start point and end point of the line consisting of the character string included in the image specified based on the auxiliary information based on the operation input. The string area extraction step to extract the character string area and
A character string area joining step in which the character string area joining portion joins the extracted character string area and another character string area adjacent to the character string area.
A character string area / character rectangle extraction method characterized by having. On the computer
An image data acquisition step for acquiring image data indicating an image including a character string, and
A display step for displaying an image based on the acquired image data, and
An operation input step that accepts operation input from the user, and
A character that extracts a character string area that encloses the entire character string included in the line based on the start point and end point of the line consisting of the character string included in the image specified based on the auxiliary information based on the operation input. Column area extraction step and
A character string area joining step that joins the extracted character string area and another character string area adjacent to the character string area,
A program to execute.

Images