JP2024078158A - Character extraction method - Google Patents

Abstract

[Problem] To suppress the time required to find a portion of an image containing characters indicating the content of predetermined information. [Solution] A character extraction method causes a computer to execute the following steps: divide an image into a plurality of sections (S13), select from the plurality of sections a section that is most likely to contain characters indicating the content of the predetermined information based on the predetermined information (S17, S19), and extract characters indicating the content of the predetermined information from the selected section (S21, S23). [Selected Figure] Figure 4

Description

The present invention relates to a character extraction method for extracting characters from an image.

Patent document 1 discloses an example of a method for converting an image of text into a text format using optical character recognition technology. Specifically, in this method, a bounding box of a text area is determined from a picture to be identified, and a text area picture is extracted from the picture to be identified based on the bounding box. Next, a bounding box of a text line is determined from the text area picture, and a text line picture is extracted from the text area picture based on the bounding box. Then, text sequence recognition is performed on the text line picture to obtain a recognition result.

When obtaining the classification results, a lightweight text sequence classification model is used. This classification model is a trained model that has been subjected to machine learning.

JP 2021-197190 A

With the above method, it may take a long time to find the part of the picture to be identified that contains the characters indicating the content of the desired information as a text line picture.

The character extraction method for solving the above problem is a method for extracting characters indicating the content of specified information from an image shown by image data. The character extraction method causes a computer to execute the following steps: divide the image into a plurality of sections; select, based on the specified information, from among the plurality of sections, the section that is most likely to contain characters indicating the content of the specified information; and extract the characters indicating the content of the specified information from the selected section.

In the above character extraction method, an image is divided into a plurality of sections, and from the plurality of sections, a section that is most likely to contain characters related to predetermined information is selected. At this time, the section is selected based on the predetermined information. Characters indicating the content of the predetermined text information are then extracted from the section. Therefore, the above character extraction method can prevent the time required to find a portion of an image containing characters indicating the content of the predetermined information from being lengthened.

In addition, when selecting the section that is most likely to contain characters indicating the content of the specified information from among multiple sections, a trained model that has been subjected to machine learning may be used.

FIG. 1 is a block diagram showing an outline of an extraction device for implementing a character extraction method according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing an image of a design drawing. FIG. 3 is a schematic diagram showing an image of a report. FIG. 4 is a flowchart showing a character extraction method according to the embodiment.

An embodiment of a character extraction method will be described below with reference to FIGS.
The character extraction method of the present embodiment is a method for extracting characters written in an image represented by image data from the image. More specifically, the character extraction method is a method for extracting characters indicating the content of predetermined information from an image.

<Extraction device>
With reference to FIG. 1, an extraction device 10 for implementing a character extraction method will be described.
The extraction device 10 includes a user interface 11 and a computer 20 .

The user interface 11 has an operation unit 12 operated by the worker, and a display unit 13 that displays the contents of information received from the computer 20. The operation unit 12 has at least one of physical buttons or switches, and buttons displayed on a display screen having a touch panel. The user interface 11 outputs requests to the computer 20 in response to the operation of the operation unit 12 by the worker.

The computer 20 is, for example, an electronic control device. In this case, the computer 20 has a CPU 21, a first storage device 22, and a second storage device 23. The first storage device 22 stores a control program CP executed by the CPU 21. When the CPU 21 executes the control program CP, the computer 20 executes the series of processes shown in FIG. 4. That is, the computer 20 divides an image represented by image data into a plurality of sections. The computer 20 ranks the plurality of sections in order of the likelihood that characters indicating the contents of the specified information are written therein. The computer 20 then performs a process of extracting characters indicating the contents of the specified information from the section with the highest rank among the plurality of sections using OCR. Note that OCR stands for optical character recognition.

The second storage device 23 stores a learned model LM and a plurality of texts TX.
The trained model LM is a learning model that has been subjected to machine learning to rank a plurality of sections into which an image is divided in order of the likelihood that characters indicating the contents of predetermined information are written. For example, one example of the trained model LM is an approximator of a multidimensional polynomial. For example, the trained model LM is configured by a fully connected forward propagation type neural network with one intermediate layer.

When the trained model LM receives specified information, image data, and the type of document as input variables, it outputs, as output variables, values indicating the likelihood that characters indicating the content of the specified information are written in multiple sections. If the image has four sections, the trained model LM outputs output variables for each of the four sections.

For example, the learned model LM was generated by applying machine learning to a learning model as shown below.
FIG. 2 illustrates an image IMGA of a design drawing. In the example illustrated in FIG. 2, the image IMGA is divided into four sections SC1, SC2, SC3, and SC4. In the image IMGA of a design drawing, the document type is "design drawing", and an example of the specified information is general tolerance. In the image IMGA illustrated in FIG. 2, "JISG100", which is a character indicating the content of the general tolerance (specified information), is present in the section SC4. Therefore, when the general tolerance is used as the specified information, the design drawing is used as the document type, and the image data of the image IMGA are used as input variables of the learning model, machine learning is performed on the learning model so that the output variable of the section SC4 is larger than the output variables of the other sections SC1, SC2, and SC3.

In FIG. 3, an image IMGB of a report is shown. The report shown in FIG. 3 has six pages. Therefore, in the example shown in FIG. 3, the image IMGB is divided into six sections SC1, SC2, SC3, SC4, SC5, and SC6. In the image IMGB of a report, the document type is "report", and an example of the specified information is the publication date. In the image IMGB shown in FIG. 3, characters indicating the contents of the publication date (specified information) are present in section SC1. Therefore, when the publication date is the specified information, the report is the document type, and the image data of the image IMGB are input variables of the learning model, machine learning is performed on the learning model so that the output variable of section SC1 is larger than the output variables of the other sections SC2, SC3, SC4, SC5, and SC6.

Text TX is used to identify characters extracted from an image. The multiple text TX include text TX for design drawings and text TX for reports. For example, the text TX for general tolerances in design drawings is "JIS G100."

<Character extraction method>
A flowchart showing the execution procedure of a plurality of processes constituting the character extraction method is shown in Fig. 4. When the computer 20 acquires image data, the computer 20 starts the series of processes shown in Fig. 4.

In step S11, the computer 20 acquires the document type and predetermined information in the image of the acquired image data. The computer 20 acquires the document type and predetermined information based on, for example, an input operation of the operation unit 12 of the user interface 11 by the worker.

In step S13, computer 20 divides the image of the image data into a number of sections. At this time, computer 20 may divide the image into a number according to the type of document acquired in step S11. For example, when the image is an image IMGA of a design drawing as shown in FIG. 2, computer 20 divides image IMGA into four sections SC1 to SC4. Also, when the image is an image IMGB of a report as shown in FIG. 3, computer 20 divides image IMGB into the same number of sections as the number of pages.

In step S15, the computer 20 inputs the document type, the specified information, and the image data as input variables to the trained model LM. In the following step S17, the computer 20 ranks the multiple sections in order of likelihood that the section contains characters indicating the contents of the specified information, based on the output variables of the trained model LM. For example, if an image is divided into four sections SC1 to SC4, the output variable of section SC4 is the highest, the output variable of section SC2 is the second highest, the output variable of section SC3 is the third highest, and the output variable of section SC1 is the fourth highest. In this case, the computer 20 ranks section SC1 first, section SC2 second, section SC3 third, and section SC1 fourth. The computer 20 then selects the section ranked first from the multiple sections as the section most likely to contain characters indicating the contents of the specified information.

In step S19, the computer 20 sets the number N to 1. Then, the computer 20 shifts the process to step S21.
In step S21, the computer 20 extracts characters from the Nth-ranked section by OCR among the multiple sections. If the number N is set to 1, the computer 20 performs a process of extracting characters from the first section, i.e., the section that is most likely to contain characters indicating the content of the specified information. When the computer 20 completes the process, it transitions to step S23.

In step S23, the computer 20 calculates the similarity X between the characters extracted in step S21 and the characters of the text TX. Specifically, the computer 20 reads out the text TX corresponding to the specified information and document type from the second storage device 23. Then, the computer 20 calculates the similarity X by comparing the characters extracted in step S21 with the characters of the text TX read out from the second storage device 23.

Here, an example will be described in which the characters extracted in step S21 are "JISG106" and the characters in text TX used for comparison are "JISG100". In this case, among the characters extracted in step S21, the first "J", the second "I", the third "S", the fourth "G", the fifth "1" and the sixth "0" match the text TX. On the other hand, the seventh "6" does not match the text TX. Therefore, the computer 20 can obtain that the number of characters that match the text TX is six. Then, the computer 20 calculates the similarity X using, for example, the following relational expression (F1). In the relational expression (F1), "Z1" is the number of characters that match the text TX, "Y1" is the number of characters that could be extracted by OCR, and "Y2" is the number of characters in the text TX.

X = (2 x Z1) / (Y1 + Y2) (F1)
In the case described here, the number of characters extracted by OCR is 7, and the number of characters in the text TX is 7. Therefore, the computer 20 assigns 7 to "Y1" and assigns 7 to "Y2". Then, the computer 20 assigns 6 to "Z1", thereby calculating the similarity X as "0.857".

If the characters extracted in step S21 are "JISG100," the computer 20 assigns 7 to "Z1" in the relational expression (F1), and calculates the similarity X to be "1."

After calculating the similarity X, the computer 20 proceeds to step S25. In step S25, the computer 20 determines whether the similarity X calculated in step S23 is greater than the judgment value Xth. The judgment value Xth is a criterion for determining whether a character indicating the content of the specified information is written in the image. When extracting characters using OCR, there is a possibility of extraction errors. On the other hand, if the similarity X is less than 0.5, it is difficult to say that the character extracted by OCR and the text TX match. Therefore, it is preferable to set a value greater than 0.5 and less than 1 as the judgment value Xth. Then, if the similarity X is greater than the judgment value Xth, it is considered that a character indicating the content of the specified information is written in the image. On the other hand, if the similarity X is equal to or less than the judgment value Xth, it is considered that a character indicating the content of the specified information is not written in the image. If the similarity X is greater than the judgment value Xth (S25: YES), the computer 20 proceeds to step S27. On the other hand, if the similarity X is equal to or less than the judgment value Xth (S25: NO), the computer 20 transitions to step S31.

In step S27, the computer 20 causes the display unit 13 of the user interface 11 to display that the image contains characters indicating the contents of the specified information. The computer 20 also writes the "characters indicating the contents of the specified information" contained in the image to a specified work file. After that, the computer 20 ends the series of processes.

In step S31, the computer 20 determines whether the process of extracting characters has been performed for all sections. For example, if the number N matches the number of sections in the image, the computer 20 determines that the process of extracting characters has been performed for all sections (S31: YES). The computer 20 then transitions to step S35. On the other hand, if the number N is smaller than the number of sections in the image, the computer 20 determines that there is a section among the multiple sections for which the process of extracting characters has not yet been performed (S31: NO). The computer 20 then transitions to step S33.

In step S33, the computer 20 increments the number N by 1. After updating the number N in this manner, the computer 20 transitions to step S21.

In step S35, the computer 20 notifies the worker that the image does not include characters indicating the content of the specified information. In this case, the computer 20 causes the display unit 13 of the user interface 11 to display a message indicating that the image does not include characters indicating the content of the specified information. The computer 20 then ends the series of processes.

<Action and Effects>
When image data is input to computer 20, the series of processes shown in Fig. 4 are executed. When computer 20 acquires the type of document and the specified information of the image represented by the image data, computer 20 processes the image and divides the image into a number of sections based on the type of document. For example, if the type of document is a design drawing, the image is divided into four sections SC1 to SC4. Then, from among the multiple sections, the section that is most likely to contain characters indicating the content of the specified information is selected. Then, the characters indicating the content of the specified information are extracted from the selected section.

In other words, in this embodiment, based on the specified information, the part of the image that is most likely to contain characters indicating the content of the specified information is searched for, and the process of extracting the characters is performed on that part rather than on other parts. This makes it possible to prevent the time required to search for the part of the image containing characters indicating the content of the specified information from being long.

In this embodiment, the following effects can be further obtained.
(1) In this embodiment, the learned model LM is used to rank the multiple sections into which an image is divided in order of the likelihood that characters indicating the content of the specified information are written. There are cases where the first-ranked section does not have characters indicating the content of the specified information. In this case, a process is executed to determine whether or not the second-ranked section has characters indicating the content of the specified information. Then, if the second-ranked section does not have characters indicating the content of the specified information, a process is executed to determine whether or not the third-ranked section has characters indicating the content of the specified information. Therefore, if any of the images has characters indicating the content of the specified information, the characters can be extracted.

(2) The input variables of the trained model LM include the document type in addition to the specified information and image data. Increasing the number of input variables in this way can improve the accuracy of ranking multiple sections into which an image is divided in order of the likelihood that they contain characters indicating the content of the specified information.

<Example of change>
The above embodiment can be modified as follows: The above embodiment and the following modifications can be combined with each other to the extent that no technical contradiction occurs.

When extracting characters from an image using OCR, misreading of characters may occur. For example, the following characters are prone to being misread:
The capital letter "O" and the number "0 (zero)."

The capital letter "I" and the number "1".
The capital letter "Z" and the lowercase letter "z".
The capital letter "G" and the number "6".

In this way, it is a good idea to register characters that are prone to misreading as confirmation characters in advance. For example, if the character actually written in the image is "I (capital letter)," OCR may extract "I (capital letter)" or "1 (number one)." Therefore, if the character extracted by OCR is the confirmation character and the character in text TX is "I (capital letter)," it is good to determine that the extracted character matches the character in text TX regardless of whether the character extracted by OCR is "I" or "1." This makes it possible to prevent the similarity X from being calculated to be low due to the accuracy of character extraction by OCR.

- The input variables of the learned model LM do not need to include the document type as long as they include specified information and image data.
The neural network is not limited to a feedforward network with one intermediate layer. For example, the neural network may be a network with two or more intermediate layers, or may be a convolutional neural network or a recurrent neural network.

The trained model by machine learning does not have to be a neural network. For example, a support vector machine may be used as the trained model.
When ranking multiple sections into which an image is divided in order of the likelihood that characters indicating the contents of predetermined information are written, it is not necessary to use the learned model LM. For example, multiple sections may be ranked based on a rule base created in advance.

The computer 20 is not limited to having a CPU and ROM and executing software processing. In other words, the computer 20 may have any of the following configurations (a) to (c).

(a) Computer 20 includes one or more processors that perform various processes according to a computer program. The processor includes a CPU and memory such as RAM and ROM. The memory stores program code or instructions configured to cause the CPU to perform processes. Memory, i.e., computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.

(b) The computer 20 is equipped with one or more dedicated hardware circuits that execute various processes. Examples of dedicated hardware circuits include application specific integrated circuits, i.e., ASICs or FPGAs. Note that ASIC is an abbreviation for "Application Specific Integrated Circuit" and FPGA is an abbreviation for "Field Programmable Gate Array."

(c) The computer 20 includes a processor that executes some of the various processes according to a computer program, and a dedicated hardware circuit that executes the remaining processes.

Note that the expression "at least one" used in this specification means "one or more" of the desired options. As an example, the expression "at least one" used in this specification means "only one option" or "both of two options" if the number of options is two. As another example, the expression "at least one" used in this specification means "only one option" or "any combination of two or more options" if the number of options is three or more.

20...Computer, 21...CPU, 23...Second storage device, IMGA, IMGB...Images, LM...Trained model, SC1 to SC6...Partitions.

Claims (4)

A character extraction method for extracting characters indicating content of predetermined information from an image represented by image data, comprising the steps of:
Segmenting the image into a plurality of sections;
selecting, from among the plurality of sections, the section that is most likely to contain characters indicating the content of the predetermined information based on the predetermined information;
and extracting characters indicating the content of the predetermined information from the selected section. When selecting the section having the highest possibility of containing characters indicating the content of the predetermined information from among the plurality of sections, ranking the plurality of sections in order of the possibility of containing characters indicating the content of the predetermined information based on the predetermined information, and causing the computer to select the section having the highest ranking;
selecting the section with the second highest rank from among the plurality of sections when characters indicating the content of the predetermined information cannot be extracted from the section with the highest rank;
The character extraction method according to claim 1 , further comprising the step of: extracting characters indicating a content of the predetermined information from the section having the second highest rank. The computer includes a storage device that stores a trained model that has been subjected to machine learning, and the trained model uses the predetermined information and the image data as input variables, and outputs, as an output variable, a value indicating the degree of likelihood that characters indicating the content of the predetermined information are written in the plurality of sections into which the image is divided;
The character extraction method according to claim 2, wherein when ranking the plurality of sections, the computer ranks the plurality of sections based on output variables of the trained model when the specified information and the image data are input as input variables to the trained model. The input variables of the trained model further include the type of document depicted in the image;
The character extraction method according to claim 3, wherein when ranking the multiple sections, the computer ranks the multiple sections based on output variables of the trained model when the specified information, the image data, and the document type are input to the trained model as input variables.

Images