JP2021015490A - Image processing device and method for improving recognition accuracy - Google Patents

Abstract

To effectively improve accuracy related to pattern recognition of an image processing device through machine learning.SOLUTION: An image processing device includes a pattern recognition part for recognizing an image related to image processing or a user performing operation related to the image processing, a recognition evaluation part for performing evaluation about the correctness of a recognition result or receiving evaluation from the user, a recognition complement part for presenting the whole or a part of a recognition object for determining that recognition accuracy is not enough on the basis of evaluation, and acquiring an input of the user about the correctness, and a machine learning part for performing learning related to recognition about the whole or a part of the recognition object by using the acquired input. The recognition complement part presents the recognition object during a waiting period when the user waits for a response to an end or operation of the image processing.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing apparatus having a function of recognizing an image related to image processing or a user who performs an operation related to the image processing, and a method for improving recognition accuracy related to the recognition function.

As a technique related to character recognition of document images, there is a technique called OCR (optical character recognition). Further, as a method of improving the recognition rate, it is known to use a user dictionary in addition to a dictionary prepared in advance, and a character recognition device that learns the recognition result when the recognition result is modified by the user (for example,). See Patent Document 1). However, Patent Document 1 does not specifically describe what kind of timing or method is applied to accept corrections from users.
On the other hand, for users who use the image forming device installed in stores such as convenience stores, the users who answered the questionnaire by using the waiting time of the image forming device for advertising and sales promotion A mechanism for reducing printing charges has been proposed (see, for example, Patent Document 2).

JP-A-2009-193387 Japanese Unexamined Patent Publication No. 2016-143110

An image processing device having a function of recognizing characters in a scanned original image is known. Further, an image processing device that authenticates a user and accepts an operation of the authenticated user is known. Various methods can be applied to user authentication, but in recent years, in addition to the conventional methods using a personal identification number or a card, an authentication method applying a pattern recognition technology to, for example, an image or a voice has become widespread. Character recognition and voice recognition are technically in the category of pattern recognition.
The progress of pattern recognition technology in recent years is remarkable, and the improvement of recognition accuracy is remarkable. However, it cannot be said that there is no misrecognition. In addition, the correct answer may not be unique in the first place. For example, in handwritten character recognition, the character (correct answer) intended by the user may differ even if the pattern is substantially the same, depending on the habit of the user who handwritten the character and the situation when handwriting.
In order to improve the recognition accuracy, it is considered effective to perform machine learning in order to have a correct answer according to the usage situation (for example, for each user). However, if the learning method is such that the user finds it bothersome, the learning function will not be used, which may not lead to improvement in recognition accuracy.
The present invention has been made in consideration of the above circumstances, and provides a method capable of effectively improving the accuracy related to pattern recognition of an image processing apparatus through machine learning.

The present invention includes a pattern recognition unit that recognizes an image related to image processing or a user who performs an operation related to the image processing, a recognition evaluation unit that evaluates whether the recognition result is correct or incorrect, or receives an evaluation from the user. A recognition complementing unit that presents all or part of the recognition target that is judged not to have sufficient recognition accuracy based on the evaluation and requests the input of the user regarding correctness, and all or one of the recognition targets using the obtained input. The recognition complementing unit includes a machine learning unit that performs learning related to the recognition of the unit, and the recognition complementing unit presents the recognition target during a waiting period in which the user waits for the end of the image processing or a response to the operation. Provide the device.

Further, from different viewpoints, the present invention includes a step of recognizing an image related to image processing or a user who performs an operation related to the image processing, and a step of evaluating the correctness of the recognition result or receiving an evaluation from the user. A recognition complementing step that presents all or part of the recognition target that is judged not to have sufficient recognition accuracy based on the evaluation and asks for the user's input related to correctness, and all or part of the recognition target using the obtained input. The computer executes a process including a step of learning about the recognition for a part of the recognition, and the recognition complement step is executed during a waiting period in which the user waits for the end of the image processing or a response to the operation. Provide a method for improving accuracy.

In the image processing apparatus according to the present invention, the recognition complementing unit presents all or a part of the recognition target determined to be insufficient recognition accuracy during the waiting period in which the user waits for the end of the image processing or the response to the operation. Since the input of the user related to the correctness is requested, the accuracy related to the pattern recognition of the image processing device can be effectively improved through machine learning without causing the user to feel annoyed.

It is a block diagram which shows the structure of the image processing apparatus in this embodiment. It is a perspective view which shows the appearance of the digital multifunction device shown in FIG. In this embodiment, it is a flowchart which shows an example of the process which a control part executes with respect to learning of a recognition target. FIG. 1 is a first explanatory diagram showing an example in which an input relating to correctness of recognition is requested during a waiting period for the end of copying in the first embodiment. FIG. 2 is a second explanatory diagram showing an example in which an input relating to correctness of recognition is requested during a waiting period for the end of copying in the first embodiment. FIG. 3 is a third explanatory diagram showing an example in which an input relating to correctness of recognition is requested during a waiting period for the end of copying in the first embodiment. FIG. 4 is a fourth explanatory diagram showing an example in which an input relating to correctness of recognition is requested during a waiting period for the end of copying in the first embodiment. FIG. 5 is a fifth explanatory diagram showing an example in which the input relating to the correctness of recognition is requested during the waiting period for the end of copying in the first embodiment. FIG. 6 is a sixth explanatory diagram showing an example in which the input relating to the correctness of recognition is requested during the waiting period for the end of copying in the first embodiment. It is 1st explanatory drawing which shows the example of requesting the input which concerns on the correctness of recognition during the waiting period of the copy completion in Embodiment 2. FIG. 2 is a second explanatory diagram showing an example in which input relating to correctness of recognition is requested during a waiting period for the end of copying in the second embodiment. FIG. 3 is a third explanatory diagram showing an example in which input relating to correctness of recognition is requested during a waiting period for the end of copying in the second embodiment. FIG. 3 is a first explanatory diagram showing an example in which an input relating to correctness of recognition is requested during a waiting period for the end of copying in the third embodiment. FIG. 2 is a second explanatory diagram showing an example in which an input relating to correctness of recognition is requested during a waiting period for the end of copying in the third embodiment. FIG. 3 is a third explanatory diagram showing an example in which an input relating to the correctness of recognition is requested during the waiting period for the end of copying in the third embodiment.

Hereinafter, the present invention will be described in more detail with reference to the drawings. It should be noted that the following description is exemplary in all respects and should not be construed as limiting the invention.
(Embodiment 1)
<< Configuration of image processing device >>
FIG. 1 is a block diagram showing a configuration of a digital multifunction device, which is one aspect of the image processing device according to this embodiment. FIG. 2 is a perspective view showing the appearance of the digital multifunction device shown in FIG.
In this embodiment, a digital multifunction device is mentioned as an example of an image processing device, but the present invention is not limited to this. For example, it may be a scanner device, a copy device, a facsimile device, a printer device, or the like, and is not limited to these as long as it is a device related to image processing.

As shown in FIG. 1, the image processing device 100 includes an operation unit 10, a control unit 11, a display unit 12, a printing unit 13, a communication interface circuit 14, a scanner unit 15, and an image data generation circuit 16. Further, it is connected to an external information processing device 20 via a communication interface circuit 14. In this embodiment, the information processing device 20 is a personal computer that stores the original image read by the image processing device 100 and the user processes the stored original image. However, the information processing device 20 is not limited to a personal computer, and may be, for example, a smartphone, a file server, or the like.

Further, as shown in FIG. 2, the image processing device 100 includes a paper feed tray 17a, discharge trays 18a, 18b and 18c, and a manual feed tray 17b.
The control unit 11 and the printing unit 13, and the control unit 11 and the scanner unit 15 are connected by a bus so that they can communicate with each other.
The control unit 11 controls the operation of each unit of the image processing device 100 shown in FIG. Specifically, it is composed of hardware resources such as a memory, an input / output interface circuit, and a timer circuit, centering on a CPU or MPU (hereinafter, both are collectively referred to as a CPU).
At least a part of the ROM included in the control unit 11 may be a rewritable non-volatile memory. The CPU included in the control unit 11 reads the control program stored in the ROM and appropriately expands it into the RAM. Then, the process according to the control program expanded in the RAM is executed.

The CPU causes the display unit 12 to display a display related to the user interface according to the contents of the control program stored in the ROM. Then, the operation input performed by the user to the operation unit 10 is accepted. Further, the CPU controls the hardware of the image processing device 100 according to the control program to realize functions such as printing processing.
The control unit 11 includes a pattern recognition unit 11a, a recognition evaluation unit 11b, a recognition complementing unit 11c, and a machine learning unit 11d. When the CPU executes the control program stored in the ROM, the functions of the pattern recognition unit 11a, the recognition evaluation unit 11b, the recognition complementing unit 11c, and the machine learning unit 11d are realized.
The pattern recognition unit 11a performs image recognition on the image read by the scanner unit 15. However, the present invention is not limited to this, and instead of or in addition to the image read by the scanner unit 15, for example, an image of a user taken by a camera (not shown in FIG. 1) may be recognized, or a microphone (not shown in FIG. 1) may be recognized. You may recognize the user's voice picked up by.

The recognition evaluation unit 11b evaluates the correctness of recognition of the pattern recognition such as image recognition and voice recognition performed by the pattern recognition unit 11a. Alternatively, an evaluation by the user regarding the correctness of the recognition is obtained.
In the evaluation of correctness of recognition, the recognition objects that have been pattern-recognized in the past are classified into the same group, and the fluctuation of the correct answer input to the recognition objects belonging to the same group is evaluated. ..
For example, it is assumed that the pattern recognition unit 11a recognizes the characters of the document image read by the scanner unit 15. Further, when the user determines that the result of character recognition by the pattern recognition unit 11a is erroneous recognition, the user shall have a function of correcting the erroneous recognition portion. The user may operate the display unit 12 to correct the erroneous recognition, or may operate the information processing device 20 capable of communicating via the communication interface circuit 14 to correct the erroneous recognition.

The recognition evaluation unit 11b stores the recognition target for which character recognition has been performed and the recognition result. At that time, the same or similar recognition targets are grouped and stored. If the user corrects the misrecognition, it is stored together with the corrected content. Then, the recognition target belonging to the group having a large number of corrections with respect to the number of character recognitions is given a low evaluation as having a high correction rate. On the contrary, the recognition target belonging to the group in which the number of corrections is small for the number of recognitions is given a high evaluation. For example, in this way, it is possible to evaluate the correctness of the recognition target belonging to each group.
Alternatively, the recognition evaluation unit 11b may obtain an evaluation related to the correctness of recognition from the user instead of evaluating the correctness of recognition by itself. As a method of acquiring the evaluation, for example, it is conceivable to let the user select a recognition target that is considered to have many false recognitions.

As a specific example, the read image and the character recognition result are displayed side by side on the display unit 12 or the information processing device 20. If there is a misrecognition, the user can correct the part and edit the recognition result. The recognition evaluation unit 11b stores the recognition target modified by the user. Then, for example, during the waiting period of the user, the recognition targets that have been modified in the past are displayed in a list on the display unit 12 or the information processing device 20 as feedback candidates. Alternatively, feedback candidates are displayed one by one. Then, from the displayed candidates, the recognition target that the user thinks that there are many false recognitions is selected as the feedback input target.
Here, the waiting period is a period in which the user waits for the image processing (hereinafter, also referred to as a job) executed by the image processing device 100, the response of communication, or the like to be completed. The waiting period also includes the period until the image processing apparatus 100 returns from the power saving mode in response to the image processing job start instruction. Examples of job types include jobs such as copy, scanner, printer, and fax. These jobs involve scanning documents and printing data.
During the waiting period of the user, the recognition complementing unit 11c presents to the user a recognition target that is determined to have insufficient recognition accuracy among the recognition targets that have already been recognized. Then, the user is requested to input the correctness of recognition. Here, the recognition target presented is not limited to the original image related to the job being executed by the image processing device 100, the recognition of the user, and the like. For example, it includes a recognition target related to a job executed in the past.
The machine learning unit 11d performs machine learning related to pattern recognition executed by the pattern recognition unit 11a.

The operation unit 10 is provided on the housing of the image processing device 100, and is composed of a plurality of operation buttons for receiving user operations, a touch panel arranged on the display surface of the display unit 12, and the like. The control unit 11 recognizes a signal indicating an input operation to the operation unit 10.

The display unit 12 is composed of, for example, a liquid crystal display device. The display unit 12 can display various information, images, and the like based on, for example, the input received by the operation unit 10. The control unit 11 generates and updates the contents to be displayed on the display unit 12. Along with this, the display unit 12 displays various information and images.
A scanner unit 15 for reading an image of a document is connected to the control unit 11.
The scanner unit 15 executes image reading processing in copy, fax, and scanner jobs under the control of the control unit 11. That is, the original image is read and converted into an image signal.
The image data generation circuit 16 generates image data based on the image signal output by the scanner unit 15.

The paper feed tray 17a is composed of a plurality of trays that individually store various sizes of paper.
The manual feed tray 17b is a tray capable of feeding various sizes and types of paper.
Under the control of the control unit 11, the paper feed mechanism (not shown in FIGS. 1 and 2) feeds the paper in the designated paper feed tray into the printing apparatus and conveys it to the printing unit 13.
Under the control of the control unit 11, the printing unit 13 prints the designated image data on the paper fed from the paper feed tray 17a or the manual feed tray 17b.

The paper ejection mechanism (not shown in FIGS. 1 and 2) ejects the paper printed by the printing unit 13 to any of the ejection trays 18a, 18b, and 18c.
The communication interface circuit 14 is an interface for communicating data with an external device via a network. In this embodiment, the image processing device 100 communicates with the information processing device 20 connected via the network. The information processing device 20 stores the image data read by the scanner unit 15 and generated by the image data generation circuit 16. Further, the image data stored in the information processing apparatus 20 is provided for the printing unit 13 to print.

≪Machine learning related to pattern recognition≫
Next, the process executed by the control unit regarding the learning of the recognition target will be described.
FIG. 3 is a flowchart showing an example of processing executed by the control unit regarding learning of the recognition target in this embodiment. 4 to 9 are explanatory views showing an example in which input relating to the correctness of recognition is requested during the waiting period for the end of copying in this embodiment.
Hereinafter, the processing of the control unit will be described with reference to FIGS. 4 to 9 according to the flowchart of FIG.

As shown in FIG. 3, the control unit 11 determines whether or not the waiting period of the user has arrived. (Step S11)
In this embodiment, the waiting time of the user is a period in which the user who has come to the side of the image processing device 100 and started the copy job waits for the end of the copy job.
That is, the user shall instruct the start of the copy job after setting the copy using the operation unit 10.

FIG. 4 is an operation screen of the copy job displayed on the display unit 12, and shows a state in which the user operates the [Start] key with the finger 30 to start the copy job.
At the left end of the operation screen shown in FIG. 4, a setting key 31 for performing various settings related to the copy job and displaying the setting status is arranged. Further, a copy number area 32 in which the number of copies is set and the set number of copies is displayed is arranged in the central portion.
Below that, a machine illustration 33 is arranged in which the presence / absence of a document and the size of paper loaded in each tray are displayed, and when touched, the paper setting screen of each tray is displayed. The [Job Status] key 34 is arranged in the upper right corner of the operation screen, and when touched, the jobs being executed or waiting to be executed are displayed.

A [reset] key 35 and a [start] key 36 are arranged below the [job status] key 34. When the [Reset] key 35 is touched, all the settings (including the number of copies) related to the copy job are reset. When the [Start] key 36 is touched, the execution of the set copy job is started.
The control unit 11 executes the processing of the response to these displays and operations.

As shown in FIG. 4, when the control unit 11 recognizes that the [start] key 36 has been touched, the control unit 11 replaces the [start] key 36 with the [copy stop] key 37 in response. When the [Cancel Copy] key 37 is touched, the control unit 11 cancels the copy job being executed.
The control unit 11 that has started the copy job executes the copy job by causing the scanner unit 15 to read the document and the printing unit 13 to print the image of the scanned document. The period from the reading of the original to the end of printing is a waiting period for the user to wait for the end of the copy job (Yes in step S11). When the operation related to the copy setting is completed, the user waits for the end of the started copy job near the image processing device 100.

When the waiting period is reached, the control unit 11 as the recognition complementing unit 11c causes the display unit 12 to display the feedback input screen related to the pattern recognition (step S13).
FIG. 6 shows an example of the screen displayed in this case. Here, the word selected by the recognition complementing unit 11c from the image of the manuscript whose characters have been recognized in the past is displayed. That is, in the embodiment shown in FIG. 6, the recognition complementing unit 11c displays some characters in the image of the document to be recognized. In the present embodiment, the character to be recognized is not a handwritten character but a printed character.
The recognition complementing unit 11c selects a plurality of words (9 in the example shown in FIG. 6) as candidates for the correct recognition result for the displayed character, "Please select the same character displayed in the frame from the bottom." Present to the user with the message. Then, it waits for the input from the user (step S15).

The control unit 11 confirms whether the waiting period has ended (step S19) or the timeout (step S23) until the feedback input is received from the user (No in step S17).
When the waiting period ends while waiting for input (No in step S19), that is, when the copy job ends while waiting for input, the control unit 11 as the recognition complementing unit 11c waits for input as shown in FIG. The screen is closed (step S21) to end the process. Then, the control unit 11 causes the display unit 12 to display a standby screen waiting for the setting of the next copy job.

On the other hand, if a timeout arrives while waiting for feedback input (Yes in step S23), that is, if there is no input from the user within a predetermined period, the control unit 11 as the recognition complementing unit 11c will perform the above-mentioned step S13. The character displayed in is switched to the recognition target for receiving the next feedback (step S25). For example, when the time-out occurs while the screen shown in FIG. 6 is displayed, the recognition complementing unit 11c switches to the display of the next character recognition target shown in FIG. Then, the input related to the user's feedback is requested for the newly displayed character to be recognized (step S13).

The recognition evaluation unit 11b determines, for example, the characters for which input is requested for feedback as follows. The recognition evaluation unit 11b stores the recognition target for which character recognition has been performed, and groups and stores similar recognition targets at that time. The user's corrections are stored together with the corrections. As a specific method of correction, the read image and the result of character recognition are displayed side by side on the display unit 12, and if there is an erroneous recognition, the user can correct the part and edit the recognition result.
The recognition evaluation unit 11b gives a low evaluation to the recognition target belonging to the group having a large number of corrections with respect to the number of character recognitions, assuming that the correction rate is high. On the contrary, the recognition target belonging to the group with a small number of corrections for the number of recognitions is given a high evaluation.
Then, when the waiting time arrives, the display unit 12 is displayed in order from the one with the lowest evaluation. If the user inputs feedback and obtains the correct answer for the recognition target, the evaluation of the recognition target is given as a perfect score. In this way, it is possible to prevent repeated request for feedback on the recognition target in which the correct answer is input.

Another aspect of determining the character for which feedback is requested is described. The recognition evaluation unit 11b stores the recognition target modified by the user. Then, when the waiting period of the user arrives, the recognition targets modified by the user in the past are displayed in a list on the screen of the display unit 12 or the information processing device 20, or are sequentially displayed. Then, the user selects a recognition target that the user thinks that there are many false recognitions from among them, and requests input related to feedback for the selected one.
Hereinafter, when input from the user is made in the loop of steps S15, S17, S19 and S23 or the loop of steps S13, S15, S17, S19, S23 and S25 waiting for the user's feedback input (Yes in step S17). The processing will be described.

For example, when the user selects one of the candidates displayed on the screen of FIG. 6 (see FIG. 7) and confirms the recognition result (see FIG. 8), the control unit 11 acts as the machine learning unit 11d. Learn that the correct answer for the displayed character is the selected candidate.
However, in this embodiment, the selected candidate is not unconditionally accepted as the correct answer, but it is determined whether or not the input is valid (step S27).
Of course, there may be a mode in which learning is performed for all inputs without determining whether or not the inputs are valid. However, in that case, learning may be affected by erroneous input by the user. Preferably, as shown in step S27, the control unit 11 determines whether or not the input from the user is valid, and only the valid input is the correct answer.

Whether or not the input is valid can be determined, for example, as follows. For example, the recognition target to be learned is presented to the user as a set with an image (test image) in which the correct answer of the recognition result is known in advance. Only when the input from the user for the test image matches the correct answer, it is determined that the candidate selected for the other recognition target is the recognition result of the correct answer.
Alternatively, seek feedback on the same recognition target at different times or from different users. Then, when the rate of obtaining the same feedback exceeds a predetermined standard, it is determined that a valid input related to the recognition target has been obtained.
On the contrary, when the ratio of obtaining the same feedback for a certain recognition target is less than or equal to the above standard, it is judged that the obtained input is not valid, and learning using the accepted input is not performed. However, the ratio is updated and stored based on the received input to prepare for the next input.

The recognition complementing unit 11c shall determine the recognition target for which focus is sought for feedback based on the above ratio. That is, it is assumed that the input related to the feedback is requested more frequently for the recognition target having the ratio of the reference value or less and the smaller value. In this way, the recognition target having a small ratio is required to input feedback more frequently than the recognition target having a large ratio, but learning is not performed on the obtained input until the ratio of obtaining the same feedback exceeds the reference value. .. Learning is performed when the rate of obtaining the same feedback exceeds the reference value.
If it is determined in step S27 that the input is valid, the machine learning unit 11d performs learning based on the received input (step S29), and ends the process.
On the other hand, if it is determined that the input is not valid, the machine learning unit 11d ends the process (step S31) without performing learning based on the received input.

For example, the recognition result candidate shown in FIG. 6 includes a correct answer, and the user selects the correct answer from the candidates. However, there may be cases where the correct answer is not included in the recognition result candidates. In order to deal with such cases, in addition to the list of candidates, it is conceivable to prepare options such as "Other" and "There is no correct answer in this".
When "Other" or "There is no correct answer in this" is selected, the recognition complementing unit 11c can input the correct answer intended by the user and provides a means for accepting the input.
For example, it is conceivable to enable the user to input by handwriting, or to display a soft key for input on the screen and enable the user to input Kana-Kanji conversion using the soft key.
Further, for example, when a candidate as shown in FIG. 6 is displayed as a first-level candidate having the highest probability of correct answer and "Other" or "There is no correct answer in this" is selected, the probability of correct answer is next. It is also possible to display high second level candidates.
Alternatively, if the candidates do not fit on one screen, a page feed button or a scroll bar may be displayed so that the candidates that are not displayed on the current screen can be displayed by their operations.

(Embodiment 2)
In the first embodiment, a case where a printed image is recognized as a character has been described as an example. However, the target of character recognition and learning is not limited to printed characters, and may be handwritten characters.
10 to 12 are explanatory views showing an example in which input relating to the correctness of handwritten character recognition is requested during the waiting period for the end of copying in this embodiment.
When the user selects one of the candidates displayed on the screen shown in FIG. 10 (see FIG. 11) and confirms the recognition result (see FIG. 12), the control unit 11 is displayed as the machine learning unit 11d. Learn that the correct answer for handwritten characters is the selected candidate. This makes it possible to improve the recognition accuracy of the user's habitual characters and handwritten characters that are difficult to distinguish.

(Embodiment 3)
In the first embodiment, the case where the pattern recognition unit 11a recognizes the image read by the scanner unit 15 as characters has been described as an example.
In this embodiment, the image processing device 100 authenticates the user and accepts the operation of the authenticated user. As a means of user authentication, the pattern recognition unit 11a shall perform biometric authentication.
As a specific example of biometric authentication, a case where the image processing device 100 is provided with a camera and performs face authentication of a user will be described. However, the biometric authentication is not limited to this, and can be applied to fingerprint authentication, iris authentication, fingerprint authentication, and voice authentication. Face recognition, fingerprint recognition, and iris recognition are pattern recognition for images. On the other hand, voice authentication is pattern recognition for voice. However, machine learning can be applied to image recognition, speech recognition, and other pattern recognition.

13 to 15 are explanatory views showing an example of requesting user feedback input for an example of face recognition. Instead of or in addition to FIGS. 6 to 8 described in the first embodiment, the recognition complementing unit 11c displays the screens shown in FIGS. 13 to 15 during the waiting period, and machine learning is performed based on the user's input. Part 11d learns. This makes it possible to improve the accuracy of face recognition.
Further, not limited to user authentication, image recognition other than the character recognition described in the first embodiment may be performed by using the image read by the scanner unit 15 or the image received via the communication interface circuit 14 as the recognition target.
For example, a process such as recognizing a logo or a seal image included in an image received by facsimile and performing a journal entry of the sender may be performed.

(Embodiment 4)
Regarding the character recognition of the type shown in the first embodiment, it is preferable to obtain an input related to feedback from an unspecified user. It is thought that targeting unspecified users can obtain more feedback during the same period than identifying users, and can provide more objective feedback.
On the other hand, the pattern recognition related to user authentication as shown in the third embodiment is information individual to the user. Therefore, it is preferable to perform learning in association with each user.

Regarding the handwritten character recognition shown in the second embodiment, the user, the administrator, or the like may be able to select whether to perform learning as related to an unspecified user or as individual information for the user. ..
If the handwritten character to be recognized is the user's own handwritten character, it is preferable to learn as individual information for the user. However, if it is a handwritten character of another person, it is preferable to perform learning as if it is unspecified. You can ask the user to select the handwritten characters of the user or the handwritten characters of another person.

As mentioned above
(I) The image processing apparatus according to the present invention has a pattern recognition unit that recognizes an image related to image processing or a user who performs an operation related to the image processing, and evaluates the correctness of the recognition result or receives an evaluation from the user. Using the recognition evaluation unit, the recognition complementing unit that presents all or part of the recognition target that is judged not to have sufficient recognition accuracy based on the evaluation and requests the input of the user related to correctness, and the obtained input. The recognition complementing unit includes a machine learning unit that performs learning related to the recognition for all or a part of the recognition target, and the recognition complementing unit waits for the user to finish the image processing or respond to the operation. It is characterized by presenting an object.

In the present invention, the image related to image processing is an image to be image-processed by the image processing apparatus. A specific embodiment thereof is, for example, a scanned original image or a printed image to be printed.
Further, the operation by the user related to the image processing may be performed directly on the image processing device, but the operation is not limited thereto. For example, the user may operate an information processing device such as a PC or a smartphone that can communicate with the image processing device via a network, and the image processing device may accept the operation via communication.

Furthermore, receiving an evaluation from the user regarding the correctness of the recognition result means that, for example, when the object of recognition is a manuscript image in which a sentence is written, the recognition result is erroneous recognition for some characters of the written sentence. You may get the evaluation. Alternatively, the recognition result may be evaluated for the entire sentence. The recognition complementing unit determines whether or not the recognition accuracy of all or part of the recognition target is sufficient based on the obtained evaluation.
The evaluation regarding the correctness of the recognition result may be received from the user who has performed the operation related to the image processing of the image to be evaluated. However, the evaluation may be received from other users. That is, the user who performs the evaluation is not always limited to the user who has performed the operation related to the image processing of the image to be evaluated. Depending on the evaluation target, a user who performs an operation related to other image processing may perform the evaluation.

Since the recognition accuracy can be said to be a statistical index of whether the recognition result is correct or incorrect, it is extremely preferable that the recognition complementing unit makes a judgment based on a plurality of evaluations to determine whether or not the recognition accuracy is sufficient. That is, it is extremely preferable to make a judgment using a plurality of evaluations for the same or similar recognition target.
The recognition complement unit asks the user to input the correct answer for all or part of a certain recognition target. Various means can be considered as a means for inputting the correct answer. For example, a method in which a plurality of candidates are presented and one of the candidates is selected as the correct answer can be considered.
The waiting period is a period in which the user waits for the end of image processing, a response to an operation, or the like.
For example, when the recognition target is a document image of a manuscript and the character recognition is performed, the recognition complementing unit may present a recognition target other than the manuscript during the waiting period for image processing related to the manuscript. That is, the recognition target processed in the past may be presented instead of the manuscript. Of course, the document being image-processed may be presented as a recognition target.

Further, preferred embodiments of the present invention will be described.
(Ii) The recognition target may be an image related to the image processing or the user, and the recognition complementing unit may further include a display unit that presents all or a part of the image.
In this way, the recognition target whose recognition accuracy is not sufficient can be displayed on the display unit provided in the image processing device, and the user can input the correct answer.

(Iii) The recognition target is a voice related to the user, and the recognition complementing unit may further include a voice output unit that presents all or a part of the voice.
In this way, it is possible to output a recognition target having insufficient recognition accuracy from the voice output unit included in the image processing device and have the user input the correct answer. There are various possible means for inputting the correct voice, but it is not always necessary to reply by voice. For example, a plurality of correct answer candidates may be displayed as characters on a display unit provided in the image processing device, and the correct answer may be selected from the correct answer candidates, or the user may input the correct answer in characters.
As a case where the recognition target is the voice related to the user, for example, a scene in which a specific speaker voice recognition is applied to user authentication and a predetermined word or sentence is used as a password or a passphrase can be considered.

(Iv) The waiting period may be a period that does not include the state during the power saving mode and the state in which the power saving mode can be entered.
In this way, it is possible to prevent the effect of energy saving from being diminished by learning.
The state in which the power saving mode can be shifted refers to a state in which the power saving mode can be shifted to the power saving mode, for example, waiting for a job instruction.
On the other hand, the state in which the power saving mode cannot be shifted is related to, for example, the state of returning from the power saving mode in response to a job execution instruction during the power saving mode, or image processing such as copying, scanning, and printer. The job is being executed. It can be said that these states belong to the waiting period in which the user is waiting for the job to finish, but cannot shift to the power saving mode.

(V) The recognition complementing unit may identify a user who inputs correct / incorrect input, and the machine learning unit may perform the learning for each identified user.
In this way, since learning is performed for each user, it is possible to improve the recognition accuracy in response to the habits and characteristics peculiar to the user.

(Vi) The recognition complementing unit may determine whether or not the obtained input should be learned, and may perform learning only for the input determined to be learned.
In this way, when the user makes a biased input or an erroneous input, it is possible to judge it and select an appropriate input for learning.

(Vii) Further, as a mode different from the above-mentioned image processing apparatus, the present invention performs a step of recognizing an image related to image processing or a user who performs an operation related to the image processing, and evaluates the correctness of the recognition result. A step of receiving an evaluation from the user and a recognition complementing step of presenting all or a part of the recognition target judged to be not sufficient recognition accuracy based on the evaluation and requesting the input of the user related to the correctness were obtained. The computer executes a process including a step of learning about the recognition for all or a part of the recognition target using the input, and in the recognition complement step, the user finishes the image processing or responds to the operation. It provides a method for improving recognition accuracy that is executed during the waiting period.

Preferred embodiments of the present invention include a combination of any of the plurality of embodiments described above.
In addition to the embodiments described above, there may be various variations of the present invention. These variations should not be construed as not belonging to the scope of the present invention. The present invention should include the meaning equivalent to the claims and all modifications within the said scope.

10: Operation unit, 11: Control unit, 11a: Pattern recognition unit, 11b: Recognition evaluation unit, 11c: Recognition complement unit, 11d: Machine learning unit, 12: Display unit, 13: Printing unit, 14: Communication interface circuit, 15: Scanner unit, 16: Image data generation circuit, 17a: Paper feed tray, 17b: Bypass tray, 18a, 18b, 18c: Ejection tray, 20: Information processing device 30: Fingers, 31: Setting key, 32: Number of copies Area, 33: Machine illustration, 34: [Job status] key, 35: [Reset] key, 36: [Start] key, 37: [Cancel copy] key 100: Image processing device

Claims (7)

A pattern recognition unit that recognizes an image related to image processing or a user who performs an operation related to the image processing.
The recognition evaluation department that evaluates the correctness of the recognition result or receives the evaluation from the user,
A recognition complementing unit that presents all or part of the recognition target that is judged not to have sufficient recognition accuracy based on the evaluation and requests the input of the user regarding correctness.
It is provided with a machine learning unit that performs learning related to the recognition for all or a part of the recognition target using the obtained input.
The recognition complementing unit is an image processing device that presents the recognition target during a waiting period in which the user waits for the end of the image processing or a response to the operation. The recognition target is the image processing or the image related to the user.
The image processing apparatus according to claim 1, wherein the recognition complementing unit further includes a display unit that presents all or a part of the image. The recognition target is the voice related to the user,
The image processing device according to claim 1, wherein the recognition complementing unit further includes an audio output unit that presents all or part of the audio. The image processing apparatus according to claim 1, wherein the waiting period does not include a state during the power saving mode and a state in which the mode can be shifted to the power saving mode. The recognition complementing unit identifies the user who inputs the correctness and error, and identifies the user.
The image processing device according to any one of claims 1 to 4, wherein the machine learning unit performs the learning for each identified user. The image according to any one of claims 1 to 4, wherein the recognition complementing unit determines whether or not the obtained input should be learned, and learns only the input determined to be learned. Processing equipment. A step of recognizing an image related to image processing or a user who performs an operation related to the image processing,
The step of evaluating the correctness of the recognition result or receiving the evaluation from the user,
A recognition complementing step that presents all or part of the recognition target that is judged not to have sufficient recognition accuracy based on the evaluation and requests the input of the user regarding correctness.
Using the obtained input, the computer executes a process including a step of learning related to the recognition for all or a part of the recognition target.
The recognition complementing step is a recognition accuracy improving method executed during a waiting period in which the user waits for the end of the image processing or a response to the operation.