JP6944127B2 - Image processing equipment, computer programs, and image processing methods - Google Patents

Description

The present specification relates to image processing for image data, and more particularly to image processing for specifying character pixels in an image.

In image data, for example, image data generated by reading a printed matter using an image sensor, halftone dots included in the printed matter appear in the image indicated by the image data. Pixels constituting such halftone dots are likely to be erroneously specified as character pixels when specifying character pixels in an image.

The image processing apparatus disclosed in Patent Document 1 executes edge determination for determining whether or not each pixel is an edge, and halftone dot determination for determining whether or not each pixel is a halftone dot. The image processing device identifies pixels that are edges and are not halftone dots as pixels that indicate characters.

Japanese Unexamined Patent Publication No. 6-164928 Japanese Unexamined Patent Publication No. 2016-38732

As described above, there has been a demand for a technique capable of accurately identifying the character pixels constituting the characters in the target image.

The present specification discloses a technique capable of accurately identifying character pixels in a target image.

The techniques disclosed herein have been made to solve at least a portion of the above problems and can be realized as the following application examples.

[Application Example 1] An image processing device, an image acquisition unit that acquires target image data indicating a target image, and a plurality of character candidate pixels that are candidates for character pixels that constitute characters using the target image data. A candidate pixel extraction unit for extracting The block-by-block determination is performed using a machine learning model trained with a plurality of character image data indicating characters and a plurality of non-character image data not indicating characters. An image processing device including a determination unit and a character pixel identification unit that identifies a plurality of character pixels indicating characters from the plurality of character candidate pixels by using the determination result by the determination unit.

According to the above configuration, a plurality of character pixels are specified from a plurality of character candidate pixels by using the judgment result of determining whether or not the character block is a character block using a machine learning model. NS. As a result, the character pixels in the target image can be specified with high accuracy.
[Application example 2]
The image processing apparatus according to Application Example 1.
The candidate pixel extraction unit is an image processing device that extracts a plurality of character candidate pixels without executing edge adjustment processing for adjusting the intensity of edges in the target image with respect to the target image data.
[Application example 3]
The image processing apparatus according to Application Example 1 or 2.
The target image data includes color values of a plurality of pixels.
The color value includes a plurality of component values and contains a plurality of component values.
The candidate pixel extraction unit
Using the target image data, it is the first image data including a plurality of first values corresponding to the color values of the plurality of pixels, and each of the plurality of first values is the corresponding color. The first image data, which is a value based on either the minimum value or the maximum value among the plurality of component values of the value, is generated.
An image processing device that extracts the plurality of character candidate pixels using the first image data.
[Application example 4]
The image processing apparatus according to any one of Application Examples 1 to 3.
The candidate pixel extraction unit
Using the target image data, second image data including a plurality of second values indicating the brightness of the corresponding pixel among the plurality of pixels in the target image is generated.
An image processing device that binarizes the second image data so as to extract pixels having a brightness higher than the reference as the character candidate pixels.
[Application example 5]
The image processing apparatus according to any one of Application Examples 1 to 4.
The candidate pixel extraction unit
The first extraction process is executed to extract a plurality of first character candidate pixels among the plurality of character candidate pixels.
A second extraction process different from the first extraction process is executed to extract a plurality of second character candidate pixels among the plurality of character candidate pixels.
The character pixel identification unit is
Using the judgment result by the judgment unit, a plurality of first pixels are identified from the plurality of first character candidate pixels.
Using the judgment result by the judgment unit, a plurality of second pixels are identified from the plurality of second character candidate pixels.
An image processing device that identifies the plurality of character pixels including the plurality of first pixels and the plurality of second pixels.
[Application example 6]
The image processing apparatus according to Application Example 5.
The machine learning model used by the determination unit includes a first machine learning model and a second machine learning model different from the first machine learning model.
The judgment unit
Using the first machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
Using the second machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
The character pixel identification unit is
Using the judgment result using the first machine learning model, the plurality of first pixels are identified, and the plurality of first pixels are identified.
An image processing device that identifies a plurality of the second pixels by using the determination result using the second machine learning model.
[Application 7]
The image processing apparatus according to Application Example 6.
The plurality of character image data includes a plurality of first character image data and a plurality of second character image data different from the plurality of first character image data.
The plurality of non-character image data includes a plurality of first non-character image data and a plurality of second non-character image data different from the plurality of first non-character image data. ,
The first machine learning model is the machine learning model trained using the plurality of first character image data and the plurality of first non-character image data.
The second machine learning model is an image processing apparatus that is the machine learning model trained using the plurality of second character image data and the plurality of second non-character image data. ..
[Application Example 8]
The image processing apparatus according to Application Example 7.
The target image data includes color values of a plurality of pixels.
The color value includes a plurality of component values and contains a plurality of component values.
The first extraction process is
Using the target image data, it is the first image data including a plurality of first values corresponding to the color values of the plurality of pixels, and each of the plurality of first values is the corresponding color. The process of generating the first image data, which is a value based on either the minimum value or the maximum value of the plurality of component values of the value, and
A process of identifying the plurality of first pixels by binarizing the first image data, and
Including
The second extraction process is
A process of using the target image data to generate second image data including a plurality of second values indicating the brightness of the corresponding pixel among the plurality of pixels in the target image.
A process of specifying the plurality of second pixels by binarizing the second image data so that a pixel having a brightness higher than the reference is specified as the character candidate pixel.
Including
In the first machine learning model, the plurality of first character image data showing the first character, which is a character having a brightness lower than that of the background, and the plurality of first characters not showing the first character. The machine learning model trained using the non-character image data of
In the second machine learning model, the plurality of second character image data showing the second character, which is a character having a brightness higher than that of the background, and the plurality of second characters not showing the second character. An image processing apparatus, which is the machine learning model trained using the non-character image data of.
[Application example 9]
The image processing apparatus according to any one of Application Examples 1 to 8.
Among the target image data, the first image processing is executed on the identified values of the plurality of character pixels, and the first image is applied to the values of pixels different from the plurality of character pixels. An image processing apparatus including an image processing unit that executes a second image processing different from the processing to generate the target image data that has been image-processed.
[Application Example 10]
The image processing apparatus according to Application Example 9.
An image processing apparatus including a print data generation unit that generates print data using the target image data that has been image-processed.

The techniques disclosed in the present specification can be realized in various forms, for example, a multifunction device, a scanner, a printer, an image processing method, a function of these devices, or a computer for realizing the above method. It can be realized in the form of a program, a recording medium on which the computer program is recorded, or the like.

It is a block diagram which shows the structure of the multifunction device 200 which is an example of an image processing apparatus. It is a flowchart of image processing. FIG. 1 is a first diagram showing an example of an image used in image processing. It is a figure which shows an example of the image used in the character identification processing. It is a flowchart of the 1st binary image data generation processing. It is explanatory drawing of the minimum component value and the maximum component value of scan data. FIG. 2 is a second diagram showing an example of an image used for image processing. It is a flowchart of the 2nd binary image data generation processing. It is a flowchart of a block determination process. It is explanatory drawing of a plurality of blocks BL arranged on a scan image SI. It is a figure which shows an example of the judgment for each block BL. It is a figure which shows an example of setting of the pixel value in a block determination data. It is a figure explaining the effect of an Example.

A. Example:
A-1: Configuration of the multifunction device 200 An embodiment will be described based on an embodiment. FIG. 1 is a block diagram showing a configuration of a multifunction device 200, which is an example of an image processing device. The multifunction device 200 includes a CPU 210 which is a processor for controlling an image processing device, a volatile storage device 220 such as a DRAM, a non-volatile storage device 230 such as a flash memory and a hard disk drive, and a display unit 240 such as a liquid crystal display. An operation unit 250 including a touch panel and buttons superimposed on a liquid crystal display, an interface (communication IF) 270 for communicating with an external device such as a user's terminal device 100, a print execution unit 280, and a reading execution unit 290. , Is equipped.

The scanning execution unit 290 generates scan data by optically scanning the document using the one-dimensional image sensor under the control of the CPU 210. According to the control of the CPU 210, the print execution unit 280 uses a plurality of types of toners, specifically, cyan (C), magenta (M), yellow (Y), and black (K) toners as coloring materials. The image is printed on a printing medium such as paper by the laser method. Specifically, the print execution unit 280 exposes the photosensitive drum to form an electrostatic latent image, and attaches toner to the electrostatic latent image to form a toner image. The print execution unit 280 transfers the toner image formed on the photosensitive drum to the paper. In the modified example, the print execution unit 280 may be an inkjet type print execution unit that ejects ink as a coloring material to form an image on paper.

The volatile storage device 220 provides a buffer area for temporarily storing various intermediate data generated when the CPU 210 performs processing. The computer program PG is stored in the non-volatile storage device 230. The computer program PG is a control program that allows the CPU 210 to control the multifunction device 200. In this embodiment, the computer program PG is provided in a form that is pre-stored in the non-volatile storage device 230 at the time of manufacturing the multifunction device 200. Instead, the computer program PG may be provided in a form downloaded from a server, or may be provided in a form stored in a DVD-ROM or the like. The CPU 210 can execute image processing described later by executing the computer program PG.

A-2: Image processing FIG. 2 is a flowchart of image processing. This image processing is executed, for example, when the user places the document on the platen of the scanning execution unit 290 and inputs a copy execution instruction. In this image processing, scan data generated by scanning a document using a scanning execution unit 290 is acquired, and the scan data is used to generate print data indicating the document, thereby producing a so-called copy of the document. It is a process to be realized.

In S10, the CPU 210 generates scan data as target image data by reading the document placed on the platen by the user using the scanning execution unit 290. The manuscript is, for example, a printed matter in which an image is printed by a multifunction device 200 or a printer (not shown). The generated scan data is stored in the buffer area of the volatile storage device 220 (FIG. 1). The scan data includes the values of a plurality of pixels, and each of the values of the plurality of pixels represents the color of the pixel as an RGB color value (also referred to as an RGB value). That is, the scan data is RGB image data. The RGB value of one pixel includes, for example, the values of three color components (hereinafter, also referred to as R value, G value, and B value) of red (R), green (G), and blue (B). I'm out. In this embodiment, the number of gradations of each component value is 256 gradations.

The scan data, which is RGB image data, includes three component image data (R component image data, G component image data, B component image data) corresponding to the three color components constituting the RGB color system. Can be said. Each component image data is image data in which the value of one type of color component is used as the pixel value.

FIG. 3 is a first diagram showing an example of an image used in image processing. FIG. 3A shows an example of the scanned image SI shown by the scan data. The scanned image SI includes a plurality of pixels. The plurality of pixels are arranged in a matrix along a first direction D1 and a second direction D2 orthogonal to the first direction D1.

The scanned image SI of FIG. 3A shows a white background Bg1 indicating the background color of the paper of the original, objects Ob1 to Ob3 different from the three characters, four characters Ob4 to Ob7, and four characters. The backgrounds Bg2 and Bg3 of the characters Ob4 to Ob7 are included. Objects that are different from text are, for example, photographs. The backgrounds Bg2 and Bg3 are uniform images having colors different from white. The characters Ob4 and Ob5 on the background Bg2 are characters having a darker color than the background Bg2, that is, characters having a lower brightness than the background Bg2. The characters Ob6 and Ob7 on the background Bg3 are characters having a lighter color than the background Bg3, that is, characters having a higher brightness than the background Bg3.

In S20, the CPU 210 executes character identification processing on the scan data. The character identification process is a process for identifying character pixels by classifying a plurality of pixels in the scanned image SI into a plurality of character pixels indicating characters and a plurality of non-character pixels not indicating characters. Is.

By the character identification process, for example, binary image data (also referred to as character identification data) in which the value of the character pixel is set to "1" and the value of the non-character pixel is set to "0" is generated. FIG. 3B shows an example of the character identification image TI indicated by the character identification data. In this character identification image TI, a plurality of pixels constituting the four characters Ob4 to Ob7 in the scanned image SI are specified as character pixels Tp4 to Tp7. The details of the character identification process will be described later.

In S30, the CPU 210 executes a halftone dot smoothing process on the scan data to generate smoothed image data indicating the smoothed image. Specifically, the CPU 210 executes a smoothing process using a smoothing filter such as a Gaussian filter on each of the values of a plurality of non-character pixels included in the scan data, and the smoothing process has been completed. Calculate the values of a plurality of non-character pixels. The non-character pixel to be smoothed is specified by referring to the character identification data generated by the classification process of S20. The CPU 210 generates smoothed image data including the values of the plurality of character pixels included in the scan data and the values of the plurality of non-character pixels that have been smoothed.

FIG. 3C shows a smoothed image GI represented by the smoothed image data. The smoothed image GI includes a white background Bg1g, objects Ob1 to Ob7 in the scanned image SI, objects Ob1g to Ob7g in which the backgrounds Bg2 and Bg3 are smoothed, backgrounds Bg2g, and Bg3g. Of these objects Ob1g to Ob7g, background Bg2g, and Bg3g, parts other than the characters Ob4g to Ob7g (also referred to as non-character parts) are smoothed as compared with the scanned image SI.

In S40, the CPU 210 executes character sharpening processing on the smoothed image data to generate processed image data. Specifically, the CPU 210 executes a sharpening process such as an unsharp mask process or a process of applying a sharpening filter on each of the values of a plurality of character pixels included in the smoothed image data. Calculate the values of a plurality of character pixels that have been sharpened. The character pixel to be sharpened is specified by referring to the character identification data generated by the classification process of S20. Then, the CPU 210 includes the values of the plurality of non-character pixels included in the smoothed image data (values of the plurality of non-character pixels that have been smoothed) and the values of the plurality of character pixels that have been sharpened. Generates processed image data including ,. Since the values of the plurality of character pixels included in the smoothed image data are not subject to the smoothing process, they are the same as the values of the plurality of character pixels included in the scan data. Therefore, it can be said that the character sharpening process of this step is executed for the values of a plurality of character pixels included in the scan data.

FIG. 3D shows a processed image FI represented by the processed image data. The processed image FI includes a white background Bg1f, objects Ob1 to Ob7 in the scanned image SI, objects Ob1f to Ob7f corresponding to the backgrounds Bg2 and Bg3, and backgrounds Bg2f and Bg3f. Of these objects Ob1f to Ob7f, backgrounds Bg2f, and Bg3f, the edges of the characters Ob4f to Ob7f are sharpened as compared with the characters Ob4 to Ob7 in the scanned image SI and the characters Ob4g to Ob7g in the smoothed image GI. Has been done. Further, the edges of the objects Ob1f to Ob3f, the background Bg2f, and the Bg3f other than the characters are not sharpened.

As can be seen from the above description, the objects Ob1f to Ob7f, the background Bg2f, and the Bg3f in the processed image FI include sharpened characters and smoothed non-characters.

In S50, the CPU 210 executes a print data generation process for generating print data using the processed image data. Specifically, a color conversion process is executed on the processed image data which is RGB image data, and a color having color components (C, M, Y, K components) corresponding to the color material used for printing is performed. CMYK image data indicating the color for each pixel is generated by the CMYK value which is a value. The color conversion process is executed with reference to, for example, a known look-up table. Halftone processing is executed on the CMYK value image data, and dot data indicating a dot formation state is generated for each color material used for printing and for each pixel. The dot formation state can be, for example, two types of states with or without dots, and four types of states of large dots, medium dots, small dots, and no dots. The halftone processing is executed according to, for example, a dither method or an error diffusion method. The dot data is rearranged in the order in which they are used at the time of printing, and print data is generated by adding a print command to the dot data.

In S60, the CPU 210 executes the print process and ends the image process. Specifically, the CPU 210 supplies print data to the print execution unit 280, and causes the print execution unit 280 to print the processed image.

According to the image processing described above, the first image processing (specifically, the edge sharpening process) is executed for the values of the plurality of specified character pixels in the scan data (S40). A second image process (specifically, halftone dot smoothing process) different from the first image process is executed for the values of the plurality of non-character pixels (S30), and the processed image data is generated. NS. As a result, different image processing is executed for the value of the character pixel and the value of the pixel different from the character pixel, so that appropriate image processing for the scan data can be realized. In the modified example, the character sharpening process of S40 may be executed first, and then the halftone dot smoothing process of S30 may be executed.

More specifically, processed image data including the values of the plurality of character pixels that have been sharpened and the values of the plurality of non-character pixels that have been smoothed is generated (S30, S40). .. As a result, it is possible to generate processed image data showing the processed image FI that looks good.

For example, as shown in the processed image FI of FIG. 3D, in the processed image data, the sharpening-processed value is used as the value of the character pixel. As a result, the characters of the processed image FI look sharp, so that the appearance of the processed image FI to be printed can be improved, for example.

Further, in the processed image data, the smoothed value is used as the value of the background Bg2 in the processed image FI and the non-character pixel constituting an object different from the character such as a photograph. As a result, for example, it is possible to suppress the appearance of halftone dots that cause moire in a portion different from the characters of the processed image FI, so that it is possible to prevent problems such as moire from occurring in the printed processed image FI. can. As a result, the appearance of the processed image FI to be printed can be improved. In addition, since the edges in the photograph are suppressed from being excessively emphasized, the appearance of the processed image FI to be further printed can be improved.

For example, the original used to generate the scan data is a printed matter on which an image is printed. Therefore, for example, a uniform portion such as a background Bg2 having a color different from white in a document forms halftone dots when viewed at the dot level forming an image. The halftone dots include a plurality of dots and a portion where the dots are not arranged (a portion indicating the ground color of the document). For this reason, halftone dots are shown in the region showing the backgrounds Bg2 and Bg3 in the scanned image SI when viewed at the pixel level. The dots in the halftone dots are arranged with periodicity due to the influence of the dither matrix used when printing the original. For this reason, when printing is performed using scan data, the periodic components of halftone dot dots existing in the original image (scan image SI) before halftone processing and the dots of halftone dots that make up the printed image Moire is likely to appear due to interference with the periodic component. In the processed image FI of this embodiment, the smoothing process reduces the periodic component of dots in a portion different from the edge in the original image (scanned image SI). As a result, when the processed image FI is printed using the processed image data, for example, it is possible to suppress the occurrence of moire in the printed processed image FI.

In particular, in the above image processing, print data is generated using the processed image data (S50). Therefore, for example, appropriate print data capable of suppressing moire that tends to occur in the processed image FI to be printed is generated. can do.

A-3: Character identification process The character identification process of S20 in FIG. 2 will be described. In S21, the CPU 210 uses the scan data to execute the first binary image data generation process to generate the first binary image data. The first binary image data is binary data indicating a character candidate pixel and a pixel different from the character candidate pixel. The character candidate pixel is a candidate for a character pixel to be specified by the character identification process. Here, the character candidate pixel indicated by the first binary image data is also referred to as a first character candidate pixel.

FIG. 4 is a diagram showing an example of an image used in the character identification process. FIG. 4A shows an example of the first binary image CI1 represented by the first binary image data. In the first binary image CI1, the black portion indicates a pixel specified as a character candidate pixel, and the white portion indicates a pixel different from the character candidate pixel.

In the first binary image CI1, a plurality of pixels Cp1 to Cp3 constituting the objects Ob1 to Ob3 different from the characters in the scanned image SI, and a plurality of pixels Cp4 and Cp5 constituting the characters Ob4 and Ob5 are used. The pixel Cpb that constitutes the entire background Bg3 and the characters Ob6 and Ob7 is specified as the first character candidate pixel. Further, in the first binary image CI1, the pixels constituting the characters Ob6 and Ob7 are not specified separately from the background Bg3. As described above, the specified first character candidate pixel may include pixels forming an object or background different from the character. This is because it is difficult to extract all character candidate pixels by only one type of binarization processing, and it is also difficult to exclude pixels that do not constitute characters. Details of the first binary image data generation process will be described later. In this embodiment, in the first binary image data generation process, first character candidate pixels including character pixels constituting characters having a brightness lower than that of the background are extracted.

In S22, the CPU 210 executes a first block determination process on the scan data to indicate binary image data (first block) indicating a character block indicating characters and a non-character block indicating no characters. (Also called judgment data) is generated. The first block determination data is binary data in which the value of the pixel constituting the character block is "1" and the value of the pixel individualizing the non-character block is "0". The first block determination process is a process of determining for each block whether or not each of the plurality of blocks arranged in the scan image SI is a character block indicating a character by using the scan data. One block is a rectangular area containing N pixels (N is an integer of 2 or more). The details of the first block determination process will be described later.

FIG. 4B shows an example of the first block determination image BI1 shown by the first block determination data. In the first block determination image BI1, the character blocks Bk4 and Bk5 corresponding to the area in which the characters Ob4 and Ob5 are arranged in the scanned image SI are specified. As described above, the character block specified by the first block determination data does not include the block corresponding to the area including the object different from the character. The character block specified by the first block determination data is a block indicating a character having a lower brightness than the background. Therefore, in the first block determination image BI1, the character blocks Bk6 and Bk7 (FIG. 4 (D)) corresponding to the area in which the characters Ob6 and Ob7 having higher brightness than the background are arranged are not specified.

In S23, the CPU 210 executes the logical product composition process using the first binary image data generated in S21 and the first block determination data generated in S22. As a result, first character identification data indicating a plurality of first character pixels is generated. Specifically, the CPU 210 generates binary image data as the first character identification data by taking the logical product of each pixel of the first binary image data and the first block determination data. do. In other words, among the plurality of pixels in the scanned image SI, the CPU 210 selects the pixel specified as the first character candidate pixel in S21 and located in the character block identified in S22 as the first character block. Specify as a character pixel. The CPU 210 does not specify the pixel that is not specified as the first character candidate pixel and the pixel in the non-character block as the first character pixel among the plurality of pixels in the scanned image SI.

As shown in the first binary image CI1 of FIG. 4A, the plurality of first character candidate pixels identified by the first binary image data have the characters Ob4 and Ob5 in the scan image SI. In addition to the constituent pixels Cp4 and Cp5, the pixels Cp1 to Cp3 and Cpb constituting the other objects Ob1 to Ob3 and the background Bg3 are included. As shown in the first block determination image BI1 of FIG. 4B, by taking the logical product of the first binary image data and the first block determination data, the first character identification data can be scanned. The pixels Cp4 and Cp5 constituting the characters Ob4 and Ob5 in the image SI are selectively specified as the first character pixel. That is, the plurality of first character pixels include pixels Cp4 and Cp5 constituting characters Ob4 and Ob5 in the scanned image SI, and pixels Cp1 to Cp3 and Cpb constituting other objects Ob1 to Ob3 and background Bg3. Not included

In S24, the CPU 210 uses the scan data to execute the second binary image data generation process to generate the second binary image data. The second binary image data is binary data indicating a character candidate pixel and a pixel different from the character candidate pixel. Here, the character candidate pixel indicated by the second binary image data is also referred to as a second character candidate pixel.

FIG. 4C shows an example of the second binary image CI2 represented by the second binary image data. In the second binary image CI2, the black portion indicates a pixel specified as the second character candidate pixel, and the white portion indicates a pixel different from the character candidate pixel. The same applies to FIG. 4 (C).

In the second binary image CI2, the characters Ob6 and Ob7 having higher brightness than the background Bg3 and the Cp8 constituting the background Bg1g are specified as the second character candidate pixels. Further, in the first binary image CI1, the pixels constituting the characters Ob4 and Ob5 are not specified. As described above, the second character candidate pixel may include pixels forming an object or background different from the character, like the first character candidate pixel. Details of the second binary image data generation process will be described later. In this embodiment, in the second binary image data generation process, a second character candidate pixel including a character pixel constituting a character having a brightness higher than that of the background is extracted.

In S25, the CPU 210 executes a second block determination process on the scan data to indicate binary image data (second block) indicating a character block indicating characters and a non-character block indicating no characters. (Also called judgment data) is generated. In the second block determination data, similarly to the first block determination data, the value of the pixel constituting the character block is set to "1", and the value of the pixel individualizing the non-character block is set to "0". Value data. In the second block determination process, as in the first block determination process, whether or not each of the plurality of blocks arranged in the scan image SI using the scan data is a character block indicating a character. Is a process for determining each block. The details of the second block determination process will be described later.

FIG. 4D shows an example of the second block determination image BI2 shown by the second block determination data. In the second block determination image BI2, the character blocks Bk6 and Bk7 corresponding to the area in which the characters Ob6 and Ob7 are arranged in the scanned image SI are specified. As described above, the character block specified by the second block determination data does not include the block corresponding to the area including the object different from the character. The character block specified by the second block determination data is a block indicating a character having a brightness higher than that of the background. Therefore, in the second block determination image BI2, the character blocks Bk4 and Bk5 (FIG. 4B) corresponding to the area in which the characters Ob4 and Ob5 having lower brightness than the background are arranged are not specified.

In S26, the CPU 210 executes the logical product composition process using the second binary image data generated in S24 and the second block determination data generated in S25. As a result, second character identification data indicating a plurality of second character pixels is generated. Specifically, the CPU 210 generates binary image data as the second character identification data by taking the logical product of each pixel of the second binary image data and the second block determination data. do. In other words, among the plurality of pixels in the scanned image SI, the CPU 210 selects the pixel specified as the second character candidate pixel in S24 and located in the character block identified in S25 as the second character block. Specify as a character pixel. The CPU 210 does not specify the pixel that is not specified as the second character candidate pixel and the pixel in the non-character block as the second character pixel among the plurality of pixels in the scanned image SI.

As shown in the second binary image CI2 of FIG. 4C, the plurality of second character candidate pixels identified by the second binary image data have the characters Ob6 and Ob7 in the scan image SI. In addition to the constituent pixels Cp6 and Cp7, the constituent pixels Cp8 including the background Bg1 and the like are included. As shown in the second block determination image BI2 of FIG. 4D, by taking the logical product of the second binary image data and the second block determination data, the second character identification data can be scanned. The pixels Cp6 and Cp7 constituting the characters Ob6 and Ob7 in the image SI are selectively specified as the second character pixels. That is, the plurality of second character pixels include the pixels Cp6 and Cp7 constituting the characters Ob6 and Ob7 in the scanned image SI, and do not include the pixels Cp8 forming the background Bg1 and the like.

In S27, the CPU 210 executes the OR composition process using the first character identification data generated in S23 and the second character identification data generated in S26. As a result, character identification data indicating a plurality of character pixels to be finally specified is generated. In other words, the CPU 210 is a pixel including a plurality of first character pixels specified by the first character identification data and a plurality of second character pixels specified by the second character identification data. A group of pixels that does not include pixels that are different from the first character pixel and the second character pixel is finally specified as a plurality of character pixels. As a result, the identification omission of a plurality of character pixels in the scanned image SI can be effectively reduced by using the first character identification data and the second character identification data. For example, as in the character identification image TI shown in FIG. 4B, the plurality of character pixels identified by the final character identification data are a plurality of characters constituting the characters Ob4 to Ob7 in the scanned image SI. It contains pixels Tp4 to Tp7, and does not include pixels that make up other objects or backgrounds.

A-4: First binary image data generation process The first binary image data generation process in S21 of FIG. 2 will be described. FIG. 5 is a flowchart of the first binary image data generation process. In S100, the CPU 210 uses the scan data to generate the minimum component data. Specifically, the CPU 210 acquires the minimum component value Vmin from each of the values (RGB values) of the plurality of pixels included in the scan data. The minimum component value Vmin is the minimum value among a plurality of component values (R value, G value, B value) included in the RGB value. The CPU 210 generates image data in which these minimum component values Vmin are the values of a plurality of pixels as the minimum component data. The minimum component data is image data indicating an image having the same size as the scanned image SI. Each of the values of the plurality of pixels included in the minimum component data is the minimum component value Vmin of the corresponding pixel value (RGB value) of the scan data.

FIG. 6 is an explanatory diagram of the minimum component value and the maximum component value of the scan data. 6 (A) to 6 (E) show a bar graph showing RGB values of cyan (C), magenta (M), yellow (Y), black (K), and white (W) as an example of RGB values. Is illustrated in. As shown in FIG. 6, the RGB values (R, G, B) of C, M, Y, K, and W are (0, 255, 255), (255, 0, 255) (255, 255, respectively). 0), (0, 0, 0), (255, 255, 255).

As described above, the luminance Y of these RGB values can be calculated using, for example, the formula Y = 0.299 × R + 0.587 × G + 0.114 × B. The luminances of C, M, Y, K, and W (represented by values from 0 to 255) are about 186, 113, 226, 0, and 255, which are different values (FIG. 6). On the other hand, the minimum component values Vmin of C, M, Y, K, and W are 0, 0, 0, 0, 255 as shown in FIG. 6, which are the same values except for white (W). ..

FIG. 7 is a second diagram showing an example of an image used for image processing. FIG. 7A is an enlarged view of the halftone dot region described above in the scanned image SI. For example, in the example of FIG. 7A, the halftone dot region in the scanned image SI includes a plurality of M dot MDs and a plurality of Y dots YD. Here, for the sake of explanation, it is assumed that the image showing the M dot MD is a uniform image having the primary color of magenta, and the image showing the Y dot YD is a uniform image having the primary color of yellow.

FIG. 7B shows an example of the minimum component image MNI represented by the minimum component data. This minimum component image MNI corresponds to the scanned image SI of FIG. 7 (A). In the minimum component image MNI, the values of the pixels in the region MDb corresponding to the Y dot MD of the scanned image SI and the values of the pixels in the region YDb corresponding to the Y dot YD are the same as each other. As a comparative example, FIG. 7C shows a luminance image YI represented by luminance image data indicating the luminance of each pixel. This luminance image YI corresponds to the scanned image SI of FIG. 7 (A). In the luminance image YI, unlike the minimum component image MNI, the values of the pixels in the region MDd corresponding to the M dot MD of the scan image SI and the values of the pixels in the region YDd corresponding to the Y dot YD are different from each other. ..

As can be seen from the above description, in the minimum component image MNI, in the scanned image SI, the difference between the values of the plurality of pixels corresponding to the portions where the C, M, Y, and K dots are formed in the document is large. It is smaller than the luminance image YI. Then, in the minimum component image MNI, in the scanned image SI, the pixel value of the ground color region corresponding to the region indicating the ground color (white of the paper) in the document is the pixel value corresponding to the portion where the dots are formed. Will be larger than.

In S110, the CPU 210 executes a binarization process on the generated minimum component data to generate binary image data. The binary image data generated in this step is the first binary image data. For example, the CPU 210 classifies a pixel having a pixel value equal to or less than a threshold value (for example, 128) as a first character candidate pixel in the minimum component data, and classifies a pixel having a pixel value larger than the threshold value. It is not classified into 1 character candidate pixel. In the binary image data, as described above, the value of the first character candidate pixel is set to "1", and the value of the other pixels is set to "0".

According to the first binary image data generation process described above, the first binary image data is generated using the minimum component data. In general, characters that are less bright (that is, darker in color) than the background are often composed mainly of dots, and the background is often composed mainly of the white color of the paper. In the minimum component data, as described with reference to FIG. 7, the pixel value of the ground color region corresponding to the region indicating the ground color (white of the paper) in the document corresponds to the portion where the dots are formed. Since it is larger than the pixel value, it is possible to suppress specific omission of pixels constituting characters having lower brightness (that is, darker color) than the background in the first binary image data.

For example, yellow (Y) has a lower density (higher brightness) than C, M, and K. For this reason, when there are yellow characters on the background of the background color (white) of the paper, for example, even if the luminance image data indicating the brightness is binarized, the character pixels constituting the yellow characters are displayed. It may not be possible to properly identify it as a character candidate pixel. In this embodiment, even in such a case, the character pixels constituting the character of the yellow can be appropriately specified as the character candidate pixels. For this purpose, by executing the identification of the character pixels constituting the characters having the brightness lower than that of the background by using the minimum component data, for example, the characters that cannot be specified only by the luminance image data can be identified. As a result, the accuracy of specifying the character candidate pixels in the scanned image SI can be improved.

A-5: Second binary image data generation process The second binary image data generation process of S24 of FIG. 2 will be described. FIG. 8 is a flowchart of the second binary image data generation process. In S200, the CPU 210 uses the scan data to generate luminance image data. Specifically, the CPU 210 calculates the brightness Y of each pixel using the R value, G value, and B value of each pixel acquired from the scan data. The brightness Y is, for example, a weighted average of the above three components, and can be specifically calculated using the formula Y = 0.299 × R + 0.587 × G + 0.114 × B. As described above, the value of the plurality of pixels of the luminance image data indicates the luminance Y of the corresponding pixel among the plurality of pixels in the scanned image SI.

In S210, the CPU 210 executes a binarization process on the generated luminance image data to generate the binary image data. For example, the CPU 210 classifies pixels having a pixel value (that is, brightness) equal to or greater than a threshold value (for example, 128) into character candidate pixels in the luminance image data, and characterizes pixels having a pixel value less than the threshold value. Classify into pixels different from the candidate pixels. In the binary image data, as described above, the value of the character candidate pixel is set to "1", and the value of the pixel different from the character candidate pixel is set to "0". The binary image data generated in S210 is the second binary image data. In the second binary image data, as described above, the second character candidate pixel including the character pixel constituting the character having higher brightness (that is, lighter color) than the background is specified.

According to the second binary image data generation process described above, the second binary image data is generated using the luminance image data. Due to the readability of the characters, there is often a relatively large difference in brightness between the color of the characters and the color of the background. Therefore, by using the luminance image data, it is possible to suppress the specific omission of the character pixel.

Further, the second binary image data and the first binary image data can identify character pixels different from each other. For example, as described above, in the second binary image data, pixels constituting characters having higher brightness than the background can be specified as character candidate pixels, whereas in the first binary image data, the brightness is higher than that of the background. Pixels constituting characters with a low value can be specified as character candidate pixels. Further, the first binary image data is located on the background of the primary colors C, M, Y, and K used for printing, and is another primary color of C, M, Y, and K, or the white color of the paper. While it is difficult to specify the character pixels constituting the character having the above, the second binary image data can specify the character pixels constituting such a character. Therefore, by using the first binary image data and the second binary image data in combination, it is possible to further suppress the omission of specific character candidate pixels.

A-6: Block determination process The first block determination process of S22 and the second block determination process of S25 of FIG. 2 will be described. These block determination processes are executed using a machine learning model trained using a plurality of character image data showing characters and a plurality of non-character image data not showing characters. The machine learning models used in the first block determination process and the second block determination process are different from each other. The machine learning model used in the first block determination process is called a first machine learning model, and the machine learning model used in the second block determination process is called a second machine learning model. In this embodiment, the first block determination process and the second block determination process are the same except for the machine learning model used, and will be described with reference to one flowchart.

FIG. 9 is a flowchart of the block determination process. FIG. 10 is an explanatory diagram of a plurality of blocks BL arranged on the scanned image SI. As described above, the block determination process determines whether or not each of the plurality of block BLs arranged in the scan image SI is a character block indicating a character based on the block determination data for each block BL. It is a process to generate.

In S400, the CPU 210 prepares canvas data for generating block determination data in a memory (specifically, a buffer area of the volatile storage device 220). The canvas (initial image) indicated by the canvas data is an image having the same size as the scanned image SI, that is, an image having the same number of pixels. The value of each pixel of the canvas data is a predetermined initial value (for example, 0).

In S405, the CPU 210 sets a block of interest in the scanned image SI. The first block of interest is the block BL (1) in the upper left of FIG. 10 in this embodiment. One block is a rectangular area containing N pixels (N is an integer of 2 or more). Here, in FIG. 10, a plurality of squares indicated by broken lines arranged in a matrix on the scanned image SI indicate a sub-block SB. One subblock SB is a rectangular area containing k pixels (k is an integer satisfying 1 ≦ k <N). In this embodiment, the sub-block SB is an area of vertical M pixels × horizontal M pixels (M is an integer of 1 or more) (k = (M × M)). For example, one block BL is an area including sub-block SBs of L vertical × L horizontal (L is an integer of 2 or more) in this embodiment. That is, each block BL of this embodiment is a region of vertical (L × M) pixels × horizontal (L × M) pixels. In this embodiment, since M = 10 and L = 5, each block BL is an area of 50 pixels in the vertical direction and 50 pixels in the horizontal direction (N = 2500).

In S410, the CPU 210 uses a machine learning model to calculate the probability that the image in the block of interest indicates a character (referred to as the character probability Txr).

The machine learning model is a model using CNN (Convolutional Neural Network). As such a machine learning model, for example, LeNet or AlexNet is used.
LeNet, for example, "Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner (1998): Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (November 1998), 2278-2324. It is disclosed in. AlexNet, for example, "Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton (2012): ImageNet classification with deep convolutional neural networks In F. Pereira, CJC Burges, L. Bottou, & KQ Weinberger, eds. Advances in Neural Information Processing Systems 25. Curran Associates, Inc., 1097-1105.5 ”.

The input of the machine learning model is a matrix in which the values of N pixels in the block of interest (for example, RGB values and luminance values) are arranged in an order according to the position of each pixel in the block of interest. That is, the values of N pixels in the attention block are input in association with the positions of the pixels in the attention block. The output of the machine learning model is the character probability Txr described above.

For example, the character probability Txr is represented by a numerical value of 0 to 100%. As shown in S415 to S430 described later, the CPU 210 determines whether the block of interest is a character block indicating a character, a non-character block indicating no character, or an unknown block indicating a character or not, based on the character probability Txr. Determine if there is. When the character probability Txr is TH1 ≦ Txr, the CPU 210 determines that the attention block is a character block. When the character probability Txr is TH2 ≦ Txr <TH1, the CPU 210 determines that the block of interest is an unknown block. When the character probability Txr is Txr <TH2, the CPU 210 determines that the attention block is a non-character block. The threshold TH1 is, for example, 75%, and the threshold TH2 is, for example, 25%.

In this way, even if the combination of the values of N pixels is the same, different character probabilities Txr are output if the value of which pixel is in which position in the block of interest is different. A judgment is made about the block of interest based on the character probability Txr. In this way, whether the attention block is a character block, a non-character block, or an unknown block can be determined according to the positions of N pixels and the values of N pixels in the attention block. I understand.

The machine learning model is trained using, for example, a predetermined number of character image data showing characters (for example, 3000) and a predetermined number of non-character image data not showing characters (for example, 3000). There is. The character image data and the non-character image data for training are images having the same size as the block BL including N pixels. A non-character image is an image showing an object (for example, a photograph) different from characters, a solid background, or the like.

Here, in the first machine learning model used for the first block determination process and the second machine learning model used for the second block determination process, the character image data and the non-character image used for training are used. The set of data is different. The first machine learning model includes a plurality of first character image data showing characters having a brightness lower than the background, a plurality of first non-character image data showing characters having a brightness lower than the background, and a plurality of first non-character image data. Have been trained using. The second machine learning model includes a plurality of second character image data showing characters having a brightness higher than the background, a plurality of second non-character image data showing characters having a brightness higher than the background, and a plurality of second non-character image data. Have been trained using. Therefore, in the first block determination process, a block indicating a character having a brightness lower than that of the background is determined to be a character block, but a block indicating a character having a brightness higher than the background is a character block. Is not judged. Then, in the second block determination process, the block indicating a character having a brightness lower than the background is not determined to be a character block, and the block indicating a character having a brightness higher than the background is determined to be a character block. NS. In FIG. 11 below, black characters are illustrated on a white background for the purpose of explaining the operation of the processing, and this example is an example for the first block determination processing. In the second block determination process, the same operation is performed in the case where the characters are the same as those in FIG. 11 and the characters are white on a black background.

FIG. 11 is a diagram showing an example of determination for each block BL. For example, when the block BL (1) of FIG. 11 (A) and the block BL (2) of FIG. 11 (B) are attention blocks, the characters occupy a relatively wide range in the attention block, so that the characters occupy a relatively wide range. The attention block is determined to be a character block. For example, when the block BL (3) of FIG. 11C is the attention block, the attention block is an unknown block because the character is included in the attention block but the range occupied by the characters is relatively narrow. Is judged to be. For example, when the block BL (4) of FIG. 11 (D) is the attention block, the attention block is determined to be a non-character block because no character is included in the attention block. Hereinafter, the processes of S415 to S430 will be specifically described.

In S415, the CPU 210 determines whether or not the character probability Txr calculated in S410 is equal to or greater than the threshold value TH1. When the character probability Txr is equal to or greater than the threshold value TH1 (S415: YES), the attention block is determined to be a character block. Therefore, in this case, in S420, the CPU 210 sets the values of all the pixels in the block of interest to the values indicating the characters. If the character probability Txr is less than the threshold TH1 (S415: NO), S420 is skipped.

FIG. 12 is a diagram showing an example of setting a pixel value in the block determination data. 12 (A) to 12 (D) conceptually show the block determination image BI indicated by the block determination data. When the block BL (1) of FIG. 11 (A) and the block BL (2) of FIG. 11 (B) are attention blocks, it is determined that the attention block is a character block. Also, as shown in FIGS. 12A and 12B, the values of all the pixels in the blocks BL (1) and BL (2) are set to the value "1" indicating the character.

In S425, the CPU 210 determines whether or not the character probability Txr is less than the threshold TH2. When the character probability Txr is less than the threshold value TH2 (S425: YES), the attention block is determined to be a non-character block. Therefore, in this case, in S430, the CPU 210 sets the values of all the pixels in the block of interest to values indicating non-characters. When the character probability Txr is equal to or higher than the threshold value TH2 (S425: NO), S430 is skipped.

When the block BL (4) of FIG. 11 (D) is the block of interest, it is determined that the block of interest is a non-character block. Therefore, also in the block determination image BI, as shown in FIG. 12 (D). , The values of all the pixels in the block BL (4) are set to the value "2" indicating non-characters.

When the character probability Txr is equal to or higher than the threshold value TH2 and less than the threshold value TH1 (S415: NO and S425: NO), the attention block is determined to be an unknown block. Therefore, in this case, the values of all the pixels in the block of interest are not changed. That is, at this point, the pixel having the value "1" indicating the character is maintained as the value indicating the character, and the pixel having the value "2" indicating the non-character is maintained as the value indicating the non-character. Pixels having a value "0" indicating unknown are maintained at the value indicating unknown.

When the block BL (3) of FIG. 11 (C) is the block of interest, it is determined that the block of interest is an unknown block. Therefore, in the block determination image BI, as shown in FIG. 12 (C), the block is blocked. The values of all the pixels in BL (4) are maintained unchanged.

In S435, the CPU 210 moves the attention block to the right by M pixels. That is, the attention block is moved to the right by one subblock SB. For example, when the block BL (1) in FIG. 10 is the block of interest, the block BL (2) is set as a new block of interest. When the block BL (q-1) of FIG. 10 is the block of interest, the block BL (q) is set as a new block of interest.

In S440, the CPU 210 determines whether or not the right end of the attention block has moved to the right side of the right end of the scan image SI as a result of moving the attention block by M pixels to the right. That is, it is determined whether or not the new attention block after the movement protrudes to the right side of the scanned image SI. For example, when the new attention block is the block BL (q) or the block BL (e) of FIG. 10, it is determined that the right end of the attention block has moved to the right side of the right end of the scanned image SI.

When the right end of the attention block has not moved to the right side of the right end of the scanned image SI (S440: NO), the CPU 210 returns to S410. In this way, for example, while shifting the attention block by M pixels to the right, the determination for each block (S410 to S430) is sequentially performed. In the example of FIG. 10, it is determined whether each block BL is a character block, a non-character block, or an unknown block in the order of blocks BL (1), BL (2), and BL (3).

When the right end of the attention block is moved to the right side of the right end of the scan image SI (S440: YES), in S445, the CPU 210 moves the attention block to the left end of the scan image SI, and in S450, attention is paid. The block is moved downward by M pixels.

In S455, the CPU 210 determines whether or not the lower end of the attention block has moved below the lower end of the scanned image SI as a result of moving the attention block downward by M pixels. That is, it is determined whether or not the new attention block after the movement protrudes below the scanned image SI. For example, when the new attention block is the block BL (e + 1) of FIG. 10, it is determined that the lower end of the attention block has moved below the lower end of the scanned image SI. For example, when the new attention block after movement is the block BL (e + 1) in FIG. 10, it is determined that the lower end of the attention block has moved below the lower end of the scanned image SI.

If the lower end of the attention block has not moved below the lower end of the scanned image SI (S455: NO), the CPU 210 returns to S410. In this way, for example, while shifting the block of interest downward by M pixels, the block BL for one line from the left end to the right end is sequentially determined line by line. For example, next to the rightmost block BL (q-1) in FIG. 10, the block of interest to be determined is the leftmost block BL (q + 1) in the lower row by M pixels.

When the lower end of the block of interest moves below the lower end of the scanned image SI (S455: YES), the determination of all the blocks BL is completed, so the CPU 210 proceeds to S460.

In S460, the CPU 210 determines whether or not a value “0” indicating unknown remains in the block determination data. If a value indicating unknown remains, the CPU 210 sets the value indicating unknown to the value "1" indicating a character in S465. As a result, the value of each pixel of the block determination data becomes either a value "1" indicating a character or a value "2" indicating a non-character.

In S470, the CPU 210 changes the non-character value "2" to "0" and converts the block determination data into binary image data having either a value of "1" or "0". As a result, it is shown that a character is a value, that is, a value "1" indicating that it is a pixel constituting the above-mentioned character block, and a value indicating a non-character, that is, a pixel constituting the above-mentioned non-character block. Block determination data having any of the values "0" and the value "0" for each pixel is generated.

According to the present embodiment described above, a plurality of character candidate pixels are extracted using the scan data as the target image data (S21 and S24 in FIG. 2), and the scan data is used on the scan image SI. Whether or not each of the plurality of blocks BL to be arranged is a character block indicating a character is determined for each block (S22 and S25 in FIG. 2). The determination for each block BL is executed using a machine learning model trained using a plurality of character image data and a plurality of non-character image data (S410 in FIG. 9). Then, using the determination result for each block, a plurality of character pixels are specified from the plurality of character candidate pixels (S23, S26, S27 in FIG. 2). As a result, as a result, the character pixels in the scanned image SI can be identified with high accuracy.

FIG. 13 is a diagram illustrating the effect of the embodiment. 13 (A) to 13 (D) show a scan image SI indicated by scan data, a first binary image CI1 indicated by first binary image data, and a first block determination data indicated by first block determination data. An example of each of the block determination image BI1 and the first character identification image TI1 shown by the first character identification data is conceptually shown. In FIG. 13, the squares shown by the broken lines of these images SI, CI1, BI1, and TI1 each indicate the pixel Px.

The scanned image SI may include a non-character object (for example, a photograph) or a colored background as well as the character Tx. In the example of FIG. 13A, the scanned image SI may include the constituent halftone dots DT together with the character Tx. This is because, as described above, the scan data is the data generated by reading the printed matter. In such a case, it is relatively difficult to specify only the character pixel. For example, edge extraction that smoothes the image of the scan data and extracts the edge pixels in the image from the smoothed scan data so that the halftone dot DT is not erroneously specified. A method of processing can be considered. In this method, the extracted edge pixels are specified as character pixels. In this method, if the halftone dot DT cannot be sufficiently smoothed, the edge of the halftone dot DT can be erroneously identified as a character pixel. Further, if the image is excessively smoothed in order to sufficiently smooth the halftone dot DT, the edges of the characters are excessively blurred, and the identification accuracy of the character pixels is lowered. For this purpose, for example, as shown in the first binary image CI1 of FIG. 13B, in the first binary image data, not only the pixel Cpt constituting the character Tx but also the halftone dot DT is configured. The pixel Cpd may also be specified as the first character candidate pixel.

On the other hand, in the block determination process of S22 and S25 of this embodiment, whether or not the block is a character block is determined for each block BL according to the positions of N pixels in the block and the values of N pixels. Since the judgment including is performed, the spatial resolution is coarser than the judgment for each pixel, but the judgment error is relatively small. Further, in this embodiment, since the judgment for each block BL is executed by using the machine learning model, it is sufficient to sufficiently train the machine learning model to determine whether or not each block is a character block. Can be judged with high accuracy. Therefore, for example, in the first block determination image BI1 of FIG. 13C, for example, the area including the character Tx is specified as the character block TB, and the area including the halftone dot DT is designated as the non-character block OB. Be identified.

As a result, if the first character identification data is generated by taking the logical product of the first character identification data and the first block determination data, the character pixels can be appropriately specified. For example, as shown in the character identification image TI1 of FIG. 13 (D), the pixels constituting the character Tx are specified as character pixels, and the pixels indicating the halftone dot DT are not specified as character pixels. The same applies to the second binary image data, the second block determination data, and the second character identification data obtained by taking the logical product of these data.

Further, according to the present embodiment, the CPU 210 adjusts the intensity of the edge in the image with respect to the target image data in the first binary image data generation process and the second binary image data generation process. A plurality of character candidate pixels are extracted without executing the adjustment process (FIGS. 5 and 8). As a result, a plurality of character candidate pixels can be extracted with high accuracy. The edge adjustment process includes a smoothing process for smoothing the image to reduce the intensity of the edges in the image and an edge enhancement process for increasing the intensity of the edges in the image. The smoothing process includes, for example, a process using a simple average filter and a process using a Gaussian filter. The edge enhancement process includes, for example, an unsharp mask process and a process using a Laplacian filter. For example, if the scan data is smoothed to eliminate halftone dot DTs as described above, or after the smoothing process, edge enhancement processing is further performed to remove the edges of blurred characters. I would like to emphasize it. By doing so, it is possible to reduce the inconvenience that the pixels constituting the halftone dot DT are erroneously specified as character pixels, but the edges of the characters also change, so that the thickness of the characters may change. As a result, there is an inconvenience that the character pixel is specified so as to form a character thicker than the character in the scanned image SI, and a character pixel is specified so as to form a character thinner than the character in the scanned image SI. This can occur and reduce the accuracy of character pixel identification. For example, it may occur that a small character is specified in a crushed state, or a character is specified in a state where the line drawing is interrupted. In this embodiment, by taking the logical product of the first binary image data or the second binary image data and the block determination data, an object other than characters such as halftone dot DT and pixels constituting the background can be obtained. Exclude. Therefore, in the first binary image data and the second binary image data, there is no problem even if the pixels constituting the halftone dot DT are erroneously specified as the character candidate pixels. Therefore, in the first binary image data generation process and the second binary image data generation process, it is not necessary to execute the smoothing process or the edge enhancement process on the target image data. As a result, since the edges of the characters are not changed, a plurality of character candidate pixels can be extracted with high accuracy.

Further, in the first binary image data generation process of this embodiment, the CPU 210 uses the scan data to generate the minimum component data (S100 in FIG. 5). Each of the values of the plurality of pixels of the minimum component data is a value based on the minimum value of the three component values of the RGB values of the corresponding scan data. Then, a plurality of first character candidate pixels are extracted using the minimum component data (S110 in FIG. 5). As a result, character candidate pixels that are difficult to extract when the target image data is binarized as it is can be extracted. For example, as described above, even when there are yellow characters on the background of the background color (white) of the paper, character candidate pixels including the pixels constituting the characters can be extracted.

Further, in the second binary image data generation process of this embodiment, the CPU 210 uses the scan data to generate luminance image data (S200 in FIG. 8). The CPU 210 uses the luminance image data to extract pixels having a luminance higher than the reference as character candidate pixels (S210 and S220 in FIG. 8). As a result, it is possible to extract character candidate pixels having relatively high brightness, which is difficult to extract when the target image data is binarized as it is. For example, as described above, character candidate pixels including pixels constituting characters constituting characters having higher brightness than the background can be extracted.

Further, in this embodiment, the CPU 210 executes the first binary image data generation process to extract a plurality of first character candidate pixels (S21 in FIG. 2), and the second binary image data. The generation process is executed to extract a plurality of second character candidate pixels (S24 in FIG. 2). Then, the CPU 210 uses the determination result for each block BL to specify a plurality of first character pixels from the plurality of first character candidate pixels (S23 in FIG. 2), and for each block BL. Using the determination result, a plurality of second character pixels are identified from the plurality of second character candidate pixels, and the plurality of first character pixels and the plurality of second character pixels are selected. A plurality of including pixels are finally specified as character pixels (S27 in FIG. 2). As a result, it is possible to suppress specific omission of character pixels.

Further, in the first block determination process (S22 in FIG. 2) of the present embodiment, the CPU 210 uses the first machine learning model to determine whether or not each of the plurality of blocks BL has a character block for each block. (Fig. 9). In the second block determination process (S25 in FIG. 2), the CPU 210 uses a second machine learning model different from the first machine learning model, and whether or not each of the plurality of blocks BL has a character block. Is determined for each block (Fig. 9). The CPU 210 identifies a plurality of first character pixels by using the judgment result using the first machine learning model (S23 in FIG. 2), and uses the judgment result using the second machine learning model. The second character pixel is specified (S26 in FIG. 2). In this way, since the determination results using different machine learning models are used when specifying the first character pixel and when specifying the second character pixel, the first character pixel and the second character pixel are used. Character pixels can be appropriately identified.

Further, in the present embodiment, the first machine learning model used in the first block determination process (S22 in FIG. 2) includes a plurality of first character image data and a plurality of first non-character image data. The second machine learning model used in the second block determination process (S25 in FIG. 2) is a machine learning model trained using the above, and is different from the plurality of first character image data. It is a machine learning model trained using the second character image data of the above and a plurality of second non-character image data different from the plurality of first non-character image data. As described above, the first machine learning model and the second machine learning model are models trained using images different from each other. As a result, the first character pixel (in this embodiment, a pixel constituting a character having a brightness lower than the background) and the second character pixel (in this embodiment, a pixel constituting a character having a brightness higher than the background) Can be identified more appropriately.

As can be seen from the above description, the minimum component data of the above embodiment is an example of the first image data, and the luminance image data is an example of the second image data. The first binary image data generation process is an example of the first extraction process, and the second binary image data generation process is an example of the second extraction process.

B. Modification example:

(1) In the above embodiment, the first block determination process (S21 in FIG. 2) and the second block determination process (S24 in FIG. 2) use different machine learning models. Instead of this, the same machine learning model may be used in the first block determination process and the second block determination process. For example, in the embodiment, a plurality of first character images showing characters having a brightness lower than the background, second character image data showing characters having a brightness higher than the background, and characters having a brightness lower than the background. A machine learning model trained using a plurality of non-character image data that does not show any of the characters having a brightness higher than the background may be used in the two block determination processes.

(2) In the above embodiment, the first machine learning model used in the first block determination process and the second machine learning model used in the second block determination process are different from each other in character image data and non-character image data. It is trained using character image data. Alternatively, the first machine learning model and the second machine learning model may be trained using the same character image data and non-character image data. In this case, the structure of the neuros network such as the number of convolution layers and the number of pooling layers may be different from each other between the first machine learning model and the second machine learning model.

(3) In the above embodiment, the final character pixel is specified by using both the first binary image data and the second binary image data. Instead of this, the character pixel may be specified by using only one of the first binary image data and the second binary image data. For example, when only the first binary image data is used, S24 to S27 in FIG. 2 may be omitted.

(4) Luminance image data is used in the second binary image data generation process (FIG. 8) (S200). Instead of this, for example, the average component value image data in which the average value of the three component values (R value, G value, B value) included in the RGB values of the corresponding pixels of the scan data is the value of each pixel. May be used.

(5) In the first binary image data generation process (FIG. 5) of the above embodiment, the minimum component data is used (S100). Instead of this, the maximum component data or the inverted minimum component data may be used.

The maximum component data includes a plurality of values corresponding to a plurality of pixels included in the scan data, and each of the plurality of values is the maximum component value Vmax of the corresponding pixel of the scan data. The maximum component value Vmax is the maximum value among a plurality of component values (R value, G value, B value) included in the RGB values of the corresponding pixels of the scan data.

The inversion minimum component data is acquired as follows. First, for each of the values (RGB values) of the plurality of pixels included in the scan data, the inverted color values in which the plurality of component values (R value, G value, B value) are inverted are generated. Assuming that the RGB values before inversion are (Rin, Gin, Bin), the inverted RGB values (Rout, Gout, Bout) are represented by the following equations (1) to (3).

Rout = Rmax-Rin ... (1)
Gout = Gmax-Gin ... (2)
Bout = Bmax-Bin ... (3)

Here, Rmax, Gmax, and Bmax are the maximum values that the R value, G value, and B value can take, respectively, and in this embodiment, Rmax = Gmax = Bmax = 255. Image data in which these inverted RGB values are the values of a plurality of pixels is generated as inverted image data. Then, the inverted minimum component data is generated using the inverted image data. Specifically, the inverted minimum component value VRmin is acquired from each of the plurality of inverted RGB values included in the inverted image data. The inverted minimum component value VRmin is the minimum value among a plurality of component values (R value, G value, B value) included in the inverted RGB value. The inversion minimum component data is image data in which these inversion minimum component values VRmin are values of a plurality of pixels.

The inverted minimum component value VRmin is an inverted value of the maximum component value, and the relationship of VRmin = (255-Vmax) is established. Therefore, both the maximum component data and the inverted minimum component data are values based on the maximum value among the plurality of component values included in the value of each pixel of the scan data (the inverted value of the maximum value, or the inverted value of the maximum value, or It can be said that the maximum value itself) is the image data in which the pixel value is used.

As shown in FIG. 6, the maximum component values Vmax of C, M, Y, K, and W are 255, 255, 255, 0, and 255, which are the same values except for black (K). Therefore, in the maximum component data and the inverted minimum component data, among the five types of elements constituting the halftone dot region, that is, the dots of C, M, Y, and K, and the background color (white) of the paper. The difference between the values indicating the four types of elements (dots C, M, Y and the background color (white) of the paper) is suppressed. As a result, when the maximum component data or the inverted minimum component data is used, it is possible to prevent the pixels constituting the halftone dots from being specified as the character candidate pixels, as in the case of using the minimum component data.

(6) In the first binary image data generation process (FIG. 5) of the above embodiment, the minimum component data is used (S100). Instead of this, for example, luminance image data may be used. That is, in the first binary image data generation process, in order to identify the character pixels constituting the character having a brightness lower than that of the background, the brightness image data that is not inverted is used, and the second binary image is used. In the data generation processing, inversion-processed brightness image data may be used in order to identify character pixels constituting characters having a brightness higher than that of the background.

(7) In each of the above embodiments, character sharpening processing is executed for character pixels (S40 in FIG. 2), and halftone dot smoothing processing is executed for non-character pixels (S30 in FIG. 2). ). Instead of this, antialiasing processing for improving the appearance of the character may be executed on the character pixel. Further, for non-character pixels, for example, in order to reduce the amount of color material used during printing, a process of skipping colors (a process of converting to white) may be executed. In general, it is preferable that character pixels and non-character pixels are subjected to different image processing. Alternatively, the specific image processing may be executed on either the character pixel or the non-character pixel, and the specific image processing may not be executed on the other.

(8) In the block determination process of FIG. 9 of the above embodiment, if a value indicating unknown remains in the block determination data after the determination for all the block BLs (S460: YES), the CPU 210 determines. In S465, the value indicating unknown is set to the value indicating characters. This is to prevent a part of the character pixel from being mistakenly identified as a non-character pixel and to avoid inconvenience such as a part of the character being blurred. For example, when it is important to prevent a part of non-character pixels from being mistakenly identified as a character pixel and to avoid inconveniences such as conspicuous halftone dots, the CPU 210 makes an unknown question in S465. The indicated value may be set to a value indicating a non-character.

(9) In the above embodiment, character identification data is finally generated using the first binary image data and the second binary image data (S21 to S27 in FIG. 2). Instead of this, character identification data may be generated using the first binary image data, the second binary image data, and the third binary image data. For example, between S26 and S27 in FIG. 2, it is generated using the above-mentioned maximum component data, the maximum component data is binarized, and a third binary image data is generated. Then, the third character identification data is generated by taking the logical product of the third binary image data and the first block determination process generated in S22. Then, in S27 of FIG. 2, the first character identification data (S23 of FIG. 2) generated using the first binary image data and the second binary image data generated using the second binary image data are used. The final character identification data is generated by taking the logical sum of the character identification data (S26 in FIG. 2) and the third character identification data generated by using the third binary image data. You may. Thereby, the specific omission of the character pixel can be further suppressed.

(10) In the block determination process of the above embodiment, it is determined whether or not the block is a character block while shifting the attention block of vertical (L × M) pixels × horizontal (L × M) pixels by M pixels. A plurality of blocks arranged on the scanned image SI overlap each other (FIG. 10). Instead of this, a plurality of blocks may be arranged on the scanned image SI so that the plurality of blocks do not overlap each other.

(11) In the block determination process of the above embodiment, the CPU 210 determines whether the block of interest is a character block, a non-character block, or an unknown block. Instead, the CPU 210 may determine whether the block of interest is a character block or a non-character block. In this case, for example, the threshold value TH1 used in S415 and the threshold value TH2 used in S425 may be set to the same value. For example, TH1 = TH2 = 50% may be set.

(12) In the above embodiment, as described above, as shown in S420 and S430 of FIG. 9, when the attention block is determined to be a character block or a non-character block, in the block determination data, the attention block is contained. The values of all the pixels are set according to the judgment result. Instead of this, in the block determination data, of the N pixels in the block of interest, only the value of the pixel having a value indicating unknown may be set according to the determination result. That is, it is determined that both the first block and the second block that overlap each other are blocks other than the unknown block (that is, a character block or a non-character block). In this case, regarding the overlapping area of the first block and the second block, even if the judgment result of the block whose judgment processing order is earlier than that of the first block and the second block is prioritized. good.

(13) In the above embodiment, the target image data is scan data, but is not limited to this. The target image data may be generated by reading the printed matter with a digital camera equipped with a two-dimensional image sensor. Further, the target image data may be image data created by using an application program such as drawing creation or document creation.

(14) The image processing device that realizes the image processing of FIG. 2 is not limited to the multifunction device 200, and may be various devices. For example, the scanner or the digital camera may execute the image processing of FIG. 2 in order to generate print data to be supplied to the printer by using the image data generated by the scanner or the digital camera. Further, for example, a connected terminal device (for example, terminal device 100) or a server (not shown) capable of communicating with the scanner or printer executes the image processing of FIG. 2 using the scan data acquired from the scanner. , Print data may be generated and the print data may be supplied to the printer. Further, a plurality of computers (for example, a cloud server) capable of communicating with each other via a network may partially share the functions required for image processing and execute image processing as a whole. In this case, the entire plurality of computers is an example of an image processing device.

(15) In each of the above embodiments, a part of the configuration realized by the hardware may be replaced with software, and conversely, a part or all of the configuration realized by the software may be replaced with the hardware. You may do so. For example, the process of calculating the character probability Txr using the machine learning model of S410 in FIG. 9 may be executed by dedicated hardware such as ASIC.

Although the present invention has been described above based on Examples and Modifications, the above-described embodiments of the invention are for facilitating the understanding of the present invention and do not limit the present invention. The present invention can be modified and improved without departing from the spirit and claims, and the present invention includes equivalents thereof.

100 ... Terminal device, 200 ... Complex machine, 210 ... CPU, 220 ... Volatile storage device, 230 ... Non-volatile storage device, 240 ... Display unit, 250 ... Operation unit, 270 ... Communication IF, 280 ... Print execution unit, 290 ... Reading execution unit, Vmin ... Minimum component value, Vmax ... Maximum component value, VRmin ... Inverted minimum component value, D1 ... 1st direction, D2 ... 2nd direction, SB ... Subblock, TB ... Character block, OB ... Non-character Block, PG ... computer program, TI ... character specific image, GI ... smoothed image, FI ... processed image, YI ... brightness image, BI ... block judgment image, SI ... scanned image, BL ... block, DT ... halftone dot, Px ... Pixel, Tx ... Character, Ob1 to Ob7 ... Object, MNI ... Minimum component image, Bg1 to Bg3 ... Background

Claims (9)

It is an image processing device
An image acquisition unit that acquires target image data indicating the target image,
It is a candidate pixel extraction unit that extracts a plurality of character candidate pixels that are candidates for a plurality of character pixels by using the target image data, and each of the plurality of character pixels has a character on the target image. The candidate pixel extraction unit, which is a pixel constituting a character in the arranged area,
Using the target image data, it is a determination unit that determines for each block whether or not each of the plurality of blocks arranged on the target image is a character block, and the character block is on the target image. It is a block corresponding to the area where the characters of are arranged, and the judgment for each block is a machine trained using a plurality of character image data indicating characters and a plurality of non-character image data not indicating characters. The determination unit, which is executed using the learning model, includes a first machine learning model and a second machine learning model different from the first machine learning model .
Using the judgment result by the judgment unit, a character pixel identification unit that specifies the plurality of character pixels from the plurality of character candidate pixels, and a character pixel identification unit.
Equipped with a,
The candidate pixel extraction unit
The first extraction process is executed to extract a plurality of first character candidate pixels among the plurality of character candidate pixels.
A second extraction process different from the first extraction process is executed to extract a plurality of second character candidate pixels among the plurality of character candidate pixels.
The judgment unit
Using the first machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
Using the second machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
The character pixel identification unit is
Using the determination result using the first machine learning model, a plurality of first pixels are identified from the plurality of first character candidate pixels.
Using the determination result using the second machine learning model, a plurality of second pixels are identified from the plurality of second character candidate pixels.
Wherein that identifies the plurality of character pixels and a second pixel a plurality of first pixels and the plurality, the image processing apparatus. The image processing apparatus according to claim 1.
The candidate pixel extraction unit is an image processing device that extracts a plurality of character candidate pixels without executing edge adjustment processing for adjusting the intensity of edges in the target image with respect to the target image data. The image processing apparatus according to claim 1 or 2.
The plurality of character image data includes a plurality of first character image data and a plurality of second character image data different from the plurality of first character image data.
The plurality of non-character image data includes a plurality of first non-character image data and a plurality of second non-character image data different from the plurality of first non-character image data. ,
The first machine learning model is the machine learning model trained using the plurality of first character image data and the plurality of first non-character image data.
The second machine learning model is an image processing apparatus that is the machine learning model trained using the plurality of second character image data and the plurality of second non-character image data. .. The image processing apparatus according to claim 3.
The target image data includes color values of a plurality of pixels.
The color value includes a plurality of component values and contains a plurality of component values.
The first extraction process is
Using the target image data, it is the first image data including a plurality of first values corresponding to the color values of the plurality of pixels, and each of the plurality of first values is the corresponding color. The process of generating the first image data, which is a value based on either the minimum value or the maximum value of the plurality of component values of the value, and
A process of identifying the plurality of first pixels by binarizing the first image data, and
Including
The second extraction process is
A process of using the target image data to generate second image data including a plurality of second values indicating the brightness of the corresponding pixel among the plurality of pixels in the target image.
A process of specifying the plurality of second pixels by binarizing the second image data so that a pixel having a brightness higher than the reference is specified as the character candidate pixel.
Including
In the first machine learning model, the plurality of first character image data showing the first character, which is a character having a brightness lower than that of the background, and the plurality of first characters not showing the first character. The machine learning model trained using the non-character image data of
In the second machine learning model, the plurality of second character image data showing the second character, which is a character having a brightness higher than that of the background, and the plurality of second characters not showing the second character. An image processing apparatus, which is the machine learning model trained using the non-character image data of. The image processing apparatus according to claim 1 or 2.
The plurality of character image data indicates a plurality of first character image data indicating a first character which is a character having a brightness lower than that of the background, and a second character which is a character having a brightness higher than that of the background. Includes a plurality of second character image data and
The first machine learning model is the machine learning model trained using the plurality of first character image data.
The second machine learning model is an image processing device that is the machine learning model trained using the plurality of second character image data. The image processing apparatus according to any one of claims 1 to 5.
Among the target image data, the first image processing is executed on the identified values of the plurality of character pixels, and the first image is applied to the values of pixels different from the plurality of character pixels. An image processing apparatus including an image processing unit that executes a second image processing different from the processing to generate the target image data that has been image-processed. The image processing apparatus according to claim 6.
An image processing apparatus including a print data generation unit that generates print data using the target image data that has been image-processed. It ’s a computer program,
An image acquisition function that acquires target image data indicating the target image, and
It is a candidate pixel extraction function that extracts a plurality of character candidate pixels that are candidates for a plurality of character pixels by using the target image data, and each of the plurality of character pixels has a character on the target image. The candidate pixel extraction function, which is a pixel constituting a character in the arranged area,
Using the target image data, it is a determination function for determining for each block whether or not each of the plurality of blocks arranged on the target image is a character block, and the character block is on the target image. It is a block corresponding to the area where the characters of are arranged, and the judgment for each block is a machine trained using a plurality of character image data indicating characters and a plurality of non-character image data not indicating characters. The determination function, which is executed using the learning model, includes a first machine learning model and a second machine learning model different from the first machine learning model .
A character pixel identification function for identifying the plurality of character pixels from the plurality of character candidate pixels using the determination result by the determination function, and a character pixel identification function.
To the computer ,
The candidate pixel extraction function
The first extraction process is executed to extract a plurality of first character candidate pixels among the plurality of character candidate pixels.
A second extraction process different from the first extraction process is executed to extract a plurality of second character candidate pixels among the plurality of character candidate pixels.
The judgment function
Using the first machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
Using the second machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
The character pixel identification function is
Using the determination result using the first machine learning model, a plurality of first pixels are identified from the plurality of first character candidate pixels.
Using the determination result using the second machine learning model, a plurality of second pixels are identified from the plurality of second character candidate pixels.
Wherein that identifies a plurality of character pixels, the computer program and a second pixel of the plurality the first pixel of the plurality. It is an image processing method
An image acquisition process for acquiring target image data indicating a target image, and
This is a candidate pixel extraction step of extracting a plurality of character candidate pixels that are candidates for a plurality of character pixels using the target image data, and each of the plurality of character pixels has a character on the target image. The candidate pixel extraction step, which is a pixel constituting a character in the arranged area,
It is a determination step of determining for each block whether or not each of the plurality of blocks arranged on the target image is a character block using the target image data, and the character block is on the target image. It is a block corresponding to the area where the characters of are arranged, and the judgment for each block is a machine trained using a plurality of character image data indicating characters and a plurality of non-character image data not indicating characters. The determination step, which is executed using the learning model, includes a first machine learning model and a second machine learning model different from the first machine learning model .
A character pixel specifying step of specifying the plurality of character pixels from the plurality of character candidate pixels by using the judgment result in the determination step.
Equipped with a,
The candidate pixel extraction step is
The first extraction process is executed to extract a plurality of first character candidate pixels among the plurality of character candidate pixels.
A second extraction process different from the first extraction process is executed to extract a plurality of second character candidate pixels among the plurality of character candidate pixels.
The judgment step is
Using the first machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
Using the second machine learning model, it is determined for each block whether or not each of the plurality of blocks is the character block.
The character pixel identification step is
Using the determination result using the first machine learning model, a plurality of first pixels are identified from the plurality of first character candidate pixels.
Using the determination result using the second machine learning model, a plurality of second pixels are identified from the plurality of second character candidate pixels.
Wherein that identifies a plurality of character pixels, the image processing method and a second pixel of the plurality the first pixel of the plurality.

Images