JP7371466B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing device.

Patent Document 1 discloses that an input image including an object is acquired, a changing area image that is an image of a changing area is extracted from the input image using a background image, and the input image and changing area image are combined and folded. A technique is described in which a feature image of R, G, B, and O is extracted by using a built-in neural network, and the position of an object is detected from the feature image.

Japanese Patent Application Publication No. 2017-191501

However, the information processing device described in Patent Document 1 has a problem in that the accuracy of the position of the object extracted from the feature image is low. For example, when determining the type of a certain image, it is difficult to accurately indicate which characteristic part of the image was focused on in making this determination. Furthermore, if the number of characteristic parts used in the determination is not singular but plural, it becomes even more difficult to accurately indicate which characteristic part was focused on in making the determination.

The present invention has been made in view of the above-mentioned problems, and when determining the type of an image based on the object included in the image, regardless of whether the object is singular or plural, the object is The purpose is to maintain high accuracy in accurately extracting and determining the type of image.

An image processing device according to one aspect of the present invention includes a processing unit including a generation unit, a convolutional neural network unit, a GLAD cam unit, a storage unit, a comparison unit, an output unit, and a correction unit, The generation unit generates a density map from image data to be processed, and the convolutional neural network unit applies a filter to the density map to generate a feature map, and generates classification data from the feature map. The GRAD cam unit performs processing to generate an activation map from the feature map using an activation function, divides the activation map into a plurality of groups by clustering, and applies a correction function to each of the plurality of groups. , the storage unit stores training data, and the comparison unit performs a process of calculating a first loss function for the classification data using the classification data and the training data, The output unit calculates a second loss function for each of the plurality of groups by adding the correction function to the first loss function calculated by the comparison unit, and the correction unit calculates a second loss function for each of the plurality of groups. The weighting of the filter used in the convolutional neural network unit is corrected using a correction value that is the sum of the calculated second loss functions, and the convolution is performed using the new filter created by the correction unit. The filter is corrected and updated by repeating the processing by the neural network section, the grad cam section, the comparison section, the output section, and the correction section.

According to the present invention, when determining the type of an image based on an object included in the image, regardless of whether the object is singular or plural, the object is accurately extracted and the image is The accuracy of determining the type of can be maintained high.

1 is a diagram showing an internal configuration of an image forming apparatus according to an embodiment of an image processing apparatus according to the present invention. FIG. 1 is a block diagram showing the electrical configuration of an image forming apparatus. (A) is a diagram showing a manuscript, (B) is a diagram showing a density map created from image data obtained by reading the manuscript, and (C) is a diagram showing a filter used for the density map. (A) to (C) are diagrams illustrating processing by a convolutional neural network unit. (A) to (C) are diagrams illustrating processing by a convolutional neural network unit. (A) to (C) are diagrams illustrating processing by a convolutional neural network unit. It is a figure explaining activation function ReLU. FIG. 3 is a diagram showing an example of an activation map. (A) is a diagram showing an example of teacher data, classification data, and difference data, and (B) is a diagram showing a transition in which the filter used by the convolutional neural network unit is corrected. (A) to (C) are diagrams showing examples of teacher data, classification data, and difference data. (A) to (C) are diagrams showing how the activation map changes each time the filter is corrected.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing apparatus and an image processing method according to an embodiment of the present invention will be described below with reference to the drawings. In the following description, the same or similar parts will be denoted by the same reference numerals, and repeated description will be omitted.

An image processing apparatus according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram showing the internal configuration of an image forming apparatus according to an embodiment of an image processing apparatus according to the present invention.

The image forming apparatus 100 is a multifunction device that has the functions of a copying machine, a printer, and a facsimile.

As shown in FIG. 1, the image forming apparatus 100 includes a document transport section 2, a reading section 4, a feeding section 6, a transport section 8, an image forming section 10, an ejection section 12, and a processing section 20. Equipped with Furthermore, the image forming apparatus 100 includes an operation section 5 and a network interface section 91 (FIG. 2).

The document transport unit 2 transports the document G placed on the tray 80. The document transport section 2 may include a pickup roller 82 and a plurality of transport rollers 84. Further, an example of the document transport unit 2 is an ADF (Auto Document Feeder). The original G is paper or a plastic sheet used in a projector.

The reading unit 4 reads the image of the document G. The reading unit 4 reads an image and generates image data. The reading section 4 includes a light source 86, a plurality of reflective mirrors 88, a lens 90, and an imaging section 92. An example of the reading unit 4 is a scanner.

Light source 86 has a plurality of light emitting elements. The light emitting element is, for example, a light emitting diode (LED). The light emitted from the light source 86 is reflected by the original G being transported through the original transporting section 2 , is reflected by a plurality of reflecting mirrors 88 , passes through a lens 90 , and reaches the imaging section 92 .

The imaging unit 92 includes a plurality of light receiving elements that receive light from the lens 90. The imaging unit 92 is, for example, a charge coupled device (CCD). The imaging section 92 generates an analog electrical signal from the light that has reached the imaging section 92 . Thereafter, in an A/D converter (not shown), the analog signal is converted into a digital signal, and the digital signal constitutes image data. The reading section 4 outputs the image data to the processing section 20.

The feeding section 6 accommodates a plurality of sheets S, and feeds the sheets S to the conveying section 8. The sheet S is made of paper or synthetic resin, for example. The conveyance section 8 includes a plurality of conveyance roller pairs, and conveys the sheet S to the image forming section 10.

The image forming section 10 forms a toner image on the sheet S using an electrophotographic method. Specifically, the image forming section 10 includes a photosensitive drum, a charging device, an exposure device, a developing device, a replenishing device, a transfer device, a cleaning device, and a static eliminator.

The toner image shows, for example, an image of a document G. The discharge unit 12 discharges the sheet S to the outside of the image forming apparatus 100.

Next, with reference to FIG. 2, the electrical configuration of the image forming apparatus 100 according to this embodiment will be described. FIG. 2 is a block diagram showing the electrical configuration of the image forming apparatus 100 according to this embodiment. 3A is a diagram showing a document, FIG. 3B is a diagram showing a density map created from image data obtained by reading the document, and FIG. 3C is a diagram showing a filter used for the density map.

The processing unit 20 includes a processor, a RAM (Random Access Memory), a ROM (Read Only Memory), and a dedicated hardware circuit. The processor is, for example, a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or an MPU (Micro Processing Unit). The processing section 20 includes a generation section 22, a convolutional neural network section 24, a GLAD cam section 26, a storage section 28, a comparison section 30, an output section 32, a correction section 34, and a classification section 36. There is.

The processing section 20 generates the generation section 22, the convolutional neural network section 24, the GLAD cam section 26, and the memory by the operation of the processor according to the control program stored in the HDD 111 included in the image forming apparatus 100 or the ROM. It functions as a section 28, a comparison section 30, an output section 32, a correction section 34, and a classification section 36. However, the generating section 22 to the classifying section 36 may be configured by hardware circuits instead of operating according to the control program by the processor.

The generation unit 22, the convolutional neural network unit 24, the GLAD cam unit 26, the storage unit 28, the comparison unit 30, the output unit 32, and the correction unit 34 process image data obtained by reading a document by the reading unit 4, for example. Perform the processing shown below. For example, as shown in FIG. 3A, manuscript G is a document in which a corporate name is written in the title. The title of the manuscript G includes, for example, an image portion A1 in which "ABCD Corporation" is written, and an image portion A2 in which "ABCD Corporation" is written once again at the end of the sentence.

The generation unit 22 generates a density map 50 (FIG. 3(B)) from the image data.

The convolutional neural network unit 24 performs a process of applying a filter 52 (FIG. 3(C)) to the density map 50 to generate a first feature map 54 (FIG. 4(C)). Further, the convolutional neural network unit 24 performs processing to generate a second feature map (FIG. 5(C)) from the first feature map 54. Further, the convolutional neural network unit 24 performs a process of generating classification data from the second feature map.

The Grad-cam unit 26 generates an activation map 66 from the second feature map 56 by performing Grad-cam processing.

Furthermore, the GLAD cam unit 26 divides the generated activation map 66 into a plurality of groups. The GLAD cam unit 26 performs a process of dividing the activation map 66 into a plurality of groups using clustering (for example, k-means). In this embodiment, an example will be described in which the GLAD cam unit 26 performs clustering using k-means on the activation map 66 and divides the activation map 66 into a plurality of parts. For example, the GLAD cam unit 26 detects a center point with a high density for each cluster from the activation map 66, and defines a set of points existing within a predetermined distance from the center point as one group. Perform the dividing process.

Note that the GLAD cam unit 26 may perform clustering on the image data using x-means in which the number of groups is estimated, and divide the activation map 66 into the estimated number of groups.

The storage unit 28 is, for example, an HDD or a memory, and stores the teacher data 60. In this embodiment, a case will be described in which the processing unit 20 determines whether the document G (FIG. 3A) is a document created for ABCD Corporation. For this reason, for example, the generation unit 22 and the convolutional neural network unit 24 perform processing on a sample image showing the characters "ABCD Corporation" to generate classification data, and use the generated classification data as the teacher data. shall be.

The comparison unit 30 compares the classification data 62 and the teacher data 60 and calculates difference data 64. Furthermore, the comparison unit 30 calculates a first loss function using the classification data 62 and the teacher data 60.

The output unit 32 further calculates a second loss function from the first loss function.

The correction unit 34 corrects the weighting of the filter 52 using the second loss function outputted from the output unit 32.

The classification unit 36 performs image determination processing. The classification unit 36 determines the type of image based on the array of values indicated by the classification data generated by the convolutional neural network unit 24 using the latest filter 52 corrected and updated by the correction unit 34 as described above. judge.

The operation unit 5 receives input of various operation instructions from the user.

The network interface unit 91 is a communication interface including a communication module such as a LAN chip (not shown). The network interface unit 91 sends and receives various data to and from external devices within the local area or on the Internet.

Next, a specific example of the image forming apparatus 100 according to the present embodiment will be described with reference to FIGS. 3 to 11 in addition to FIG. 2.

First, an outline of the processing of the convolutional neural network unit 24 will be explained with reference to FIGS. 3(A) to 3(C). The convolutional neural network unit 24 repeats convolution processing and pooling processing to obtain classification data (FIG. 9(A)).

The convolutional neural network unit 24 multiplies the density map 50 (FIG. 3(B)) generated by the generating unit 22 by a pre-stored filter 52 (FIG. 3(C)), and generates the image in FIG. 4(C). A first feature map 54 shown in FIG. 1 is generated, and a second feature map 56 is further generated from the first feature map 54.

Subsequently, the convolutional neural network unit 24 performs pooling processing to extract representative values 56C and representative values from the second feature map 56 for each predetermined matrix (in this embodiment, 2×2 will be explained as an example). 56D is extracted (FIG. 6(A)(B)). The convolutional neural network unit 24 (i) repeats the pooling process to reduce the size of the second feature map 56, and further divides the thus reduced second feature map 56 into a predetermined line in a primary line. (ii) Developing the second feature map 56 into data with a predetermined number of pixels in a linear row using a fully connected layer, etc., as shown in FIG. 9(A). Classification data 62 shown in is obtained. The classification data 62 will be described later with reference to FIG. 9(A).

The Grad-cam unit 26 generates an activation map 66 from the second feature map 56 by performing Grad-cam processing. Furthermore, the GLAD cam unit 26 divides the generated activation map 66 into a plurality of groups. Through this division process, the GLAD cam unit 26 divides the activation map 66 into a group including the image portion A1 and a group including the image portion A2 in the image of the document G shown in FIG. 3A, for example.

Here, the above convolution processing by the convolutional neural network unit 24 will be explained in more detail. The reading unit 4 (FIG. 2) reads the original G and generates image data, an example of which is shown in FIG. 3(A).

Then, the generation unit 22 generates a density map 50 (FIG. 3(B)) for the image showing the entire document G. The density map 50 includes pixels x11, pixel x12, pixel x13, ..., pixel x21, pixel x22, pixel x23, ..., pixel x31, pixel x32, pixel x33, ..., pixel xij, ... are arranged in a matrix. Density information is given to each pixel xij.

Note that, as an example, the number of pixels of pixel xij is two digits, but it is not limited to two digits, and may be one digit, or may be three or more digits. Further, hereinafter, if there is no need to specify the pixel number, it will be described as pixel xij. In this embodiment, to simplify the explanation, the density map 50 is assumed to be a 9×9 matrix of pixels x11 to x99. i and j are positive integers. In the filter 52, pixels wij of pixel w11, pixel w12, pixel w13, pixel w21, pixel w22, pixel w23, pixel w31, pixel w32, and pixel w33 are arranged in a 3×3 matrix, for example.

First, the convolutional neural network unit 24 convolves the density map 50. Specifically, as shown in FIG. 4A, the convolutional neural network unit 24 multiplies the density map 50 by a filter 52. First, the convolutional neural network unit 24 multiplies pixels w11 to w33 of the filter 52 to pixels x11 to x33 of the density map 50.

Next, the convolutional neural network unit 24 shifts the filter 52 by one column in the direction of increasing column numbers in the density map 50, and multiplies the pixels w11 to w33 of the filter 52 to the pixels x12 to x34 of the density map 50. . Thereafter, similarly, the convolutional neural network unit 24 sequentially multiplies the filter 52 from the matrix of pixels x11 to x33 of the density map 50 to the matrix of pixels x77 to x99.

That is, the convolutional neural network unit 24 multiplies the density map 50 by the filter 52 as described above to obtain the determinant of the first feature data yij=xij×wij shown in FIG. 4(B). i and j are positive integers. Specifically, the convolutional neural network unit 24 multiplies pixels w11 to w33 of the filter 52 to pixels x11 to x33 of the density map 50 to obtain first feature data y11=x11w11+x12w12+x13w13+x21w21+x22w22+x23w23+x31w31+x32w32+x33w33. get Next, pixels w11 to w33 of the filter 52 are multiplied by pixels x12 to x34 of the density map 50 to obtain first feature data y12=x12w11+x13w12+x14w13+x22w21+x23w22+x24w23+x32w31+x33w32+x34w33. Thereafter, similarly, the convolutional neural network unit 24 sequentially multiplies the matrix of pixels x77 to x99 of the density map 50 by the filter 52.

The convolutional neural network unit 24 then generates first feature data y11, first feature data y12, . . . , first feature data yij, . A first feature map 54 (FIG. 4(C)) arranged in a matrix formed by the feature data y77 is obtained. As a result, the convolutional neural network unit 24 has performed convolution processing from the 9×9 density map 50 to the 7×7 first feature map 54.

Furthermore, with reference to FIGS. 5(A) to 5(C), a specific example of convolution processing by the convolutional neural network unit 24 will be described. The density map 50 shown in FIG. 5(A) is a specific example of the density map 50 shown in FIG. 3(B). The filter 52 shown in FIG. 5(A) is a specific example of the filter 52 shown in FIG. 3(B).

Here, in order to explain the density map 50 in an easy-to-understand manner, an example will be described in which the value of each pixel constituting the density map 50 is shown as either a binary value of "1" or "-1". Further, the filter 52 will also be explained using an example shown as either a binary value "1" or "-1".

The density map 50 shown in FIG. 5A has pixel x11=-1, pixel x12=-1, pixel x13=-1, pixel x21=-1, pixel x22=1, pixel x23=1, pixel x31=- 1, pixel x32=-1, pixel x33=-1, and so on.

The filter 52 shown in FIG. 5(A) represents the pixel wij of the filter 52 shown in FIG. 3(C) as either binary "1" or "-1". As an example, pixel w11=1, pixel w12=-1, pixel w13=-1, pixel w21=-1, pixel w22=1, pixel w23=-1, pixel w31=-1, pixel w32=1, and pixel w33=-1.

The convolutional neural network unit 24 multiplies the pixel xij of the density map 50 and the pixel wij of the filter 52, and obtains the first feature data yij=xij×wij, as described with reference to FIG. 4(B). Specifically, the first feature data y11=-1×1+-1×-1+-1×-1+-1×-1+1×1+1×-1+-1×-1+-1×1+-1×-1= 3. First feature data y12=-1×1+-1×-1+-1×-1+1×-1+1×1+1×-1+-1×-1+-1×1+-1×-1=1,... It is.

The convolutional neural network unit 24 arranges the first feature data y11=3, the first feature data y12=1, . . . , and the first feature data y77=-3 in a matrix, as shown in FIG. 5(B). Thus, the first feature map 54 is generated.

Furthermore, since the filter 52 is composed of nine 3×3 matrices in the example shown in FIG. For example, the second feature map 56 is generated by converting the value to 1/9. That is, if the second feature map 56 is composed of the second feature data zij, then the second feature data zij=first feature data yij×1/9.

Specifically, second feature data z11=3×1/9=0.33, second feature data z12=1×1/9=0.11, ..., second feature data z77=-3× 1/9=-0.33. The convolutional neural network unit 24 arranges these second feature data zij in a matrix, and generates a second feature map 56 as shown in FIG. 5(C). The above is the convolution process by the convolution neural network unit.

Next, pooling processing by the convolutional neural network unit 24 will be explained. As shown in FIG. 6A, the convolutional neural network unit 24 divides the second feature map 56 into a plurality of feature matrices 56A, feature matrices 56B, . . . Representative values 56C, 56D, . . . are extracted from 56B, .

Specifically, the convolutional neural network unit 24 divides the second feature map 56 into 2×2 feature matrices 56A, 56B, . . . . However, if there are less than two columns, it is divided into a 1×2 matrix. Then, the convolutional neural network unit 24 extracts the representative value 56C of the feature matrix 56A from the feature matrix 56A. Here, it is assumed that the convolutional neural network unit 24 extracts the maximum value of the feature matrix 56A as a representative value. For example, the representative value 56C is the maximum value 0.33 of the feature matrix 56A. However, the convolutional neural network unit 24 may extract the representative value as an average value or a median value, for example, instead of the maximum value (the same applies throughout the specification of the present application).

Continuously, the convolutional neural network unit 24 extracts the representative value 56D of the feature matrix 56B from among the values configuring the feature matrix 56B. Here, it is assumed that the convolutional neural network unit 24 extracts the maximum value of the feature matrix 56B as a representative value. For example, the representative value 56D is the maximum value 0.11 of the feature matrix 56B.

The convolutional neural network unit 24 generates a matrix in which the representative values (representative value 56C, representative value 56D, . . . ) extracted as described above are arranged. The above is the pooling process.

Furthermore, (i) the convolutional neural network unit 24 generates a matrix composed of a predetermined number of representative values (an example of a matrix 58 in which the predetermined number is 3×3 is shown in FIG. 6(B)). The convolution process and the pooling process are further repeated until . The convolutional neural network unit 24 one-dimensionally expands each representative value forming the matrix to generate classification data 62, an example of which is shown in FIG. 9(A). (ii) Alternatively, the convolutional neural network unit 24 further performs the convolution process and the pooling process until a matrix composed of representative values of a predetermined second number larger than in case (i) is obtained. Repeatedly, when the matrix is obtained, the fully connected layer is processed to create a matrix (FIG. 6(B)) consisting of the predetermined number of representative values, and each of the The representative values are expanded one-dimensionally to generate classification data 62, an example of which is shown in FIG. 9(A). The above is the processing by the convolutional neural network unit 24.

Next, the glad cam process will be described with reference to FIGS. 7, 8, and 11 in addition to FIG. 2. The Grad cam unit 26 performs Gradient-weighted Class Activation Mapping (GradCAM). The GLAD cam unit 26 performs a process of applying an activation function ReLU (Rectified Linear Unit) to the second feature map 56 generated by the convolutional neural network unit 24 as GLAD cam processing, thereby creating an activation map 66 (FIG. 8). ) is generated.

The activation function ReLU is a function that sets all output values less than 0 to 0, as shown in FIG. In other words, the activation function ReLU is a process that makes only the part above a certain threshold value meaningful information, that is, a process that emphasizes the part above the certain threshold value as a characteristic part. The activation function ReLU has yij on the horizontal axis and f(yij) on the vertical axis. When yij<0, f(yij)=0, and when yij≧0, f(yij)=yij. The GLAD cam unit 26 applies the activation function ReLU to each value of the second feature map 56 to generate a matrix-like activation map 66 made up of the applied values.

FIG. 8 is a diagram showing an example of the activation map 66. In the activation map 66, each second feature data zij constituting the second feature map 56 is further emphasized and displayed by the GLAD cam unit 26, so that the feature portions in the document data read from the original G are emphasized. Become.

The activation map 66 includes an image of the company name part of the manuscript G, and also highlights parts of the manuscript G other than the company name part. Therefore, the correction unit 34 makes the activation map 66 accurately highlight only the image portion that should be the basis for image type determination, for example, the image portion where the company name of the document G is written ( In order to clarify which image part in the entire image indicates the "part where the company name is written"), the pixel wij, which is the weighting of the filter 52 used by the convolutional neural network unit 24, is corrected.

Here, the GLAD cam unit 26 performs the above-described division process to divide the activation map 66 into a plurality of groups. Then, the GradCAM unit 26 calculates a correction function f (GradCAM) for each of the plurality of groups. The correction function f (GradCAM) is expressed by formula (1).
...(1)

In the correction function f (GradCAM) of formula (1), a specific pixel among the pixels constituting the above group is designated as pixel c, and all pixels constituting the above group other than pixel c are respectively designated as pixel r. do. The correction function f is expressed as the product of the activation function Act(r) and a distance function (r−c)^2 representing the square of the distance between the pixel c and each pixel r. That is, the correction function f calculates the product of the activation function Act(r) and the distance function (r−c)^2 for each combination of pixel c and all pixels r in the above group, and calculates these products. The sum of all the products obtained is further calculated. The grad cam unit 26 calculates a correction function f for each of the plurality of divided groups.

Here, as an example, the pixel of the group with the maximum density is designated as pixel c, and an arbitrary pixel is designated as pixel r. For example, the activation function Act(r) represents the magnitude of the response of the activation map 66 (the above group) at pixel r.

For example, the density of pixel r is used as the activation function Act(r). That is, the value of the activation function Act(r) changes depending on the density of the pixel r. Also, the larger the distance between pixel c and pixel r, the larger the value of distance function (r-c)^2, and the smaller the distance between pixel c and pixel r, the greater the value of distance function (r-c)^2. becomes smaller.

Next, the storage section 28 and the comparison section 30 will be explained with reference to FIG.

As described above, the storage unit 28 stores the teacher data 60 (FIG. 9(A)). As shown in FIG. 9(A), the teacher data 60 consists of a numerical sequence in which the same number of values as the classification data are arranged one-dimensionally. The storage unit 28 stores teacher data 60 corresponding to each image that is a target image for determining the type of document.

The comparison unit 30 compares the classification data 62 acquired from the convolutional neural network unit 24 and the teacher data 60 stored in the storage unit 28, and calculates difference data 64. Further, the comparison unit 30 uses the classification data 62 and the teacher data 60 to calculate a first loss function using the following formula (2).
...(2)

Then, the output unit 32 acquires the value of the correction function f(GradCAM) (Formula 1) from the GradCam unit 26, and the first loss function (Formula (2)) calculated by the comparison unit 30 and the correction function f(GradCAM) ) is calculated as a second loss function. That is, the second loss function is expressed by equation (3). The output unit 32 calculates a second loss function for each group using the correction function f of each group.
...(3)

The correction unit 34 uses the sum of the second loss functions of each group outputted by the output unit 32 as a correction value, and uses this correction value to correct the weighting of the filter 52 used in the convolutional neural network unit 24. do. Thereby, the correction unit 34 creates the filter 52 after the correction. This second loss function is based on the activation map 66 that accurately emphasizes the image of the company name written part of the document G, which should be the basis for the above determination. When the weighting of the filter 52 is corrected using the loss function, the filter 52 (FIG. 9(B) ) will be corrected. A filter obtained by correcting the filter 52 in this manner is referred to as a filter 52B (FIG. 9(B)).

Specifically, as shown in FIG. 10A, when the comparison unit 30 obtains the classification data 62A, it calculates the difference data 64A by calculating the difference between the teacher data 60A and the classification data 62A. Furthermore, the comparison unit 30 calculates a first loss function (Formula 2) from the teacher data 60A and the classification data 62A. The output unit 32 calculates the sum of the first loss function and the value of the correction function f for each group, and sets this sum as a second loss function (Equation 3). The correction unit 34 calculates the correction value by summing the second loss functions of each group. Based on the correction value (for example, by multiplying the filter 52 by the correction value), the correction unit 34 changes the filter 52 of the convolutional neural network unit 24 to a filter 52B, for example, as shown in FIG. 9(B). Correct to.

When the filter 52 is corrected to the filter 52B in this way, the convolutional neural network unit 24 uses the corrected filter 52B to generate new classification data (an example is shown as classification data 62B in FIG. 10(B)). The grad cam unit 26 calculates a new correction function f for each group. The comparison unit 30 then compares the teacher data 60B and the classification data 62B and outputs difference data 64B. At this time, the classification data 62B approximates the teacher data 60B compared to the classification data 62A. Further, the comparison unit 30 uses the classification data 62B and the teacher data 60B to calculate a first loss function, and the output unit 32 uses the new correction function f calculated for each group by the GLAD cam unit 26 to calculate a first loss function. A new second loss function is calculated for each group. The correction unit 34 corrects the filter 52B using a correction value obtained by summing the new second loss functions for each group to create a filter 52C.

The convolutional neural network unit 24 generates new classification data (an example is shown as classification data 62C in FIG. 10C) using each of the corrected filters 52C. The grad cam unit 26 further calculates a new correction function f for each group. The comparison unit 30 then compares the teacher data 60C and the classification data 62C and outputs difference data 64C. The classification data 64C is more similar to the teacher data 60C than the classification data 62B (FIG. 10C shows an example in which the teacher data 60C and the classification data 62C match). In the example shown in FIG. 10(C), the difference data 64C has only 0 data.

11A to 11C show the filter correction processing performed by the convolutional neural network unit 24 to the correction unit 34 as described above, and the convolutional neural network unit 24 and the GLAD cam unit using the corrected new filter. 26 is a diagram showing how the activation map 66 of each group generated by the GLAD cam section 26 is corrected by repeating the activation map generation process by 26 for each group. FIG.

For example, the first activation map 66A for one of the plurality of groups shown in FIG. However, the filter is corrected as described above, and the convolutional neural network unit 24 uses the new filter 52B for the density map 50 to create a new first feature map 54 and a new second feature map 50. When the feature map 56 is generated and the GLAD cam section 26 generates a second activation map 66B from the new second feature map 56, the second activation map 66B has the first activation map 66B as shown in FIG. 11(B). Compared to the activation map 66A, the density of pixels located at a large distance from the title portion is greatly reduced.

Further, the filter 52B is corrected to the filter 52C by the filter correction processing performed by the convolutional neural network unit 24 to the correction unit 34, and the convolutional neural network unit 24 and the GLAD cam unit 26 perform an action using the corrected new filter. When the third activation map 66C is generated by the activation map generation process, for example, as shown in FIG. 11(C), the title of the third activation map 66C is The density of pixels further away from the portion is reduced even more. In FIG. 11C, compared to the second activation map 66B, the density of pixels located at a large distance from the pixel representing the image of the company name portion of the document G is almost 0, and the density of the pixels of the document G is close to 0. An example is shown in which the density of the image where the company name is written is more emphasized.

In this way, the filter correction process from the convolutional neural network unit 24 to the correction unit 34 is repeated, and for example, when the teacher data and the classification data match and the difference data becomes 0 (or when the teacher data and the classification data The repetition is ended at the time when the values are approximated to within a predetermined range), and the convolutional neural network unit 24 updates the latest filter corrected at this time as the filter used for the above-mentioned convolution processing. The above is a learning process for extracting images to be used as the basis for image type determination.

In addition, in the process of image type determination, the classification unit 36 uses the latest filter 52 that has been corrected and updated as described above to use the convolutional neural network unit 24 to generate a document image (by reading the document by the reading unit 4). The type of image is determined based on the array of values indicated by the classification data generated from the obtained image. For example, if (A) only the first classification data indicates a value greater than 0 and all other values are 0, the classification unit 36 determines that the image portion to be used as the basis for image type determination is " (B) If only the second classification data shows a value greater than 0 and all other values are 0, the image type is determined. The type of image is determined by, for example, assuming that the image portion that should be the basis for the image is "EFGH company" and determining that it is a "document addressed to EFGH company."

Therefore, once the learning process shown in this embodiment is completed, the classification data generated by the convolutional neural network unit 24 after this matches or approximates the teacher data, so when determining the type of image, image type determination is performed. Even in the case where there is not a single object but a plurality of objects as image portions to be used as the basis for the image, it is possible to accurately extract each object and maintain high accuracy in determining the type of the image.

Furthermore, in this embodiment, if the learning process up to the correction process of the filter 52 is also performed every time the image type determination process is performed, the number of times the image forming apparatus 100 reads the document G and the image processing As the number of cases increases, the filter of the convolutional neural network unit 24 is corrected more appropriately, so that the accuracy of image type determination can be improved.

Further, the configuration and processing of the above embodiment described using FIGS. 1 to 11 are merely one embodiment of the present invention, and the present invention is not intended to be limited to the configuration and processing.

Further, in the above embodiment, an example in which an embodiment of the image processing according to the present invention is applied to an image forming apparatus (multifunction device) is described, but this is only an example, and the image processing according to the present invention may be applied to other electronic devices, such as medical equipment, personal computers, mobile phones, smartphones, tablets, hub devices, and server devices.

100 Image forming device 4 Reading section 20 Processing section 22 Generation section 24 Convolutional neural network section 26 Grad cam section 28 Storage section 30 Comparison section 32 Output section 34 Correction section 36 Classification section

Claims (3)

A processing unit including a generation unit, a convolutional neural network unit, a grad cam unit, a storage unit, a comparison unit, an output unit, and a correction unit,
The generation unit generates a density map from image data to be processed,
The convolutional neural network unit generates a feature map by filtering the density map, and generates classification data from the feature map,
The GRAD cam unit performs a process of generating an activation map from the feature map using an activation function, divides the activation map into a plurality of groups by clustering, and calculates a correction function for each of the plurality of groups. ,
The storage unit stores teacher data,
The comparison unit performs a process of calculating a first loss function for the classified data using the classified data and the teacher data,
The output unit adds the correction function to the first loss function calculated by the comparison unit to calculate a second loss function for each of the plurality of groups,
The correction unit corrects the weighting of the filter used in the convolutional neural network unit using a correction value that is the sum of the second loss functions calculated by the output unit,
From the process by the convolutional neural network unit using the new filter created by the correction unit, the process by the GLAD cam unit, the comparison unit, the output unit, and the correction unit is repeated to improve the filter. An image processing device that corrects and updates. The GLAD cam unit performs division into the plurality of groups by detecting a center point with a high density for each cluster from the activation map, and dividing each point within a predetermined distance from the center point. The image processing apparatus according to claim 1, wherein the image processing is performed by forming a set of into one group. The processing unit further includes a classification unit,
The image according to claim 1 or 2, wherein the classification unit determines the type of the image based on an array of values indicated by classification data generated by the convolutional neural network unit using the updated filter. Processing equipment.