JP3113785B2 - Image classification device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image classification device,
In particular, the present invention relates to an image classification device that determines the attribute of an input image when inputting an image database, and automatically classifies and registers the image.

[0002]

2. Description of the Related Art Japanese Patent Laid-Open No. 4-316180 discloses a method of automatically dividing a region of an image, which has been previously determined to be a document image, by a human, and determining the attribute of each region.
And JP-A-62-71379.

In the former, a binarized input image is divided into regions, and attributes such as a photograph region and a character region are determined based on the number of black and white reversals and the ratio of black pixels in each region.
In the latter case, the binarized input image is divided into regions, and attributes such as a photograph region and a character region are determined based on the run-length characteristic and the black pixel ratio of each region.

[0004]

However, the prior art only determines the attribute of each of the regions divided within the image for the document image, and determines whether or not the input image itself is a document image. No method or apparatus has been disclosed so far.

Therefore, when an image is registered in a multimedia database to which various image data are input in addition to a document image, it is necessary for a human to classify and judge the content of the input image and add an appropriate keyword. was there.

Further, such a multimedia database is disclosed in JP-A-4-316180 and JP-A-62-713.
When applying the 79 techniques, a human must first determine whether or not the input image is a document image, and then apply those techniques.

[0007] Further, since there is no device for classifying only business card images from image data as a conventional technology, when performing processing such as creating an address book from a business card image using, for example, character recognition, a human being can select a business card image from various images. And perform the character recognition process for each image.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem, and automatically determines whether or not an input image itself is a document image, and classifies the input image without any human trouble. It is an object of the present invention to provide an image classifying apparatus which can register in an image database after adding a keyword, and can automatically perform processing according to the content of the image, for example, creation of an address book using character recognition. And

[0009]

According to a first aspect of the present invention, there is provided an image classification device, comprising: an identification unit for identifying a connected component composed of a plurality of adjacent pixels representing an object from input image data; Extracting means for extracting a characteristic amount representing character-likeness from the shape of the connected component in the extracted image data, and determining means for determining whether the input image data is a document image based on the extracted characteristic amount. With the input
The image data is composed of grayscale image data.
Data areas with strong edges from the input grayscale image data
Edge detection means for detecting the area of the connected component
From the area of the overlapping area of the connected component and the data area, the feature amount
Is extracted.

[0010]

[0011] image classification apparatus according to claim 2, of the input
From adjacent image data
Identification means for identifying connected components consisting of pixels;
Character-likeness from the shape of connected components in the
Extraction means for extracting the amount of features
To determine whether the input image data is a document image or not.
The extracting means includes calculating means for calculating an area of a circumscribed rectangle for each of the connected components, and extracts a feature amount from the area of the connected component and the area of the circumscribed rectangle.

[0012]

[0013]

[0014]

[0015]

[0016]

According to a third aspect of the present invention, there is provided the image classification apparatus according to the first or second aspect , wherein the identification means includes a plurality of adjacent images representing a target object from the input image data. Pixels satisfying a predetermined condition are identified as connected components.

[0018]

According to the first aspect of the present invention, a connected component composed of a plurality of adjacent pixels representing a target object is identified from input image data, and a connected component of the input image data is identified. A feature amount representing character-likeness is extracted from the shape, and it is determined whether or not the input image data is a document image based on the extracted feature amount. In addition, grayscale image data
Input, and from the input grayscale image data,
Data area and the area of the connected component,
The feature amount is extracted from the area of the overlapping region with the data region.

[0019]

The image classification apparatus according to claim 2 is to calculate the area of the circumscribed rectangle for each of the communication formed component, extracts a feature quantity from the area of the connecting component and the area of the circumscribed rectangle.

[0021]

[0022]

[0023]

[0024]

[0025]

According to a third aspect of the present invention, in addition to the operation of the first or second aspect of the present invention, a predetermined one of a plurality of adjacent pixels representing an object from input image data is provided. Are identified as connected components.

[0027]

【Example】

(First Embodiment) FIG. 1 is a block diagram showing a system configuration of an image classification device according to a first embodiment of the present invention.

Referring to the drawings, the image classification device is roughly composed of an image input device 11 for inputting an image including a camera, a scanner, and the like, and processing the input image to convert the image into a document image and a non-document image. The image processing apparatus 12 includes an image processing apparatus 12 for performing classification (keyword assignment) and an image database apparatus 13 for storing images classified by the image processing apparatus 12.

Each device is connected by a bus 15. The image processing device 12 processes and classifies the input image, and controls the operation of the entire system. The control device 121 includes a CPU, a ROM 122 that stores various programs, constants, and the like. And a RAM 123 for storing processing data and the like.

The ROM 122 includes an image input processing section 1221 used for image input processing, an image binarization processing section 1222 used for binarizing an image, and a ROM for storing binarized image data. , A labeling processing unit 1224 for labeling (processing for assigning a label number) to connected components of data in the binarized image data, and a binarized image data , A character-like feature extraction processing unit 1225 for extracting the character-like features in the image, an image classification processing unit 1226 for classifying images, and an image registration processing unit 1227 for classifying and registering image data in the image database device 13 And

The RAM 123 has a multi-valued image memory 1231 for storing inputted image data as image data of 256 gradations, and a binary image memory for storing image data represented by binary values of "0" and "1". 1232, a label image memory 1233 for storing the label number of each connected component of the data, a fillet coordinate memory 1234 for storing the fillet coordinates and area of each connected component, and a differential image of the image data of the multi-valued image memory 1231. , A differential binary image memory 1236 for storing image data obtained by binarizing the image data of the differential image memory 1235 with a threshold value, and a binary image memory 123
For storing a logical product image (AND image) of the image data of the second and the binary image memory 1236
And an image memory 1237.

With this system configuration, the image input device 1
Various images input from the image processing device 1
By examining the feature amount of the image in, the image is automatically classified into a document image or a non-document image and recorded in the image database device 13.

For example, a document image is an image containing many characters shown in FIG. 4, and a non-document image is a landscape image shown in FIG.

FIGS. 2 and 3 are flowcharts showing a processing routine of the control device 121 of FIG.

The flowcharts of FIGS. 2 and 3 correspond to A in FIG.
And a series of processing is shown.

In step S201, the control device 12
1 inputs image data through the image input device 11 and stores it in the multi-valued image memory 1231. The image data stored in the multi-valued image memory 1231 is, as shown in FIG.
The data is composed of 512 pixels in the direction and 432 pixels in the Y direction. One pixel has a luminance value of 256 gradations (8 bits) from “0” (black pixel) to “255”. The origin pixel (0, 0) is defined as the upper left pixel with respect to the image data diagram, and X
The coordinates of 0 to 511 are added in the direction and the coordinates of 0 to 431 are added in the Y direction.

The multi-valued image memory 1231 has the structure shown in FIG. That is, the multi-valued image memory 1231 stores X
It has an 8-bit memory that stores a 256-level luminance value for each pixel indicated by coordinates and Y coordinates.
Starting from the luminance value of the pixel at the origin (0, 0), data of the luminance value of the pixel is sequentially stored in the memory in the order of increasing X-coordinate. The luminance value data is stored in the order of increasing coordinates, and the coordinates (5
11, 431) are stored.

In step S202, the control device 12
Reference numeral 1 denotes the entirety of image data (multi-valued image data) having 256-level luminance values stored in the multi-valued image memory 1231;
Value. Binary image data (binary image data)
Are stored in the binary image memory 1232.

Specifically, in the binarization, the control device 121 scans the entire multi-valued image data in the memory 1231 and sets a threshold value of the luminance for the binarization. This is performed by setting a pixel having a white pixel (“1” data) and a pixel having a luminance lower than the threshold value as a black pixel (“0” data). Here, as a method of setting the threshold, for example, supervised by Hideyuki Tamura, edited by Japan Industrial Technology Center,
"Introduction to Computer Image Processing", Soken Shuppan, 1988
Year, p. 66 to p. 69 (a discriminant analysis method, a p-tile method, a mode method, etc.) can be used. In this embodiment, the discriminant analysis method is used.

In the discriminant analysis method, when it is assumed that a set of pixels having different luminance values is divided into two classes (threshold or more and less than th) by a threshold value (th), the variance between each class is maximized. This is a method of setting the threshold value (th) so that

After the determination of the threshold value, the control device 121 scans the multivalued image data stored in the multivalued image memory 1231 from the origin, reads the luminance value of each pixel, and compares it with the set threshold value. . As a result, if the luminance value is less than the threshold value, the luminance value of the pixel is set to “0” (black pixel), and
If the luminance value is equal to or larger than the threshold value, the luminance value of the pixel is stored in the binary image memory 1232 as “1” (white pixel).

The address structure of the binary image memory 1232 is the same as that of the multi-valued image memory 1231 of FIG. 7, except that one-bit data of "0" or "1" is recorded as a luminance value.

In step S203, the control device 12
1 determines whether the number of white pixels in the binary image data of the binary image memory 1232 is greater than 1/3 of the total number of pixels. Y
If it is ES, in step S204, the binary image data “0” and “1” of the binary image
Is inverted.

If NO in step S203,
Step S204 is not performed.

The processing in steps S203 to S204 is performed for the following reason. Normally, image data is processed with a black pixel as a background and a white pixel as a target. For example, when image data composed of a black pixel target is input on a white pixel background shown in FIG. It is necessary to reverse black and white. Therefore, when the number of white pixels is 1/3 or more in all the pixels, it is determined that the background is a white pixel and the object is a black pixel, and the white pixel and the black pixel are inverted.

In step S205, connected components of white pixels in the binary image data of the binary image memory 1232 are extracted, and labeling is performed on each component.

The connected component is a group of white pixels (data of "1") in the image data which are continuous vertically, horizontally, and diagonally. For example, if the image data is the data shown in FIG. 8, four connected components of the connected components 20a to 20d are extracted as a group in which data of "1" is continuous vertically and horizontally, and each connected component is identified. Label numbers are set as shown in FIG. In FIG. 9, "1" data, "2" data, "3" data, and "4" data are respectively assigned as label numbers to the connected component 20a, the connected component 20b, the connected component 20c, and the connected component 20d of FIG. Is set. For a black pixel without a label number, data “0” is set instead of the label number.

The label number is recorded in the label image memory 1233. The label image memory 1233 has the address configuration shown in FIG. That is, the address configuration is the same as that of the binary image memory 1232. Each address has a 16-bit memory for storing a label number assigned to an image corresponding to the address. It is necessary to prepare the number of bits of each memory as much as possible to store the maximum value of the label number. In this embodiment, it is considered that 16 bits are sufficient.

As a labeling method, for example, “Digital Image Processing for Image Understanding (I
I) "Shokodo, 1988, p. 45 to p. 46 can be used.

In step S206, the controller 12
1 extracts the fillet coordinates and area of each connected component with a label number. The fillet coordinates are the upper left and lower right coordinates of the smallest rectangle circumscribing one connected component in the image. When there is a connected component as shown in FIG. 11, the upper left fillet coordinates are (x_sp, y_sp ), And the lower right fillet coordinates are (x_ep, y_ep).

That is, the largest X coordinate and the largest Y coordinate among the X coordinate and the Y coordinate of the pixels assigned the same label number
The coordinate set is the lower right fillet coordinate, and the minimum X coordinate and the minimum Y coordinate set is the upper left fillet coordinate.

The area of the connected component is represented by the number of pixels having the same label number, that is, the number of pixels of each connected component.

In FIG. 11, the connected components are shown in black and the background is shown in white for convenience of explanation. The file coordinates and area of each connected component are stored in the file coordinate memory 1234. The fillet coordinate memory 1234 has the configuration shown in FIG. That is, the X coordinate and the Y coordinate of the upper left fillet coordinate, the X coordinate and the Y coordinate of the lower right fillet coordinate, and the area of the connected component are recorded in the order of the label number. The memory for storing the coordinates and the area is 16 bits. Therefore, one label has a memory area of 80 bits.

In step S207, unnecessary connected components are removed. Unnecessary connected components are connected components described below.

(1) A minute component having an area of a connected component of 3 pixels or less. (2) The width of the rectangle circumscribing the connected component in the X direction is one-eighth of the entire width (512 pixels) of the image data in the X direction. The width in the Y direction of a rectangle that is larger or circumscribes the connected component is one-eighth of the width (432 pixels) of the entire image data in the Y direction.
Larger component (3) Elongated component whose width in one of the X and Y directions of the rectangle circumscribing the connected component is larger than four times the other width The three connected components described above are not character information Therefore, it is deleted from the binary image memory 1232.

For example, in the binary image memory 1232, FIG.
6, there are six types of connected components, labeled 1 to label 6, and correspondingly, the fillet coordinate memory 12
It is assumed that the data shown in FIG. At this time, the connected component of the label 2 is removed as the above-described giant component because the width in the Y direction is too large.

The connected component of the label 4 is removed as a minute component because the area is 3 pixels or less.

The connected component of the label 6 is removed as an elongated component because the width in the X-axis direction is four times or more the width in the Y-axis direction.

Accordingly, as a result of removing each connected component, the connected components of labels 1, 3, and 5 are left in the binary image memory 1232 as shown in FIG.

In step S208, the controller 12
Numeral 1 performs a spatial differentiation process on the multivalued image data in the multivalued image memory 1231. The image (differential image) subjected to the differential processing is recorded in the differential image memory 1235.

The differentiation of the image is performed to detect a portion where the density discontinuity is high in the image.

The portion where the density discontinuity is high (where the change in the density value is large) in the image is recognized as the boundary (edge portion) of the object.

A specific differential processing method is performed as follows. Referring to FIG. 16, if the coordinates of the image to be differentiated in multi-valued image memory 1231 are (i, j), the pixel at coordinates (i, j) and the pixel at coordinates (i, j) ,
A total of 3 × 3 pixels that are vertically and horizontally inclined are pixels used for differentiation. The luminance value of each of the 3 × 3 pixels is multiplied by a coefficient shown in FIG. Δ _x F (i, j) is the sum of all the luminance values of 3 × 3 after being multiplied by the coefficient.

On the other hand, the brightness value of each of the 3 × 3 pixels is multiplied by the coefficient shown in FIG. 18, and the sum of all the 3 × 3 brightness values after the multiplication is added. To δ
_{Let y} F (i, j).

Here, as a differential value, for example, | δ _x F
(I, j) | + | δ _y F (i, j) | can be used.

The coefficients in FIGS. 17 and 18 are called Sobel operators, supervised by Hideyuki Tamura, edited by Japan Industrial Technology Center, "Introduction to Computer Image Processing", Soken Shuppan, 1988.
Year, p. 118-p. 125.
Other methods may be used as a method for obtaining the differential value.

The pixels in the multi-valued image memory 1231 are scanned by the Sobel operator shown in FIGS. 17 and 18, and the differential value of each pixel is recorded in the differential image memory 1235.

In the multi-valued image memory 1231, the portion corresponding to the end of the image data (the portion where the X coordinate is 0 or 511 or the Y coordinate is 0 or 431) cannot be applied with the Sobel operator. 0 ”data is recorded.

In step S 209, the differential image recorded in the differential image memory 1235 is binarized by a threshold, and is recorded in the differential binary image memory 1236. 2
The value conversion process is the same as the method described in step S202, and a description thereof will be omitted.

By these processes, the differential binary image memory 1
In 236, a portion where the change in the luminance value is large in the input image is recorded as a white pixel (data of "1").

In step S 210, the image data of the binary image memory 1232 and the differential binary image memory 1236
AND operation is performed on the image data of
This is recorded in the D image memory 1237. What is AND operation?
The data at the same coordinate position of two binary images are both "1"
This is an operation in which “1” data is given only to a pixel that is a (white pixel), and “0” data is given to other pixels.
By this processing, the overlapping area between the binary image data of the effective connected component and the binary image data of the portion where the change in the luminance value is large is recorded in the AND image memory 1237 as "1" data.

FIG. 19 is a diagram showing a state in which one connected component in the document image of FIG.

In FIG. 19, for convenience of explanation, white pixels are set as the background (portion of "0" data) and black pixels are set as the object (portion of "1" data).
The black pixel is used as the background (that is, black and white inverted from FIG. 19).
It is recorded in the memory (the same applies to FIG. 24 hereinafter).

FIG. 20 is a diagram showing a state in which the same connected components as those shown in FIG. 19 are recorded in the differential binary image memory 1234. Referring to the figure, the boundary between the object having a large change in the luminance value and the background is recorded as “1” data.

FIG. 21 is a diagram for explaining the same connected components as those in FIG. 19 in a state where they are recorded in the AND image memory 1237. In the AND image memory 1237, a result obtained by performing an AND operation on the image of FIG. 19 and the image of FIG. 20 is recorded.

FIGS. 22 to 24 are views showing a state in which one connected component in the landscape image of FIG. 5 is recorded in each memory, and correspond to FIGS. 19 to 21.

A comparison between a landscape image and a document image has the following tendency. For the document image, the AND image memory 12 stores the area of the connected component recorded in the binary image memory 1232.
37, the ratio of the area of the connected component stored in the AND image memory 1237 to the area of the connected component stored in the binary image memory 1232 is relatively large in a landscape image. Relatively small. This is based on the fact that the character connected component has a large amount of edge information, and it is possible to determine whether the input image is a document image or other than the input image based on this difference.

In step S211, the control device 12
1 calculates the ratio edge ratio of the entire area of the AND image memory 1237 determined in step S210 to the entire area of the effective connected component determined in step S207. The edge ratio is calculated by equation (1).

Edge ratio = (Σedge area / Σbinary area) × 100 (1) In equation (1), edge area is the area of each area recorded in the AND image memory 1237, and
area is the area of each area included in the binary image memory 1232. Σ indicates that all areas in the memory are summed.

It is also possible to calculate the edge ratio in each area and finally take the average of the entire edge ratio.

In step S212, the control device 12
1 compares the edge ratio obtained in step S211 with the set threshold value, and determines whether the edge ratio is equal to or greater than the threshold value.

If YES in step S212, it is determined in step S213 that the image data is a document image.

If NO in step S212, the image data is determined to be non-document in step S215.

The threshold value is, for example, 60%, but can be changed by the user.

In step S 214, the image is stored in the image database 13 together with the determined keyword (attribute) of a document or a non-document.

FIG. 25 is a diagram for explaining the effect of this embodiment. The horizontal axis is the edge ratio, and the vertical axis is the edge rat
Shows the number of sample images classified into a predetermined range of io.
The edge ratio is divided in a range of 5%, and the number of sample images of documents and non-documents classified in the range is shown as a histogram.

Referring to the graph, the document image has the edge ratio
Is greater than 60% and non-document images have an edge ratio of 60%
Tends to be smaller. Thus, it is understood that the classification of the document image and the non-document image can be performed by using the value of 60% as the threshold value in the third step S212.

As described above, in the present embodiment, the ratio of the area of the differential binary image (representing the edge information) in the effective connected image is examined by utilizing the feature that the character region has a large amount of edge information. In addition, a document image and a non-document image including many character areas can be classified with high accuracy.

Therefore, such an image classification device can be used as an input device of an image database or a facsimile device. For example, when used as an input device of an image database, a document and a non-document are automatically classified, so that a keyword of “document” or “non-document” is automatically added to an input image without manual intervention. It becomes possible.

Further, a keyword can be automatically extracted from a document in an image by applying character recognition to the image classified as a document image, and can be added.

Further, by using the image classification device of the present invention at the time of inputting an image database, a user can input a document image and a non-document image to one system without being conscious of it. It becomes possible to search individually.

Further, by using the present invention as an input device of a facsimile machine, an input image can be converted into a document,
It is possible to classify the document as a non-document, for example, it is possible to automatically select a print mode.

Further, the image classification apparatus according to the present invention is used for determining the attribute of each area by applying an image in which a document is mixed with a photograph or a figure to each area after dividing the area. Is possible.

(Second Embodiment) FIG. 26 is a block diagram showing a device configuration of an image classification device according to a second embodiment of the present invention.

Referring to the drawing, this block diagram is obtained by removing the differential binary image memory 1236 and the AND image memory 1237 from the block diagram of FIG.

The basic processing performed in this embodiment is the same as that of the first embodiment. However, as a criterion for classifying document image data and non-document image data, the area of the circumscribed rectangle of the connected component and the internal It is characterized in that the ratio with the area of the effective connected component is used.

The processing in the image classification device according to the present embodiment is performed as follows. First, step S2 in FIG.
The processing from 01 to S207 is performed. This processing is substantially the same as the processing described in the first embodiment, and thus description thereof will not be repeated.

After the processing of step S207 is performed,
The processing of the flowchart shown in FIG. 27 is performed.

In step S401, the control device 12
1 calculates the ratio between the area of the effective connected component and the area of the circumscribed rectangle. The area of the effective connected component is read from the fillet coordinate memory 1234, and the area of the circumscribed rectangle is obtained from the fillet coordinates of the fillet coordinate memory 1234.

The area of the circumscribed rectangle of the effective connected component (filet
area) is the area of the effective connected component (binary area)
(Bin ratio) is obtained by the equation (2).

Bin ratio = (Σbinary area / Σfilet area) × 100 (2) In equation (2), Σ indicates the sum of the entire image.

In this embodiment, the ratio of the total area of the effective connected components to the total area of the circumscribed rectangles of the effective connected components is used as the bin ratio. After obtaining the ratio of the area of the effective connected component to the area of the circumscribed rectangle, a method of taking the average value of the whole may be used.

In step S402, the control device 12
1 bin rati obtained in step S401 by
It is determined whether o is less than a predetermined threshold value stored in advance. As the predetermined threshold value, for example, a value of 55% is used as described later, but the threshold value may be arbitrarily set by the user. At that time, the processing routine for setting the threshold by the user is as follows:
Recorded in OM122.

If YES in step S402, it is determined in step S403 that the input image is a document image.

If NO in step S402, it is determined in step S404 that the input image is a non-document image.

At step S405, the image data is
It is recorded in the image database device 13 together with the keyword (attribute) of document / non-document.

FIG. 28 shows an example of binary image data in a document image, and FIG. 29 shows an example of binary image data in a landscape image (non-document image).

In the figure, dotted lines indicate circumscribed rectangles in each connected component. Also, for convenience of explanation, the object is shown in black and the background is shown in white.

Referring to the figure, the ratio of the area of the effective connected component to the area of the circumscribed rectangle is smaller in the document image than in the non-document image. As a result, document images and non-document images can be classified.

FIG. 30 shows bins of a document image and a non-document image.
It is a figure for explaining ratio. The horizontal axis shows the bin ratio divided in 5% steps, and the vertical axis shows the number of sample images classified in each bin ratio divided in 5% steps.

Referring to the figure, the document image has a bin ratio of 55.
%, And the non-document image has a bin ratio of 55% or more. Therefore, it is understood that the document image and the non-document image can be classified by setting the threshold value in step S402 in FIG. 27 to 55%, and determining that the document image is less than the threshold value and the non-document image is greater than the threshold value. .

As described above, in this embodiment, the ratio of the area of the effective connected component to the area of the circumscribed rectangle is utilized by utilizing the characteristic of the character region that the ratio of the connected component in the circumscribed rectangle is small because the figure is a line figure. By using, the characteristics of a document image in which many character areas are arranged can be extracted well and classified with high accuracy.

Further, the edge used in the first and second embodiments is used.
It is possible to classify the image data with higher accuracy by judging by combining a plurality of feature values indicating character likeness including ratio and bin ratio.

This is, for example, in one image data.
Scores are given for each of the document image likeness by the edge ratio and the document image likeness by the bin ratio, and the image data is classified based on the total score. Data can be realized by a method such as classification using the other feature amount.

(Third Embodiment) FIG. 31 is a diagram for explaining the processing of the image classification device according to the third embodiment of the present invention.

The configuration of the image classification device according to the present invention is the same as that of the first embodiment, and a description thereof will be omitted.

The processing performed in the image classification device according to the present embodiment is as follows.

After the processing of steps S201 to S207 of FIG. 2 is performed, the processing of FIG. 31 is performed.

That is, in step S501, a feature quantity indicating the character likeness of an effective connected component is extracted. The feature amount is extracted for all the effective connected components in the image or each effective connected component. The feature amount is data having one value in the entire image. The feature quantity representing character-likeness is stored in the multi-valued image memory 1231 and the binary image memory 12.
32, label image memory 1233, fillet coordinate memory 1
234.

In step S502, the processing device compares the extracted feature value with a predetermined threshold value. The predetermined threshold value may be recorded in the ROM, or may be externally specified by the user. Further, two or more feature amounts obtained by two or more different methods may be used for comparison.

In step S503, it is determined from the compared characteristic amount and a predetermined threshold value whether the characteristic amount satisfies the condition indicating character-likeness. If YES, the input image is determined to be a document in step S504. Is done.

If NO in step S503,
In step S506, the input image is determined to be a non-document.

In step S505, the input image is recorded in the image database, but the keyword (attribute) of “document image” is determined to be a non-document for the image determined to be a document image at that time. The keyword “non-document image” is recorded together with the image.

Thus, when searching for image data from the image database, an efficient search can be performed using the keyword.

(Fourth Embodiment) FIG. 32 is a block diagram showing a system configuration of an image classification device according to a fourth embodiment of the present invention.

Referring to the figure, the image classifying device is roughly divided into an image input device 51 for inputting an image including a scanner as a camera, a business card image by processing the input image, and other image data. An image processing device 52 classifies the images (keywords are assigned) and an image database device 53 that stores images classified by the image processing device 52. Each device is connected by a bus 15. The image processing device 52 includes a control device 521 including a CPU and the like for processing and classifying input images and controlling the operation of the entire system, a ROM 522 for storing various programmable values and constants, A RAM 523 for storing processing data and the like.

The ROM 522 includes an image processing section 5221 used for image input processing, an image binarization processing section 5222 used for binarizing an image, and a An area extraction processing unit 5223 used for identifying an area, a labeling processing unit 5224 that performs labeling (processing for assigning a label number) to a connected component of data in the binarized image data, and an area of the image data A region extraction method processing unit 5225 for extracting the characteristics of the arrangement of images, an image classification processing unit 5226 for classifying images, and an image registration processing unit 5227 for registering image data in the image database device 53 are included.

A RAM 523 stores a multi-valued image memory 5231 for storing input image data as image data of 256 gradations, and a binary image memory for storing image data represented by binary values of “0” and “1”. 5232, a label image memory 5233 for storing the label number of each connected component of the data, a fillet coordinate memory 5234 for storing the fillet coordinates and area of each connected component, and storing the coordinates of the center of gravity of the effective connected component. A center-of-gravity memory 5235,
A direction frequency histogram memory 5236 for storing the direction frequency between the center of gravity and another center of gravity, and a multi-valued image memory 1231
Image memory 52 for storing the differential image of the image data
37, a density projection histogram memory 5 for storing a density histogram obtained by projecting the density of the differential image in the X and Y directions.
238, an effective connected component number memory 5239 for storing the effective connected component number, and a histogram differential value memory 5240 for storing the histogram differential value.

With this system configuration, the image input device 5
The various images input from 1 are automatically classified as business card images or not by checking the feature amounts of the images in the image processing device 52 and stored in the image database device 53.

As a criterion for determining the feature value in this embodiment,
The business card image utilizes the feature that the number of characters is small and the line spacing is wide.

FIGS. 33 to 35 are flowcharts showing the processing of the control device 521 in this embodiment.

The flow chart shows a series of processing by linking the parts A or B in the figure. Steps S601 to S60
7 is substantially the same as the processing in steps S201 to S207 in FIG. 2, and the description thereof will not be repeated.

At step S608, the control unit 52
1 measures the number of valid connected components contained in the binary image memory 5232, and stores the number in the valid connected component memory 532.
Record at 239. Since the business card image has a small number of characters, it has a feature that there are few effective connected components. Therefore, as one of the targets for determining the business card image, the number of effective connected components is measured.

In step S 609, the center of gravity of each valid connected component is determined and recorded in the center of gravity memory 5235.

The coordinates of the barycenter are obtained for the file coordinates of each effective connected component recorded in the file coordinate memory 5234.

Assuming that the upper left fillet coordinate of each effective connected component is (x_sp [i], y_sp [i]) and the lower right fillet coordinate is (x_ep [i], y_ep [i])
The barycentric coordinate grav [i] is obtained by equation (3).

Grav [i] = {(x_ep [i] −x_sp [i]) / 2, (y_ep [i] −y_sp [i]) / 2} (3) The centroid memory 5235 has the following structure. As shown, the X and Y coordinates of the center of gravity of the label are recorded in the order of the label numbers of the effective connected components.

Since the center of gravity of the connected component removed in step S607 is not recorded, the label numbers are not always continuous.

Since the memories for recording the X and Y coordinates of the center of gravity each have a memory capacity of 16 bits, a 32-bit memory area is secured for one label.

In step S610, control device 521
Finds the other centroid point closest to each centroid point and records its direction in the direction frequency histogram memory 5236.

FIG. 37 is a flowchart showing a specific routine of the processing of the center of gravity performed in step S610 of FIG.

In step S801, the value of a variable “size” representing the size (pixel unit) of the search range is set to 1.

In step 802, the value of a variable “ws” representing the rectangular width of the search range is set by the equation ws = size × 2.

In step S803, it is determined whether or not there is another barycenter on the side of the rectangle of one side ws centered on the barycenter of interest.

In step S804, there is no center of gravity (N
If it is determined as O), in step 809, the variable “size” is incremented by one.

In step S810, it is determined whether the value of the variable “size” is smaller than the maximum threshold value MAX_SIZE. MAX_SIZE is set to, for example, a value of 100.

If YES in step S810, the processing from step S802 is repeated.

If YES in step S804, the direction from the center of gravity to the center of gravity on the side is calculated in step S805. At this time, if there are a plurality of centroid points on the rectangle, the directions are calculated for all the centroid points. The direction θ is obtained by Expression (4), assuming that the direction vector from the center to the center of gravity on the rectangle is (dx, dy).

Θ = tan ⁻¹ (dx / dy) (4) FIG. 38 shows a state in which another barycentric point is found at the lower right of the focused barycentric point in the state of size = 3, ws = 6. FIG. The hatched portion in the figure indicates a rectangle with one side 6.

The direction vector from the center of gravity to the center of gravity on the rectangle is represented by the difference between the coordinates of the respective centers of gravity. In this example, (dx, dy) = (3, 2), and θ = tan
⁻¹ (dx / dy) ≒ 0.98 [rad].

In step S806, the obtained direction is quantized, and the direction frequency histogram memory 5236
Will be recorded.

The range of quantization is as shown in FIG. 39, and θ is −π with respect to the horizontal direction (X-axis direction).
The region between // 2 and π / 2 (rad) is divided into four regions from region 0 to region 3 having an angle of π / 4. The determined direction is classified and recorded in any of these areas. For example, the direction vector θ ≒ in FIG.
Since it is 0.98, it is classified into the area 3.

If the direction vector is not classified into the region shown in FIG. 39 (θ> π / 2, θ <−π / 2), the vector whose direction is reversed is counted.

Therefore, in this embodiment, the frequency histogram memory 5236 has four memory areas for counting the directional frequencies of the areas 0 to 3, respectively. Each memory area has 16 bits.

In step S807, the processing moves to the next center of gravity. In step S808, it is determined whether the direction of the latest centroid point has been checked for all centroid points.
If it is ES, this routine ends.

If NO in step S808, the processing from step S801 is repeated. N in step S810
If O, the process proceeds to step S807.

By the above processing, the cumulative frequency of all the centroid points in the direction of the nearest centroid point is recorded in the direction frequency histogram memory 5236.

In step S611 of FIG. 34, the control device 521 extracts the characteristics of the intervals and the number of character-like regions of the input image from the information on the arrangement of the edge portions. A spatial differentiation process is performed on the image data to create a differential image of 8 bits per pixel. The created differential image is stored in the differential image memory 5237. The method of creating the differential image is the same as that of the first embodiment using the Sobel operator, and therefore the description thereof will not be repeated.

In step S612, control device 52
1 creates a density projection histogram, which is density data projected on the X and Y axes, based on the differential image stored in the differential image memory 5237. The density projection histogram is stored in the density projection histogram memory 5238.

Projecting an image on a certain axis means repeating the operation of sequentially adding the densities of pixels along a straight line in a direction perpendicular to the axis and calculating the sum by translating the position of the straight line. Thus, a one-dimensional density array (waveform) is obtained. For example, assuming a circle in which the density is constant in all parts as shown in FIG.
Each of the density projection histograms on the axis and the Y axis is a semi-elliptical histogram as shown.

As shown in FIG. 41 (a), the density projection histogram memory 5238 has an address configuration in which the X-axis and Y-axis coordinates are arranged in order, and the density projection value 16 bits for each coordinate is defined as one unit. It has a memory area of the sum of the number of X coordinates and Y coordinates (512 + 432 = 944 in this case).

If the number of character strings in the image is small (that is, if there are many blank portions), the density projection histogram in the direction of the character string changes little and its value becomes small.

In step S613, control unit 52
1 differentiates the density projection histogram and records it in the histogram differential value memory 5240.

This processing is performed to emphasize the change of the density projection histogram and make it easy to understand. The specific contents of this processing will be described below.

The control device 521 applies the respective differential values x_hist 'to the density projection histogram on the X and Y axes recorded in the density projection histogram memory 5238.
[X] and y_hist '[y] are calculated by Expressions (5) and (6), and the histogram differential value memory 5240 is calculated.
To record.

X_hist ′ [x] = (x_hist [x + 1] −x_hist [x−1]) / 2, 1 ≦ x ≦ 510 (5) y_hist ′ [y] = (y_hist [y + 1] −y_hist [y− 1]) / 2, 1 ≦ y ≦ 431 (6) where x_hist [i] and y_hist [i] are data (density projection values) of the density projection histogram memory 5238 at the coordinate i.

The processing in step S613 will be described with reference to FIG. As shown in FIG. 41B, the histogram differential value memory 5238 takes addresses arranged in the order of each coordinate of the X coordinate / Y coordinate, and sets the X coordinate of the differential value 16 bits of the density projection value for each coordinate as one unit. And the number of Y coordinates (in this case, 512 + 432 = 944). As shown in FIG. 41, the control device 541
Is, for example, X =
A differential value is calculated from the contents of 0 and X = 2 using Expressions (5) and (6), and X =
Record at address 1. This calculation is performed for all values on the X and Y axes. Note that the differential value calculation requires the presence of coordinates before and after the pixel of interest, so that the differential value is not calculated for the first and last coordinates (here, coordinates 0 and 511 or 431). That is, as shown in FIG. 41B, X =
No data is recorded at each address of 0, X = 511, Y = 0, and Y = 431.

A business card image generally has a small number of characters, but the characters are clearly arranged in one direction, and the space between lines is large. Therefore, a business card image has a smaller number of effective connected components than other images, has a higher concentration of effective connected components in a specific direction (a concentration of frequencies in a specific direction of the center of gravity arrangement direction), and has a higher differential density. There is a tendency that the average value of the projection histogram is small. Based on these tendencies, it is determined whether or not the input image is a business card image.

At step S614, step S6
It is determined whether the number of valid connected components stored at 08 is equal to or less than a threshold value. The threshold value is set, for example, to a value of 200, which is stored in the ROM 522.

If YES in step S614, it is determined in step S615 whether the frequency of the direction from the center of gravity to the center of gravity closest to the center of gravity is equal to or greater than a threshold value. The threshold value is set, for example, such that the ratio of the mode value of the direction vector to the number of valid connected components is 50%.

If YES in step S615, the
In step S616, the histogram differential value memory 524
The average value in the X-axis direction of the histogram recorded at 0,
Larger and smaller average values in the Y-axis direction
First, it is determined whether the value is equal to or smaller than the threshold value. Predetermined threshold
Is, for example, 4.5 × 1 with respect to the larger average value.
0 ^Three, 1.5 × 10 with respect to the smaller average^ThreeTo
Are set and stored in the ROM 522.

Although the respective thresholds are stored in the ROM 522, the thresholds may be freely set by the user. In this case, the setting processing routine may be stored in the ROM 522.

If YES in step S616, it is determined in step S617 that the input image is a business card image.

If NO in any of steps S614, S615, and S616, it is determined in step S619 that the input image is not a business card image.

In step S618, the input image is recorded in the image database device 53 together with a keyword (attribute) as to whether or not the image is a business card image.

As a result, the attribute of a given business card can be used at the time of processing for searching each pixel. Further, by applying character recognition to an image determined to be a business card, applications such as creating an address book are possible.

Next, a specific example of the differential value of the differential density projection histogram of the business card image will be described.

FIG. 44 is a graph showing an example of a differential value of a differential density projection histogram of a business card image, and FIG. 45 is a graph showing an example of a differential value of a differential density projection histogram of a document image other than a business card image.

Referring to the figure, the business card image has the average value of the density projection histograms with respect to the X-axis and the Y-axis of 2.26 × 10 ³ and 0.86 × 10 ³ , respectively. For the document image, the larger of the average values of the density projection histograms with respect to the X axis and the Y axis is 4.3 ×
10 ³ , the smaller one is 2.5 × 10 ³ .

Thus, it can be seen that the average value of the differential values of the differential density projection histogram is smaller in the business card image than in other document images.

FIG. 42 shows actual classification results. FIG.
In the graph of FIG. 2, the X axis is the number of effective connected components of the input image, and the Y axis is the frequency ratio of the effective connected components of the input image in the direction of the nearest centroid.

As is clear from the figure, the business card image has a tendency that the number of effective connected components is small and the frequency ratio in the direction of the center of gravity is large recently.

The graph of FIG. 43 is a region surrounded by hatching in the graph of FIG. 42 (the number of valid connected components is 200 or less,
In addition, only the sample images included in the frequency ratio of 50% or more in the recent direction of the center of gravity are graphed only. The X axis of the graph is the smaller one of the average values of the differential values of the differential density projection histogram with respect to the X axis and the Y axis of the input image, and the Y axis of the graph is the larger value. In this example, 1.5 with respect to the X axis in FIG.
× 10 ^3, Y understood that it is possible to classify the business card image by a threshold value of 4.5 × 10 ³ relative to axis.

As described above, according to this embodiment, a business card image can be accurately classified from the density projection histogram of an effective connected component image and a differential image by utilizing the characteristics of the business card image. If it is determined to be a business card image,
Other applications such as address book creation, and applications such as automatically proceeding to processing are possible.

In this embodiment, there are three data: the number of effective connected components for determining a business card image, the direction of another effective connected component close to the effective connected component, and the average of the differential values of the differential density projection histogram. Is used, the accuracy may be lowered, but the determination is possible even if the determination is made based on the data of 1 or 2 among them.

(Fifth Embodiment) FIG. 46 is a flowchart showing the processing of the control device of the image classification device according to the fifth embodiment of the present invention.

The configuration of the apparatus according to this embodiment is the same as the configuration of the image classification apparatus according to the fourth embodiment shown in FIG. 32, and therefore the description thereof will not be repeated.

The processing performed in the image classification device according to the present embodiment is as follows.

After the processing of steps S601 to S607 in FIG. 33 is performed, the routine shown in FIG. 46 is executed.

In step S608, control unit 521
Calculates the number of valid connected components in the binary image memory 5232 as one of the feature amounts for classifying the business card image, considering the valid connected components in the image as a character-like region,
It is stored in the effective connected component number memory 5239. Further, in step S609, the control device 521 extracts a feature amount of the arrangement of effective connected components in the binary image memory 5232. Examples of the feature amount of the arrangement of the effective connected components include the degree of concentration of the arrangement of the effective connected components in a specific direction and the density projection of the differential image of the input image as described in the fourth embodiment of the present invention. A feature amount such as the average value of the differential value of the histogram that often shows the properties of the business card image is used.

In step S610, to determine whether or not the input image is a business card image, control device 521 first checks whether or not the number of valid connected components is equal to or smaller than a threshold value. Since the number of characters is small in the business card image, the number of effective connected components is also small. If the number of valid connected components is not equal to or smaller than the threshold (NO in step S610), it is determined in step S614 that the input image is not a business card image.

If the number of valid connected components is equal to or smaller than the threshold value (YES in step S610), then in step S611, the feature amount of the sequence of valid connected components is compared with a predetermined threshold value. The predetermined threshold value may be stored in the ROM 522, or may be externally specified by the user. The feature amounts to be compared may be a single feature amount or a combination of a plurality of feature amounts. A single threshold or a combination of a plurality of thresholds is used as the predetermined threshold corresponding to the feature quantity to be compared, and a threshold condition to be satisfied by the document image is set.
Those processing procedures are stored in the ROM 522.

In step S612, control device 52
As a result of the comparison between the feature value and a predetermined threshold value, it is determined whether or not the input image satisfies a condition for being a business card image. If the condition is satisfied (Y in S612)
ES), it is determined that the input image is a business card image, otherwise (NO in step S612), it is determined that the input image is not a business card image.

After the determination, in step S615, the control device 521 registers the input image in the image database device 53. At this time, a keyword (attribute) “business card image” is added to the image determined to be a business card image. Then, it is registered in the image database device 53 together with the input image data. Thus, the attribute of a given business card can be used at the time of processing at the time of searching each image.

As described above, according to the present invention, the number of character-like areas and their arrangement are examined to determine whether or not the entire image satisfies the conditions to be satisfied by the business card image. Of a business card image that is widely open, and can be classified with high accuracy.

When the image is classified as a business card image, application such as automatic creation of an address book by further applying character recognition becomes possible.

Although the image data input in the embodiment of the present invention is multi-valued image data, binary image data may be input.

[0198]

The image classification apparatus according to the present invention can classify an image into a document image and a document image.

[0199]

[0200]

[0201]

[0202]

[0203]

[0204]

[0205]

According to the image classification device of the third aspect , in addition to the effect of the first or second aspect , a pixel satisfying a predetermined condition among a plurality of adjacent pixels representing an object is identified as a connected component. Accuracy can be further increased.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a system configuration of an image classification device according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a processing routine of a control device 121 of FIG.

FIG. 3 is a flowchart following the flowchart of FIG. 2;

FIG. 4 is a diagram illustrating an example of a document image.

FIG. 5 is a diagram illustrating an example of a landscape image that is a non-document image.

FIG. 6 is a diagram for describing a coordinate system of image data stored in a multi-valued image memory 1231.

FIG. 7 is a diagram for explaining an address configuration of a multi-valued image memory 1231.

FIG. 8 is a diagram illustrating an example of binary image data and describing connected components.

FIG. 9 is a diagram showing a result of performing labeling on the connected component of FIG. 8;

FIG. 10 is a diagram showing an address configuration of a label image memory.

FIG. 11 is a diagram for explaining a method of setting fillet coordinates.

FIG. 12 is a diagram showing an address configuration of a fillet coordinate memory 1234.

FIG. 13 is a diagram illustrating an example of a connected component.

FIG. 14 is a diagram showing a state in which the connected components of FIG. 13 are recorded in a fillet coordinate memory.

FIG. 15 is a diagram showing a connected component after unnecessary connected components have been removed from FIG. 13;

FIG. 16 is a diagram for explaining data for performing a spatial differentiation process on multi-valued image data in a multi-valued image memory.

FIG. 17 is a first diagram illustrating coefficients used for differentiation.

FIG. 18 is a second diagram for describing coefficients used for differentiation.

19 is a diagram illustrating a state in which one connected component in the document image in FIG. 4 is recorded in a binary image memory 1232. FIG.

20 is a diagram illustrating a state where the same connected component as the connected component illustrated in FIG. 19 is recorded in the differential binary image memory 1234. FIG.

FIG. 21 is a diagram for explaining the same connected components as those in FIG. 19 in a state recorded in an AND image memory 1237.

22 is a diagram illustrating a state in which one connected component in the landscape image of FIG. 5 is recorded in a binary image memory 1232. FIG.

FIG. 23 is a diagram showing a state where the same connected component as that shown in FIG. 21 is recorded in the differential binary image memory 1234.

FIG. 24 is a diagram for explaining the same connected components as those in FIG. 23 in a state recorded in an AND image memory 1237.

FIG. 25 is a diagram for explaining effects in the first embodiment of the present invention.

FIG. 26 is a block diagram illustrating a device configuration of an image classification device according to a second embodiment of the present invention.

FIG. 27 is a flowchart illustrating a process performed by a control device of the image classification device according to the second embodiment.

FIG. 28 is a diagram illustrating an example of binary image data in a document image.

FIG. 29 is a diagram illustrating an example of binary image data in a landscape image (non-document image).

FIG. 30 is a diagram for explaining a bin ratio of a document image and a non-document image.

FIG. 31 is a diagram for describing processing of the image classification device according to the third embodiment of the present invention.

FIG. 32 is a block diagram illustrating a system configuration of an image classification device according to a third embodiment of the present invention.

FIG. 33 is a flowchart illustrating a process performed by the control device 521 of the image classification device according to the fourth embodiment of the present invention.

FIG. 34 is a flowchart following FIG. 33.

FIG. 35 is a flowchart following FIG. 34;

FIG. 36 is a diagram showing an address configuration of a centroid memory 5235.

FIG. 37 is a flowchart showing a specific routine of processing of the center of gravity performed in step S610 of FIG. 34;

38 is a diagram showing a state where another barycenter point is found at the lower right of the barycenter point of interest in the state of size = 3, ws = 6 in the processing of FIG. 37.

FIG. 39 is a diagram for describing an area for quantizing a determined direction.

FIG. 40 is a diagram for describing an example of density projection of image data on an X axis and a Y axis.

FIG. 41 is a diagram for explaining a density projection histogram memory 5238 and a histogram differential value memory 5240.

FIG. 42 is a diagram for describing effects of the fourth embodiment of the present invention.

FIG. 43 is a diagram for describing an average value of differential values of a differential density projection histogram of image data included in a hatched portion in FIG. 42;

FIG. 44 is a diagram for describing a density projection histogram of a business card image in the X-axis direction and the Y-axis direction.

FIG. 45 shows the X-axis direction and Y of a document image other than a business card image.
It is a figure for explaining density projection histogram of an axial direction.

FIG. 46 is a flowchart showing a process of the control device according to the fifth embodiment of the present invention.

[Explanation of symbols]

 11, 51 Image input device 12, 52 Image processing device 13, 53 Image database device 121, 521 Control device 122, 522 ROM 123, 523 RAM 1231, 5231 Multivalued image memory 1232, 5232 Binary image memory 1233, 5233 Label image Memory 1234, 5234 Fillet coordinate memory 1235, 5237 Differential image memory 1236 Differential binary image memory 1237 AND image memory 5235 Center of gravity memory 5236 Direction frequency histogram memory 5238 Density projection histogram memory 5239 Effective connected component number memory 5240 Histogram differential value memory

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) G06K 9/20 H04N 1/40

Claims (3)

(57) [Claims] 1. An identification means for identifying a connected component consisting of a plurality of adjacent pixels representing an object from input image data, and character-likeness based on the shape of the connected component in the input image data. Extracting means for extracting a characteristic amount representing the following: and determining means for determining whether or not the input image data is a document image based on the extracted characteristic amount , wherein the input image data is shaded. Image data, and the extracting means extracts the image data from the input grayscale image data.
Edge detection means to detect data areas with strong edge
Seen, the area of the connected component, and the connecting component the data area
An image classification device that extracts the feature amount from the area of the overlap region with 2. An object is selected from input image data.
Identify connected components consisting of adjacent pixels to represent
Identification means, and whether the shape of the connected component in the input image data is
Extracting means for extracting a characteristic amount representing character-likeness from the input image based on the extracted characteristic amount.
Determining means for determining whether or not the data is a document image, wherein the extracting means includes a circumscribed rectangle for each of the connected components.
Calculating means for calculating the area of the characteristic, and calculating the characteristic from the area of the connected component and the area of the circumscribed rectangle.
An image classification device that extracts the amount of collection . 3. The image processing apparatus according to claim 2, wherein the identification unit is configured to output the input image data.
Data of adjacent images representing the object
A pixel satisfying a predetermined condition is identified as the connected component.
The image classification device according to claim 1 or 2, wherein: