JP2017211976A - Image processing device and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly accurately extract scene characters.SOLUTION: An image processing device includes: an extreme value region extraction unit for extracting an extreme value region in which pixels of a first region and a second region continue from an image on the basis of the first region which is contained in the image and in which plural pixels with pixel values being equal to or less than a first value continue and the second region which is contained in the image and in which plural pixels with the pixel values being equal to or less than the second value larger than the first value continue; and a determination unit for determining the degree of character likeness of the shape of the region contained in the image. The determination unit determines the degree of character likeness of the extreme value region on the basis of the long sides and short sides of a plurality of rectangles externally contacting the extreme value region extracted from the image and rotated by a plurality of angles on the plane of the image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus and an image processing program.

  2. Description of the Related Art Conventionally, a technique for extracting a connected component that is a continuous cluster of pixels with similar colors from an image captured by a camera and recognizing a character (scene character) reflected in the image based on the extracted connected component is known. It has been.

  When this scene character is detected, an area including each pixel (connected component) having a minimum (or maximum) pixel value such as a luminance value in the image is set as an initial area, and pixel values included in the area are stepwise. A technique for exhaustively extracting an extreme region (Extremal Region), which is a region having a pixel value lower (or higher) than that of the surrounding area by making it larger (or smaller) is known (Non-Patent Document 1). ).

  In the technique described in Non-Patent Document 1, for example, compared to the conventional method of extracting connected components using binarization, the number of connected components to be detected increases, so the amount of calculation increases, but the contrast with the surroundings increases. Since connected components that are not so high in difference can be extracted, there are few detection errors of connected components.

  FIG. 1 is a diagram illustrating a conventional processing flow described in Non-Patent Document 1. In the following description, it is assumed that the pixel having the minimum pixel value v has the darkest color. First, connected components whose pixel value v is the minimum value (v = 0) are extracted. Subsequently, connected components whose pixel value v is 0 or 1 (v = 0 to t (t = 1)) are extracted. Subsequently, the connected component is acquired while adding t one by one until the pixel value v becomes 0 to the maximum value (v = 0 to t (t = 255)). Thereby, it is possible to comprehensively obtain connected components that are darker than the background.

  Subsequently, on the contrary to the above process, a process of extracting a connected component whose pixel value v is the maximum value (v = 255) is performed. That is, a connected component having a pixel value v of 255 or 254 (v = t to 255 (t = 254)) is extracted. Subsequently, the connected component is acquired while subtracting t by 1 until the pixel value v becomes 255 to the minimum value (v = t to 255 (t = 0)). As a result, connected components that are lighter in color than the background can be comprehensively acquired. For example, a white character region can be extracted on a black background.

  Then, a feature such as the shape of the extracted extreme value region is calculated, and it is determined whether or not it has a character-like feature. At this time, in the technique described in Non-Patent Document 1, the calculation cost is reduced by using a feature based on incrementally computable descriptors (IC descriptors).

  Note that incremental calculation is possible when there is a certain area (parent area) and multiple areas (child areas) obtained by dividing the area, using the descriptor already calculated in the child area. Says that the region descriptor can be calculated. That is, in the process of extracting the extreme value region, when calculating a descriptor of a certain parent region, the calculation can be omitted for a range already calculated in the child region. Therefore, the calculation cost is on the order of N with respect to the number N of pixels of the image to be processed, and high-speed processing is possible.

  Next, descriptors that can be calculated incrementally will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a descriptor that can be calculated incrementally in the related art.

  FIG. 2 shows a calculation example of the area (the number of pixels in the region), which is a representative descriptor that can be calculated incrementally. It is assumed that there are certain child areas c1 and c2, and there is a parent area p in which the two child areas are connected, and the number of pixels in each area is 5, 4, and 9, respectively.

  At this time, when the area of each region is obtained individually, the required number of calculations is 18 because it is the sum of the number of pixels. Here, when the areas of c1 and c2 have been calculated and are included in the parent region p, the area of p can be obtained simply by adding the areas of c1 and c2. It is said that descriptors with such characteristics can be calculated incrementally. Using the fact that the area can be calculated incrementally, the total number of calculations required to determine the area of each of the areas c1, c2, and p can be 10 times.

In the technique described in Non-Patent Document 1, using a feature based on a descriptor that can be calculated incrementally, a primary determination is made as to whether or not the character is a character, and only for an area that is determined to be a character in the primary determination, High-precision secondary determination is performed using complicated features with higher calculation costs. Thereby, high-speed scene character detection can be realized. The descriptors that can be calculated incrementally are Area descriptor [a], Bounding Box descriptor [x _min , y _min , x _max , y _max ], Perimeter length (Perimeter) Descriptor [p], Euler number descriptor [e], Horizontal crossings descriptor [ci], and the aspect ratio feature [w / h] ( Where w = x _max −x _min , h = y _max −y _min ), compactness feature [√a / p], hole number feature [1-e], and horizontal crossing number feature, Primary judgment is performed.

  Here, the feature of the number of intersections in the horizontal direction is expressed by the following equation.

In the secondary determination, for the area that broke through the primary determination, the hole area ratio feature, convex hull rate feature, and inflection point feature on the outer periphery with higher calculation cost are calculated, and the features used in the primary determination are calculated. A total of seven features are used.

L. Neumann, J. Matas, "Real-Time Scene Text Localization and Recognition", Proc. In 25th IEEE Conf. On Computer Vision and Pattern Recognition (CVPR), pp.3538-3545 (2012)

  However, the conventional technique described in Non-Patent Document 1 has a problem that the accuracy of extracting scene characters such as kanji and kana having a complicated figure shape is lower than that of numerals and alphabets.

  Therefore, an object is to extract scene characters with high accuracy.

  In the image processing apparatus, a first area included in the image, a first area where a plurality of pixels having pixel values equal to or less than the first value are continuous, and a second area included in the image. And a second region in which a plurality of pixels having a pixel value equal to or smaller than a second value greater than the first value are continuous, and a pole in which the pixels in the first region and the second region are continuous. An extreme value region extraction unit that extracts a value region from the image; and a determination unit that determines a character-like degree of the shape of the region included in the image, wherein the determination unit extracts the extreme value extracted from the image A degree of character likeness in the extreme value region is determined based on long sides and short sides of the plurality of rectangles circumscribing the region and rotated by a plurality of angles on the plane of the image.

  According to the disclosed technique, a scene character can be extracted with high accuracy.

It is a figure explaining the conventional process flow described in the nonpatent literature 1. FIG. It is a figure explaining the example of the descriptor which can be calculated incrementally in a prior art. It is a functional block diagram which shows the function structure of the image processing apparatus which concerns on this embodiment. It is a flowchart which shows an example of a process of an image processing apparatus. It is a flowchart which shows an example of an extreme value area | region extraction process. It is a figure explaining the example of calculation of a rotation boundary rectangle descriptor. It is a figure explaining the example of calculation of a rotation aspect-ratio characteristic. It is a flowchart which shows an example of the extreme value area | region primary extraction process which seems to be a character. It is a figure which shows the example of the tree structure of an extreme value area | region. It is a flowchart which shows an example of the extreme value area | region secondary extraction process which seems to be a character. It is a figure explaining the process which abbreviate | omits determination of the character degree with respect to the extreme value area | region of a child. It is a figure which shows the example of the original image in which the three-dimensional character was reflected. It is a flowchart which shows an example of a solid side surface removal process. It is a figure explaining the process which removes the shadow part which is the side of a solid character. It is a figure explaining the process which removes the shadow part which is the side of a solid character. It is a flowchart which shows an example of each extreme value area | region primary extraction process like a character string. It is a flowchart which shows an example of each extreme value area | region secondary extraction process like a character string. It is a figure explaining each extreme value area | region secondary extraction process like a character string. It is a flowchart which shows an example of each extreme value area | region primary extraction process like the character string which concerns on 2nd Embodiment. It is a figure explaining the process which searches the same character string in each of several angles. It is a flowchart which shows an example of the process which searches the same character string. It is a figure explaining the process which searches the same character string.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
<Functional configuration of image processing apparatus>
First, the functional configuration of the image processing apparatus according to the present embodiment will be described with reference to the drawings. FIG. 3 is a functional block diagram showing a functional configuration of the image processing apparatus according to the present embodiment.

  The image processing apparatus 10 includes an image acquisition unit 11, an extreme value region extraction unit 12, a calculation unit 13, a character degree determination unit 14, a removal unit 15, a character string region detection unit 16, an estimation unit 17, a detection unit 18, and a storage unit 19. And an output unit 20.

  The image acquisition unit 11 acquires data of an image (scene image) in which characters are shown. The image acquired by the image acquisition unit 11 may be, for example, an image captured by an imaging device such as a camera or a video camera, or may be an image stored in advance. Further, the above-described imaging device may be provided inside the image processing device 10 or may be provided outside. Further, the above-described image may be a still image or an image frame included in a moving image.

  The extreme value region extraction unit 12 includes a first region that is a continuous cluster of pixels having a pixel value equal to or less than the first value from the image acquired by the image acquisition unit 11, and a pixel value that is greater than the first value. A second region that is a continuous cluster of pixels that are less than or equal to the second large value is extracted. Then, the extreme value region extraction unit 12 extracts (generates) an extreme value region that is a continuous cluster of pixels in the first region and the second region.

  The calculation unit 13 is a rectangle circumscribing the first region extracted by the extreme value region extraction unit 12, and each rectangle circumscribed by the predetermined region on the plane of the image circumscribes the first region. Are calculated (first coordinates).

  The calculating unit 13 is a rectangle circumscribing the second region extracted by the extreme value region extracting unit 12, and a rectangle rotated by a predetermined angle on the plane of the image circumscribes the second region. The coordinates of each position (second coordinates) are calculated.

  The calculation unit 13 is a rectangle circumscribing the extreme value region extracted by the extreme value region extraction unit 12 based on the first and second coordinates, and has a predetermined angle on the plane of the image. Calculate the aspect ratio of the rectangle rotated by.

  The character degree determination unit 14 determines the degree of character likeness of the shape of the region.

  The character degree determination unit 14 determines the shape of the first region when the degree of character-likeness of the shape of the extreme value region is equal to or greater than a predetermined threshold and the aspect ratio calculated by the calculation unit 13 is within a predetermined range. The determination of the degree of character like is omitted.

  In addition, the character degree determination unit 14 determines the first region when the degree of character-likeness of the shape of the extreme value region is equal to or greater than a predetermined threshold and the aspect ratio calculated by the calculation unit 13 is not within the predetermined range. The degree of character of the shape is determined.

  The removal unit 15 removes the region on the side surface of the three-dimensional character from the extreme value region.

  The character string region detection unit 16 detects each extreme value region included in the character string in the image acquired by the image acquisition unit 11 based on the height (size) of the plurality of extreme value regions.

  The estimation unit 17 estimates the position and size of each character included in the character string based on the position and size of each extreme value region detected by the character string region detection unit 16.

  The detection unit 18 detects an extreme value region included in each character included in the character string from a range corresponding to the position and size of each character estimated by the estimation unit 17.

  The storage unit 19 stores extreme value region data detected by the extreme value region extraction unit 12.

  The output unit 20 outputs a region that seems to be a character string detected by the detection unit 18. Note that the output unit 20 may present the position of the region on the image to the user, or may output shape data of the region to a character recognition device or the like.

<Processing>
Next, an overview of the processing of the image processing apparatus 10 will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of processing of the image processing apparatus 10.

  First, in step S1, the image acquisition unit 11 acquires an image.

  Subsequently, in step S2, the extreme value region extraction unit 12 extracts an extreme value region from the image.

  Subsequently, in step S3, the character degree determination unit 14 primarily determines whether or not each extreme value region extracted in step S2 is likely to be a character, and extracts each extreme value region likely to be a character.

  Subsequently, in step S4, the character degree determination unit 14 secondarily determines whether or not each extreme value region extracted in step S3 is likely to be a character, and extracts each extreme value region likely to be a character.

  Subsequently, in step S5, the character string region detection unit 16 determines whether or not each extreme value region extracted in step S4 seems to be a character string, and extracts each extreme value region likely to be a character string.

  Subsequently, in step S6, the detection unit 18 determines in a relatively detailed manner whether or not each extreme value region extracted in step S5 is a character string, and extracts each extreme value region that seems to be a character string. In step S6, each extreme value region that seems to be a character string, which is missed in step S5, is extracted.

≪Extreme area extraction processing≫
Next, with reference to FIG. 5, the process of extracting the extreme value region from the image in step S2 will be described. FIG. 5 is a flowchart illustrating an example of the extreme value region extraction process.

  First, in step S101, the extreme value region extraction unit 12 sets a pixel value to be extracted as a minimum value (for example, 0), and deletes information stored in the work buffer (empty).

  Subsequently, in step S102, the extreme value region extraction unit 12 extracts all connected components that are pixel values to be extracted.

  Subsequently, in step S103, the extreme value region extraction unit 12 selects one connected component of the extracted connected components as a processing target.

  Subsequently, in step S104, the feature calculation unit 121 of the extreme value region extraction unit 12 calculates a rotation boundary rectangle descriptor for the connected component to be processed. A method for calculating the rotation boundary rectangle descriptor will be described later. Here, not only the rotation boundary rectangle descriptor but also an IC descriptor such as a conventional area descriptor may be calculated.

  Subsequently, in step S105, the extreme value region extraction unit 12 determines whether or not a reference to one or more extreme value regions adjacent to the connected component to be processed is stored in the work buffer.

  If a reference to the extreme value region adjacent to the connected component to be processed is not stored in the work buffer (S105: NO), in step S106, the connected component to be processed is set as a new extreme value region in the extreme value region storage buffer. The reference to the new extremum area is stored in the work buffer, and the process proceeds to step S112 described later.

  If a reference to the extreme value region adjacent to the connected component to be processed is stored in the work buffer (S105: YES), in step S107, the extreme value region extraction unit 12 sets each of the adjacent adjacent components to be processed. Delete the reference to the extreme value area from the working buffer.

  In step S108, the extreme value region extraction unit 12 connects each extreme value region adjacent to the connected component to be processed and the connected component to be processed to form a new extreme value region.

  Subsequently, in step S109, the feature calculation unit 121 of the extreme value region extraction unit 12 calculates a rotation boundary rectangle descriptor for the new extreme value region.

  Subsequently, in step S110, the extreme value region extraction unit 12 stores the new extreme value region in the extreme value region storage buffer as a parent of each extreme value region adjacent to the connected component to be processed.

  Subsequently, in step S111, the extreme value region extraction unit 12 stores a reference to the new extreme value region in the work buffer.

  Subsequently, in step S112, the extreme value region extraction unit 12 determines whether there is an unselected connected component as a processing target among the connected components extracted in step S102.

  If there is an unselected connected component as a processing target (S112: YES), an unselected connected component is selected as a processing target in step S113, and the process proceeds to step S104.

  If there is no unselected connected component to be processed (S112: NO), in step S114, the extreme value region extracting unit 12 increases the value of the pixel value to be extracted by one.

  Subsequently, in step S115, the extreme value region extraction unit 12 determines whether or not the value of the pixel value to be extracted exceeds a maximum value (for example, 255).

  When the pixel value to be extracted does not exceed the maximum value (S115: NO), the process proceeds to step S102.

  If the pixel value to be extracted exceeds the maximum value (S115: YES), the process ends.

  In the above, for example, in order to extract black characters, the initial value of the pixel value to be extracted is set to the minimum value and incremented by 1 from the initial value, thereby comprehensively acquiring connected components that are darker than the background. The processing to be described has been described.

  On the other hand, for example, white characters are also extracted, so that the initial value of the pixel value to be extracted is set to the maximum value and is reduced by 1 from the initial value, so that connected components that are lighter in color than the background are also comprehensively acquired. May be.

≪Rotation boundary rectangle descriptor calculation process≫
Next, the process of calculating the rotation boundary rectangle descriptor in step S104 and step S109 will be described.

The feature calculation unit 121 calculates a rotation boundary rectangle descriptor having four values of d _min, d _max, d ⁻ _{min, and} d ⁻ _min .

Assuming that each pixel included in the extreme value region P is p _i , the coordinates of p _i are x _i , y _i , and the height (length in the x direction) of the input image is h, d _min, d _max, d ⁻ _{min ,} D ⁻ _min are calculated by the following equations, respectively.

Here, when the rotation angle of the rotation boundary rectangle descriptor to be obtained is θ, c = tan (θ).

FIG. 6 is a diagram for explaining a calculation example of the rotation boundary rectangle descriptor. FIG. 6 shows a calculation example when the rotation angle of the rotation boundary rectangle descriptor is 45 degrees. When obtaining the rotation boundary rectangle descriptor in the coordinate system as shown in FIG. 6, the calculation of d _min, d _max, d ⁻ _{min and} d ⁻ _min is the upper left, the lower right, the lower left, and the highest, respectively. Equivalent to finding the pixel on the upper right.

Further, when calculating the rotation boundary rectangle descriptor of the extreme value region p such that a certain pixel value is v = 0 to t (t = i), v = 0 to t (t < i) If the extreme value region c is included, a pixel that overlaps the extreme value region c is not newly calculated, and the calculated extreme value region c descriptor and a region other than the extreme value region c are calculated. It is only necessary to perform calculation with the pixels. Therefore, the calculation cost for _obtaining d _min, d _max, d ⁻ _min, d ⁻ _min of all the extracted extreme value regions is N orders (O (N)) with respect to the number N of pixels.

  A rotation aspect ratio feature (rotated_aspect) that can be calculated by the following equation will be described as a feature obtained from the rotation boundary rectangle descriptor.

Here, a case where the aspect ratio when Expression (6) is rotated by 45 degrees will be described. First _, the vertical and horizontal lengths of the boundary rectangle when the extreme value region is rotated 45 degrees to the right from d _min, d _max, d ⁻ _{min, and} d ⁻ _min will be examined. Here, the boundary rectangle is a rectangle circumscribing the target region, and refers to a rectangle that surrounds the region and has the smallest area. d _{min and} d _max are the minimum and maximum values of x + y of the constituent pixels in the extreme value region. That is, it represents the value of n of the straight line closest to the origin and the farthest straight line among the straight lines of x + y = n passing through the pixels in the extreme value region. Therefore, the upper side of the boundary rectangle rotated 45 degrees to the right overlaps the straight line x + y = d _min , and the lower side overlaps the straight line x + y = d _max . That is, the distance between the two straight lines is the height of the boundary rectangle when rotated by 45 degrees.

Here, the following equation is used to obtain the distance between the point (x _i , y _i ) and the straight line ax + by + c = 0.

Thus, for example, when substituting (0, d _max ) for a point and x + y−d _min = 0 for a straight line in equation (7), the following is obtained.

Here, since d _max is always larger than d _min , Equation (7) can be modified as follows.

And this is the height of the boundary rectangle when rotated 45 degrees.

Considering d ^- _{min and} d ^- _{min in the} same manner, the horizontal width of the rotation boundary rectangle when rotated 45 degrees is as follows.

Therefore, equation (6) is obtained from the following equation. Further, the rotation boundary rectangle when rotated to an angle other than 45 degrees can be similarly obtained by Expression (6).

FIG. 7 is a diagram for explaining a calculation example of the rotation aspect ratio feature. FIG. 7 shows a calculation example of the 45-degree rotation aspect ratio feature obtained from Equation (6). As shown in FIG. 7A, a region having a contour shape close to a square with a low degree of elongation can obtain values close to 1 for both the conventional aspect ratio and the 45-degree rotation aspect ratio.

  On the other hand, as shown in FIG. 7 (B), when the region having a long and slender outline shape with a high degree of elongation is slanted, the conventional aspect ratio is close to 1, but the 45 degree rotation aspect ratio is It is about 0.53, and there is a large difference (difference) between the conventional aspect ratio and the 45 ° rotation aspect ratio.

  Thus, by looking at the difference between the conventional aspect ratio and the rotated aspect ratio when rotated by 45 degrees, it is possible to estimate the approximate extension degree of the contour shape of the extreme value region. In addition to the case where the rotation angle is 45 degrees, if the rotation aspect ratios of a plurality of angles are used in combination, the estimation accuracy of the extension degree is improved. In character string detection, for example, even when a rotation aspect ratio of about 22.5 degrees or 15 degrees is obtained, character detection can be performed with a certain accuracy. In the case of 22.5 degree increments, three of a 22.5 degree rotation aspect ratio, a 45 degree rotation aspect ratio, and a 67.5 degree rotation aspect ratio are obtained. In the case of 15 degree increments, five of 15 degree rotation aspect ratio, 30 degree rotation aspect ratio, 45 degree rotation aspect ratio, 60 degree rotation aspect ratio, and 75 degree rotation aspect ratio are obtained.

  In order to facilitate handling of the value of the rotation aspect ratio feature, the rotation aspect ratio feature may always be a value of 1 or less by the following equation, for example.

<< Characteristic extreme value area primary extraction process >>
Next, with reference to FIG. 8, the process of first determining whether or not a character appears in step S3 and extracting each extreme value region likely to be a character will be described. FIG. 8 is a flowchart showing an example of extreme value region primary extraction processing that seems to be a character.

  First, at this stage, since the number of each extreme value region stored in the extreme value region storage buffer is relatively large, it takes a very long time to calculate a feature amount with high calculation cost for each extreme value region. Therefore, a feature value using an IC descriptor with a low calculation cost is calculated for each extreme value region, and an extreme value region that is clearly not a character from the calculated feature value is deleted from the tree. In addition, about the determination of the character degree using the feature-value used here, what has been trained in advance using machine learning may be used, or a determination rule in which a fixed threshold value is manually set for each feature. You may make it.

  In step S <b> 201, the character degree determination unit 14 extracts an extreme value region from the extreme value region extraction unit 12 and selects one extreme value region as a processing target from a plurality of extreme value regions stored in the extreme value region storage buffer. To do.

  Subsequently, in step S202, the character degree determination unit 14 calculates a primary feature amount from the IC descriptor including the rotation boundary rectangle descriptor of the extreme value region to be processed. Thereby, since the feature-value used for primary extraction can be increased by one compared with a prior art, the performance of primary extraction improves.

  Subsequently, in step S203, the character degree determination unit 14 calculates the character-like degree (character degree) of the extreme value region to be processed based on the calculated primary feature amount.

  Subsequently, in step S204, the character degree determination unit 14 determines whether or not the calculated character degree is greater than a predetermined threshold.

  When the calculated character degree is not greater than the predetermined threshold (S204: NO), the process proceeds to step S207.

  When the calculated character degree is larger than the predetermined threshold (S204: YES), in step S205, the character degree determination unit 14 determines the processing target extreme value region and the parent extreme value region of the processing target extreme value region. It is determined whether the similarity is equal to or greater than a predetermined threshold. The similarity may be calculated using the similarity between the IC descriptor value of the parent extreme value region and the IC descriptor value of the child extreme value region.

  If the similarity with the parent extreme value region is not equal to or greater than the predetermined threshold (S205: NO), the process proceeds to step S208.

  If the degree of similarity with the parent extreme value region is equal to or greater than a predetermined threshold (S205: YES), in step S206, the character degree determination unit 14 determines that the character degree of the parent extreme value region is the extreme value region to be processed. It is judged whether it is larger than the character degree.

  When the character degree of the parent extreme region is larger than the character degree of the extreme region to be processed (S206: YES), the extreme region to be processed is deleted from the extreme region storage buffer in step S207.

  When the character degree of the parent extremum area is not greater than the character degree of the extremum area to be processed (S206: NO), in step S208, the character degree determination unit 14 saves each character stored in the extremum area storage buffer. It is determined whether or not there is an unselected extreme value region among the extreme value regions.

  If there is an unselected extreme value region (S208: YES), in step S209, an unselected extreme value region is selected as a processing target, and the process proceeds to step S202.

  If there is no unselected extreme value region (S208: NO), the process is terminated.

  Thereby, an extreme value region with a low character degree can be deleted from the extreme value region tree. In addition, when there is a parent and child of a similar extreme value region, the character with the lower character degree can be deleted from the extreme value region tree.

  By the above-described extreme value region extraction process, each extreme value region is stored in the extreme value region storage buffer in association with the parent extreme value region. Then, in the extreme value region primary extraction process that seems to be a character, the character degree is low and the IC descriptor value is close from the extreme value region storage buffer based on each extreme value region associated with the parent extreme value region Is deleted. FIG. 9 is a diagram illustrating an example of a tree structure of an extreme value region indicating an association between each extreme value region stored in the extreme value region storage buffer and a parent extreme value region at this time.

≪Extreme area secondary extraction process that seems to be a character≫
Next, with reference to FIG. 10, a process of secondly determining whether or not a character appears in step S <b> 4 and extracting each extreme value region likely to be a character will be described. FIG. 10 is a flowchart showing an example of extreme value region secondary extraction processing that seems to be a character.

  In step S301, the character degree determination unit 14 first deletes the information stored in the work buffer (empty).

  Subsequently, in step S302, the character degree determination unit 14 determines the extreme value of the parent from the extreme value region storage buffer in which the extreme value region extracted by the extreme value region primary extraction process that seems to be the character is stored. All extremal areas that are not associated and stored are stored in the work buffer. It should be noted that the extreme value region where the parent extreme value region is not associated and stored is each extreme value region 501 at the highest level (root) in the tree diagram of FIG.

  Subsequently, in step S303, the character degree determination unit 14 selects one extreme value region from among a plurality of extreme value regions stored in the work buffer as a processing target.

  Subsequently, in step S304, the character degree determination unit 14 deletes the reference to the extreme value area to be processed from the work buffer.

  Subsequently, in step S305, the character degree determination unit 14 calculates the character degree of the extreme value region to be processed. Note that in step S305, a feature amount that is more costly to calculate may be used than in the process of step S203 described above. For example, the hole area rate feature, the convex hull rate feature, and the inflection point number feature on the outer periphery are used as in the prior art. In addition to this, an IC descriptor including a rotation boundary rectangle descriptor may be used. As a result, by the above-described primary extraction process of extreme values like characters, only the number of extreme values reduced to a certain extent is used to calculate the character degree using the feature amount that is expensive to calculate.

  Subsequently, in step S306, the character degree determination unit 14 determines whether or not the calculated character degree is greater than a predetermined threshold.

  If the calculated character degree is not greater than the predetermined threshold (S306: NO), a solid side surface removal process described later is performed in step S307.

  Subsequently, in step S308, the character degree determination unit 14 calculates the character degree of the processing target extreme value region by setting the extreme value region after the three-dimensional side surface removal processing as the processing target extreme value region. This process may be the same as the process in step S305.

  Subsequently, in step S309, the character degree determination unit 14 determines whether or not the calculated character degree is larger than a predetermined threshold value. This process may be the same as the process in step S306.

  When the calculated character degree is larger than the predetermined threshold (S309: YES), the process proceeds to step S310 described later.

  If the calculated character degree is not greater than the predetermined threshold (S309: NO), the process proceeds to step S313 described later.

  When the character degree calculated in step S306 is larger than the predetermined threshold (S306: YES), in step S310, the character degree determination unit 14 determines whether or not the shape of the extreme value region to be processed is elongated (the degree of expansion is large). To determine. A process for determining whether or not the shape of the extreme value region to be processed is elongated will be described later.

  If the extreme region to be processed is not elongated (S310: NO), in step S311, the character degree determination unit 14 adds a reference to the extreme region to be processed to the first list, and the processing in step S314 described later. Proceed to As a result, extreme values regions that are likely to be characters are stored in the first list.

  If the extreme region to be processed is elongated (S310: YES), in step S312, the character degree determination unit 14 adds the extreme region to be processed to the second list. As a result, extremal areas that are character-like and elongated are stored in the second list.

  Subsequently, in step S313, the character degree determination unit 14 adds an extreme value region, which is a child of the extreme value region to be processed, to the work buffer.

  Subsequently, in step S314, the character degree determination unit 14 determines whether or not an unselected extreme value region exists in the work buffer.

  If there is an unselected extremal area in the work buffer (S314: YES), in step S315, the unselected extremal area is selected as a processing target, and the process proceeds to step S304.

  If there is no unselected extremal area in the work buffer (S315: NO), the process ends.

  Thus, for each extreme value region stored in a tree structure in the extreme value region storage buffer, the character degree is determined in order from the parent of the tree, and for an extreme value region with a high character degree, a child of the extreme value region is determined. The character degree determination process for the extreme value region is omitted. This is because it can be assumed that the extreme value region that is a child of the extreme value region having a high character degree includes only a part of the character.

  Further, at that time, for the long extremum area, it is possible to prevent erroneous determination of the extremum area where a plurality of characters are connected by not omitting the character degree determination process for the extremum area of the child of the extremum area. it can.

  Referring to FIG. 11, an example in which the character degree determination process for the child extreme value region is omitted will be described. FIG. 11 is a diagram illustrating a process of omitting the character degree determination for the child extreme value region. As shown in FIG. 11, by using the character degree and determination of whether or not the extreme value region is elongated, detection of a region for one character while preventing erroneous detection of a region where a plurality of characters are connected, Judgment processing can be omitted.

≪3D side removal processing≫
Next, the three-dimensional side surface removal process in step S307 will be described with reference to FIGS. FIG. 12 is a diagram illustrating an example of an original image in which a three-dimensional character is shown. In the prior art described in Non-Patent Document 1, it is possible to extract only a region where all background pixels adjacent to a connected component pixel are darker or lighter than the connected component color.

  Therefore, for example, as shown in FIG. 12, when a black character object is three-dimensionally reflected on a white wall, the character background (wall surface) is lighter than the character color, but the shadow portion of the raised character (three-dimensional character) The side) is darker than the text color. In this case, assuming that the initial value of the pixel value v when extracting the connected components is the minimum value (v = 0), the shadow portion of the three-dimensional character is extracted at a relatively initial stage. For this reason, among the obtained extreme value regions, those including a three-dimensional character always include a shadow portion of the three-dimensional character.

  On the other hand, if the initial value of the pixel value v when extracting the connected components is the maximum value (v = 255), the background portion (wall surface) of the character is extracted at a relatively early stage. For this reason, among the obtained extreme value regions, those including a three-dimensional character always include the background portion of the three-dimensional character.

  Thus, since the three-dimensional character is not separated from the shadow portion or the background portion, for example, when character recognition is attempted based on the detected shape of the extreme value region, the character may not be recognized correctly.

  Therefore, the removal unit 15 performs the following three-dimensional side surface removal process. FIG. 13 is a flowchart illustrating an example of the three-dimensional side surface removal process.

  In step S307-1, the removing unit 15 divides a plurality of pixels included in the processing target extreme value region (a plurality of pixels configuring the processing target extreme value region) into a plurality of groups according to the pixel value. To do. For example, the removal unit 15 analyzes a histogram of a plurality of pixels included in the extreme value region to be processed, has a small intra-group deviation, and a large difference in pixel values between groups (for example, an average value of pixels in each group). With such a threshold value, it is divided into a plurality of groups according to the pixel value. Note that the number of groups is not limited to two. For example, it may be divided into three or more.

  Subsequently, in step S307-2, the removal unit 15 removes the group including the shadow portion of the three-dimensional character from the separated groups (components).

  Here, when the extreme value region to be processed is extracted by setting the initial value of the pixel value v when extracting the connected components to the minimum value (v = 0), first, the group having a low pixel value is removed. This is because an extreme value region obtained by removing a group having a high pixel value is already obtained at a relatively early stage when extracting connected components.

  When the extreme value region to be processed is extracted by setting the initial value of the pixel value v when extracting the connected component to the maximum value (v = 255), the group having a high pixel value is removed.

  FIG. 14 is a diagram for explaining processing for removing a shadow portion that is a side surface of a three-dimensional character. FIG. 14 shows an example of a histogram in which the horizontal axis represents the values of a plurality of pixels included in the extreme value region to be processed and the vertical axis represents the number of pixels. The removal unit 15 separates a plurality of pixels included in the processing target extreme value region into a shadow portion 511 and a character shape portion 512, and removes the shadow portion 511 pixel from the processing target extreme value region. To do.

  FIG. 15 is a diagram for explaining processing for removing a shadow portion that is a side surface of a three-dimensional character. FIG. 15A is an example of an original image in a case where a three-dimensional object of the letter “I” is added to a white wall. FIG. 15B is an example of the extreme value region to be processed extracted from the original image. FIG. 15C is an example when a plurality of pixels included in the extreme value region to be processed are divided into a plurality of groups 511 and 512 according to the pixel values. FIG. 15D is an example of the extreme value region after the group including the shadow portion 511 of the three-dimensional character is removed.

  In addition, in the process of step S307-1, when it divides | segments into 3 or more groups, the extreme value area | region which deleted one group with the lowest (or high) pixel value, or the order of the low (or high) pixel value. A plurality of extreme value regions may be extracted, such as an extreme value region in which two groups are deleted.

  Note that the solid side surface removal process may be omitted if it can be determined from the shape of the extreme value area to be processed that the character is clearly not included.

≪Modification of 3D side face removal process≫
Instead of removing the connected components after separating them, the extreme value regions may be extracted from the extreme value region to be processed in the reverse order to the extraction of the extreme value region to be processed. For example, when the extreme value region to be processed is extracted with the initial value of the pixel value v as the minimum value (v = 0), the initial value of the pixel value v is maximized for each pixel included in the extreme value region to be processed. An extreme value region is performed as a value (v = 255).

  In this case, instead of step S308 and step S309, the above-described character-like extreme value region is used for the data of the tree structure of each extreme value region extracted in the reverse order to the extraction of the extreme value region to be processed. A primary extraction process is performed. Then, the extracted region is set as an extreme value region as a child of the extreme value region to be processed, and the process proceeds to step S313.

≪Elongation (long or narrow) judgment process≫
Next, the process of determining whether or not the extreme value region is elongated in step S310 will be described.

  When a plurality of characters are adjacent to each other, an extreme value region where the characters are connected may be included in the tree. Since such an area includes many features that are likely to be characters, it may be determined as likely to be a character with high accuracy. Therefore, when it is desired to detect an extreme value region for each character, when the aspect ratio is not close to 1, that is, when the shape of the region is an elongated extreme value region, the determination of the extreme value region (lower region) of the child is not omitted. To.

  When the camera or the subject is photographed in a non-vertical state, characters such as rotated kanji and kana are included in the image, and the aspect ratio is close to 1 even though it is an extreme region where multiple characters are connected. There is a case. Therefore, by using the rotation aspect ratio feature described above, it is possible to determine that the extreme value region in which a plurality of characters are connected and rotated is elongated.

  For example, when both the aspect ratio and all the rotation aspect ratios fall within the range of 0.8 to 1.2, it may be determined that the length is not elongated.

≪Each extreme value area primary extraction process that seems to be a character string≫
Next, with reference to FIG. 16, the process of determining whether or not a character string is likely in step S <b> 5 and extracting each extreme value region likely to be a character string will be described. FIG. 16 is a flowchart showing an example of each extreme value region primary extraction process that seems to be a character string.

  In step S401, the character string area detection unit 16 deletes the information stored in the work buffer.

Subsequently, in step S402, the character string region detection unit 16 sets one extreme value region ER _i among the plurality of extreme value regions stored in the first list and the second list to be processed. select. In step S402, only the first list may be used. When both the first list and the second list are used in step S402, the accuracy of extracting scene characters is relatively high, but the processing time is relatively long. When only the first list is used in step S402, the processing time is relatively fast, but the accuracy of extracting scene characters is relatively lowered.

  Specifically, when only the first list is used in step S402, the first list does not include an elongated extreme value region, and thus, for example, an elongated shape such as “iii” or “Ini”. A character string composed only of the extreme region of the shape cannot be detected. However, since there are practically no such character strings in Japanese scene characters, it is considered that there is actually no problem. The same applies to English and Chinese. If the character string includes an extreme value area having a shape that is not long and narrow, the processing of steps S404 to S405, which will be described later, causes each of the elongated extreme value areas located around the extreme value area to be Since it can be detected from the list of 2, the accuracy of extracting scene characters does not decrease.

Subsequently, in step S403, the character string region detection unit 16 creates an array ERline _i having the extreme value region to be processed as an element.

Subsequently, in step S404, the character string region detecting unit 16, among the extreme region stored in the first list and the second list, to select another extreme region ER _j to be processed.

Subsequently, in step S405, the character string region detection unit 16 determines whether or not the other extreme value region ER _j seems to be the same character string as any of the elements of the array ERline _i .

Note that the character string region detection unit 16 determines that the two extreme value regions are likely to be the same character string when the distance between the position coordinates is equal to or less than a predetermined threshold value and the positions of the extreme value regions are close, for example. judge.
The character string region detection unit 16 also determines that the two extreme value regions are likely to be the same character string even when the heights of the extreme value regions are close. In this case, as the height of each extreme value region, the height in the oblique direction of each extreme value region calculated by the rotation boundary rectangle descriptor described above may be used. Furthermore, the character string region detection unit 16 determines that the two extreme value regions are likely to be the same character string even when the colors (pixel values) of the extreme value regions are close.

When the other extreme value region ER _j is not likely to be the same character string as any of the elements of the array ERline _i (S405: NO), the process proceeds to step S407 described later.

When the other extreme value region ER _j seems to be the same character string as one of the elements of the array ERline _i (S405: YES), in step S406, the character string region detection unit 16 arranges the other extreme value region ER _j into the array. Store as ERline _i element.

Subsequently, in step S407, the character string region detection unit 16 determines whether there is another unselected extreme value region ER _j among the extreme value regions stored in the first list and the second list. Determine whether.

If there is another extreme value region ER _{j that} has not been selected (S407: YES), in step S408, the character string region detection unit 16 selects another extreme value region ER _j that has not been selected as a processing target, and step S405 is performed. Proceed to the process.

If there is no other extreme value region ER _j that has not been selected (S407: NO), in step S409, the character string region detection unit 16 determines whether the number of elements of the array ERline _i is a predetermined number (eg, 3) or more. judge.

If the number of elements of the array ERline _i is not greater than or equal to the predetermined number (S409: NO), in step S410, the character string region detection unit 16 discards the array ERline _i and proceeds to the process of step S412 described later.

If the number of elements of the array ERline _i is equal to or greater than the predetermined number (S409: YES), in step S411, the character string region detection unit 16 stores the array ERline _i in the array list.

Subsequently, in step S412, the character string region detection unit 16 determines whether or not there is one unselected extreme value region ER _i among the extreme value regions stored in the first list and the second list. Determine whether.

If there is one unselected extreme value region ER _i (S412: YES), in step S413, the character string region detection unit 16 selects one unselected extreme value region ER _i as a processing target, and step S403. Proceed to the process.

If there is no unselected extreme value region ER _i (S412: NO), the character string region detection unit 16 ends the process.

≪Secondary extraction process for extremum regions that seems to be character strings (projection distortion support) ≫
Next, with reference to FIG. 17, a description will be given of a process of extracting each extreme value region that seems to be a character string by determining whether or not the character string seems to be relatively detailed in step S <b> 6. FIG. 17 is a flowchart showing an example of each extreme value region secondary extraction process that seems to be a character string.

  For example, when a scene character is detected from an image included in a moving picture of recorded news material, a signboard displaying a court name or the like may be taken from an oblique angle. In this case, a projection distortion occurs in which characters on the near side as viewed from the camera appear large and characters on the far side appear small. Therefore, if a character string is detected on the premise that the size of each character included in one character string is approximately the same, an extreme value area of a character that appears small on the back side or an extreme value area that is a part of the character There is a problem that sometimes cannot be detected. Therefore, in the present embodiment, in the character string in which such a projection distortion is caused by the following processing, the extremum constituting the character string is based on the position and size of each character included in the character string. Re-search the area.

In step S501, the estimation unit 17 selects one array ERline _i from among the arrays stored in the above-described array list as a processing target.

Subsequently, in step S502, the estimation unit 17 selects a plurality of extreme value regions serving as a reference for the character string from among the extreme value regions that are elements of the array ERline _{i to be} processed. Details of this process will be described later.

  Subsequently, in step S503, the estimation unit 17 calculates two straight lines that are close to a plurality of reference extreme value regions.

  Subsequently, in step S504, the estimation unit 17 determines that the extreme value region that is not an element of each array stored in the array list among the extreme value regions stored in the first list and the second list described above. Is selected as a processing target.

  Subsequently, in step S505, the estimation unit 17 determines whether or not the processing target extreme value region is located between the two straight lines calculated in step S503.

  If the extreme value region to be processed is not located between the two straight lines calculated in step S503 (S505: NO), the process proceeds to step S508 described later.

If the processing target extreme value region is located between the two straight lines calculated in step S503 (S505: YES), in step S506, the detection unit 18 determines that the processing target extreme value region is the processing target array ERline. _It is determined whether or not the extreme value region of any element of _i is close to the color or position.

If the extreme region to be processed is not close to the color, position, or the like of any of the elements in the array ERline _{i to be} processed (S506: NO), the process proceeds to step S508 described later.

If the extreme region to be processed is close in color, position, etc. to any one of the elements of the array ERline _{i to be} processed (S506: YES), in step S507, the detection unit 18 detects the extreme region to be processed. The value area is stored as an element of the processing target array ERline _i .

In step S405 described above, for example, it is determined whether or not the heights of the respective extreme value regions are close. However, in step S506, it is not determined whether the height is close if it is located between two straight lines. Thereby, an extreme value region that is extremely small compared to other extreme value regions is also stored as an element of the array ERline _i to be processed.

  Subsequently, in step S508, the estimation unit 17 includes each extreme value region stored in the first list and the second list, and any of the arrays stored in the array list. It is determined whether or not there is an unselected extreme value region as a processing target among extreme value regions that are not elements.

  If there is an unselected extreme value region as a processing target (S508: YES), in step S509, the estimating unit 17 selects the unselected extreme value region as a processing target, and the process proceeds to step S503.

If there is no unselected extreme value region as a processing target (S508: NO), in step S510, the estimation unit 17 has an unselected array ERline _i as a processing target among the arrays stored in the array list. It is determined whether or not.

If there is an unselected array ERline _i as a processing target (S510: YES), the unselected array ERline _i is selected as a processing target in step S511, and the process proceeds to step S502.

If there is no unselected array ERline _i to be processed (S510: NO), the estimating unit 17 ends the process.

  FIG. 18 is a diagram for explaining each extreme value region secondary extraction process that seems to be a character string. FIG. 18A is a diagram illustrating an example of an original image subjected to projective distortion in which a scene character is shown with a large character on the near side and a small character on the back side.

  FIG. 18B is a diagram illustrating an example of the extreme value region extracted from the original image by the extreme value secondary extraction process that seems to be the character described above.

  FIG. 18C is a diagram showing an example of the extreme value region extracted by each extreme value region primary extraction process that seems to be the character string described above. Compared to FIG. 18B, in FIG. 18C, a part of characters is not extracted.

  FIG. 18D is a diagram illustrating an example of calculating the height (size) of each extreme value region included in the character string. For example, the radius or diameter of each extreme value region included in the character string is the height (size) of each extreme value region included in the character string.

  FIG. 18E is a diagram illustrating an example when a plurality of extreme value regions serving as a reference for a character string are selected in step S502. In FIG. 18E, an extreme value region circumscribed by the circles 521, 522, and 523 is selected as a character string reference.

  FIG. 18F is a diagram illustrating an example of estimating the position and size of each character included in the character string based on the character string reference. In FIG. 18F, a range between two straight lines 524 and 525 adjacent to circles 521, 522, and 523 is estimated as the position and size of each character included in the character string. Note that the estimation unit 17 may estimate the position and size of each character constituting the character string based on changes in the position and size of a plurality of extreme value regions serving as a reference for the character string. Then, if there is an extreme value region in which a feature such as a color and the extreme value region of each character is close in the range based on the estimated position and size of each character, the detection unit 18 has an extreme value region in which the feature is close. Are detected as extreme regions constituting each character.

  The estimation unit 17 may estimate the position and size of the character at a location where the character string detected by the character string region detection unit 16 is extended by one character left and right. And when the extremum area which seems to be a new character is detected in the extended part by the detection part 18, the part which estimates the position and size of a character is repeatedly extended to the part where a new character is no longer detected. Also good.

  FIG. 18G is a diagram illustrating an example in which the extreme value region of each character included in the character string is extracted from the estimated position and size.

≪Process to select multiple extremum areas that serve as the reference for character string≫
Next, a process of selecting a plurality of extreme value regions serving as a character string reference from among the extreme value regions that are elements of the processing target array ERline _i in step S502 will be described.

The estimation unit 17 uses, as a reference, an extreme value region whose height (size) is higher (larger) than the average of the extreme value regions among the extreme value regions that are elements of the array ERline _{i to be} processed. Select as value area. As a result, even if the element of the array ERline _{i to be} processed includes an extreme value region that constitutes a small part of the character, the character string reference can be correctly selected. When there are not a plurality of extreme value regions whose height is higher than the average of the extreme value regions, for example, the first and second extreme value regions may be selected as reference extreme value regions. .

  In addition, the character string is not always subject to projective distortion. Therefore, the following processing may be performed.

The estimation unit 17 uses the extreme value region having the highest (large) height (size) as a reference among the extreme value regions that are elements of the array ERline _{i to be} processed.

The estimation unit 17 is each extreme value region stored in the first list and the second list described above, and is not an element of any array among the arrays stored in the array list. Among the extreme value areas, the extreme value area whose height (size) is close to that of the reference is stored as an element of the array _ERline _{i to be} processed.

Then, the detection unit 18 calculates the character degree of the array ERline _i and the array _ERline _i extracted in each extreme value region secondary extraction process that seems to be the character string described above, and deletes the array having the smaller character degree. .

[Second Embodiment]
In the first embodiment, in the process of step S5 in FIG. 4, an example of the process of determining whether or not a character string is likely and extracting each extreme value region that seems to be a character string has been described.

  In the second embodiment, another example of the process of determining whether or not a character string appears in the process of step S5 in FIG. 4 and extracting each extreme value region likely to be a character string will be described.

  Note that the second embodiment is the same as the first embodiment except for a part thereof, and thus description thereof will be omitted as appropriate.

≪Each extreme value area primary extraction process that seems to be a character string≫
Next, with reference to FIG. 19, an example of processing for determining whether or not a character string is likely to be performed in step S <b> 5 of FIG. 4 and extracting each extreme value region likely to be a character string will be described. FIG. 19 is a flowchart showing an example of each extreme value region primary extraction process that seems to be a character string according to the second embodiment.

  In step S1401, the character string region detection unit 16 sorts each extreme region that seems to be a character extracted in the processing up to step S4 in the preceding stage in descending order of the character likelihood. Here, the character string area detection unit 16 sorts, for example, the extreme value areas included in the first list and the second list in descending order of the character likelihood determined by the character degree determination unit 14. To do. Note that, similarly to the description of step S402 of the first embodiment, only the first list may be used in step S1401. When both the first list and the second list are used in step S1401, the accuracy of extracting scene characters is relatively high, but the processing time is relatively long. When only the first list is used in step S1401, the processing time becomes relatively fast, but the accuracy of extracting scene characters is relatively lowered. Note that if the character string includes an extreme value area having a shape that is not elongated even for one character, the process of step S2001 to be described later extracts each of the elongated extreme value areas located around the extreme value area from the second list. Since it can be detected, the accuracy of extracting scene characters is not lowered.

  Subsequently, in step S1402, the character string region detection unit 16 selects an extreme value region having the highest character-likeness from the sorted list as a processing target. Hereinafter, the extreme value region selected as the processing target is referred to as “extreme value region to be processed”.

  Subsequently, in step S1403, the character string region detection unit 16 determines whether or not the degree of character character of the extreme value region to be processed is lower than a predetermined threshold value.

  If the degree of character-likeness of the extreme value area to be processed is lower than a predetermined threshold (step S1403: YES), the process ends.

  If the degree of character-likeness of the extreme region to be processed is not lower than the predetermined threshold (step S1403: NO), in step S1404, the character string region detection unit 16 includes a region including the extreme region to be processed ( One extreme value area or a combination of multiple extreme value areas) is searched for the same character string at each of a plurality of angles. A process for searching for the same character string at each of a plurality of angles will be described later.

  Subsequently, in step S1405, the character string region detection unit 16 has the highest “degree of likelihood of the same character string” between the characters constituting the character string among the character strings searched at each of a plurality of angles. Extract things. Here, the character string region detection unit 16 stores the extracted character strings in the above-described array list. In addition, you may calculate "the degree of likeness of the same character string" between each character based on dispersion | distribution of each characteristics, such as the position of each said character, height, horizontal width, stroke width, a color, for example. In this case, for example, the character string region detection unit 16 determines the character string having the smallest sum of the values obtained by multiplying the variance value of each feature by the weighting coefficient corresponding to each feature amount, as “the degree of likelihood of being the same character string”. "May be determined to be the highest character string.

  Subsequently, in step S1406, the character string region detection unit 16 determines whether there is an extreme value region having the next highest character-likeness degree in the sorted list.

  If there is an extreme value region with the next highest character likelihood (step S1406: YES), in step S1407, the character string region detection unit 16 selects an extreme value region with the next highest character likelihood as a processing target. The process proceeds to step S1403 described above.

  If there is no extreme value region with the next highest character-like degree (step S1406: NO), the process is terminated.

  In the present embodiment, a character in which one character is separated into a plurality of extreme value regions in kanji (including Chinese kanji), kana (including hiragana and katakana), hangul, and the like is also extracted. In this case, a region (composite region) in which a plurality of extreme value regions are combined is also a target for calculating the degree of character string identity. In this case, if the same character string degree is calculated for all combinations of a plurality of extreme value regions, the amount of calculation becomes enormous. For this reason, the character string region detection unit 16 performs the processing in steps S1404 and S1405 only for the extreme value region in which the degree of characterness is equal to or greater than a predetermined threshold value.

  Next, the process for searching for the same character string at each of a plurality of angles in step S1404 will be described with reference to FIG. FIG. 20 is a diagram illustrating processing for searching for the same character string at each of a plurality of angles. FIG. 20 shows an example in which an extreme value region that is a candidate for an adjacent character is searched for along an angle of 15 degrees in an image that includes three Chinese characters indicating a parking lot in Chinese. In this case, the angle in the direction of searching for a candidate for an adjacent character may be aligned with the rotation angle when the extreme boundary region descriptor 12 is calculated by the extreme value region extraction unit 12 in steps S103 and S107. In other words, when the extreme boundary region descriptor 12 calculates the rotation boundary rectangle descriptor such as the rotation aspect ratio in increments of 15 degrees, the character string region detection unit 16 is adjacent along the angle in increments of 15 degrees. Search for extreme regions that are candidates for characters. Thus, using the rotation boundary rectangle calculated in the process of calculating the rotation boundary rectangle descriptor, a “score indicating the degree of the same character string” (hereinafter also simply referred to as “score”) is calculated. Therefore, the processing amount for calculation can be reduced.

  Here, the “score” is, for example, a value obtained by multiplying the difference between the feature amounts such as the position, height, width, stroke width, color, and the like of the two extreme value regions to be compared by a weighting factor corresponding to each feature. Calculated by the total value. In this case, each feature such as position, height, width, stroke width, etc. indicates that the smaller the difference in feature amount, the more likely it is to be the same character string. Therefore, the smaller the “score” value, the more the same character string. ("Similarity of the same character string" is high). The weighting factor described above may be determined using a machine learning method or the like.

  When calculating the “score”, the character string region detection unit 16 calculates the rotation boundary calculated in the process of calculating the rotation boundary rectangle descriptor using the difference between the position, height, and width of the two extreme value regions to be compared. Calculate using a rectangle.

  The difference between the positions of the two extreme value regions may be a distance between the rotation boundary rectangles for each of the two extreme value regions to be compared. In the example of FIG. 20, the distance between the right end of the rotation boundary rectangle 601a for the leftmost character 601 and the left end of the rotation boundary rectangle 602a for the central character 602 is the difference between the positions of the two extreme value regions. The

  The difference in height between the two extreme value regions and the difference in horizontal width may be a difference in height between the rotation boundary rectangles and a difference in horizontal width for each of the two extreme value regions to be compared. In the example of FIG. 20, the difference between the height and width of the rotation boundary rectangle 601a for the leftmost character 601 and the height and width of the rotation boundary rectangle 602a for the center character 602 are two poles, respectively. The difference between the height of the value area and the difference between the widths are set. As a result, as compared with the conventional method using the non-rotating boundary rectangles (601b, 602b), the vertical alignment (alignment) is aligned, so that the detection accuracy of characters included in the same character string is detected. Will improve.

  Note that the stroke width and color are characteristics unrelated to rotation. The stroke width is the line width of the character. The color is the color of the extreme value region. For the color of the extreme value region, an HSI color space composed of hue, saturation, and luminance may be used, an RGB color space may be used, or luminance may be used. Good.

  Next, a process for searching for the same character string in step S1404 will be described with reference to FIG. FIG. 21 is a flowchart illustrating an example of processing for searching for the same character string.

  In step S2001, the character string region detection unit 16 sets the extreme value region group that is a candidate for a character adjacent to the right side in a predetermined angular direction in the processing target region as the first list and the second list described above. Search from. Here, the “region to be processed” is a region including the extreme value region to be processed in step S1404, and is a region in which one extreme value region or a plurality of extreme value regions are combined. Even in the case where only the first list is used in step S1401, in step S2001, the long and narrow extreme value regions located around the extreme value region included in the first list are also searched. Both the first list and the second list are used.

  Here, the character string area detection unit 16 searches for a plurality of extreme value areas whose characteristics such as color, position, and stroke width are close to the area to be processed. More specifically, when the difference between the feature of the region to be processed and the feature of the other extreme value region is less than a predetermined threshold, the character string region detection unit 16 determines the other extreme value region, It is included in the extreme value region group that is a candidate for a character adjacent to the region to be processed. Thereby, since the number of extreme value regions included in the candidate extreme value region group can be narrowed down, the amount of subsequent calculation processing can be reduced. The predetermined threshold value may be determined using a machine learning method or the like.

  Subsequently, in step S2002, the character string region detection unit 16 determines “the degree of likelihood of the same character string” between the region to be processed and each extreme value region included in the candidate extreme value region group described above. The score shown is calculated.

Subsequently, in step S2003, the character string region detection unit 16 has an extreme value region having the smallest “score” value for the region to be processed among the extreme value regions included in the candidate extreme value region group described above. _Are selected as “adjacent character candidates (C _next )”.

Subsequently, in step S2004, the character string region detection unit 16 selects one extreme value region that is not included in the “adjacent character candidate (C _next )” from the candidate extreme value region group described above. .

Subsequently, in step S2005, the character string region detection unit 16 sets a region obtained by combining the selected extreme value region and the “adjacent character candidate (C _next )” as a “C _next candidate (C _candinate )”. for _{"C next} candidate _{(C candidate)",} to calculate the value of the "score" for the region to be processed.

  Subsequently, in step S2006, the character string region detection unit 16 determines whether or not there is an unselected extreme value region in the candidate extreme value region group described above.

  When there is an unselected extreme value region (step S2006: YES), in step S2007, the character string region detection unit 16 selects an unselected extreme value region.

Subsequently, in step S2008, the character string region detection unit 16 sets the “score” value for the region to be processed for a region obtained by combining the selected extreme value region and “adjacent character candidate (C _next )”. Is calculated.

Subsequently, in step S2009, the character string region detection unit 16 determines that the “score” value of the combined region calculated in step S2008 is larger than the “score” value of “C _{next candidate} ” (C _candidate ). It is determined whether or not it is small.

  If not smaller (step S2009: NO), the process proceeds to step S2006.

If it is smaller (step S2009: YES), in step S2010, the character string region detection unit 16 sets the combined region as a new “C _next candidate (C _candidate )”, and proceeds to the processing of step S2006.

In step S2006, when there is no unselected extreme value region (step S2006: NO), in step S2011, the character string region detection unit 16 determines that the value of “score” of “C _next candidate (C _candidate )” is It is determined whether or not the value is smaller than the “score” value of “adjacent character candidate (C _next )”.

If it is smaller (step S2011: YES), in step S2012, the character string region detection unit 16 sets “C _next candidate (C _candidate )” as a new “adjacent character candidate (C _next )”, and step S2004. Proceed to the process.

  If not small (step S2011: NO), in step S2013, the character string region detection unit 16

It is determined whether or not the “score” value of “adjacent character candidate (C _next )” is smaller than a predetermined threshold. The predetermined threshold value may be determined using a machine learning method or the like.

If it is smaller than the predetermined threshold (step S2013: YES), in step S2014, the character string region detection unit 16 sets the processing target region and “adjacent character candidate (C _next )” to the same character string. It is determined that

Subsequently, in step S2015, the character string region detection unit 16 sets “adjacent character candidate (C _next )” as a new “region to be processed”, and proceeds to the processing in step S2001 described above.

  If it is not smaller than the predetermined threshold (step S2013: NO), the process is terminated.

  Next, processing for searching for the same character string in FIG. 21 will be described with reference to FIG. FIG. 22 is a diagram illustrating processing for searching for the same character string.

In FIG. 22, the example of the image in which the signboard of the Chinese parking lot is reflected is shown. FIG. 22A shows an example in which the leftmost character 601 and the middle character 602 among the three Chinese characters are determined to be the same character string in step S2014. In step S2015, a region obtained by combining one or more extreme value regions constituting the character 601 and one or more extreme value regions constituting the character 602 that is the “adjacent character candidate (C _next )” is a new “ The process proceeds to step S2001 described above.

In step S2003, a “score” for “region to be processed” of each extreme value region is calculated. Here, the rotation boundary rectangle 601a for the character 601 and the rotation boundary rectangle 602a for the character 602 (an example of “first rectangle”) and a part 604 of the rightmost character 603 shown in FIG. A “score” is calculated based on the rotation boundary rectangle 604a with respect to. Similarly, a “score” is calculated for the rotation boundary rectangles 605 a to 608 a for the regions 605 to 608. Further, it is assumed that a partial area 605 of the rightmost character 603 having the smallest “score” value is set as “adjacent character candidate (C _next )”.

Then, in steps S2005 to S2010, an area obtained by combining another extreme area with the “adjacent character candidate (C _next )” that is one extreme area having the smallest “score” value. A “score” for the “region to be processed” is calculated. Of the combined regions, the region having the smallest “score” value is a “C _{next candidate} ”.

  Here, as shown in FIG. 22C, the rotation boundary rectangle 601a for the character 601 and the rotation boundary rectangle 602a for the character 602, and the rotation boundary rectangle 603a (“second rectangle” for the region 603 in which the region 604 is combined with the region 604 are shown. "Score" is calculated based on the above. Similarly, the “score” is also calculated for the rotation boundary rectangle for the area obtained by combining the area 605 and the areas 606 to 608.

Then, among the “score” values of the region combining the two extreme regions, the “score” value of the region combining the region 604 is added to the “adjacent character candidate (C _next )” that is the region 605. Assuming that the area is the smallest, the area obtained by combining the area 605 and the area 604 is a “C _{next candidate} ”.

Then, in step S2011, if the “score” value of the region combining the region 605 and the region 604 is smaller than the “score” value of the “adjacent character candidate (C _next )” that is the region 605, In step S2012, a region obtained by combining the region 605 and the region 604 is set as a new “adjacent character candidate (C _next )”.

  Then, in steps S2005 to S2010 again, the “score” for the “region to be processed” of the region obtained by combining the region 605 and the region 604 with another extreme region is calculated.

Then, in step S2011 again, the “score” of the region obtained by combining the region 605 and the region 604 with another extreme value region is the region where the region 605 and the region 604 are combined. It is assumed that it is not smaller than the value of “score” of “C _next )”. In this case, if the “score” value of “adjacent character candidate (C _next )” is smaller than a predetermined threshold value in step S2013, in step S2015, extreme values constituting the character 601, character 602, and character 603 are formed. A region obtained by combining the region groups is set as a new “region to be processed”, and the process proceeds to step S2001 described above. Then, based on the rotation boundary rectangles 601a to 603a for the characters 601 to 603 shown in FIG. 22C, an extreme value region that is a candidate for a character adjacent in a predetermined angular direction is searched, and If the “score” value of “adjacent character candidate (C _next )” is smaller than a predetermined threshold value in step S2013, the process ends.

<About the program>
The image processing apparatus 1 according to the present embodiment includes, for example, a volatile storage medium such as a CPU (Central Processing Unit) and a RAM (Random Access Memory), a nonvolatile storage medium such as a ROM (Read Only Memory), a mouse, You may comprise by the computer provided with input devices, such as a keyboard and a pointing device, the display part which displays an image and data, and the interface for communicating with the exterior.

  In this case, each function of the image processing apparatus 1 can be realized by causing the CPU to execute a program (image processing program) describing these functions. The program can also be stored and distributed on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like. That is, the above-described processing can be realized by installing a program for causing a computer (hardware) to execute the processing in each configuration described above, for example, on a general-purpose PC or server.

  Moreover, you may implement | achieve part or all of the image processing apparatus 1 in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each functional block of the image processing apparatus 1 may be individually made into a processor, or a part or all of them may be integrated into a processor.

  For example, the image processing apparatus 1 recognizes characters in an image in the moving image from a moving image database in which the moving image of the coverage content or the moving image of the past broadcast program is stored, and a tag for searching for the moving image. As described above, the present invention can be applied to an archive system that manages a recognized character string in association with the moving image. The image processing apparatus 1 can also be applied to, for example, a smartphone that recognizes characters from an image captured by a camera.

<Summary>
In the technique described in Non-Patent Document 1, there is a problem in the accuracy of extraction when scene characters are extracted for a language in which one character is composed of a plurality of separated regions such as kanji, kana, and hangul. In the alphabet and numbers targeted by the technology described in Non-Patent Document 1, one stroke can be written except for “i” and “j” (the area constituting one character is not separated). When scene characters such as “river” and “i” are extracted, one character is separated into a plurality of extreme value regions. There are about 100 alphabets and numbers. For example, Japanese has about 6,000 characters. Furthermore, when considering the case where the scene character is rotated and the like, there is a problem that the processing cost when extracting the scene character region becomes enormous.

  On the other hand, according to the present embodiment, it is determined whether or not the extreme value region is elongated using the rotation aspect ratio feature. And if there is an extremum area that is character-like and not elongated on the tree structure of the extracted extremum area, the extremum area belonging to the lower level of the extremum area is an area that represents a part of the character. It is not a target for character character judgment. As a result, feature calculation processing with a high calculation cost can be reduced, and character area detection processing can be executed at high speed even for scene characters such as rotated kanji, kana, and hangul.

  In the present embodiment, if there is a character-like and elongated extremal area on the tree structure of the extracted extremal area, the extremal area belonging to the lower level of the extremal area is a target for character-likeness determination. And Thereby, it is possible to prevent an extreme value region in which a plurality of characters are connected from being erroneously detected.

  In the prior art, a character string is found by utilizing the fact that the upper line and the lower line of the character string are aligned. For alphabets and numbers, etc., except for “i” and “j”, each character is one extreme value region (connected component), so the upper and lower lines are aligned to a certain extent. If it sees, even if a character string is reflected diagonally, it can detect. However, for example, when a character string such as Japanese “crab” is shown obliquely, only one of the two sides included in “ni” may be detected as a character adjacent to “ka”. is there. In addition, when the character string is two lines, the extreme value area that constitutes a character that is included in the lower line and is composed of a plurality of extreme value areas such as “O” is the upper line. It may be detected as a character adjacent to the character.

  On the other hand, according to the present embodiment, a region of characters included in the same character string is searched for by calculating the degree of likelihood of the same character string using a rotation boundary rectangle along a plurality of angular directions. To do. Thereby, it is possible to improve the accuracy of detecting a character string even for scene characters such as kanji, kana, and hangul.

  The above-described embodiment is not limited to kanji, kana, hangul, etc., but is particularly suitable for extracting scene characters in a language in which one character is constituted by a plurality of separated areas. It is also suitable for extracting scene characters in which the area constituting one character is not separated.

  As described above, the embodiments have been described in detail with reference to the drawings. However, the specific configuration is not limited to that described above, and various design changes and the like can be made without departing from the scope of the invention. Is possible.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 11 Image acquisition part 12 Extreme value area extraction part 13 Calculation part 14 Character degree determination part 15 Removal part 16 Character string area | region detection part 17 Estimation part 18 Detection part 19 Storage part 20 Output part

Claims (10)

A first region included in the image, a first region in which a plurality of pixels having pixel values equal to or less than the first value are continuous, and a second region included in the image, wherein the pixel value is Based on a second region in which a plurality of pixels that are equal to or smaller than a second value greater than the first value are continuous, the extreme region in which the pixels in the first region and the second region are continuous is defined as the image. An extreme value region extraction unit that extracts from
A determination unit that determines the degree of character-likeness of the shape of the region included in the image,
The determination unit determines the degree of character of the extreme value region according to a long side and a short side of a rectangle circumscribed by the extreme value region and rotated by a predetermined angle on the plane of the image. An image processing apparatus. The image processing apparatus according to claim 1.
The determination unit circumscribes the extreme value region, and when the aspect ratio between the long side and the short side of the rectangle rotated by a predetermined angle on the plane of the image is not within a predetermined range, the determination unit Judge the degree of character of the shape,
When the aspect ratio is within the predetermined range, the character processing degree of the shape of the first region is not determined. The image processing apparatus according to claim 2.
A rectangle circumscribing the first region and rotated by the predetermined angle on the plane of the image, and a rectangle circumscribing the second region and rotated by the predetermined angle on the plane of the image An image processing apparatus comprising: a calculation unit that calculates the aspect ratio of a rectangle circumscribing the extreme value region and rotated by a predetermined angle on a plane of the image. In the image processing device according to any one of claims 1 to 3,
Of the extreme value region, comprising a removal unit for removing the region of the side of the three-dimensional character,
The said determination part determines the degree of character likeness of the shape of the extreme value area | region from which the area | region of the side surface of the solid character was removed, The image processing apparatus characterized by the above-mentioned. The image processing apparatus according to claim 4.
The removing unit divides each pixel included in the extreme value region into a plurality of groups based on the value of each pixel included in the extreme value region, and removes a group including a shadow portion of a three-dimensional character.
An image processing apparatus. In the image processing device according to any one of claims 1 to 5,
A first rectangle circumscribing the first extremum region and rotated by the predetermined angle on the plane of the image, and a second rectangle circumscribing the second extremum region and the plane of the image by the predetermined angle An image processing apparatus comprising: a character string region detection unit that detects the second extreme value region, which is the same character string as the first extreme value region, based on the rotated second rectangle. The image processing apparatus according to claim 6.
The character string area detection unit
Based on at least one of a distance difference, a height difference, and a width difference between the first rectangle and the second rectangle, the first extreme value region, and the second rectangle An image processing apparatus that determines whether or not an extreme value region is the same character string. In the image processing device according to any one of claims 1 to 7,
An estimation unit for estimating the position and size of each character included in the character string based on the position and size of each extreme value region included in the character string in the image;
From a range according to the position and size of each character estimated by the estimation unit, a detection unit for detecting an extreme value region included in each character included in the character string;
Providing
An image processing apparatus. The image processing apparatus according to claim 8.
The estimation unit is configured to determine a position of each character included in the character string based on positions and sizes of a plurality of extreme value regions larger than an average of the extreme value regions among the extreme value regions included in the character string. And estimating the size,
An image processing apparatus. Computer
An image processing program that functions as the image processing apparatus according to claim 1.