JP2013041315A - Image recognition device and image recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and a method for accurately extracting an area corresponding to an object forming a specific geometric figure from an image.SOLUTION: An image recognition device comprises: an edge extraction unit for extracting edge segments from an image; an acquisition unit for acquiring combinations of candidates for predetermined geometric figures to be formed by using the edge segments extracted by the edge extraction unit; a calculation unit for calculating a recall ratio indicating a level at which the extracted edge segments cover outer circumference of figure candidates and a relevance ratio indicating a level at which the extracted edge segments are to be used as the figure candidates, for each combination acquired by the acquisition unit; and an image extraction unit for extracting an area corresponding to figure candidates included in a combination with the highest evaluation value determined on the basis of the recall ratio and relevance ratio.

Description

  The present invention relates to an image recognition apparatus and an image recognition method for recognizing an area corresponding to a predetermined geometric figure from an image.

  There is a need to recognize and extract a region corresponding to an object constituting a specific geometric figure from an image. For example, there is an application in which a white board with a rectangular note sticker is photographed with a digital camera and an area corresponding to the note sticker is extracted from the obtained image. In this case, for example, by performing character recognition on the extracted image, it is possible to save the characters described on the note sticker as electronic data. For such applications, there is known a method of extracting an area corresponding to a target object by extracting an edge from an image and recognizing an area surrounded by the edge.

  As a related technique, the following graphic cutout method has been proposed. This method reads an image including a rectangular two-dimensional code figure in which data is arranged in a matrix having a configuration in which at least two sides are straight lines or a figure similar thereto, and reads the two-dimensional code figure or the like from the read image. In the image recognition apparatus for cutting out and recognizing a figure similar to it, the position of two straight lines intersecting each other by the Hough transform method and the least square approximation method for the image of the two-dimensional code figure or a figure similar thereto , A step of detecting the lengths of the two straight lines detected in this step, and the remaining two intersecting with each other based on the positions and lengths of the two straight lines detected in the respective steps. A step of detecting the position of the straight line of the book is provided, and the two-dimensional code figure or a figure similar thereto is cut out by each step. (For example, Patent Document 1)

  As another related technique, the following image matching method has been proposed. In this method, an image is processed to obtain a candidate area of a building, and the vertical and horizontal dimensions of the area are prototyped by a binary image representation having a value of 1 for pixels including the building area and 0 for pixels not including the building area. Size test a set of building dimensions and if a dimension is too large or too small, determine that it is not a building. An XY pixel list of contour lines of each building candidate is obtained based on the binary image representation, and the main axes of the contour lines of each region are aligned with the pixel grid of the binary image, and the horizontal and vertical edge portions of the contour list are obtained. The direction histogram is calculated, and if the peak concentration rate of the histogram is smaller than the current step, the area is determined not to be a building. A peak in the histogram of the edge portion is assumed as a corner candidate in the coordinate list of the region pixels, and a combination in which the most corner candidates are valid is selected as the entire circumference of the building. (For example, Patent Document 2)

JP-A-7-220081 Japanese Patent Laid-Open No. 5-101183

  In the prior art, in the process of extracting a region corresponding to an object constituting a specific geometric figure from an image, when a plurality of objects overlap each other, each edge extracted from the image constitutes which object. It is difficult to determine whether it is an edge to be performed. In this case, the accuracy of extracting a region corresponding to each object decreases. Further, when the color of the object is similar to the background color, one edge may be extracted in a plurality of portions. Also in this case, it is difficult to correctly extract a region corresponding to each object.

  An object of the present invention is to provide an apparatus and a method for accurately extracting a region corresponding to an object constituting a specific geometric figure from an image.

  An image recognition apparatus according to one aspect of the present invention includes an edge extraction unit that extracts an edge segment from an image, and a predetermined geometric figure formed by using the edge segment extracted by the edge extraction unit. An acquisition unit for acquiring a combination of candidates, a reproducibility representing the extent to which an outer periphery of the candidate for the figure is covered by the extracted edge segment for each combination acquired by the acquisition unit, and the extracted A calculation unit that calculates a precision that represents a degree to which the edge segment is used as a candidate for the graphic, and a graphic candidate included in a combination that has a maximum evaluation value determined based on the recall and the precision An image extraction unit for extracting a corresponding region;

  According to the above aspect, it is possible to accurately extract a region corresponding to an object constituting a specific geometric figure from an image.

It is a block diagram which shows the function of the image recognition apparatus of embodiment. It is a flowchart which shows the image recognition method of embodiment. It is a figure which shows a Sobel filter. It is a figure which shows the example of the binarization edge image produced | generated from an input image. It is a figure explaining the direction decomposition of an image. It is a figure explaining a direction decomposition process. It is a figure which shows the example of the binarization edge image produced | generated by direction decomposition | disassembly. It is a figure explaining a labeling and a circumscribed rectangle. It is a figure explaining the overlap integration of a black pixel connection component. It is a figure which shows the example of the edge segment extracted by the edge extraction part. It is a flowchart which shows the process which extracts a rectangular area candidate. FIG. 6 is a diagram (part 1) illustrating a condition for an edge segment to form a rectangular region. It is FIG. (2) explaining the conditions for which an edge segment comprises a rectangular area. FIG. 11 is a diagram (part 3) for explaining the conditions under which an edge segment forms a rectangular area. It is FIG. (4) explaining the conditions for which an edge segment comprises a rectangular area. It is a figure explaining the graph and clique for acquiring a rectangular area candidate. It is a flowchart which shows the process which acquires the combination of a rectangle area | region candidate. It is a figure explaining the graph and clique for acquiring the combination of a rectangle area candidate. It is a figure which shows the extracted edge segment. (A) is a graph about an edge segment, (b) is a figure which shows the extracted clique. It is a figure which shows a rectangular area candidate. (A) is a graph about a rectangular area | region candidate, (b) is a figure which shows the extracted clique. It is a figure which shows the combination of a rectangular area | region candidate. It is a figure explaining direction decomposition for extracting an equilateral triangle field. It is FIG. (1) explaining the conditions for which an edge segment comprises an equilateral triangle area | region. It is FIG. (2) explaining the conditions for which an edge segment comprises an equilateral triangle area | region. FIG. 6 is a diagram (part 3) for explaining conditions for an edge segment to form an equilateral triangle region; It is a figure which shows the hardware constitutions of the computer system for implement | achieving an image recognition apparatus.

  FIG. 1 is a block diagram illustrating functions of the image recognition apparatus according to the embodiment. The image recognition apparatus 1 according to the embodiment includes an image data storage unit 2, a processing unit 3, an extraction result storage unit 8, and an output unit 9.

  The image data storage unit 2 stores image data obtained by a digital camera or a scanner. Here, the image recognition apparatus 1 may include an interface for receiving image data from a digital camera, a scanner, or the like. Alternatively, the image recognition device 1 may be built in a digital camera or the like. The image data is color image data in this embodiment. In the following description, the image data may be simply referred to as “image”.

  The processing unit 3 extracts a predetermined geometric figure (in this embodiment, a rectangle) from the image stored in the image data storage unit 2. The processing unit 3 includes an edge extraction unit 4, an acquisition unit 5, a calculation unit 6, and an image extraction unit 7 in order to extract a geometric figure from an image. The operations of the edge extraction unit 4, the acquisition unit 5, the calculation unit 6, and the image extraction unit 7 will be described later.

  The extraction result storage unit 8 stores image data of an area corresponding to the graphic extracted by the processing unit 3. Then, the output unit 9 outputs the image data of the area extracted by the processing unit 3 stored in the extraction result storage unit 8. The output unit 9 outputs image data to a display device, for example. Alternatively, the output unit 9 may output image data to an external storage device.

  FIG. 2 is a flowchart illustrating an image recognition method according to the embodiment. The processing of this flowchart is executed by the processing unit 3 when an extraction instruction is given to the image recognition apparatus 1, for example. In this embodiment, the extraction instruction is an instruction to extract a rectangular area from the image. Further, the extraction instruction is input to the image recognition apparatus 1 by the user, for example. When an extraction instruction is given, the processing unit 3 acquires a color image from the image data storage unit 2.

  In step S <b> 1, the edge extraction unit 4 grays the color image acquired from the image data storage unit 2. In step S2, the edge extraction unit 4 performs a Sobel filter operation on the grayed image. By this Sobel filter calculation, an image with an enhanced edge (hereinafter referred to as an edge image) is obtained. In step S3, the edge extraction unit 4 performs binarization processing on the edge image to generate a binarized edge image. In step S4, the edge extraction unit 4 decomposes the binarized edge image into a plurality of predetermined directions using the result of the Sobel filter calculation. In step S5, the edge extraction unit 4 extracts edge segments from the plurality of binarized edge images decomposed in the respective directions.

  In step S <b> 6, the acquisition unit 5 lists geometric figure candidates formed using the edge segments extracted in steps S <b> 1 to S <b> 5. That is, the acquisition unit 5 extracts one or more rectangular regions from the edge segments extracted in steps S1 to S5 by taking out edge segments that may form the outer periphery (ie, the side) of the rectangular region. Get candidates. In step S7, the acquisition unit 5 acquires a combination of rectangular area candidates. At this time, the acquisition unit 5 selects a combination that does not contradict the relationship between the rectangular regions and does not contradict the relationship between the rectangular region candidate and the edge segment from among the combinations of the rectangular region candidates. .

  In step S8, the calculation unit 6 calculates a recall rate and a matching rate for each combination obtained in step S7. In this example, the recall represents the degree or ratio of the sides of the rectangular area candidate covered by the edge segment. The relevance ratio represents the degree or proportion of all edge segments extracted by the edge extraction unit 4 that are used as sides of rectangular area candidates.

  In step S <b> 9, the image extraction unit 7 specifies a combination that maximizes the evaluation value determined based on the recall rate and the matching rate. The evaluation value is, for example, an F value. Then, the image extraction unit 7 extracts rectangular area candidates included in the specified combination as rectangular areas to be acquired. When a black and white image is input to the processing unit 3, the graying process in step S1 is omitted.

  Next, processing of each step of the flowchart shown in FIG. 2 will be described in detail with reference to the drawings. In the following description, it is assumed that the image recognition device 1 extracts a rectangular area from an input image.

<Step S1: Graying>
Graying a color image corresponds to a process of projecting the pixel value of each pixel onto an arbitrary straight line passing through the origin of the RGB space. Therefore, various graying is possible according to the setting of the direction vector in the RGB space. For example, graying that represents the pixel value of each pixel by brightness is widely performed in image processing, and is calculated by the following equation. The pixel value of each pixel is represented by coordinates (R, G, B) in the RGB space.
Lightness = 0.299R + 0.587G + 0.114B

  The edge extraction unit 4 may gray out the color image by other methods. For example, the edge extraction unit 4 can gray the color image using the color difference.

<Steps S2 to S3: Sobel filter and binarization process>
The edge extraction unit 4 performs a Sobel filter operation on the gray image obtained in step S1. The Sobel filter is one of edge operators that emphasizes the edge of an image, and performs an X-direction filter operation and a Y-direction filter operation on each pixel of a gray image. The X direction filter and the Y direction filter are as shown in FIG. That is, the result Sx (x, y) of the X direction filter operation for the pixel (x, y) is obtained by the following equation.
g (x + 1, y + 1) + 2g (x, y + 1) + g (x-1, y + 1) -g (x + 1, y-1) -2g (x, y-1) -g (x-1, y-1)
Further, the result Sy (x, y) of the Y-direction filter operation for the pixel (x, y) is obtained by the following equation.
g (x + 1, y + 1) + 2g (x + 1, y) + g (x + 1, y-1) -g (x-1, y + 1) -2g (x-1, y) -g (x-1, y-1)
G (i, j) represents the density value of the pixel (i, j) calculated by the graying process.

Subsequently, the edge extraction unit 4 calculates the intensity and direction for each pixel using the result of the Sobel filter calculation. The intensity and direction of the pixel (x, y) is calculated by the following equation.
Strength = √ (Sx (x, y) 2 + Sy (x, y) 2)
Direction = arctan (Sy (x, y) / Sx (x, y))

  Here, considering the intensity value obtained for each pixel as a density value, the output from the Sobel filter can be processed as a gray image. Then, the edge extracting unit 4 generates a binarized edge image by performing binarization processing on the gray image. For the binarization processing, for example, the binarization method of Otsu can be used.

  FIG. 4 shows an example of the binarized edge image generated from the input image. In this example, an input image is obtained by photographing a white boat with a digital camera. In addition, four note stickers are affixed to the photographed whiteboard.

  In the input image, as shown in FIG. 4A, an area corresponding to the whiteboard 11 and an area corresponding to the note stickers 12a to 12d are formed. Note that the note stickers 12a to 12d have a color different from that of the whiteboard 11, and are represented by hatched areas in FIG. In this example, note stickers 12a and 12b partially overlap each other, and note stickers 12b and 12c also partially overlap each other. Note that characters and the like are written on the memo pad seals 12a to 12d, but the characters and the like are omitted here for easy viewing of the drawings.

  FIG. 4B shows a binarized edge image generated from the input image shown in FIG. In this binarized edge image, the pixels in the area corresponding to the end of the whiteboard 11, the pixels in the area corresponding to the ends of the note stickers 12a to 12d, and the note stickers 12a to 12d are described. The density value (or pixel value) of the pixel in the area corresponding to the character or the like is “1”, and the pixel value in the other area is “0”. That is, there is an edge in the area corresponding to the end of the whiteboard 11, the area corresponding to the end of the note stickers 12a to 12d, and the area corresponding to the characters written on the note stickers 12a to 12d. doing.

<Step S4: Direction decomposition>
As described above, the edge extraction unit 4 calculates the intensity and direction for each pixel using the result of the Sobel filter calculation. Here, the intensity is binarized by the binarization process described above. That is, a binarized edge image as shown in FIG. 4B is generated. Then, the extraction unit 4 decomposes the binarized edge image into a plurality of predetermined directions.

In this embodiment, the binarized edge image is decomposed into eight directions dir0 to dir7 shown in FIG. In this case, the following angle ranges are set for the disassembly directions dir0 to dir7, respectively.
Dir0: -π / 8 <θ ≦ π / 8
Dir1: π / 8 <θ ≦ 3π / 8
Dir2: 3π / 8 <θ ≦ 5π / 8
Dir3: 5π / 8 <θ ≦ 7π / 8
Dir4: 7π / 8 <θ ≦ 9π / 8 (-7π / 8)
Dir5: -7π / 8 <θ ≦ -5π / 8
Dir6: -5π / 8 <θ ≦ -3π / 8
Dir7: -3π / 8 <θ ≦ -π / 8

  FIG. 5B shows the relationship between the direction of the image area and the decomposition direction. In the example shown in FIG. 5B, two rectangular areas 12e and 12f are formed on the image. In this case, the lower side of the rectangular area 12e belongs to the angle range of the decomposition direction dir0. Further, the right side, the upper side, and the left side of the rectangular area 12e belong to the angular ranges of the decomposition directions dir2, dir4, and dir6, respectively. Similarly, each side of the rectangular area 12f belongs to the angular range of the decomposition directions dir1, dir3, dir5, and dir7.

  FIG. 6 is a diagram illustrating the direction decomposition process. In FIG. 6, each square corresponds to one pixel. In the binarized edge image shown in FIG. 6A, the value written in the upper part of each pixel represents the intensity obtained based on the result of the Sobel filter calculation. Here, the intensity is binarized. Further, the value written in the lower part of each pixel represents the direction obtained based on the result of the Sobel filter calculation. However, the value indicating the direction is omitted in a pixel having an intensity of zero.

  The edge extraction unit 4 extracts pixels having an intensity of 1 in each decomposition direction dir0 to dir7 and belonging to the angle range of the corresponding decomposition direction. For example, for the decomposition direction dir4, pixels whose intensity is 1 and whose direction belongs to 7π / 8 to 9π / 8 (that is, 157.5 to 202.5 degrees) are extracted. As a result, five pixels are extracted from the binarized edge image shown in FIG. 6A, and the image shown in FIG. 6B is obtained as the binarized edge image in the direction dir4. Similarly, the binarized edge images are generated in the other decomposition directions, respectively.

  FIG. 7 shows an example of a binarized edge image generated by direction decomposition. Here, FIG. 7A shows a binarized edge image in the decomposition direction dir2 obtained from the binarized edge image shown in FIG. 4B. This binarized edge image includes an edge corresponding to the right end of each of the note stickers 12a to 12d. FIG. 7B shows a binarized edge image in the decomposition direction dir6 obtained from the binarized edge image shown in FIG. 4B. This binarized edge image includes an edge corresponding to the left end of each of the note stickers 12a to 12d. Similarly, FIG. 7C shows a binarized edge image in the decomposition direction dir4 and includes an edge corresponding to the upper end of each of the note stickers 12a to 12d. FIG. 7D shows a binarized edge image in the decomposition direction dir0 and includes an edge corresponding to the lower end of each of the note stickers 12a to 12d.

<Step S5: Extraction of Edge Segment>
The edge extraction unit 4 extracts edge segments from the binarized edge image in each direction. An edge segment is an element constituting an edge. In this example, the edge segment is an area surrounded by four points, and is represented by the coordinates of these four points. Edge segment extraction includes labeling processing, overlap integration processing, noise removal processing, and integration processing described below.

(1) The labeling edge extraction unit 4 gives a label to each black pixel connected component in the binarized edge image. The black pixel connected component is an area where more than a predetermined number of black pixels are connected. A black pixel is a pixel whose binarized pixel value (or density value) is 1. The label is an identification number for identifying each black pixel connected component. In the example shown in FIG. 8A, labels L1 and L2 are assigned to each black pixel connected component.

  The edge extraction unit 4 projects each black pixel connected component in the coordinate system of the binarized edge image to be processed. The coordinate system of the binarized edge image to be processed is an orthogonal coordinate system rotated by an angle in the decomposition direction with respect to the coordinate system of the input image. For example, the coordinate system of the binarized edge image in the decomposition direction dir1 is rotated by π / 4 with respect to the coordinate system of the input image. Then, as shown in FIG. 8B, the edge extraction unit 4 projects the maximum projection value obtained by projecting the black pixel connected component onto each projection axis of the coordinate system of the binarized edge image to be processed. And get the minimum value.

  The edge extraction unit 4 obtains intersections of straight lines that pass through the above-described maximum values and minimum values and are orthogonal to the corresponding projection axes. Here, in FIG. 8B, if the maximum value and the minimum value on one projection axis are a and b, respectively, and the maximum value and the minimum value on the other projection axis are c and d, respectively, there are four intersections. Coordinates (a, c) (a, d) (b, c) (b, d) are obtained. These four intersection coordinates represent the coordinates of the four vertices of the smallest rectangle (that is, the circumscribed rectangle) surrounding the black pixel connected component. Then, as a result of the labeling process, the edge extraction unit 4 outputs, for each black pixel connected component, a label for identifying the black pixel connected component and four intersection coordinates representing the circumscribed rectangle of the black pixel connected component.

(2) The overlap integrated edge extraction unit 4 determines whether or not the circumscribed rectangles overlap each other for any two black pixel connected components in the binarized edge image. In the example shown in FIG. 9A, the circumscribed rectangles of the black pixel connected components L3 and L4 overlap each other. In this case, the edge extraction unit 4 integrates the black pixel connected components L3 and L4 into one black pixel connected component. That is, the same label is given to these two black pixel connected components. In FIG. 9B, the same label L3 is given to these two black pixel connected components. Also, the coordinates of the vertices of the minimum rectangle (the circumscribed rectangle of the black pixel connection components L3 and L4) surrounding these two black pixel connection components are calculated. Then, the edge extraction unit 4 repeats the overlapping integration process until there are no black pixel connected components that overlap each other.

(3) The noise removal edge extraction unit 4 performs noise removal processing on a set of black pixel connected components obtained after overlap integration processing. For example, when the size of the black pixel connected component obtained after the overlap integration process is smaller than a predetermined value, the black pixel connected component is determined as noise and removed from the above set. The size of the black pixel connected component is defined by, for example, the length of the long side of the circumscribed rectangle of the black pixel connected component.

(4) The integrated edge extraction unit 4 integrates adjacent black pixel components in the binarized edge image. That is, the black pixel connected components close to each other are integrated into one black pixel connected component. Here, the distance between the black pixel connected components is represented by, for example, a difference in projection values when each black pixel connected component is projected onto the projection axis described above. In this case, if the difference between the projection values on at least one projection axis is smaller than a preset threshold value, it is determined that the black pixel connected components should be integrated. In addition, when two black pixel connection components are integrated, the same label is given to these two black pixel connection components similarly to the above-described overlap integration processing. Also, the coordinates of the vertices of the minimum rectangle surrounding the two integrated black pixel connected components are calculated.

  The edge extraction unit 4 repeats the integration process until there are no black pixel connected components close to each other. Each black pixel connected component (or a circumscribed rectangle of each black pixel connected component) obtained by this integration processing is extracted as an edge segment.

  FIG. 10 is a diagram illustrating an example of edge segments extracted by the edge extraction unit 4. FIG. 10A shows edge segments extracted from the binarized edge image in the decomposition direction dir2 shown in FIG. In this example, edge segments E1 to E5 are extracted. Each of the edge segments E1 to E4 corresponds to the right end (or part thereof) of the note stickers 12a to 12d shown in FIG. The edge segment E5 corresponds to the end portion of the whiteboard 11. FIG. 10B shows all the edge segments respectively extracted from the binarized edge images in the decomposition directions dir0 to dir7. In this example, edge segments E1 to E18 are extracted.

  As described above, the edge extraction unit 4 extracts edge segments from the binarized edge images in the respective decomposition directions dir0 to dir7. Each edge segment is identified by a label. The position and shape of each edge segment are represented by the coordinates of the four vertices of the circumscribed rectangle of the black pixel connected component in the edge segment.

<Step S6: Acquisition of Rectangular Area Candidate>
The acquiring unit 5 lists all the rectangular area candidates based on the edge segments extracted in steps S1 to S5. A rectangular area candidate is represented by a set of edge segments that may form a rectangular area.

  FIG. 11 is a flowchart illustrating a process of extracting rectangular area candidates. This flowchart is executed by the acquisition unit 5 after the edge segment is extracted by the edge extraction unit 4 as described above. The acquisition unit 5 receives edge segment information from the edge extraction 4. The edge segment information includes information indicating the number of edge segments, the coordinates of the circumscribed rectangle of each edge segment, and information indicating the decomposition direction (dir0 to dir7) from which each edge segment is extracted.

  In step S11, the acquisition unit 5 creates a graph from the input edge segment information. In step S12, the acquisition unit 5 extracts a clique from the graph to obtain a set of edge segments that may form a rectangular area as a rectangular area candidate. In step S <b> 13, the acquiring unit 5 removes the rectangular area candidate larger than the predetermined maximum size and the rectangular area candidate smaller than the predetermined minimum size as noise. Thereby, a final rectangular area candidate is obtained. Then, the acquisition unit 5 outputs information indicating the number of rectangular area candidates and identification numbers (that is, labels) of edge segments constituting each rectangular area candidate.

(1) In step S11, the graph creation acquisition unit 5 creates a graph from the input edge segment information. The graph is composed of nodes and paths connecting the nodes. In this example, each node corresponds to one edge segment. In addition, the path connecting the nodes represents the possibility that the corresponding two edge segments form a rectangular area.

  The graph is created for each edge segment by determining whether or not a condition for forming a rectangular area is satisfied using the edge segment and each of the other edge segments. In the example shown in FIG. 10B, the edge segment E1 is combined with each of the edge segments E2 to E18 to determine whether or not the condition for forming the rectangular area is satisfied. For example, in the determination between the edge segments E1 and E2, when it is determined that the edge segment E1 corresponds to one side of the rectangular area, it is checked whether the edge segment E2 corresponds to an arbitrary side of the same rectangular area. The And the acquisition part 5 produces a graph by performing this determination about the combination to all the edge segments.

  An example of conditions for two edge segments to form a rectangular region is shown. Here, an edge segment in the decomposition direction dir2 shown in FIG. 5A will be described as an example. The edge segment in the decomposition direction dir2 corresponds to the right side of the rectangular area.

  In the following description, the centroid coordinates of the edge segment L are expressed as (L.ave_x, L.ave_y). The barycentric coordinates of the edge segment are calculated from the four vertex coordinates of the circumscribed rectangle that specifies the shape of the edge segment. The circumscribed rectangle that specifies the shape of the edge segment is as described in connection with the overlap integration process and the integration process when extracting the edge segment. For the four vertices of the circumscribed rectangle that specify the shape of the edge segment, the maximum x coordinate is L.max_x, the maximum y coordinate is L.max_y, the minimum x coordinate is L.min_x, and the minimum y coordinate is L. Expressed as .min_y.

  The acquisition unit 5 sets a virtual rectangular area. Then, it is assumed that one of the edge segments in the decomposition direction dir2 corresponds to the right side (or part thereof) of the virtual rectangular area. In the example shown in FIGS. 12 to 15, a virtual rectangular area 21 is set, and for the edge segment L1 in the decomposition direction dir2, other edge segments (hereinafter referred to as search target edge segments) constituting the rectangular area candidate are searched. The

When the search target edge segment (L2) is extracted from the decomposition direction dir0, the acquisition unit 5 determines that the edge segments L1 and L2 may form a rectangular area if the following condition is satisfied. In this case, the edge segments L1 and L2 correspond to the right side and the lower side of the rectangular area 21, respectively, as shown in FIG.
L1.ave_x> = L2.max_x and L1.max_y <= L2.ave_y
In the second inequality, L2.min_y may be used instead of L2.ave_y.

When the search target edge segment (L3) is extracted from the decomposition direction dir2, the acquisition unit 5 determines that the edge segments L1 and L3 may form a rectangular area if the following condition is satisfied. In this case, the edge segments L1 and L3 both correspond to the right side of the rectangular area 21, as shown in FIG. TH1 is a predetermined threshold value determined in advance.
| L1.ave_x-L3.ave_x | <TH1

When the search target edge segment (L4) is extracted from the decomposition direction dir4, the acquisition unit 5 determines that the edge segments L1 and L4 may form a rectangular area if the following condition is satisfied. In this case, the edge segments L1 and L4 correspond to the right side and the upper side of the rectangular area 21, respectively, as shown in FIG.
L1.ave_x> = L4.max_x and L1.min_y> = L4.ave_y
In the second inequality, L4.max_y may be used instead of L4.ave_y.

When the search target edge segment (L5) is extracted from the decomposition direction dir6, the acquisition unit 5 determines that the edge segments L1 and L5 may form a rectangular area if the following condition is satisfied. In this case, the edge segments L1 and L5 correspond to the right side and the left side of the rectangular area 21, respectively, as shown in FIG.
L1.ave_x> = L5.ave_x

  When the search target edge segment is extracted from a decomposition direction other than the decomposition directions dir0, dir2, dir4, and dir6, the acquisition unit 5 has no possibility that the edge segment L1 and the search target edge segment form a rectangular area. judge. Here, the determination condition in the case where one edge segment is the right side of the rectangular area has been described with reference to FIGS. 12 to 15, but one edge segment is the left side, the upper side, or the lower side of the rectangular area. The determination condition in a certain case can be obtained similarly.

  Thus, the acquisition unit 5 determines whether each edge segment has a possibility of forming a rectangular area together with other edge segments. Therefore, the graph created by the above determination is represented by an n × n matrix when the total number of extracted edge segments is n. In this case, when the combination of the i-th edge segment and the j-th edge segment satisfies the condition for forming the rectangular area, the acquisition unit 5 and the (i, j) component and (j, i When the combination does not satisfy the above condition, 0 is set for each of the (i, j) and (j, i) components of this matrix. An example of the created graph is shown in FIG.

(2) The clique extraction acquisition unit 5 extracts a clique from the graph created as described above. A clique corresponds to a maximal complete subgraph of the graph. The complete graph means that all nodes constituting the graph are connected to all other nodes by paths. In addition, the maximal complete subgraph means a complete subgraph and a subgraph in which there is no other complete subgraph that truly includes the subgraph. Therefore, there is a possibility that the set of edge segments constituting the clique forms a rectangular area with all of the edge segments other than itself. An example of a clique extracted from the graph shown in FIG. 16A is shown in FIG. In FIG. 16B, “−1” means the end of the clique component.

  In the embodiment shown in FIG. 16B, for example, the clique 1 has L25, L24, L23, L18, and L1 as elements as a set of edge segments that may form a rectangular area. In this case, when any two edge segments are extracted from L25, L24, L23, L18, and L1, the two extracted edge segments always satisfy the condition for forming the rectangular area described above. Become.

  Thus, the acquisition unit 5 creates a graph from the edge segment information, and further extracts a clique from the graph. Here, each clique is a set of edge segments that may form a rectangular region. That is, the acquisition unit 5 acquires one or a plurality of rectangular area candidates expressed by a set of a plurality of edge segments.

<Step S7: Acquisition of Combination of Rectangular Area Candidates>
The acquisition unit 5 lists combinations of rectangular area candidates based on the rectangular area candidates extracted in step S6. A combination of rectangular area candidates is represented by a set of compatible rectangular area candidates.

  FIG. 17 is a flowchart showing a process for acquiring a combination of rectangular area candidates. This flowchart is executed by the acquisition unit 5 after the rectangular area candidate is extracted as described above. At this time, the acquisition unit 5 uses the rectangular area candidate information. The rectangular area candidate information includes information indicating the number of rectangular area candidates, a number for identifying each rectangular area candidate, and a number of edge segments constituting each rectangular area candidate.

  In step S21, the acquisition unit 5 creates a graph from the rectangular area candidate information. In step S22, the acquisition unit 5 obtains a combination of rectangular area candidates by extracting a clique from the graph. Then, the acquisition unit 5 outputs information indicating the number of combinations of rectangular area candidates, and identification numbers of the rectangular area candidates that constitute the combinations of the rectangular area candidates.

(1) The graph creation acquisition unit 5 creates a graph from the rectangular area candidate information in step S21. As described above, the graph includes nodes and paths connecting the nodes. However, when obtaining a combination of rectangular area candidates, each node corresponds to one rectangular area candidate. In addition, the path connecting the nodes represents the possibility that two corresponding rectangular area candidates are compatible with each other.

The graph is created for each rectangular area candidate by determining whether or not the rectangular area candidate and each of the other rectangular area candidates satisfy a mutually compatible condition. For example, the following two conditions satisfy the two rectangular area candidates.
Condition 1: One rectangular area candidate is not completely included by the other rectangular area candidate Condition 2: Two rectangular area candidates do not share the same edge segment

  For example, it is assumed that the rectangular area candidate 1 is formed in the rectangular area candidate 2. In this case, since the rectangular area candidate 1 is completely included in the rectangular area candidate 2, the condition 1 is not satisfied. That is, it is determined that the rectangular area candidates 1 and 2 are not compatible.

  In this case, the rectangular area candidate 1 is composed of edge elements L1, L2, and L3, and the rectangular area candidate 3 is composed of edge elements L3, L5, and L6. Since L3 is shared, Condition 2 is not satisfied. That is, it is determined that the rectangular area candidates 1 and 3 are not compatible.

  In this manner, the acquisition unit 5 determines whether each rectangular area candidate can be compatible with other rectangular area candidates. Therefore, the graph created by the above determination is represented by an m × m matrix when the total number of extracted rectangular area candidates is m. In this case, when the i-th rectangular area candidate and the j-th rectangular area candidate can be compatible, the acquiring unit 5 sets 1 to each of the (i, j) component and the (j, i) component of this matrix. When these rectangular area candidates are not compatible, 0 is set to each of the (i, j) component and (j, i) component of this matrix. An example of the created graph is shown in FIG.

(2) The clique extraction acquisition unit 5 extracts a clique from the graph created as described above. As described above, the clique corresponds to the maximum complete subgraph of the graph. Therefore, each clique is a set of rectangular area candidates that can be compatible with each other. FIG. 18B shows an example of a clique extracted from the graph shown in FIG.

  Thus, the acquisition unit 5 creates a graph from the rectangular area candidate information, and further extracts a clique from the graph. Here, each clique is a set of compatible rectangular regions. That is, the acquisition unit 5 acquires a combination of one or more rectangular area candidates expressed by a set of one or more rectangular area candidates.

<Steps S8 to S9: Evaluation and Extraction>
The calculation unit 6 calculates a recall rate and a matching rate for each compatible combination of rectangular area candidates, and further calculates an evaluation value determined based on the recall rate and the matching rate. The evaluation value is a so-called F value. Then, the image extraction unit 7 specifies a combination of rectangular area candidates having the highest evaluation value, and extracts an image of the rectangular area included in the combination.

(1) Recall Rate Calculation The calculation unit 6 calculates a recall rate for each combination of rectangular area candidates. The recall represents how much the combination of rectangular area candidates is explained by the extracted edge segment. In this embodiment, the recall represents the degree or ratio of the outer periphery of each rectangular area included in the combination of rectangular area candidates covered by the extracted edge segment.

(2) The precision calculation unit 6 calculates the precision for each combination of rectangular area candidates. The relevance ratio represents how much the combination of rectangular area candidates can explain the extracted edge segment. In this embodiment, the relevance ratio represents the degree or proportion of all the edge segments extracted by the edge extraction unit 4 that are used as the sides of the rectangular area included in the combination of the rectangular area candidates.

(3) The F value calculation unit 6 calculates the F value for each combination of rectangular area candidates. The F value is an evaluation scale that takes into consideration the recall and precision, and is obtained by the harmonic average of the recall and precision (including the harmonic average multiplied by a constant). That is, when the recall is represented by R and the precision is represented by P, the F value is calculated by the following equation.
F value = 2 × R × P / (R + P)

(4) Image extraction The image extraction unit 7 identifies a combination of rectangular area candidates with the highest evaluation value, and extracts an image of one or more rectangular areas included in the combination. The extracted image data is stored in the extraction result storage unit 8. The extracted image data stored in the extraction result storage unit 8 is output by the output unit 9 in accordance with, for example, an instruction from the user.

<Example>
In the following embodiment, as shown in FIG. 19, nine edge segments L1 to L9 are extracted from the input image. The edge segment extraction is realized by steps S1 to S5 of the flowchart shown in FIG.

  In FIG. 19, “direction” and “length” are shown for each of the edge segments L1 to L9. The “direction” corresponds to an angle calculated based on the output of the Sobel filter, and is represented by dir0 to dir7 shown in FIG. 5A in this embodiment. The “length” is the length of the long side of the rectangular area forming the edge segment, and is represented by the number of pixels, for example.

  First, the acquisition unit 5 refers to the edge segments L1 to L9 and acquires rectangular area candidates. In order to acquire a rectangular area candidate, the acquisition unit 5 determines whether any two edge segments extracted from the edge segments L1 to L9 have a possibility of forming a rectangular area. At this time, the acquisition unit 5 determines whether or not all combinations have a possibility of forming a rectangular area. As a result, the graph shown in FIG.

  As an example, the edge segment L3 will be described. That is, it is determined whether or not the pair of the edge segment L3 and each of the other edge segments may form a rectangular area. The direction of the edge segment L3 is dir2.

(1) Edge segment L1
The direction of the edge segment L1 is dir4. Therefore, in order for the edge segments L3 and L1 to form a rectangular area, the following conditions must be satisfied.
L3.ave_x> = L1.max_x and L3.min_y> = L1.ave_y
Here, the edge segment L3 is located on the right side of the edge segment L1, and the barycentric coordinate of the edge segment L3 in the X direction is larger than the maximum coordinate of the edge segment L1 in the X direction. The edge segment L3 is positioned below the edge segment L1, and the minimum coordinate in the Y direction of the edge segment L3 is larger than the barycentric coordinate in the Y direction of the edge segment L1. That is, the above two conditions are satisfied, and the edge segments L3 and L1 can form a rectangular area. Accordingly, in the graph shown in FIG. 20A, “1” is set for the edge segments L3 and L1.

(2) Edge segment L2
The direction of the edge segment L2 is the same as that of the edge segment L1, and is dir4. The positional relationship between the edge segments L3 and L2 is the same as the positional relationship between the edge segments L3 and L1. Therefore, the edge segments L3 and L2 can form a rectangular area, and “1” is set for the edge segments L3 and L2.

(3) Edge segment L4
The direction of the edge segment L4 is also the same as the edge segment L1 and is dir4. Therefore, the conditions for the edge segments L3 and L4 to form a rectangular region are similar to the conditions for the edge segments L3 and L1 described above, and are as follows.
L3.ave_x> = L4.max_x and L3.min_y> = L4.ave_y
However, the edge segment L3 is located on the left side of the edge segment L4, and the barycentric coordinate of the edge segment L3 in the X direction is smaller than the maximum coordinate of the edge segment L4 in the X direction. That is, the above condition is not satisfied, and the edge segments L3 and L4 cannot form a rectangular area. Therefore, “0” is set for the edge segments L3 and L4 in the graph shown in FIG.

(4) Edge segment L5
The direction of the edge segment L5 is the same as the edge segment L3 and is dir2. Therefore, in order for the edge segments L3 and L5 to form a rectangular area, the following conditions must be satisfied.
| L3.ave_x-L5.ave_x | <TH1
The threshold value TH1 is a small value such that the two edge segments are arranged on substantially the same straight line. Here, the edge segment L3 is located on the left side of the edge segment L5, and the difference between the barycentric coordinate in the X direction of the edge segment L3 and the barycentric coordinate in the X direction of the edge segment L5 is larger than the threshold value TH1. That is, the above condition is not satisfied, and the edge segments L3 and L5 cannot form a rectangular area. Therefore, “0” is set for the edge segments L3 and L5 in the graph shown in FIG.

(5) Edge segment L6
The direction of the edge segment L6 is dir0. Therefore, in order for the edge segments L3 and L6 to form a rectangular area, the following conditions must be satisfied.
L3.ave_x> = L6.max_x and L3.max_y <= L6.ave_y
Here, the edge segment L3 is located on the left side of the right tip of the edge segment L6, and the barycentric coordinate of the edge segment L3 in the X direction is smaller than the maximum coordinate of the edge segment L6 in the X direction. That is, the above condition is not satisfied, and the edge segments L3 and L6 cannot form a rectangular area. Therefore, “0” is set for the edge segments L3 and L6 in the graph shown in FIG.

(6) Edge segments L7 to L8
The direction of the edge segment L7 is also dir0. Therefore, in order for the edge segments L3 and L7 to form a rectangular area, the following conditions must be satisfied.
L3.ave_x> = L7.max_x and L3.max_y <= L7.ave_y
Here, the edge segment L3 is located on the right side of the right tip of the edge segment L7, and the barycentric coordinate of the edge segment L3 in the X direction is larger than the maximum coordinate of the edge segment L7 in the X direction. The edge segment L3 is positioned above the edge segment L7, and the maximum coordinate in the Y direction of the edge segment L3 is smaller than the barycentric coordinate in the Y direction of the edge segment L7. That is, the above two conditions are satisfied, and the edge segments L3 and L7 can form a rectangular area. Therefore, “1” is set for the edge segments L3 and L7 in the graph shown in FIG. Similarly, “1” is set for the edge segments L3 and L8.

(7) Edge segment L9
The direction of the edge segment L9 is dir6. Therefore, in order for the edge segments L3 and L9 to form a rectangular area, the following conditions must be satisfied.
L3.ave_x> = L9.ave_x
Here, the edge segment L3 is located on the right side of the edge segment L9, and the barycentric coordinate in the X direction of the edge segment L3 is larger than the barycentric coordinate in the X direction of the edge segment L9. That is, the above condition is satisfied, and the edge segments L3 and L9 can form a rectangular area. Accordingly, in the graph shown in FIG. 20A, “1” is set for the edge segments L3 and L9.

  Similarly, the acquisition unit 5 determines whether a rectangular area can be configured for all edge segment pairs. As a result, the graph shown in FIG.

  Subsequently, the acquisition unit 5 extracts a clique from the graph created as described above. That is, a maximal complete partial graph is extracted from the graph shown in FIG. As a result, four cliques C1 to C4 shown in FIG. 20B are extracted.

  Each clique represents one rectangular area candidate. For example, the clique C1 represents a rectangular area candidate in which five edge segments L1, L2, L3, L8, and L9 are used as components on the outer periphery (ie, sides). Thus, in this embodiment, four rectangular area candidates are obtained.

  FIGS. 21A to 21D show rectangular area candidates corresponding to the cliques C1 to C4, respectively. For example, the rectangular area candidate REC1 represented by a broken line in FIG. 21A is formed by edge segments L1, L2, L3, L8, and L9 that are elements of the clique C1. Similarly, rectangular area candidates REC2 to REC4 represented by broken lines in FIGS. 21 (b) to 21 (d) are formed by elements of cliques C2 to C4.

  In this embodiment, the acquisition unit 5 extracts only cliques having three or more edge segments as elements. That is, even in the maximal complete subgraph, when the number of elements (that is, the number of edge segments) is 2 or less, the acquisition unit 5 does not extract such a clique. For example, the edge segments L4 and L7 may constitute a rectangular area. However, it is difficult to specify the shape of the rectangular area with two edge segments. Therefore, the edge segments L4 and L7 are not extracted as cliques. Similarly, the edge segments L4 and L8 are not extracted as cliques. However, the acquisition unit 5 may extract a clique having two elements.

  Furthermore, the acquisition unit 5 determines whether the combinations of the rectangular area candidates can be compatible with each other for the rectangular area candidates REC1 to REC4 obtained as described above. Here, each edge segment belongs to one rectangular area candidate, and is not shared by a plurality of rectangular area candidates.

  For example, the elements belonging to the rectangular area candidate REC1 are edge segments L1, L2, L3, L8, and L9, and the elements belonging to the rectangular area candidate REC3 are edge segments L1, L2, L5, L6, and L9. That is, the rectangular area candidates REC1 and REC3 share the edge segments L1, L2, and L9. Therefore, the rectangular area candidates REC1 and REC3 are not compatible. Similarly, the rectangular area candidates REC1 and REC4, the rectangular area candidates REC2 and REC3, and the rectangular area candidates REC3 and REC4 are not compatible.

  In other words, in the rectangular area candidates REC1 to REC4, only the combination of the rectangular area candidates REC1 and REC2 and the combination of the rectangular area candidates REC2 and REC4 can be compatible. FIG. 22A shows a graph created based on the determination result.

  Subsequently, the acquisition unit 5 extracts a clique from the graph created as described above. That is, a maximal complete subgraph is extracted from the graph shown in FIG. As a result, three cliques C11 to C13 shown in FIG. 22B are extracted.

  Each clique represents a combination of one rectangular area candidate. For example, the clique C11 represents an image in which two rectangular area candidates REC1 and REC2 exist. In this embodiment, even a subgraph having only one element is extracted as one clique when the element does not belong to another clique. For example, the element of the clique C12 is only the rectangular area candidate REC3.

  Here, for example, the rectangular area candidate REC1 belongs to the clique C11. For this reason, a partial graph having only the rectangular region candidate REC1 as an element is not a maximum graph. Therefore, a subgraph having only the rectangular area candidate REC1 as an element is not extracted as a clique. The same applies to the rectangular area candidates REC2 and REC4.

  FIG. 23A to FIG. 23C show combinations of rectangular area candidates corresponding to the cliques C11 to C13, respectively. FIG. 23A shows an image in which a rectangular area candidate REC1 having edge segments L1, L2, L3, L8, and L9 as elements and a rectangular area candidate REC2 having edge segments L4, L5, and L6 as elements exist. FIG. 23B shows an image in which a rectangular region candidate REC3 having edge segments L1, L2, L5, L6, and L9 as elements is present. FIG. 23C shows an image in which a rectangular area candidate REC2 having the edge segments L4, L5, and L6 as elements, and a rectangular area candidate REC4 having the edge segments L1, L2, L3, L7, and L9 as elements.

  23A to 23C show the shape of each rectangular area candidate. For example, the size of the rectangular area candidate REC1 is “80 × 60”. This notation indicates that the length of the rectangular area candidate REC1 in the X direction is “80” and the length in the Y direction is “60”. The same applies to the other rectangular area candidates REC2 to REC4.

  The calculation unit 6 calculates the reproduction rate R and the relevance rate P for each combination of rectangular area candidates shown in FIGS. 23A to 23C, and further calculates the F value from the reproducibility R and the relevance rate P. calculate.

  The recall ratio R is calculated by “the sum of the lengths of the edge segments constituting the rectangular area candidate / the sum of the peripheral lengths of the rectangular area candidates”. The matching rate P is calculated by “the sum of the lengths of the edge segments constituting the rectangular region candidate / the sum of the lengths of all the extracted edge segments”. The F value is calculated by “2RP / (R + P)”. Note that the perimeters of the rectangular area candidates REC1, REC2, REC3, and REC4 are “280”, “280”, “480”, and “320”, respectively, as shown in FIGS. 23 (a) to 23 (c). Further, the sum of the lengths of all the edge segments L1 to L9 extracted by the edge extraction unit 4 is “411” as shown in FIG.

The recall ratio R, precision ratio P, and F value for the combination shown in FIG. 23A are calculated as follows.
Recall rate R = {(35 + 25 + 20 + 55 + 55) + (60 + 58 + 78)} / (280 + 280) = 0.689
Precision P = {(35 + 25 + 20 + 55 + 55) + (60 + 58 + 78)} / 411 = 0.939
F value = 2 * 0.689 * 0.939 / (0.689 + 0.939) = 0.795

The recall ratio R, precision ratio P, and F value for the combinations shown in FIG. 23B are calculated as follows.
Recall rate R = (35 + 25 + 58 + 78 + 55) /480=0.523
Precision P = (35 + 25 + 58 + 78 + 55) /411=0.611
F value = 2 * 0.523 * 0.611 / (0.523 + 0.611) = 0.564

The recall rate R, the matching rate P, and the F value for the combinations shown in FIG. 23C are calculated as follows.
Recall rate R = {(60 + 58 + 78) + (35 + 25 + 20 + 25 + 55)} / (280 + 320) = 0.593
Precision P = {(60 + 58 + 78) + (35 + 25 + 20 + 25 + 55)} / 411 = 0.866
F value = 2 * 0.593 * 0.866 / (0.593 + 0.866) = 0.704

  The image extraction unit 7 identifies the combination having the highest F value from the combinations of the rectangular area candidates shown in FIGS. 23 (a) to 23 (c). In this embodiment, the F value for the combination of rectangular area candidates shown in FIG. Therefore, the image extraction unit 7 extracts and outputs images corresponding to the rectangular area candidates REC1 and REC2 shown in FIG.

<Extraction of other geometric figures>
In the above-described embodiment, the image recognition device 1 extracts a rectangular image region from the input image. However, the image recognition apparatus 1 is not limited to the configuration for extracting a rectangular image region, and may extract an image region corresponding to another geometric figure. Hereinafter, a configuration and a method for extracting an equilateral triangle image area from an input image will be described.

  A method of extracting an equilateral triangle image region from the input image is almost the same as the procedure of the flowchart shown in FIG. However, when an equilateral triangle image region is extracted, the processes in steps S4 and S6 are different from the process of extracting a rectangular region.

  When extracting an equilateral triangle image region, the edge extraction unit 4 decomposes the binarized edge image into 24 directions dir0 to dir23 as shown in FIG. The angle range assigned to each disassembly direction is 15 degrees.

  The acquiring unit 5 extracts equilateral triangle region candidates configured using the edge segments obtained by the edge extracting unit 4. Here, conditions for determining whether or not any two edge segments can form an equilateral triangle region will be described. In the following description, it is assumed that the direction of one edge segment L1 is dir0.

When the search target edge segment (L2) is extracted from the decomposition direction dir0, the acquisition unit 5 determines that the edge segments L1 and L2 may constitute an equilateral triangle area when the following condition is satisfied. In this case, the edge segments L1 and L2 both correspond to the lower side of the equilateral triangle region 31, as shown in FIG.
| L1.ave_y-L2.ave_y | <TH1

When the search target edge segment (L3) is extracted from the decomposition direction dir8, the acquisition unit 5 determines that the edge segments L1 and L3 may constitute an equilateral triangle area when the following condition is satisfied. In this case, the edge segments L1 and L3 respectively correspond to the lower side and the upper right side of the equilateral triangle region 31, as shown in FIG. “Sqrt” represents a square root.
L1.ave_x <= (L1.ave_y-L3.ave_y) / (sqrt (3)) + L3.ave_x

When the search target edge segment (L4) is extracted from the decomposition direction dir16, when the following condition is satisfied, the acquisition unit 5 determines that the edge segments L1 and L4 may constitute an equilateral triangle region. In this case, the edge segments L1 and L4 correspond to the lower side and the upper left side of the equilateral triangular region 31, respectively, as shown in FIG.
L1.min_x> =-(L1.ave_y-L4.ave_y) / (sqrt (3)) + L4.ave_x

  When the search target edge segment is extracted from a decomposition direction other than the decomposition directions dir0, dir8, and dir16, the acquisition unit 5 determines that the edge segment L1 and the search target edge segment do not have a possibility of forming an equilateral triangle area. To do. Here, the determination condition when one edge segment is the lower side of the equilateral triangle area has been described with reference to FIGS. 25 to 27. However, one edge segment is the upper right side or the left side of the equilateral triangle area. The determination conditions for the oblique upper side can be obtained similarly.

  Thereafter, the image recognition apparatus 1 acquires compatible equilateral triangle region combinations, and calculates an F value for each combination. Then, the image recognition apparatus 1 extracts one or a plurality of equilateral triangle area candidate images belonging to the combination having the highest F value.

<Hardware configuration of image recognition device>
FIG. 28 is a diagram illustrating a hardware configuration of a computer system for realizing the image recognition apparatus 1. As shown in FIG. 28, the computer system 100 includes a CPU 101, a memory 102, a storage device 103, a reading device 104, a communication interface 106, and an input / output device 107. The CPU 101, the memory 102, the storage device 103, the reading device 104, the communication interface 106, and the input / output device 107 are connected to each other via a bus 108, for example.

  The CPU 101 can provide some or all of the functions of the edge extraction unit 4, the acquisition unit 5, the calculation unit 6, and the image extraction unit 7 by executing an image recognition program using the memory 102. At this time, the CPU 101 may provide the functions of the edge extraction unit 4, the acquisition unit 5, the calculation unit 6, and the image extraction unit 7 by executing a program describing the processing of the flowchart shown in FIG.

  The memory 102 is a semiconductor memory, for example, and includes a RAM area and a ROM area. The storage device 103 is, for example, a hard disk, and stores an image recognition program related to image recognition according to the embodiment. Note that the storage device 103 may be a semiconductor memory such as a flash memory. The storage device 103 may be an external recording device. The image data storage unit 2 and the extraction result storage unit 8 are realized using the memory 102 and / or the storage device 103.

  The reading device 104 accesses the removable recording medium 105 in accordance with an instruction from the CPU 101. The detachable recording medium 105 includes, for example, a semiconductor device (USB memory or the like), a medium to / from which information is input / output by a magnetic action (magnetic disk or the like), a medium to / from which information is input / output by an optical action (CD-ROM, For example, a DVD). The communication interface 106 transmits / receives data via a network according to instructions from the CPU 101. The input / output device 107 corresponds to, for example, a device that receives an instruction from a user, an interface that receives image data from a digital camera, an interface that outputs a recognition result, and the like.

The image recognition program of the embodiment is provided to the computer system 100 in the following form, for example.
(1) Installed in advance in the storage device 103.
(2) Provided by the removable recording medium 105.
(3) Provided from the program server 110.
Note that the image recognition method of the embodiment may provide the above-described processing using a plurality of computers. In this case, a certain computer may request a part of the above-described processing to another computer via a network and receive the processing result.

  Furthermore, a part of the image recognition apparatus according to the embodiment may be realized by hardware. Alternatively, the image recognition apparatus of the embodiment may be realized by a combination of software and hardware.

<Effect of embodiment>
As described above, according to the image recognition apparatus of the embodiment, using the edge segments extracted in the input image, all the combinations of compatible area candidates corresponding to the objects having the predetermined geometric shapes are extracted. Is done. Therefore, even when the objects overlap each other or when the extracted edge segment is interrupted, the correct object (that is, the area corresponding to the actual object) is the above-mentioned area candidate. Included in the combination. Therefore, according to the image recognition device of the embodiment, when an input image is recognized, there is a low possibility that an object will leak without being extracted.

  Further, according to the image recognition apparatus of the embodiment, for each combination of region candidates to be extracted, an evaluation value that is determined based on the recall rate and the matching rate regarding the edge segment and the region candidate is calculated. Then, an area to be extracted is determined according to the evaluation value. As a result, one or more correct areas to be extracted can be identified with high accuracy from among a plurality of area candidates. Therefore, even when the objects overlap each other or when the object and the background are similar in color, the image of the object can be accurately extracted.

  The following additional notes are further disclosed with respect to the embodiments including the examples described above. Note that the present invention is not limited to the following supplementary notes.

(Appendix 1)
An edge extractor for extracting edge segments from the image;
An acquisition unit that acquires a combination of predetermined geometric figure candidates formed by using the edge segment extracted by the edge extraction unit;
For each combination acquired by the acquisition unit, the recall representing the extent to which the outer periphery of the graphic candidate is covered by the extracted edge segment, and the extracted edge segment are used as the graphic candidate. A calculation unit for calculating the relevance ratio representing the degree of
An image extraction unit for extracting a region corresponding to a graphic candidate included in a combination having a maximum evaluation value determined based on the reproduction rate and the matching rate;
An image recognition apparatus.
(Appendix 2)
The calculation unit calculates the recall by dividing the sum of the lengths of the edge segments used for the graphic candidates by the sum of the outer perimeters of the graphic candidates, and determines the graphic candidates. The relevance ratio is calculated by dividing the sum of the lengths of the edge segments used by the sum of the lengths of all the edge segments extracted by the edge extraction unit. Image recognition device.
(Appendix 3)
The image recognition apparatus according to appendix 1 or 2, wherein the evaluation value is a harmonic average of the recall and the precision.
(Appendix 4)
The acquisition unit extracts a geometric figure candidate formed using the edge segment extracted by the edge extraction unit, and the figure candidate is compatible among the combinations of the extracted figure candidates. The image recognition device according to any one of appendices 1 to 3, wherein a combination that can be acquired is acquired.
(Appendix 5)
The image recognition apparatus according to appendix 4, wherein the acquisition unit acquires combinations of graphic candidates that are not in an inclusion relationship among the combinations of extracted graphic candidates.
(Appendix 6)
The acquisition unit acquires a combination of graphic candidates in which the edge segment is not shared by a plurality of graphic candidates among combinations of extracted graphic candidates. Image recognition device.
(Appendix 7)
An edge extractor for extracting edge segments from the image;
A combination of predetermined geometric figure candidates formed by using the edge segment extracted by the edge extraction unit is extracted, and two or more figure candidates from the combination have an inclusion relation. An acquisition unit that acquires combinations that do not have and share two or more graphic candidates that do not share the same edge segment;
For each combination acquired by the acquisition unit, a calculation unit that calculates an evaluation value indicating the validity of the graphic candidate for the extracted edge segment;
An image extraction unit for extracting a region corresponding to a graphic candidate included in the combination having the maximum evaluation value calculated by the calculation unit;
An image recognition apparatus.
(Appendix 8)
Computer
Extract edge segments from the image,
Obtaining a combination of predetermined geometric figure candidates formed using the extracted edge segments;
For each of the combinations, a reproduction rate indicating the extent to which the outer periphery of the graphic candidate is covered by the extracted edge segment, and a matching rate indicating the extent to which the extracted edge segment is used as the graphic candidate Respectively,
An image recognition method comprising extracting a region corresponding to a graphic candidate included in a combination having a maximum evaluation value determined based on the reproduction rate and the matching rate.
(Appendix 9)
Extract edge segments from the image,
Obtaining a combination of predetermined geometric figure candidates formed using the extracted edge segments;
For each of the combinations, a reproduction rate indicating the extent to which the outer periphery of the graphic candidate is covered by the extracted edge segment, and a matching rate indicating the extent to which the extracted edge segment is used as the graphic candidate Respectively,
An image recognition program for causing a computer to execute a process of extracting a region corresponding to a graphic candidate included in a combination having a maximum evaluation value determined based on the reproduction rate and the matching rate.

DESCRIPTION OF SYMBOLS 1 Image recognition apparatus 3 Processing part 4 Edge extraction part 5 Acquisition part 6 Calculation part 7 Image extraction part

Claims (5)

An edge extractor for extracting edge segments from the image;
An acquisition unit that acquires a combination of predetermined geometric figure candidates formed by using the edge segment extracted by the edge extraction unit;
For each combination acquired by the acquisition unit, the recall representing the extent to which the outer periphery of the graphic candidate is covered by the extracted edge segment, and the extracted edge segment are used as the graphic candidate. A calculation unit for calculating the relevance ratio representing the degree of
An image extraction unit for extracting a region corresponding to a graphic candidate included in a combination having a maximum evaluation value determined based on the reproduction rate and the matching rate;
An image recognition apparatus. The calculation unit calculates the recall by dividing the sum of the lengths of the edge segments used for the graphic candidates by the sum of the outer perimeters of the graphic candidates, and determines the graphic candidates. The precision is calculated by dividing the sum of the lengths of the edge segments used by the sum of the lengths of all the edge segments extracted by the edge extraction unit. Image recognition device. An edge extractor for extracting edge segments from the image;
A combination of predetermined geometric figure candidates formed by using the edge segment extracted by the edge extraction unit is extracted, and two or more figure candidates from the combination have an inclusion relation. An acquisition unit that acquires combinations that do not have and share two or more graphic candidates that do not share the same edge segment;
For each combination acquired by the acquisition unit, a calculation unit that calculates an evaluation value indicating the validity of the graphic candidate for the extracted edge segment;
An image extraction unit for extracting a region corresponding to a graphic candidate included in the combination having the maximum evaluation value calculated by the calculation unit;
An image recognition apparatus. Computer
Extract edge segments from the image,
Obtaining a combination of predetermined geometric figure candidates formed using the extracted edge segments;
For each of the combinations, a reproduction rate indicating the extent to which the outer periphery of the graphic candidate is covered by the extracted edge segment, and a matching rate indicating the extent to which the extracted edge segment is used as the graphic candidate Respectively,
An image recognition method comprising extracting a region corresponding to a graphic candidate included in a combination having a maximum evaluation value determined based on the reproduction rate and the matching rate. Extract edge segments from the image,
Obtaining a combination of predetermined geometric figure candidates formed using the extracted edge segments;
For each of the combinations, a reproduction rate indicating the extent to which the outer periphery of the graphic candidate is covered by the extracted edge segment, and a matching rate indicating the extent to which the extracted edge segment is used as the graphic candidate Respectively,
An image recognition program for causing a computer to execute a process of extracting a region corresponding to a graphic candidate included in a combination having a maximum evaluation value determined based on the reproduction rate and the matching rate.