JP5185072B2 - Frame detection method, frame detection apparatus, and frame detection program - Google Patents

Description

  The present invention relates to a frame detection method, a frame detection apparatus, and a frame detection program for detecting a frame from an “image including a frame” taken by a camera or the like.

  Up to now, there are the following methods as means for selecting a frame region from an image.

  The first method is a method of displaying a guide at the time of shooting and aligning the object with the guide.

  The second method is a method in which the user specifies a frame to be imaged.

Patent Document 1 describes an invention in which unnecessary portions are deleted by performing a histogram process when extracting a license plate character of a vehicle, but this method has a straight line such as the number “1”. There is a possibility of misjudging characters as unnecessary parts.
JP 2005-352705 A

  In the first method described above, there are many devices that cannot be guided, and may not be supported. Further, since the user must match the object to be photographed with the guide, it is inconvenient for the user. In addition, the second method described above also inconveniences the user.

  Therefore, the present invention provides a frame detection method and a frame detection device that can automatically select a frame from a captured image even if the object to be imaged is rotated with respect to the camera or the size is not constant. And it aims at providing a frame detection program.

  According to the present invention, a binarization step for binarizing an input grayscale image into black and white to obtain a binarized image, and a vertical direction from the binarized image passing through the vicinity of the center of the binarized image. A black point that passes through the first straight line, a first point that is parallel to the first straight line and that is separated from the first straight line in one direction by a distance that is set to be shorter than a lateral side of the frame to be detected. A third point separated from the first straight line in the other direction by a distance determined so as to be shorter than a black point passing through two straight lines or a horizontal side of the frame to be detected that is parallel to the first straight line. If there are black spots that pass through a straight line, a first black spot detecting step that detects all of the black spots; a black spot that passes from the binarized image through a fourth straight line in the lateral direction passing near the center of the binarized image; The vertical side of the frame to be detected that is parallel to the fourth straight line From a black spot passing through a fifth straight line that is separated from the fourth straight line in one direction by a predetermined distance or a side in the horizontal direction of the frame that is parallel to the fourth straight line and to be detected. If there are black spots that pass through a sixth straight line separated from the fourth straight line in the other direction by a distance determined so as to be shorter, a second black spot detecting step for detecting all the black spots, and the first black spot A first combination generating step for finding all combinations of black spots passing through the second straight line detected in the detecting step and black spots passing through the third straight line detected in the first black spot detecting step; and detecting in the second black spot detecting step. A second combination generation step for obtaining all combinations of the black points passing through the fifth straight line and the black points passing through the sixth straight line detected in the second black point detection step; and the set generated in the first combination generation step. From the combination generated in the first selection step for selecting only the combination having the black point passing through the first straight line at the midpoint between the two black points forming the combination and the combination generated in the second combination generation step, From the second selection step of selecting only a combination having a black point passing through the fourth straight line at the middle point of the black point, and from the combination selected in the first selection step, all pixels between the two black points forming the combination are black. A third selection step for selecting only a combination that forms a line segment, and a combination that selects only a combination in which all pixels between the two black dots forming the combination form a black line segment from the combination selected in the second selection step. A first straight line end point detecting step for obtaining points at both ends of a line segment passing through the two points forming the combination for the combination selected in the fourth selecting step; and the fourth selecting step. For the combinations selected in (2), the second straight line end point detecting step for obtaining points at both ends of the line segment passing through the two points forming the combination, and the line segments for all pairs of the line segments related to the combination selected in the third selection step When the first frame determination process is performed in order from the longest and a frame is determined by the first frame determination process for a certain line segment pair, the frame determined by the first frame determination process for the certain line segment pair is It is checked whether or not the image is near the center of the binary image, and if so, a frame determined by the first frame determination process for the certain line segment pair is set as a finally detected frame. The second frame determination process is performed for all pairs of line segments related to the combination selected in the frame detection step and the fourth selection step in descending order of the line segments, and a pair of line segments is subjected to the second frame determination process. If the frame is fixed, the certain line segment Then, it is checked whether or not the frame determined by the second frame determination process is near the center of the binary image. If so, the frame finally determined by the second frame determination process is detected. And a second frame detection step. A frame detection method is provided.

  Further, according to the present invention, binarization means for binarizing an input grayscale image into black and white to obtain a binarized image, and a vertical path from the binarized image passing through the vicinity of the center of the binarized image. A black spot that passes through the first straight line in the direction, and is separated from the first straight line in one direction by a distance that is defined to be shorter than the horizontal side of the frame to be detected that is parallel to the first straight line. A distance from the first straight line in the other direction by a predetermined distance so as to be shorter than a black spot passing through the second straight line or a horizontal side of the frame to be detected that is parallel to the first straight line. If there are black spots that pass through the third straight line, first black spot detection means for detecting all of the black spots, and black spots that pass from the binarized image through the fourth straight line in the lateral direction passing through the vicinity of the center of the binarized image. The vertical side of the frame to be detected that is parallel to the fourth straight line Than the black point passing through the fifth straight line separated in one direction from the fourth straight line by a predetermined distance or the horizontal side of the frame to be detected that is parallel to the fourth straight line. If there are black spots that pass through a sixth straight line that is separated from the fourth straight line in the other direction by a predetermined distance, a second black spot detecting means that detects all of the black spots, and the first black spot detection First combination generating means for obtaining all combinations of black spots passing through the second straight line detected by the means and black spots passing through the third straight line detected by the first black spot detecting means, and detected by the second black spot detecting means A second combination generating means for obtaining all combinations of black spots passing through the fifth straight line and black spots passing through the sixth straight line detected by the second black spot detecting means, and combinations generated from the combinations generated by the first combination generating means. Of the two sunspots A first selection means for selecting only a combination having a black point passing through the first straight line at a point, and a combination of the two black points forming the combination through the fourth straight line from the combination generated by the second combination generation means. Second selection means for selecting only a combination with a black dot, and third selection for selecting only a combination in which all pixels between the two black dots forming the combination form a black line segment from the combination selected by the first selection means. A selection unit; a fourth selection unit that selects only a combination in which all pixels between two black dots forming the combination form a black line segment from the combinations selected by the second selection unit; and a third selection unit. For the selected combination, the first straight line end point detecting means for obtaining the points at both ends of the line segment passing through the two points forming the combination, and the both ends of the line segment passing through the two points forming the combination for the combination selected by the fourth selecting means Find the point A first frame determination process is performed for all pairs of line segments related to the combination selected by the second straight line end point detection means and the third selection means in descending order of line segments, and the first frame for a certain line segment pair. If the frame is confirmed by the confirmation process, it is checked whether or not the frame confirmed by the first frame confirmation process for the certain line segment pair is near the center of the binary image. A first frame detection unit that uses the frame determined by the first frame determination process as a frame finally detected for a pair of line segments, and all the line segments related to the combination selected by the fourth selection unit; The second frame determination process is performed in order of the longest line segment for the pair, and if the frame is determined by the second frame determination process for a pair of line segments, the second frame determination process is determined for the certain line segment. Check whether the frame is near the center of the binary image, and If the second frame detection means for the confirmed frame by the second frame confirmation process and finally detected frame, the frame detecting device characterized by comprising a are provided.

  Furthermore, according to the present invention, there is provided a frame detection program for causing a computer to execute the above frame detection method.

  In an image processing system that uses a digital image taken by a user under various conditions with a camera such as a mobile phone, only a frame is recognized from an image including unnecessary information such as a background without the assistance of the user. As a result, it is possible to extract only the area within the frame of the image. In addition, the image processing accuracy of the post-process can be improved from the information (rotation angle, enlargement / reduction ratio, distortion) of the frame.

  The best mode for carrying out the present invention will be described below in detail with reference to the drawings.

  As a method for selecting only the frame region from the image, since the frame is rectangular, the frame region is selected by finding two parallel lines from within the image. There is a possibility that two parallel lines may also enter the background other than the frame. On the other hand, the frame is selected by utilizing the fact that the object to be photographed is almost at the center.

  A specific frame acquisition method will be described.

  First, binarization that separates the frame from the background is performed. However, it is rare that a unique binarization threshold can be applied to the entire image, and it is difficult to select this value.

The reason is
This is because there is an influence of light (illumination / external light). It is because it is not known what color and shape the area other than the image frame will be. It is not difficult to determine a unique binarization threshold if the shooting environment is in a state where light is blocked, the shooting target image is black and white, or the location of the frame is constant. However, assuming a variety of environments such as magazines, posters, product packages, etc., regardless of whether the target shooting object is indoors or outdoors with a mobile phone, it is difficult to uniquely determine the binarization threshold as described above is there.

  Therefore, a binarization program was created to separate the frame from the background. This is hereinafter referred to as area-based binarization. This binarized image is intended to leave a frame and make the frame stand out from the surroundings, and may contain information (noise) such as background other than the frame.

This binarization procedure is
1. Convert a color image to a grayscale image.
2. Divide the image into smaller blocks.
3. A binarization threshold is dynamically determined for each block image by a discriminant analysis method or the like.
4). If there is a threshold value larger than itself compared with the threshold value around the block, it is brought close to that value.
5. The image is converted into binary data of white and black using a threshold value for each block.
It is.

  The division size of 2 is too small or too large. If it is too small, the frame fills the block and an appropriate threshold cannot be obtained. If it is too large, many pixels are included in the block, so the meaning of increasing the binarization accuracy by dividing the block is diminished. Therefore, it is necessary to select an appropriate value depending on the image size and the size of the frame to be imaged (see FIG. 5).

  In 4, the threshold is raised using information around the block. There are two reasons for doing this. One is to make the boundary of the block inconspicuous when binarized by bringing the threshold close to the surroundings. Second, the larger the threshold, the more frame areas remain. In this embodiment, an average of its own threshold value and a value larger than that is used as compared with the threshold value of 3 × 3 blocks around a certain block (see FIG. 6).

  Next, a block image (noise) in which a frame does not surely exist for each block is removed. This processing is performed in order to shorten the subsequent processing time and increase the accuracy, and is not essential.

  Here are three methods for discriminating blocks that do not have a frame. First, it can be determined that there is no frame if the difference between the maximum and minimum pixel values in the block is not more than a certain value. The reason for this is that if a frame and background exist in the block, the pixel value of one point in the frame is selected as the minimum value, and the pixel value of one point in the background is selected as the maximum value. Become. Therefore, if the difference is small, it can be determined that there is no frame in the block. Second, if black pixels in a block are equal to or greater than a certain ratio, it can be determined that there is no frame. This is because if the thickness of the frame is small relative to the block size, the entire block in which the frame exists cannot be black. These blocks are painted white, assuming that there is no frame, that is, a background portion. Third, it can be said that a block having a large distance from the center has no frame. For this distance, when a frame is arranged at the center and the frame is copied to the full camera preview screen, a block of a distance not including the frame is removed (see FIG. 7).

  Next, lines that are candidates for the sides of the frame are extracted from the binary image.

  As the premise of the image, the upper left of the image is (0, 0), the right direction is the positive direction of x, and the lower direction is the positive direction of y.

  First, the center coordinates are set to (CX, CY), and x = CX is scanned in the y-axis direction. If a black pixel is found, save this coordinate. In the case of consecutive black pixels, the middle point of those pixels is saved. Similarly, scanning of x = CX ± A (A is an integer) is performed. The value A is sufficiently smaller than the length of the short side of the frame.

  For all combinations of the point x = CX + A and the point x = CX−A, the combinations in which the midpoint between the two points exist at the point x = CX are listed. There is a possibility that a line connecting these two points is a part of the frame line.

  The image is scanned in the y-axis direction. Next, the same operation is performed for y = CY, y = CY ± B (B is an integer) in the x-axis direction, and a combination of points is listed. When shooting in the normal position (when the image is not rotated), it is often the case that shooting is performed with the shooting target slightly below the center, so the CY value should be corrected somewhat below the center. Is also possible.

  Next, it is checked whether a black pixel line exists between the two points listed above. At this time, if the line is interrupted, it is excluded. If it becomes a straight line, investigate how far this line extends. By these processes, the points that are the ends of the straight line and the lengths between them are listed.

  By the processing so far, several line segments as a result of scanning in the y-axis direction and the x-axis direction have been acquired. Some of these line segments are the sides of the frame, but unrelated line segments included in the background are also included. In addition, when a plurality of frames are arranged in the image, the sides of the frames other than the frame to be searched are also included. Therefore, it is necessary to select what becomes a frame line from these line segments.

  It should be noted here that the above candidates may not include all four sides of the frame. When shooting without a guide, it is difficult to shoot with the subject at the center. This is especially true when the edges of the frame are aligned. However, even if it is difficult to acquire all four, it is possible to acquire one opposite side. For example, even if one side is short, it is possible to capture the center axis of X or Y at the longer side, and it is difficult to center the object vertically but to capture the center at the side. It is done unconsciously. Therefore, there is a high possibility that one of the opposite sides can be acquired by the above method with a normal captured image. In FIG. 8A, the x-axis is not parallel, but the y-axis is properly captured. On the other hand, in FIG. 8B, the y-axis is out of the frame depending on the value of A of x = CX + A to the extent that the center is framed, but the x-axis captures the frame properly. Thus, if the frame is not too small, there is a high possibility that a pair of opposing sides can be obtained even if the four sides are impossible by the conventional methods.

  In the processing from now on, a line segment that is a pair of opposite sides of the frame is searched for from the line segments listed in the X-axis direction and the Y-axis direction.

  The obtained line segment is divided into a line segment in the X-axis direction and a line segment in the Y-axis direction, and two long lines are combined for each. The reason for combining from long lines is that the shorter the line segment, the more likely it is noise from the background or characters in the frame. Also, near-rectangular characters such as “□” at the center of the frame (“Kuchi” or “RO” in the reading) may be mistaken for a frame, but if a long line is first, the frame is better. Since the order comes first, the frame can be specified first. As another line segment combination order, there is a method of first selecting line segments having the same length and then combining long sides.

Next, check whether these two line segments are opposite sides. This is done by checking whether the line segments are parallel . Therefore, the rotation angle of each side with respect to the reference line is examined to see if this value is close. Since it is a line segment taken from the image, the angles do not completely coincide with each other. Therefore, it is sufficient to omit what can be said to be absolutely non-parallel here. Also, it is checked that these two line segments are not the same line segment. This is done by extracting one point from one line segment, which is determined by the distance from the other line segment or its extension.

  If a pair of sides is found, it is checked whether there are two sides that intersect perpendicularly with the two sides. If not found, the pair of edges is not part of the frame.

  The sides facing each other are referred to as sides 1 and 2. The end points of side 1 are called points 1 and 2, and the end points of side 2 are called points 3 and 4. Points 1 and 3 and points 2 and 4 correspond to each other (points connected by a frame line). Therefore, points 1 and 4 and points 2 and 3 are diagonal points.

The method for finding a new edge is as follows.
(1). Check if there is a line between point 1 and point 3 and between point 2 and point 4.
(2). If both lines are found in (1), check if the four lines are rectangular.
(3a). If one line is found in (1), it is checked whether there is a line in the normal direction of the side from one of the two points on the side that was not found. If not found, the same processing is performed for the other point.
(3b). If a line is found, it is checked whether the four form a rectangle, and if not found, it is determined that the pair of side 1 and side 2 is not part of the frame.
(4a). If a line is not found in (1), it is checked whether there is a line in the normal direction of either side for each of the points 1 and 3. The same is done for points 2 and 4.
(4b). As a result of the above, if both lines are found, check whether they become rectangular by four. Otherwise, it is determined that the pair of side 1 and side 2 is not part of the frame.

  The processing of (3) and (4) is not necessary if the end points of the original two sides can be acquired properly. However, in practice, the line may be interrupted during the line search. Therefore, if either one of the end points is taken normally, the processes (3) and (4) are performed so that the side can be found.

  The frame obtained as a result of the processing so far should basically be a rectangle (or a shape close to a rectangle). However, just in case, you can check if this is a rectangle, and if it is not a rectangle, it can be a frame. There are two possible conditions for examining the rectangle: four conditions for the four intersections. The first is the length of points 1 and 2, and the length of points 3 and 4. It is examined whether the lengths of the points 1 and 3 are substantially equal to the lengths of the points 2 and 4. Second, it is only necessary that the two pairs of sides facing each other are almost parallel.

  Next, it is checked whether this frame is in the center. If it is not the center, this frame may be a frame other than the object that was mixed in at the time of shooting, or noise such as the background. In this discrimination method, since the object to be photographed is basically in the middle, the distance between the center of the image and the center of the frame is determined to be less than a certain value, and the center of the image is included in the frame area. There is a case.

  If it is off the center, this is stored as a frame candidate closest to the frame together with the distance between the center of the image and the center of the frame. Then, candidates for other sides are examined, and if there is no corresponding frame, this is the correct answer. Similarly, when a frame out of the center is found, the distance from this and the center of the frame candidate is obtained, and the smaller value is set as the frame candidate.

  Finally, correction processing is performed to take the outermost point of the frame. This is because the vertex area is wide because the line is thick, and the vertex coordinates of the frame may be different even if the frame in the same state is photographed twice. In order to make this take almost the same vertex, the outermost vertex of the frame is taken.

  This method looks at the two sides that touch each point, and determines the direction of vertex movement from here. See also FIG. 10 for a description of the direction of movement. The frame is expanded toward the direction of the vector obtained by adding the vectors in the vertex direction of each side, and this is represented by <+, 0, −> of the x and y components. In the conversion, as shown in FIG. 9, the previous vector direction is represented by the positive and negative directions of the x and y components. If the current vertex is (X, Y) and the vector is x + and y is-, it is checked whether (X + 1, Y) is a frame for the x component, and then (X, Y-) for the y component. Check 1). If either is a frame, move the vertex coordinates to that coordinate. The same processing is performed for the vertex coordinates until the coordinates stop moving. Do this for all vertices.

  Thereby, the vertex coordinates of the four points of the frame in the image can be specified.

  Next, the frame detection method according to the embodiment of the present invention will be described with reference to the flowcharts shown in FIGS.

  Referring to FIG. 1, first, binarization for each area is performed on a grayscale image to obtain a binarized image (step S101). Details of step S101 will be described later.

  Next, if there are black spots that pass on the straight line x = CX, black spots that pass on the straight line x = CX + A, or black spots that pass on the straight line x = CX-A, they are detected (step S103). Here, CX is a coordinate near the center in the X direction of the image, and A is a value determined to be shorter than the short side of the frame in a normal captured image.

  Next, if there are black spots that pass on the straight line y = CY, black spots that pass on the straight line y = CY + B, or black spots that pass on the straight line y = CY-B, these are detected (step S105). Here, CY is a coordinate near the center of the image in the Y direction, and B is a value determined so as to be shorter than the short side of the frame in a normal captured image.

  Next, all combinations of black spots passing through the line x = CX-A detected in step S103 and black spots passing through the line x = CX-A are obtained (step S107). For example, if the number of black spots passing through the straight line x = CX-A is 3, and the number of black spots passing through the straight line x = CX-A is 4, 3 × 4 = 12 combinations are obtained.

  Next, all combinations of black spots passing through the y = CY-B straight line detected at step S105 and black spots passing through the y = CY-B straight line are obtained (step S109). For example, if the number of black spots passing on the line y = CY-B is 2 and the number of black spots passing on the line y = CY-B is 5, 2 × 5 = 10 combinations are obtained.

  Next, of all the combinations obtained in step S107, only the combination having a black point on x = CX at the midpoint of the two points forming the combination is selected (step S111).

  Next, out of all the combinations obtained in step S109, only the combination having a black point on y = CY at the midpoint between the two points forming the combination is selected (step S113).

  Next, of all the combinations selected in step S111, only the combination in which all the pixels between the two points forming the combination form a black line segment is selected (step S115).

  Next, of all the combinations selected in step S113, only the combination in which all the pixels between the two points forming the combination form a black line segment is selected (step S116).

  Next, for all combinations selected in step S115, points at both ends of a straight line passing through the two points forming the combination are obtained (step S117).

  Next, for all combinations selected in step S116, points at both ends of a straight line passing through the two points forming the combination are obtained (step S118).

  Referring to FIG. 2, next, the processes from step S119S to step S119E are repeated in the order of longer line segments for all combinations of the line segments related to the combination selected in step S115.

  In each repetition, first, a frame determination process is performed (step S121). Details of the frame determination process will be described later.

  Next, it is determined whether or not the frame has been confirmed in the frame confirmation process (step S123). If not confirmed, the process proceeds to the next iteration. If confirmed (YES in step S123), it is determined whether or not the frame is near the center of the image (step S125).

  If the frame is near the center of the image (YES in step S125), the process proceeds to step S139. In other words, the process repeats between step S119S and step S119E.

  If the frame is not near the center of the image (NO in step S125), the frame determined in step S121 is stored as a frame candidate (step S127). At this time, the distance between the center of the frame and the center of the image is measured, and this distance is also stored in association with the frame candidate. When step S127 ends, the process proceeds to the next iteration.

  When the repetition from step S119S to step S119E is completed, the processing from step S129S to step S129E is repeated in the order of long line segment for all combinations of the line segments related to the combination selected in step S116.

  In each repetition, first, a frame determination process similar to step S121 is performed (step S131).

  Next, it is determined whether or not the frame has been confirmed in the frame confirmation process (step S133). If not confirmed, the process proceeds to the next iteration. If determined (YES in step S133), it is determined whether or not the frame is near the center of the image (step S135).

  If the frame is near the center of the image (YES in step S135), the process proceeds to step S139. That is, the process repeats between step S129S and step S129E.

  If the frame is not near the center of the image (NO in step S135), the frame determined in step S131 is stored as a frame candidate (step S137). At this time, the distance between the center of the frame and the center of the image is measured, and this distance is also stored in association with the frame candidate. When step S137 ends, the process proceeds to the next iteration.

  If the repetition from step S129S to step S129E is completed, it is checked whether or not there is a frame determined in the frame determination process (step S121 or step S131) (step S139).

  If there is no fixed frame (NO in step S139), error processing is performed (step S141), and the operation is terminated.

  If there is a confirmed frame (YES in step S139), among the confirmed frames, the frame having the shortest distance between the center of the frame and the center of the binary image is determined as the finally detected frame (step S137). ), The process proceeds to step S139.

  In step S139, the frame is expanded and the vertex coordinates are specified. That is, although the frame line has a width, the coordinates of the outermost four corners of the frame are specified.

  Next, details of the binarization step by area (step S101) will be described with reference to FIG.

  Referring to FIG. 3, first, a color image is subjected to gray scale conversion to obtain a gray scale image (step S101-1).

  Next, the grayscale image is divided into, for example, blocks of horizontal 20 pixels × vertical 20 pixels (step S101-3).

  Next, a temporary threshold for binarization is determined for each block (step S101-5). Here, for example, a threshold value selection method by Otsu is used to obtain a threshold value for each block, which is set as a provisional threshold value.

  Next, the threshold value for binarization is corrected (step S191-7). Here, for each block, the average value of the tentative thresholds, which is the sum of the tentative threshold values of the neighboring blocks and the tentative threshold value of the own block, among the tentative threshold values of the neighboring blocks, The threshold is used.

  Next, the grayscale image is converted into a binarized image for each block using the threshold value obtained in step S101-7 (step S101-9).

  Next, details of the steps (steps S121 and S131) of the frame determination process will be described with reference to FIG. Here, the left endpoint of one line is point 1, the right endpoint is point 2, the left endpoint of the other line is point 3, and the right endpoint is point 4.

  Referring to FIG. 4, first, it is examined whether or not an edge exists between the points 1 and 3 (step S121-1). Here, if all the pixels between the points 1 and 3 are black, it is determined that there is an edge.

  Next, it is examined whether or not an edge exists between the points 2 and 4 (step S121-3). Here, if all the pixels between the points 2 and 4 are black, it is determined that there is an edge.

  Next, it is determined whether or not an edge has been found in steps S121-1 and S121-3 (step S121-5). If the determination result of step S121-5 is affirmative (if two sides are found), the process proceeds to step S121-7.

  If the determination result in step S121-5 is negative, it is determined whether an edge is found in one of steps S121-1 and S121-3 (step S121-11). If the determination result in step S121-11 is affirmative (if one edge is found), the process proceeds to step S121-13. If the determination result in step S121-11 is negative (if no edge is found), the process proceeds to step S121-21.

  In step S121-13, it is checked whether or not there is a line segment that passes through the end point of one line segment and extends in the normal direction of the line on the side where no side is found.

  Next, it is determined whether or not a line segment satisfying the requirements in step S121-13 has been found (step S121-15). If found (YES in step S121-15), the process proceeds to step S121-7.

  If not found (NO in step S121-15), the side where no side is found passes through the end point of the other line segment (the end point of the line not checked in step S121-13), and the normal direction of the line It is checked whether or not there is a line segment extending (step S121-17).

  Next, it is determined whether or not a line segment satisfying the requirements in step S121-17 has been found (step S121-19). If found (YES in step S121-19), the process proceeds to step S121-7.

  If not found (NO in step S121-19), the process proceeds to step S121-37.

  In step S121-21, about one end point (for example, point 1) among the four end points (point 1, point 2, point 3, point 4), the normal direction of the line passing through the end point and having the end point Check to see if there is a line segment that extends.

  In step S121-23, it is determined whether or not a line segment satisfying the requirement in step S121-21 is found. If found (YES in step S121-23), the process proceeds to step S121-25.

  If not found (NO in step S121-23), whether there is a line extending through the end point (for example, point 3) facing the end point examined in step S121-21 and extending in the normal direction of the line having the end point (Step S121-27).

  In step S121-29, it is determined whether a line segment satisfying the requirements in step S121-27 has been found. If found (YES in step S121-29), the process proceeds to step S121-25.

  If not found (NO in step S121-29), the process proceeds to step S121-37.

  In step S121-25, it is checked whether there is a line segment that passes through the other end point (eg, point 2) having the end point checked in step S121-21 and extends in the normal direction of the line having the end point. .

  In step S121-31, it is determined whether a line segment satisfying the requirement in step S121-25 is found. If found (YES in step S121-31), the process proceeds to step S121-7.

  If not found (NO in step S121-31), whether there is a line segment that passes through a point (for example, point 4) facing the end point examined in step S121-25 and extends in the normal direction of the line having the end point. (Step S121-33).

  In step S121-35, it is determined whether a line segment satisfying the condition of step S121-33 has been found. If found (YES in step S121-35), the process proceeds to step S121-7.

  If not found (NO in step S121-35), the process proceeds to step S121-37.

  In step S121-7, it is checked whether or not the four sides found form a rectangle. Here, if the line segment is not connected, the line segment is extended until it is connected to form a side.

  When the found four sides form a rectangle (YES in step S121-7), the found four sides are determined as the four sides constituting the frame (step S121-9). That is, it is determined that a frame has been found.

  If the found four sides do not form a rectangle (NO in step S121-7), the process proceeds to step S121-37.

  In step S121-37, it is determined that the opposing line segment is not part of the frame.

  The above-described method can be realized only by hardware. However, the method described above can also be realized by a computer. In this case, the above-described method can be realized by causing the computer to read and execute a program for causing the computer to function as a functional unit that performs each step.

  FIG. 5 shows how the binarized image changes depending on the block size in the process of making a grayscale image into blocks and then binarizing it to obtain a binarized image.

  (A)-(e) shows the example in case the thickness of a frame is 3 points | pieces and the image | photographed image is not rotating. (F)-(j) shows the example in case the thickness of a frame is 6 points | pieces and the image | photographed image is rotating. (A), (f) shows the binarized image when the block size is 12 horizontal pixels × 12 vertical pixels, and (b), (g) are two images when the block size is 20 horizontal pixels × 20 vertical pixels. (C) and (h) show binarized images when the block size is horizontal 32 pixels × vertical 32 pixels, and (d) and (i) show block sizes of horizontal 48 pixels × vertical. The binarized image in the case of 48 pixels is shown, and (e) and (j) show the binarized image when not divided into blocks.

  Experiments have shown that the frame is most easily detected when the block size is 20 pixels wide × 20 pixels vertical.

  FIG. 6 shows how the binarized image changes depending on the threshold value in the process of blocking the grayscale image and binarizing it to obtain the binarized image.

  (A)-(c) shows the example in case the thickness of a frame is 3 points | pieces and the image | photographed image is not rotating. (D)-(f) shows the example in case the thickness of a frame is 6 points | pieces and the imaged image is rotating. The block size is 20 horizontal pixels × 20 vertical pixels. (A), (d) shows the binarized image when the temporary threshold described in the embodiment is used as it is, and (b), (g) are the temporary threshold of the own block and the upper, lower, left, and right temporary thresholds. (B) shows the binarized image when the average value of the five provisional thresholds is used as the final threshold in combination, (c) and (f) are the provisional threshold of the own block and the provisional threshold of the surrounding eight blocks The binarized image in the case where the average value of the temporary thresholds combined with the lower level than the temporary threshold value of the own block is used as the final threshold value is shown.

  In the experiment, the frame is most easily detected when the average value of the temporary thresholds combined with the low threshold value of the temporary threshold value of the own block and the temporary threshold values of the surrounding 8 blocks is used as the final threshold value. The result is out.

  FIG. 7 shows a binarized image after noise removal when the provisional threshold described in the embodiment is used as it is, and the provisional threshold of the own block and the surrounding eight blocks based on the provisional threshold described in the embodiment. The binarized image after noise removal in the case where the average value of the temporary thresholds combined with the temporary threshold value of the current block, which is lower than the temporary threshold value of the own block, is used as the final threshold value.

  FIG. 8A shows an example in which a frame can be detected because three lines in the Y-axis direction can catch the longer line.

  FIG. 8B shows an example in which a frame can be detected because three lines in the X-axis direction can catch the longer line.

  FIG. 9 shows a direction in which the frame is expanded in step S139 of the embodiment. The outer angle of the two intersecting sides is 270 degrees, but a direction of 135 degrees, which is a half of this angle, is the direction in which the frame is expanded. However, since the pixels are arranged in a grid, the frame cannot be expanded in the direction in which the frame is to be expanded with a small number of pixels included in the frame. Therefore, as shown in FIG. 9, the direction in which the frame is to be expanded is divided into eight, and the direction in which the frame is actually expanded is determined for each division direction.

  For example, for the range (a) in the direction in which it is desired to expand the frame of plus or minus 15 degrees around the plus direction of the x axis, the direction in which the frame is actually expanded is set to zero degrees. Further, for the range (c) in the direction in which the frame of +/- 15 degrees is desired to be expanded around the positive direction of the y axis, the direction in which the frame is actually expanded is set to -90 degrees. Furthermore, with respect to the range (e) in the direction in which the frame of +/- 15 degrees is desired to be expanded about the minus direction of the x axis, the direction in which the frame is actually expanded is set to 180 degrees. Further, with respect to the range (g) in the direction in which the frame of +/- 15 degrees is desired to be expanded around the negative direction of the y axis, the direction in which the frame is actually expanded is set to 90 degrees. Further, for the range (b) sandwiched between the range (a) and the range (c), the direction in which the frame is actually expanded is set to −45 degrees. Furthermore, for the range (d) sandwiched between the range (c) and the range (e), the direction in which the frame is actually expanded is set to -135 degrees. Further, for the range (f) sandwiched between the range (e) and the range (g), the direction in which the frame is actually expanded is set to 135 degrees. Further, for the range (b) sandwiched between the range (a) and the range (g), the direction in which the frame is actually expanded is set to 45 degrees.

  FIG. 10A shows an example in which the direction in which the frame is to be expanded and the direction in which the frame is actually expanded are both 135 degrees. FIG. 10B shows an example in which the direction in which the frame is to be expanded and the direction in which the frame is actually expanded are both 90 degrees.

It is a flowchart (1/2) which shows the whole frame detection method by embodiment of this invention. It is a flowchart (2/2) which shows the whole frame detection method by embodiment of this invention. It is a flowchart which shows the detail of the binarization classified by area shown in FIG. It is a flowchart which shows the detail of the frame determination process shown in FIG. It is a figure which shows how a binarized image changes with block sizes in the process which blocks a grayscale image and binarizes it and obtains a binarized image on it. It is a figure which shows how a binarized image changes with a threshold value in the process which blocks a gray scale image and binarizes and obtains a binarized image on it. (C) shows the binarized image after noise removal when the provisional threshold described in the embodiment of the present invention is used as it is, and (f) shows the provisional threshold described in the embodiment of the present invention. Based on the provisional threshold of the current block and the provisional threshold of the surrounding eight blocks, the average value of the provisional threshold combined with the lower threshold than the provisional threshold of the own block is used as the final threshold. The binarized image after removal is shown. (A), (d) shows an original image. (B) shows an image immediately after binarization when the provisional threshold described in the embodiment of the present invention is used as it is, and (e) shows the provisional threshold described in the embodiment of the present invention. In addition, when the average value of the tentative thresholds combined with the tentative threshold value of the own block and the tentative threshold values of the surrounding 8 blocks that are lower than the tentative threshold value of the own block is used as the final threshold value, An image immediately after conversion is shown. (A) shows an example in which a frame can be detected because three lines in the Y-axis direction can capture the longer line. (B) shows an example in which a frame can be detected because three lines in the X-axis direction can catch the longer line. In step S139 of the embodiment of the present invention, the direction in which the frame is expanded is shown. (A) shows an example in which the direction in which the frame is to be expanded and the direction in which the frame is actually expanded are both 135 degrees. (B) shows an example in which the direction in which the frame is to be expanded and the direction in which the frame is actually expanded are both 90 degrees.

Claims (17)

A binarization step of binarizing the input grayscale image into black and white to obtain a binarized image;
From the binarized image, a black spot that passes through the first straight line in the vertical direction passing near the center of the binarized image, and is shorter than the horizontal side of the frame to be detected that is parallel to the first straight line. In this way, the black point passing through the second straight line separated from the first straight line in one direction by a predetermined distance or parallel to the first straight line is shorter than the lateral side of the frame to be detected. If there are black spots that pass through a third straight line that is separated from the first straight line in the other direction by a distance determined in (1), a first black spot detecting step that detects all of the black spots;
From the binarized image, a black spot that passes through the fourth straight line in the horizontal direction passing near the center of the binarized image, and is shorter than the vertical side of the frame to be detected that is parallel to the fourth straight line. In such a manner, the black point passing through the fifth straight line separated from the fourth straight line in one direction by a predetermined distance or the horizontal side of the frame to be detected that is parallel to the fourth straight line is shorter. If there are black spots that pass through a sixth straight line that is separated from the fourth straight line in the other direction by a distance determined in (2), a second black spot detecting step that detects all of the black spots;
A first combination generating step for obtaining all combinations of black spots passing through the second straight line detected in the first black spot detecting step and black spots passing through the third straight line detected in the first black spot detecting step;
A second combination generating step for obtaining all combinations of black spots passing through the fifth straight line detected in the second black spot detecting step and black spots passing through the sixth straight line detected in the second black spot detecting step;
A first selection step of selecting, from the combinations generated in the first combination generation step, only a combination having a black point passing through the first straight line at a midpoint between two black points forming the combination;
A second selection step of selecting, from the combinations generated in the second combination generation step, only a combination having a black point passing through the fourth straight line at a midpoint between two black points forming the combination;
A third selection step of selecting only a combination in which all pixels between two black dots forming the combination form a black line segment from the combinations selected in the first selection step;
A fourth selection step for selecting, from the combinations selected in the second selection step, only a combination in which all pixels between two black dots forming the combination form a black line segment;
For the combination selected in the third selection step, a first straight line end point detection step for obtaining points at both ends of the line segment passing through the two points forming the combination;
For the combination selected in the fourth selection step, a second straight line end point detecting step for obtaining points at both ends of the line segment passing through the two points forming the combination;
If the first frame determination process is performed for all pairs of line segments related to the combination selected in the third selection step in descending order of the line segments, and the frame is determined by the first frame determination process for a certain line segment pair For example, it is checked whether or not the frame determined by the first frame determination process for the certain line segment pair is near the center of the binary image. A first frame detection step in which the frame determined by the first frame determination process is the frame finally detected;
If the second frame determination process is performed for all pairs of line segments related to the combination selected in the fourth selection step in descending order of the line segments, and the frame is determined by the second frame determination process for a certain line segment pair For example, it is checked whether or not the frame determined by the second frame determination process for the certain line segment is near the center of the binary image. If so, the frame determined by the second frame determination process is determined. A second frame detection step as a finally detected frame;
A frame detection method comprising: In the first frame detection step, if it is determined that the frame determined by the first frame determination process is not near the center of the binary image, the center of the frame determined by the first frame determination process and the second frame are determined. A first frame candidate storing step of measuring a distance between the center of the value image and storing the frame determined by the first frame determining process and the measured distance;
In the second frame detection step, if it is determined that the frame determined by the second frame determination process is not at the center of the binary image, the center of the frame determined by the second frame determination process and the binary value are determined. A second frame candidate storing step of measuring the distance between the center of the image and storing the frame determined by the second frame determining process and the measured distance;
Of the frames stored in the first frame candidate storage step and the frames stored in the second frame candidate storage step, the frame having the shortest distance between the center of the frame and the center of the binary image is finally determined. A third frame detecting step to detect the detected frame;
The frame detection method according to claim 1, further comprising: The frame determination process includes
A first combined edge existence confirmation step for confirming whether or not an edge exists between one end point of one of the target sides and one end point of the other side of the target;
A second combined edge existence confirmation step for confirming whether or not there is an edge between the other end point of the one side being the target and the other end point of the other side being the target ;
If the confirmation results of both the first combined edge existence confirmation step and the second combined edge existence confirmation step are affirmative, the one side that is the target and the other side that is the target Confirming whether or not the side confirmed to exist in the first coupled-side existence confirmation step and the side confirmed to exist in the second coupled-side existence confirmation step constitute a rectangle. 1 rectangular configuration confirmation step;
If the confirmation result in the first rectangular configuration confirmation step is affirmative, the one side that is the target, the other side that is the target, and the first combined side presence confirmation step exist A first confirmation step for confirming that the side confirmed to be present and the side confirmed to be present in the second combined edge existence confirmation step constitute a frame;
The frame detection method according to claim 1, further comprising: If only one confirmation result of the first coupled edge existence confirmation step and the second coupled edge existence confirmation step is affirmative, the end point of the one side of the target and the other side of the target A line extending in the normal direction of one of the sides passing through an end point where no side exists between or the end point of one side of the target of the other side of the target A first line existence confirmation step for confirming whether or not there is a line extending in the normal direction of the other side of the object through an end point where no side exists between
If the confirmation result of the first line presence confirmation step is affirmative, the one side that is the target, the other side that is the target, and the first combined side presence confirmation step or the first Confirm whether or not the side confirmed to exist in the 2-joint side existence confirmation step and the line segment expanded from the line segment confirmed to exist in the first line existence confirmation step constitute a rectangle. A second rectangular configuration confirmation step,
If the confirmation result in the second rectangular configuration confirmation step is affirmative, the one side that is the target, the other side that is the target, and the first combined side existence confirmation step or the The second determined that the side confirmed to exist in the second connected side existence confirmation step and the line segment expanded from the line segment confirmed to exist in the first line existence confirmation step constitute a rectangle. A confirmation step;
The frame detection method according to claim 3, further comprising: If the confirmation results of both the first combined edge existence confirmation step and the second combined edge existence confirmation step are negative, one of the targeted edges and one of the other targeted edges A line that passes through one end point that does not exist between the end points and extends in the normal direction of the target side, or the target side of the target side A second line existence confirmation step for confirming whether or not there is a line passing through one end point where no side exists between one end point of the side and extending in the normal direction of the other side of interest; ,
If the confirmation results of both the first combined edge existence confirmation step and the second combined edge existence confirmation step are negative, the one side of the target is the other of the other sides of the target A line extending in the normal direction of one side of the target passing through the other end point where no side exists between the end points, or one of the target of the other side of the target A third line existence confirmation step for confirming whether or not there is a line passing through the other end point where no side exists between the other end point of the side and extending in the normal direction of the other side of interest; ,
If both the confirmation result of the second line existence confirmation step and the confirmation result of the third line existence confirmation step are affirmative, the one side that is the target and the other side that is the target A line segment expanded from the line segment confirmed to exist in the second line existence confirmation step, and a line segment expanded from the line segment confirmed to exist in the third line existence confirmation step. A third rectangular composition confirmation step for confirming whether or not to constitute a rectangle;
If the confirmation result in the third rectangular configuration confirmation step is affirmative, the one side that is the target, the other side that is the target, and the second line existence confirmation step exist. A third determination step for determining that the line segment expanded from the line segment confirmed to be present and the line segment expanded from the line segment confirmed to exist in the third line presence confirmation step constitute a rectangle;
The frame detection method according to claim 3 or 4, further comprising: The frame detection method according to claim 1, further comprising a conversion step of converting the input color image into the grayscale image. 7. The binarizing step divides the input grayscale image into a plurality of areas, and binarizes the image into black and white using a threshold value for each area for each area. The frame detection method according to claim 1. The threshold value is an average value of a temporary threshold value obtained for the own area and an average value of the temporary threshold values of the eight areas around the own area that are lower than the temporary threshold value obtained for the own area. The frame detection method according to claim 7. Binarization means for binarizing the input grayscale image into black and white to obtain a binarized image;
From the binarized image, a black spot that passes through the first straight line in the vertical direction passing near the center of the binarized image, and is shorter than the horizontal side of the frame to be detected that is parallel to the first straight line. In this way, the black point passing through the second straight line separated from the first straight line in one direction by a predetermined distance or parallel to the first straight line is shorter than the lateral side of the frame to be detected. If there are black spots that pass through a third straight line that is separated from the first straight line in the other direction by a distance determined in (1), first black spot detecting means for detecting all of the black spots;
From the binarized image, a black spot that passes through the fourth straight line in the horizontal direction passing near the center of the binarized image, and is shorter than the vertical side of the frame to be detected that is parallel to the fourth straight line. In such a manner, the black point passing through the fifth straight line separated from the fourth straight line in one direction by a predetermined distance or the horizontal side of the frame to be detected that is parallel to the fourth straight line is shorter. If there are black spots that pass through a sixth straight line that is separated from the fourth straight line in the other direction by a distance determined in (2), a second black spot detecting means that detects all the black spots;
First combination generating means for obtaining all combinations of black spots passing through the second straight line detected by the first black spot detecting means and black spots passing through the third straight line detected by the first black spot detecting means;
Second combination generating means for obtaining all combinations of black spots passing through the fifth straight line detected by the second black spot detecting means and black spots passing through the sixth straight line detected by the second black spot detecting means;
First selection means for selecting, from the combinations generated by the first combination generation means, only a combination having a black point passing through the first straight line at a midpoint between two black points forming the combination;
Second selection means for selecting, from the combinations generated by the second combination generation means, only a combination having a black point passing through the fourth straight line at a midpoint between two black points forming the combination;
Third selection means for selecting, from the combinations selected by the first selection means, only a combination in which all pixels between the two black dots forming the combination form a black line segment;
Fourth selection means for selecting only a combination in which all pixels between two black dots forming the combination form a black line segment from the combinations selected by the second selection means;
For the combination selected by the third selection means, first linear end point detection means for obtaining points at both ends of the line segment passing through the two points forming the combination;
For the combination selected by the fourth selection means, second linear end point detection means for obtaining points at both ends of the line segment passing through the two points forming the combination;
If the first frame determination process is performed for all pairs of line segments related to the combination selected by the third selection unit in the order of the longest line segment, and the frame is fixed by the first frame determination process for a certain line segment pair For example, it is checked whether or not the frame determined by the first frame determination process for the certain line segment pair is near the center of the binary image. A first frame detection means that uses the frame determined by the first frame determination process as the finally detected frame;
If the second frame determination process is performed in order of the longest line segment for all pairs of line segments related to the combination selected by the fourth selection means, and the frame is determined by the second frame determination process for a certain line segment pair For example, it is checked whether or not the frame determined by the second frame determination process for the certain line segment is near the center of the binary image. If so, the frame determined by the second frame determination process is determined. A second frame detecting means as a finally detected frame;
A frame detection apparatus comprising: If it is determined by the first frame detection means that the frame determined by the first frame determination process is not near the center of the binary image, the center of the frame determined by the first frame determination process and the second frame are determined. First frame candidate storage means for measuring the distance between the center of the value image and storing the frame determined by the first frame determination process and the measured distance;
If the second frame detection means determines that the frame determined by the second frame determination process is not at the center of the binary image, the center of the frame determined by the second frame determination process and the binary value are determined. Second frame candidate storing means for measuring the distance between the center of the image and storing the frame determined by the second frame determination process and the measured distance;
Of the frames stored by the first frame candidate storage unit and the frames stored by the second frame candidate storage unit, the frame having the shortest distance between the center of the frame and the center of the binary image is finally determined. A third frame detecting means for detecting the detected frame;
The frame detection device according to claim 9, further comprising: The means for performing the frame determination process is:
A first combined edge existence confirmation means for confirming whether or not an edge exists between one end point of one of the target sides and one end point of the other side of the target;
A second combined edge existence confirmation means for confirming whether or not an edge exists between the other end point of the one side that is the target and the other end point of the other side that is the target ;
If the confirmation results of both the first combined edge existence confirmation unit and the second combined edge existence confirmation unit are affirmative, the one side that is the target and the other side that is the target First, it is confirmed whether or not the side confirmed by the first coupled-side presence confirmation unit and the side confirmed by the second coupled-side presence verification unit form a rectangle. 1 rectangular configuration confirmation means;
If the confirmation result by the first rectangular configuration confirmation means is affirmative, the one side as the object, the other side as the object, and the first combined edge existence confirmation means exist. First confirmation means for confirming that the side confirmed to be present and the side confirmed to be present by the second combined edge existence confirmation means constitute a frame;
The frame detection device according to claim 9, further comprising: If only one of the confirmation results of the first coupled edge existence confirming means and the second coupled edge existence confirming means is affirmative, the end point of the targeted one edge of the other edge A line extending in the normal direction of one of the sides passing through an end point where no side exists between or the end point of one side of the target of the other side of the target A first line existence confirmation means for confirming whether or not there is a line extending in the normal direction of the other side of the object passing through an end point where no side exists between
If the confirmation result of the first line existence confirmation unit is affirmative, the one side that is the target, the other side that is the target, the first combined side presence confirmation unit, or the first Confirm whether or not the side confirmed by the two-joint side existence confirmation unit and the line segment expanded from the line segment confirmed to exist by the first line existence confirmation unit form a rectangle. Second rectangular configuration confirmation means for
If the confirmation result by the second rectangular configuration confirmation unit is affirmative, the one side that is the target, the other side that is the target, and the first combined side presence confirmation unit or the The second determined that the side confirmed by the second combined side presence checking means and the line segment expanded from the line checked by the first line presence checking means form a rectangle. A confirmation means;
The frame detection device according to claim 11, further comprising: If the confirmation result of both the first combined edge existence confirming means and the second combined edge existence confirming means is negative, one of the targeted edges and the other of the targeted edges A line that passes through one end point that does not exist between the end points and extends in the normal direction of the target side, or the target side of the target side Second line existence confirmation means for confirming whether or not there is a line passing through one end point where no side exists between one end point of the side and extending in the normal direction of the other side of interest; ,
If the confirmation result of both the first combined edge existence confirming means and the second combined edge existence confirming means is negative, the one of the targeted edges and the other of the targeted edges A line extending in the normal direction of one side of the target passing through the other end point where no side exists between the end points, or one of the target of the other side of the target Third line existence confirmation means for confirming whether or not there is a line passing through the other end point where no side exists between the other end point of the side and extending in the normal direction of the other side of interest; ,
If both the confirmation result of the second line existence confirmation unit and the confirmation result of the third line existence confirmation unit are affirmative, the one side that is the target and the other side that is the target And a line segment expanded from the line segment confirmed to exist by the second line existence confirmation unit and a line segment expanded from the line segment confirmed to exist by the third line presence confirmation unit Third rectangular configuration confirmation means for confirming whether or not to constitute a rectangle;
If the confirmation result by the third rectangular configuration confirmation unit is affirmative, the one side that is the target, the other side that is the target, and the second line existence confirmation unit exist. A third confirming means for confirming that the line segment expanded from the line segment confirmed to be and the line segment expanded from the line segment confirmed to exist by the third line presence confirming means constitute a rectangle;
The frame detection device according to claim 11, further comprising: The frame detection device according to any one of claims 9 to 13, further comprising conversion means for converting an input color image into the grayscale image. 15. The binarization unit divides an input grayscale image into a plurality of areas, and binarizes the image into black and white using a threshold value for each area for each area. The frame detection apparatus of Claim 1. The threshold value is an average value of a temporary threshold value obtained for the own area and a threshold value that is lower than the temporary threshold value obtained for the own area among the temporary threshold values of the eight areas around the own area. The frame detection device according to claim 15.   A frame detection program for causing a computer to execute the frame detection method according to claim 1.