JP6107372B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that are used for, for example, a camera calibration process for calculating the position and orientation of a camera and that are used for extracting marker coordinates on a captured image.

  A technology that makes it easy for the driver to check the surroundings of the host vehicle and contributes to the driver's safe driving by acquiring a surrounding image of the host vehicle with a camera and presenting the acquired image to the driver of the vehicle. Proposed. For example, by providing a plurality of cameras around the host vehicle and synthesizing images captured by the cameras, it becomes possible for the driver to visually recognize a wide range of images around the host vehicle.

  As a result of combining a plurality of captured images, in order to obtain an image that does not feel uncomfortable for the driver, i.e., a combined image in which the seams between images are not shifted or the images are not tilted, It is necessary to be installed in the position and the predetermined camera orientation. However, since the camera is attached to the vehicle, the camera installation position, angle, and the like may change due to vibration caused by traveling or the like. Therefore, at any time after the start of use, such as at the time of shipment or vehicle inspection, check whether there is any deviation in the camera installation position and angle, and perform the camera installation position and angle calibration process (calibration process). There is a need to do.

  In the calibration processing of the camera installation position and angle, a marker (jig) placed at a specified position in the camera shooting range is shot, and the marker position on the shot image, in other words, the coordinates of the marker feature point on the captured image, Associate with the specified position. As a marker used for the calibration process, it is desirable to have a pattern from which a feature point can be easily extracted on the captured image. For example, a checkered marker is generally used. For example, the checkerboard marker is extracted as a feature point at the center point where the checkerboard pattern intersects.

  As a method of extracting feature points of a checkered pattern marker, the feature of the marker is obtained by detecting the color boundary line of the color area constituting the checkered pattern on the captured image and obtaining the intersection using the detected color boundary line. A technique for extracting points is disclosed.

JP 2010-87743 A

  By the way, the coordinates of the feature points of the marker on the image are used as parameters for the calibration processing described above. Therefore, if the extraction accuracy when extracting the feature points from the marker on the captured image is low, the accuracy of the calibration processing of the camera installation position and angle is deteriorated. Therefore, it is necessary to extract the marker feature points on the captured image with high accuracy.

  For example, since it is desirable that a camera mounted on the vehicle to capture the situation around the host vehicle can capture the surroundings of the vehicle with a small number, for example, a wide-angle camera is used. However, the wide-angle camera tends to blur the shape of the marker on the captured image, for example, the shape of the marker that is an object on the captured image is distorted as the distance from the center of the captured image increases. Even if the marker is photographed by a camera other than the wide-angle camera, the shape of the marker is not a little distorted near both ends of the captured image, and the shape of the marker on the captured image becomes unclear.

  Furthermore, depending on the state of the lighting environment in which the calibration process is performed, the checkered pattern of the marker on the captured image may not be captured like the checkered pattern. For example, in a checkered marker having two color regions, a white region and a black region, when the illumination light is strong and bright, an image is captured in a state where the checkered white region of the marker is expanded. When the illumination light is weak and dark, the image is taken in a state where the checkered white area of the marker is contracted. It is difficult to detect a checkered pattern color boundary line and extract a marker feature point from such an unclear marker image. Therefore, it is difficult to extract the marker feature points with sufficient accuracy in the above-described conventional technology.

  For this reason, in the image processing apparatus, it is necessary to extract the feature point of the marker with high accuracy. An object of this invention is to provide the image processing apparatus which can extract the feature point of a marker with high precision.

  An image processing apparatus disclosed in the present invention includes a plurality of first color regions, a plurality of second color regions having a color different from the first color region, a plurality of first color regions, and a plurality of second color regions. An acquisition unit that acquires an image including markers whose color regions face each other is provided. The image processing apparatus further includes an extraction unit that extracts the first color area or the second color area from the image. Further, the image processing apparatus performs the first color region or the first color region until the first similarity between the shape of the extracted first color region or the second color region and the predetermined elliptical shape satisfies a predetermined first threshold value. A reduction unit for reducing the two-color area is provided. A calculation unit is provided that calculates a second similarity between the first color region or the second color region that satisfies the first threshold and a predetermined butterfly shape.

  The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. It should also be understood that both the above general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  With the image processing device disclosed in this specification, it is possible to extract marker feature points with high accuracy even in a blurred image.

(A) is a figure showing the 1st example of a marker. (B) is a figure showing the 2nd example of a marker. (C) is a figure which shows the 3rd example of a marker. (A) is a conceptual diagram when the color region which comprises a marker is recognized correctly. (B) is a conceptual diagram when the color area which comprises a marker is recognized accidentally. It is a functional block diagram of image processing device 1 by one embodiment. It is a figure which shows an example of the arrangement | positioning condition of a marker. It is a determination flowchart of the butterfly shape by a calculation part. (A) is a figure which shows the image before the rotation containing the 1st color area | region which is not a butterfly shape. (B) is a figure which shows the image after the rotation containing the 1st color area | region which is not a butterfly shape. (C) is a projection histogram of the first color region which is not a butterfly shape. (A) is a figure which shows the image before rotation containing the butterfly-shaped 1st color area | region. (B) is a figure which shows the image after the rotation containing a butterfly-shaped 1st color area | region. (C) is a projection histogram of the butterfly-shaped first color region. It is a conceptual diagram of the area | region pair of the 1st color area | region and 2nd color area | region which comprise a marker. 3 is a flowchart of image processing by the image processing apparatus 1. It is a hardware block diagram of the image processing apparatus 1 by one Embodiment.

  Fig.1 (a) is a figure which shows the 1st example of a marker. FIG.1 (b) is a figure which shows the 2nd example of a marker. FIG.1 (c) is a figure which shows the 3rd example of a marker. 1A to 1C, the marker has a checkered pattern. As shown to Fig.1 (a), a marker has a circular area | region in the center part of the square area | region of about 40 centimeters, for example. Moreover, as shown in FIG.1 (b) and FIG.1 (c), a marker has a square area | region further in the center part of the square area | region of about 40 centimeters, for example.

  In FIGS. 1A to 1C, the marker has a plurality of first color regions and a plurality of second color regions having colors different from the first color regions, and the plurality of first color regions are The plurality of second color regions face each other. In other words, the marker has, for example, a checkered pattern in which white areas and black areas are alternate. In addition, the circular area or the square area located at the center of the marker has a pair of white areas and a pair of black areas facing each other across the center of the marker. The marker is, for example, a black and white checkered pattern, for example, to extract the center of the circular area or square area of the marker as a feature point. In this embodiment, the description will be made using the marker shown in FIG. 1 (a), but the present invention can also be implemented using the marker shown in FIG. 1 (b) or FIG. 1 (c). For convenience of explanation, the first color area corresponds to the white area and the second color area corresponds to the black area, but the correspondence relationship may be defined in reverse.

  The following problems have been clarified in an image processing apparatus that extracts feature points of a marker from an unclear image by vigorous verification by the present inventors. For example, International Publication No. 2012/061205 pamphlet discloses an image processing apparatus that extracts feature points of a marker even with a blurred image. The image processing apparatus focuses on the fact that the feature point of the marker is the intersection of a pair of white areas of a checkered pattern and a pair of black areas. A method for extracting and extracting marker feature points based on the positional relationship is disclosed.

  As described above, one color region of the marker is expanded due to the influence of the illumination environment in which the calibration process is performed, and the other color region is degenerated and photographed. For this reason, the marker of the captured image can appear on the captured image in a state where one color region is eroded (may be referred to as a connected state). In the image processing apparatus disclosed in the pamphlet of International Publication No. 2012/061205, in order to separate the eroded areas, similarity to a predetermined convex shape (elliptical shape) with respect to the candidate areas of the color areas constituting the marker It is disclosed that the determination of sex is performed, and if they are not similar, a degeneration process is performed. Specifically, for example, the image processing apparatus calculates a pseudo ellipse shape from candidate areas of color areas constituting the marker. The image processing apparatus compares the area of the ellipse with the area of the candidate area, and determines that the area has a convex shape (elliptical shape) when the similarity of the areas is high. Identify the region pair).

  However, as a result of diligent verification by the present inventors, when one color area of the marker of the captured image is eroded to a certain size, the candidate of the color area constituting the marker observed on the captured image It became clear that the region was observed as a butterfly shape rather than an elliptical shape. FIG. 2A is a conceptual diagram when the color region constituting the marker is correctly recognized. FIG. 2B is a conceptual diagram when the color area constituting the marker is erroneously recognized. As shown in FIG. 2A, when erosion does not occur in a color area, a pair of white areas constituting a marker is recognized by determining the color area (for example, white area) constituting the marker as an elliptical shape. I can do it.

  On the other hand, as shown in FIG. 2B, when erosion occurs in a color area (for example, when illumination light is strong and bright), a pair of white areas constituting a marker are connected to form a butterfly shape. In such a butterfly shape, since the similarity of the area can satisfy the condition in the comparison of the area of the above-mentioned elliptical shape and the candidate region, the degeneration process is not performed, so the pair of white regions constituting the marker is specified. It became clear that there was a problem that could not be done.

  It has been verified by the present inventors that the determination condition for similarity to a predetermined elliptical shape is set to a stricter determination condition so that the butterfly shape can be excluded. However, it has become clear that robustness is affected when the judgment conditions are set strictly for unclear images. For example, when comparing using an area as a determination condition, it is conceivable to set a high threshold for determining that they are similar. However, in the case of an unclear image, since the checkered marker itself is distorted and imaged, there is a problem that the threshold value is not satisfied even if the color regions constituting the marker are separated. In addition, the present inventors have prepared a butterfly-shaped template in advance and verified that similarity is determined. However, in a blurred image, the butterfly shape is captured in various distortions, so the template is applied. It became clear that it was difficult.

  In consideration of the above-mentioned restrictions, in the present embodiment, in the unclear image as shown in FIG. 2B, the feature points of the marker even when the pair of white areas constituting the marker are connected to form a butterfly shape. Disclosed is an image processing apparatus that extracts the image with high accuracy.

  Hereinafter, examples of an image processing apparatus, an image processing method, and an image processing program according to an embodiment will be described in detail with reference to the drawings. Note that this embodiment does not limit the disclosed technology.

Example 1
FIG. 3 is a functional block diagram of the image processing apparatus 1 according to one embodiment. The image processing apparatus 1 includes an acquisition unit 2, an extraction unit 3, a decline unit 4, a calculation unit 5, and a recognition unit 6. The image processing apparatus 1 includes a communication unit (not shown), and can use network resources by bidirectionally transmitting and receiving data to and from various external devices via a communication line.

  The acquisition unit 2 is a hardware circuit based on wired logic, for example. The acquisition unit 2 may be a functional module realized by a computer program executed by the image processing apparatus 1. The acquisition unit 2 receives an image captured by the external device. An external device that captures an image is, for example, an image sensor. The imaging device is an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) camera, for example. Note that the image sensor can be included in the image processing apparatus 1 as necessary. The imaging element images, for example, a checkered pattern marker. The acquisition unit 2 outputs an image including the acquired marker to the extraction unit 3.

  FIG. 4 is a diagram illustrating an example of a marker arrangement state. As shown in FIG. 4, a plurality of image sensors 8 a to 8 d are installed on the vehicle 7 at positions where the entire periphery of the vehicle 7 can be imaged, for example. In addition, 8A-8D shown in FIG. 4 corresponds to the imaging | photography range of the image pick-up elements 8a-8d installed in the vehicle 7. FIG. As shown in FIG. 4, a part of the imaging range 8A of the image sensor 8a, a part of the imaging range 8B of the image sensor 8b, and a part of the imaging range 8C of the image sensor 8c overlap. Further, a part of the imaging range 8D of the image sensor 8d, a part of the imaging range 8B of the image sensor 8b, and a part of the imaging range 8C of the image sensor 8c overlap.

  In FIG. 4, the vehicle 7 stops at a predetermined stop position. And the marker 9 is arrange | positioned around the vehicle 7 so that each of the image pick-up elements 8a-8d installed in the vehicle 7 can image | photograph at least four markers, for example. In this case, the markers 9 may be arranged so that two markers 9 out of the four markers 9 photographed by a certain image sensor are photographed also by an adjacent image sensor. For example, the marker 9 may be arranged so that one marker 9 out of the four markers 9 photographed by the image sensor 8a is photographed by the image sensor 8b adjacent to the image sensor 8a.

  Further, the marker 9 may be arranged so that one marker 9 out of the four markers 9 photographed by the image sensor 8a is photographed by the image sensor 8c adjacent to the image sensor 8a. In other words, the marker 9 may be arranged at a position where the imaging ranges of the plurality of image sensors overlap. However, the physical arrangement position of the marker 9 with respect to the vehicle 7 including the overlapping position is assumed to be measured in advance. In addition, since the image sensor and the marker 9 may be one, in Example 1, it demonstrates as what has one image sensor and the marker 9. FIG. In the following description, the reference numerals of the markers 9 are omitted for convenience of description.

但し、上述の（数１）において、（ｘ、ｙ）は、原画像の位置（原点は、例えば、画像の左上端であれば良い）である。Ｒ（ｘ、ｙ）、Ｇ（ｘ、ｙ）、ならびにＢ（ｘ、ｙ）は、原画像の位置（ｘ、ｙ）におけるＲ成分値、Ｇ成分値、ならびにＢ成分値である。 The extraction unit 3 in FIG. 3 is, for example, a hardware circuit based on wired logic. Further, the extraction unit 3 may be a functional module realized by a computer program executed by the image processing apparatus 1. The extraction unit 3 receives an image from the acquisition unit 2 and extracts a first color region or a second color region from the image. Specifically, the extraction unit 3 first converts an image (for example, an RGB color original image) including the marker received from the acquisition unit 2 into a grayscale image by changing luminance according to the following equation. .

However, in the above (Equation 1), (x, y) is the position of the original image (the origin may be, for example, the upper left corner of the image). R (x, y), G (x, y), and B (x, y) are an R component value, a G component value, and a B component value at the position (x, y) of the original image.

  Next, the extraction unit 3 generates a binary image from the grayscale image. For example, the extraction unit 3 compares the pixel value of the grayscale image with a predetermined threshold value, and binarizes the value equal to or higher than the threshold value to “1” and the value less than the threshold value to “0”. A binary image is generated from the grayscale image. For example, the checkered white area candidate is set to “1” and the black area candidate is set to “0”. Note that the predetermined threshold value described above may not be a fixed value, and may be dynamically determined locally or globally based on the target image. Known methods include, for example, Otsu's discrimination method and NiBlack method.

  The extraction unit 3 extracts a first color area and a second color area (in other words, a white area or a black area) by labeling the binarized binarized image. As a labeling method, for example, a 4-connection (4 neighborhood) method or an 8-connection (8 neighborhood) method which is a known method can be used. The extraction unit 3 may include an image buffer (not shown) that stores the extraction result. The image buffer is, for example, an image having the same size as the above-described binarized image, and is initialized with white (1) or black (0). Note that the image buffer may be provided not by the extraction unit 3 but by the reduction unit 4 or the like. The extraction unit 3 outputs the extracted first color region and second color region to the reduction unit 4.

  The reduction unit 4 receives the first color region and the second color region extracted by the extraction unit 3 from the extraction unit 3. The reduction unit 4 calculates a first similarity between the shapes of the first color region and the second color region and a predetermined elliptical shape. In the following processing, a method for processing the first color region (white region) will be described. The second color region (black region) can also be processed in the same manner as the first color region by performing, for example, black and white reversal processing, and thus detailed description thereof is omitted.

  The reduction unit 4 determines whether each of the first color regions included in the binarized image has an elliptical shape. Specifically, the reduction unit 4 first calculates the major and minor axes of a pseudo ellipse corresponding to the first color region extracted by labeling in the following procedure.

上述の（数２）において、モーメントＭ_ｉｊは、（ｉ＋ｊ）次のモーメントとも称される。ここで、次式で表現される０次モーメントは、第１色領域の面積となる。

When the binarized image at the position (x, y) of the binarized image is set to BW (x, y) and the determination process is performed for any first color region included in the binarized image, the first The color region moment M _ij is defined by the following equation.

In the above (Equation 2), the moment M _ij is also referred to as an (i + j) -th moment. Here, the 0th-order moment expressed by the following equation is the area of the first color region.

更に、重心周りのモーメントを、次式を用いて算出することが出来る。

Based on the above (Equation 2) and (Equation 3), the barycentric coordinates (x _G , y _G ) of the first color region can be calculated.

Furthermore, the moment around the center of gravity can be calculated using the following equation.

In the above (Equation 5), the second moments around the center of gravity (three of MG ₂₀ , MG ₀₂ , and MG ₁₁ ) are called inertia moments. Here, the normalized inertia moment μ _ij normalized by the 0th-order moment (the area of the first color region) is expressed by the following equation.

Only the moment M _ij on the right side of the above (Formula 6) can be expressed by the following equation.

Next, in order to perform the approximation to the elliptical shape (the area is M ₀₀ ) of the first color region, the long axis length a and the short axis length b of the pseudo elliptical shape to be approximated are set as described above ( Using the equation (7), calculation is performed as follows.

The pseudo-elliptical area S can be calculated by the following equation using the above (Equation 8).

なお、減退部４は、例えば、国際公開第２０１２／０６１２０５号パンフレットに開示される減退方法や評価方法を適宜適用することが出来る。 As shown in FIG. 2A, in the case where erosion does not occur in the color area constituting the marker, the color area constituting the marker (for example, the white area) is determined to be elliptical, so that the white area constituting the marker Can be recognized. In other words, when the pair of white areas of the marker is captured in a normally separated state, the area of the white area and the pseudo elliptical area do not differ greatly. Therefore, the reduction unit 4 in FIG. 3 has a predetermined threshold value that is a ratio between the area of the first color area to be processed and the area of the pseudo elliptical shape that applies to the area (may be referred to as the first similarity). It is determined whether or not the first color region to be processed is an elliptical shape depending on whether or not (may be referred to as a first threshold) is satisfied. The reduction unit 4 calculates the evaluation value E using, for example, the following expression for evaluating the area ratio (first similarity), sets the first threshold value to 0.3, for example, and E <0. If it is 3, it is determined that the shape is elliptical, and if E ≧ 0.3, it is determined that the shape is not elliptical.

In addition, the decline part 4 can apply suitably the decline method and evaluation method which are disclosed by the international publication 2012/061205 pamphlet, for example.

  The calculation unit 5 in FIG. 3 is, for example, a hardware circuit based on wired logic. The calculation unit 5 may be a functional module that is realized by a computer program executed by the image processing apparatus 1. The calculation unit 5 accesses the reduction unit 5, and in the first color regions included in the binarized image, the first color region determined to be elliptical by satisfying the above (Equation 10) condition; A second similarity with a predetermined butterfly shape is calculated. Here, the technical significance of calculating the second similarity between the first color region determined to be elliptical by the calculation unit 5 and the predetermined butterfly shape will be described. As shown in FIG. 2B, the inventors have verified that when the color area is eroded (for example, when the illumination light is strong and bright), the pair of the first color shape constituting the marker is connected. It became clear that it became a butterfly shape. In such a butterfly shape, since the condition of the above (Equation 10) can be satisfied, the degeneration process is not performed, and thus a problem of false detection that cannot identify the pair of white areas constituting the marker occurs. However, when the condition of the above (Equation 10) is satisfied, if it is possible to determine whether or not the shape is a butterfly shape, erroneous detection can be prevented and robustness is improved. A method for calculating the second similarity with the predetermined butterfly shape by the calculation unit 5 will be described later.

  When the calculation unit 5 determines that the first color region to be processed is not a butterfly shape, the calculation unit 5 sets the first color region as a candidate for the first color region constituting the marker. The calculation unit 5 accesses the image buffer included in the extraction unit 3, copies the first color area to the image buffer, and erases the range of the first color area of the original image (fills with black). You may do it. In addition, when the calculation unit 5 determines that the first color region to be processed is a butterfly shape, the calculation unit 5 determines whether the first color region constituting the marker is a candidate even if the first color region is an elliptical shape. do not do. When the calculation unit 5 calculates the second similarity with respect to the plurality of first color regions and determines whether or not the shape is a butterfly shape, the first color region candidate constituting the marker is extracted in the image buffer. Thus, a first color region that is not a candidate remains in the original binarized image.

  After the processing of the calculation unit 5 described above is performed, the reduction unit 4 performs a reduction process on the original binarized image. In other words, the reduction unit 4 performs the reduction process on the first color region that has not been determined to be elliptical or the first color region that has been determined to be elliptical but has been determined to have a butterfly shape. Note that the reduction processing includes various methods such as a method of deleting (deleting) pixels on the outer periphery of the first color region (white region) one by one, a method of deleting pixels at the edge portion of the first color region by a predetermined filtering process, and the like. Any known technique can be used. When the reduction process by the reduction unit 4 is completed, the extraction unit 3 performs the labeling again on the binarized image after the reduction, so that the first color region and the second color region (in other words, the white region or the black region). ) Are extracted again, and the reduction unit 4 and the calculation unit 5 repeat the series of processes described above until the first color region disappears. When the iterative process is completed, candidates for the first color region constituting the marker are extracted from the image buffer. Note that the second color region (black region) can be processed in the same manner as the first color region by performing, for example, black-and-white reversal processing, and thus detailed description will be omitted in the following processing.

  Here, a method for determining whether or not the first color region determined to be an elliptical shape by the calculation unit 5 has a predetermined butterfly shape will be described. FIG. 5 is a flowchart for determining a butterfly shape by the calculation unit 5. The calculation unit 5 calculates the barycentric position of the first color area using the first color area determined to be elliptical as input data (step S501). In step S501, the calculation unit 5 can calculate the barycentric position of the first color region based on the above (Equation 4).

なお、慣性主軸は、ステップＳ５０１で算出した重心位置を通過する上述の（数１１）で算出される角度θの傾きを有する直線である。 Next, the calculation unit 5 calculates the inertia main axis of the first color region to be processed (step S502). In addition, the calculation part 5 can calculate an inertia principal axis based on following Formula.

The inertial main axis is a straight line having the inclination of the angle θ calculated by the above (Equation 11) passing through the center of gravity calculated in step S501.

Next, the calculation unit 5 generates a projection histogram on the calculated inertial main axis in the first color region to be processed (step S503). In addition, the calculation part 5 can produce | generate a projection histogram in the following procedures, for example in step S503.
(1) An image including the first color region to be processed is placed around the barycentric coordinates (x _G , y _G ) calculated using the above (Equation 4) based on the above (Equation 11). It is rotated by the calculated angle (−θ). In the rotated image, the direction of the principal axis of inertia is the horizontal direction of the image (in other words, the x-axis direction).
(2) The number of pixels in the first color area is counted in each column direction (vertical direction, in other words, y-axis direction) of the rotated image. The result of counting the number of pixels for each column is a projection histogram.

FIG. 6A is a diagram illustrating an image before rotation including a first color region that is not a butterfly shape. FIG. 6B is a diagram illustrating an image after rotation including a first color region that is not a butterfly shape. FIG. 7A is a diagram showing an image before rotation including a butterfly-shaped first color region. FIG. 7B is a diagram illustrating an image after rotation including a first color region having a butterfly shape. As shown in FIGS. 6 (a) and 7 (a), in the first color region included in the image with the upper left corner as the origin, the center of inertia coordinates (x _G , y _G ) and the inclination θ are defined. The Further, as shown in FIGS. 6B and 7B, the first color region is rotated so that the direction of the principal axis of inertia is parallel to the x-axis of the image.

  FIG. 6C is a projection histogram of the first color region that is not a butterfly shape. FIG. 7C is a projection histogram of the butterfly-shaped first color region. The projection histogram shown in FIG. 6C has a unimodal shape, and it can be confirmed that the projection histogram shown in FIG. 7C has a bimodal multimodality. For this reason, the calculation unit 5 has a projection histogram whose degree of similarity with a predetermined bimodal shape (may be referred to as a second similarity) is equal to or higher than a predetermined similarity (may be referred to as a second threshold). It can be determined that the first color region has a butterfly shape.

  In FIG. 5, the calculation unit 5 calculates the position of the center of gravity of the projection histogram generated in step S503 (step S504). The barycentric position may be, for example, the x coordinate of the barycentric position of the first color area of the rotated image. Incidentally, as shown in FIGS. 7A to 7C, it can be understood that if the first color region is a butterfly shape, there is a minimum value of the projection histogram near the center of gravity position. Therefore, next, the calculation unit 5 searches for the minimum value and the minimum value position within a predetermined search range from the barycentric position of the projection histogram (step S505). For example, the predetermined search range may be set to 0.6 times the distance from the center of gravity position of the projection histogram to the left end (or right end). If the projection histogram is a bimodal shape, it is considered that there is a maximum value near the half of the distance from the center of gravity position to the left end (or right end), and a slightly larger value of 0.6 is set. This is because there is a high possibility that the minimum value to be searched is a minimum value.

  In step S505, the calculation unit 5 obtains the minimum value position from the projection histogram by searching, and then searches for the maximum value on the left side and the right side of the minimum value position. The right maximum value is set (step S506). Next, the calculation unit 5 calculates the ratio between the left maximum value and the right maximum value and the minimum value searched earlier, and sets these as the left ratio and the right ratio (may be referred to as second similarity). (Step S507).

  If the projection histogram is a butterfly area, there are convex areas on the left and right sides of the searched minimum value position, so the left ratio and right ratio (second similarity) are compared to the case where the butterfly shape is not used. Great value. Therefore, the calculation unit 5 determines whether or not both the left ratio and the right ratio are equal to or greater than a predetermined threshold (for example, 3.0) that is the second threshold (step S508). When the second similarity of the first color region to be processed is equal to or greater than the second threshold (Yes in step S508), the calculation unit 5 outputs a result of the butterfly shape affirmation as having bimodality. (Step S509). In addition, when the second similarity of the first color region to be processed is less than the second threshold (No in step S508), the calculation unit 5 determines that the bifality is not present, and the result of the butterfly shape denial Is output (step S510). Upon completion of the processing in step S509 or S510, the calculation unit 5 ends the butterfly shape determination processing shown in the flowchart of FIG. The calculation unit 5 outputs the processing result of the butterfly shape determination to the recognition unit 6 in FIG.

  The recognition unit 6 in FIG. 3 is, for example, a hardware circuit based on wired logic. The recognition unit 6 may be a functional module that is realized by a computer program executed by the image processing apparatus 1. The recognition unit 6 receives the processing result of the butterfly shape determination from the calculation unit 5. Based on the processing result of the butterfly shape determination, the recognition unit 6 sets the first color region or the second color region whose second similarity does not satisfy the second threshold as the first color region or the second color region constituting the marker. recognize. Specifically, the recognition unit 6 detects a feature point of the marker, that is, an intersection of a checkered pattern, using the first color region or the second color region constituting the recognized marker. In other words, the recognizing unit 5 does not satisfy the second threshold and faces the first line segment connecting the predetermined positions of the plurality of first color regions facing each other and does not satisfy the second threshold and faces each other. The intersection of the second line segment connecting the predetermined positions of the plurality of second color regions is recognized as a feature point of the marker. Specifically, the recognition unit 6 executes the following processing, for example.

  There may be a case where a plurality of color area candidates constituting the marker are extracted, but in order to be the first color area and the second color area constituting the marker, it is necessary to include two white areas and two black areas. is there. In addition, the line segment connecting the centroids of the white area and the line segment connecting the centroids of the black area need to intersect. When there are a plurality of extracted area pairs, the recognition unit 6 may further narrow down the area pairs. The recognition unit 6 may use, for example, an approximate size of the marker on the image, and narrow down to a region pair whose distance between the centroids is within a predetermined value (for example, the approximate size of the marker). Next, the recognition unit 6 performs a verification process on the extracted plurality of region pairs. For example, since the shape of the marker is known, a marker-shaped template is prepared in advance, and an area pair that seems to be a marker is selected based on the template matching score. Since the marker on the image may appear distorted, when performing template matching, the image of the region pair is set so that the centroid position of each region of the region pair matches the centroid position of each region of the marker-shaped template. For example, it is preferable to correct the distortion by affine transformation.

  Through the above-described processing, the recognition unit 6 can select the first color region and the second color region region pair constituting the marker. The recognizing unit 6 extracts intersections of line segments connecting the centroids of the selected region pairs as intersections of a checkered pattern and recognizes them as marker feature points.

  The recognition unit 6 recognizes a combination of four regions (hereinafter referred to as region pairs) including a white region pair and a black region pair that satisfy the above conditions. FIG. 8 is a conceptual diagram of an area pair of a first color area and a second color area constituting a marker. As shown in FIG. 8, the recognition unit 6 has an intersection of a first line segment connecting the centroid positions of the first color regions facing each other and a second line segment linking the centroid positions of the second color regions facing each other. It can be recognized as a feature point of the marker. For example, the recognition unit 6 can appropriately apply the feature point determination method disclosed in International Publication No. 2012/061205 pamphlet.

  FIG. 9 is a flowchart of image processing in the image processing apparatus 1. In FIG. 9, the acquisition unit 2 acquires a captured image captured by the external device (step S901). The extraction unit 3 receives the image from the acquisition unit 2, and extracts the first color region or the second color region from the image by the above-described method (step S902). For convenience of explanation, the first color area and the second color area are collectively referred to as a color area. When the number of extracted first color areas or second color areas is greater than 0 (step S903-Yes), the reduction unit 4 selects one color area from the plurality of extracted color areas (step S903). S904). The reduction unit 4 calculates the first similarity between the shape of the selected color region and the predetermined elliptical shape using the above-described method (step S905). For example, the reduction unit 4 compares the first similarity with the first threshold based on the above (Equation 10), and determines whether or not the selected color region satisfies the elliptical shape (step S906).

  When the first similarity is less than the first threshold (step S906-Yes), the calculation unit 5 calculates the second similarity by the above-described method, assuming that the selected color region satisfies the elliptical shape (step S907). . The calculation unit 5 determines whether or not the selected color region satisfies the butterfly shape by comparing the second similarity and the second threshold value (step S908). When the second similarity is less than the second threshold (step S908-No), the recognition unit 6 assumes that the selected color area does not satisfy the butterfly shape, in other words, the color that the selected color area constitutes the marker. It is recognized as an area (step S909). The image processing apparatus 1 determines that the first similarity is greater than or equal to the first threshold in step S906 (step S906-No), or the second similarity is greater than or equal to the second threshold in step S908 (step S908-Yes). In the case of, the process proceeds to step S910 with the selected color area as a color area that does not constitute a marker. The image processing apparatus 1 determines whether or not the comparison processing of the first similarity and the first threshold and the second similarity and the second threshold has been completed in all color regions extracted by the extraction unit 3 in step S902. (Step S910). In step S910, when the comparison process is not completed (step S910-No), the image processing apparatus 1 repeats the processes of steps S904 to S910.

  In step S910, when the comparison process is completed (step S910-Yes), the reduction unit 4 determines that the color region has not been determined to be elliptical in step S906, or has been determined to be elliptical in step S908 but has a butterfly shape. A reduction process is executed for the color area that has been set (step S911). When the process of step S911 is completed, the extraction unit 3 extracts the color area again. Next, the reduction unit 4 determines whether or not the number of extracted color regions is 0 or more (S903). Since the color area gradually disappears because the reduction process is performed in step S911, the number of color areas is eventually reduced to 0 by repeating the processes in steps S902 to S911 (step S903-No).

  In step S903, when the number of color regions becomes 0 (step S903-No), the recognition unit recognizes feature points from the color regions constituting the marker recognized in step S909 by the method described above (step S912). . With the completion of the processing in step S912, the image processing apparatus 1 completes the image processing shown in the flowchart of FIG.

  In the image processing apparatus according to the first embodiment, it is possible to extract a marker feature point with high accuracy even when a pair of white regions constituting a marker is connected to form a butterfly shape in a blurred image.

(Example 2)
FIG. 10 is a hardware configuration diagram of the image processing apparatus 1 according to one embodiment. As illustrated in FIG. 10, the image processing apparatus 1 includes a control unit 10, a main storage unit 11, an auxiliary storage unit 12, a drive device 13, a network I / F unit 15, an input unit 16, and a display unit 17. These components are connected to each other via a bus so as to be able to transmit and receive data.

  The control unit 10 is a CPU that controls each device, calculates data, and processes in a computer. The control unit 10 is an arithmetic device that executes a program stored in the main storage unit 11 or the auxiliary storage unit 12. The control unit 10 receives data from the input unit 16 or the storage device, calculates, and processes the display unit 17. Or output to a storage device.

  The main storage unit 11 is a ROM, a RAM, or the like, and is a storage device that stores or temporarily stores programs and data such as an OS and application software that are basic software executed by the control unit 10.

  The auxiliary storage unit 12 is an HDD or the like, and is a storage device that stores data related to application software and the like.

  The drive device 13 reads the program from the recording medium 14, for example, a flexible disk, and installs it in the auxiliary storage unit 12. A predetermined program is stored in the recording medium 14, and the program stored in the recording medium 14 is installed in the image processing apparatus 1 through the drive device 13. The installed predetermined program can be executed by the image processing apparatus 1.

  The network I / F unit 15 has a communication function connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) constructed by a data transmission path such as a wired and / or wireless line. This is an interface between the device and the image processing apparatus 1.

  The input unit 16 includes a keyboard having cursor keys, numeric input, various function keys, and the like, and a mouse and a slice pad for selecting keys on the display screen of the display unit 17. The input unit 16 is a user interface for a user to give an operation instruction to the control unit 10 and input data.

  The display unit 17 is composed of a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), or the like, and performs display according to display data input from the control unit 10.

  The image processing method described above may be realized as a program for causing a computer to execute. The above-described image processing method can be realized by installing this program from a server or the like and causing the computer to execute it.

  It is also possible to record the program on the recording medium 14 and cause the computer or portable terminal to read the recording medium 14 on which the program is recorded, thereby realizing the above-described image processing. The recording medium 14 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, flexible disk, magneto-optical disk, etc. Various types of recording media such as a semiconductor memory for recording can be used.

  In addition, each component of each illustrated apparatus does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation.

The following supplementary notes are further disclosed with respect to the embodiment described above.
(Appendix 1)
A plurality of first color regions and a plurality of second color regions having colors different from the first color region are included, and the plurality of first color regions and the plurality of second color regions face each other. An acquisition unit for acquiring an image including a marker;
An extraction unit for extracting the first color region or the second color region from the image;
Until the first similarity between the extracted shape of the first color region or the second color region and a predetermined elliptical shape satisfies a predetermined first threshold, the first color region or the second color region is A decline department to decline,
A calculation unit that calculates a second similarity between the first color region or the second color region that satisfies the first threshold and a predetermined butterfly shape;
An image processing apparatus comprising:
(Appendix 2)
A recognizing unit that recognizes the first color area or the second color area in which the second similarity does not satisfy a predetermined second threshold as the first color area or the second color area constituting the marker; The image processing apparatus according to appendix 1, further comprising:
(Appendix 3)
The calculation unit calculates an inertia main axis of the first color area or the second color area that satisfies the first threshold, generates a projection histogram for the inertia main axis, and generates the projection histogram and a predetermined bimodal shape. The image processing apparatus according to appendix 2, wherein the similarity is calculated as the second similarity.
(Appendix 4)
The reduction unit reduces the first color area or the second color area until the first color area or the second color area that satisfies the second threshold satisfies the first threshold. The image processing apparatus according to appendix 3.
(Appendix 5)
The recognizing unit does not satisfy the second threshold and connects a predetermined position of the plurality of first color regions facing each other;
Any one of appendix 2 to appendix 4 that recognizes an intersection of a second line segment that does not satisfy the second threshold and connects a predetermined position of the plurality of second color regions facing each other as a feature point of the marker The image processing apparatus according to claim 1.
(Appendix 6)
5. The image processing apparatus according to claim 1, wherein the marker is a checkered pattern.
(Appendix 7)
The reduction unit is configured to delete the first color region or the second color region by deleting pixels on the outer periphery of the first color region or the second color region or filtering an edge portion. The image processing apparatus according to 1.
(Appendix 8)
A plurality of first color regions and a plurality of second color regions having colors different from the first color region are included, and the plurality of first color regions and the plurality of second color regions face each other. Get an image that includes a marker,
Extracting the first color region or the second color region from the image;
Until the first similarity between the extracted shape of the first color region or the second color region and a predetermined elliptical shape satisfies a predetermined first threshold, the first color region or the second color region is Decline,
An image processing method comprising: calculating a second similarity between the first color region or the second color region that satisfies the first threshold and a predetermined butterfly shape.
(Appendix 9)
Recognizing the first color region or the second color region in which the second similarity does not satisfy a predetermined second threshold as the first color region or the second color region constituting the marker. The image processing method according to claim 8, further comprising:
(Appendix 10)
The calculating calculates an inertia principal axis of the first color area or the second color area that satisfies the first threshold, generates a projection histogram for the inertia principal axis, and generates a projection histogram and a predetermined bimodal shape The image processing method according to appendix 9, wherein the similarity is calculated as the second similarity.
(Appendix 11)
The reduction is characterized in that the first color area or the second color area is reduced until the first color area or the second color area that satisfies the second threshold satisfies the first threshold. The image processing method according to appendix 10.
(Appendix 12)
The recognizing means that a first line segment that does not satisfy the second threshold and connects predetermined positions of the plurality of first color regions facing each other;
Any one of Supplementary Notes 9 to 11 that recognizes an intersection point of a second line segment that does not satisfy the second threshold and connects a predetermined position of the plurality of second color regions facing each other as a feature point of the marker An image processing method according to claim 1.
(Appendix 13)
12. The image processing method according to claim 8, wherein the marker is a checkered pattern.
(Appendix 14)
The reduction is characterized in that the first color area or the second color area is reduced by deleting pixels on the outer periphery of the first color area or the second color area or by filtering an edge portion. The image processing method according to appendix 8.
(Appendix 15)
On the computer,
A plurality of first color regions and a plurality of second color regions having colors different from the first color region are included, and the plurality of first color regions and the plurality of second color regions face each other. Get an image that includes a marker,
Extracting the first color region or the second color region from the image;
Until the first similarity between the extracted shape of the first color region or the second color region and a predetermined elliptical shape satisfies a predetermined first threshold, the first color region or the second color region is Decline,
An image processing program that causes a calculation of a second similarity between the first color region or the second color region that satisfies the first threshold and a predetermined butterfly shape.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Acquisition part 3 Extraction part 4 Decline part 5 Calculation part 6 Recognition part

Claims (7)

A plurality of first color regions and a plurality of second color regions having colors different from the first color region are included, and the plurality of first color regions and the plurality of second color regions face each other. An acquisition unit for acquiring an image including a marker;
An extraction unit for extracting the first color region or the second color region from the image;
Until the first similarity between the extracted shape of the first color region or the second color region and a predetermined elliptical shape satisfies a predetermined first threshold, the first color region or the second color region is A decline department to decline,
A calculation unit that calculates a second similarity between the first color region or the second color region that satisfies the first threshold and a predetermined butterfly shape;
An image processing apparatus comprising:   A recognizing unit that recognizes the first color area or the second color area in which the second similarity does not satisfy a predetermined second threshold as the first color area or the second color area constituting the marker; The image processing apparatus according to claim 1, further comprising:   The calculation unit calculates an inertia main axis of the first color area or the second color area that satisfies the first threshold, generates a projection histogram for the inertia main axis, and generates the projection histogram and a predetermined bimodal shape. The image processing apparatus according to claim 2, wherein the similarity is calculated as the second similarity.   The reduction unit reduces the first color area or the second color area until the first color area or the second color area that satisfies the second threshold satisfies the first threshold. The image processing apparatus according to claim 3. The recognizing unit does not satisfy the second threshold and connects a predetermined position of the plurality of first color regions facing each other;
5. The crossing point of the second line segment that connects the predetermined positions of the plurality of second color regions facing each other and does not satisfy the second threshold value is recognized as a feature point of the marker. The image processing apparatus according to any one of the above. A plurality of first color regions and a plurality of second color regions having colors different from the first color region are included, and the plurality of first color regions and the plurality of second color regions face each other. Get an image that includes a marker,
Extracting the first color region or the second color region from the image;
Until the first similarity between the extracted shape of the first color region or the second color region and a predetermined elliptical shape satisfies a predetermined first threshold, the first color region or the second color region is Decline,
An image processing method comprising: calculating a second similarity between the first color region or the second color region that satisfies the first threshold and a predetermined butterfly shape. On the computer,
A plurality of first color regions and a plurality of second color regions having colors different from the first color region are included, and the plurality of first color regions and the plurality of second color regions face each other. Get an image that includes a marker,
Extracting the first color region or the second color region from the image;
Until the first similarity between the extracted shape of the first color region or the second color region and a predetermined elliptical shape satisfies a predetermined first threshold, the first color region or the second color region is Decline,
An image processing program that causes a calculation of a second similarity between the first color region or the second color region that satisfies the first threshold and a predetermined butterfly shape.

Images