JP2015178961A - Image processing device, three-dimensional shape measurement device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of correctly detecting the position of a hole part having a circular contour from an image, and to provide a three-dimensional shape measurement device and an image processing method.SOLUTION: An image processing device 10 according to an embodiment of this invention includes an image acquisition unit 21, a pixel type estimation unit 22 and a feature amount calculation unit 23. The image acquisition unit 21 acquires an image of a surface provided with a hole part having a circular contour. The pixel type estimation unit 22 estimates pixels included in the contour of the hole part and pixels with the contour of the hole part belonging thereto in accordance with the density of each pixel of the acquired image, that is, which the density belongs to between two types of density ranges: the density range of pixels belonging to the surface; and the density range of the pixels included in the contour of the hole part and the pixels with the contour of the hole part belonging thereto. The feature amount calculation unit 23 calculates the radius and the center position of the hole part on the basis of a density distribution of the pixels included in the contour of the hole part and the pixels with the contour of the hole part belonging thereto.

Description

  Embodiments described herein relate generally to an image processing apparatus, a three-dimensional shape measurement apparatus, and an image processing method for detecting an object having a circular outline from an image.

  Conventionally, various techniques are known as techniques for detecting an object having a circular outline from an image. For example, the technique described in Japanese Patent Laid-Open No. 2002-140713 (Patent Document 1) first sets a temporary center of a circle on a grayscale image and extracts an edge on the image. In edge detection, it is obtained by zero-crossing of the second derivative or Gaussian fitting of the first derivative. Next, based on the distance from the temporary center to the edge and the angle between the tangential direction of the edge and the direction from the temporary center to the edge, each edge constitutes the circumference of a circle that represents the outline of the observation object It is determined whether or not.

  With the technique described in Patent Document 1, it is possible to extract an edge having a high possibility of forming a contour using a set temporary center, and to obtain a center and a radius that are more accurate than the temporary center using the extracted edge. Can do. For this reason, according to this type of technology, for example, the position of a solder ball provided so as to rise on the surface of a plate-like member can be easily detected.

JP 2002-140713 A

  However, the conventional technique does not consider the case where the object is a hole. For this reason, in the prior art, it is difficult to extract a hole from an image obtained by imaging a surface provided with a hole having a circular outline.

  The conventional technology assumes that an object having a circular outline is provided so as to rise on the surface of a plate-like member. In the case of a hemispherical object provided so as to swell on the surface, the density (luminance) of the pixel to which the object belongs in the image is highest near the center of the object, has the lowest contour, and is near the contour. The concentration distribution in the concentration gradient direction has a differentiably smooth distribution. For this reason, it is possible to easily detect the edge using the first and second derivatives as in the conventional technique.

  However, in the case of an image provided with a hole, the density of each pixel is substantially zero from the hole center to the contour, while the density of the surface part is a high density with a substantially constant value. For this reason, the concentration changes very steeply near the contour of the hole, and the concentration distribution in the concentration gradient direction near the contour becomes a non-differentiable distribution.

  Therefore, when a hole is extracted from an image of a surface provided with a hole having a circular contour, it is difficult to detect a stable and accurate edge position by a method using a primary differential or a secondary differential. . For this reason, it is difficult to accurately detect the position of the object. Further, depending on the size of the hole, there are cases where the number of pixels belonging to the hole is as small as about 10-20 pixels, for example. In this case, the contours of the pixels belonging to the hole are in a mosaic shape, which makes it more difficult to detect edges using the conventional technique.

  The present invention has been made in consideration of the above-described circumstances, and provides an image processing apparatus, a three-dimensional shape measuring apparatus, and an image processing method that can accurately detect the position of a hole having a circular outline from an image. The purpose is to do.

  In order to solve the above-described problem, an image processing apparatus according to an embodiment of the present invention includes an image acquisition unit, a pixel type estimation unit, and a feature amount calculation unit. The image acquisition unit acquires an image of a surface provided with a hole having a circular outline. The pixel type estimation unit has two types of density ranges in which the density of each pixel of the acquired image is a density range of pixels belonging to the surface, a pixel included in the outline of the hole, and a density range of the pixel to which the outline of the hole belongs. The pixel included in the outline of the hole and the pixel to which the outline of the hole belongs are estimated. The feature amount calculation unit obtains the radius and center position of the hole based on the density distribution of the pixels included in the outline of the hole and the pixel to which the outline of the hole belongs.

  According to the image processing device, the three-dimensional shape measurement device, and the image processing method according to the present invention, the position of a hole having a circular outline can be accurately detected from an image.

1 is a block diagram showing a configuration example of an image processing apparatus according to a first embodiment of the present invention. Explanatory drawing which shows an example of the object used as the imaging target of the image acquired by the image acquisition part. Explanatory drawing which shows an example of the positional relationship of the hole which has a circular outline, and a pixel. The flowchart which shows an example of the procedure at the time of detecting correctly the position of the hole which has a circular outline from the image by CPU of the main-control part shown in FIG. FIG. 5 is a subroutine flowchart showing an example of a procedure of hole belonging pixel estimation processing executed by a pixel type estimation unit 22 in step S2 of FIG. 4. Explanatory drawing which shows an example of the result of a labeling process. FIG. 5 is a subroutine flowchart illustrating an example of a procedure for calculating a radius r and a center position C, which is executed by a feature amount calculation unit in step S3 of FIG. 4. Explanatory view showing an example of a state in which the concentration G of surface pixels are replaced with the normalized value T _0. Explanatory drawing which shows an example of the normalization density | concentration T of a hole affiliation pixel. Explanatory drawing which shows an example of the reverse normalization density | concentration Tinv of a surface pixel and a hole affiliation pixel. FIG. 5 is a subroutine flowchart illustrating an example of a procedure for correcting a radius r and a center position C executed in step S4 of FIG. 4. FIG. The figure for demonstrating the feature-value correction | amendment process by a correction | amendment part using the partially expanded view of FIG. The whole block diagram which shows an example of the solid-shape measuring apparatus containing the image processing apparatus which concerns on 2nd Embodiment of this invention. The perspective view which shows the example of 1 structure of a focus position change part. The block diagram which shows the example of 1 structure of the image processing apparatus which concerns on 2nd Embodiment of this invention. The figure for demonstrating the image of the object obtained in each of discrete position h1, h2, ..., hp in the height h direction perpendicular | vertical to the surface. The flowchart which shows an example of the procedure at the time of calculating | requiring the density | concentration in the surface position of each pixel by CPU of the main-control part shown in FIG. Explanatory drawing which shows an example of the density distribution of the height h direction of each pixel. (A) is a figure for demonstrating the zero crossing of the 1st-order differential R 'of R and R, (b) is for enlarging (a) partially, and demonstrating in detail about the zero crossing of R'. Figure. Explanatory drawing which shows an example of the height distribution of the surface obtained by calculating height (micro | micron | mu) which gives the estimated maximum density | concentration in each pixel.

  Embodiments of an image processing apparatus, a three-dimensional shape measurement apparatus, and an image processing method according to the present invention will be described with reference to the accompanying drawings.

  An image processing apparatus according to an embodiment of the present invention accurately detects the position of a hole having a circular outline from an image without performing edge detection using primary differentiation or secondary differentiation. In the following description, the surface is assumed to be a plane. Moreover, the hole provided in the surface refers to a portion that is sufficiently less reflected than the reflection of light on the surface when light is irradiated from above the surface, and includes a recess provided in the surface. For example, the holes according to the present embodiment include non-through or through holes such as through-silicon vias (TSVs), substrate blind vias, and through-hole vias, and have a depth corresponding to the depth of the hole. And the like having a small diameter.

(First embodiment)
FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 10 according to the first embodiment of the present invention.

  As illustrated in FIG. 1, the image processing apparatus 10 includes a display unit 11, an input unit 12, a storage unit 13, a network connection unit 14, and a main control unit 15.

  The display unit 11 is configured by a general display output device such as a liquid crystal display or an OLED (Organic Light Emitting Diode) display, and displays various images according to the control of the main control unit 15.

  The input unit 12 includes at least a pointing device and is configured by a general input device such as a mouse, a trackball, a keyboard, a touch panel, and a numeric keypad, and outputs an operation input signal corresponding to a user operation to the main control unit 15. .

  The storage unit 13 includes a recording medium that can be read and written by the CPU of the main control unit 15 such as a magnetic or optical recording medium or a semiconductor memory. Some or all of the programs and data in these storage media may be configured to be downloaded via an electronic network.

  The network connection unit 14 implements various information communication protocols according to the network form. The network connection unit connects the image processing apparatus 10 and other electrical devices according to these various protocols. For this connection, an electrical connection via an electronic network can be applied. Here, an electronic network means an information communication network using telecommunications technology in general, in addition to a wireless / wired LAN (Local Area Network) and the Internet network, a telephone communication line network, an optical fiber communication network, a cable communication network, Includes satellite communications networks.

  The main control unit 15 includes a storage medium such as a CPU, RAM, and ROM, and controls the operation of the image processing apparatus 10 according to a program stored in the storage medium. The CPU of the main control unit 15 loads an image processing program stored in a storage medium such as a ROM and data necessary for executing the program into the RAM. The CPU of the main control unit 15 executes processing for accurately detecting the position of the hole having a circular outline from the image according to this program.

  The RAM of the main control unit 15 provides a work area for temporarily storing programs and data executed by the CPU. The storage medium such as the ROM of the main control unit 15 stores a startup program for the image processing apparatus 10, an image processing program, and various data necessary for executing these programs.

  A storage medium such as a ROM has a configuration including a recording medium readable by a CPU, such as a magnetic or optical recording medium or a semiconductor memory, and a part of programs and data in the storage medium. Or you may comprise so that all may be downloaded via an electronic network.

  As shown in FIG. 1, the CPU of the main control unit 15 functions as at least an image acquisition unit 21, a pixel type estimation unit 22, a feature amount calculation unit 23, and a correction unit 24 by an image processing program. These function realization units 21 to 24 use a required work area of the RAM as a temporary data storage location. Moreover, you may comprise these function implementation parts 21-24 by hardware logics, such as a circuit, without using CPU.

  FIG. 2 is an explanatory diagram illustrating an example of an object 30 that is an imaging target of an image acquired by the image acquisition unit 21.

  The image acquisition unit 21 acquires an image of the surface 33 provided with the hole 32 having the circular outline 31. This image is an image obtained by imaging the object 30 from the information on the surface 33, and may be a distance image.

  FIG. 3 is an explanatory diagram illustrating an example of a positional relationship between the hole 32 having the circular outline 31 and the pixel.

  Among the pixels of the image acquired by the image acquisition unit 21, a pixel (hereinafter referred to as a surface pixel) 41 belonging to the surface 33 has a high density, and a pixel (hereinafter referred to as a hole belonging pixel) 42 belonging to the hole 32. The concentration is low. Among the hole-affiliated pixels 42, the density of a pixel 44 (hereinafter referred to as a hole-included pixel) 44 included in the contour 31 of the hole 32 is substantially zero, and the pixel to which the contour 31 of the hole 32 belongs (hereinafter referred to as a hole contour pixel). 43) has a slightly higher density than the hole inclusion pixel 44. This is because the hole inclusion pixels 44 are all configured by an image of the hole portion 32, whereas the hole contour pixel 43 is a pixel to which the contour 31 belongs and a part thereof is configured by an image of the surface 33.

  Therefore, the pixel type estimation unit 22 determines whether the density G (xm, yn) of each pixel (xm, yn) of the acquired image belongs to the density range of the surface pixel 41 or the density range of the hole-affiliated pixel 42. Accordingly, the surface pixel 41 and the hole assigned pixel 42 are estimated. Further, the pixel type estimation unit 22 determines whether the hole G pixel 42 and the hole inclusion pixel 42 have a density G of the hole belonging pixel 42 depending on which of the density range of the hole contour pixel 43 and the density range of the hole inclusion pixel 44 belongs. 44 is estimated.

  The feature amount calculation unit 23 obtains the radius r and the center position C (xc, yc) of the hole 32 based on the density distribution of the hole belonging pixel 42.

  The correction unit 24 assumes a straight line 45 connecting the center of each pixel and the center position C obtained by the feature amount calculation unit 23 for each of the hole outline pixels 43. Further, the correcting unit 24 is a pixel dividing line segment 46 that is orthogonal to the straight line 45 and divides the pixel into two, and the division area on the center position C side has an area corresponding to its density. Ask for. Then, the correction unit 24 corrects the radius r and the center position C of the hole 32 based on the position of the pixel dividing line segment 46.

  Next, an example of the operation of the image processing apparatus 10 according to the present embodiment will be described.

  FIG. 4 is a flowchart showing an example of a procedure when the CPU of the main control unit 15 shown in FIG. 1 accurately detects the position of the hole 32 having the circular contour 31 from the image. In FIG. 4, reference numerals with numbers added to S indicate steps in the flowchart.

  First, in step S <b> 1, the image acquisition unit 21 acquires an image of the surface 33 provided with the hole 32 having the circular outline 31.

  Next, in step S <b> 2, the pixel type estimation unit 22 determines that the density G (xm, yn) of each pixel (xm, yn) of the acquired image is the density range of the surface pixel 41 and the density range of the hole-affiliated pixel 42. The surface pixel 41 and the hole-affiliated pixel 42 are estimated according to which one they belong to.

  Next, in step S <b> 3, the feature amount calculation unit 23 obtains the radius r and the center position C (xc, yc) of the hole 32 based on the density distribution of the hole belonging pixel 42.

  Next, in step S <b> 4, the correction unit 24 corrects the radius r and the center position C obtained by the feature amount calculation unit 23 based on the density of the hole outline pixel 43. Note that step S4 is not necessarily executed. When step S4 is not executed, the image processing apparatus 10 may not include the correction unit 24.

  With the above procedure, the position of the hole 32 having the circular contour 31 can be accurately detected from the image without performing edge detection using primary differentiation or secondary differentiation.

  Next, a procedure for estimating the hole belonging pixel 42 from the acquired image will be described.

  FIG. 5 is a subroutine flowchart showing an example of the procedure of the hole belonging pixel estimation process executed by the pixel type estimation unit 22 in step S2 of FIG.

  In step S <b> 21, the pixel type estimation unit 22 estimates a region including at least the hole inclusion pixel 44 by performing a labeling process focusing on a pixel having a density G of almost zero.

  FIG. 6 is an explanatory diagram illustrating an example of a result of the labeling process. By the labeling process in step S21, a mosaic area including at least the hole inclusion pixels 44 is extracted (see FIG. 6).

  The position of the hole 32 is estimated by labeling. In other words, a pixel in a predetermined region surrounding the labeled pixel is considered to be the surface pixel 41.

  Next, in step S22, the pixel type estimation unit 22 obtains an average density Gsur_ave of a plurality of pixels in a predetermined area surrounding the labeled pixels. It should be noted that the predetermined area surrounding the labeled pixel (to be extracted from the surface pixel 41) may be an area surrounding the upper, lower, left and right end pixels of the labeled pixel. For example, assuming that the upper, lower, left and right end pixels are (x2, y1), (x3, y4), (x1, y2), (x4, y3), (x1-1, y1-1) and (x4 + 1, y4 + 1) are diagonal. It is preferable that the rectangular area and the area equal to the expanded two pixels of the labeled pixels are defined as the predetermined areas to be extracted from the surface pixels.

Next, in step S <b> 23, the pixel type estimation unit 22 calculates the normalized density T by normalizing the density G of the pixels in the predetermined region using the following formula (1).

The normalized value T ₀ may be a value larger than the maximum density value of the hole belonging pixel 42 (the hole outline pixel 43 and the hole inclusion pixel 44). Further, when normalizing the density of each pixel using the equation (1), the normalized value T ₀ is preferably a value larger than the maximum normalized density of the hole-affiliated pixel 42. In the following description, an example in which the density of each pixel is normalized using Expression (1) and the normalized value T ₀ is 100 will be described.

Next, in step S <b> 24, the pixel type estimation unit 22 estimates a pixel having a normalized density T equal to or greater than a predetermined threshold thT as the surface pixel 41. Note that the threshold thT is preferably a value equal to or smaller than the normalized value T ₀ (for example, when the normalized value T ₀ is 100, thT is set to 95). Further, the pixel type estimation unit 22 estimates the pixel surrounded by the front surface pixel 41 as the hole belonging pixel 42.

  Through the above procedure, the hole-affiliated pixel 42 of the acquired image can be estimated.

  Next, a procedure for obtaining the radius r and the center position C of the hole 32 based on the density distribution of the hole belonging pixel 42 will be described.

  FIG. 7 is a subroutine flowchart showing an example of the procedure of the radius r and center position C calculation process executed by the feature amount calculation unit 23 in step S3 of FIG. This procedure starts when step S24 of the procedure shown in FIG. 5 is executed.

Figure 8 is an explanatory diagram showing an example of a state in which the concentration G of the surface pixel 41 is substituted with a normalized value T _0.

In step S31, the feature amount calculation unit 23 replaces the density G of the surface pixel 41 estimated in step S24 a normalized value _{T 0} (see Figure 8). In step S24, a pixel having a normalized density T equal to or greater than a predetermined threshold thT is estimated as the surface pixel 41. For this reason, the process of step S24 and step S31 can be put together like the following formula | equation (2).

  FIG. 9 is an explanatory diagram showing an example of the normalized density T of the hole belonging pixel 42. FIG. 10 is an explanatory diagram showing an example of the inverted normalized density Tinv of the front surface pixel 41 and the hole belonging pixel 42.

When the density G of the front surface pixel 41 is replaced with the normalized value T ₀ , the feature amount calculation unit 23 sets the density G of the hole belonging pixel 42 (the hole outline pixel 43 and the hole inclusion pixel 44) to the maximum density value of these pixels. Invert by subtracting from a larger normalized value.

Specifically, the feature amount calculation unit 23 obtains the normalized density T of the hole-affiliated pixel 42 using Equation (1) in step S32 (see FIG. 9). Next, the feature amount calculation unit 23 subtracts the normalized density T from the normalized value T ₀ using the following equation (3) in step S33 to obtain an inverted normalized density (inverted normalized density) Tinv (FIG. 10).

Next, the feature amount calculating unit 23 obtains the area A of the hole 32 by dividing the sum of the inverted densities by the normalized value, and obtains the radius r of the hole 32 using the area A. Specifically, as shown in the following equation (4), the feature amount calculation unit 23 obtains the area A of the hole 32 by dividing the sum of the inverted normalized density Tinv by the normalized value T ₀ (Step S4). S34) The radius r is obtained from the area A using the following equation (5) (step S35).

Next, in step S36, the feature amount calculating unit 23 uses the inverted density or inverted normalized density Tinv of the hole belonging pixel 42 to calculate the center of the hole 32 by the centroid calculation shown in the following equations (6) and (7). A position C (xc, yc) is obtained.

  According to the above procedure, the radius r and the center position C of the hole 32 can be obtained based on the density distribution of the hole belonging pixel 42. The main control unit 15 may generate an image such as character information indicating information on the obtained radius r and center position C and superimpose it on the image of the surface 33 and display the image on the display unit 11.

  Next, a procedure for correcting the radius r and the center position C obtained by the feature amount calculation unit 23 based on the density of the hole outline pixel 43 will be described.

  FIG. 11 is a subroutine flowchart showing an example of a procedure for correcting the radius r and the center position C executed in step S4 of FIG. This procedure is started when step S36 of the procedure shown in FIG. 7 is executed and the radius r and the center position C (xc, yc) are obtained by the feature amount calculation unit 23. FIG. 12 is a diagram for explaining the feature amount correction processing by the correction unit 24 using the partially enlarged view of FIG.

  In step S <b> 41, the pixel type estimation unit 22 estimates the hole outline pixel 43. For example, the pixel type estimation unit 22 is a pixel whose normalized density T is smaller than a threshold thT (for example, 95) used at the time of extracting the front surface pixel 41 and that is larger than a value (for example, 5) that belongs to the hole inclusion pixel 44. Is estimated as the hole outline pixel 43.

  Next, in step S <b> 42, the correction unit 24 assumes a straight line 45 connecting the center of each pixel and the center position C obtained by the feature amount calculation unit 23 for each of the hole outline pixels 43.

  Next, in step S43, the correction unit 24 is a pixel division line segment 46 that is orthogonal to the straight line 45 and divides the pixel into two, and the division region on the center position C side has an area corresponding to its own density. The position of the dividing line segment 46 is obtained. Here, as the area according to its own concentration, for example, the inverted normalized concentration Tinv can be used. This is because the inverted normalized density Tinv has a value corresponding to the ratio of the hole portions 32 included in the pixel (see FIG. 12).

Next, in step S <b> 44, the correction unit 24 corrects the radius r and the center position C of the hole 32 based on the position of the pixel dividing line segment 46. Specifically, the correction unit 24 obtains the inclination θi of the straight line 45 and the distance Ri from the center position C to the pixel dividing line segment 46 for each of the hole outline pixels 43, and the inclination θi and the distance Ri. Based on this, the radius r is corrected by the following equation (8) and the center position C is corrected by using the following equations (9)-(11). In equations (8), (10), and (11), n represents the total number of hole outline pixels 43 used for correction calculation.

  With the above procedure, the radius r and the center position C obtained by the feature amount calculation unit 23 based on the density of the hole outline pixel 43 can be corrected.

  The image processing apparatus 10 according to the present embodiment performs the position of the hole 32 having the circular contour 31 based on the density distribution of the hole-affiliated pixel 42 without performing edge detection using primary differentiation or secondary differentiation. Can be accurately detected from the image. Further, when the image processing apparatus 10 includes the correction unit 24, the position of the hole 32 is obtained more accurately by correcting the radius r and the center position C of the hole 32 based on the density of the hole outline pixel 43. Can do.

  Further, the image processing apparatus 10 according to the present embodiment obtains the radius r and the center position C of the hole 32 using only the density of each pixel without performing edge detection using primary differentiation or secondary differentiation. be able to. For this reason, the image processing apparatus 10 is particularly suitable when the number of pixels occupied by the circular hole 32 is small and the hole outline pixels 43 are arranged in a mosaic pattern.

(Second Embodiment)
FIG. 13 is an overall configuration diagram illustrating an example of a three-dimensional shape measurement apparatus 50 including an image processing apparatus 10A according to the second embodiment of the present invention.

  The image processing apparatus 10 </ b> A shown in the second embodiment uses the confocal optics of the three-dimensional shape measurement apparatus 50 for images obtained by imaging the object 30 at discrete positions in the height h direction perpendicular to the surface 33. This is different from the image processing apparatus 10 </ b> A shown in the first embodiment in that it is obtained from the imaging unit 60 having a system and is configured to be able to determine the surface position of each pixel from these images. Since other configurations and operations are not substantially different from those of the image processing apparatus 10 illustrated in FIG. 1, the same components are denoted by the same reference numerals and description thereof is omitted. In the following description, the height h direction perpendicular to the surface 33 of the object 30 is defined as the direction along the optical axis direction, and the optical axis direction (height h direction perpendicular to the surface 33 of the object 30) is defined as Z. An example in which the direction perpendicular to the optical axis direction is the X axis and the Y axis will be described. That is, in the following example, the optical axis direction, the height h direction of the object 30, and the Z direction are parallel to each other.

  As illustrated in FIG. 13, the three-dimensional shape measurement apparatus 50 includes an image processing apparatus 10 </ b> A and an imaging unit 60.

  The imaging unit 60 is for placing an illumination optical system 61 having a light source 61a, an aperture plate 62, an objective lens 63, a focal position changing unit 64, and an object 30 arranged so that the main surface is perpendicular to the optical axis direction. The stage 65, the stage drive unit 66 for moving the stage 65 in each of the XYZ directions, the support stage 67 that supports the stage 65 and the stage drive unit 66, and the reflected light from the object 30 The light detection unit 68 and the image processing apparatus 10A. The mounting table driving unit 66 includes a mounting table Z displacement unit 69 and a mounting table XY displacement unit 70.

  For example, a halogen lamp or a laser can be used as the light source 61a. The light emitted from the light source 61 a becomes a planar illumination light beam via the illumination lens 71. This light illuminates the aperture plate 62 via the polarization beam splitter 72.

  The aperture plate 62 has a plurality of apertures 73 each functioning as a confocal aperture. The aperture plate 62 may be formed with a plurality of apertures 73. The aperture plate 62 may be one in which the apertures 73 are two-dimensionally arranged to form an aperture array. It may be used. When using a Nipkow disc, the Nipkow disc is rotationally driven by a drive mechanism (not shown).

  The light that has passed through each opening 73 irradiates the objective lens 63 via the focal position changing unit 64, and is projected onto the object 30 by the objective lens 63 and is a spot conjugate with the opening 73 (hereinafter, object-side condensing). Focused on a point). Each object-side condensing point is a surface perpendicular to the optical axis direction at a predetermined position in the Z direction (height h direction perpendicular to the surface 33 of the object 30 and the optical axis direction) (hereinafter referred to as object-side condensing surface). Located on the top. Note that the objective lens 63 may be configured by, for example, a plurality of lenses and a diaphragm to form a double-sided telecentric optical system.

  The mounting table Z displacement unit 69 is configured by a general driving device such as a stepping motor, a servo motor, or a piezo motor, and displaces the mounting table 65 in the Z direction. The amount, direction, and timing of this displacement are controlled via the Z-axis driver 74 by the image processing apparatus 10A. The stage table Z displacement unit 69 roughly displaces the stage table 65 in the Z direction before the start of measurement, for example.

  FIG. 14 is a perspective view illustrating a configuration example of the focal position changing unit 64.

  When a parallel plate-shaped optical path changing piece 75 made of a transparent body is arranged in the optical path of the objective lens 63, the position of the object-side condensing surface of the objective lens 63 moves in the Z direction. This focal movement width can be controlled by the refractive index and thickness of the optical path changing piece 75.

  For this reason, as shown in FIG. 14, optical path changing pieces 75 having different focal movement widths are arranged at equal intervals along the rotation direction with respect to the rotating plate 76. By rotating the rotating plate 76 continuously at a predetermined speed by a driving unit 77 such as a motor, the objective lens 63 is discretely (stepwise) each time the optical path changing piece 75 crosses the optical axis of the objective lens 63. The position of the object side condensing surface can be moved in the Z direction. A part of the optical path changing piece 75 disposed on the rotating plate 76 may have the same refractive index and thickness as the other optical path changing pieces 75 and the same focal point movement width.

  The rotation state of the rotating plate 76 is detected by the timing sensor 78. The output of the timing sensor 78 is given to the image processing apparatus 10A. The image processing apparatus 10A holds information that associates each optical path changing piece 75 with the Z-axis coordinate of the object-side condensing point in advance. The image processing apparatus 10 </ b> A repeats exposing the light detection unit 68 at the timing at which each optical path changing piece 75 and the optical axis intersect based on the output of the timing sensor 78, so that the image processing apparatus 10 </ b> A is discrete in the height h direction perpendicular to the surface 33 The object 30 can be imaged easily and at high speed at each appropriate position.

  The drive unit 77 may be configured to be controllable by the image processing apparatus 10A. In this case, the image processing apparatus 10 </ b> A can control the rotation speed of the rotating plate 76.

  In addition, although the example in case the shape of the optical path change piece 75 is circular was shown in FIG. 14, these shapes may be an ellipse, a polygon, etc.

  Of the reflected light from the object 30, the reflected light at the object-side condensing point, in particular, is condensed by the objective lens 63 onto a point that is optically conjugate with the object-side condensing point (hereinafter referred to as image-side condensing point). The The opening 73 serving as a point light source and the object-side condensing point correspond one-to-one. In this embodiment, an example in which the image-side condensing point coincides with the opening 73 serving as a point light source will be described. In this case, the light that has passed through the opening 73 is condensed at the object-side condensing point, reflected by the object-side condensing point, and incident on the opening 73 again.

  The light that has entered the opening 73 again is deflected by the polarization beam splitter 72, enters the imaging optical system 79, and enters the image sensor that constitutes the light detection unit 68. Here, the imaging optical system 79 is configured to connect the image of the opening 73 to the photoelectric conversion surface of the imaging element. The aperture 73 (image-side condensing point) and the image sensor disposed at the position corresponding to the aperture 73 are conjugated with each other by the polarization beam splitter 72 and the imaging optical system 79.

  The light detection unit 68 is configured by a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and outputs a signal corresponding to the intensity of irradiated light to the image processing apparatus 10A. In addition, the light detection unit 68 is controlled by the image processing apparatus 10A for light detection timing.

  Based on the signal received from the light detection unit 68, the image processing apparatus 10 </ b> A acquires images obtained by capturing the object 30 at discrete positions in the height h direction perpendicular to the surface 33. If the image processing apparatus 10A can obtain the output signal of the light detection unit 68 for each exposure, the image processing apparatus 10A can measure the three-dimensional shape of the object 30 based on the output signal.

  The mounting table XY displacement unit 70 of the mounting table driving unit 66 displaces the mounting table 65 in a direction perpendicular to the Z direction. For example, the stage XY displacement unit 70 is used to move the measurement target region in the XY plane between measurements.

  The mounting table XY displacement unit 70 includes an X-axis displacement mechanism 80 and a Y-axis displacement mechanism 81 that position the mounting table 65 in the X-axis direction and the Y-axis direction. The X-axis displacement mechanism 80 and the Y-axis displacement mechanism 81 are configured by, for example, servo motors, and the amount, direction, and timing of displacement are controlled by the image processing apparatus 10A via the XY-axis driver 82.

  FIG. 15 is a block diagram showing a configuration example of an image processing apparatus 10A according to the second embodiment of the present invention.

  The main control unit 15A of the image processing apparatus 10A functions as an image acquisition unit 21A, a pixel type estimation unit 22A, a feature amount calculation unit 23, a correction unit 24, an imaging control unit 25, and a surface specifying unit 26 according to an image processing program.

  The imaging control unit 25 controls the focal position changing unit 64, the light detection unit 68, the Z-axis driver 74, and the XY-axis driver 82, and the imaging unit 60 at each of discrete positions in the height h direction perpendicular to the surface 33. To image the object 30.

  FIG. 16 is a diagram for describing images of the object 30 obtained at discrete positions h1, h2,..., Hp in the height h direction perpendicular to the surface 33.

  The image acquisition unit 21A acquires a plurality of images obtained by imaging the object 30 at discrete positions in the height h direction perpendicular to the surface 33 (see FIG. 16).

  The surface specifying unit 26 acquires the density distribution in the height h direction of each pixel based on a plurality of images, and specifies the surface position and the density at the surface position of each pixel based on the acquired density distribution.

  Each of the plurality of images is associated with different heights h1, h2,. In addition, it is assumed that the imaging areas in the xy plane of the plurality of images are the same. In this case, the pixel (xm, yn) is included in each of the plurality of images. Therefore, by extracting the density G of the pixel (xm, yn) from each of the plurality of images, the density distribution (G (xm, yn, h1), G () in the height h direction of the pixel (xm, yn) is extracted. xm, yn, h2),..., G (xm, yn, hp)). These densities G (xm, yn, h1), G (xm, yn, h2),..., G (xm, yn, hp) are densities measured from the image, and are hereinafter referred to as measured densities as appropriate.

  Now, let hs be the position that gives the highest measured density among the measured densities of each pixel. It is estimated that the highest density of each pixel is obtained at the surface position. For this reason, it can be estimated that the surface position of each pixel is in the vicinity of the position of the height hs.

  Therefore, the surface specifying unit 26 may estimate the position that gives the highest density of each pixel based on the distribution of the measured density of each pixel, and set the position μ that gives this estimated highest density as the surface position. When the position interval (sampling interval) in the height h direction to be imaged is sufficiently small, the surface specifying unit 26 may estimate the height hs itself as the surface position.

  The pixel type estimation unit 22A estimates the surface pixel 41 and the hole assigned pixel 42 using the density at the surface position of each pixel specified by the surface specifying unit 26.

  Next, an example of the operation of the three-dimensional shape measurement apparatus 50 including the image processing apparatus 10A according to the present embodiment will be described.

  FIG. 17 is a flowchart showing an example of a procedure for obtaining the density at the surface position of each pixel by the CPU of the main control unit 15A shown in FIG. In FIG. 17, reference numerals with numbers added to S indicate steps in the flowchart. This procedure is executed in place of step S1 in FIG.

  First, in step S51, the imaging control unit 25 causes the imaging unit 60 to image the object 30 at discrete positions h1, h2,..., Hp in the height h direction perpendicular to the surface 33. . Then, the image acquisition unit 21A acquires an image at each position.

  Next, in step S52, the surface specifying unit 26 extracts the measured maximum density G (hs) of each pixel.

  Next, in step S <b> 53, the surface specifying unit 26 obtains a height μ that gives the estimated maximum density for each pixel whose measured maximum density G (hs) is equal to or greater than a threshold.

  Here, a method of calculating the height μ that gives the estimated maximum density will be described in detail.

FIG. 18 is an explanatory diagram illustrating an example of a density distribution in the height h direction of each pixel. In obtaining the height μ that gives the estimated maximum density, first, among the density distribution in the height h direction of each pixel, the distribution in the vicinity of the maximum density is approximated by a Gaussian distribution represented by the following equation (12). Consider (see FIG. 18).

However, μ represents the vertex position and σ ² represents the variance. In the following description, the distribution of the density G is represented by g.

このとき、式（１３）の１階微分は、次のように書ける。

As a method for obtaining the vertex μ, for example, a method of performing zero crossing on the first derivative of the equation (12) can be cited. However, this method can be applied only when the result of the first-order differentiation is linear, whereas the result of the first-order differentiation of Equation (12) is not necessarily linear.
Now consider the logarithm of equation (12).

At this time, the first derivative of equation (13) can be written as follows.

As is apparent from the equation (14), the first derivative of the equation (13) (logarithm of the equation (12)) is a linear equation. For this reason, it turns out that a zero crossing is applicable with respect to Formula (13). Here, when it is replaced with R (xm, yn, h) = ln [g (xm, yn, h)], the first derivative of R (Equation (13)) can be expressed as follows.

  FIG. 19A is a diagram for explaining zero crossing of the first derivative R ′ of R and R, and FIG. 19B is a partially enlarged view of FIG. It is a figure for demonstrating.

Due to the zero crossing of R ′, μ uses, as the position of the apex of R, the height hs that gives the highest measured density G (hs) and the measurement positions hs−1 and hs + 1 before and after that, as shown in the following equation (16): (See FIGS. 19A and 19B).

Further, when the interval (height pitch) between the imaging positions h1, h2,..., Hp is a constant value Δh, Expression (16) is modified as follows.

Here, since R (xm, yn, h) = ln [g (xm, yn, h)], the following equation (18) holds.

When Expression (18) is substituted into Expression (17), μ is given by the following Expression (19).

  FIG. 20 is an explanatory diagram showing an example of the height distribution of the surface 33 obtained by calculating the height μ that gives the estimated maximum density in each pixel.

  As described above, the surface specifying unit 26 can obtain the height μ (xm, yn) that gives the estimated maximum density for each pixel (xm, yn), and can obtain the height distribution of the surface 33. (See the hatched portion in FIG. 20).

  Next, in step S54, the surface specifying unit 26 obtains the surface position Z based on μ.

Specifically, the surface specifying unit 26 uses the height distribution of the surface 33 to perform a first-order approximation by a method such as a least square method represented by the following equation (20), thereby obtaining the position Z (xm , Yn).

  However, Z (x, y) represents the position of the surface 33 in the height h direction at (x, y), and α, β, and γ represent coefficients obtained by the first-order approximation.

  Next, in step S55, the surface specifying unit 26 obtains the density G (xm, yn, Z) at the estimated surface position Z for each pixel (xm, yn).

  Specifically, the surface specifying unit 26 estimates the density G of each point on the first-order approximated plane using Expression (12). Now, μ in the equation (12) is obtained by the equation (17) or the equation (19). For this reason, if σ can be obtained, the density G at an arbitrary height h can be obtained using the equation (12).

By subtracting the logarithm of the values g (hs) and g (hs + 1) on the concentration distribution of the two points hs and hs + 1 across μ, the following equation (21) is obtained (see equation (13)).

It can be seen that σ can be expressed as the following equation (22) by modifying this equation (21).

  Since the normalized values of the measured densities G and g are known, if the μ and σ of the density distribution g shown in Expression (12) can be obtained, the density G at an arbitrary height h of each pixel can be estimated. it can. Of course, it is also possible to estimate the density G at the estimated surface position of each pixel (see equation (20)).

  In this way, in step S52-55, the surface specifying unit 26 acquires the density distribution in the height h direction of each pixel based on the plurality of images, and based on the acquired density distribution, the surface position and each pixel. Specify the concentration at the surface location.

  After the surface position and the density at the surface position are estimated, the processing after step S2 in FIG. 4 is executed in the same manner as the image processing apparatus 10 according to the first embodiment. Note that the image processing apparatus 10A according to the present embodiment determines that the density of the surface position Z is smaller than the threshold in step S52 in step S21 (labeling process) in FIG. 5 illustrating an example of the process in step S2 in FIG. It is better to focus on the pixels that did not become.

  In addition, when it takes time to estimate the concentration G at an arbitrary height h using the equation (12), such as it takes time to calculate σ in the equation (22), the measured maximum concentration G (hs) is set to The given measurement position hs itself may be estimated as the surface position. In this case, the threshold thT used in step S24 of FIG. 5 is determined according to the height pitch, such as lowering the threshold when the intervals (height pitch) of the imaging positions h1, h2,. Also good.

  The image processing apparatus 10A according to the present embodiment also has the same effects as the image processing apparatus 10 according to the first embodiment. In addition, the image processing apparatus 10A according to the present embodiment is provided on the surface 33 of the object 30 by obtaining the surface position from a plurality of height-direction images captured by the imaging unit 60 having a confocal optical system. The position of the formed hole 32 can be obtained accurately.

  In addition, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

10, 10A Image processing device 21, 21A Image acquisition unit 22, 22A Pixel type estimation unit 23 Feature amount calculation unit 24 Correction unit 26 Surface identification unit 30 Object 31 Outline 32 Hole 33 Surface 41 Surface pixel 42 Hole belonging pixel 43 Hole outline Pixel 44 Hole inclusion pixel 46 Pixel division line segment 50 Three-dimensional shape measurement device 60 Imaging unit A Area r Radius C Center position G Measurement density Gsur_ave Average density g Density distribution T Normalized density thT Threshold T ₀ Normalized value Tinv Inverted normalized density

Claims (10)

An image acquisition unit for acquiring an image of a surface provided with a hole having a circular outline;
The density of each pixel of the acquired image is one of the two density ranges: the density range of the pixel belonging to the surface, the pixel included in the outline of the hole, and the density range of the pixel to which the outline of the hole belongs. A pixel type estimation unit that estimates a pixel included in the outline of the hole and a pixel to which the outline of the hole belongs, according to whether it belongs,
A feature amount calculating unit for obtaining a radius and a center position of the hole based on a density distribution of a pixel included in the outline of the hole and a pixel to which the outline of the hole belongs;
An image processing apparatus. The feature amount calculation unit includes:
The density of the pixel included in the outline of the hole and the density of the pixel to which the outline of the hole belongs are reversed by subtracting from a normalized value larger than the maximum density value of these pixels, and included in the outline of the hole. The area of the hole is obtained by dividing the sum of the inverted densities of the pixel to which the pixel belongs and the outline of the hole by the normalized value, and the radius of the hole is obtained using the obtained area. ,
The image processing apparatus according to claim 1. The feature amount calculation unit includes:
The center position of the hole is obtained by a centroid calculation using the inverted density of each pixel included in the outline of the hole and the pixel to which the outline of the hole belongs.
The image processing apparatus according to claim 2. The pixel type estimation unit
A region including pixels included in the outline of the hole is estimated by performing a labeling process on the acquired image, an average density of a plurality of pixels in a predetermined region surrounding the estimated region is obtained, and the predetermined region Among the plurality of pixels, a pixel having a normalized density obtained by dividing the pixel density by the average density and multiplying by the normalized value is estimated as a pixel belonging to the surface,
The normalized value is greater than the predetermined threshold;
The image processing apparatus according to claim 3. The feature amount calculation unit includes:
Obtaining the normalized density for the pixel contained in the outline of the hole and the pixel to which the outline of the hole belongs, and subtracting from the normalized value to obtain the inverted density;
The image processing apparatus according to claim 4. The pixel type estimation unit
The density of each pixel of the pixel included in the estimated outline of the hole and the pixel to which the outline of the hole belongs belongs to the density range of the pixel included in the outline of the hole and the pixel to which the outline of the hole belongs. According to which of the density ranges, further estimate the pixels included in the contour of the hole,
For each pixel to which the outline of the hole belongs, a straight line connecting the center of each pixel and the center position of the hole obtained by the feature amount calculation unit is assumed, and the pixel is orthogonal to the straight line and divided into two. A pixel dividing line segment, which is a pixel dividing line segment whose area on the center position side of the hole portion has an area corresponding to its own density, is determined based on the position of the pixel dividing line segment. A correction unit for correcting the radius and the center position,
The image processing apparatus according to claim 1, further comprising: The image acquisition unit
Obtaining a plurality of images of the object obtained by imaging the object including the surface for each discrete position in a height direction perpendicular to the surface;
A surface specifying unit that acquires the density distribution in the height direction of each pixel based on the plurality of images, and specifies the surface position and the density at the surface position of each pixel based on the acquired density distribution;
Further comprising
The pixel type estimation unit
Estimating a pixel included in the outline of the hole and a pixel to which the outline of the hole belongs according to the density of each pixel specified by the surface specifying unit at the surface position;
The image processing apparatus according to claim 1. The surface specifying portion is
The surface position and the density of each pixel at the surface position are specified by approximating the discrete density distribution of each pixel in the image at each discrete position in the height direction with a Gaussian distribution.
The image processing apparatus according to claim 7. An image processing device according to claim 7 or 8,
An imaging unit that has a confocal optical system, and generates a plurality of images of the object by imaging the object at discrete positions in the height direction;
A three-dimensional shape measuring apparatus. Obtaining an image of a surface provided with a hole having a circular outline;
The density of each pixel of the image belongs to any one of two density ranges: a density range of the pixel belonging to the surface, a pixel included in the outline of the hole, and a density range of the pixel to which the outline of the hole belongs. Accordingly, estimating a pixel included in the outline of the hole and a pixel to which the outline of the hole belongs,
Obtaining a center position and a radius of the hole based on a density distribution of a pixel included in the outline of the hole and a pixel to which the outline of the hole belongs;
An image processing method.

Images