JP5443303B2 - Appearance inspection apparatus and appearance inspection method - Google Patents

Description

  The present invention relates to an appearance inspection apparatus and an appearance inspection method.

  Patent Document 1 describes an appearance inspection apparatus that determines the quality of solder application using an appearance inspection image and a height measurement image. This apparatus calculates | requires the volume of cream solder from the area and height of the application area | region of cream solder, and test | inspects whether it is an appropriate application quantity.

JP 2004-226316 A

  By the way, as a method for measuring the three-dimensional shape of an object, a light cutting method is known in which a height is obtained from a shift amount of light when a sheet-like light is applied from a direction different from the imaging direction. . Since the height can be obtained by this method only in the portion where the light hits, in order to obtain the height distribution over the entire object, it is necessary to photograph many times while changing the irradiation position. For this reason, it takes time to shoot and it is difficult to quickly obtain the height distribution of the entire object.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an appearance inspection apparatus and an appearance inspection method capable of promptly obtaining the height of an object to be inspected with a small number of imaging.

  One embodiment of the present invention is an appearance inspection apparatus that inspects an object to be inspected based on the height of the surface of the object to be inspected that includes a substrate and components mounted on the substrate. The apparatus includes a projection unit for projecting a pattern onto an object to be inspected, an imaging unit for imaging the object to be inspected on which the pattern is projected, and a height of a surface of the object to be inspected based on the captured image. A height measuring unit for obtaining the height. The projection unit projects a first periodic pattern whose brightness periodically changes onto the object to be inspected with a plurality of different phases, and the imaging unit includes a plurality of first projection images whose phases of the first periodic pattern are different from each other. Image. The projection unit projects a second periodic pattern whose brightness changes with a period different from the first periodic pattern onto the object to be inspected with a plurality of different phases, and the imaging unit has phases of the second periodic pattern different from each other. A plurality of second projection images are captured. The height measuring unit determines the phase of periodic brightness fluctuations according to the second period pattern of the height measurement point of the object to be inspected, the brightness of the measurement point in the first projection image and the second projection image, and the first period. The calculation is performed based on the known relationship of the periodic brightness variation corresponding to each of the pattern and the second periodic pattern.

  Another aspect of the present invention is an appearance inspection method for inspecting an object to be inspected based on the height of the surface of the object to be inspected comprising a substrate and components mounted on the substrate. In this method, a first imaging step of projecting a first periodic pattern whose brightness periodically changes onto an object to be inspected with a plurality of different phases to capture a plurality of first projection images, and the first periodic pattern, A second imaging step of projecting a second periodic pattern whose brightness changes in different periods onto the object to be inspected at a plurality of different phases and capturing a plurality of second projected images; and a target corresponding to the second periodic pattern The phase of the brightness fluctuation at the measurement point of the inspection object is determined based on the brightness fluctuation corresponding to each of the brightness of the measurement point in the first projection image and the second projection image and each of the first periodic pattern and the second periodic pattern. And a phase calculation step obtained based on a known relationship.

  Yet another aspect of the present invention is an appearance inspection apparatus that inspects an object to be inspected based on the height of the surface of the object to be inspected that includes a substrate and components mounted on the substrate. The apparatus has a first periodic pattern having a first average intensity and a periodic brightness change, and a second average intensity associated with the first average intensity and having a brightness different from the first periodic pattern. A projection unit for projecting a second periodic pattern having a varying length onto the object to be inspected, a first image when the first periodic pattern is projected onto the object to be inspected, and a first captured image of the first periodic pattern. An imaging unit for imaging the second image when projected onto the object to be inspected with a phase different from the above, and the third image when projecting the second periodic pattern onto the object to be inspected, and a second periodic pattern The brightness at the measurement point when projected onto the object to be inspected with a phase different from that of the third image is the brightness of the measurement point in the first to third images, the first average intensity, and the second average intensity. And a calculation unit obtained based on the relationship.

  Yet another embodiment of the present invention is a method for determining the height of a surface of an object to be inspected in order to inspect the object to be inspected comprising a substrate and a component mounted on the substrate. In this method, a first periodic pattern having a first average intensity and periodically changing brightness is projected onto an object to be inspected, and the first modulation pattern modulated by the surface shape of the object to be inspected is used as a first image. A first imaging step for imaging and the first periodic pattern are projected onto the object to be inspected at a phase different from that of the first imaging step, and a second modulation pattern modulated by the surface shape of the object to be inspected is second. A second imaging step of imaging as an image, and a second periodic pattern having a second average intensity associated with the first average intensity and having a brightness that changes in a period different from the first periodic pattern on the object to be inspected A third imaging step of projecting and imaging a third modulation pattern modulated by the surface shape of the object to be inspected as a third image, and the second periodic pattern in a phase different from that of the third imaging step. Obtaining the brightness at the measurement point when projected onto the body based on the brightness of the measurement point in the first to third images and the relationship between the first average intensity and the second average intensity; including.

  In addition, what converted the expression of this invention between the method, an apparatus, a system, a computer program, a data structure, a recording medium, etc. is also effective as an aspect of this invention.

  According to the present invention, the height of the object to be inspected can be obtained with a small number of imaging.

It is a figure which shows typically the external appearance inspection apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically the projection unit which concerns on one Embodiment of this invention. It is a figure which shows typically the projection pattern correction | amendment which concerns on one Embodiment of this invention. It is a figure which shows an example of the to-be-inspected object image which concerns on one Embodiment of this invention. It is a figure which shows an example of the height map which concerns on one Embodiment of this invention. It is a flowchart for demonstrating the inspection process which concerns on one Embodiment of this invention. It is a figure for demonstrating the method to obtain | require the initial phase of each measurement point which concerns on one preferable Example of this invention. It is a flowchart for demonstrating the height measurement process which concerns on one Embodiment of this invention.

  FIG. 1 is a diagram schematically illustrating an appearance inspection apparatus 10 according to an embodiment of the present invention. The appearance inspection apparatus 10 inspects the inspection object 12 using an inspection object image obtained by imaging the inspection object 12. The device under test 12 is, for example, an electronic circuit board on which a large number of electronic components are mounted. The appearance inspection apparatus 10 determines the quality of the mounting state of the electronic component based on the inspected object image. This inspection is usually performed for a plurality of inspection items for each part. An inspection item is an item that requires pass / fail judgment. The inspection items include, for example, inspection items for component placement such as missing parts, misalignment, and polarity reversal of the components themselves, and inspection items for the connection part between the component and the board, such as soldered state and lead pin floating of the component. Is included.

  The appearance inspection apparatus 10 includes an inspection table 14 for holding the inspection object 12, an imaging unit 16 that illuminates and images the inspection object, an XY stage 18 that moves the imaging unit 16 relative to the inspection table 14, and an imaging And a control unit 20 for controlling the unit 16 and the XY stage 18. For convenience of explanation, as shown in FIG. 1, the inspected object arrangement surface of the inspection table 14 is an XY plane, and a direction perpendicular to the arrangement surface (that is, an imaging direction by the imaging unit 16) is a Z direction.

  The imaging unit 16 is attached to a moving table (not shown) of the XY stage 18 and can be moved in the X direction and the Y direction by the XY stage 18. The XY stage 18 is, for example, a so-called H-type XY stage. Therefore, the XY stage 18 supports the Y linear motor that moves the moving table in the Y direction along the Y direction guide extending in the Y direction, the Y direction guide at both ends thereof, and the moving table and the Y direction guide in the X direction. Two X-direction guides and an X linear motor configured to be movable are provided. The XY stage 18 may further include a Z moving mechanism that moves the imaging unit 16 in the Z direction, or may further include a rotation mechanism that rotates the imaging unit 16. The appearance inspection apparatus 10 may further include an XY stage that allows the inspection table 14 to move. In this case, the XY stage 18 that moves the imaging unit 16 may be omitted.

  The imaging unit 16 includes a camera unit 22, an illumination unit 24, and a projection unit 26. In one embodiment, the camera unit 22, the illumination unit 24, and the projection unit 26 may be configured as an integrated imaging unit 16. In the integrated imaging unit 16, the relative positions of the camera unit 22, the illumination unit 24, and the projection unit 26 may be fixed, or each unit may be configured to be relatively movable. In addition, the camera unit 22, the illumination unit 24, and the projection unit 26 may be separate and configured to be separately movable.

  The camera unit 22 includes an image sensor that generates a two-dimensional image of an object, and an optical system (for example, a lens) for forming an image on the image sensor. The camera unit 22 is, for example, a CCD camera. The maximum field of view of the camera unit 22 may be smaller than the inspection object placement area of the inspection table 14. In this case, the camera unit 22 divides the image into a plurality of partial images and images the entire inspection object 12. The control unit 20 controls the XY stage 18 so that the camera unit 22 is moved to the next imaging position every time the camera unit 22 captures a partial image. The control unit 20 synthesizes the partial images to generate an entire image of the inspection object 12.

  Note that the camera unit 22 may include an image sensor that generates a one-dimensional image instead of the two-dimensional image sensor. In this case, the entire image of the inspection object 12 can be acquired by scanning the inspection object 12 with the camera unit 22.

  The illumination unit 24 is configured to project illumination light for imaging by the camera unit 22 onto the surface of the inspection object 12. The illumination unit 24 includes one or a plurality of light sources that emit light having a wavelength or wavelength range selected from a wavelength range that can be detected by the imaging element of the camera unit 22. The illumination light is not limited to visible light, and ultraviolet light, X-rays, or the like may be used. When a plurality of light sources are provided, each light source is configured to project light of different wavelengths (for example, red, blue, and green) onto the surface of the inspection object 12 at different projection angles.

  In one embodiment, the illumination unit 24 includes an epi-illumination source 24 a that projects illumination light perpendicularly to the inspection surface of the inspection object 12 (that is, the surface facing the imaging unit 16), and the inspection surface of the inspection object 12. On the other hand, a side illumination source 24b that projects illumination light from an oblique direction may be provided. Each of the epi-illumination source 24a and the side illumination source 24b may be a ring illumination source. In other words, the epi-illumination source 24 a is a ring illumination source that surrounds the optical axis of the camera unit 22, and is disposed in the vicinity of the camera unit 22 so as to project illumination light substantially perpendicular to the inspection surface of the inspection object 12. Has been. The side illumination source 24b is a ring illumination source that surrounds the optical axis of the camera unit 22, and is disposed outside the epi-illumination source 24a so as to project illumination light obliquely with respect to the inspection surface of the inspection object 12. Yes.

  As shown in FIG. 1, the epi-illumination source 24a may include one ring illumination source, and the side illumination source 24b may include a plurality (two in the figure) of ring illumination sources. For example, the epi-illumination source 24a may be a blue illumination source, and the side illumination source 24b may be a green illumination source and a red illumination source. Unlike this, the epi-illumination source 24a may include a plurality of illumination sources, and the side illumination source 24b may include one illumination source. Alternatively, each of the epi-illumination source 24a and the side illumination source 24b may include a red illumination source, a blue illumination source, and a green illumination source.

  In FIG. 1, for reference, a light beam projected from the epi-illumination source 24a, reflected by the inspection surface of the inspection object 12 and projected onto the camera unit 22 is indicated by a broken-line arrow. Further, the projections from the side illumination source 24b and the projection unit 26 are similarly indicated by broken arrows. In the illustrated embodiment, the projection unit 26 is provided between the epi-illumination source 24a and the side illumination source 24b. However, the arrangement of the projection unit 26 is not limited to this, for example, the side illumination source 24b. The projection unit 26 may be provided on the outer side.

  The projection unit 26 projects a pattern onto the inspection surface of the inspection object 12. The inspected object 12 on which the pattern is projected is imaged by the camera unit 22. The appearance inspection apparatus 10 creates a height map of the inspection surface of the inspection object based on the imaged pattern image of the inspection object 12. The control unit 20 detects a local mismatch of the pattern image with respect to the projection pattern, and obtains the height of the part based on the local mismatch. That is, the change of the imaging pattern with respect to the projection pattern corresponds to the height change on the inspection surface.

  The projection pattern is preferably a one-dimensional stripe pattern in which bright lines and dark lines are alternately and periodically repeated. The projection unit 26 is disposed so as to project a fringe pattern from an oblique direction with respect to the inspection surface of the inspection object 12. The height discontinuity on the inspection surface of the object to be inspected appears as a pattern shift in the fringe pattern image. Therefore, the height difference can be obtained from the pattern shift amount. In one embodiment, the control unit 20 creates a height map by a PMP (Phase Measurement Profilometry) method using a fringe pattern whose brightness changes according to a sine curve. In the PMP method, the amount of deviation of the fringe pattern corresponds to the phase difference of the sine curve.

  The projection unit 26 includes a pattern forming device, a light source for illuminating the pattern forming device, and an optical system for projecting the pattern onto the inspection surface of the inspection object 12. For example, the pattern forming apparatus may be a variable patterning apparatus that can dynamically generate a desired pattern such as a liquid crystal display, or a fixed patterning in which a pattern is fixedly formed on a substrate such as a glass plate. It may be a device. When the pattern forming apparatus is a fixed patterning apparatus, the projection position of the pattern can be made variable by providing a moving mechanism for moving the fixed patterning apparatus or by providing an adjustment mechanism in the optical system for pattern projection. preferable. The projection unit 26 may be configured to be able to switch between a plurality of fixed patterning devices having different patterns.

  A plurality of projection units 26 may be provided around the camera unit 22. The plurality of projection units 26 are arranged so as to project patterns onto the inspection object 12 from different projection directions. In this way, it is possible to reduce an area where a pattern is not projected due to a shadow due to a height difference on the inspection surface.

  In one embodiment, three projection units 26 may be provided around the camera unit 22. For example, as shown in the drawing, the second projection unit 26b is disposed to face the first projection unit 26a, and the camera unit 22 is disposed between the first projection unit 26a and the second projection unit 26b. The third projection unit (not shown) is arranged out of the plane of arrangement of the first projection unit 26a and the second projection unit 26b (the sheet of FIG. 1). For example, the third projection unit is arranged along a plane that is perpendicular to the arrangement plane of the first projection unit 26 a and the second projection unit 26 b and includes the optical axis of the camera unit 22. Each projection unit is arranged at an equal distance from the camera unit 22.

  If the first projection unit 26a and the second projection unit 26b project the pattern onto the inspection object 12 from the north-south direction, the third projection unit projects the pattern onto the inspection object 12 from the east (or west). . By arranging at least three projection units 26 in this way, it is possible to substantially eliminate blind spots caused by electronic components mounted on the board. The third projection unit may pick up an image of the inspected object 12 when there is a mounted component whose height from the substrate surface exceeds a predetermined value.

  FIG. 2 is a diagram schematically showing the configuration of the projection unit 26 according to an embodiment of the present invention. The projection unit 26 includes a light source 42, a pattern forming device 44, and a projection optical system 46. The light source 42 is, for example, a white LED light source. The pattern forming device 44 is a reflective liquid crystal display. The pattern forming apparatus 44 may be a transmissive device. The projection optical system 46 includes at least one projection lens, and forms an image of the pattern formed on the pattern forming apparatus 44 on the inspection object 12.

  The illumination beam 48 emitted from the light source 42 is given a pattern by the pattern forming device 44. The illumination beam provided with the pattern is projected onto the object 12 as a projection beam 50 by the projection optical system 46. The projection beam 50 has, for example, a fringe pattern having an illuminance distribution corresponding to the fringe pattern having a sinusoidal reflectance distribution formed in the pattern forming device 44. As illustrated, the projection beam 50 is a one-dimensional fringe pattern in which a bright line 52 and a dark line 54 are periodically repeated, and brightness, that is, illuminance has a sinusoidal distribution. The pattern forming device 44 can control the stripe pattern to be displayed by the control unit 20. For example, the stripe pitch of the stripe pattern and the projection position on the inspection object 12 are controlled. That is, the cycle and phase of the sine wave pattern are controlled.

  The projection unit 26 is not limited to a method in which the pattern forming device 44 directly forms a stripe pattern in gray scale and forms an image on the surface of the object to be inspected. For example, a focal beam is intentionally given to the projection beam obtained by illuminating a black and white binary grating pattern, or a pseudo sine wave fringe pattern is obtained by using a diffraction effect by the grating pattern. Classical methods may be used.

  The pattern forming device 44 may correct non-uniformity in illuminance in the cross section of the illumination beam 48 or the projection beam 50 due to the optical characteristics of the projection unit 26. The pattern forming apparatus 44 may form a corrected fringe pattern obtained by superimposing a correction pattern for correcting nonuniformity on the fringe pattern to be projected on the inspection object 12.

  FIG. 3 is a diagram schematically showing correction of a projection pattern according to an embodiment of the present invention. 3 shows the non-uniformity 56 of the optical characteristics of the projection unit 26, the corrected fringe pattern 58 at the center, and the corrected projection beam 60 at the left side. The non-uniformity 56 of the projection unit 26 is, for example, the illuminance distribution of the projection beam 50 when no pattern is formed on the pattern forming device 44. In the illustrated example, the distribution is relatively bright at the center and relatively dark at the periphery. Therefore, the corrected fringe pattern 58 is formed in the pattern forming apparatus 44 so that the central portion is relatively dark and the peripheral portion is relatively bright so as to cancel out this non-uniformity. In this way, it is possible to obtain a regular fringe pattern 60 in which the nonuniformity of the illuminance distribution is corrected and the local average illuminance is uniformed as a whole.

  For example, the pattern forming apparatus 44 may be a so-called LCOS (Liquid Crystal On Silicon) type reflection device in which a liquid crystal element is directly formed on a substrate. In the LCOS type device, necessary wirings and switches are formed behind the liquid crystal element, and the liquid crystal elements are densely arranged. Therefore, it is possible to project a high resolution pattern, which is preferable.

  In this case, it is preferable that the pattern forming apparatus 44 alternately or at a predetermined timing form a set of fringe patterns in which the brightness brought about by the reflecting elements to the projection beam 50 is in opposite phases. In the antiphase fringe pattern, the control amounts of the reflecting elements are equal in opposite directions. Therefore, by forming the antiphase fringe pattern with the same frequency, the average of the control amount can be made almost zero over time and the deviation in one direction can be eliminated.

  The control unit 20 may control the imaging unit 16 so that only one of the set of stripe patterns is imaged by the camera unit 22 and the other is not imaged. The control unit 20 moves the object 12 to be inspected relative to the imaging unit 16 to the imaging position for imaging the next partial image when the non-imaged stripe pattern is formed in the pattern forming device 44. The imaging unit 16 and the XY stage 18 may be controlled. In addition, the control unit 20 may execute the arithmetic processing in the inspection control unit 28 or the height measuring unit 32 when the fringe pattern that is not imaged is formed in the pattern forming device 44. In this way, it is possible to achieve both control requirements and improvement of inspection throughput by using the LCOS type device.

  The control unit 20 shown in FIG. 1 controls the entire apparatus as a whole. The hardware is realized by a CPU, memory, or other LSI of any computer, and the software is loaded into the memory. It is realized by a program or the like, but here, functional blocks realized by their cooperation are drawn. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  FIG. 1 shows an example of the configuration of the control unit 20. The control unit 20 includes an inspection control unit 28 and a memory 30. The inspection control unit 28 includes a height measuring unit 32, an inspection data processing unit 34, and an inspection unit 36. In addition, the appearance inspection apparatus 10 includes an input unit 38 for receiving input from a user or another device, and an output unit 40 for outputting information related to the inspection. The input unit 38 and the output unit 40 are each connected to the control unit 20. The input unit 38 includes, for example, input means such as a mouse and a keyboard for receiving input from the user, and communication means for communicating with other devices. The output unit 40 includes known output means such as a display and a printer.

  The inspection control unit 28 is configured to execute various control processes for inspection based on the input from the input unit 38 and the inspection related information stored in the memory 30. The inspection related information includes a two-dimensional image of the inspection object 12, a height map of the inspection object 12, and substrate inspection data. Prior to the inspection, the inspection data processing unit 34 creates the substrate inspection data using the two-dimensional image and the height map of the inspection object 12 that are guaranteed to pass all inspection items. The inspection unit 36 performs an inspection based on the generated substrate inspection data and the two-dimensional image and height map of the inspection object 12 to be inspected.

  The board inspection data is inspection data created for each type of board. The board inspection data is a collection of inspection data for each component mounted on the board. The inspection data of each part includes an inspection item necessary for the part, an inspection window that is an inspection area on the image for each inspection item, and an inspection standard that is a criterion for pass / fail judgment for each inspection item. One or a plurality of inspection windows are set for each inspection item. For example, in an inspection item for determining whether or not a part is soldered, the same number of inspection windows as the number of soldering areas of the part are usually set in an arrangement corresponding to the arrangement of the soldering areas. In addition, for inspection items that use an image that has been subjected to predetermined image processing on the object image, the contents of the image processing are also included in the inspection data.

  The inspection data processing unit 34 sets each item of inspection data according to the board as the board inspection data creation process. For example, the inspection data processing unit 34 automatically sets the position and size of each inspection window for each inspection item so as to match the component layout of the board. The inspection data processing unit 34 may accept user input for some items of the inspection data. For example, the inspection data processing unit 34 may accept tuning of inspection standards by the user. The inspection standard may be set using height information.

  The inspection control unit 28 executes an imaging process for the object to be inspected 12 as a pre-process for creating board inspection data. As this inspected object 12, one that passes all inspection items is used. As described above, the imaging process is performed by sequentially taking partial images of the inspection object 12 by controlling the relative movement between the imaging unit 16 and the inspection table 14 while illuminating the inspection object 12 by the illumination unit 24. . A plurality of partial images are photographed so that the entire inspection object 12 is covered. The inspection control unit 28 synthesizes the plurality of partial images and generates a whole substrate image including the entire inspection surface of the inspection object 12. The inspection control unit 28 stores the entire board image in the memory 30.

  In addition, the inspection control unit 28 controls the relative movement between the imaging unit 16 and the inspection table 14 while projecting a pattern onto the inspection object 12 by the projection unit 26 as pre-processing for creating a height map. The pattern image of the body 12 is divided and photographed sequentially. The projected pattern is preferably a stripe pattern whose brightness changes according to a sine curve based on the PMP method. The inspection control unit 28 combines the captured divided images and generates a pattern image of the entire inspection surface of the inspection object 12. The inspection control unit 28 stores the pattern image in the memory 30. Note that the pattern image may be generated for a part of the inspection surface instead of the whole.

  The height measuring unit 32 creates a height map of the entire inspection surface of the inspection object 12 based on the pattern on the captured pattern image. First, the height measuring unit 32 obtains a phase difference map of the inspection surface of the inspection object 12 by obtaining a local phase difference between the captured pattern image and the reference pattern image for the entire image. The reference pattern image is a pattern image projected by the projection unit 26 (that is, an image generated by a pattern forming device built in the projection unit 26). The height measurement unit 32 creates a height map of the object 12 to be inspected based on a reference plane serving as a reference for height measurement and a phase difference map. The reference surface is, for example, the substrate surface of the electronic circuit board to be inspected. The reference surface is not necessarily a flat surface, and may be a curved surface reflecting deformation such as warpage of the substrate.

  Specifically, the height measuring unit 32 obtains the phase difference of the fringe pattern between each pixel of the captured pattern image and the pixel of the reference pattern image corresponding to the pixel. The height measuring unit 32 converts the phase difference into a height. Conversion to height is performed using a local stripe width in the vicinity of the pixel. This is to compensate for the fact that the stripe width on the captured pattern image varies depending on the location. Since the distance from the projection unit 26 differs depending on the position on the inspection surface, the stripe width linearly changes from one end to the other end of the pattern projection area on the inspection surface even if the stripe width of the reference pattern is constant. Because it will end up. The height measuring unit 32 obtains the height from the reference surface based on the converted height and the reference surface, and creates a height map of the device under test 12.

  The inspection control unit 28 creates an inspection object image having a height distribution by associating the height information of the height map of the inspection object 12 with each pixel of the two-dimensional image of the inspection object 12. Also good. The inspection control unit 28 may perform three-dimensional modeling display of the inspection object 12 based on the inspection object image with height distribution. Further, the inspection control unit 28 may superimpose the height distribution on the two-dimensional object image and display it on the output unit 40. For example, the inspected object image may be displayed in different colors according to the height distribution. FIG. 4 is an example of the inspection object image, and FIG. 5 is a gray scale display of the height distribution of the inspection object image shown in FIG.

  FIG. 6 is a flowchart for explaining inspection processing according to an embodiment of the present invention. The appearance inspection apparatus 10 inspects an electronic circuit board to be inspected using the generated board inspection data. First, the appearance inspection apparatus 10 creates an entire substrate image (S10). The entire board image is created in the same manner as described above as the preprocessing for creating the board inspection data. That is, the appearance inspection apparatus 10 divides and shoots the electronic circuit board with the imaging unit 16 and synthesizes the obtained images. In this way, the appearance inspection apparatus 10 creates a whole board image that is a two-dimensional image of the whole electronic board facing the imaging unit 16. The entire substrate image is the same as the image shown in FIG.

  Next, the appearance inspection apparatus 10 creates a substrate whole surface height map (S11). The entire board height map is created in the same manner as described above. That is, the appearance inspection apparatus 10 divides and shoots images while projecting a pattern onto the electronic circuit board by the imaging unit 16. A pattern image of the entire substrate is synthesized from the obtained divided images. The appearance inspection apparatus 10 creates a height map of the entire surface of the electronic circuit board facing the imaging unit 16 based on the pattern image. The entire board height map is displayed on the display unit of the appearance inspection apparatus 10 in the format shown in FIG. 5, for example.

  The appearance inspection apparatus 10 applies the substrate inspection data to the entire substrate image and inspects the substrate (S12). The appearance inspection apparatus 10 sets the inspection window necessary for each inspection item on the entire substrate image by applying the substrate inspection data to the entire substrate image. Depending on the inspection item, an inspection may be performed using a dedicated image obtained by performing image processing on the entire substrate image. In this case, the appearance inspection apparatus 10 sets an inspection window on the dedicated image based on the substrate inspection data. The appearance inspection apparatus 10 determines pass / fail of each inspection item according to the inspection standard for each inspection window.

  The inspection criterion may be a threshold for height. In other words, the appearance inspection apparatus 10 may determine pass / fail based on the height distribution in the inspection window obtained from the entire board height map. By using the height information as an inspection standard, it is possible to accurately determine an inspection item that is not always easy to determine from a two-dimensional image. For example, the floating of the lead of the electronic component can be accurately determined with a simple algorithm. It is not always easy to identify whether or not the lead pin floats upward without being connected to the substrate surface by a two-dimensional image captured from above. According to the present embodiment, since the height of the back surface of the lead pin can be measured, it is possible to accurately determine the presence or absence of floating. The appearance inspection apparatus 10 outputs the inspection result and ends the process.

  Instead of creating the substrate whole surface height map after creating the substrate whole surface image, the substrate whole surface height map may be created first. Further, the appearance inspection apparatus 10 may create the entire substrate image and the entire substrate height map in parallel. In this case, the imaging unit 16 and the inspection table 14 for imaging the next partial area after sequentially imaging a partial area of the inspection object 12 under illumination and pattern projection of the illumination unit 24 and the projection unit 26. The imaging unit 16 and the XY stage 18 may be controlled so as to be moved relative to each other. Further, the appearance inspection apparatus 10 may create a two-dimensional image and a height map for a part of the area to be inspected instead of the whole object 12 and inspect the area.

  The height measuring method according to an embodiment of the present invention will be described in more detail. Focusing on one measurement point in the stripe pattern projection area, when the stripe pattern is projected while spatially shifting the phase (in other words, when the stripe pattern is scanned in the direction in which the stripe repeats), the measurement point The brightness varies periodically. Although the average brightness differs for each measurement point according to the surface characteristics of the measurement point such as color and reflectance, periodic brightness fluctuations corresponding to the fringe pattern occur at any measurement point. Therefore, for example, the phase of the fringe pattern when the measurement point is brightest represents the height information of the measurement point. In general, it can be said that the initial phase of the periodic brightness fluctuation gives the height information of each measurement point.

  The height measurement method according to an embodiment of the present invention obtains the height using at least two types of patterns having different periods. A rough height is obtained from a first pattern having a long period (that is, a thick stripe), and a precise height is obtained from a second pattern having a short period (a thin stripe). As described above, since the PMP method obtains the height based on the phase, the height cannot be uniquely specified if the height difference is large and the fringes are shifted by one cycle or more. By obtaining the height in combination with the first fringe pattern, the height can be uniquely specified even when the fringes are shifted by one cycle or more when the second fringe pattern is projected.

  However, when the height is measured with a plurality of different patterns and the result is combined to obtain a height map, it is generally considered that each pattern is individually projected and imaged. In principle, the PMP method requires imaging at least three times, typically four times, with respect to one type of fringe pattern with the phase shifted. Corresponding to the fringe pattern being a sine wave, the brightness fluctuation at each measurement point is also a sine wave. Since the pitch of the fringes is known, a sine wave representing a variation in brightness is specified if the average value, amplitude, and initial phase of brightness are clarified.

By obtaining the brightness measurement value of the measurement point from at least three captured images, it is possible to determine the three variables of brightness average value, amplitude, and initial phase. Also, assuming that the luminance of a pixel when a single fringe pattern is imaged four times with a phase shift of 90 degrees is l _{0 to} l ₃ , the initial phase φ of the measurement point corresponding to that pixel is tan (φ) = ( It is known that l ₃ −l ₁ ) / (l ₂ −l ₀ ). When the pitch of the fringe pattern is P, the amount of deviation of the fringe pattern is obtained by P * (φ / 2π). The height of the position can be obtained from the pattern deviation amount using the projection angle of the pattern.

  If two types of patterns are used, at least 6 times or 8 times of images are taken. In order to inspect one object 12 to be inspected, a plurality of partial areas are imaged to obtain an entire image. One partial area is imaged by each of the plurality of projection units 26. For example, if 10 partial areas are each imaged by 3 projection units, 10 * 3 * 6 (or 8) = 180 (or 240) imaging is required. Thus, the number of imaging required for one inspection tends to increase. The influence of the number of imaging operations on the time required for inspection is large.

  Therefore, in one embodiment of the present invention, when a height is obtained using a certain pattern, an image obtained by projecting another pattern on at least one image is used instead of capturing at least three times with the pattern. Use and estimate. When obtaining the height using a certain pattern, the number of times of imaging is reduced by sharing the luminance information of the image when another pattern is projected.

  In one embodiment, the appearance inspection apparatus 10 projects a first periodic pattern whose brightness periodically changes periodically onto a projection area including the height measurement point of the object 12 to be inspected. A first measurement step for measuring brightness and a second measurement for projecting a second periodic pattern whose brightness changes with a period different from the first periodic pattern to the projection area and measuring the brightness at the height measurement point And step.

  The appearance inspection apparatus 10 determines the phase of the measurement point brightness variation corresponding to at least one of the first and second periodic patterns, the measurement results of the first measurement step and the second measurement step, and the first and second periods. A phase calculation step is performed based on the known relationship of brightness fluctuations corresponding to each of the patterns. The appearance inspection apparatus 10 executes a height information calculation step for obtaining height information of each measurement point based on the phase of brightness variation at each measurement point. By using a known relationship, it is possible to share the measurement result of the other periodic pattern for phase calculation when one of the first and second periodic patterns is projected.

  If necessary, the appearance inspection apparatus 10 projects a third periodic pattern whose brightness changes with a period different from the first and second periodic patterns onto the projection area, and measures the brightness of the height measurement point. A third measurement step may be performed. Similarly, the measurement results of other periodic patterns may be shared for phase calculation when one periodic pattern is projected from the known relationship of brightness fluctuations corresponding to each periodic pattern.

  The brightness measurement of the height measurement point is performed, for example, by imaging the pattern projection area. The first measurement step may include a first imaging step in which the first periodic pattern is projected onto the object to be inspected with a plurality of different phases to capture a plurality of first projection images. The second measurement step may include a second imaging step in which the second periodic pattern is projected onto the object to be inspected with a plurality of different phases to capture a plurality of second projection images. The periodic pattern is shifted in phase by an equal amount, for example 90 degrees or 120 degrees (ie, the pattern is spatially moved in the direction in which the pattern repeats, for example, by 1/4 period or 1/3 period). You may project on the to-be-inspected object 12. FIG. If adjustment is appropriately made in the subsequent arithmetic processing, it is not always necessary to shift by an equal amount.

  In one embodiment, the first and second periodic patterns may be patterns represented by a common periodic function. The first periodic pattern and the second periodic pattern may be made different by changing the parameters of the periodic function. The two periodic patterns may be at least different in period, for example, one period may be at least twice as long as the other period, and preferably greater than 10 times, for example. The two periodic patterns may further differ from each other in at least one of brightness amplitude and average value.

  When the period of the first periodic pattern is larger than the period of the second periodic pattern, the appearance inspection apparatus 10 determines the phase of the periodic brightness fluctuation due to the first periodic pattern at the height measurement point of the inspection object 12, You may calculate based on the brightness of the said measurement point in the some 1st projection image obtained at the 1st imaging step. In this case, the appearance inspection apparatus 10 may obtain the phase of the periodic brightness variation by the second periodic pattern from the first projection image and the second projection image using the known relationship as described above. In this way, the appearance inspection apparatus 10 may obtain the height of the measurement point based on the phase of the periodic brightness fluctuation due to the first periodic pattern and the phase of the periodic brightness fluctuation due to the second periodic pattern.

  In one embodiment, the known relationship between the brightness fluctuation at each measurement point corresponding to the first periodic pattern and the brightness fluctuation corresponding to the second periodic pattern is the average intensity of the first periodic pattern and the second periodic pattern. May be associated with the average intensity of. By associating the average intensities of the two patterns, the brightness of the measurement point when each pattern is projected can be related.

  Preferably, the average intensity of the first periodic pattern and the average intensity of the second periodic pattern may be substantially equal. Therefore, it is preferable that the appearance inspection apparatus 10 includes a projection unit including at least a variable pattern forming apparatus capable of changing the pattern cycle, and the projection unit has a common light source and optical system for each pattern. Is preferred. The variable pattern forming apparatus is preferably configured to be able to change the stripe pitch of the stripe pattern. With this configuration, it is possible to project the pattern under the same projection conditions by changing only the stripe pitch of the stripe pattern. Thus, it is considered that at least the average brightness value is maintained when the pattern period is changed. The projection unit 26 and the pattern forming apparatus 44 shown in FIGS. 1 and 2 are examples of the projection unit and the variable pattern forming apparatus configured as described above.

  Further, the above-described known relationship may include that the brightness amplitude of the measurement point when the first periodic pattern is projected is associated with that of the second periodic pattern. Preferably, the brightness amplitude of each measurement point may be substantially equal between the first periodic pattern and the second periodic pattern. For example, it is also possible to adjust the period of each pattern so that the brightness amplitude when the two patterns are projected is equal in the two patterns by the projection unit and the variable pattern forming device.

  FIG. 7 is a diagram for explaining a method for obtaining the initial phase of each measurement point according to a preferred embodiment of the present invention. The left side of FIG. 7 shows a periodic brightness fluctuation l (x, y) at a certain measurement point (x, y) when the first fringe pattern having a large stripe width is projected while shifting the phase. On the right side, the brightness fluctuation m (x, y) at the same measurement point (x, y) for the second stripe pattern with a small stripe width is shown. Both the first stripe pattern and the second stripe pattern are the sinusoidal stripe patterns shown in FIG. 2, and only the stripe width is different. Due to the optical performance of the projection unit 26, as shown in FIG. 7, the brightness amplitude when projected onto the inspection object 12 is smaller in the second stripe pattern having a smaller stripe width.

In this embodiment, the pattern projection region is sequentially imaged by moving the fringe pattern relative to the inspection object 12 by 90 degrees (that is, by 1/4 period). The brightness of the pixel corresponding to the measurement point (x, y) in each of the plurality of images thus obtained is defined as the brightness of the measurement point (x, y). As described above, the phase φ (x, y) of each measurement point for the first stripe pattern can be obtained by tan (φ) = (l ₃ −l ₁ ) / (l ₂ −l ₀ ). . l ₀ , l ₁ , l ₂ , and l ₃ are the luminances of pixels corresponding to the respective measurement points when the first fringe pattern is projected at a phase of 0 degree, 90 degrees, 180 degrees, and 270 degrees, respectively. Similarly, the phase θ (x, y) of each measurement point for the second stripe pattern can be obtained by tan (θ) = (m ₃ −m ₁ ) / (m ₂ −m ₀ ). . m ₀ , m ₁ , m ₂ , and m ₃ are the luminances of pixels corresponding to the measurement points when the second stripe pattern is projected at a phase of 0 degree, 90 degrees, 180 degrees, and 270 degrees, respectively.

Since the average value of the brightness is constant for each of the first stripe pattern and the second stripe pattern, and the average value of the brightness is equal between the first stripe pattern and the second stripe pattern,
(L ₀ + l ₂ ) / 2 = (l ₁ + l ₃ ) / 2 = (m ₀ + m ₂ ) / 2 = (m ₁ + m ₃ ) / 2
This relationship is established. Therefore,
l ₃ = l ₀ -l ₁ + l ₂
m ₂ = l ₀ + l ₂ −m ₀
m ₃ = l ₀ + l ₂ -m ₁
Is guided.

Therefore, tan (φ) and tan (θ) are respectively
tan (φ) = (l ₀ + l ₂ −2l ₁ ) / (l ₂ −l ₀ )
tan (θ) = (l ₀ + l ₂ −2m ₁ ) / (l ₀ + l ₂ −2m ₀ )
It can ask for. That is, as shown by solid circles in FIG. 7, the phases φ and θ for the first and second fringe patterns are determined from the five measured values of l ₀ , l ₁ , l ₂ , m ₀ , and m _1. Can be sought. L ₃ , m ₂ , and m ₃ indicated by broken-line circles do not need to be measured. Instead of measuring all of l ₀ , l ₁ , l ₂ , l ₃ , m ₀ , m ₁ , m ₂ , m _{3 in} a total of 8 imaging, the phase for both two patterns in 5 imaging Can be sought. This is a smaller number of times than 6 which is the minimum number of times of imaging that can be conventionally considered. Furthermore, if the brightness amplitudes of the first stripe pattern and the second stripe pattern are equal, m ₁ can be omitted and two phases can be obtained by imaging four times of l ₀ , l ₁ , l ₂ , and m _0. I will.

  Here, the image to be captured preferably includes a set of images in which the phases of the fringe patterns are opposite to each other. That is, it is preferable to include an image captured by shifting the fringe pattern by a half cycle. In this way, the average value of brightness can be easily expressed by the simple average of the two measured luminances as in the above relational expression.

  Moreover, it is preferable that the fringe pitch of the first fringe pattern is set to be larger than the maximum value of the component height assumed for the inspection object 12. In this way, the amount of deviation of the stripe pattern caused by the mounting component with the maximum height of the device under test 12 can be kept within one cycle. Therefore, the phase φ can be uniquely determined from tan (φ).

  The fringe pattern does not necessarily have to be a sinusoidal fringe pattern. Any periodic pattern in which the periodic function of the brightness variation is determined based on the brightness of the measurement point obtained by imaging a plurality of times with different phases may be used. By utilizing the known relationship of brightness fluctuations for at least two periodic patterns having different periods, the total number of imaging operations can be reduced in the same manner as in the preferred embodiment described above.

FIG. 8 is a flowchart for explaining the height measurement process according to the embodiment of the present invention. The projection unit 26 of the appearance inspection apparatus 10 projects a first fringe pattern having a first average intensity and periodically changing brightness on a projection area of the inspection object 12. The camera unit 22 captures a first modulation pattern modulated by (step between the substrate surface by eg mounted parts) surface shape of the inspection object 12 as a first image L ₀ (S30). The brightness l ₀ (x, y) of the measurement point (x, y) corresponding to the luminance of each pixel of the image L ₀ is represented. The image L ₀ is stored in the memory 30 of the control unit 20.

Projection unit 26 that projects to the device under test 12 at a different phase with a first fringe pattern first image L _0. Projection unit 26 is, for example, the first image L ₀ projecting the first fringe pattern to the same projection area by shifting the phase by 90 degrees. The camera unit 22 captures a second modulation pattern modulated by the surface shape of the inspection object 12 as the second image L ₁ (S32). The brightness l ₁ (x, y) of the measurement point (x, y) corresponding to the luminance of each pixel of the image L ₁ is represented. Image _{L 1} is stored in the memory 30.

Projection unit 26 that projects to the device under test 12 at a different phase from the first image L ₀ and the second image L ₁ a first fringe pattern. Projection unit 26 projects a first fringe pattern to the same projection area by shifting the further 90 ° phase from the second image L _1. The camera unit 22 captures the third modulation pattern modulated by the surface shape of the inspection object 12 as a third image L ₂ (S34). The brightness l ₂ (x, y) of the measurement point (x, y) corresponding to the luminance of each pixel of the image L ₂ is represented. Image _{L 2} is stored in the memory 30.

The projection unit 26 projects the second stripe pattern having a narrower stripe pitch than the first stripe pattern onto the same projection area under the same projection conditions as the first stripe pattern. The camera unit 22 captures the fourth modulation pattern modulated by the surface shape of the inspection object 12 as the fourth image M ₀ (S36). The brightness m ₀ (x, y) of the measurement point (x, y) corresponding to the luminance of each pixel of the image M ₀ is represented. The image M ₀ is stored in the memory 30.

Projection unit 26 that projects to the device under test 12 at a different phase with a second fringe pattern fourth image M _0. Projection unit 26 projects a second fringe pattern at the same projection area by shifting the phase by 90 degrees and the fourth image M _0. The camera unit 22 captures a fifth modulation pattern modulated by the surface shape of the inspection object 12 as the fifth image M ₁ (S38). The brightness m ₁ (x, y) of the measurement point (x, y) corresponding to the luminance of each pixel of the image M ₁ is represented. Image _{M 1} is stored in the memory 30.

The control unit 20 determines whether or not the projection area has been imaged by all the projection units 26 (S40). If it is not already captured by using the area of all projection units 26 (S40 of N), repeating the first captured image L ₀ to fifth image M ₁ using the following projection unit 26. In this embodiment, imaging may be performed using two projection units 26a and 26b in the north-south direction, and the other projection unit may be used as necessary.

When imaging using all the projection units 26 to be used is completed (Y in S40), the control unit 20 determines whether imaging of the entire inspection object 12 is completed (S42). When there is an area that has not been shot yet (N in S42), the imaging is repeated as described above for the next projection area. During the movement to the next projection region, at least one antiphase pattern of the image L ₁ , the image M ₀ , and the image M ₂ may be formed in the pattern forming apparatus. This antiphase pattern is not imaged.

If the entire imaging of the device under test 12 is completed (S42 of Y), the high of the device under test 12 based on the height measurement unit 32 first image L ₀ to fifth image M ₁ of the control unit 20 A map is created (S44). In the height map creation process, the brightness at the measurement point when the second fringe pattern is projected onto the inspected object 12 with a phase different from that of the fourth image M ₀ and the fifth image M ₁ is represented by the first image L _{0 to} L _0. and determining from the brightness of the measuring point of the fifth image M _1. Further, in the height map creation process, the brightness at the measurement point when the first fringe pattern is projected onto the object 12 to be inspected with a phase different from that of the first image L _{0 to} the third image L ₂ is obtained. ₀ to and determining from the brightness of the measurement points in the third image L _2.

For example, as described above, the brightness of the measurement point when the fourth image M ₀ and projecting the second fringe pattern in opposite phase, utilizing the average intensity between the first fringe pattern and the second fringe pattern is equal to Then, it is obtained from the first image L ₀ , the third image L ₂ , and the fourth image M ₀ . The first image L ₀ is obtained by using the brightness of the measurement point when the second stripe pattern is projected in the opposite phase to the fifth image M ₁ using the fact that the average intensity is equal between the first stripe pattern and the second stripe pattern. , From the third image L ₂ and the fifth image M ₀ . Moreover, the brightness of the measurement point when the first fringe pattern is projected in the opposite phase to the second image L ₁ is obtained from the first image L ₀ , the second image L ₁ , and the third image L ₂ .

The height measuring unit 32 obtains a first phase map based on the first fringe pattern and a second phase map based on the second fringe pattern based on the first image L _{0 to} the fifth image M ₁ . These phase maps consist of phase information at each measurement point. This phase information represents the relative height information of each measurement point. As described above, the height measuring unit 32 obtains the height of each measurement point relative to the substrate surface from the height of the surface position of the substrate and the relative height of each measurement point.

  In the present embodiment, since the same measurement point is imaged using a plurality of projection units 26, the height measurement unit 32 most appropriately determines the measurement value of any one of the projection units when creating the phase map. As a correct value, it may be selected for each measurement point. For example, when the brightness of a pixel representing the measurement point is saturated in an image when a certain projection unit is used, the saturated pixel may be replaced with a corresponding pixel in the image when another projection unit is used. Good. In addition, even when the local average brightness and amplitude near the measurement point deviate from the projection pattern beyond the reference, the pixel that represents the measurement point is a corresponding pixel when another projection unit is used May be replaced.

  Further, when the height measurement unit 32 cannot determine which measurement value by the projection unit is appropriate, the height measurement unit 32 interpolates the measurement value of the measurement point based on the measurement value of the surrounding measurement point. It may be. For example, when the variation in measured values when using a plurality of projection units is large, the height measuring unit 32 may determine that the measured values by any projection unit are not appropriate.

  DESCRIPTION OF SYMBOLS 10 Appearance inspection apparatus, 12 Inspected object, 14 Inspection table, 16 Imaging unit, 18 XY stage, 20 Control unit, 22 Camera unit, 24 Illumination unit, 26 Projection unit, 28 Inspection control part, 30 Memory, 32 Height measurement Part, 34 inspection data processing part, 36 inspection part, 38 input part, 40 output part.

Claims (7)

An appearance inspection apparatus that inspects the object to be inspected based on the height of the surface of the object to be inspected comprising a substrate and a component mounted on the substrate,
A projection unit for projecting a pattern onto the inspection object, an imaging unit for imaging the inspection object on which the pattern is projected, and a height for obtaining the height of the surface of the inspection object based on the captured image A measuring unit,
The projection unit projects a first periodic pattern whose brightness periodically changes onto the object to be inspected with a plurality of different phases, and the imaging unit includes a plurality of first projection images whose phases of the first periodic pattern are different from each other. Image
The projection unit projects a second periodic pattern whose brightness changes with a period different from the first periodic pattern onto the object to be inspected with a plurality of different phases, and the imaging unit has phases of the second periodic pattern different from each other. Capturing a plurality of second projection images;
The height measuring unit determines the phase of periodic brightness fluctuations according to the second period pattern of the height measurement point of the object to be inspected, the brightness of the measurement point in the first projection image and the second projection image, and the first period. An appearance inspection apparatus that performs calculation based on a known relationship between periodic brightness fluctuations corresponding to each of a pattern and a second periodic pattern.   The visual inspection apparatus according to claim 1, wherein the known relationship includes that the average intensity of the first periodic pattern is equal to the average intensity of the second periodic pattern.   The visual inspection apparatus according to claim 1, wherein the plurality of first projection images include a set of images in which phases of the first periodic pattern are opposite to each other. The period of the first periodic pattern is greater than the period of the second periodic pattern;
The height measuring unit calculates a phase of periodic brightness variation based on a first periodic pattern of a height measurement point of the object to be inspected based on brightness of the measurement point in the plurality of first projection images, 4. The height of the measurement point is obtained based on a phase of periodic brightness fluctuation caused by the periodic pattern and a phase of periodic brightness fluctuation caused by the second periodic pattern. 5. Visual inspection equipment. A visual inspection method for inspecting an object to be inspected based on a height of a surface of the object to be inspected comprising a substrate and a component mounted on the substrate,
A first imaging step of projecting a first periodic pattern, the brightness of which periodically changes, onto the object to be inspected at a plurality of different phases and capturing a plurality of first projection images;
A second imaging step of projecting a plurality of second projection images by projecting a second periodic pattern whose brightness changes with a period different from that of the first periodic pattern onto a test object with a plurality of different phases;
The phase of the brightness fluctuation at the measurement point of the inspection object corresponding to the second periodic pattern, the brightness of the measurement point in the first projection image and the second projection image, the first periodic pattern, and the second periodic pattern And a phase calculation step that is obtained based on a known relationship of brightness fluctuations corresponding to each of the visual inspection methods. An appearance inspection apparatus that inspects the object to be inspected based on the height of the surface of the object to be inspected comprising a substrate and a component mounted on the substrate,
The first periodic pattern having the first average intensity and the brightness periodically changing, and the second average intensity associated with the first average intensity and the brightness changing at a different period from the first periodic pattern. A projection unit for projecting the second periodic pattern onto the object to be inspected;
The first image when the first periodic pattern is projected onto the object to be inspected, the second image when the first periodic pattern is projected onto the object under inspection with a phase different from that of the first captured image, and the second periodic pattern An imaging unit for imaging a third image when projected onto the object to be inspected;
The brightness at the measurement point when the second periodic pattern is projected onto the object to be inspected with a phase different from that of the third image, the brightness of the measurement point in the first image to the third image, and the first average intensity An appearance inspection apparatus comprising: a calculation unit that is obtained based on a relationship with the second average intensity. A method for determining the height of the surface of the object to be inspected in order to inspect the object to be inspected comprising a substrate and a component mounted on the substrate,
A first periodic pattern having a first average intensity and periodically changing in brightness is projected onto an object to be inspected, and a first modulation pattern modulated by the surface shape of the object to be inspected is imaged as a first image. Imaging step;
A second imaging step of projecting the first periodic pattern onto the object to be inspected with a phase different from that of the first imaging step, and capturing a second modulation pattern modulated by the surface shape of the object to be inspected as a second image. When,
A second periodic pattern having a second average intensity associated with the first average intensity and having a brightness that changes in a period different from the first periodic pattern is projected onto the object to be inspected, and the surface shape of the object to be inspected A third imaging step of imaging the third modulation pattern modulated by the third image as a third image;
The brightness at the measurement point when the second periodic pattern is projected onto the object to be inspected with a phase different from that of the third imaging step is the brightness of the measurement point in the first to third images. And a step of obtaining based on a relationship between the first average intensity and the second average intensity.