JP2020197511A - Measurement device, imaging device, measurement system, and program - Google Patents

Abstract

To acquire an image taken of a measurement target in a desirable state when making a measurement based on the image.SOLUTION: The measurement device includes: acquisition means for acquiring an image taken of an object with a flat surface as a measurement target on which a predetermined pattern is projected, information of the predetermined pattern, and information on the relative positional relation between a projection optical system which projected the predetermined pattern and an imaging optical system; determination means for determining the direction of taking the image on the basis of the image taken by the acquisition means, the information on the predetermined pattern, and the information on the relative positional relation; and first notification means for issuing a notification according to the direction of imaging determined by the determination means.SELECTED DRAWING: Figure 3

Description

The present invention relates to a measuring device, an imaging device, a measuring system, and a program, and more particularly to a technique for performing various measurements on a subject based on a captured image.

One of the dimensional measurement methods is to take an image of a subject to be measured and derive the dimensions based on the size of the subject image on the captured image. For example, in industrial applications, such a dimensional measurement method is used for visual inspection to determine whether parts and products are processed and assembled according to specifications. In addition, in recent years, communication terminals such as smartphones have a function of superimposing the distance between predetermined points in the real space and a measuring rod that serves as a reference for grasping the distance on the captured image of a subject such as furniture or fish. There are also things.

In such dimensional measurement based on a captured image, the distance to the subject (subject distance, depth information) is required to derive the result in addition to the imaging magnification and the imaging angle of view at the time of imaging. One method for deriving the subject distance is to project an image of a known pattern onto the subject from a position different from that of the imaging optical system, and analyze the projection mode in the captured image to derive the subject distance and the surface shape of the subject. is there. In Patent Document 1, distance measurement is performed by projecting parallel light at a known interval and imaging the parallel light.

JP-A-2017-191024

By the way, in a situation where the object to be dimensionally measured based on the captured image as described above is inclined in the yaw direction or the pitch direction with respect to the optical axis of the imaging optical system, due to the change in the imaging magnification due to the difference in the depth position, The measurement accuracy may decrease. This is because there is a difference in shooting magnification between the measurement position in the foreground and the measurement position in the back, and the spatial resolution (resolution) in the captured image is different. Therefore, the unit length assigned to the subject in the back compared to the subject in the foreground in the captured image. This is due to the fact that the number of pixels around the edge is reduced.

On the other hand, in the irradiation device of Patent Document 1, the subject is irradiated with three types of reference light points having a predetermined relationship at the time of imaging, and the measurement target is tilted according to the positional relationship of these intersections appearing in the captured image. Is acquired and the measurement result is corrected. However, since the method of Patent Document 1 corrects the change in the imaging magnification in the depth direction after imaging, there is a possibility that the decrease in accuracy due to the decrease in the number of pixels cannot be fundamentally solved even if the dimensions are derived.

The present invention has been made in view of the above-mentioned problems, and provides a measuring device, an imaging device, a measuring system, and a program for acquiring an captured image in a state in which a measurement target is suitable for measurement based on an captured image. With the goal.

In order to achieve the above-mentioned object, the measuring device of the present invention projects a captured image of a subject having a plane to be measured on which a predetermined pattern is projected, information on the predetermined pattern, and a predetermined pattern. Imaging of the captured image based on the acquisition means for acquiring the information on the relative positional relationship between the projected projection optical system and the imaging optical system, the captured image acquired by the acquisition means, the information of a predetermined pattern, and the information on the relative positional relationship. It is characterized by having a determination means for determining a direction and a first notification means for giving a notification according to an imaging direction determined by the determination means.

According to the present invention, it is possible to acquire an image captured in a state in which the measurement target is suitable for measurement based on the captured image.

External view showing the measuring device 100 according to the first embodiment of the present invention. The figure for demonstrating the functional structure of the measuring apparatus 100 which concerns on Embodiment 1 of this invention. A flowchart illustrating a measurement process executed by the measuring device 100 according to the first embodiment of the present invention. The figure for demonstrating the input of the measurement point which concerns on the measurement item in the measurement apparatus 100 which concerns on Embodiment 1 of this invention. Another figure for explaining the input of the measurement point which concerns on the measurement item in the measurement apparatus 100 which concerns on Embodiment 1 of this invention. Yet another diagram for explaining the input of the measurement points related to the measurement items in the measuring device 100 according to the first embodiment of the present invention. The figure for demonstrating the relationship between the plane normal and the imaging direction of the object subject which concerns on embodiment and modification of this invention. Another diagram for explaining the relationship between the plane normal and the imaging direction of the target subject according to the embodiment and the modified example of the present invention. The figure for demonstrating the pattern projection mode to the object subject which concerns on Embodiment 1 of this invention. The figure for demonstrating the state which the measuring apparatus 100 which concerns on Embodiment 1 of this invention and a plane of a target subject face each other. The figure for demonstrating the relationship between the projection angle and the amount of image displacement which concerns on Embodiment 1 of this invention. The figure which illustrated the notification mode of the state which does not face a plane of a target subject in the measuring apparatus 100 which concerns on Embodiment 1 of this invention. The figure which illustrated the presentation mode of the measurement result in the measuring apparatus 100 which concerns on Embodiment 1 of this invention. The figure for demonstrating the presentation mode of the guide display for improving the measurement accuracy in the measuring apparatus 100 which concerns on Embodiment 1 of this invention. Another diagram for explaining the presentation mode of the guide display for improving the measurement accuracy in the measuring device 100 according to the first embodiment of the present invention. External view showing the measuring device 100 according to the modified example of the present invention. The figure for demonstrating the pattern projection mode to the target subject which concerns on the modification of this invention. A flowchart illustrating a measurement process executed by the measuring device 100 according to the second embodiment of the present invention. The figure which illustrated the distribution state of the measurement point which concerns on the measurement item in Embodiment 2 of this invention. The figure which illustrated the captured image stored for dimension derivation in the measuring apparatus 100 which concerns on Embodiment 2 of this invention. The figure which illustrated the developed image for dimension derivation in the measuring apparatus 100 which concerns on Embodiment 2 of this invention.

[Embodiment 1]
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

One embodiment described below describes an example in which the present invention is applied to a measuring device having a projection optical system for projecting a pattern for distance measurement and an imaging optical system for measurement as an example of a measurement system. To do. However, the present invention is configured to be capable of transmitting and receiving information by being connected to a projection device and an imaging device arranged in a known positional relationship by communication, and can be an arbitrary device capable of analyzing an imaging mode of a measurement target, or any device thereof. Applicable to the configured system.

<< Configuration of measuring device 100 >>
FIG. 1 is a diagram showing the appearance of the measuring device 100 according to the present embodiment. As shown in the figure, in the measuring device 100 of the present embodiment, an imaging unit 102 for imaging and a projection unit 101 for pattern projection are provided in the housing of the measuring device 100, and pattern projection is performed by one unit. It is configured to enable imaging and derivation of subject distance. In the present embodiment, the measuring device 100 is described as including the imaging unit 102 and the projection unit 101 in this way, but the implementation of the present invention is not limited to this. The measuring device 100 grasps the information of the pattern projected by the projection unit 101 on the subject of the captured image acquired from the imaging unit 102, and analyzes the state of the pattern in the captured image to determine the imaging direction of the imaging unit 102. It does not have to be integrated as long as it can be identified.

Next, the functional configuration of the measuring device 100 of the present embodiment will be described with reference to the block diagram of FIG.

The calculation unit 104 is a control device composed of, for example, a CPU, a microprocessor, and a work memory, and controls the operation of each block included in the measurement device 100. The arithmetic unit 104 controls the operation of each block by reading the operation program of each block stored in the storage unit 105, expanding it into a work memory (not shown), and executing the program. The arithmetic unit 104 also performs control processing of the imaging unit 102 such as autofocus (AF: automatic focusing), change of focus position, and change of F value (aperture) at the time of imaging. Further, the calculation unit 104 of the present embodiment will be described as performing various image processing on the image signal captured by the image pickup unit 102.

The storage unit 105 is a recording device included in the measuring device 100 such as a non-volatile memory. The storage unit 105 is used to store information that requires permanent recording, such as an operation program of each block of the measuring device 100. On the other hand, the work memory described above is a storage device such as a volatile memory, and is not only used as an expansion area for an operation program but also as a storage area for temporarily storing intermediate data output in the operation of each block. Function.

The display 103 is, for example, a display device such as an LCD, and presents various types of information in the measuring device 100. The display 103 functions as a digital viewfinder by displaying (through display) the A / D converted image data when the image pickup unit 102 is performing imaging. Further, in the present embodiment, the display 103 is also used for displaying a GUI when notifying a suitable imaging direction and displaying a measurement result.

The operation input unit 106 is a user input interface included in the measuring device 100 such as a release switch, a setting button, and a mode setting dial. When the operation input unit 106 detects the operation input made to various user input interfaces, the operation input unit 106 outputs a control signal corresponding to the operation input to the calculation unit 104. Further, in the embodiment in which the display 103 is provided with a touch panel sensor, the operation input unit 106 also functions as an interface for detecting a touch operation performed on the display 103.

The projection unit 101 projects a predetermined pattern on the subject to be measured. The projection unit 101 is composed of a projection optical system 111, a pattern display unit 112, and a light projection unit 113, and is controlled to be driven at a timing required for measurement processing described later.

The light projecting unit 113 realizes a light projecting function having, for example, a light source or lighting. The luminous flux emitted by the light projecting unit 113 is projected onto the subject to be measured via the pattern display unit 112 and the projection optical system 111. The pattern display unit 112 is an image forming device such as a liquid crystal panel or DLP, and forms a pattern to be projected onto a subject. That is, when the pattern display unit 112 is irradiated by the light projecting unit 113, the pattern formed on the pattern display unit 112 is projected onto the subject via the projection optical system 111. In the present embodiment, the pattern display unit 112 is described as a liquid crystal element in order to make the projected pattern variable, but other elements such as a space modulator may be adopted.

The imaging unit 102 images an imaging range including the subject to be measured, and outputs an image signal related to the captured image. The image pickup unit 102 has an image pickup element 122 that may be, for example, a CCD or a CMOS sensor. The image pickup element 122 photoelectrically converts an optical image formed on the image pickup surface of the image pickup element 122 by a luminous flux incident through the image pickup optical system 121, and outputs an analog image signal. In the present embodiment, the analog image signal acquired by imaging is subjected to various image processing such as A / D conversion processing, adjustment processing, and development processing by the arithmetic unit 104, and is converted into a captured image of digital data. explain.

However, the practice of the present invention is not limited to this. For example, such image processing and conversion processing may be realized by a functional configuration (not shown) provided in the imaging unit 102, and the captured image may be output from the imaging unit 102, which is different from the arithmetic unit 104. It may be realized by a dedicated functional configuration. In other words, in the present embodiment, it is described that the function is realized by executing the program that realizes these functions by the arithmetic unit 104, but even if the hardware that realizes the function is provided. Good.

<< Measurement processing >>
In the measuring device 100 of the present embodiment having such a configuration, a specific process will be described with reference to the flowchart of FIG. 3 for a subject to be measured (a measurement process for performing various measurements on the target subject). The corresponding processing can be realized by, for example, the arithmetic unit 104 reading the corresponding processing program stored in the storage unit 105, expanding the corresponding processing program into the work memory, and executing the measurement processing. It is assumed that the operation is started when the 100 is set to the measurement mode and the operation is started.

In S301, the calculation unit 104 determines the item to be measured. In the measuring device 100 of the present embodiment, with respect to the subject existing in the imaging range, the spatial distance of a straight line or a curve connecting two points on an arbitrary plane set on the captured image, or three or more points on the plane. It is assumed that the area of the area surrounded by is provided as a measurable item. The item to be measured may be determined based on the detection of the operation input related to the selection operation by the user, preset for each mode, or determined based on the state of the subject. It may be what is done. In the following description, a mode of deriving the space distance between two points as a dimension will be described.

In S302, the calculation unit 104 determines two measurement points for dimension measurement. At the time of the determination, the captured image captured by the imaging unit 102 is displayed through the display 103, and the user can perform an operation input to specify an arbitrary measurement point in the captured image. At this time, for example, as shown in FIG. 4, a notification 402 prompting the selection of two measurement points for dimensional measurement is superimposed and displayed on the captured image including the target subject 401 which is displayed through. It shall be. When the calculation unit 104 acquires the information of the measurement point to be measured based on the operation input related to the designation, the calculation unit 104 stores the information in the work memory.

Here, when the measurement points are input, the measurement points and the straight lines connecting the two measurement points are superimposed and displayed on the captured image on the display 103 after the designation of the measurement points and the two measurement points. It may be configured to notify that the input has been made. In the example of FIG. 4, after the measurement points 403 and the measurement points 404 are designated, it is clearly shown that the straight line 405 connecting the measurement points is the target of the dimension derivation.

As described above, in the measurement process of the present embodiment, the linear distance between the two measurement points is mainly derived as a dimension, but in the embodiment in which the curved distance is selected as the measurement item, for example, FIG. Such a presentation may be made on the display 103. As shown in the figure, a notification 501 prompting the input of a curve for dimension measurement is superimposed and displayed on the captured image on the display 103, and the start point and the end point are measured by the trajectory input of the curve 502 to be measured by the user. It shall be stored as a point. In this case, the locus information is also stored. Further, in the embodiment in which the area is selected as the measurement item, the display 103 may be presented as shown in FIG. 6, for example. As shown, on the display 103, a notification 601 prompting the input of each vertex of the polygonal region for which the area is to be derived is superimposed and displayed on the captured image, and three or more vertices are input as measurement points by the user. And the area 602 connecting these is presented as an area derivation target.

In S303, the calculation unit 104 determines an image (projected image) to be projected on the projection unit 101 based on the positions of the two measurement points determined in S302. In the measuring device 100 of the present embodiment, a predetermined pattern is arranged in the area where the measuring points are distributed in the projected image in order to determine whether or not the projection image faces the plane where the measuring points actually exist in the target subject. It is configured to be. In other words, in the projected image, the predetermined pattern is arranged so as to include the region where the measurement points are distributed.

In S304, the projection unit 101 projects the projection image determined in S303 under the control of the calculation unit 104. More specifically, the pattern display unit 112 forms a projection image, and the formed projection image is irradiated by the projection unit 113, so that a pattern for facing determination is projected onto the target subject via the projection optical system 111. Will be done.

In S305, the imaging unit 102 images the target subject on which the projected image is projected under the control of the calculation unit 104, and outputs the captured image. In the captured image obtained by imaging, when the pattern is not projected so as to cover the region where the measurement points are distributed, the calculation unit 104 derives a focal length suitable for the projection optical system 111, and the pattern is derived. After making adjustments to change the focal length, projection and imaging may be performed again. In this case, in a mode in which the imaging unit 102 can derive the subject distance, the focal length of the projection optical system 111 may be adjusted based on the subject distance information.

In S306, the calculation unit 104 extracts a region corresponding to the pattern in the projected image determined in S303 from the captured image captured in S305. Here, the extraction of the region corresponding to the pattern is not limited to the extraction of the pattern itself, but includes the extraction of the region from which the similarity of the pattern and the predetermined value or more is derived. That is, since the extraction in this step is performed to determine whether or not it faces the plane on which the measurement points to be measured are distributed, depending on the orientation of the image pickup unit 102 on the plane with respect to the optical axis. The pattern may be deformed and appear in the captured image. Therefore, in this step, the calculation unit 104 extracts a region in which a projection image similar to the pattern determined from the captured image appears for the face-to-face determination.

In S307, the calculation unit 104 determines whether or not the imaging direction of the current measuring device 100 is within a range that can be regarded as facing the target subject, based on the image of the region extracted in S306. The range that can be regarded as facing each other is predetermined as a range in which the angle formed by the optical axis of the imaging unit 102 related to the imaging direction and the normal of the plane of the target subject in which the measurement point exists falls within a predetermined threshold value. It is assumed that

For example, consider a case where the dimensional measurement line segment 702 composed of the two measurement points 701a and b and the measurement device 100 are distributed as shown in FIG. 7. Here, when the measuring device 100 exists at the position 705, a plane including the dimensional measurement line segment 702 and defined by a vector from the measuring point 701a to the measuring point 701b and a vector along the optical axis of the imaging unit 102. The angle formed above is determined with reference to 703. That is, there is no optical axis of the imaging unit 102 and the normal of the plane of the target subject, and the angle θ is the angle formed by the surface normal of the target subject having the dimensional measurement line segment 702 and the plane 703 as shown in the figure. It has become. Similarly, when the measuring device 100 is present at the position 706, the angle θ formed is determined with reference to the plane 704.

Here, when θ ≠ 0, the measurement accuracy may decrease because the subject distance from the measuring device 100 differs between the measuring points as described above. On the other hand, when θ = 0, such a factor related to the decrease in accuracy does not occur, so that the measurement accuracy can be easily guaranteed. Therefore, in the measuring device 100 of the present embodiment, the measurement is performed based on the captured image obtained in the state where the plane of the target subject in which the measuring point exists and the measuring device 100 face each other (θ = 0). , Judge whether or not it falls within the range that can be regarded as facing in this step.

Since the relationship with the dimensional measurement line segment 702 does not change depending on whether the measuring device 100 is on the plane 703 or the plane 704, it does not matter which plane the measuring device 100 is on. .. Therefore, in the following description, the imaging direction is assumed to be a plane including the dimensional measurement line segment 702 and defined by a vector from the measurement point 701a to the measurement point 701b and a vector along the optical axis of the imaging unit 102. , Read as appropriate and explain.

<Drivation of deviation from the facing direction>
Hereinafter, the principle of deriving the deviation from the direction in which the current imaging direction faces is described with reference to the drawings.

As illustrated in FIG. 8, a case where the projection unit 101 and the imaging unit 102 (measurement device 100) and the plane 801 of the target subject having the measurement point are arranged will be described. In the figure, in order to facilitate understanding of the aspect in which the plane 801 is tilted relative to the measuring device 100, the tilt of the plane 801, the direction in which the dimensional measurement line segment 702 extends, and the pattern projection position are all. It is shown as being within the xz cross section. In the example of the figure, the measuring device 100 is oriented in the z direction, and the position of the imaging unit 102 is the origin of the coordinate system. The projection unit 101 and the imaging unit 102 are arranged so as to be separated by a distance d in the x-axis direction, and the optical axis of the projection optical system 111 and the optical axis of the imaging optical system 121 are parallel to each other. It is assumed that the information on the relative arrangement (relative positional relationship) between the projection unit 101 and the imaging unit 102 is stored in the storage unit 105 in advance.

In FIG. 8, the image plane 802 schematically represents the image plane of the imaging unit 102, and is distributed at the position of z = −f. Here, f is a focal length set in the imaging optical system 121 of the imaging unit 102. Further, since the plane 801 passes through z = z _ob at x = 0, the subject distance of the target subject is expressed as z _ob . At this time, when the plane 801 is distributed on the plane of z = z _ob , the plane 801 and the measuring device 100 are in a state of facing each other. Here, it is assumed that the plane 801 is inclined in the direction rotated by an angle α _y around the y-axis, and the normal 803 of the plane 801 is represented by (sin α _y , 0, cos α _y ) ^T. Further, in FIG. 8, since the dimensional measurement line segment 702 is within the xz cross section, θ = α _y .

Further, the projection unit 101 projects at least a pattern on the projection position 804 on the plane 801. If the plane 801 faces the measuring device 100, the pattern projected at the projection position 804 will be projected at the position 805. Therefore, if the plane 801 faces the measuring device 100, the image of the projected pattern is formed at the position 806 corresponding to the projected position 804 on the image plane 802 by imaging. At this time, the coordinates (X ₀ , Y ₀ , Z ₀ ) of the position 806 are the following numbers, where ω _x is the projection angle (x direction: around the y axis) with respect to the optical axis of the imaging optical system 121 at the projection position 804. It can be represented by 1.

On the other hand, since the plane 801 is actually inclined, an image of the same pattern is formed at a position corresponding to the position 805 on the image plane 802 by imaging. At this time, the coordinates (X, Y, Z) of the position 807 can be represented by the following equation 2.

As can be seen from Equation 2, if the plane 801 is tilted about the y-axis, the tilt affects only the x component of the imaging position of the image of the pattern. This is also due to the fact that the projection unit 101 and the imaging unit 102 are displaced in the x direction. Therefore, by deriving the deviation amount of the x component for at least two patterns based on the captured image acquired by the imaging unit 102, α _y can be calculated back. That is, the deviation (angle) from the facing direction can be derived.

More specifically, for example, as shown in FIG. 9, a pattern 901 in which straight lines orthogonal to the dimensional measurement line segment 702 are arranged at equal intervals is projected to derive a deviation from the facing direction. do it. In the present embodiment, when a subject on which a projected image is projected is imaged, a pattern of a straight line group is projected in order to make the pattern easily detectable, but the embodiment of the present invention is limited to this. It is not something that can be done. That is, any pattern may be projected as long as the pattern can be detected in the captured image. Here, according to Equation 2, if f, d, z, and ω _x are known, α _y can be derived if there is only one straight line projected as a pattern. However, in the case of the measuring device 100 of the present embodiment in which the subject distance z cannot be derived from one straight line pattern, at least two straight lines are required for the pattern, so that the projected image formed in S303 is also included. , Includes two or more straight lines as a pattern.

Therefore, for the projection angles ω _{x, 1} and ω _{x, 2} for the two linear patterns, the deviation amounts X ₁ and X ₂ are
Therefore, by excluding z from these equations, the deviation angle α _y from the facing direction can be derived.

Incidentally, in order to reduce the influence of the pattern detection error is not limited to the two rectilinear patterns, be one that the final alpha _y an average value derived multiple alpha _y for 3 or more straight line patterns You may. Or
Α _y that minimizes may be derived by the method of least squares or numerical operation. Here, ω _{x and i} are the projection angles of the i-th pattern, X _i is the position of the image of the i-th pattern, and N _p is the total number of straight lines included in the pattern.

The calculation unit 104 may make a determination in this step based on the deviation angle α _y from the facing direction derived in this way. That is, if, for example, the absolute value of the deviation angle α _y does not exceed the threshold value that is regarded as facing the target subject, the calculation unit 104 determines that the measuring device 100 faces the plane of the target subject. .. On the other hand, when the absolute value of the deviation angle α _y exceeds the threshold value deemed to face the target subject, the calculation unit 104 determines that the measuring device 100 does not face the plane of the target subject. When the calculation unit 104 determines that the measuring device 100 is facing the plane of the target subject, the calculation unit 104 processes the display 103 by displaying a notification 1001 indicating that the measuring device 100 is facing the plane of the target subject as shown in FIG. To S309. Further, when the calculation unit 104 determines that the measuring device 100 does not face the plane of the target subject, the calculation unit 104 shifts the processing to S308.

In S308, the calculation unit 104 presents a notification via the display 103 that the direction is deviated from the facing direction. The notification of deviation from the facing direction is performed based on the image deviation amount ΔX in the x direction in the measurement process of the present embodiment in order to facilitate the movement to a suitable imaging position. The image shift amount ΔX is the image shift amount 808 shown on the image plane 802 in FIG. 8, and corresponds to the difference between Equation 1 and Equation 2. Therefore, in the illustrated embodiment, only the x component has a non-zero value, and the image shift amount ΔX can be represented by Equation 3.

In order to derive α _y from the position X of the pattern with high accuracy, it is preferable that | ΔX | appears as a value as large as possible. Further, for the sake of simplicity, it has been described above that the plane 801 is inclined only around the y-axis and that the projection position of the pattern is only the projection angle ω _{x in the} x direction. In a mode in which the plane 801 is inclined by an angle α _x around the x-axis and the projection position also includes a projection angle ω _y in the y direction, the above equation 3 is expanded as in the equation 4. it can.

FIG. 11 is a graph plotting the change of ΔX with respect to the change of ω _x and ω _y of Equation 4. The parameter values are z = 2 [m], d = 10 [cm], α _x = 5 [deg], α _y = 5 [deg], and f = 20 [mm]. Further, when ω _x is changed, ω _y = 0, and when ω _y is changed, ω _x = 0. Although variation of is gradual, whereas, for the change of ω _x, | | ω _y to changes in | ΔX ω _x | depending on | [Delta] X | varies greatly. Therefore, at the time of measurement, it is preferable to project a pattern having an angle of view as large as possible in the arrangement direction of the projection unit 101 and the imaging unit 102. In particular, when | ω _x | ≈25 [deg], an image shift amount of about 40 [μm] is observed for | ΔX |. Therefore, it is suitable for the measurement that the parameters of the projection unit 101 and the imaging unit 102 are configured so that the image shift amount of 40 [μm] can be detected. Here, the parameters include, for example, the F value of the projection optical system 111 of the projection unit 101, the F value of the image pickup optical system 121 of the image pickup unit 102, the pixel pitch of the image sensor 122, and the like.

As shown in FIG. 12, for example, the notification based on the image shift amount ΔX is superimposed on the captured image on the display 103 to display the image 1201 when the target subject is imaged from the direction facing the subject. It may be done by. The image 1201 may be generated by projective transformation of the captured image using, for example, the deviation angle θ from the current facing direction of the measuring device 100. The superimposed display may be realized, for example, by displaying the captured image and the image 1201 on the display 103 every other pixel. In FIG. 12, only the image on the plane 801 is superimposed and displayed in consideration of the distinctiveness of the figure, but since the image 1201 is generated by converting the captured image, the pattern is imaged from the facing position. It is easy to understand that an image equivalent to a state is originally presented. Note that the notification 1202 that prompts the user to move the imaging position may be displayed on the display 103.

In addition, the notification based on the image shift amount ΔX may present quantitative information such as the image shift amount ΔX and the shift angle θ (α _y ). Alternatively, the notification may include the movement direction and movement amount of the user in the facing direction.

After the notification is given in this way, the calculation unit 104 waits for processing until an operation input for completing the movement to a different imaging position is made. Then, when the movement completion instruction is given, the calculation unit 104 causes the imaging unit 102 to perform preliminary imaging, identifies the measurement point from the captured image obtained, and then returns the process to S303. By doing so, it is possible to construct and project a projection image for facing judgment at a new imaging position for the same measurement point.

On the other hand, when it is determined in S307 that the measuring device 100 faces the plane of the target subject, the imaging unit 102 controls the arithmetic unit 104, and in S309, the captured image for deriving the dimensions between the measurement points. Take an image of the above. The imaging may be performed based on the notification by giving a notification prompting the imaging for dimension derivation, or upon determining that the image is in a facing state, the calculation unit 104 May be performed by the imaging unit 102.

In S310, the calculation unit 104 derives the dimensions between the measurement points based on the captured image captured in S309, and presents them via the display 103. Derivation of dimensions between measurement points uses Equation 4 from the relationship between the size between measurement points in the captured image and the imaging magnification of the imaging optical system 121.
Can be derived with.

Here, L is the dimension between the measurement points (spatial distance in the real world), l is the size between the measurement points in the captured image for dimension derivation, z is the subject distance on the plane of the target subject where the measurement point exists, and f is the subject distance. This is the focal length of the imaging optical system 121. l can be derived from the number of pixels on the captured image and the pixel pitch of the image sensor 122. Further, if the measuring device 100 has a subject distance detection configuration, z may be acquired from the configuration, but if it does not, z can be obtained by removing α _x and α _y from Equation 4. Can be derived.

Then, when the calculation unit 104 derives the dimensions between the measurement points, it presents them via the display 103. At this time, the calculation unit 104 may indicate that the dimensions have been derived by superimposing the measurement points and the derived dimensions on the captured image, for example, as shown in FIG.

By the way, as is clear from Equation 4, when α _y = 0, ΔX = 0, and α _x cannot be derived. That is, in such an embodiment, when the direction in which the dimensional measurement line segment 702 extends has the y component and the z component, the direction facing the opposite cannot be derived. Therefore, the direction in which the dimensional measurement line segment 702 extends needs to be the x-axis direction. In other words, the direction in which the dimensional measurement line segment 702 extends and the arrangement direction of the projection unit 101 and the imaging unit 102 (the direction connecting the optical axes of the projection optical system 111 and the imaging optical system 121) must be in the same plane. There is. Therefore, a notification may be given to urge the user to change the posture of the measuring device 100 so as to have such a relationship.

The notification is obtained by superimposing a guide display 1401 indicating a horizontal line connecting straight lines connecting measurement points as shown in FIG. 14, and a notification 1402 prompting a posture change of the measurement device 100 on an captured image. It may be done. Here, it is assumed that the guide display 1401 matches the arrangement direction of the projection unit 101 and the image pickup unit 102. By doing so, it is possible to promote imaging in a state where the imaging direction is correctly derived. Then, when the posture of the measuring device 100 is in an appropriate state, as shown in FIG. 15, information is presented via the display 103 for taking an image from the direction facing the measurement device 100 as described above. It should be done.

Further, in the present embodiment, it has been described that in S307 of the measurement process, a notification indicating that the current measuring device 100 does not face the target subject is given only when it is determined that the current imaging direction of the measuring device 100 does not face the target subject. , The practice of the present invention is not limited to this. Regarding the notification according to the imaging direction of the current measuring device 100, the notification may be given regardless of whether or not it is in a non-facing state. That is, for example, when facing each other, a notification that the imaging direction does not need to be changed may be given. In addition, the current imaging direction may be simply notified, such as the number of directions from the facing position.

As described above, according to the measuring device 100 of the present embodiment, it is possible to acquire a captured image in a state in which the measurement target is suitable for measurement based on the captured image. More specifically, in order to determine whether or not to perform measurement after determining whether or not the imaging direction faces the plane of the subject to be measured, a situation in which a low-precision measurement result is derived is described. It can be preferably avoided.

[Modification example]
By the way, when the area is selected as the measurement item, it must face the surface of the target subject. Therefore, for example, as shown in FIG. 16, the projection unit 101 and the imaging unit 102 need to be arranged in two different directions. In the example of FIG. 16, in addition to the measuring device 100 of the first embodiment, a new projection unit 1601 which may have the same configuration as the projection unit 101 is added, with respect to the arrangement direction of the imaging unit 102 and the projection unit 101. The imaging unit 102 and the projection unit 1601 are arranged so as to be orthogonal to each other. In the embodiment in which the measurement target is a region, a configuration in which both arrangement directions are orthogonal to each other as shown in the figure is preferable so that it can be used for general purposes. However, in the implementation of the invention according to the present modification, both arrangement directions are different. You just have to.

When the projection unit 101 and the projection unit 1601 project a projection image for facing determination by the measuring device 100 of the present modification having such a configuration, the result is as shown in FIG. In FIG. 17, in addition to the pattern 901 projected by the projection unit 101, the pattern 1701 projected by the projection unit 1601 is formed on the plane of the target subject. By doing so, it is possible to derive the imaging direction for two different types of directions and expand the judgment as to whether or not they are facing each other. As a result, the measurement accuracy for the two-dimensional region is also improved. It becomes possible.

In this modification, as shown in FIG. 16, a configuration example in which the measuring device 100 includes one imaging unit 102, a projection unit 101, and a projection unit 1601 has been described, but the implementation of the present invention is limited to this. It is not something that can be done. That is, it suffices if the direction connecting the optical axes of the projection optical system and the imaging optical system can be analyzed based on pattern projection with at least two different combinations, and the configuration includes one projection unit 101 and two imaging units. It is easy to understand that the same effect can be obtained as an example.

[Embodiment 2]
In the above-described embodiment and modification, the measurement points and regions existing on the same plane are notified to be imaged from the direction facing the plane in order to realize highly accurate measurement based on the captured image. Aspects have been described. In the present embodiment, a mode for realizing measurement with suitable accuracy will be described even in a mode in which measurement targets are distributed over a plurality of planes. In the description of the present embodiment, the configuration of the measuring device 100 is assumed to be a configuration capable of grasping the direction facing the plane as in the above-described modification, and the description thereof will be omitted.

<< Measurement processing >>
Hereinafter, specific processing of the measurement processing of the measuring device 100 of the present embodiment will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the arithmetic unit 104 reading, for example, the corresponding processing program stored in the storage unit 105, expanding it into the work memory, and executing it. This measurement process will be described as being started, for example, when the measuring device 100 is set to the mode for performing measurement and starts the operation. In the description of the measurement process of the present embodiment, the steps for performing the same process as that of the first embodiment are given the same reference number and the description is omitted. In the following, the process characteristic of the measurement process of the present embodiment is omitted. Only the steps to do this will be described.

Here, in the measurement process of the present embodiment, as shown in FIG. 19, measurement points 1901a and b, which are reference points for dimensional measurement, exist on two adjacent planes 1903 and 1904 having different normal directions from each other. It is assumed that there is. In addition, the dimensions of the dimensional measurement line segment 1902 between the measurement points derived by measurement are not derived as the spatial distance linearly connecting the three-dimensional spaces, but as the shortest distance along the plane group. It shall be.

When the region corresponding to the pattern in the projected image is extracted from the captured image in S306, the calculation unit 104 in S1801 determines whether the measurement points exist in different planes based on the change in the pattern appearing in the extracted region. Judge whether or not. The determination in this step may be performed including a part of the same processing as in S307, for example, by sampling the angle formed by the normal of the plane in the region and the imaging direction at a plurality of points and measuring the measurement point from the distribution of the formed angle. You just have to determine how many planes exist in between. For example, in the example of FIG. 19, θ ₁ related to the plane 1903 and θ ₂ related to the plane 1904 are derived from the pattern projected on the dimensional measurement line segment 1902 between the measurement points 1901a and b. When the calculation unit 104 determines that the measurement points exist on different planes, the processing is transferred to S1802, and when it is determined that the measurement points exist on one plane, the processing is transferred to S307.

In S1802, the calculation unit 104 selects, among the plane groups existing between the measurement points, a plane that has not been imaged from the direction facing the measurement point as the target plane. When the measurement points are distributed and exist on different planes as shown in FIG. 19, the calculation unit 104 divides the image of the region extracted in S306 into regions that can be determined to be the same plane, and in this step, Processing is performed by treating each of the divided regions as a target plane.

In S1803, the calculation unit 104 determines the projection image to be projected on the projection unit 101 with respect to the target plane selected in S1802. Then, in S1804, the projection unit 101 projects the projected image under the control of the calculation unit 104, and in S1805, the imaging unit 102 captures the target subject related to the target plane on which the projected image is projected and outputs the captured image. To do. Then, in S1806, the calculation unit 104 extracts a region corresponding to the pattern included in the captured image, and determines in S1807 whether or not the imaging direction of the measuring device 100 is within a range that can be regarded as facing the target plane. The processing of S1803 to S1807 has the same processing content as that of S303 to S307, but is performed with reference to the target plane. When the calculation unit 104 determines that the image pickup direction faces the target plane, the calculation unit 104 displays a notification indicating that the image pickup direction faces the target plane, and shifts the processing to S1808. If the calculation unit 104 determines that the imaging direction does not face the target plane, the processing is transferred to S1809, and after presenting a notification that the imaging direction deviates from the facing direction, the movement to a different imaging position is completed. The process is returned to S1803 according to the operation input.

On the other hand, when it is determined in S1807 that the measuring device 100 faces the target plane, the imaging unit 102 relates to the captured image for dimension derivation related to the target plane in S1809 under the control of the calculation unit 104. Take an image. The captured image is stored in, for example, a work memory in association with the information for identifying the target plane.

In S1810, the calculation unit 104 determines whether or not there is a plane that has not yet been selected as the target plane in the plane group existing between the measurement points. If the calculation unit 104 determines that a plane that has not yet been selected as the target plane exists, the process returns to S1802, and if it determines that the plane does not exist, the calculation unit 104 moves to S1811.

In S1811, the calculation unit 104 combines the captured images stored for each plane existing between the measurement points to generate an image (expanded image) in which the plane group between the measurement points is expanded. In the aspect shown in FIG. 19, at the time of reaching this step, the work memory stores the images 2001 and 2002 captured from the directions facing the planes 1903 and 1904 as shown in FIG. 20. In this step, the calculation unit 104 extracts the regions 2003 and 2004 related to each plane in the image stored in this way, adjusts and joins them so that the continuity of the boundary portion of the plane is ensured. A developed image as shown in FIG. 21 is generated. Here, the combination related to the developed image may be performed by aligning using the texture of the image near the boundary portion of the regions 2003 and 2004. When the imaging magnifications of the image 2001 and the image 2002 are different, a process of scaling one of the images at the time of combining is included.

In S1812, the calculation unit 104 derives the dimensions between the measurement points based on the developed image generated in S1811, and presents them via the display 103.

By doing so, even if planes having different inclinations are included between the dimension measurement line segments connecting the measurement points, according to the measurement device 100 of the present embodiment, the dimension measurement lines along the planes. A suitable imaging direction can be presented so that the length of the line segment can be derived.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Measuring device, 101: Projection unit, 102: Image pickup unit, 103: Display, 104: Calculation unit, 105: Storage unit, 106: Operation input unit, 111: Projection optical system, 112: Pattern display unit, 113: Throw Optical unit, 121: Imaging optical system, 122: Image sensor

Claims (18)

An image of an image of a subject having a plane to be measured on which a predetermined pattern is projected, information on the predetermined pattern, and a relative relationship between the projection optical system on which the predetermined pattern is projected and the imaging optical system on which the subject is imaged. An acquisition method for acquiring positional relationship information,
A determination means for determining the imaging direction of the captured image based on the captured image acquired by the acquisition means, the information on the predetermined pattern, and the information on the relative positional relationship.
A first notification means that gives notification according to the imaging direction determined by the determination means, and
A measuring device characterized by having. The determination means is characterized in that it determines whether or not the imaging direction faces the plane of the measurement target based on the state of the pattern in the region where the predetermined pattern is projected in the captured image. The measuring device according to claim 1. The determination means determines the imaging direction based on the angle formed by the normal line of the plane of the measurement target and the imaging direction derived based on the state of the pattern in the region where the predetermined pattern is projected in the captured image. The measuring device according to claim 2, wherein it is determined whether or not is facing the plane of the measurement target. In the determination means, when the angle formed by the normal of the plane of the measurement target and the imaging direction does not exceed the threshold value of the angle predetermined to be regarded as facing each other, the imaging direction is the measurement target. The measuring device according to claim 3, wherein it is determined that the surface is facing the plane of the above. The first notification means is any one of claims 2 to 4, wherein the first notification means notifies the result of determination by the determination means whether or not the imaging direction faces the plane of the measurement target. The measuring device described in. 5. The first notification means is characterized in that, when the determination means determines that the imaging direction does not face the plane of the measurement target, the first notification means notifies that the image pickup direction does not face the plane of the measurement target. The measuring device described in. The notification to the effect that they are not facing each other by the first notification means includes presenting an image of a subject and a captured image when the imaging direction and the plane of the measurement target are facing each other. The measuring device according to claim 6. When it is determined by the determination means that the imaging direction faces the plane of the measurement target, the measurement related to the plane of the measurement target is performed based on the captured image captured in the facing state. The measuring device according to any one of claims 5 to 7, further comprising a measuring means. The measuring device according to claim 8, further comprising a presenting means for presenting a measurement result by the measuring means. 5. The fifth aspect of the present invention is further comprising a second notifying means for notifying that the imaging direction is facing the plane of the measurement target when the determining means determines that the imaging direction is facing the plane of the measurement target. 9. The measuring device according to any one of 9. Selection means for selecting measurement items and
An input means that accepts an input that is a measurement point related to the measurement item and specifies a measurement point distributed on the plane of the measurement target according to the measurement item selected by the selection means.
An adjusting means that adjusts the predetermined pattern projected on the plane of the measurement target based on the input received by the input means.
The measuring device according to any one of claims 1 to 10, further comprising. The measurement item includes the length of a straight line or curve connecting the two measurement points on a plane, or the area of a region composed of at least three measurement points.
The measuring device according to claim 11, wherein the adjusting means adjusts the predetermined pattern so as to include a region in which the designated measuring points are distributed. The adjusting means is characterized in that when a straight line connecting two measurement points is selected as the measurement item by the selection means, the predetermined pattern is adjusted so as to include a plurality of straight lines orthogonal to the straight line. The measuring device according to claim 12. When a straight line connecting two measurement points is selected as the measurement item by the selection means, imaging is promoted so that the straight line extends in the arrangement direction of the projection optical system and the imaging optical system in the captured image. The measuring device according to claim 12 or 13, further comprising a third notifying means for giving notification. When the specified plurality of measurement points are distributed on different planes,
The adjusting means adjusts the predetermined pattern for each of the planes existing between the plurality of measurement points.
One of claims 11 to 14, wherein the determination means determines whether or not the imaging direction faces the plane for each of the planes existing between the plurality of measurement points. The measuring device described in. The measuring device according to any one of claims 1 to 15.
A projection means for projecting the predetermined pattern onto the subject via the projection optical system,
An imaging means that images the subject via the imaging optical system and outputs the captured image.
An imaging device characterized by having. The measuring device according to any one of claims 1 to 15.
A projection device that projects the predetermined pattern onto the subject via the projection optical system, and
An imaging device that images the subject via the imaging optical system and outputs the captured image.
A measurement system characterized by having. A program for causing a computer to function as each means of the measuring device according to any one of claims 1 to 15.