JP2023128043A - Displacement measurement device and displacement measurement method - Google Patents

Abstract

To provide a displacement measurement device and a displacement measurement method that can measure displacement of a plurality of markers, that is, a plurality of measurement points at a low cost.SOLUTION: A marker image creation unit 30 extracts, for a panorama image 25#1 obtained through imaging at a time point #1 and a panorama image 25#2 obtained through imaging at a time point #2, a plurality of known marker areas where a plurality of markers are present to create a plurality of marker images R#1, M1#1 to Mn#1, R#2, M1#2 to Mn#2. A marker image analysis unit 28a calculates the amount of displacement D1 (x, y) to Dn (x, y) for each of the plurality of markers generated between the time point #1 and the time point #2 based on the plurality of marker images R#1, M1#1 to Mn#1 created from the panorama image 25#1, and the plurality of marker images R#2, M1#2 to Mn#2 created from the panorama image 25#2.SELECTED DRAWING: Figure 5

Description

The present invention relates to a displacement measuring device and a displacement measuring method.

Patent Document 1 discloses a method of measuring dimensions without providing shape data of a target space based on an omnidirectional image taken from a single viewpoint taken from a limited opening such as an inspection port. Specifically, a panoramic camera is used to take omnidirectional photographs, information about vertical or horizontal lines in the target space is given to adjust the level, and the results are projected onto a regular hexahedron to create a cubic panoramic image. Three-dimensional coordinate measurement is performed using the synthesized cubic panoramic image and camera height information.

Patent Document 2 discloses a method that can suppress unnecessary measurements compared to the case where the distance to the subject is measured once for each of all the images required for one panoramic image capture. Specifically, in panoramic imaging, directional light is emitted to a subject within a specific imaging range among multiple imaging ranges, and the distance to the subject is measured by receiving the reflected light, and the measurement is successful. In this case, measurements targeting subjects within multiple imaging ranges are prohibited.

Patent Document 3 discloses that time-series image data obtained by photographing a predetermined pattern attached to a columnar structure is input, the displacement occurring in the columnar structure is determined from the time-series image data, and the displacement of the columnar structure is determined from the displacement. A method of determining the natural frequency of the columnar structure and determining the state of the columnar structure based on the natural frequency is shown. When determining the displacement occurring in the columnar structure, a digital image correlation method or a moire fringe phase analysis method is used.

Patent Document 4 discloses a method that makes it possible to reduce errors due to the inclination of the target surface or the measurement direction when measuring the displacement of a measurement point on the target surface using the sampling moiré method. Further, Patent Document 5 discloses a method of measuring in-plane displacement and out-of-plane displacement of an object using a sampling moiré method based on an image obtained from a single camera.

Japanese Patent Application Publication No. 2015-125002 International Publication No. 2017/149850 Japanese Patent Application Publication No. 2018-141663 Japanese Patent Application Publication No. 2019-11984 International Publication No. 2017/029905

In recent years, technology for measuring the displacement of an object based on images captured by an imaging device has been developed. In this technology, the displacement of the marker and, by extension, the object is calculated by capturing an image of a marker attached to an object and performing various image processing on the captured image. By using such technology, compared to using a dedicated surveying device such as a total station, equipment costs can be reduced and work efficiency can be improved due to automation. Furthermore, various advantages can be obtained, such as being able to record measurement results in the form of images as well as numerical values, being able to perform constant measurements with an infrared camera, and being able to use the imaging device as a surveillance camera.

Here, for example, at a construction site or the like, there are cases where it is desired to measure displacement at a plurality of measurement points on a single object or a plurality of objects. In this case, it is sufficient to image a plurality of markers attached to a plurality of measurement points, respectively. On the other hand, in order to capture images of multiple markers, it is generally necessary to increase the shooting distance. In this case, there was a risk that the angle of view would become narrower or that the resolution of the photographing surface would decrease. Therefore, a method of distributing a plurality of markers to a plurality of imaging devices and capturing images may be considered. However, in this case, there was a risk of an increase in cost.

The present invention has been made in view of the above, and one of its objects is to provide a displacement measuring device and a displacement measuring method that can measure the displacement of multiple markers and, by extension, multiple measurement points at low cost. It is about providing.

A brief overview of typical inventions disclosed in this application is as follows.

A displacement measuring device according to a typical embodiment of the present invention includes an imaging device that images a marker, and a displacement measuring device that measures the displacement of the marker based on an image captured by the imaging device. An imaging device images a plurality of markers attached to a single object or a plurality of objects, and creates a panoramic image including a plurality of markers in one image by composing the plurality of captured images. The displacement measuring device includes a marker image creation section and a marker image analysis section. The marker image creation unit includes a plurality of markers for a first panoramic image obtained by imaging at a first time point and a second panoramic image obtained by imaging at a second time point. By extracting a plurality of known marker regions, an image in which a plurality of markers are photographed is created. The marker image analysis unit determines the first point in time and the second point in time based on the plurality of marker images created from the first panoramic image and the plurality of marker images created from the second panoramic image. The amount of displacement for each of the plurality of markers that occurred during that time is calculated.

Of the inventions disclosed in this application, the effects obtained by typical ones will be briefly described. Displacements of a plurality of markers and, by extension, a plurality of measurement points can be measured at low cost.

1 is a schematic diagram showing a configuration example of a displacement measurement system according to Embodiment 1 of the present invention. FIG. 2 is a schematic diagram showing a configuration example of a marker in FIG. 1. FIG. FIG. 2 is a block diagram showing a schematic configuration example of the displacement measuring device in FIG. 1. FIG. 4 is a schematic diagram showing an example of a panoramic image captured by the imaging device in FIG. 3. FIG. FIG. 4 is a block diagram showing a schematic configuration example of main parts of the displacement measuring instrument in FIG. 3. FIG. 6 is a block diagram showing a configuration example of a sampling moiré section in FIG. 5. FIG. FIG. 7 is a schematic diagram illustrating an example of processing contents of the phase detection section in FIG. 6; FIG. 2 is a schematic diagram showing a configuration example of a marker in FIG. 1 in a displacement measuring device according to a second embodiment. 4 is a block diagram illustrating a schematic configuration example of a main part of a displacement measuring device in FIG. 3 in a displacement measuring device according to a second embodiment. FIG. 10 is a block diagram showing a configuration example of a phase-only correlation section in FIG. 9. FIG. FIG. 2 is a schematic diagram showing an example of the configuration of a marker in FIG. 1 in a displacement measuring device according to a third embodiment. 4 is a block diagram illustrating a schematic configuration example of a main part of a displacement measuring device in FIG. 3 in a displacement measuring device according to a third embodiment. FIG. FIG. 13 is a conceptual diagram illustrating an example of processing contents of a displacement amount calculation unit in FIG. 12. FIG.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings. It should be noted that the disclosure is merely an example, and any modifications that can be easily made by those skilled in the art while maintaining the gist of the invention are naturally included within the scope of the present invention. In addition, in order to make the explanation clearer, the drawings may schematically represent the width, thickness, shape, etc. of each part compared to the actual aspect, but these are only examples, and the interpretation of the present invention is It is not limited.

In addition, in this specification and each figure, the same elements as those described above with respect to the previously shown figures are denoted by the same reference numerals, and detailed explanations may be omitted as appropriate.

(Embodiment 1)
<Outline of displacement measurement system>
FIG. 1 is a schematic diagram showing a configuration example of a displacement measurement system according to Embodiment 1 of the present invention. FIG. 2 is a schematic diagram showing an example of the structure of the marker in FIG. 1. The displacement measurement system shown in FIG. 1 includes a plurality of markers 1r, 1m[1] to 1m[n] attached to one or more objects 3[0] to 3[n], and a displacement measurement device including an imaging device. 2. In the specification, the plurality of markers 1r, 1m[1] to 1m[n] are collectively referred to as marker 1. The objects 3[0] to 3[n] are, for example, buildings installed on the ground at a construction site such as shaft construction, or the ground itself.

The displacement measuring device 2 images the marker 1 using an imaging device, and measures the displacement of the marker 1 and, by extension, the displacement of the building, the ground, etc., based on the captured image. This makes it possible to successively monitor the displacement status of buildings, the ground, etc., and for example, to detect early signs of accidents such as collapse of buildings and landslides. As a result, accidents can be prevented and worker safety can be ensured.

The marker 1r is a reference marker attached to a measurement point treated as a fixed point. On the other hand, markers 1m[1] to 1m[n] excluding the reference marker 1r are measurement markers attached to actual measurement points. For example, there is a possibility that a displacement occurs not in the marker 1 but in the installation angle of the imaging device in the displacement measuring device 2, etc. By providing the reference marker 1r, it becomes possible to detect such displacement of the imaging device itself. However, if it can be guaranteed that the imaging device itself will not be displaced, all the markers 1r, 1m[1] to 1m[n] may be treated as measurement markers.

Here, the imaging device in the displacement measuring device 2 is a panoramic camera capable of panoramic photography. As a result, the imaging device images the plurality of markers 1r, 1m[1] to 1m[n], and creates a panoramic image including the plurality of markers 1r, 1m[1] to 1m[n] in one image. Create an image. In general, a panoramic camera, for example, images multiple imaging areas so that some areas overlap while switching the imaging area, and stitches the multiple images obtained from the multiple imaging areas. A single panoramic image is created by combining the images through processing, for example by joining them together. As a result, the panoramic image becomes an image obtained by capturing a wide range of angle of view, for example, exceeding 180°, without reducing the resolution.

In the specification, as shown in FIG. 1, the optical axis direction of the imaging device in the displacement measuring device 2 is defined as the Z axis, and one direction, in this case, the horizontal direction, is defined as the X axis in the plane direction of the plane perpendicular to the Z axis. The direction perpendicular to the one direction, in this case the vertical direction, is the Y-axis. The plurality of markers 1r, 1m[1] to 1m[n] are attached so that the XY plane is imaged. For example, a pattern as shown in FIG. 2 is written on the XY plane of each of the plurality of markers 1r, 1m[1] to 1m[n].

A periodic pattern is written on the marker 1A shown in FIG. By using the periodic pattern, the displacement of the marker can be measured using the sampling moiré method, which will be described later. In this example, the periodic pattern is a lattice pattern, and black level squares are arranged at regular intervals with white level squares in between in the X-axis direction and the Y-axis direction. The pitch of the black level or white level squares in the X-axis direction is Wx [mm], the pitch in the Y-axis direction is Wy [mm], and the pitches Wx and Wy are, for example, several mm to several tens of mm. It's good.

Further, the lattice pattern shown in FIG. 2 is substantially equivalent to a striped pattern arranged in the X-axis direction or the Y-axis direction. That is, by performing image processing on the grid pattern, specifically averaging processing along the Y-axis direction, it can be converted into a striped pattern lined up in the X-axis direction. Similarly, by performing image processing on the grid pattern, specifically averaging processing along the X-axis direction, it can be converted into a striped pattern arranged in the Y-axis direction. Note that the periodic pattern is not limited to a lattice pattern, and may be such a striped pattern.

<Outline of displacement measuring device>
FIG. 3 is a block diagram showing a schematic configuration example of the displacement measuring device in FIG. 1. As shown in FIG. The displacement measuring device 2 shown in FIG. 3 includes an imaging device 10 that is a panoramic camera, and a displacement measuring device 20 that is realized by an information processing device such as a PC (Personal Computer) or a dedicated image processing device. Be prepared. However, the imaging device 10 and the displacement measuring device 20 may be implemented in the same device, for example, in the form of an information processing device with a panoramic camera. As described above, the imaging device 10 images a plurality of markers 1 and creates a panoramic image including the plurality of markers 1 in one image. The displacement measuring device 20 measures the amount of displacement for each of the plurality of markers 1, and further for each of the plurality of measurement points, based on the panoramic image created by the imaging device 10.

The imaging device 10 includes a lens 11, an image sensor 12, a computing unit 13, an internal memory 14, a communication interface 15, and a panoramic mechanism 16. Arithmetic unit 13, internal memory 14, and communication interface 15 are connected to each other via a bus. The arithmetic unit 13, internal memory 14, and communication interface 15 may be implemented in, for example, one microcontroller.

Lens 11 focuses light from the imaging area onto image sensor 12 . The image sensor 12 is typically a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, or the like, and includes a plurality of pixels arranged in an array. Each pixel of the image sensor 12 generates an electrical signal according to the amount of light focused by the lens 11. The image sensor 12 transmits electrical signals generated at each pixel to the arithmetic unit 13.

The internal memory 14 is, for example, a nonvolatile memory such as a flash memory, and corresponds to a built-in memory in a microcontroller or an external memory such as a memory card. The panorama mechanism 16 switches the imaging area by, for example, mechanically moving the lens 11, the imaging device 10 itself, etc., in response to instructions from the computing unit 13. Note that the method of the panoramic camera is not limited to such a mechanically moving method, and may be any of various known methods including, for example, a method including a plurality of lenses.

The arithmetic unit 13 includes a processor 13a such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor), and a RAM (Random Access Memory) 13b. The processor 13a executes various processes necessary for creating a panoramic image, for example, by executing a program loaded from the internal memory 14 into the RAM 13b. Specifically, the processor 13a creates an image for each imaging area based on the electrical signal from the image sensor 12 while switching the imaging area via the panorama mechanism 16. Further, the processor 13a creates one panoramic image by stitching the images of each imaging area, and stores the panoramic image in the internal memory 14.

The communication interface 15 sends and receives data to and from the displacement measuring device 20, specifically, the communication interface 21 therein. As one example, the communication interface 15 transmits the panoramic image stored in the internal memory 14 to the displacement measuring device 20. Communication interface 15 and communication interface 21 are connected by wire or wirelessly. In this case, for example, a configuration in which the connection is made via an external network such as the Internet may be used.

When using an external network, for example, an imaging device 10 equipped with a communication interface 15 for wireless communication may be fixedly installed at a construction site, and a displacement measuring device 20 may be mounted on an in-house server device of a construction company. The form is beneficial. In this case, the imaging device 10 can sequentially transmit the created panoramic images to the in-house server device via the external network, and the in-house server device can perform displacement measurement based on the panoramic images.

The displacement measuring device 20 includes a computing unit 22, an internal memory 23, and a communication interface 21. Arithmetic unit 22, internal memory 23, and communication interface 21 are connected to each other via a bus. For example, when the displacement measuring device 20 is configured with a dedicated image processing device or the like, the arithmetic unit 22, internal memory 23, and communication interface 21 may be implemented in one microcontroller. The internal memory 23 is, for example, a nonvolatile memory such as a flash memory or a hard disk drive. The communication interface 21 receives, for example, a panoramic image from the communication interface 15 of the imaging device 10 and stores it in the internal memory 23.

The arithmetic unit 22 includes a processor 22a such as a CPU, GPU, or DSP, and a RAM 22b. The computing unit 22 calculates the amount of displacement for each of the plurality of markers 1, for example, by performing predetermined image processing on the panoramic image stored in the internal memory 23. At this time, the processor 22a calculates the amount of displacement, for example, by executing a displacement measurement program loaded from the internal memory 23 into the RAM 22b. Note that the arithmetic unit 22 is not limited to the processor 22a, but may be partially or entirely configured with hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). That is, the arithmetic unit 22 may be configured as appropriate using a software method, a hardware method, or a combination thereof. This also applies to the arithmetic unit 13 within the imaging device 10.

<Outline of displacement measuring instrument>
FIG. 4 is a schematic diagram showing an example of a panoramic image captured by the imaging device in FIG. 3. The panoramic image 25 shown in FIG. 4 includes a plurality of markers 1, that is, a reference marker 1r and measurement markers 1m[1] to 1m[n] in one image. The panoramic image 25 is created, for example, by combining captured images 26[0] to 26[n] of a plurality of imaging areas set so that some areas overlap.

FIG. 5 is a block diagram showing a schematic configuration example of the main parts of the displacement measuring instrument in FIG. 3. As shown in FIG. The displacement measuring device 20 shown in FIG. 5 includes a marker image creation section 30 and a marker image analysis section 28a. Each of these parts is realized, for example, by the processor 22a executing a displacement measurement program. In addition, the internal memory 23 in the displacement measuring device 20 stores a panoramic image 25#1 obtained by imaging at time point #1 and a panoramic image 25#2 obtained by imaging at a subsequent time point #2. Panoramic images 25 at each point in time are stored.

The marker image creation unit 30 creates a plurality of marker images by extracting a plurality of known marker regions in which a plurality of markers 1 exist, from the plurality of panoramic images 25 stored in the internal memory 23 . That is, the marker region where marker 1 exists in panoramic image 25 is fixedly determined in advance based on the positional relationship at the stage of constructing the displacement measurement system shown in FIG. In the example shown in FIG. 4, for each marker 1r, 1m[1] to 1m[n], a pixel range slightly inside the outer frame is fixedly determined as a marker area, and an image of this marker area is extracted. In this way, a marker image 27 is created.

The marker image creation unit 30 stores the created marker image 27 in, for example, the RAM 22b. The created marker image 27 includes a reference marker image R#1 and measurement marker images M1#1 to Mn#1 at time point #1, and a reference marker image R#2 and measurement marker image M1# at time point #2. 2 to Mn#2 are included. For example, the reference marker image R#1 and the measurement marker images M1#1 to Mn#1 are the reference marker 1r and the measurement markers 1m[1] to 1m[n] from the panoramic image 25#1 at time point #1. It is created by extracting each marker area.

In the specification, reference marker images R#1 and R#2 are collectively referred to as reference marker image R, or simply marker image R. Furthermore, the measurement marker images M1#1 and M1#2 are collectively referred to as the measurement marker image M1 or simply the marker image M1. Similarly, measurement marker images Mn#1 and Mn#2 are collectively referred to as measurement marker image Mn, or simply marker image Mn.

The marker image analysis unit 28a includes a plurality of marker images R#1, M1#1 to Mn#1 created from the panoramic image 25#1, a plurality of marker images R#2, created from the panoramic image 25#2, Based on M1#2 to Mn#2, the amount of displacement for each of the plurality of markers 1 occurring between time point #1 and time point #2 is calculated. Specifically, the marker image analysis section 28a includes a sampling moire section 31 and a displacement amount calculation section 32.

The sampling moire unit 31 uses a sampling moire method to determine the phase of the moire fringe, in this case PH#1, for each of the plurality of marker images R#1, M1#1 to Mn#1 created from the panoramic image 25#1. to detect. Similarly, the sampling moire unit 31 detects the phase of the moire fringe, which is PH#2 in this case, for each of the plurality of marker images R#2, M1#2 to Mn#2 created from the panoramic image 25#2. do. Based on the difference between the detected phases PH#1 and PH#2, the sampling moiré unit 31 extracts each of the plurality of marker images R, M1 to Mn that occurred between time point #1 and time point #2. Calculate the phase difference of moire fringes.

That is, the sampling moire unit 31 calculates the phase difference ΔPHr(x,y) generated in the reference marker image R and the phase difference ΔPHm1(x,y) to ΔPHmn(x,y) generated in the measurement marker images M1 to Mn. Calculate. Each of the phase differences ΔPHr (x, y), ΔPHm1 (x, y) to ΔPHmn (x, y) represents the phase difference ΔPH (x) in the X-axis direction and the phase difference ΔPH (y) in the Y-axis direction. include.

The displacement calculation unit 32 calculates the phase differences ΔPHr (x, y), ΔPHm1 (x, y) to ΔPHmn (x, y) of the moire fringes for each of the marker images R, M1 to Mn calculated by the sampling moire unit 31 [ rad], the displacement amount for each marker 1, for example, the displacement amount D1 (x, y) to Dn (x, y) [mm] for each measurement marker 1 m [1] to 1 m [n] is calculated. Each of the displacement amounts D1(x,y) to Dn(x,y) includes a displacement amount D(x) in the X-axis direction and a displacement amount D(y) in the Y-axis direction.

Specifically, the phase difference ΔPH (x) [rad] in the X-axis direction and the phase difference ΔPH (y) [rad] in the Y-axis direction from the sampling moiré section 31 and the X calculated by the displacement calculation section 32 The axial displacement amount D(x) [mm] and the Y-axis displacement amount D(y) [mm] have the relationship shown in equation (1) and equation (2). In equations (1) and (2), Wx [mm] and Wy [mm] are the pitches on the marker 1A shown in FIG. 2, respectively. The displacement calculation unit 32 calculates the displacement D1 (x, y) to Dn (x, y) for each measurement marker 1 m [1] to 1 m [n] using equations (1) and (2). do.
D(x)=(ΔPH(x)/2π)×Wx…(1)
D(y)=(ΔPH(y)/2π)×Wy...(2)

However, at this time, the displacement calculation unit 32 calculates the displacement D1(x,y) to Dn(x,y) for each measurement marker 1m[1] to 1m[n] as the displacement of the reference marker 1r. Calculate using the difference value. In detail, the displacement calculation unit 32 calculates, for example, the difference value between the phase difference ΔPHm1 (x, y) generated at the measurement marker 1m[1] and the phase difference ΔPHr (x, y) generated at the reference marker 1r. “ΔPHm1(x,y)−ΔPHr(x,y)” is regarded as the phase difference generated at the measurement marker 1m[1]. Then, the displacement calculation unit 32 calculates the displacement D1 (x, y) of the measurement marker 1m[1] by substituting the considered phase difference into Equation (1) and Equation (2).

Similarly, the displacement calculation unit 32 measures the difference value between the phase difference ΔPHmn(x,y) generated at the measurement marker 1m[n] and the phase difference ΔPHr(x,y) generated at the reference marker 1r. The displacement amount Dn(x, y) of the measurement marker 1m[n] is calculated by regarding it as the phase difference generated at the marker 1m[n]. In this way, by calculating the phase difference between the measurement markers 1m[1] to 1m[n], and thus the amount of displacement, using the difference value from the phase difference of the reference marker 1r and, ultimately, the amount of displacement, as described in Fig. 1, In addition, the displacement of the imaging device 10 can be detected, and the displacement of each measurement marker can be measured after reflecting the displacement of the imaging device 10.

<Details of sampling moiré part>
FIG. 6 is a block diagram showing an example of the configuration of the sampling moiré section in FIG. 5. FIG. 7 is a schematic diagram illustrating an example of the processing contents of the phase detection section in FIG. 6. The sampling moire section 31 shown in FIG. 6 includes a phase detection section 35 and a phase difference calculation section 36. The phase detection unit 35 uses the sampling moiré method to detect the phase PH(x,y)#1 of the moire fringes for the marker image 27#1 at time point #1, and similarly detects the phase PH(x,y)#1 of the moire fringe at time point #2. The phase PH(x,y)#2 of the moiré fringe is detected for the marker image 27#2. The marker image 27#1 corresponds to one of the reference marker image R#1 and the measurement marker images M1#1 to Mn#1 at the time point #1, and the marker image 27#2 corresponds to the reference marker image R#1 at the time point #1 and one of the measurement marker images M1#1 to Mn#1. This corresponds to one of the reference marker image R#2 and the measurement marker images M1#2 to Mn#2.

Here, each of the phases PH(x,y)#1 and PH(x,y)#2 includes a phase PH(x) in the X-axis direction and a phase PH(y) in the Y-axis direction. The phase PH(x) in the X-axis direction is obtained from the periodic patterns arranged in the X-axis direction, and the phase PH(y) in the Y-axis direction is obtained from the periodic patterns arranged in the Y-axis direction. As described in FIG. 2, when detecting the phase PH(x) in the X-axis direction, the phase detection unit 35 converts the grid pattern represented by the marker image 27 into a striped pattern arranged in the X-axis direction as a preprocessing. do. Similarly, when detecting the phase PH(y) in the Y-axis direction, the phase detection unit 35 converts the lattice pattern represented by the marker image 27 into a striped pattern arranged in the Y-axis direction as preprocessing.

FIG. 7 shows the principle of the sampling moiré method used in the phase detection section 35. In FIG. 7, an imaging device 10 having a plurality of pixels P arranged at a pitch p creates a marker image 27 including a periodic pattern by imaging the marker 1A. The phase detection unit 35 detects the phase of the moire fringes by performing the following processing on this periodic pattern based on the sampling moire method.

First, the phase detection unit 35 creates a sampling image by sampling pixels P at predetermined intervals, typically one every four pixels, for the periodic pattern in the marker image 27. At this time, the phase detection unit 35 creates four sampling images 40[0] to 40[3] by sequentially shifting the sampling pixel positions.

Next, the phase detection unit 35 interpolates each of the four sampling images 40[0] to 40[3] to obtain four moire images 41[0] to 41 including moiré fringes. Create [3]. Next, the phase detection unit 35 uses the formula ( 3) is calculated. In equation (3), I0, I1, I2, and I3 are the brightness values of the moire images 41[0], 41[1], 41[2], and 41[3] at each pixel P, respectively.
tan(PH)=-(I3-I1)/(I2-I0)...(3)

Here, the brightness value In of each pixel P in the moire image 41[n] (n=0, 1, 2, 3) is generally calculated using the formula ( 4). In equation (4), if the unknowns A0 and B0 are known, it is possible to determine the phase PH. Therefore, in the sampling moire method, for example, as shown in FIG. 7, by creating four moire images 41[0] to 41[3] whose phases are sequentially shifted by π/2, brightness values I0 to I3 can be obtained. If four luminance values I0 to I3 can be obtained, four simultaneous equations can be created based on equation (4). Equation (3) is an equation for determining the phase PH from these four simultaneous equations.
In=A0×cos(PH+(n×π/2))+B0…(4)

Using such a sampling moiré method, the phase detection unit 35 detects the phase PH(x,y)#1 for each marker image 27#1 at time point #1 for each plurality of pixels P, Phase PH(x,y)#2 is detected for each marker image 27#2 at time point #2. As shown in FIG. 7, the method of creating a plurality of grating images (moiré images in this case) whose phases are sequentially shifted and determining the phase PH of the gratings (here moiré fringes) using equation (3) etc. is based on the phase shift method. Also called shift method. The sampling moire method utilizes this phase shift method and uses a moire image created by sampling pixels P as a grid image.

Returning to FIG. 6, the phase difference calculation section 36 includes an average value calculation section 37. The phase difference calculation unit 36 receives the input from the phase detection unit 35 and calculates the difference between the phase PH(x,y)#1 at time point #1 and the phase PH(x,y)#2 at time point #2. Calculate the phase difference. Specifically, the phase difference calculation unit 36 calculates such a phase difference for each of the plurality of pixels P using equation (3). The average value calculation unit 37 calculates the average value of the phase differences obtained from the plurality of pixels P, and outputs the calculated phase difference ΔPH (x, y) to the displacement amount calculation unit 32 shown in FIG.

Note that the sampling moiré method is generally used to measure in-plane displacement in the mounting area of the marker 1, in other words, the deformation state of an object, etc., based on the phase difference for each pixel P. On the other hand, for example, at a construction site such as shaft construction, when measuring the displacement of a structure or the ground, information on the in-plane displacement of the marker 1 is not particularly required, and the displacement information of the marker 1 as a whole, and even more , movement information of the object itself is often required. In order to measure the displacement of the marker 1 as a whole in this way, in this example, the average value calculation unit 37 is used to calculate the average value. Furthermore, the lower the resolution of the marker image 27, the lower the accuracy and resolution of displacement measurement, so the marker image 27 needs to have a somewhat high resolution. For this reason, it is beneficial to create a panoramic image 25 as shown in FIG.

<Main effects of Embodiment 1>
As described above, by using the method of Embodiment 1, it is typically possible to measure the displacements of a plurality of markers 1 and, by extension, a plurality of measurement points at low cost. Specifically, in the method of the first embodiment, the imaging device 10 creates a panoramic image 25 including a plurality of markers 1 in one image, and the displacement measuring device 20 uses the panoramic image 25 to The displacement of each of the plurality of markers 1 can be measured based on the extracted plurality of marker images 27 having sufficient resolution. As a result, there is no need to provide a plurality of imaging devices to obtain sufficient resolution, and by providing one imaging device 10, it becomes possible to perform high-resolution or highly accurate displacement measurement.

Furthermore, in the method of the first embodiment, a reference marker 1r is provided, and the displacement amount for each measurement marker 1m[1] to 1m[n] is calculated by the difference value from the displacement amount of the reference marker 1r. As a result, it becomes possible to reflect the displacement of the imaging device 10 and measure the displacement of each measurement marker 1m[1] to 1m[n]. At this time, the reference marker image R and the measurement marker images M1 to Mn are not images taken by different imaging devices, but are images taken by panoramic photography by one imaging device 10, that is, substantially Since the images were captured under the same conditions as before, a situation in which the displacement of the imaging device 10 is erroneously determined is unlikely to occur.

(Embodiment 2)
<Outline of displacement measuring instrument>
FIG. 8 is a schematic diagram showing a configuration example of the marker in FIG. 1 in the displacement measuring device according to the second embodiment. A geometric pattern is written on the marker 1B shown in FIG. By using the geometric pattern, the displacement of the marker can be measured using a phase-only correlation method, which will be described later. In this example, the geometric pattern is six circles arranged at 60° intervals. However, the geometric pattern is not limited to this, and may be any of various commonly known patterns, especially if it is a pattern that does not include periodic components in the X- and Y-axis directions. good.

FIG. 9 is a block diagram showing a schematic configuration example of the main part of the displacement measuring device in FIG. 3 in the displacement measuring device according to the second embodiment. The displacement measuring device 20 shown in FIG. 9 includes a marker image creation section 30 and a marker image analysis section 28b. As in the case of FIG. 5, the marker image creation unit 30 extracts a plurality of known marker areas from the panoramic images 25#1, 25#2, . . . stored in the internal memory 23, thereby creating a plurality of Create a marker image. However, in FIG. 4, the marker image 27 of the reference marker 1r and measurement markers 1m[1] to 1m[n] included in the panoramic image 25 changes from the periodic pattern shown in FIG. 2 to the geometric pattern shown in FIG. It will be replaced.

The marker image analysis unit 28b, as in the case of FIG. Based on the plurality of marker images R#2, M1#2 to Mn#2, the amount of displacement of each of the plurality of markers 1 occurring between time point #1 and time point #2 is calculated. However, unlike the case in FIG. 5, the marker image analysis section 28b includes a phase-only correlation section 45 and a displacement amount calculation section 46.

The phase-only correlation unit 45 generates a plurality of marker images R#1, M1#1 to Mn#1 created from the panoramic image 25#1, and a plurality of marker images R#2, created from the panoramic image 25#2. The correlations with M1#2 to Mn#2 are calculated using the phase-only correlation method. Thereby, the phase only correlation unit 45 calculates the shift amount for each of the plurality of marker images R, M1 to Mn that occurred between time point #1 and time point #2.

That is, the phase-only correlation unit 45 calculates the shift amount ΔSFr (x, y) that occurred in the reference marker image R and the shift amount ΔSFm1 (x, y) to ΔSFmn (x, y) that occurred in the measurement marker images M1 to Mn. Calculate. Each of the shift amounts ΔSFr(x, y), ΔSFm1(x, y) to ΔSFmn(x, y) represents the shift amount ΔSF(x) in the X-axis direction and the shift amount ΔSF(y) in the Y-axis direction. include.

The displacement calculation unit 46 calculates the shift amounts ΔSFr (x, y), ΔSFm1 (x, y) to ΔSFmn (x, y) [pixel ( px)], calculate the displacement amount for each marker 1, for example, the displacement amount D1 (x, y) to Dn (x, y) [mm] for each measurement marker 1 m [1] to 1 m [n]. . Each of the displacement amounts D1(x,y) to Dn(x,y) includes a displacement amount D(x) in the X-axis direction and a displacement amount D(y) in the Y-axis direction.

Specifically, the shift amount ΔSF(x) [px] in the X-axis direction and the shift amount ΔSF(y) [px] in the Y-axis direction from the phase-only correlation unit 45 are calculated by the displacement amount calculation unit 46. The displacement amount D(x) [mm] in the X-axis direction and the displacement amount D(y) [mm] in the Y-axis direction have the relationship shown in Equation (5) and Equation (6). In equations (5) and (6), Lx [mm] and Ly [mm] are the length in the X-axis direction and the length in the Y-axis direction on the marker 1 corresponding to one pixel of the image sensor 12. , is fixedly determined based on the number of pixels of the image sensor 12, the setting of the optical magnification at the time of imaging, and the like. The displacement calculation unit 46 calculates the displacement D1 (x, y) to Dn (x, y) for each measurement marker 1 m [1] to 1 m [n] using equations (5) and (6). do.
D(x)=ΔSF(x)×Lx…(5)
D(y)=ΔSF(y)×Ly...(6)

However, at this time, the displacement calculation unit 46 calculates the displacement D1(x,y) to Dn(x,y) for each measurement marker 1m[1] to 1m[n], as in the case of FIG. , and the displacement amount of the reference marker 1r. In detail, the displacement calculation unit 46 calculates, for example, the difference value between the shift amount ΔSFm1 (x, y) generated at the measurement marker 1m[1] and the shift amount ΔSFr (x, y) generated at the reference marker 1r. “ΔSFm1(x,y)−ΔSFr(x,y)” is regarded as the amount of shift caused by the measurement marker 1m[1]. Then, the displacement calculation unit 46 calculates the displacement D1 (x, y) of the measurement marker 1m[1] by substituting the considered shift amount into equations (5) and (6). The same applies to the other measurement markers 1m[n].

<Details of phase-only correlation section>
FIG. 10 is a block diagram showing a configuration example of the phase-only correlation section in FIG. 9. A method is known in which the degree of matching between a plurality of images is determined by correlating frequency components of the images. When determining the frequency components of an image, information about the shape of the image within the image is generally held by the phase spectrum rather than the amplitude spectrum. Utilizing this, in the phase-only correlation method, when determining the correlation between images, the amplitude spectrum of the frequency component of the image is normalized to 1, for example, and then the correlation is determined mainly based on the phase spectrum.

The phase-only correlation unit 45 shown in FIG. 10 includes a Fourier transform unit 50, a composite function calculation unit 51, and a correlation function calculation unit 52. Image data f1 (m, n) of marker image 27#1 at time point #1 and image data f2 (m, n) of marker image 27#2 at time point #2 are input to the Fourier transform unit 50. be done. The marker image 27#1 corresponds to one of the reference marker image R#1 and the measurement marker images M1#1 to Mn#1 at the time point #1, and the marker image 27#2 corresponds to the reference marker image R#1 at the time point #1 and one of the measurement marker images M1#1 to Mn#1. This corresponds to one of the reference marker image R#2 and the measurement marker images M1#2 to Mn#2. Here, assuming that the pixel area of the marker image 27 is M×N pixels, m is one of M integers, and n is one of N integers.

The Fourier transform unit 50 performs two-dimensional discrete Fourier transform on the image data f1 (m, n), f2 (m, n), for example, to obtain the Fourier transform value F1 (u , v) and F2(u, v), respectively. In formulas (7) and (8), u is any one of M integers, and v is any one of N integers. Further, A(u,v) and B(u,v) are amplitude spectra, and ^ejθ1(u,v) and ^ejθ2(u,v) are phase spectra.
F1(u,v)=A(u,v)×e ^jθ1(u,v) ...(7)
F2(u,v)=B(u,v)×e ^jθ2(u,v) ...(8)

The composition function calculation unit 51 multiplies the phase spectrum of the Fourier transform value F1 (u, v) by the complex conjugate of the phase spectrum of the Fourier transform value F2 (u, v) to obtain the result shown in equation (9). A composite function C12(u,v) is calculated. The correlation function calculation unit 52 calculates the correlation function c12(m, n) by subjecting the composite function C12(u,v) of Equation (9) to two-dimensional inverse discrete Fourier transform.
C12(u,v)=e ^{j(θ1(u,v)−θ2(u,v))} …(9)

Here, for example, assume that the positions of the images of image data f1 (m, n) and image data f2 (m, n) are the same. In this case, the correlation function c12(m,n) between image data f1(m,n) and image data f2(m,n) is close to a delta function with a peak value at the origin position (0,0). Become. On the other hand, assume that the image data f2 (m, n) is image data in which the image position is shifted by Δm in the m direction with respect to the image data f1 (m, n). In this case, when the correlation function c12(m,n) between image data f1(m,n) and image data f2(m,n) (=f1(m-Δm,n)) is calculated, the position of the peak value is Shift by Δm in the m direction from the origin position. The same applies to the n direction.

In this way, when the phase-only correlation unit 45 is used, the marker image 27 that occurs between time point #1 and time point #2 is calculated in the m direction and the n direction based on the position where the peak value occurs. Each shift amount, that is, the shift amount ΔSF(x,y) in the X-axis direction and the Y-axis direction can be calculated. The phase-only correlation unit 45 outputs the calculated shift amount ΔSF(x,y) to the displacement amount calculation unit 46 shown in FIG. Note that the lower the resolution of the marker image 27, the lower the accuracy and resolution of displacement measurement, so the marker image 27 needs to have a somewhat high resolution. For this reason, it is beneficial to create a panoramic image 25 as shown in FIG.

<Main effects of Embodiment 2>
As described above, by using the method of Embodiment 2, various effects similar to those described in Embodiment 1 can be obtained. becomes measurable. Note that when comparing the sampling moiré method and the phase-only correlation method, the sampling moiré method is superior in terms of resolution, and the phase-only correlation method is superior in terms of ease of image processing, cost, and the like. Therefore, for example, if high resolution is required for displacement measurement, the method of Embodiment 1 may be applied, and if not, the method of Embodiment 2 may be applied.

That is, the resolution is in pixel units in the phase-only correlation method, whereas in the sampling moiré method, for example, the periodic pattern shown in the marker image 27 in FIG. 7 is expanded to the periodic pattern shown in the moiré image 41[0]. Since the displacement is measured by using a sub-pixel, the displacement is effectively measured in sub-pixel units. Furthermore, in the phase-only correlation method, displacement can be measured through simple arithmetic processing as described in Figure 10, whereas in the sampling moiré method, complex image processing as described in Figure 7 is required. . Furthermore, since the phase-only correlation method does not normally require high resolution, it does not require high performance, ie, high-cost image sensor 12, etc., compared to the sampling moiré method.

(Embodiment 3)
<Outline of displacement measuring instrument>
FIG. 11 is a schematic diagram showing a configuration example of the marker in FIG. 1 in the displacement measuring device according to the third embodiment. The marker 1C shown in FIG. 11 is a combination of the marker 1A shown in FIG. 2 and the marker 1B shown in FIG. 8. That is, on the marker 1C, a geometric pattern and a periodic pattern are written side by side in, for example, the X-axis direction so as not to overlap with each other. Geometric patterns are used in the phase-only correlation method, and periodic patterns are used in the sampling moiré method.

FIG. 12 is a block diagram showing a schematic configuration example of the main parts of the displacement measuring device in FIG. 3 in the displacement measuring device according to the third embodiment. The displacement measuring device 20 shown in FIG. 12 includes a marker image creation section 30 and a marker image analysis section 28c. As in the case of FIG. 5, the marker image creation unit 30 extracts a plurality of known marker areas from the panoramic images 25#1, 25#2, . . . stored in the internal memory 23, thereby creating a plurality of Create a marker image.

However, in FIG. 4, the marker images 27 of the reference marker 1r and measurement markers 1m[1] to 1m[n] included in the panoramic image 25 vary from the periodic pattern shown in FIG. 2 to the geometric pattern shown in FIG. It is replaced by a combination of periodic patterns. When using a marker 1C as shown in FIG. 11, the area of the marker image 27 may expand, so it is usually difficult to maintain resolution. On the other hand, when the panoramic image 25 is used, it becomes possible to sufficiently increase the resolution of each of the geometric pattern and the periodic pattern. Note that the marker areas are individually set for each of the geometric pattern and the periodic pattern.

As in the case of FIGS. 5 and 9, the marker image analysis unit 28c analyzes the plurality of marker images R#1, M1#1 to Mn#1 created from the panoramic image 25#1, and the panoramic image 25#2. Based on the plurality of created marker images R#2, M1#2 to Mn#2, the amount of displacement of each of the plurality of markers 1 occurring between time point #1 and time point #2 is calculated. However, unlike the cases shown in FIGS. 5 and 9, the marker image analysis section 28c includes a sampling moire section 31, a phase-only correlation section 45, and a displacement calculation section 55.

The sampling moire unit 31 has the same configuration as shown in FIGS. 5 and 6, and uses the sampling moire method to calculate the phase difference ΔPHr (x, y) generated in the reference marker image R and the measurement marker images M1 to Mn. The phase differences ΔPHm1 (x, y) to ΔPHmn (x, y) that occur are calculated. The phase-only correlation unit 45 has the same configuration as shown in FIGS. 9 and 10, and uses the phase-only correlation method to calculate the shift amount ΔSFr (x, y) generated in the reference marker image R and the measurement marker image M1. The shift amount ΔSFm1 (x, y) to ΔSFmn (x, y) caused by ~Mn is calculated.

Here, in the sampling moiré method, since a periodic pattern is used, if a displacement exceeding the length of one period of the periodic pattern occurs in marker 1, that is, if a periodic deviation occurs, the periodic deviation is detected. This becomes difficult. On the other hand, the phase-only correlation method uses a geometric pattern, so this problem of periodic deviation does not occur. Therefore, the displacement calculation section 55 calculates the shift amount ΔSF(x,y) for each of the plurality of marker images 27 calculated by the phase-only correlation section 45, and calculates The magnitude of the periodic shift added to the phase difference ΔPH(x,y) of each moire fringe is determined. Then, the displacement amount calculation unit 55 calculates the displacement amount D1(x,y) to Dn(x, y).

FIG. 13 is a conceptual diagram illustrating an example of the processing content of the displacement amount calculation unit in FIG. 12. FIG. 13 shows the phase of the moire fringe brightness distribution 42#1 at time point #1 and the phase of the moire fringe brightness distribution 42#2 at time point #2, which are detected by the sampling moire unit 31. . The sampling moiré unit 31 calculates a phase difference ΔPH [rad] between the phase of the brightness distribution 42#1 and the phase of the brightness distribution 42#2 in the range from −π to +π. Therefore, the actual phase difference is not necessarily ΔPH+0, but may be ΔPH+2π, ΔPH+4π, or the like. These 0, 2π, 4π, etc. are determined based on the shift amount ΔSF(x,y) from the phase only correlation unit 45.

As a specific example, the pitch Wx of the periodic pattern in marker 1C shown in FIG. Assume that the length is 12 mm. When the sampling moiré method is used, a displacement amount of, for example, 2 mm, 7 mm, 12 mm, 17 mm, etc. can be obtained. On the other hand, when the phase-only correlation method is used, a result is obtained in which the amount of displacement is within a range of, for example, 10 mm to 15 mm. The displacement calculation unit 55 can calculate the actual displacement of 12 mm based on the overlapping portion of these results.

In this way, the displacement calculation unit 55 calculates the actual displacement of the reference marker 1r and the actual displacement of each of the measurement markers 1m[1] to 1m[n]. Further, as in the case shown in FIG. Correct displacement amounts D1(x,y) to Dn(x,y) for each marker 1m[1] to 1m[n] are calculated.

<Main effects of Embodiment 3>
As described above, by using the method of Embodiment 3, various effects similar to those described in Embodiment 1 and Embodiment 2 can be obtained. displacement can be measured at low cost. Furthermore, the magnitude of the periodic deviation caused by the sampling moiré method can be detected using the phase-only correlation method, making it possible to realize high-resolution measurement based on the sampling moiré method over a wide measurement range.

As above, the invention made by the present inventor has been specifically explained based on the embodiments, but the present invention is not limited to the embodiments described above, and various changes can be made without departing from the gist thereof.

1, 1A to 1C...Marker, 1r...Reference marker, 1m[1] to 1m[n]...Measurement marker, 2...Displacement measuring device, 3[0] to 3[n]...Object, 10...Imaging device, 20 ...Displacement measuring device, 25...Panorama image, 26[0] to 26[n]...Pictured image, 27...Marker image, 28a to 28c...Marker image analysis section, 30...Marker image creation section, 31...Sampling moire section, 32, 46, 55...Displacement amount calculation unit, 45...Phase only correlation unit, R...Reference marker image, M1 to Mn...Measurement marker image

Claims (7)

an imaging device that images the marker;
a displacement measuring device that measures the displacement of the marker based on an image captured by the imaging device;
A displacement measuring device having:
The imaging device images a plurality of markers attached to a single object or a plurality of objects, and creates a panoramic image including the plurality of markers in one image by combining the plurality of captured images,
The displacement measuring device is
A first panoramic image obtained by imaging at a first point in time and a second panoramic image obtained by imaging at a second point in time are targeted for a known plurality of known markers in which the plurality of markers exist. a marker image creation unit that creates a plurality of marker images by extracting each marker area;
The first point in time and the second point in time are determined based on the plurality of marker images created from the first panoramic image and the plurality of marker images created from the second panoramic image. a marker image analysis unit that calculates the amount of displacement for each of the plurality of markers that occurred during
Equipped with
Displacement measuring device. The displacement measuring device according to claim 1,
One of the plurality of markers is a reference marker attached to a measurement point treated as a fixed point,
Markers other than the reference marker are measurement markers,
The marker image analysis unit calculates the displacement amount of each measurement marker using a difference value from the displacement amount of the reference marker.
Displacement measuring device. The displacement measuring device according to claim 1 or 2,
A periodic pattern is written on each of the plurality of markers,
The marker image analysis unit detects a first phase of a moire fringe using a sampling moire method for each of the plurality of marker images created from the first panoramic image, and detects a first phase of a moire fringe created from the second panoramic image. A second phase of the moire fringes is detected for each of the plurality of marker images using the sampling moire method, and based on the difference between the first phase and the second phase, the second phase of the moire fringe is detected for each of the plurality of marker images. and calculating a phase difference of the moiré fringes for each of the plurality of marker images that occurs between and the second time point.
Displacement measuring device. The displacement measuring device according to claim 1 or 2,
A geometric pattern is written on each of the plurality of markers,
The marker image analysis unit calculates the correlation between the plurality of marker images created from the first panoramic image and the plurality of marker images created from the second panoramic image using a phase-only correlation method. calculating a shift amount for each of the plurality of marker images that occurred between the first time point and the second time point;
Displacement measuring device. The displacement measuring device according to claim 1 or 2,
Each of the plurality of markers has a periodic pattern and a geometric pattern written on it,
The marker image analysis unit includes:
A sampling moire method is used to detect a first phase of moire fringes for each of the plurality of marker images created from the first panoramic image for the periodic pattern, and a first phase of moire fringes created from the second panoramic image is detected. A second phase of the moire fringes is detected for each of the plurality of marker images using the sampling moire method, and a difference between the first phase and the second phase is used to determine the difference between the first time point and the second phase. a sampling moire unit that calculates a phase difference of the moire fringes for each of the plurality of marker images occurring between the second time point;
A phase-only correlation method is used to calculate the correlation between the plurality of marker images created from the first panoramic image and the plurality of marker images created from the second panoramic image, targeting the geometric pattern. a phase-only correlation unit that calculates a shift amount for each of the plurality of marker images that occurred between the first time point and the second time point by calculating each using the phase-only correlation unit;
Based on the shift amount for each of the plurality of marker images calculated by the phase-only correlation unit, the size of a periodic shift added to the phase difference of the moire fringes for each of the plurality of marker images calculated by the sampling moiré unit. a displacement calculation unit that calculates a displacement amount for each of the plurality of markers based on the phase difference of the moire fringes and the magnitude of the periodic shift;
Equipped with
Displacement measuring device. A displacement measurement method that uses an imaging device that images a marker and measures the displacement of the marker based on an image captured by the imaging device, the method comprising:
a step of capturing images of a plurality of markers attached to a single object or a plurality of objects, and creating a panoramic image including the plurality of markers in one image by composing the plurality of captured images;
A first panoramic image obtained by imaging at a first point in time and a second panoramic image obtained by imaging at a second point in time are targeted for a known plurality of known markers in which the plurality of markers exist. creating a plurality of marker images by respectively extracting marker regions;
The first point in time and the second point in time are determined based on the plurality of marker images created from the first panoramic image and the plurality of marker images created from the second panoramic image. a step of calculating the amount of displacement for each of the plurality of markers that occurred between;
Equipped with
Displacement measurement method. The displacement measuring method according to claim 6,
One of the plurality of markers is a reference marker attached to a measurement point treated as a fixed point,
Markers other than the reference marker are measurement markers,
further comprising the step of calculating the amount of displacement for each of the measurement markers by a difference value from the amount of displacement of the reference marker;
Displacement measurement method.

Images