JP2023082603A - Displacement measurement device, image correction device, and system for the same, displacement measurement method, image correction method, and program for the same - Google Patents

Abstract

To achieve alignment of images and measurement of displacement with high precision using images picked up at different times.SOLUTION: An image correction unit acquires an image representing two or more reference markers used as a reference of an amount of displacement and a measurement marker representing a pattern spatially repeated at a fixed pitch from an imaging unit for each frame to correct a position of the measurement marker so as to compensate for displacement between frames of the reference marker. An amount of displacement calculation unit calculates the amount of displacement of the measurement marker from a phase difference between frames of a moire image generated from a pattern of the measurement marker.SELECTED DRAWING: Figure 3

Description

The present invention relates to a displacement measuring device using image displacement measurement, an image correcting device, a system thereof, a displacement measuring method, an image correcting method, and a program thereof. The present invention relates, for example, to structural health monitoring and image processing.

Displacement detection has become an urgent issue in measures against aging of public facilities (infrastructure, hereinafter referred to as "infrastructure") composed of structures such as bridges. Displacement sensing may use ring displacement gauges, Doppler sensors, fixed cameras, and other fixed displacement measurement devices. Optical techniques such as a sampling moire method and a digital image correlation method (DIC: Digital Image Correlation method) are sometimes used using an image captured by a fixed camera. Installation of a fixed-type displacement measuring device may be difficult depending on topography such as hills and waterfronts, and the arrangement of buildings and structures around the object to be measured.

Under such circumstances, displacement measurement using a mobile displacement measuring device installed on a moving object such as an unmanned aerial vehicle (drone) or a robot has attracted attention. Attempts have been made to equip unmanned aerial vehicles with various displacement measuring devices, such as laser Doppler vibrometers and global positioning systems (GPS). However, laser Doppler vibrometers are generally expensive and difficult to economically implement. GPS may not provide the accuracy required for infrastructure maintenance.

Recently, a vision-based structural health inspection system using images captured by a camera mounted on a drone has been proposed. For example, Non-Patent Documents 1 and 2 relate to measurement of displacement of infrastructure using the DIC method, respectively. According to the DIC method, displacement can be measured with an accuracy of 1 mm or less. However, since speckles are used as markers in the DIC method, accuracy tends to be significantly degraded due to noise. Further, the image capturing management method described in Patent Document 1 is characterized in that an image is captured when the overlapping rate of the immediately captured image with the land portion reaches a predetermined overlapping rate. Therefore, this method is not suitable for measuring continuous small displacements of infrastructure. The inspection method described in Patent Literature 2 is characterized by detection of a corroded state of a transmission line in inspection of a live transmission line. Therefore, this method is also not suitable for measuring minute displacements of infrastructure.

JP 2017-15704 A Japanese Patent Application Laid-Open No. 2021-107992 Japanese Patent No. 4831703 Japanese Patent No. 6565037

Reagan, D., Sabato, A. and Niezrecki, C.; Feasibility of using digital image correlation for unmanned aerial vehicle structural health monitoring of bridges, Structural Health Monitoring, 2018, 17 (5), pp.1056-1072. Kalaizakis, M., Vitzilaios, N., Rizos, D.C. and Sutton, M.A., Drone-Based Stereo DIC: System Development, Experimental Validation and Infrastructure Application, Experimental Mechanics, 2021, 61, pp.981-996.

Patent Literatures 3 and 4 describe a method of measuring the displacement of infrastructure using a sampling moire method using images recorded for a short period of time with a camera fixed. However, if the camera cannot be fixed and installed, the position and shape of the object to be measured appearing in the image change as the camera moves, so the correct amount of displacement cannot be obtained as it is. In other words, it has been difficult to measure displacement with high precision using images captured by cameras mounted on moving bodies such as drones, robots, and ships, and cameras built into mobile devices such as mobile phones. In addition, when monitoring displacement over a long period of time, the position of the camera, which is supposed to be fixed, may fluctuate due to passing vehicles, wind, and the like. This can also be a factor in reducing the accuracy of the measured displacement.

The present invention has been made in view of the above points, and includes a displacement measuring device, an image correcting device, a system thereof, and a method for aligning images and measuring displacement with high accuracy using images captured at different times. An object of the present invention is to provide a displacement measurement method, an image correction method, and a program therefor.

(1) The present invention has been made to solve the above-described problems, and one aspect of the present invention is to provide two or more reference markers that are used as references for the amount of displacement, and spatially repeated markers with a constant pitch. an image correcting unit that acquires an image representing a measurement marker representing a pattern from an imaging unit for each frame and corrects the position of the measurement marker so as to compensate for the displacement of the reference marker between frames; and the pattern of the measurement marker. and a displacement amount calculation unit that calculates the displacement amount of the measurement marker from the inter-frame phase difference of the moiré image generated from.

(2) Another aspect of the present invention acquires images representing two or more reference markers each representing a pattern that is spatially repeated at a constant pitch from an imaging unit for each frame, and The image correcting apparatus includes an image correction unit that corrects the image to compensate for displacement, and further corrects the image to compensate for inter-frame phase differences of a moire image generated from the pattern of the reference marker. .

(3) Another aspect of the present invention may be a program for causing a computer to function as the displacement measuring device (1) or the image correcting device (2).

(4) Another aspect of the present invention may be a system comprising an imaging unit and the displacement measuring device of (1) or the image correcting device of (2).

(5) Another aspect of the present invention is that the displacement measuring device captures an image representing two or more reference markers used as a reference for the amount of displacement and a measurement marker representing a pattern repeated at a constant pitch for each frame. from the image correction step of correcting the position of the measurement marker to compensate for the inter-frame displacement of the reference marker obtained from a unit, and the inter-frame phase difference of the moire image generated from the pattern of the measurement marker, and a displacement amount calculation step of calculating the displacement amount of the measurement marker.

(6) Another aspect of the present invention is that the image correction device obtains, for each frame, an image representing two or more reference markers each representing a pattern that is spatially repeated at a constant pitch, and performing an image correction step of correcting the image to compensate for the frame-to-frame displacement of the markers, and further correcting the image to compensate for the frame-to-frame phase difference of moire images generated from the pattern of the reference markers. This is an image correction method for

According to the present invention, image alignment and displacement measurement can be achieved with high accuracy using images captured at different times.

BRIEF DESCRIPTION OF THE DRAWINGS It is a side view for demonstrating the outline|summary of the displacement measuring system which concerns on this embodiment. BRIEF DESCRIPTION OF THE DRAWINGS It is a front view for demonstrating the outline|summary of the displacement measuring system which concerns on this embodiment. 1 is a schematic block diagram showing a functional configuration example of a displacement measuring system according to an embodiment; FIG. 4 is a flow chart showing a first example of displacement measurement processing according to the present embodiment; FIG. 10 is a diagram showing an example arrangement of markers before and after deformation of the measurement object; FIG. 10 is a diagram showing an example of change in positional relationship between reference markers due to deformation; 8 is a flowchart showing a second example of displacement measurement processing according to the present embodiment; It is a display example of an image of each frame. It is a figure which illustrates the time change of the displacement amount in each marker. It is a figure which illustrates an experimental setup. It is a figure which illustrates the time change of the position of each marker. FIG. 4 is a diagram showing an example of trajectories of respective markers; FIG. 10 is a diagram showing an example of temporal change in the positional relationship between reference markers; FIG. 10 is a diagram showing an example of changes in observed marker positions over time; FIG. 5 is a diagram showing an example of changes in observed deflection amount over time; FIG. 5 is a diagram showing an example of a locus of displacement amounts; It is a figure which shows the experimental optical system of displacement measurement by aerial photography. It is a figure which shows the displacement measurement example by aerial photography. FIG. 4 is an explanatory diagram for explaining homography transformation; 9 is a flowchart showing a third example of displacement measurement processing according to the present embodiment; 1 is a schematic block diagram showing a functional configuration example of an image correction system according to an embodiment; FIG. FIG. 10 is an explanatory diagram showing a first example of alignment of images; FIG. 11 is an explanatory diagram showing a second example of alignment of images; FIG. 10 is an explanatory diagram illustrating an example of an experimental result of alignment correction of images;

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
1 and 2 are a side view and a front view for explaining the outline of a displacement measuring system 1 according to this embodiment. In the example of FIG. 1, the displacement measurement system 1 is used to measure the displacement of a bridge Br1 as the object to be measured. On the side of the bridge Br1, two reference markers Mk-A, Mk-B and a measurement marker Mk-C are arranged in that order in the longitudinal direction. The reference markers Mk-A and Mk-B are used as references for the amount of displacement. The measurement marker Mk-C is set at a predetermined measurement point as a target position for displacement measurement.

The imaging unit 20 is supported by a drone Dn flying in the air at a position facing the side of the bridge Br1. At this position, the imaging unit 20 captures an image showing the reference markers Mk-A and Mk-B and the measurement marker Mk-C for each frame. The position of the imaging unit 20 may be a position where the reference markers Mk-A and Mk-B and the measurement marker Mk-C are included in its field of view.
A road is laid on the surface of the bridge Br1. When the inspection vehicle Vc passes on the road, the weight of the inspection vehicle Vc deforms the bridge Br1. Under this setting, the displacement measurement system 1 can measure the amount of displacement at the measurement marker Mk-C that accompanies the deformation of the bridge Br1. Alternatively, the displacement measurement system 1 may acquire images captured over a longer period of time to observe temporal changes in the shape of the bridge Br1. In the following description, the "reference marker" and the "measurement marker" may be collectively referred to simply as the "marker."

Next, an outline of a method of measuring displacement by the displacement measuring system 1 will be described with reference to FIG. The displacement measurement system 1 illustrated in FIG. 2 compensates for changes in position of each of the reference markers Mk-A, Mk-B that occur between frames imaged at different times. The horizontal direction and vertical direction of the captured image correspond to the x direction and y direction, respectively. The x direction corresponds to the longitudinal direction of the bridge girder in the three-dimensional space in which the bridge Br1 is actually installed (in the following description, it may be referred to as the “subject space”), and the y direction is the vertical direction in the subject space. handle. Also, the space on the image of each frame is sometimes called "image space" to distinguish it from the object space.

FIG. 2(a) shows an image captured at time _t1 . At this point, the bridge Br1 is not deformed. This image is used as a reference image. The reference marker Mk-A, the reference marker Mk-B, and the measurement marker Mk-C are arranged in a straight line from left to right with respect to the drawing. That is, the position (x _C , y _C ) of the measurement marker Mk-C passes through the position (x _A , y _A ) of the reference marker Mk-A and the position (x _B , y B ) of the reference marker Mk- _B . It corresponds to the internal dividing point of a straight line.

FIG. 2(b) illustrates an image captured at time _t2 . The shape of the bridge Br1 appearing in the image at time _t2 is deformed from that appearing in the image at time _t1 . The position (x _A ', y _A ') of the reference marker Mk-A at time t ₂ is displaced downward from the position (x _A , y _A ) of the reference marker Mk-A at time t ₁ . On the other hand, the position (x _B ', y _B ') of the reference marker Mk-B at time t ₂ is displaced above the position (x _B , y _B ) of the reference marker Mk-B at time t ₁ . . However, the deformation of the bridge Br1 appearing in the image at time _t2 includes apparent deformation due to changes in the position and orientation of the imaging unit 20 . Apparent deformation is observed as image "blur" and causes measurement error.

Therefore, as illustrated in FIG. 2(c), the displacement measurement system 1 calculates the corrected position (x _A ^* , y _A ^* ) of the reference marker Mk-A at time t ₂ and the position of the reference marker Mk-B. The corrected position (x _B ^* , y _B ^* ) is the position (x _A , y _A ) of the reference marker Mk-A and the position (x _B , y _B ) of the reference marker Mk-B at time t ₁ . Correct the position of the measurement marker Mk-C in the measurement frame to match. The displacement measurement system 1 determines, for example, coordinate transformation parameters for correcting the image at time _t2 by coordinate transformation. The displacement measurement system 1 determines the position (x _C ^* , y _C ^* ) of the measurement marker Mk-C in the image corrected using the determined coordinate transformation parameters as the position of the measurement marker Mk-C after correction at time t ₂ . can be calculated by transforming the coordinates. As a method of coordinate transformation, for example, linear transformation such as affine transformation can be used.

Then, the displacement measurement system 1 generates a moiré image of the measurement marker Mk-C for each of the corrected image captured at time t ₂ and the image captured at time t ₁ . The displacement measurement system 1 calculates the amount of displacement of the measurement marker Mk-C installed on the bridge Br1 based on the inter-frame phase difference of the generated moiré image. Even in a situation where the imaging unit 20 moves, the change in the position of the measurement marker Mk-C appearing in the image of the measurement frame caused by the movement is compensated. Therefore, the displacement amount of the measurement marker Mk-C is measured with high accuracy.

It should be noted that each marker may represent any pattern as long as it is a pattern in which luminance is regularly distributed. In the examples of FIGS. 1 and 2, the brightness of each marker forms a lattice pattern that fluctuates at a constant period (pitch) in each of the horizontal and vertical directions. The imaging unit 20 may be installed at a position where the pitch of the marker pattern appearing in the captured image is at least two pixels or more.

When measuring displacement in a certain direction (for example, vertical direction), a marker representing a striped image arranged in a direction intersecting that direction is used. The marker does not have to be a dedicated object whose main purpose is displacement measurement, and a regular pattern displayed on the surface of the structure to be measured may be used. For example, a pattern formed by boundaries such as window frames of individual window panes arranged on a building surface, bricks arranged on a wall surface, tiles, and the like may be used. The object to be measured may be a bridge, a building, a dam, an inner surface of a tunnel, a micro test piece, or any other scale of displacement measurement.

However, the position of the reference marker is set at a position that can be regarded as stationary with the passage of time, or at a position that fluctuates relatively less than the position of the measurement marker. In the examples of FIGS. 1 and 2, the bridge pier portion on which the reference markers Mk-A and Mk-B are installed has relatively less fluctuation due to external force than the bridge girder portion on which the measurement marker Mk-C is installed. External forces applied to bridges include, for example, running of vehicles and passers-by, wind, water currents, earthquakes, and the like. Also, the number of reference markers is not limited to two, and may be three or more. The number of measurement markers is not limited to one, and may be two or more.

Next, a functional configuration example of the displacement measuring system 1 according to this embodiment will be described. FIG. 3 is a schematic block diagram showing a functional configuration example of the displacement measuring system 1 according to this embodiment.
A displacement measurement system 1 includes a displacement measurement device 10 and an imaging unit 20 . The displacement measuring device 10 includes a parameter input section 12 , an arithmetic processing section 14 and a display section 16 .

Various parameters are input to the parameter input unit 12 . These parameters include parameters used to identify the position of each reference marker, measure the displacement of the measurement marker, generate a moiré image from the reference and measurement markers, calculate the displacement from the phase difference of the moiré image, etc. is included. These parameters will be described together with the functions of the arithmetic processing unit 14. FIG. The parameter input unit 12 may include a data input interface, or may include an input device such as a mouse, touch sensor, or keyboard for inputting various types of information according to user operations.

The arithmetic processing unit 14 realizes processing for exerting the functions of the displacement measuring device 10 and processing for controlling these functions. The arithmetic processing unit 14 may be configured as a computer system including, for example, a processor such as a CPU (Central Processing Unit) and storage media such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The processor may implement the function of each functional unit by reading out a predetermined control program pre-stored in the storage medium and executing processing indicated by instructions written in the read control program. The arithmetic processing unit 14 includes, as functional units, a marker detection unit 142, a first image correction unit 144, a first displacement amount calculation unit 146, a second image correction unit 148, a second displacement amount calculation unit 150, and a displacement amount output unit. 152.

The marker detection section 142 detects the reference marker and the measurement marker from the image of each frame shown in the image data input from the imaging section 20 . The marker detection unit 142 can detect markers by executing known image recognition processing. The marker detection unit 142 uses, for example, a known machine learning model (for example, a neural network) to indicate the degree of possibility that a pattern of a known marker will appear for each block, which is a partial region of the image to be processed. Calculate confidence. In the marker detection unit 142, pattern data related to a marker pattern, which is the form of a reference marker, is set in advance, and the degree of similarity (similarity) between a part of the image to be processed and the marker pattern is relied upon. It may be calculated as degrees. The marker detection unit 142 can define a predetermined number of regions whose calculated reliability is higher than a preset reliability threshold as marker regions (hereinafter sometimes referred to as “marker regions”). The predetermined number corresponds to the sum of the number of reference markers and the number of measurement markers. The marker detection unit 142 outputs to the first image correction unit 144 marker information including the center coordinates of the marker indicating the marker area detected from the image of each frame.

The first image correction unit 144 identifies the marker area indicated by the marker information input from the marker detection unit 142 for each frame. Due to differences in the position and orientation of the imaging unit 20 for each imaging time, the distribution of the identified marker regions may differ between frames. The first image correction unit 144 calculates coordinate transformation parameters for correcting the image shown in the image data input from the imaging unit 20 so as to compensate for the difference in the position of the reference marker between frames. The first image correction unit 144, for example, adjusts the coordinates so that the position of the reference marker in the measurement frame to which attention is paid as the target for measuring the displacement of the measurement marker matches the position of the reference marker in the reference frame used as the reference for the displacement of the measurement marker. Transformation parameters can be calculated. As the image of the reference frame, for example, an image captured when no deformation has occurred may be used. The first image correction unit 144 outputs first corrected image data representing an image corrected by coordinate conversion using the calculated coordinate conversion parameters to the first displacement amount calculation unit 146 and the second image correction unit 148 . The first corrected image data includes the corrected image of the measurement frame. The first corrected image data may include an image of the reference frame that has not been corrected.

At this stage, the positional accuracy of each reference marker is about the same as the interval (pixel pitch) between adjacent pixels (pixels), so the positional accuracy of the measurement markers in the corrected image is also the pixel pitch has the same accuracy as In the present application, the correction of the image or the position of the measurement marker on the image executed by the first image correction unit 144 is sometimes called "coarse correction", and the accuracy of the correction is sometimes called "pixel accuracy".

The first displacement amount calculator 146 calculates the displacement amount of the measurement marker in the subject space represented in the image of each frame indicated by the first corrected image data input from the first image corrector 144 as the first displacement amount. . That is, the first displacement amount calculator 146 calculates the actual displacement amount of the measurement marker based on the displacement between the position of the measurement marker in the reference frame and the position of the measurement marker in the corrected measurement frame.

The first displacement amount calculator 146 can use the sampling moire method in calculating the first displacement amount. More specifically, the first displacement amount calculator 146 generates a moiré image of the measurement marker represented in the image of each frame. For example, the first displacement amount calculation unit 146 calculates, for a partial image including at least the area of the measurement marker, a period T that is the same as or different from the period P of the pattern of the measurement marker (T is a preset 2 (integer above) to generate T thinned images obtained by thinning. The phase of each of the T pixels to be thinned (coordinates of the pixel that is the starting point of thinning) is shifted to a value between 0 and T−1. The first displacement amount calculator 146 can generate a moiré image by interpolating pixel values between adjacent pixels after thinning out for each of the T thinned images. The first displacement amount calculator 146 performs a discrete Fourier transform on the phase-shifted T moire images to calculate a phase distribution at a predetermined spatial frequency. The first displacement amount calculator 146 can also calculate the phase distribution for other frames using a similar technique. The first displacement amount calculator 146 calculates the displacement distribution by multiplying the distribution period (pitch) p of the phase difference, which is the phase distribution difference between the frames, by the frequency and a coefficient obtained by dividing by 2π. Then, the first displacement amount calculator 146 averages the displacement amount of each pixel forming the displacement distribution between frequencies and individual pixels, and determines the displacement amount of the measurement marker to which the pixel belongs as the first displacement amount. can. The first displacement amount calculation unit 146 outputs first displacement amount data indicating the determined first displacement amount to the displacement amount output unit 152 . Therefore, the first displacement amount calculator 146 can roughly calculate the displacement amount using the image corrected with pixel precision. The periods P, T, and p are preset in the first displacement amount calculator 146 as parameters for calculating the first displacement amount. Note that the period P is the pitch of the grid markers on the captured image, and the unit is, for example, pixels. On the other hand, the period p is the physical pitch of the actual grating markers, in units of mm, for example. The period T may be different from the period P, but a value approximate to the period P may be used. Since the reciprocal 1/Q of the period Q of the moire image corresponds to the absolute value of the difference between the reciprocal 1/T of the period T and the reciprocal 1/P of the period P, the smaller the difference between the period T and the period P, the , the period Q of the phase of the moiré image increases. Therefore, the period T may be determined in advance according to a desired magnification of the period Q with respect to the period T. FIG. Note that the sampling moire method is described in more detail in, for example, International Publication No. 2017/138414.

The second image corrector 148 identifies the marker region of each marker from the image for each frame indicated by the first corrected image data input from the first displacement amount calculator 146 . The second image correction unit 148 generates a moire image for each of the identified reference markers. The second image correction unit 148 can use the same technique as the first displacement amount calculation unit 146 in generating the moiré image. The second image correction unit 148 calculates the phase distribution of the moiré image generated for the predetermined frequency for each frame. As with the first image correction unit 144, the second image correction unit 148 corrects the image shown in the first corrected image data by coordinate transformation so that the difference in the position of the reference marker is compensated for between frames. Calculate the coordinate transformation parameters of Since the period of the phase of the moire image expands more than the period of the brightness of the markers, the image correction at this stage is done with fine precision of less than one pixel (sub-pixel precision). The second image corrector 148 outputs to the second displacement amount calculator 150 second corrected image data representing an image corrected by coordinate transformation using the calculated coordinate transformation parameters. The second corrected image data may include an uncorrected reference frame image in addition to the corrected measurement frame. In the present application, the image correction performed by the second image correction unit 148 may be called "micro correction", and the precision of the correction may be called "sub-pixel precision". The second image correction unit 148 can use the same technique as the first image correction unit 144 in coordinate transformation.

Based on the image of each frame indicated by the second corrected image data input from the second image correction unit 148, the second displacement amount calculation unit 150 uses the sampling moire method similarly to the first displacement amount calculation unit 146 to correct the frame. A displacement amount of the measurement marker between is calculated as a second displacement amount. The second displacement amount calculation section 150 outputs second displacement amount data indicating the determined second displacement amount to the displacement amount output section 152 . Therefore, in the second displacement amount calculation unit 150, a minute displacement amount correction amount is obtained as the second correction amount using the image corrected with sub-pixel precision.

The displacement amount output unit 152 outputs the first displacement amount of each measurement marker indicated by the first displacement amount data input from the first displacement amount calculation unit 146 or the second displacement amount input from the second displacement amount calculation unit 150 . Any one of the second displacement amounts of each measurement marker indicated by the displacement amount data is selected as the final displacement amount of the measurement marker. The displacement amount output unit 152 outputs information on the selected displacement amount to the display unit 16 . Either the first displacement amount or the second displacement amount may be set in advance in the displacement amount output unit 152, or the first displacement amount and the second displacement amount may be selected in accordance with an input signal input from an input device. Any amount of displacement may be selected.

The display unit 16 displays information indicating the amount of displacement of the measurement marker input from the arithmetic processing unit 14 . The display unit 16 may display the displacement in any manner. The display unit 16 may, for example, represent the displacement numerically, or may graphically show the change in magnitude. The display unit 16 may include, for example, a liquid crystal display, a dial, or the like.

The imaging unit 20 captures an image representing a subject within a predetermined field of view centered on the imaging direction. As illustrated in FIGS. 1 and 2, the imaging unit 20 may be installed at a position and in an orientation in which a subject on which a plurality of reference markers and at least one measurement marker are installed is included in the field of view. In this embodiment, the position of the imaging unit 20 does not have to be fixed. As illustrated in FIGS. 1 and 2, it may be installed or supported by a moving object such as a drone. The mobile object is not limited to a drone, and may be a robot, a trolley, a vehicle, a ship, a living thing (for example, hand-held photographing by a person), or the like. The imaging unit 20 may be provided in a device whose main function is not necessarily imaging, such as a multifunction mobile phone (smartphone), a microscope, or the like.

The image capturing unit 20 is a digital video camera that sequentially captures one frame image every predetermined time (eg, 1/960 to 1/15 second). A typical digital video camera has pixels arranged in a two-dimensional imaging plane and can capture a two-dimensional image. A plurality of still images captured in succession form a moving image. The imaging unit 20 includes an output data interface that outputs image data representing a captured image to the displacement measuring device 10 wirelessly or by wire.

Next, an example of displacement measurement processing based on the position of the reference marker will be described with reference to FIGS. 4 to 6. FIG. However, two reference markers and one or more measurement markers are used (see FIGS. 1 and 2). At the reference time (reference frame), the measurement markers are arranged on a straight line with two reference markers as endpoints. However, the case of using affine transformation in the coordinate transformation will be taken as an example.

(affine transformation)
FIG. 4 is a flowchart showing a first example of displacement measurement processing according to this embodiment. When executing the processing illustrated in FIG. 4, the arithmetic processing unit 14 determines a reference frame as a reference for the amount of displacement. As the image of the reference frame, for example, an image representing the object to be measured before being displaced is used. Typically, an image at the beginning of observation is set as a reference frame, and frames captured after that are set as measurement frames to be corrected.

(Step S<b>102 ) The marker detection unit 142 detects markers for each frame shown in the image data input from the imaging unit 20 . The marker detection unit 142 determines the coordinates of the center of gravity of each detected marker as the center coordinates.
(Step S104) The first image correction unit 144 uses the arrangement of the two reference markers in the reference frame and the arrangement of the two reference markers in the measurement frame different from the reference time to convert the reference frame to the measurement frame. Analyze translation, rotation, and scaling factors. The translation amount, rotation amount, and scale ratio correspond to coordinate transformation parameters by affine transformation from the reference frame to the measurement frame. The first image correction unit 144 corrects the image of the measurement frame with pixel accuracy so that the translation by the analyzed translation amount, the rotation by the rotation amount, and the magnification by the scale factor are compensated (coarse correction).
(Step S106) The first displacement amount calculator 146 analyzes the phase difference of the moiré images of the measurement markers in the reference frame and the measurement frame using the sampling moiré method, and calculates the actual displacement of the measurement marker from the analyzed phase difference. amount is calculated as the first displacement amount.

(Step S108) The second image correction unit 148 analyzes, for each reference marker, the phase difference between the moiré image in the reference frame and the moiré image in the measurement frame after position correction with pixel accuracy. The second image correction unit 148 analyzes the amount of translation, the amount of rotation, and the scale ratio from the reference frame to the measurement frame based on the analyzed phase difference. The second image correction unit 148 re-corrects the image of the measurement frame after correction with sub-pixel accuracy so that the translation by the analyzed translation amount, the rotation by the rotation amount, and the magnification by the scale factor are compensated. (Minor correction).
(Step S110) The second displacement amount calculator 150 uses the sampling moire method to analyze the phase difference between the moiré images of the measurement markers in the reference frame and the measurement frame, and calculates the actual displacement of the measurement marker from the analyzed phase difference. amount is calculated as the second displacement amount.
(Step S112) The displacement amount output unit 152 selects either the first displacement amount obtained in step S106 or the second displacement amount obtained in step S110 as the final displacement amount. The displacement amount output unit 152 causes the display unit 16 to display information on the selected displacement amount.

FIG. 5(a) shows an arrangement example of the reference markers Mk-A and Mk-B and the measurement marker Mk-C on the reference frame imaged before deformation of the object to be measured. The position of the measurement marker Mk-C corresponds to the internal dividing point of the two reference markers Mk-A and Mk-B. The marker detection unit 142 detects the coordinates (x A , y A ) and (x _B , y _B ) of the centers of gravity A, B, and _C of the reference markers Mk- _A , Mk-B and the measurement marker C from the image of the reference frame. , (x _C , y _C ) can be specified in pixels (pixel precision).
FIG. 5(b) shows an arrangement example of the reference markers Mk-A, Mk-B and the measurement marker Mk-C on the measurement frame imaged after the deformation of the measurement object. The marker detection unit 142 detects the coordinates (x _A ', y A '), (x _A '), (x _B ', y _B '), (x _C ', y _C ') can be specified in pixel units (pixel accuracy).

Factors that vary the positions of the fiducial markers Mk-A and Mk-B between frames include translation, scale (change in size, shift in magnification) and rotation.
The first image correction unit 144 calculates the displacement of any one reference marker between frames, for example, the displacement from the center of gravity A to the center of gravity A' (x _A '-x _A , y _A '-y _A ) as a reference marker. It can be calculated as the translation amount (Δx, Δy) of Mk−A.
The first image correction unit 144 calculates a magnification r' from the distance r between the two reference markers Mk-A and Mk-B in the reference frame to the distance r' between the two reference markers A and Mk-B in the measurement frame. /r can be calculated as a scale factor Δs (magnification deviation, see FIG. 6B). Distance r is √(x _B −x _A ) ² +(y _B −y _A ) ² , distance r′ is √(x _B '−x _A ') ² +(y _B '−y _A ') ² becomes.

The first image correction unit 144 calculates the angle formed by the straight line AB passing through the centers of gravity A and B in the reference frame and the straight line A'B' passing through the centers of gravity A' and B' in the measurement frame as a rotation amount Δθ (rotation deviation, (see FIG. 6(a)). The rotation amount Δθ is obtained by normalizing the inner product of the vector AB from the center of gravity A to the center of gravity B and the vector A'B' from the center of gravity A' to the center of gravity B' by the absolute values of the vectors AB and A'B'. An inverse cosine function value (arccos) for the obtained cosine value (cos) can be calculated as the amount of rotation Δθ. FIG. 6(c) shows a situation in which a rotational shift and a magnification shift are superimposed. In general, translational shifts may be included as well as rotational shifts and magnification shifts.

The calculated scale ratio, rotation amount, and translation amount correspond to coordinate transformation parameters for transforming an arbitrary position on the reference frame onto the measurement frame. The first image correction unit 144 uses the calculated scale ratio, amount of rotation, and amount of translation to convert the position of the reference marker between the reference frame and the measurement frame to match the position of the reference marker on the measurement frame. can be corrected for the positions of the measurement markers.

(AB correction)
It should be noted that the position of the measurement markers can also be achieved using simpler techniques. The method described below is also called AB correction (Absolute Blurring Compensation). The first image corrector 144 uses the positions of the two reference markers of the measurement frame to determine the positions of the measurement markers in the measurement frame without explicitly calculating the scale, rotation, and translation. can be corrected.

In the AB correction, the center of gravity C of the measurement marker Mk-C is placed on a straight line passing between the centers of gravity A and B of the two reference markers Mk-A and Mk-B in the reference frame, and the first image correction unit 144 are the weights of the inter-frame displacements (x _A '-x _A , y _A '-y _A ) and (x _B '-x _B , y _B '-y _B ) of the reference markers Mk-A and Mk-B, respectively The average value is determined as the correction amount for the center of gravity (x _C ', y _C ') of the measurement marker Mk-C. However, when the center of gravity C of the measurement marker Mk-C is the point that internally divides the center of gravity A and B of the two reference markers Mk-A and Mk-B, the displacement (x _A '-x _A , y _A ' -y _A ), (x _B '-x _B , y _B '-y _B ), the reciprocal of the internal ratio from each of the centers of gravity A and B to the center of gravity C can be used. For example, if the distance between the centers of gravity A and C and the distance between the centers of gravity C and B are L _A and L _B , respectively, the first image correction unit 144 corrects the measurement marker Mk-C in the measurement frame in the y direction. can be ^calculated according to _equation (1). Similarly, the first image correction unit 144 can calculate the corrected coordinate value x _C ^* in the x direction of the measurement marker Mk-C in the measurement frame according to Equation (2).

In particular, when the center of gravity C of the measurement marker Mk-C is the middle point of the centers of gravity A and B of the two reference markers Mk-A and Mk-B, the first image correction unit 144 sets the reference marker Mk-A , Mk-B, the coordinates of the center of gravity of the measurement frame (x _A ', y _A '), (x _B ', y _B ') are used as correction values, and the center of gravity of the measurement marker Mk-C It can be corrected by subtracting from the coordinate values (x _C ', y _C ').

The measurement frame can also be regarded as being formed by in-plane deformation of the image of the reference frame with the center of gravity C as a base point. In that case, rotation, magnification (scale), and lens aberration appear point-symmetrically with respect to the center of gravity C. When the centers of gravity A and B are in the horizontal direction (x direction), the centers of gravity A and B are displaced in the y direction and -y direction by rotation, and the changes in scale cause the centers of gravity A and B to move in the -x direction and +x direction. is displaced to Therefore, when the center of gravity C is the point that internally divides the centers of gravity A and B, the weighted average corresponding to the reciprocal of the distance to each of the centers of gravity A and B cancels out the displacements occurring in each of the centers of gravity A and B ( canceled). Further, according to the parallel movement, the centers of gravity A, B, and C are all displaced in the same direction by the same amount of displacement. Displacement is canceled.

Note that when the center of gravity C of the measurement marker Mk-C is the external dividing point of the centers of gravity A and B of the two reference markers Mk-A and Mk-B, the displacement (x _A '-x _A , y _A ' −y _A ), (x _B ′−x _B , y _B′ −y _B ), the reciprocal of the external division ratio from each of the centers of gravity A and B to the external division point can be used. Again, translation, rotation and scale changes are compensated for by simple arithmetic. However, since the error due to the aberration of the lens is not canceled, the accuracy is lower than when the center of gravity C of the measurement marker Mk-C becomes the internally dividing point of the center of gravity A and B of the two reference markers Mk-A and Mk-B. Become.

Note that the sampling moire method cannot measure a displacement amount larger than the half period of the pattern represented by the measurement marker. This is because the sampling moiré method uses the phase difference of the moiré generated from the pattern, and therefore cannot distinguish between displacement amounts that differ by an integer multiple of the period of the pattern. In this method, the displacement between frames is corrected with pixel accuracy, and based on the displacement between the position C' of the center of gravity in the corrected measurement frame of the measurement marker Mk-C and the position C of the center of gravity in the reference frame, the measurement marker Mk- The frame-to-frame displacement of C can be contained within half a period of the pattern pitch of the measurement marker Mk-C. Therefore, the first displacement amount calculator 146 can measure the actual displacement amount of the measurement marker Mk-C between frames after correction with pixel accuracy as the first displacement amount using the sampling moire method.

In addition, the second image correction unit 148 can correct the position of the measurement frame in the measurement frame with sub-pixel precision using AB correction for the position of the reference frame in the measurement frame corrected with pixel precision. When performing correction with sub-pixel accuracy, the second image correction unit 148 generates a moire image for each marker in the measurement frame and determines the phase distribution of the generated moire image. The second image corrector 148 generates a phase distribution for each reference marker in the measurement frame. The second image correction unit 148 calculates the phase distribution obtained by weighted averaging the phase distribution for each reference marker as the correction amount of the measurement marker. On the other hand, the second image correction unit 148 generates a phase distribution after correction by subtracting the correction amount from the phase distribution in the measurement markers of the measurement frame. The second displacement amount calculation unit 150 uses the sampling moire method to sub-calculate the second displacement amount based on the phase difference between the corrected phase distribution of the measurement markers of the measurement frame and the phase distribution of the measurement markers of the reference frame. It can be measured with pixel precision.

Next, an example of displacement measurement processing using AB correction will be described with reference to FIG. FIG. 7 is a flowchart showing a method using AB correction as a second example of displacement measurement processing according to the present embodiment. When executing the process illustrated in FIG. 7, similarly to the example of FIG. 4, the arithmetic processing unit 14 sets a reference frame as a reference for the amount of displacement, and sets other frames as measurement frames to be corrected. do. However, in the reference frame, the measurement marker and the reference marker are placed so that the center coordinates of the measurement marker are on a straight line passing through the center coordinates of the two reference markers; back.

(Step S<b>152 ) The marker detection section 142 detects a marker for each frame shown in the image data input from the imaging section 20 . The marker detection unit 142 determines the coordinates of the center of gravity of each detected marker as the center coordinates.
(Step S154) The first image correction unit 144 calculates the inter-frame displacement of the center coordinates of each reference marker between the reference frame and the measurement frame. The first image correction unit 144 corrects the center coordinates of the measurement markers in the measurement frame according to formulas (1) and (2) based on the weighted average value of inter-frame displacements of the center coordinates of each reference marker. This corrects the position of the measurement marker with pixel accuracy (coarse correction).
(Step S156) The first displacement amount calculator 146 uses the sampling moire method to analyze the phase difference between the moiré images of the measurement marker of the reference frame and the measurement marker of the position-corrected measurement frame, and A displacement amount of the actual measurement marker is calculated as a first displacement amount from the phase difference.

(Step S158) For each reference marker, the second image correction unit 148 analyzes the phase difference between the moiré image in the reference frame and the moiré image in the measurement frame after position correction with pixel precision. The analyzed phase difference corresponds to frame-to-frame displacement with sub-pixel accuracy. The second image correction unit 148 converts the center coordinates of the measurement markers after correction with pixel accuracy in the measurement frame to the inter-frame displacement of the center coordinates of each reference marker with sub-pixel accuracy according to equations (1) and (2). is corrected based on the weighted average of This corrects the position of the measurement marker with sub-pixel precision (fine correction).
(Step S160) The second displacement amount calculator 150 uses the sampling moire method to determine the positions of the moire images of the measurement markers in the reference frame and the moire images of the measurement markers in the measurement frame whose positions are corrected with sub-pixel precision. The phase difference is analyzed, and the actual displacement amount of the measurement marker is calculated as the second displacement amount from the analyzed phase difference.
(Step S162) Either the first displacement amount obtained in step S156 or the second displacement amount obtained in step S160 is selected. The displacement amount output unit 152 causes the display unit 16 to display information on the selected displacement amount.

(simulation)
Next, a simulation performed on the displacement measuring device 10 according to this embodiment will be described. The simulation verifies the measurements using known displacements.
In the first simulation, image a illustrated in FIG. 8A was used as the image of the reference frame before displacement, and image b illustrated in FIG. 8B was used as the image of the measurement frame after displacement. Then, the coordinates of the measurement markers in image b were corrected, and the coordinates of the measurement markers after correction in image c (FIG. 8(c)) were calculated. Image c is obtained by transforming the coordinates in image b so that the positions of the reference markers match those of image a. The size of images ac is 4000 pixels×1600 pixels, respectively.

Image a is a synthesized image simulating the front of a bridge on which seven markers A to G are installed. Center coordinates (X, Y) of markers AG are as follows.
A(400,360), D(1200,960), C(2000,360), E(2800,360), B(3600,360), F(400,960), G(3600,960)
Markers A to G represent lattice patterns in which the luminance fluctuates for six periods at a constant pitch P in the horizontal and vertical directions. The luminance value at the coordinates (i, j) of each marker is set to Bias+Amp(cos(2π(i/P+X)+cos(2π(j/P+Y))), where the pitch P is 20 pixels, The bias value Bias was set to 128, and the amplitude Amp was set to 50.
Among markers AG, markers A and B were used as reference markers, and markers C to E were used as measurement markers. The positions of markers C to E correspond to internal dividing points of straight lines passing through markers A and B, respectively. The position of marker C corresponds to the midpoint of markers A and B. FIG.

Image b is synthesized by giving known displacements to the markers C to E represented in image a, applying affine transformation to the images obtained by giving the displacements, and adding translation, scale, and rotation. The displacements given to the markers CE are as follows.
D (0.025, -0.05), C (0, -0.1), E (0, -0.01)
The parallel displacement amounts Δx and Δy are set to 6.6 pixels and 13.2 pixels, respectively. The rotation angle Δθ was set to 2.2°.

For comparison, the image b' obtained by applying a known displacement before the affine transformation was used as the image of the measurement frame after deformation, and the sampling moire method was used to calculate the displacement of the markers CE. Here, the actual pitch p of the pattern represented by the markers is 100 mm, and the thinning period T is 20 pixels. A low-pass filter with a filter size of 43 pixels and a cutoff frequency of 0.1 was applied to separate the pattern component in the x direction and the pattern component in the y direction from the two-dimensional lattice pattern. The size of the area for averaging the amount of displacement was set to 30 pixels×30 pixels. As a result, the amounts of displacement of the markers D, C, and E before and after deformation in the y direction were 5.0, 10.0, and 1.021 mm, respectively.

In a second simulation, the following approach was taken to define the y-direction displacement of the markers D, C, E. AB correction was used as the image correction method unless otherwise specified.
(1) Correct the coordinate values (pixel accuracy) of markers D, C, and E in image c obtained by performing rough correction based on the coordinate values of markers A and B and the internal division ratio, and perform the sampling moiré method. was used to define the displacement in the y-direction. As a result, the displacement amounts of the markers D, C, and E before and after deformation in the y direction were 5.007, 9.992, and 1.020 mm, respectively.
(2) Using the coordinate values (sub-pixel precision) of the markers D, C, and E in the image c obtained by performing coarse correction and fine correction, the displacement in the y direction was determined using the sampling moire method. As a result, the displacement amounts of the markers D, C, and E before and after deformation in the y direction were 5.064, 10.034, and 1.071 mm, respectively.
(3) Correct the coordinate values of the markers D, C, and E in the image c obtained by performing the coarse correction and the fine correction based on the coordinate values of the markers A and B and the internal division ratio, and use the sampling moire method. determined the displacement in the y direction. As a result, the amounts of displacement of markers D, C, and E before and after deformation in the y direction were 5.019, 9.988, and 1.024 mm, respectively.

Comparing (1) and (2), the error in the displacement amount obtained in (1) is smaller than in (2). Even if the displacement amount is obtained with high accuracy using the sampling moire method, the accuracy of correction for the imaging direction simulated by the affine transformation is not necessarily high. Comparing (1) and (3), (1) has a slightly smaller error than (3). However, the error in the amount of displacement is about the same between (1) and (2), and there is no significant difference.

In the third simulation, the amount of displacement at markers CE was calculated assuming the following cases (i)-(v) for each measurement frame after displacement. The period for each case was 20 frames.
(i) when the position of the imaging unit 20 is stationary (still, no conversion), (ii) when parallel to the x direction (horizontal direction) (Δx), (iii) when translated in the y direction (Δy), (iv) rotating (Δθ), (v) changing magnification (Δs)

In (i), in order to simulate camera shot noise, the brightness value of each marker represented in the image of the reference frame shown in FIG. Random noise with σ was added and the position was varied for each measurement frame. In (ii), the initial value of the amount of translation in the x direction added to the image obtained by adding random noise to the coordinate values of each marker is set to 0, and the amount is increased by 0.1 pixels for each frame, and the maximum value is 1 pixel. After reaching , it was decreased by 0.1 pixels per frame.
In (iii), the initial value of the amount of translation in the y direction added to the image obtained by adding random noise to the coordinate values of each marker is set to 0, and is increased by 0.1 pixels for each frame, and the maximum value is 1 pixel. After reaching , it was decreased by 0.1 pixels per frame.
In (iv), the initial value of the amount of rotation to be applied to the image obtained by adding random noise to the coordinate values of each marker is set to 0, and is increased by 0.001 degrees for each frame, reaching the maximum value of 0.01 degrees. and then decreased by 0.01 degrees every frame. The reference point for rotation was the center of the image.
In (v), the initial value of the magnification to be added to the image obtained by adding random noise to the coordinate values of each marker is 1 (same magnification), and the magnification is increased by 0.0001 times for each frame, and the maximum value is 1.001. After doubling, it was decreased by 0.001 every frame. The reference point for magnification was taken as the center of the image.

9(a) and 9(b) show the amount of displacement in the x and y directions of the markers AC in each measurement frame. This displacement amount is a value obtained using the sampling moire method without correction. However, the thinning period T is 20, the pitch period p of the marker in the object space is 100 mm, the cutoff frequency of the low-pass filter is 0.1, and the size of the average area when averaging the phase difference of the moiré image is 40×40. pixel. As shown in FIGS. 9A and 9B, the displacement amounts in the x and y directions significantly fluctuate due to coordinate changes that mimic changes in the imaging direction.
FIG. 9(c) shows the amount of displacement of the marker C in the x direction and the amount of displacement in the y direction after correction in each measurement frame using marks ◯ and ●. The correction is made by subtracting the mean value of the y-coordinates of the markers A and B on the image in the measurement frame. As a result, in all cases (i)-(v), the average values in the x and y directions were 0.017 and -0.017, and the standard deviations were 0.011 and 0.012, indicating a significant difference. do not have. The simulation results in FIG. 9(c) show that the same measurement accuracy can be obtained in the case of translation, rotation or scale change as in the case of no change.

(experiment)
Next, an experiment conducted on the displacement measuring device 10 according to this embodiment will be described.
The first experiment is primarily aimed at verifying registration of images captured from a moving camera. In the first experiment, images of three markers Mk-A to Mk-C installed on a structure were taken at regular time intervals from a camera mounted on a drone. Markers Mk-A to Mk-C were arranged at regular intervals in the horizontal direction. FIG. 10(a) is an example of an image captured toward the front of the structure. FIG. 10(b) is an enlarged view of the area where the markers Mk-A to Mk-C are shown in FIG. 10(a). In the experiment, among markers Mk-A to Mk-C, marker Mk-C was used as a measurement marker, and markers Mk-A and Mk-B were used as reference markers. No external force was applied to the structure during the flight of the drone. Therefore, the amount of displacement at each marker is ideally zero.

FIG. 11 shows the x-coordinate and y-coordinate of the center point of each of the markers Mk-A, Mk-C, and Mk-B in the image captured at each time with pixel accuracy. In FIG. 11, each column corresponds to a marker and each row corresponds to coordinates. The markers Mk-A, Mk-C, and Mk-B are actually stationary, but the change in position shown in FIG. .

FIG. 12 shows the trajectory of the center of gravity of each of the markers Mk-A, Mk-C, and Mk-B in the image from the start to the end of observation with pixel precision in the left column, and with sub-pixel precision in the right column. In FIG. 12, each column corresponds to a marker. The shapes of the trajectories of the markers are similar to each other in both pixel precision and sub-pixel precision. This indicates that displacements common to the entire image are significant, and that local displacements can be hidden without correction. Also, by correcting the position of each marker with sub-pixel accuracy, a smooth trajectory can be obtained. It is expected that the displacement of each marker can be obtained with high precision by measuring with sub-pixel accuracy.

13A, 13B, 13C, and 13D show the amount of parallel movement Δx in the x direction, the amount of parallel movement Δy in the y direction, and the amount of rotation calculated from the measurement image captured at each time. Δθ, scale factor Δs. However, the initial (time t=0) frame was used as the reference frame, and the subsequent frames at each time t were used as the measurement frames. The translation amounts Δx and Δy, the rotation amount Δθ, and the scale ratio Δs are calculated from the center of gravity of the reference marker in each of the measurement frame and the reference frame at each time.

14(a) and (b) show the x-coordinate and y-coordinate of the center of gravity of the marker Mk-A at each time. Values obtained with pixel precision and sub-pixel precision are represented by light and dark lines, respectively. In pixel precision, values change stepwise over time, while in sub-pixel precision, values change smoothly over time. This stepwise change is more conspicuous in the y-coordinate value, which changes relatively little over time. It is shown that quantization of the position of marker Mk-A in units of pixels (pixel accuracy) has a large effect on error for small displacements.

FIG. 15 shows the sub-pixel precision deflection values of the marker Mk-C at each time. The exemplified deflection values correspond to the y-direction displacement of marker Mk-C corrected using the positions of markers Mk-A and Mk-B. As for the amount of displacement shown in FIG. 15, the average deflection value and the standard deviation were 0.004 mm and 0.048 mm, respectively. This result shows that even when using images taken by a camera mounted on a drone flying (hovering) in front of the structure, the deflection value can be obtained with a high accuracy of 0.1 mm or less, regardless of changes in position and orientation due to flight. can be measured.

FIG. 16 shows the amount of image blurring at each time. The exemplified amount of image blur corresponds to the amount of displacement from the initial coordinates of the center of gravity of the marker Mk-C to the coordinates of the center of gravity of the marker Mk-C at each subsequent time. FIG. 16(a) shows the trajectory of the amount of image blur obtained without correcting the position of the marker Mk-C. The trajectory starts at the origin, moves right about 20 pixels, moves left about 40 pixels, and then moves toward the origin.
FIG. 16(b) shows the trajectory of the image blur amount obtained by correcting the position of the marker Mk-C with pixel accuracy. Due to the correction, the amount of image blur converges near the origin.
FIG. 16(c) corresponds to an enlarged view of FIG. 16(b). The amount of image blurring is within the range of about ±3 pixels in the x direction and about ±1 pixel in the y direction.
FIG. 16(d) shows the trajectory of the image blur amount obtained by correcting the position of the marker Mk-C with sub-pixel precision. Correction with sub-pixel precision converges the amount of image blur closer to the origin than correction with pixel precision.
Both FIGS. 16(e) and 16(f) correspond to enlarged views of FIG. 16(d). The amount of image blurring is within a range of about ±0.2 pixels in the x direction and about ±0.03 pixels in the y direction.
From the example shown in FIG. 16, even in a situation where the imaging unit 20 moves according to the present embodiment, image blur caused by movement is corrected with high accuracy. Therefore, the accuracy of the displacement amount measured using the captured image is improved.

The main purpose of the second experiment is to verify the displacement measured using the images captured by the camera moved using the drone.
FIG. 17 is a diagram showing an experimental optical system for displacement measurement by aerial photography. In the example shown in FIG. 17, the displacement in the y-direction was measured using an image captured using a camera mounted on a drone (deflection measurement). However, for comparison, a monofocal lens with a focal length of 35 mm was attached, and the displacement in the y direction was measured using an image (Fig. 17(a)) taken using a digital camera fixed to a tripod. . FIG. 17(a) represents three markers AC. The markers A and B on the left and right sides of the drawing were used as reference markers, and the central marker C was used as a measurement marker. The pitches in the x and y directions of the patterns of the markers A to C were set to 50 mm. The marker C was fixed to the moving stage, and the stage controller was used to control the height of the moving stage (displacement in the y direction (opposite to the vertical direction, sometimes called "upward")) by the operator's operation. The distance from the camera or drone to marker C was set to about 7 m, and the distance between marker A and marker C and the distance between marker B and marker C were both set to 3.1 m. This assumes deflection measurement at the center of a bridge with a total length of 6.2 m.

In this experiment, using a camera mounted on a small drone, a moving image (resolution: 3840 pixels × 2160 pixels) was captured at a height of about 2.5 m. One imaging time is about 50 seconds. No displacement in the y direction for the first 10 seconds, followed by an increase over 5 seconds until the displacement in the y direction is 5 mm, then the height is maintained for about 20 seconds, and a displacement in the y direction of 10 mm for 5 seconds. and then the height was maintained for 10 seconds.
Fig. 17(b) and Fig. 17(c) respectively show an image captured during flight of the small drone with a displacement of 0 at the start of imaging (0 seconds) and a displacement of 5 mm after 25 seconds from the start of imaging. 4 shows a captured image after deformation. In this case, the position of the camera with respect to the marker C, which is the object to be measured, is not the front, but the moving image with the tilt is taken downward from above. The positions of the left and right arrows in FIGS. 17B and 17C indicate that the position of the image is shifted vertically and horizontally. The difference in the position of the central arrow in each of FIGS. 17(b) and (c) includes the effect of parallax due to the difference in drone height. This means that it is not always necessary to position the drone in front of the object structure.

FIG. 18 shows experimental results of deflection measurement by aerial photography. FIG. 18(a) shows the xy plane trajectory of the position of each marker measured with sub-pixel accuracy. These trajectories show that hovering small drones change position over time. FIGS. 18B and 18C show the time series of displacement in the y direction measured using images captured by the camera mounted on the drone and the camera fixed to the tripod, respectively. The known displacement amount is 0 mm from the start of imaging until 10 seconds later, 5 mm from 15 seconds to 35 seconds later, and 10 mm after 40 seconds later. The displacement amount using the image captured using the camera mounted on the drone using this embodiment (FIG. 18(b)) is 0.095 mm for 15 seconds from the start of imaging until 10 seconds later. The amount of displacement was 4.744 mm after 35 seconds and 10.540 mm after 40 seconds. The amount of displacement analyzed by the conventional method using an image captured using a camera fixed to a tripod (Fig. 18(c)) is 0.034 mm for 10 seconds after the start of imaging. The amount of displacement was 5.035 mm from seconds to 35 seconds, and 10.051 mm after 40 seconds. From this experimental result, it was confirmed that an accurate amount of displacement can be obtained by using an image captured by a moving camera according to the present embodiment, as in the case of using an image captured by a fixed camera. . From this experiment, it was verified that according to this embodiment, the deflection measurement of a structure can be realized with high accuracy using an image taken by a drone.

(When there are 3 or more reference markers)
Next, an embodiment in which three or more reference markers are used will be described. The above AB correction requires placing the measurement marker on a straight line between the two reference markers in the reference frame. Depending on the placement of the reference marker, it may not be possible to place the measurement marker at the position where the displacement is desired to be measured. Further, as the angle formed by the direction of the measurement marker from the imaging unit 20 and the direction of the optical axis of the imaging unit 20 (that is, the imaging direction) increases, the distortion of the shape of the measurement marker appearing in the captured image becomes significant. Tend. Therefore, it is desirable to ease restrictions on the position of the imaging unit 20 and the placement of each marker.

Therefore, the first image correction unit 144 or the second image correction unit 148 uses an image representing three or more reference markers in each frame, and matches the positions of the individual reference markers between the reference frame and the measurement frame. Transformation parameters for coordinate transformation to coordinates in the measurement frame are calculated as follows. The first image correction unit 144 or the second image correction unit 148 corrects the positions of the measurement markers by coordinate transformation using the calculated transformation parameters. When three or more reference markers are used, the position of the measurement marker may be any position in the image of each frame as long as it differs from the position of the reference marker. Therefore, restrictions on the positions of the measurement markers are relaxed. More specifically, the positions of the measurement markers do not have to be on the straight line between each two reference markers as in the case of AB correction. It doesn't have to be inside. In addition, even if the direction of the measurement marker from the imaging unit 20 is an oblique direction that intersects the front direction, the shear distortion (skew, shear) of the form of the measurement marker that appears in the captured image is compensated. A decrease in accuracy is suppressed. When the number of reference markers is three, the first image correction unit 144 or the second image correction unit 148 can use linear transformation such as affine transformation as a method of coordinate transformation. In that case, translation, rotation, and scale as well as shear strain are taken into account.

When the number of reference markers is four, the first image correction unit 144 or the second image correction unit 148 can use homography transformation as the coordinate transformation method. The homography transformation is a linear transformation in which the four vertices P1-P4 of the quadrangle P become the vertices Q1-Q4 of the corrected quadrangle Q having different shapes (see FIG. 19), and is also called trapezoidal transformation. Point P5 in quadrilateral P is transformed to Q5 in quadrilateral Q by homographic transformation. When the point P5 is converted to the point Q5, the positional relationship with each vertex is maintained, and displacement due to more complicated deformation is compensated for by a simple calculation.

The first image correction unit 144 or the second image correction unit 148 performs the following operations so that the positions Q1 to Q4 of the four reference markers Mk-1 to Mk-4 in the corrected image of each frame are common between frames. It is sufficient to determine the coordinate transformation parameters used for homography transformation. The first image correction unit 144 or the second image correction unit 148 performs homography transformation on the coordinates (x, y) indicating the position P5 of the measurement marker Mk-5 before correction using the determined coordinate transformation parameters. The coordinates (u, v) of the position Q5 of the measurement marker Mk-5 after correction can be calculated.

For example, when the initial frame is used as the reference frame, the first image correction unit 144 or the second image correction unit 148 has four reference markers Mk-1 to Mk-4 represented in the reference frame. Set as position. The first image correction unit 144 or the second image correction unit 148 uses each other frame as a measurement frame, and converts the positions of the four reference markers Mk-1 to Mk-4 represented in each measurement frame to the reference positions. Define the coordinate transformation parameters for The first image correcting unit 144 or the second image correcting unit 148 uses the determined coordinate transformation parameters to calculate the position of the measurement marker Mk-5 represented in the measurement frame using Equation (3). to determine the position of the measurement marker Mk-5.

Equation (3) represents the coordinate transformation from coordinates (x, y) to coordinates (u, v). In equation (3), a to h are real numbers corresponding to coordinate transformation parameters. Equation (4) may be used instead of Equation (3). Coordinate transformation parameters a to h satisfy both equations (5) and (6).

Therefore, the first image correction unit 144 or the second image correction unit 148 sets the coordinates (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x _{3 )} indicating the positions of the four reference markers in the measurement frame. , y ₃ ), (x ₄ , y ₄ ) and the corresponding reference position coordinates (u ₁ , v ₁ ), (u ₂ , v ₂ ), (u ₃ , v ₃ ), (u ₄ , v ₄ ), coordinate transformation parameters a to h can be calculated so as to satisfy equations (5) and (6). More specifically, the first image correction unit 144 or the second image correction unit 148 calculates eight equations ( The coordinate transformation parameters a to h can be calculated by combining equations (7) and using a technique such as Gaussian elimination.

Next, an example of displacement measurement processing to which homography transformation is applied will be described. In the following description, the position of the reference marker in the measurement frame is corrected with sub-pixel accuracy, and the measurement marker in the measurement frame is calculated based on the coordinate transformation from the position of the reference marker in the reference frame to the position of the reference marker in the corrected measurement frame. A case of correcting the position of is taken as an example.

FIG. 20 is a flowchart showing a method using homography transformation as a third example of displacement measurement processing according to this embodiment. When executing the process illustrated in FIG. 20, similarly to the example of FIG. 4, the arithmetic processing unit 14 sets a reference frame as a reference for the amount of displacement, and the other frames are measurement frames to be corrected. do.
(Step S<b>202 ) The marker detection unit 142 detects markers for each frame shown in the image data input from the imaging unit 20 . The marker detection unit 142 determines the coordinates of the center of gravity of each detected marker as the center coordinates.
(Step S204) The first image correction unit 144 uses the arrangement of the four reference markers in the reference frame and the arrangement of the four reference markers in the measurement frame different from the reference time, and uses the arrangement of the individual reference markers in the measurement frame. Define the coordinate transformation parameters for the homography transformation of the position to the position of the corresponding reference marker in the reference marker. The first image correction unit 144 performs homography transformation on the positions of the measurement markers in the measurement frame using the determined coordinate transformation parameters to determine positions after correction with pixel accuracy (coarse correction).
(Step S206) The first displacement amount calculator 146 uses the sampling moiré method to analyze the phase difference between the moire images of the measurement marker in the reference frame and the measurement marker whose position in the measurement frame is corrected, and the analyzed phase difference , the displacement amount of the actual measurement marker is calculated as the first displacement amount.

(Step S208) For each reference marker, the second image correction unit 148 analyzes the phase difference between the moiré image of the reference frame and the moiré image of the measurement frame after position correction with pixel accuracy. The second image correction unit 148 calculates coordinate transformation parameters for homography transformation from the measurement frame to the reference frame based on the analyzed phase difference. The second image correction unit 148 uses the calculated coordinate transformation parameters to perform homography transformation on the position of the measurement marker in the measurement frame corrected with pixel accuracy, and obtains the position of the measurement marker after correction with sub-pixel accuracy. Determine the position of the measurement marker.
(Step S210) The second displacement amount calculation unit 150 uses the sampling moiré method to analyze the phase difference between the moiré image of the measurement marker of the reference frame and the moiré image of the measurement marker whose position is corrected with sub-pixel accuracy. , the displacement amount of the actual measurement marker is calculated as the second displacement amount from the analyzed phase difference.
(Step S212) The displacement amount output unit 152 selects either the first displacement amount obtained in step S206 or the second displacement amount obtained in step S210. The displacement amount output unit 152 causes the display unit 16 to display information on the selected displacement amount.

(Image correction device)
In the above description, the displacement measuring system 1 and the displacement measuring device 10 are exemplified, but they may be implemented in the form of the image correcting system 3 and the image correcting device 30 . The following description mainly focuses on the points of difference from the above embodiment, and unless otherwise specified, the same reference numerals are used and the above description is used.

FIG. 21 is a schematic block diagram showing a functional configuration example of the image correction system 3 according to this embodiment. The image correction system 3 includes an image correction device 30 and an imaging section 20 . The image correction device 30 includes a parameter input section 12 , an arithmetic processing section 34 and a display section 16 . The arithmetic processing section 34 includes a marker detection section 142 , a first image correction section 144 and a second image correction section 148 . The calculation processing unit 34 illustrated in FIG. 21 is obtained by omitting the first displacement amount calculation unit 146, the second displacement amount calculation unit 150, and the displacement amount output unit 152 from the calculation processing unit 14 illustrated in FIG. there is The first image correction unit 144 corrects the position with pixel accuracy by converting the coordinates of each position in the measurement frame so that the displacement of each reference marker is compensated between the measurement frame and the reference frame. An image of the measured frame is obtained. The second image correction unit 148 compensates for the displacement corresponding to the phase difference between the moiré image of the reference marker in the image of the measurement frame whose position is corrected with pixel accuracy and the moiré image of the reference marker in the image of the reference frame. By transforming the coordinates of each position in the measurement frame as described above, an image of the measurement frame in which the position is corrected with sub-pixel precision is obtained.

With this configuration, even when the imaging unit 20 is mounted on a moving object, image registration is realized so that the positions of a plurality of reference markers match between frames with sub-pixel accuracy. Alignment can be applied to monitoring various surface conditions based on temporal changes in luminance distribution in a predetermined evaluation area of a captured image. Further, the image is not limited to a visible image based on visible light emitted from a subject, and may be an infrared image, an ultraviolet image, an X-ray image, or the like. In addition, by applying a stress-stimulated luminescent material in advance to the evaluation region, the pattern (stimulated-stimulated luminescent image) that appears by the stress-stimulated luminescence can be used to analyze the stress distribution.

Next, an example of alignment of images by the image correction device 30 will be described. FIG. 22 is an explanatory diagram of a first example of alignment of images. In the example of FIG. 22, four reference markers Mk-A to Mk-D are placed on the surface of the subject within the imaging region (field of view) of the imaging unit 20, and the observations appearing on the surface within the evaluation region included in the imaging region As an example of the target object, a case of application to observation of changes in the shape of cracks (cracks) in a wall surface will be taken as an example. In that case, it is possible to observe the temporal evolution of the crack length, width, branching, etc. in the evaluation area using the image captured for each frame. The evaluation area may be any area included in the imaging area in each frame within the observation period.

FIG. 23 is an explanatory diagram of a second example of image registration. FIG. 23 is an application example for splicing multiple frame images into a single frame image. The first image correction unit 144 and the second image correction unit 148 acquire images of a plurality of frames, and adjust the positions in the acquired frames so that the positions of two or more reference frames commonly included in the acquired images match. to correct. Thereby, images of multiple frames can be spatially connected with sub-pixel precision. FIG. 23(a) illustrates two frames of images captured at different times. Image A in one frame represents a field of view A containing markers Mk-A1, Mk-B1, Mk-B2, Mk-A2, and Image B in the other frame represents markers Mk-C1, Mk-B1, Mk- B2, represents field of view B containing Mk-C2. The first image correcting unit 144 and the second image correcting unit 148 connect the image A and the image B so that the positions of the markers Mk-B1 and Mk-B2, which are common markers for both frames, match each other. Large area images can be formed. A plurality of frames may be captured simultaneously using separate imaging units 20, or may be captured using a single imaging unit 20 at different times. In addition, the first image correction unit 144 and the second image correction unit 148 sequentially connect each set of two-frame images including two or more reference markers in common in the field of view, and select one frame image from three or more frame images. Images with larger areas may be formed.

24A and 24B are explanatory diagrams illustrating experimental results of image alignment correction. In the experiment, images obtained by aerial photography at regular time intervals from an imaging unit 20 mounted on a small flying drone were used for alignment. FIG. 24A shows an image obtained by simply superimposing and synthesizing the image of the first frame and the image of the 90th frame. FIG. 24(a) shows that the positions of common graphics and characters appearing between two images taken at different times before correction are significantly different. Using the two markers Mk-A and Mk-B that appear, the corrected image obtained by performing AB correction on the image of the later frame of the two frames and the frame of the earlier time. An image obtained by synthesizing the captured image is shown. FIG. 24(b) shows that there is no difference in the layout of common figures and characters between the two images taken at different times, and the patterns of the two images almost completely overlap. As a result, it is possible to correct the blurring of the captured image taken from the drone. As an example, the image information (the letters NEW NORMAL! Changes can be easily detected. This embodiment can be applied not only to visible light-based images captured by a general camera, but also to images captured by an infrared camera, stress luminescence images, etc. It can be used for various image evaluations.

As described above, the displacement measuring apparatus 10 according to the present embodiment provides an image representing two or more reference markers that serve as a reference for the amount of displacement, and a measurement marker representing a pattern that is spatially repeated at a constant pitch. from the imaging unit 20 for each frame, and corrects the position of the measurement marker so as to compensate for the inter-frame displacement of the reference marker (for example, the first image correction unit 144, the second image correction unit 148 ). The displacement measuring apparatus 10 includes a displacement amount calculator (a first displacement amount calculator 146, a second displacement amount calculator 146, a 150).
According to this configuration, even if the position and orientation of the imaging unit 20 change for each frame, the change in each coordinate in the captured image is compensated. Therefore, it is possible to measure the displacement amount of the site where the measurement marker is placed from the images captured at different times without impairing the accuracy.

Further, in the image of the reference frame, which is the frame used as the reference for the amount of displacement, the position of the measurement marker is on a straight line passing between the two reference markers, and the image correction unit and the respective positions of the two reference markers in the image of the measurement frame, which is the frame for which the amount of displacement is to be measured. The amount of movement may be analyzed and the position of the measurement marker in the measurement frame may be corrected based on the scale factor, rotation amount and translation amount.
This configuration compensates for changes in size, rotation, and translation of the image of the subject that occur between frames with the positions of the two reference markers as references. Therefore, fluctuations in the position and orientation of the imaging unit 20 are compensated for by a simple calculation.

Further, in the image of the reference frame, which is the frame used as the reference for the amount of displacement, the position of the measurement marker is the internal division point between the two reference markers, and the image correcting unit performs the measurement on the frame that is the object of measurement of the amount of displacement. The positions of the measurement markers in the image of a measurement frame may be corrected based on the positions of the internal division points in the measurement frame.
This configuration compensates for changes in the position of the measurement marker that occur between frames with respect to the positions of the two reference markers. Therefore, fluctuations in the position and orientation of the imaging unit 20 can be compensated by a simple calculation without impairing accuracy.

Further, the number of reference markers is 3 or more, and the image correction unit calculates transformation parameters for coordinate transformation that matches the positions of the reference markers between frames, and corrects the positions of the measurement markers using the transformation parameters. good.
This configuration compensates for changes in the positions of the measurement markers between frames based on the positions of three or more reference markers. The position of the measurement marker can be arbitrarily set within the field of view of the imaging unit 20, and even if the direction from the imaging unit 20 to the measurement marker intersects the imaging direction of the imaging unit 20, the change in the position of the measurement marker is compensated. be. Therefore, the degree of freedom in the installation positions of the measurement markers and the imaging unit 20 is relaxed.

Also, the number of reference markers may be four, and the coordinate transformation may be homography transformation.
According to this configuration, it is possible to correct complicated deformation of an image based on the positions of the reference markers, as compared with the case where the number of reference markers is three. Therefore, the optical system of the imaging unit 20, noise, and the like can be corrected with higher resistance.

Each reference marker represents a pattern that is spatially repeated at a constant pitch. The positions of the measurement markers may be further corrected to compensate for the phase difference.
According to this configuration, the position of the reference marker can be corrected with finer accuracy than a pixel by using the moire image in which the luminance distribution representing the pattern of the reference marker is enlarged. Therefore, the accuracy of the measured displacement amount of the reference marker can be improved.

Further, the displacement measurement system 1 may include a moving body (for example, a drone) on which the imaging unit 20 is installed.
According to this configuration, even in a usage environment in which the imaging unit 20 cannot be fixed, it is possible to measure the displacement amount at the site where the measurement marker is installed using the image captured by the imaging unit 20 with high accuracy.

(Modification)
The displacement measurement system 1 and the image correction system 3 according to the above embodiment may be modified as follows. The imaging unit 20 and the displacement measuring device 10 or the image correcting device 30 may be connected via a wired or wireless network.
The displacement measuring device 10 or the image correcting device 30 does not necessarily have to be integrated with the parameter input section 12 and the display section 16 . In the displacement measuring device 10, one or both of the parameter input section 12 and the display section 16 may be omitted.
Further, the displacement measuring device 10 or the image correcting device 30 may include the imaging unit 20 and be configured as a single displacement measuring device 10 or image correcting device 30 .

In the above description, in the displacement measuring device 10 or the image correcting device 30, the first image correcting unit 144 and the second image correcting unit 148 adjust the position of each reference marker in the measurement frame to that of the corresponding reference marker in the reference frame. Although the case where the position of the measurement marker of the measurement frame is corrected so as to match the position has been mainly described, the present invention is not limited to this. The first image correction unit 144 and the second image correction unit 148 use the frame at each time as a measurement frame, and the position of each reference marker in each measurement frame is adjusted to the position of the corresponding reference marker in a reference frame separate from the measurement frame. The positions of the measurement markers for each measurement frame may be corrected to match the positions.

Moreover, although the case where the 1st displacement amount calculating part 146 and the 2nd displacement amount calculating part 150 used the luminance value as a signal value for each pixel was exemplified, the present invention is not limited to this. The first displacement amount calculation unit 146 and the second displacement amount calculation unit 150 use color signal values, for example, signal values of each color such as red, green, and blue, or combinations of these signal values, as signal values for each pixel. good too.
The first image correction unit 144 and the second image correction unit 148 may perform AB correction using two predetermined reference markers among the three or more reference markers.
In the above description, the center of gravity was used as an example of the representative point representing the position of each marker, but the present invention is not limited to this. The representative point may be one predetermined vertex, for example, the lower left vertex in the drawing.

Further, in the displacement measuring device 10 or the image correcting device 30, the first image correcting section 144 and the second image correcting section 148 may be integrated to form a single image correcting section.
In the displacement measuring device 10, the first displacement amount calculator 146 and the second displacement amount calculator 150 may be integrated to form a single displacement amount calculator.

In addition, in the displacement measuring device 10, the second image correcting section 148 and the second displacement amount calculating section 150 may be omitted. In that case, correction with sub-pixel accuracy is not performed. The signal values of the patterns represented by the reference markers do not necessarily fluctuate regularly. The first displacement amount calculator 146 outputs information on the first displacement amount calculated with pixel accuracy to the display unit 16 as information indicating the displacement amount of the measurement marker.
Also, one or both of the parameter input section 12 and the display section 16 may be omitted. Various parameters used to calculate the displacement amount may be set in advance or may be input from another device. Information on the first displacement amount calculated by the first displacement amount calculation unit 146 and the information on the displacement amount acquired by the displacement amount output unit 152 may be accumulated in the device itself or may be output to another device.

In the above description, it is assumed that the pitch of the repeating pattern represented by each marker is common among the markers, but it may be different for each marker. For example, the pitch may be larger for measurement points that are farther from the imaging unit 20 . In this case, even if the distance from the imaging unit 20 increases, the period of the pattern displayed on the image does not decrease, so deterioration of measurement accuracy due to the distance from the imaging unit 20 can be prevented or mitigated.

A part of the displacement measuring device 10 or the image correcting device 30 in the above-described embodiments, for example, the arithmetic processing unit 14 may be realized by a computer. In that case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. The "computer system" here is a computer system built into the displacement measuring device 10 or the image correcting device 30, and includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically stores a program for a short period of time, such as a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include a volatile memory inside a computer system that serves as a server or client in that case, which holds the program for a certain period of time. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.
Further, part or all of the displacement measuring device 10 or the image correcting device 30 in the above-described embodiments may be realized as an integrated circuit such as LSI (Large Scale Integration). Each functional block of the displacement measuring device 10 or the image correcting device 30 may be individually processorized, or part or all of them may be integrated and processorized. Also, the method of circuit integration is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integration circuit technology that replaces LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configurations are not limited to those described above, and various design changes can be made without departing from the gist of the present invention. It is possible to

DESCRIPTION OF SYMBOLS 1... Displacement measuring system 3... Image correction system 10... Displacement measuring device 12... Parameter input part 14, 34... Arithmetic processing part 16... Display part 30... Image correction apparatus 142... Marker detection part 144 146 : first displacement amount calculation unit 148 : second image correction unit 150 : second displacement amount calculation unit 152 : displacement amount output unit

Claims (12)

acquiring an image representing two or more reference markers used as a reference for the amount of displacement and a measurement marker representing a pattern spatially repeated at a constant pitch from the imaging unit for each frame;
an image correction unit that corrects the position of the measurement marker to compensate for inter-frame displacement of the reference marker;
a displacement amount calculation unit that calculates a displacement amount of the measurement marker from a phase difference between frames of moire images generated from the pattern of the measurement marker;
A displacement measuring device comprising: In the image of the reference frame, which is the frame used as the reference for the amount of displacement, the position of the measurement marker is on a straight line passing between the two reference markers,
The image correcting unit is configured to, based on the position of each of the two reference markers in the reference frame and the position of each of the two reference markers in the image of the measurement frame, which is the frame to be measured for the amount of displacement, , analyzing the scale ratio, the amount of rotation, and the amount of translation of the image of the measurement frame from the reference frame;
The displacement measuring device according to claim 1, wherein the positions of the measurement markers in the measurement frame are corrected based on the scale factor, the amount of rotation, and the amount of translational movement. In the image of the reference frame, which is the frame used as the reference for the amount of displacement, the position of the measurement marker is an internal dividing point between the two reference markers,
2. The displacement according to claim 1, wherein the image correction unit corrects the position of the measurement marker in the image of the measurement frame, which is the frame to be measured for the amount of displacement, based on the position of the internal division point in the measurement frame. measuring device. The number of reference markers is 3 or more,
The image correction unit
2. The displacement measuring device according to claim 1, wherein a transformation parameter for coordinate transformation that matches the position of the reference marker between frames is calculated, and the position of the measurement marker is corrected using the transformation parameter. The number of reference markers is 4,
The displacement measuring device according to claim 4, wherein the coordinate transformation is homography transformation. each of the reference markers represents a pattern that is spatially repeated at a constant pitch;
6. The image correction unit further corrects the position of the measurement marker so as to compensate for a phase difference between frames of a moire image generated from the pattern of the reference marker. Displacement measuring device as described. acquiring images representing two or more reference markers each representing a pattern that is spatially repeated at a constant pitch from the imaging unit for each frame;
correcting the image to compensate for the frame-to-frame displacement of the fiducial marker;
An image correcting apparatus comprising an image correcting unit that further corrects the image so as to compensate for a phase difference between frames of a moire image generated from the pattern of the reference marker. to the computer,
A program for functioning as the displacement measuring device according to any one of claims 1 to 6 or the image correcting device according to claim 7. A system comprising the imaging unit, the displacement measuring device according to any one of claims 1 to 6, or the image correcting device according to claim 7. 10. The system according to claim 9, comprising a moving body on which said imaging unit is installed. A displacement measuring device
Acquiring an image representing two or more reference markers used as a reference for the amount of displacement and a measurement marker representing a pattern repeated at a constant pitch from the imaging unit for each frame;
an image correction step of correcting the positions of the measurement markers to compensate for inter-frame displacements of the reference markers;
a displacement amount calculation step of calculating a displacement amount of the measurement marker from a phase difference between frames of moiré images generated from the pattern of the measurement marker;
Displacement measurement method to perform. The image correction device
acquiring images representing two or more reference markers each representing a pattern that is spatially repeated at a constant pitch from the imaging unit for each frame;
correcting the image to compensate for the frame-to-frame displacement of the fiducial marker;
an image correction step of further correcting the image so as to compensate for phase differences between frames of a moire image generated from the pattern of the reference marker.

Images