JP6970893B2 - Displacement detection device and displacement detection method - Google Patents

Description

The present disclosure relates to a displacement detection device and a displacement detection method for detecting the displacement of an object to be measured.

Patent Document 1 provides a single image by providing a plurality of grid patterns on a measurement object, photographing the measurement object at predetermined time intervals, and calculating the temporal displacement of each lattice pattern reflected in the photographed image. Obtain multiple points of displacement from the row.

Japanese Unexamined Patent Publication No. 2015-141151

The present disclosure provides a displacement output device that detects the displacement of the entire object from the captured image obtained by capturing the object.

The mutation detection device in the present disclosure detects spatial displacement with the passage of time at each of a plurality of measurement points set on the object by using a plurality of captured images in which the object is imaged at a plurality of times. The object is extracted from the detection unit, the extraction unit that extracts the characteristic displacement indicating the characteristic displacement of the object from the displacement detected by the detection unit, and the characteristic displacement extracted by the extraction unit. It is characterized by including a calculation unit for calculating the total displacement indicating the total displacement.

The mutation detection method in the present disclosure is a displacement calculation method performed by a displacement detection device including a detection unit, an extraction unit, and a calculation unit, and the detection unit captures a plurality of captured images of an object at a plurality of times. From the detection step of detecting the spatial displacement with the passage of time with respect to a plurality of measurement points set in the object and the amount of change detected by the extraction unit in the detection step, the object is described. An extraction step for extracting a characteristic displacement indicating a characteristic displacement of an object, and a calculation step for the calculation unit to calculate an overall displacement indicating the displacement of the entire object from the characteristic displacement extracted by the extraction step. It is characterized by having and.

The displacement detection device and the displacement detection method in the present disclosure can detect the displacement of the entire object from a plurality of captured images of the object.

It is an external view which shows one configuration example of the displacement detection system in embodiment. It is a block diagram which shows one configuration example of the displacement detection device in embodiment. It is a block diagram which shows the other configuration example of the displacement detection apparatus in embodiment. It is a flowchart which shows the operation of the displacement detection device in embodiment. It is a flowchart which shows the other operation of the displacement detection apparatus in embodiment. It is a figure which shows an example of the captured image of a bridge. It is a figure which shows another example of the captured image of a bridge. It is a figure which shows the arrangement example of the measurement point set on a bridge. It is a figure which shows an example of the displacement detected by the detection part. It is a figure which shows an example which visualized the 1st principal component. It is a figure which shows an example which visualized the 2nd principal component. It is a figure which shows an example which visualized the 3rd principal component. It is a figure which shows an example which visualized the 4th principal component. It is a figure which shows an example which visualized the 5th principal component. It is a figure which shows the detected displacement and the calculated total displacement of an object. It is a figure which shows an example which visualized the main component when the object is a power transmission tower.

The displacement detection device according to one embodiment of the embodiment uses a plurality of captured images in which an object is captured at a plurality of times, and a space with the passage of time for each of a plurality of measurement points set on the object. A detection unit that detects a specific displacement, an extraction unit that extracts a characteristic displacement indicating a characteristic displacement of the object from the displacement detected by the detection unit, and a characteristic displacement extracted by the extraction unit. Therefore, it is characterized by including a calculation unit for calculating the total displacement indicating the displacement of the entire object.

According to the displacement detection device, the displacement of the entire object can be detected.

For example, further, the extraction unit may perform the extraction by performing a principal component analysis on the displacement detected by the detection unit to extract the main component.

As a result, the displacement detection device can express the characteristic displacement extracted by the extraction unit as a physical quantity having no linear correlation with each other.

For example, further, it is assumed that the calculation unit performs the calculation by synthesizing the principal components extracted by the extraction unit, excluding one or more principal components on the lowest side in descending order of eigenvalues. May be good.

As a result, the displacement detection device can make the calculated total displacement a physical quantity obtained by excluding the non-major main component, that is, the noise component.

For example, further, the calculation unit may perform the calculation by calculating the temporal intensity change of the principal component for each principal component to be calculated.

As a result, the displacement detection device can make the calculated total displacement a physical quantity useful for analyzing the vibration mode in the object.

For example, the calculation unit may further perform the calculation by calculating the frequency characteristic of the temporal intensity change of the principal component for each principal component to be calculated.

This makes it possible for this displacement detection device to use the calculated total displacement as a physical quantity useful for analyzing the natural frequency of the object.

For example, further, with respect to the plurality of measurement points detected by the detection unit based on the displacement of at least one or more reference measurement points among the plurality of measurement points detected by the detection unit. A correction unit for correcting the displacement may be provided, and the extraction unit may perform the extraction using the displacement corrected by the correction unit.

As a result, the displacement detection device can correct the influence of the shaking of the image pickup device used for taking an image.

For example, the detection unit further detects the displacement in the plurality of captured images as the displacement, and further, the displacement for each of the plurality of measurement points detected by the detection unit is the plurality of measurement points. The calculation unit may perform the calculation by using the displacement scaled by the scaling unit, which is provided with a scaling unit for scaling correction so as to reflect the ratio of the actually displaced distances.

As a result, this displacement detection device can calculate the total displacement with higher accuracy.

For example, further, each of the plurality of captured images is an acceleration image showing acceleration, the detection unit performs the calculation so that the displacement to be detected is expressed by using the acceleration, and the extraction unit extracts. The extraction may be performed so that the characteristic displacement to be expressed is expressed by using acceleration.

This makes it possible for this displacement detection device to use the calculated total displacement as a physical quantity useful for analyzing the spatial acceleration distribution of the entire object.

For example, further, each of the plurality of captured images is a velocity image indicating a velocity, the detection unit performs the calculation so that the displacement to be detected is expressed using the velocity, and the extraction unit extracts. The extraction may be performed so that the characteristic displacement to be expressed is expressed by using the velocity.

This makes it possible for this displacement detection device to use the calculated total displacement as a physical quantity useful for analyzing the spatial velocity distribution of the entire object.

For example, further, each of the plurality of captured images is a distance image indicating a distance, and the detection unit may detect the displacement in the three-dimensional space as the displacement.

As a result, the displacement detection device can make the calculated total displacement a physical quantity useful for analyzing the three-dimensional displacement of the entire object.

For example, it may further include a superimposed image generation unit that generates a superimposed image in which an image based on the displacement calculated by the calculation unit is superimposed on at least one of the plurality of captured images.

As a result, the user who uses this displacement detection device can visually grasp the total displacement of the object.

For example, further, the plurality of captured images may include images in which the object is synchronously imaged by a plurality of imaging devices.

As a result, this displacement detection device can calculate the total displacement even for an object having a shape or range that cannot be imaged by one camera.

The displacement detection method according to one aspect of the embodiment is a displacement calculation method performed by a displacement detection device including a protrusion, an extraction unit, and a calculation unit, and the detection unit captures an object at a plurality of times. A detection step for detecting spatial displacement with the passage of time with respect to a plurality of measurement points set on the object by using the plurality of captured images, and a change detected by the detection step in the extraction unit. An extraction step that extracts a characteristic displacement indicating a characteristic displacement of the object from the quantity, and a total displacement indicating the displacement of the entire object from the characteristic displacement extracted by the extraction step by the calculation unit. It is characterized by having a calculation step for calculating.

According to the above displacement detection method, the displacement of the entire object can be detected.

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art.

It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims. ..

(Embodiment)
Hereinafter, embodiments will be described with reference to FIGS. 1 to 9.

[1-1. composition]
[1-1-1. Imaging of the object]
FIG. 1 is an external view showing a configuration example of the displacement detection system 1 according to the embodiment. The displacement detection system 1 includes an image pickup device 101 and a displacement detection device 200. The image pickup apparatus 101 images the object 102 a plurality of times in a predetermined period. The image pickup apparatus 101, for example, takes an image of the object 102 at predetermined time intervals. A plurality of captured images captured by the image pickup device 101 are input to the displacement detection device 200. The displacement detection device 200 calculates the total displacement indicating the displacement of the entire object 102 from the plurality of input captured images. In the present embodiment, the case where the image pickup device 101 is a camera and the object 102 is a bridge will be described as an example.

[1-1-2. Configuration of change amount detection device]
FIG. 2 is a block diagram showing a configuration example of the displacement detection device 200 according to the embodiment. As shown in FIG. 2, the displacement detection device 200 includes an input / output I / F 210, a CPU 220, a detection unit 230, an extraction unit 240, a calculation unit 250, and a memory 260.

The input / output I / F 210 accepts inputs of a plurality of captured images obtained by capturing the bridge 102 for a predetermined period. Then, the total displacement of the bridge 102 calculated by the calculation unit 250 is output. The input / output I / F 210 accepts inputs of a plurality of captured images captured by the camera 101 via wireless, wired, recording media, or the like, and stores them in the memory 260. Further, the input / output I / F 210 outputs the total displacement of the bridge 102 calculated by the calculation unit 250 to the display unit (not shown) or the like via wireless, wired or recording medium. The display unit displays the total displacement output from the displacement detection device 200.

The CPU 220 controls the operation of each part. The CPU 220 has, for example, a non-volatile memory in which a program is stored, a volatile memory which is a temporary storage area for executing a program, an input / output port, a processor for executing the program, and the like.

The detection unit 230 detects spatial displacement with the passage of time for each of the plurality of measurement points set on the object by using a plurality of captured images in which the object is imaged at a plurality of times. More specifically, the detection unit 230 detects the bridge 102 existing in the captured image for each captured image for the plurality of captured images captured by the camera 101 stored in the memory 260. Then, the detection unit 230 detects the displacement, which is the amount of spatial change at the plurality of measurement points set on the bridge 102, and stores it in the memory 260.

The extraction unit 240 extracts a characteristic displacement indicating a characteristic displacement of the object from the detected displacement. More specifically, the extraction unit 240 performs principal component analysis (PCA: Principal) of the spatial displacement principal component that is spatially simultaneously displaced with respect to the displacements obtained at a plurality of measurement points at a plurality of time by the detection unit 230. It is extracted using Component Analysis) and stored in the memory 260.

The calculation unit 250 calculates the total displacement indicating the displacement of the entire object from the characteristic displacement extracted by the extraction unit 240. More specifically, the calculation unit 250 calculates the total change amount of the bridge 102 from one or more main components extracted by the extraction unit 240 and stores it in the memory 260. Here, the calculation unit 250 performs the above calculation by synthesizing the five principal components on the uppermost side among the principal components extracted by the extraction unit 240.

The memory 260 stores the captured image input from the input / output I / F 210. Further, the memory 260 is used as a working memory of each part. For example, the memory 260 stores the displacement detected by the detection unit 230. The memory 260 stores the main components extracted by the extraction unit 240. The memory 260 stores the total displacement of the bridge 102 calculated by the calculation unit 250. The memory 260 is composed of a semiconductor storage element capable of high-speed operation, such as a DRAM.

[1-1-3. Other configurations of displacement detector]
In a plurality of captured images captured by the camera 101, the bridge 102 is not always captured at the same position. In such a case, an error occurs in the displacement detected by the detection unit 230. In order to solve this problem, the displacement detection device may be provided with a function of correcting the displacement detected by the detection unit 230.

FIG. 3 is a block diagram showing another configuration of the displacement detection device according to the embodiment. In the displacement detection device 201 of FIG. 3, the same reference numerals are given to the components that perform the same operation as the displacement detection device 200 of FIG. 2, and the description thereof will be omitted.

The CPU 221 controls the operation of each part. The CPU 221 has, for example, a non-volatile memory in which a program is stored, a volatile memory which is a temporary storage area for executing a program, an input / output port, a processor for executing the program, and the like.

The correction unit 270 relates to the plurality of measurement points detected by the detection unit 230 based on the displacement of at least one or more reference measurement points among the plurality of measurement points detected by the detection unit 230. Correct the displacement. More specifically, the correction unit 270 corrects the displacements of other measurement points based on the displacements of the fixed measurement points set on the bridge 102 in the captured image, and stores them in the memory 260. The fixed measurement point is, for example, a point that is assumed to have the smallest amount of change among the measurement points.

The extraction unit 241 extracts the spatial displacement principal component that is spatially simultaneously displaced with respect to the displacements obtained at the plurality of measurement points at a plurality of times by using the principal component analysis, and stores it in the memory 260.

[1-2. motion]
[1-2-1. Operation without correction]
FIG. 4 is a flowchart showing the operation of the displacement detection device 200 according to the embodiment.

(Step 310)
The CPU 220 acquires a captured image. The CPU 220 acquires an image captured by the camera 101 for a predetermined period of time via the input / output I / F 210 and stores it in the memory 260.

(Step 320)
The CPU 220 causes the detection unit 230 to detect temporal displacements at a plurality of measurement points set on the bridge 102. The detection unit 230 takes out a plurality of captured images stored in the memory 260 in the order of imaging time, and detects the displacement of the bridge 102 for each captured image. The detection unit 230 stores the detected displacement in the memory 260 (detection step).

(Step 340)
The CPU 220 causes the extraction unit 240 to extract the spatial main component with respect to the displacement at the plurality of measurement points detected by the detection unit 230. The extraction unit 240 reads out the displacements at the plurality of measurement points stored in the memory 260 and extracts the spatial main component. The extraction unit 240 stores the extracted main component in the memory 260 (main component extraction step).

(Step 350)
The CPU 220 causes the calculation unit 250 to calculate the total displacement of the bridge 102 using the main component extracted by the extraction unit 240. The calculation unit 250 reads out the main component stored in the memory 260, calculates the total displacement of the bridge 102, and stores it in the memory 260. The CPU 220 outputs the total displacement stored in the memory 260 via the input / output I / F 210 (calculation step).

[1-2-2. Operation when making correction]
FIG. 5 is a flowchart showing another operation of the displacement detection device according to the embodiment. FIG. 5 shows the operation of the displacement detection device 201.

In FIG. 5, the steps that perform the same operation as the flowchart of FIG. 4 are designated by the same reference numerals, and the description thereof will be omitted.

(Step 330)
The CPU 220 causes the correction unit 270 to correct the displacement at the plurality of measurement points detected by the detection unit 230. The correction unit 270 reads out the temporal displacements at the plurality of measurement points stored in the memory 260, and corrects each displacement with reference to the structure included in the plurality of captured images. The correction unit 270 stores the corrected displacement in the memory 260.

(Step 341)
The CPU 220 causes the extraction unit 241 to extract the spatial main component with respect to the displacement at the plurality of measurement points corrected by the correction unit 270. The extraction unit 241 reads out the displacements at the plurality of measurement points stored in the memory 260, and extracts the spatial main component. The extraction unit 241 stores the extracted main component in the memory 260.

The processes of steps 330 and 341 may use different procedures that are mathematically equivalent, and as a result, they may be collectively processed as an integrated procedure.

[1-2-3. Operation example]
Here, an operation example of the displacement detection device 201 will be described.

As shown in FIG. 1, the CPU 221 stores a plurality of captured images captured by the camera 101 on the bridge 102 in the memory 260 via the input / output I / F 210.

FIG. 6A shows an example of a captured image of the bridge 102. Further, FIG. 6B shows another example of the captured image of the bridge 102. The captured image 400 and the captured image 401 are captured images obtained by capturing the bridge 102 at different times. The captured images 400 and 401 show that the vehicle 402 that serves as a load exists on the bridge 102. In the captured image 400, the vehicle 402 is on the support of the bridge 102, and the bridge 102 is not displaced. On the other hand, in the captured image 401, the vehicle 402 is near the center of the bridge 102, and the bridge 102 is displaced. In this way, an object different from the bridge 102 (for example, a vehicle 402) may be captured in the captured image.

The detection unit 230 detects the bridge 102 existing in the captured image by using the existing image recognition technique. The detection unit 230 detects the coordinates of a plurality of measurement points set on the detected bridge 102.

FIG. 7 is a diagram showing an arrangement example of measurement points set on the bridge 102.

In FIG. 7, black rectangles 501 to 511 indicate measurement points set on the bridge 102. The measurement point may be set by the user in advance, or may be set after the bridge 102 is automatically detected by image recognition. In FIG. 7, the measurement points are set at substantially equal intervals, but the effect of the present embodiment can be obtained even at non-equal intervals. Here, in the present embodiment, at least one measurement point from the plurality of measurement points is set as a fixed measurement point. A fixed measurement point is a measurement point that is least affected by weight and has a smaller displacement than other measurement points. In the embodiment, as fixed measurement points, measurement points 501 and measurement points provided near the contact points of the pier 103 of the bridge 102 with an abutment (not shown) installed in the ground supporting the pier 103. 511 is used.

The detection unit 230 takes out a plurality of captured images stored in the memory 260 in the order of imaging time, and detects the displacement of the bridge 102 for each captured image. The detection unit 230 detects, for example, the displacement at each measurement point between the captured image 400 and the captured image 401. As a displacement detection method in the captured image, the detection unit 230 includes block matching, a correlation method such as a normalized cross correlation method and a phase correlation method, a sampling moire method, a feature point tracking method, and a laser. A general displacement detection method such as the speckle correlation method can be used. The accuracy of displacement detection may be in pixel units or sub-pixel units.

FIG. 8 is a diagram showing an example of the displacement detected by the detection unit 230. FIG. 8 shows captured images (Flame1, Frame2, Frame3, ..., Frame) obtained by capturing the bridge 102 for a predetermined period.
The position coordinates (x, y) of the measurement points 501 to 511 in n) are shown.

Here, the position coordinates (x, y) of the i-th measurement point Pi in the captured image Frame t captured at time t are expressed as Pi (x, y, t). Further, the displacement of the i-th measurement point Pi in Frame t is expressed as Di (x, y, t). The displacement Di (x, y, t) is the difference in the position coordinates Pi of the measurement points between different captured images. In this embodiment, i = 1 to 11, and the measurement points P1 to P11 correspond to the measurement points 501 to 511.

For example, the displacement Di (x, y, t) can be calculated by the following equation using the position coordinates Pi (x, y, t) between the captured images that are adjacent in time.

(Number 1)
Di (x, y, t) = Pi (x, y, t) -Pi (x, y, t-1)

Further, for the displacement Di (x, y, t), for example, the first captured image or the captured image in which the object can be regarded as a steady state is defined as the reference captured image, and the position coordinates between the reference captured image and each captured image are defined. It may be calculated by the following equation.

(Number 2)
Di (x, y, t) = Pi (x, y, t) -Pi (x, y, 0)
Here, Pi (x, y, 0) is a position coordinate in the reference captured image.

The detection unit 230 corrects the image distortion of the imaging optical system as necessary. Further, the detection unit 230 is necessary when the ratio of the displacement on the captured image and the displacement on the real space is caused by the difference in the real space distance from the image pickup position of the camera 101 to the measurement point on the bridge 102. Therefore, scale correction is performed so that this ratio becomes equal. Such correction may be performed on the captured image or may be performed on the calculated displacement.

As an example, the displacement detection device 201 is a scaling unit that scale-corrects the displacement of each of the plurality of measurement points detected by the detection unit 230 so as to reflect the ratio of the distances actually displaced at the plurality of measurement points. You can think of an example that has. In this case, the scaling unit may, for example, store the coordinates in the real space for each measurement point and perform the scale correction using the stored coordinates in the real space.

The correction unit 270 reads out the displacement Di (x, y, t) of each measurement point stored in the memory 260. The correction unit 270 is based on the displacement D1 (x, y, t) of the fixed measurement point P1 (measurement point 501) among the measurement points, and starts from the displacement Di (x, y, t) of each measurement point. , The displacement D1 (x, y, t) of the fixed measurement point is subtracted for each captured image. This makes it possible to eliminate the influence of image displacement that occurs when the orientation of the camera in the x and y directions changes during imaging.

Further, the correction unit 270 sets a fixed measurement point P11 (measurement point 511) different from the fixed measurement point P1 so that the value of the displacement D11 (x, y, t) of the fixed measurement point P11 approaches 0. The x, y coordinate values of the displacement Di (x, y, t) of each measurement point may be rotationally transformed around the position of the fixed measurement point P1. This makes it possible to eliminate the influence of displacement of each captured image caused by a change in the rotation direction of the camera during imaging. The correction unit 270 stores the corrected displacement of each measurement point in the memory 260.

The fixed measurement point may be set on the bridge 102 or may be set other than the bridge 102. For example, a fixed measurement point may be set to a stationary object (building or the like) in the background of the captured image. Further, the number of fixed measurement points may be increased, and the translation correction and rotation correction amount of each measurement point in the x and y directions may be optimized so that the total displacement of each fixed measurement point is minimized. This makes it possible to reduce the influence on displacement detection due to changes in the rotation and orientation of the camera during imaging. Further, the movement of the frame images at different times may be analyzed to detect the dominant movement (global movement) of the entire image, and the points in the image following this movement may be set as fixed measurement points.

If the displacement caused by the orientation or rotation of the camera is expected to be within the allowable range in the captured image, the correction of the displacement by the correction unit 270 may be omitted.

The extraction unit 241 sets u (t) as a characteristic displacement that is a set of displacements at time t of a plurality of measurement points with respect to the displacement Di (x, y, t) at time t, and u (u (t) at different times. The principal component ei (i = 1 to n, n is the number of principal components) with respect to t) is obtained. Then, the extraction unit 241 stores m (m ≦ n) of the extracted n main components in the memory. The number m of the principal components to be stored may be specified from the one having the largest eigenvalue, or may be set using the cumulative contribution rate. Alternatively, a predetermined eigenvalue order component may be selected. As a method for calculating the principal component, a general method such as diagonalization of the covariance matrix can be used. The extraction unit 240 stores the calculated m main components in the memory 260.
Here, for example, m is 5, and the extraction unit 241 will explain that the top five principal components are stored in the memory in descending order of the eigenvalues.

9A to 9E are diagrams showing an example of visualizing the top five principal components (first principal component to fifth principal component) in descending order of eigenvalues. 9A to 9E show that each of the first principal component to the fifth principal component of the total displacement is divided for each measurement point, and the vector is superimposed and displayed on the captured image captured by the bridge 102.

In FIG. 9A, the vectors 902a to 910a show the vector of the first principal component at the measurement points 502 to 510. Similarly, in FIG. 9B, the vectors 902b to 910b indicate the vector of the second principal component of the measurement points 502 to 511. Similarly, in FIG. 9C, the vectors 902c to 910c indicate the vector of the third principal component of the measurement points 502 to 511. Similarly, in FIG. 9D, the vectors 902d to 910d indicate the vectors of the fourth principal components of the measurement points 502 to 511. Similarly, in FIG. 9E, the vectors 902e to 910e indicate the vector of the fifth principal component of the measurement points 502 to 511. However, in these figures, the measurement points without the vector display indicate that the size of the extracted vector is 0.

As can be seen from these figures, these five principal components show the characteristic displacements of the bridge 102, which is the object.

The calculation unit 250 synthesizes the total displacement u'(t) from the m main components stored in the memory 260. The calculation unit 250 calculates using u'(t) = Σi {ai (t) ei}.

Here, ai (t) is set to a coefficient with respect to ei, and ai (t) is set so that | u'(t) -u (t) | ^ 2 is the minimum. The output unit 207 outputs the calculated total displacement u'(t) as the displacement Di'(x, y, t) of each measurement point on the bridge 102.

The CPU 220 displays the total displacement u'(t) calculated by the calculation unit 250 on the display unit (not shown) via the input / output I / F 210.

FIG. 10 is a diagram showing the detected displacement and the calculated total displacement of the bridge 102. In FIG. 10, the dotted line L1 indicates the displacement Di (x, y, t) at the measurement point. Further, the solid line L2 indicates the total displacement u'(t) which is the total change amount of the bridge 102. As shown in FIG. 10, the total displacement of the bridge 102 by the vehicle 402 can be shown more accurately on the solid line L2 than on the dotted line L1, that is, the influence of the error component can be reduced.

If there is a sudden influence of noise in the above-mentioned principal component extraction, the influence of noise may be suppressed by performing noise removal in advance or by using robust principal component analysis.

[1-3. Effect, etc.]
As described above, in the change amount detecting device of the present embodiment, the detection unit 230 changes with time at a plurality of measurement points set on the bridge 102 in a plurality of captured images obtained by capturing the bridge 102 for a predetermined period. The extraction unit 240 detects the amount, the spatial main component is extracted with respect to the temporal change amount at a plurality of measurement points, and the calculation unit 250 calculates the total change amount of the bridge 102 from the extracted main component. do.

As a result, the displacement of multiple measurement points measured at the same time is regarded as the total change amount of the object, and the spatial principal component of the change amount is extracted to obtain prior knowledge about the geometric structure and mechanical characteristics of the object. Without using it, it is possible to obtain the characteristics of the total amount of change peculiar to the object.

Therefore, with a simple and low-cost configuration, displacements are synthesized based on the characteristics of the total amount of change of the object, displacement detection errors at individual measurement points are suppressed, and more accurate displacement detection becomes possible. ..

Further, by capturing a wide range including the measurement points and fixed measurement points on the object with a camera, it is possible to detect a large number of displacements of the object while suppressing the influence of camera shake.

(Other embodiments)
As described above, embodiments have been described as an example of the techniques disclosed in this application. However, the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, replacements, additions, omissions, etc. have been made. It is also possible to combine the components described in the above embodiment to form a new embodiment.

Therefore, other embodiments will be exemplified below.

As shown in FIG. 1, the image pickup apparatus 101 may be a separate body from the displacement detection apparatus 200, or may be provided in the displacement detection apparatus 200. Further, the captured image may be a monochrome image or a color image (including multispectral). Further, the image pickup apparatus 101 may output array data obtained by sensing an object using a distance measuring sensor or an acceleration sensor instead of a normal camera as an image. That is, the captured image used by the displacement detection device 200 may be, for example, an acceleration image showing acceleration or a velocity image showing speed. For example, when the captured image used by the displacement detection device 200 is an acceleration image, the detection unit 230 calculates so that the displacement to be detected is expressed using the acceleration, and the extraction unit 240 extracts the characteristic. Extraction may be performed so that the displacement is expressed using acceleration. For example, when the captured image used by the displacement detection device 200 is a velocity image, the detection unit 230 calculates so that the displacement to be detected is expressed using the velocity, and the extraction unit 240 extracts the characteristic. Extraction may be performed so that the displacement is expressed using velocity.

In the embodiment, the image pickup apparatus 101 is composed of one camera, but a plurality of cameras that capture the same object may be used. In this case, using the captured images synchronously captured by a plurality of cameras, processing is performed up to step 330 for each captured image captured by each camera, and the amount of change obtained from the plurality of camera images is used as a set for steps 340 and thereafter. Similar processing can be performed. As a result, it is possible to calculate the total displacement with high accuracy even for an object having a shape or range that cannot be imaged by one camera.

Further, although the two-dimensional displacement Di (x, y) is detected in the embodiment, even if the distance image (depth image) is acquired and the three-dimensional displacement Di (x, y, z) is detected. good. After the displacement is detected, a highly accurate three-dimensional displacement can be obtained by executing the same procedure as that of the embodiment. As a distance image acquisition method, a general distance image acquisition method such as a stereo camera by synchronous imaging of a plurality of cameras, a multi-view camera stereo method, a pattern projection method, a Time of flight (TOF) camera, or a laser displacement meter can be used. ..

Further, in the embodiment, a bridge is exemplified as an object, but the same effect can be obtained not only with a bridge but also with a building such as a building, a steel tower, a chimney, a wall surface, a floor material, a plate material, a steel scaffold, a road surface, a railroad track, a car body, and the like. ..

FIG. 11 shows the top five principal components (first principal component to fifth principal component) in the order of the highest eigenvalues among the principal components extracted by the extraction unit 241 when the object is the power transmission tower 110. It is a figure which shows an example which visualized.

In the figure, the first principal component 111 is a vector group in which the first principal component of the total displacement is visualized by dividing it into measurement points (black rectangles in the power transmission tower 110), and the second principal component 112 is the total displacement. The second principal component is a vector group that is visualized by dividing it into measurement points, and the third principal component 113 is a vector group that visualizes the third principal component of the total displacement by dividing it into measurement points. 114 is a vector group in which the fourth principal component of the total displacement is divided and visualized for each measurement point, and 115 is a vector group in which the fifth principal component of the total displacement is divided and visualized for each measurement point. be. However, in the figure, the measurement point without the vector display indicates that the size of the extracted vector is 0.

As can be seen from the figure, these five main components show the characteristic displacements of the transmission tower 110, which is the object.

Further, the wavelength band of the light imaged by the image pickup apparatus 101 may be ultraviolet rays, near infrared rays, or far infrared rays in addition to visible light.

Further, the main component of the amount of change calculated by the calculation unit 250 may be visualized. For example, the displacement detection device 200 further includes a superimposed image generation unit that generates a superimposed image in which an image based on the displacement calculated by the calculation unit 250 is superimposed on at least one of a plurality of captured images. May be.

As shown in FIGS. 9A to 9E, for example, the superimposed image generation unit divides each of the principal components into measurement points and displays a superimposed image in which a vector is superimposed and displayed on the captured image captured by the bridge 102. Generate. By displaying in this way, it is possible to easily grasp the spatial distribution of the total displacement of the object 102. In the case of three dimensions, a similar display can be realized by using a 3D display. If only the extraction of the characteristic displacement is required, the processing from the calculation unit 250 may be omitted.

Further, the time change of the component intensity (time change of ai (t)) may be displayed for each main component. In this case, for example, the calculation unit 250 calculates the temporal intensity change of the main component for each main component to be calculated. Further, this may be frequency-analyzed and displayed. In this case, for example, the calculation unit 250 calculates the frequency characteristic of the temporal intensity change of the main component for each main component to be calculated. This makes it possible to grasp the time characteristics of the behavior of the entire object for each component of the amount of change. If only such a display is required, the synthesis of the amount of change after step 350 may be omitted.

Further, the position and speed of the load source (for example, the vehicle 402 in FIG. 6A) may be detected by using the captured image, and the type of the load source may be recognized as an image. By specifying the type of load source, it can be associated with the load of the load source. This makes it possible to analyze the relationship between the magnitude of the load, the position, the velocity, and the displacement only from the image.

Further, in the embodiment, the displacement is detected from the captured image to obtain the total change amount of the object, but this is replaced with the velocity (differential amount of displacement) and acceleration (second-order differential amount of displacement) in the same manner. The procedure may be applied. When replaced with velocity, the spatial distribution of velocity of the entire object can be obtained, and a highly accurate velocity distribution can be obtained. The same applies to acceleration. The velocity image obtained by calculating the velocity of such an object may be obtained by differentiating the displacement obtained from the normal image, but can be directly acquired by using a laser Doppler meter. Further, the acceleration image obtained by calculating the acceleration of the object may be obtained by differentiating the displacement obtained from the normal image. For example, an acceleration sensor is installed at a measurement point on the object and the measured value of each acceleration sensor is obtained. Can be obtained directly from. When using an acceleration image, the configuration of the displacement detection device 200 is applied.

Further, the calculation unit 250 may spatially interpolate the calculated change amount of the entire object and estimate the change amount of points other than the measurement points.

Further, the correction unit 270 may correct the amount of change detected by the captured image or the detection unit 230 so that the actual scales of the structures included in the captured image are equal. When correcting the captured image, the correction unit 270 performs the correction process before the detection unit 230 detects the amount of change.

Since the above-described embodiment is for exemplifying the technique in the present disclosure, various changes, replacements, additions, omissions, etc. can be made within the scope of claims or the equivalent thereof.

The present disclosure can be used for measurement, measurement, analysis, diagnosis, inspection, etc. of the dynamic behavior of a structure.

1 Displacement detection system 101 Imaging device (camera)
102 Object (bridge)
200, 201 Displacement detector 210 Input / output I / O
220, 221 CPU
230 Detection unit 240, 241 Extraction unit 250 Calculation unit 260 Memory 270 Correction unit

Claims (12)

Using a plurality of captured images of an object captured at a plurality of times, a detection unit that detects spatial displacement with the passage of time at each of a plurality of measurement points set for the object, and a detection unit.
From the displacement detected by the detection unit, an extraction unit that extracts the characteristic displacement in which the vector indicates the characteristic displacement at the displacements at the plurality of measurement points, and the extraction unit.
A calculation unit that calculates the total displacement indicating the displacement of the entire object from the characteristic displacement extracted by the extraction unit.
A displacement detection device including a superimposed image generation unit that generates a superimposed image in which a vector image based on the displacement calculated by the extraction unit is superimposed on at least one of the plurality of captured images. Further, the displacement of the plurality of measurement points detected by the detection unit is based on the displacement of at least one reference measurement point among the plurality of measurement points detected by the detection unit. Equipped with a correction unit to correct
The displacement detection device according to claim 1, wherein the extraction unit uses the displacement corrected by the correction unit to perform the extraction. The detection unit detects the displacement in the plurality of captured images as the displacement, and determines the displacement.
Further, the calculation unit is provided with a scaling unit that scale-corrects the displacement of each of the plurality of measurement points detected by the detection unit so as to reflect the ratio of the distances actually displaced at the plurality of measurement points. The displacement detection device according to claim 1 or 2, wherein the calculation is performed by using the displacement scaled by the scaling unit. Further, the global movement of the entire captured image is detected from the spatial displacement detected by the detection unit, and the detection unit is based on the displacement from the plurality of measurement points to the plurality of reference measurement points following the movement. A correction unit for correcting the displacement of the plurality of measurement points detected by the above-mentioned is provided.
The displacement detection device according to claim 1, wherein the extraction unit uses the displacement corrected by the correction unit to perform the extraction. The displacement detection device according to claim 4, wherein the correction unit optimizes translation correction and rotation correction amount of the plurality of measurement points. The displacement detection device according to any one of claims 1 to 5, wherein the detection unit detects the spatial displacement by block matching. The displacement detection device according to claim 6, wherein the detection unit detects the plurality of captured images in pixel units or sub-pixel units. Each of the plurality of captured images is a speed image indicating the speed, and is a speed image.
The detection unit performs the detection so that the displacement to be detected is expressed by using the velocity.
The displacement detection device according to any one of claims 1 to 7, wherein the extraction unit performs the extraction so that the characteristic displacement to be extracted is expressed by using a velocity. Each of the plurality of captured images is a distance image indicating a distance, and is a distance image.
The displacement detection device according to any one of claims 1 to 7, wherein the detection unit detects a displacement in a three-dimensional space as the displacement. The extraction unit performs the extraction by performing principal component analysis on the displacement detected by the detection unit and extracting a plurality of principal components.
The displacement detection device according to any one of claims 1 to 9, wherein the superimposed image generation unit generates a plurality of superimposed images corresponding to each of the plurality of principal components. The displacement detection device according to any one of claims 1 to 10, wherein the plurality of captured images include images in which the object is synchronously imaged by a plurality of imaging devices. It is a displacement calculation method performed by a displacement detection device including a detection unit, an extraction unit, and a calculation unit.
A detection step in which the detection unit detects spatial displacement with the passage of time at a plurality of measurement points set on the object by using a plurality of captured images in which the object is captured at a plurality of times. ,
An extraction step in which the extraction unit extracts a characteristic displacement in which the vector indicates a characteristic displacement at the displacements at the plurality of measurement points from the amount of change detected by the detection step.
A calculation step in which the calculation unit calculates an overall displacement indicating the displacement of the entire object from the characteristic displacement extracted by the extraction step.
A displacement detection method comprising a superimposed image generation step of generating a superimposed image in which a vector image based on the displacement calculated by the extraction unit is superimposed on at least one of the plurality of captured images.

Images