JP2020170907A - Monitoring system and monitoring method - Google Patents

Abstract

To make it possible to grasp the deformed state of a member constituting a steel bridge only on the basis of a moving image of the steel bridge.SOLUTION: A monitoring system according to the present invention extracts a plurality of feature points from each of a plurality of frame images included in a moving image of a steel bridge on which a vehicle or the like travels, calculates the vibration frequency of the feature point group on the basis of the displacement between the feature points included in the feature point group including a plurality of feature points, identifies the vibration frequency included in the frequency band unique to the vehicle, etc., from among the calculated vibration frequencies, identifies the feature point group of the identified vibration frequency, amplifies the displacement between the feature points included in the identified feature point group, changes each of the plurality of frame images by moving the feature points included in the specified feature point group on the basis of the amplified displacement, and outputs a moving image including a plurality of frame images after the change.SELECTED DRAWING: Figure 1

Description

The present invention relates to a monitoring system and a monitoring method.

Damage to a steel bridge is often caused by the live load caused by the running of a vehicle or the like. For example, a fatigue crack in a steel bridge causes the members constituting the steel bridge to be deformed by the running of a vehicle or the like to a specific location. It is generated by the occurrence of stress concentration. Therefore, it is important to identify the deformed state of the member in formulating a repair and reinforcement plan. In addition, since the running of vehicles and fatigue cracks are closely related, it is also important to specify the position of external force acting on the members that make up the steel bridge when formulating a repair and reinforcement plan.

In order to specify the deformation of the members constituting the steel bridge, for example, the deformation state of the members is estimated based on the measurement results of the strain response and / or the displacement response generated by the running of a vehicle or the like, or a finite element. It is necessary to estimate the deformation state of the bridge members by the method.

For example, in Patent Document 1, a strain gauge is attached to a structure or a building to be measured for prevention and maintenance of a large-scale structure such as a pier or a building, and the strain generated in the structure or the building is attached. A method of measuring the response with a strain gauge is disclosed.

Further, Patent Document 2 discloses a method of creating a finite element (FEM) model of a bridge and analytically estimating the deformation state of a member of the bridge using known finite element analysis software. There is.

Japanese Unexamined Patent Publication No. 2005-291999 JP-A-2007-270552

However, in the conventional estimation method based on the measurement results such as strain response, the coating film at the corresponding portion of the bridge member must be peeled off in order to install the strain gauge, and a contact type displacement gauge or the like is used. Since displacement measurement is often performed, the time and cost required for the measurement are long, and the measurement itself may be difficult.

In addition, the conventional estimation method using the finite element method requires a large amount of time to create an FEM model, which causes a problem that the estimation itself takes time, and a hypothetical structure different from the actual structure of the target bridge is used. Since the analysis is performed on the premise, the analysis result may differ from the actual phenomenon.

Furthermore, in order to identify the traveling position of a vehicle or the like, it is necessary to photograph the road surface, but it is not possible to install a camera on a highway or the like where it is difficult to secure a shooting location such as a sidewalk. Occasionally occurred.

The present invention has been made to solve such a conventional problem, and is capable of grasping the deformed state of the members constituting the steel bridge based only on the moving image of the steel bridge. The purpose is to provide a system and monitoring method.

The monitoring system according to the present invention has an acquisition unit that acquires a moving image of a steel bridge on which a vehicle or the like is traveling from a photographing device, and an extraction that extracts a plurality of feature points from each of a plurality of frame images included in the moving image. In the unit and two frame images of the plurality of frame images, one feature point included in another frame, which is considered to be the same as one feature point included in the predetermined frame image, is determined. , A creation unit that creates a feature point group containing a plurality of feature points that are considered to be the same, and each feature point group based on the displacement between the feature points included in each feature point group. A calculation unit that calculates the vibration frequency of the group, and a specific unit that specifies the vibration frequency included in the frequency band unique to the vehicle, etc., and specifies the feature point group of the specified vibration frequency among the calculated vibration frequencies. By amplifying the displacement between the feature points included in the specified feature point group and moving the feature points included in the specified feature point group based on the amplified displacement, each of the plurality of frame images. It has an image processing unit that changes the number of frames, and an output unit that outputs a moving image including a plurality of frame images after the change.

Further, in the monitoring system according to the present invention, a storage unit that stores information on the deformation mode of the steel bridge member due to the running of a vehicle or the like and information on the repair and reinforcement method of the member corresponding to the deformation mode, and after the change. The output unit further has a mode determination unit for determining the deformation mode of the steel bridge member based on the frame image of the above, and the output unit extracts information on the repair and reinforcement method corresponding to the determined deformation mode from the storage unit. , It is preferable to output information on the extracted repair and reinforcement method.

Further, in the monitoring system according to the present invention, feature points included in the feature point group are extracted for each of the plurality of frame images, and among the extracted feature points, two features whose distances from each other are within the first distance. Create a short-distance group including points, extract multiple short-distance groups that have at least one feature point in common, create a new short-distance group that integrates the extracted multiple short-distance groups, and create multiple short-distance groups. A position where a new short-distance group is repeatedly created until the distance group is no longer extracted, and the feature point having the largest corresponding displacement among the feature points included in the short-distance group is specified as the wheel position of the vehicle or the like. It is preferable that the output unit has a specific unit and outputs a moving image in which the wheel position of a vehicle or the like is specified by superimposing a locus image indicating the specified wheel position on the corresponding frame image.

Further, in the monitoring system according to the present invention, the acquisition unit acquires the damage occurrence position of the steel bridge member included in the moving image input by the user, and the position identification unit is the feature point included in the short-distance group. Of these, it is preferable to specify the feature point having the largest corresponding displacement and within the second distance from the damage occurrence position as the wheel position of the vehicle or the like that has influenced the damage occurrence.

Further, in the monitoring system according to the present invention, the acquisition unit acquires the strain response data output from the strain measuring device installed near the damage occurrence position together with the measurement time, and the output unit determines the shooting time of the moving image. It is preferable to output the strain response data synchronized with the moving image based on the measurement time.

The monitoring method according to the present invention is a monitoring method executed by a monitoring system, in which a moving image of a steel bridge on which a vehicle or the like is traveling is acquired from a photographing device and is taken from each of a plurality of frame images included in the moving image. A plurality of feature points are extracted, and in two frame images of the plurality of frame images, one feature point included in another frame, which is considered to be the same as one feature point included in a predetermined frame image. The feature points are determined, a feature point group including a plurality of feature points considered to be the same is created, and each feature point group is based on the displacement between the feature points included in each feature point group. The vibration frequency of the feature point group was calculated, and among the calculated vibration frequencies, the vibration frequency included in the frequency band peculiar to the vehicle or the like was specified, and the feature point group of the specified vibration frequency was specified and specified. By amplifying the displacement between the feature points included in the feature point group and moving the feature points included in the specified feature point group based on the amplified displacement, each of the plurality of frame images is changed. It includes outputting a moving image including a plurality of frame images after the change.

According to the monitoring system and the monitoring method according to the present invention, it is possible to grasp the deformed state of the members constituting the steel bridge based only on the moving image of the steel bridge.

It is a schematic diagram for demonstrating the outline of a monitoring system. It is a schematic diagram for demonstrating the outline of a monitoring system. It is a figure which shows an example of the schematic structure of the monitoring system 1. It is a figure which shows an example of the schematic structure of the information processing apparatus 3. (A) is a diagram showing an example of a frame image included in a moving image, and (b) is a diagram showing an example of a data structure of the extraction feature point table T1. (A) is a diagram showing an example of the data structure of the corresponding feature point table T2, and (b) is a diagram showing an example of the data structure of the feature point group table T3. (A) is a diagram showing an example of the data structure of the vibration frequency band table T4, and (b) is a diagram showing an example of the data structure of the amplification feature point table T5. (A) is a diagram showing an example of the data structure of the wheel position table T6, and (b) is a diagram showing an example of a frame image on which a locus image is superimposed. (A) is a schematic view for explaining an example of the frame image after the change in which the in-plane deformation of the main girder web is photographed, and (b) is the change after the in-plane deformation of the floor plate is photographed. It is a figure which shows an example of a frame image. It is a figure which shows an example of the data structure of the repair reinforcement method table T7. It is a figure which shows an example of the operation flow of the monitoring method by the monitoring system 1.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

(Overview of monitoring system)
1 and 2 are schematic views for explaining the outline of the monitoring system of the present embodiment.

The monitoring system includes a photographing device for photographing a part or all of the steel bridge and an information processing device for processing moving image information photographed by the photographing device. For example, the photographing device is installed on the ground under the road bridge or the like to be photographed so that the back surface of the floor slab under the road surface of the steel bridge on which the vehicle or the like travels can be photographed. When the object to be photographed is a box girder bridge, the photographing device may be installed inside the box girder of the box girder bridge. The photographing device may photograph any member of the steel bridge.

The photographing device photographs the space to be photographed in a predetermined shooting direction at predetermined time intervals. The predetermined time interval is 1/30 second, and the photographing device outputs moving image information obtained by photographing the imaging target space in the predetermined imaging direction every 1/30 second. The predetermined time interval may be 1 second, 1/5 second, 1/50 second, or the like. The information processing device is, for example, a computer such as a personal computer (PC) or a server device. The information processing device may be any information processing device as long as it has a function of acquiring moving image information from a photographing device and executing various types of information processing.

The information processing device acquires moving image information captured by the photographing device from the photographing device via a storage medium. When the photographing device and the information processing device have a wired or wireless communication function, the information processing device may acquire moving image information from the photographing device by using the communication function. Hereinafter, the moving image information may be simply referred to as a moving image.

FIG. 1 (1) is a diagram showing an example of a frame image included in a moving image. The frame image is a still image taken at a predetermined time interval. For example, when the predetermined time interval of the moving image is 1/30 second, the moving image includes 30 frame images per second. In the example of the frame image shown in FIG. 1 (1), the main girder web, the U rib, the deck plate, the horizontal rib, etc. arranged on the back surface of the steel deck plate are photographed.

When the information processing device acquires a moving image from the photographing device, the information processing device extracts a plurality of feature points from each of the plurality of frame images included in the moving image. For example, the information processing device extracts feature points from the first frame image included in the moving image by using a predetermined feature point extraction method, and thereafter, according to the order in which the images are taken, a predetermined feature point extraction method is used. The feature points are sequentially extracted using. When the information processing apparatus extracts the feature points from the last frame image included in the moving image by using a predetermined feature point extraction method, the information processing apparatus ends the feature point extraction process. The predetermined feature point extraction method is, for example, a SIFT (Scale-Invariant Feature Transform) method, a SURF (Speeded Up Robust Features) method, or the like.

When the feature point extraction process is completed, the information processing device identifies two frame images out of the plurality of frame images. The two frame images are two consecutive frame images. The two consecutive frame images are, for example, the first (first) frame image and the second frame image, or the second frame image and the third frame image. When the number of frame images is 1000 and all the frame images are the processing targets, the number of sets of 2 frame images is 999.

The two frame images may be two frame images whose shooting time is within a predetermined time (for example, 1/10 second). For example, if the first frame image of the two frame images is the first (first) frame image, the second frame image of the two frame images is from the shooting time when the first frame image was taken. It may be a frame image taken after 1/10 second. The predetermined time may be any time as long as the vibration frequency described later can be calculated. Hereinafter, each of the two specified frame images may be referred to as a target frame image.

The information processing device determines the feature points extracted from the later target frame image that correspond to the feature points extracted from the earlier target frame image. The information processing device calculates the brightness of the feature points extracted from the previous target frame image and the brightness of the feature points extracted from the later target frame image, and extracts the feature points from the previous target frame image by using the Lucas-Kanade method or the like. The feature points extracted from the later target frame image, which can be regarded as the same as the feature points, are determined.

The information processing device creates a group including the corresponding two feature points. Hereinafter, the group may be referred to as a corresponding feature point group.

The information processing device refers to the created plurality of corresponding feature point groups, and integrates the plurality of groups including the common feature points into one group. Correspondence including feature point A of the first frame image and feature point B of the second frame image Correspondence feature point group X including feature point B of the second frame image and feature point C of the third frame image The case where the feature point group Y is created will be described as an example. Since both feature point groups X and Y include feature points B, a new feature point group is created by integrating both feature point groups X and Y. The new feature point group includes feature point A, feature point B and feature point C. When a new feature point group contains a feature point that is common to another corresponding feature point group or another new feature point group, a new feature point group is created by integrating the groups having the common feature point. Will be done. Hereinafter, the corresponding feature point group and the feature point group may be simply referred to as a group.

The information processing apparatus creates a feature point group including a plurality of feature points, each of which can be regarded as the same, by repeatedly executing the above-mentioned integrated process.

The information processing device calculates the vibration frequency (frequency per second) of each feature point group based on the displacement between the feature points included in each feature point group for each feature point group. FIG. 1 (2) shows a plurality of feature points A, B, C, and D included in the feature point group, and displacements between the feature points. In the example shown in FIG. 1 (2), the plurality of feature points A, B, C, and D are feature points extracted from different frame images and can be regarded as the same. , It is shown that the vehicle vibrates at a predetermined vibration frequency due to a vehicle traveling on the road surface or the like.

The information processing device calculates the vibration frequency of the feature point group based on the displacement between the feature points A and B, the displacement between the feature points B and C, and the displacement between the feature points C and D.

The frequency band of vibration generated due to a vehicle or the like traveling on a road surface is, for example, a frequency band of 0.4 Hz to 3.0 Hz and a known frequency band of 2.0 Hz to 20 Hz. The information processing device specifies the vibration frequency included in the frequency band peculiar to the vehicle or the like among the calculated vibration frequencies, and specifies the feature point group of the specified vibration frequency.

The information processing device amplifies the displacement between the feature points included in the identified identified feature point group. For example, in the example shown in FIG. 1 (3), when the vibration frequency is included in the frequency band of 0.4 Hz to 3.0 Hz, the displacement between the feature points included in the feature point group of the vibration frequency is amplified. For example, the information processing device moves the pixels corresponding to each feature point for each target frame image based on the displacement after amplification, and the pixels between the feature points for amplifying the displacement correspond to each feature point. Move according to the movement of the pixel to be processed.

FIG. 1 (4) is an example of the target frame image before the amplification process, and FIG. 1 (5) is an example of the target frame image after the amplification process. Since the vibration in the target frame image before the amplification process is minute as in the target frame image shown in FIG. 1 (4), the user (road manager or the like) cannot grasp the vibration in the image. On the other hand, in the target frame image shown in FIG. 1 (5), the displacement due to vibration is large, so that the user can grasp the vibration in the image.

The information processing device performs amplification processing on all or part of the target frame images, creates a moving image including the target frame image after the amplification processing, and displays and outputs the created moving image.

(1), (2) and (3) of FIG. 2 are examples in which the display and output screens of moving images are shown in chronological order. As shown in (1), (2), and (3) of FIG. 2, since the moving image displayed and output includes the target frame image changed by the amplification process, the user (road manager, etc.) can display and output the moving image. It is possible to grasp that the displacement portion is moving so as to follow the running of the vehicle or the like from the moving image.

As described in detail above, in the monitoring system, it is possible to grasp the deformed state of the members constituting the steel bridge based only on the moving image of the steel bridge. Further, since the deformed state of the members constituting the steel bridge can be grasped, the user (road manager or the like) can easily determine the repair and reinforcement method of the steel bridge.

The above-mentioned explanations of FIGS. 1 and 2 are merely explanations for deepening the understanding of the contents of the present invention. Specifically, the present invention may be carried out in each of the embodiments described below, and may be carried out by various modifications without substantially exceeding the principles of the present invention. All such variations are included within the scope of the present invention and the present specification.

(Monitoring system 1)
FIG. 3 is a diagram showing an example of a schematic configuration of the monitoring system 1. The monitoring system 1 includes a photographing device 2 and an information processing device 3.

The photographing device 2 includes an optical system having a predetermined angle of view, an image pickup device, and an image processing unit. The optical system is, for example, an optical lens, and a light flux from a subject is imaged on an image pickup surface of an image pickup device. A plurality of optical systems may be provided. For example, the photographing device 2 may be a stereo camera provided with two optical systems or a trinocular camera provided with three optical systems.

The image pickup device is a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), or the like, and outputs an image of a subject image formed on the image pickup surface. The image processing unit continuously creates and outputs a moving image in a file format from the images generated by the image sensor.

The photographing device 2 includes a photographing mounting portion (not shown) that holds the portable recording medium M detachably. When the portable recording medium M is an SD memory card, the photographing mounting unit includes an SD memory card slot. The photographing mounting unit has a function of recording a moving image on the mounted portable recording medium M. When the portable recording medium M is a DVD-ROM, the photographing mounting unit includes a DVD-ROM drive.

The photographing device 2 may include a photographing transmission unit (not shown) that transmits the output moving image to the information processing device 3 via a wireless or wired communication network at predetermined transmission time intervals. The predetermined transmission time interval is 10 seconds. The predetermined transmission time interval may be 5 seconds, 15 seconds, 30 seconds, or 1 minute.

The information processing device 3 is a personal computer (PC) or a server device. The information processing device 3 may be the information processing device 3 as long as the present invention can be applied. For example, a multifunctional mobile phone (so-called "smartphone"), a mobile phone (so-called "feature phone"), or a personal digital assistant (Personal). Digital Assistant, PDA), portable game machines, portable music players, tablet terminals, tablet PCs, notebook PCs, etc. may be used.

The information processing device 3 acquires a moving image from a portable recording medium M on which a moving image of a part (such as a floor slab under the road surface on which a vehicle or the like travels) or the entire back surface of the steel bridge is recorded. It executes various information processing on the moving image.

(Information processing device 3)
FIG. 4 is a diagram showing an example of a schematic configuration of the information processing device 3.

The information processing device 3 has a function of changing a plurality of frame images included in the moving image acquired from the photographing device 2. In order to realize this function, the information processing device 3 includes a storage unit 31, an operation unit 32, a display unit 33, a mounting unit 34, and a processing unit 35.

The storage unit 31 has, for example, at least one of a semiconductor memory, a magnetic disk device, and an optical disk device. The storage unit 31 stores a driver program, an operating system program, an application program, data, and the like used for processing by the processing unit 35. For example, the storage unit 31 stores, as a driver program, a display device driver program that controls the display unit 33, an information recording / reading device driver program that controls the mounting unit 34, and the like. The various programs may be installed in the storage unit 31 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like. The storage unit 31 stores various tables and the like, which will be described later, as data.

The operation unit 32 is a keyboard and a mouse. The operation unit 32 may be a touch panel or the like. The user can input characters, numbers, symbols, etc. using the operation unit 32. When the operation unit 32 is operated by the user, the operation unit 32 generates a signal corresponding to the operation. Then, the generated signal is supplied to the processing unit 35 as a user's instruction.

The display unit 33 is a liquid crystal display. The display unit 33 may be an organic EL (Electro-Luminescence) display or the like. The display unit 33 displays an image corresponding to the image data supplied from the processing unit 35, an image corresponding to the image data, and the like.

The mounting unit 34 includes an input / output device that detachably holds the portable recording medium M. When the portable recording medium M is an SD memory card, the mounting unit 34 includes an SD memory card slot. The mounting unit 34 has a function of reading various information stored in the mounted portable recording medium M and recording various information on the mounted portable recording medium M. When the portable recording medium M is a DVD-ROM, the mounting unit 34 includes a DVD-ROM drive.

The information processing device 3 may include a communication unit (not shown) in place of or together with the mounting unit 34. When the photographing device 2 includes a photographing transmitting unit, the communication unit of the information processing device 3 receives the moving image transmitted from the photographing transmitting unit via a wireless or wired communication network.

The processing unit 35 includes one or more processors and peripheral circuits thereof. The processing unit 35 is a CPU (Central Processing Unit) that comprehensively controls the overall operation of the information processing device 3. The processing unit 35 has the second communication unit 51 and the display unit 33 so that various processes of the information processing device 3 are executed in an appropriate procedure based on the program stored in the storage unit 31 and the operation of the operation unit 32. Etc. are controlled. The processing unit 35 executes processing based on a program (operating system program, driver program, application program, etc.) stored in the storage unit 31. Further, the processing unit 35 can execute a plurality of programs (application programs and the like) in parallel.

The processing unit 35 includes an acquisition unit 351, an extraction unit 352, a creation unit 353, a calculation unit 354, a specific unit 355, an image processing unit 356, a determination unit 357, and an output processing unit 358. Each of these units included in the processing unit 35 is a functional module realized by a program executed by the processor included in the processing unit 35. Alternatively, each of these units included in the processing unit 35 may be mounted on the information processing device 3 as firmware.

Hereinafter, with reference to FIGS. 5 to 10, the acquisition unit 351, the extraction unit 352, the creation unit 353, the calculation unit 354, the specific unit 355, the image processing unit 356, the determination unit 357, and the output processing unit 358 will be described.

When a portable recording medium M for recording a moving image obtained by photographing a part (such as the back surface of the floor slab under the road surface on which a vehicle or the like travels) or the whole of the steel bridge is mounted on the mounting section 34. , Acquires a moving image from the portable recording medium M according to the acquisition instruction input in response to the operation of the operation unit 32 by the user. The acquisition unit 351 passes the moving image acquired from the portable recording medium M to the extraction unit 352.

When the moving image is acquired from the acquisition unit 351, the extraction unit 352 extracts a plurality of feature points from each of the plurality of frame images included in the moving image.

The feature points are extracted from the image information using a predetermined feature point extraction method. The predetermined feature point extraction method is the SIFT method, the SURF method, or the like. Further, the feature point extraction method may be a corner detection method. In the corner detection method, corners and isolated points are detected as feature points based on the amount of change in the pixel value when the pixel is moved.

Hereinafter, the feature point extraction method by the SIFT method will be described. First, the extraction unit 352 detects candidate feature points in the frame image by executing DoG (Difference-of-Gaussian) processing on the frame image. In the DoG process, the extraction unit 352 generates a plurality of DoG images based on the frame image, and identifies pixels in which extreme values are detected from the generated plurality of DoG images as candidates for feature points.

Next, the extraction unit 352 extracts the feature points by deleting the candidates satisfying the predetermined deletion conditions from the feature point candidates. The predetermined deletion condition is that the candidate feature points are the same as the points detected by the corner detection method or the like. Further, the predetermined deletion condition may be a condition that the candidate pole value of the feature point is equal to or less than a predetermined numerical value. Candidates for the remaining feature points deleted in this way are extracted as feature points. For the feature point extraction method by the SIFT method, refer to, for example, Japanese Patent Application Laid-Open No. 2015-213615.

FIG. 5A is a diagram showing an example of a frame image included in the moving image acquired by the acquisition unit 351. In the example shown in FIG. 5A, a plurality of feature points superimposed on the frame image are shown.

FIG. 5B is a diagram showing an example of the data structure of the extraction feature point table T1. The extraction feature point table T1 is a table for managing feature points extracted from a plurality of frame images included in the moving image acquired by the acquisition unit 351. The extraction unit 352 stores in the storage unit 31 an extraction feature point table T1 including information on the feature points extracted for each of the plurality of frame images after the feature point extraction process or during the extraction process.

In the extraction feature point table T1, the frame number, shooting time, feature point, etc. of each frame image are stored in association with each other for each of the plurality of frame images. The frame number is an example of identification information for uniquely identifying each frame image, and is, for example, an integer value that increases in the order in which the images are taken. The shooting time is the time when each frame image was shot. The feature points are the positions in the frame image of each feature point extracted from the frame image. For example, as feature points, a two-dimensional coordinate value represented by the number of pixels from the upper side of the frame image and the number of pixels from the left side of the frame image is stored.

The creation unit 353 reads the feature points associated with each of the two target frame images out of the plurality of frame images from the extraction feature point table T1. For example, the creation unit 353 specifies a predetermined frame image among the plurality of frame images as the target frame image ahead, and sets the frame image at the shooting time after the predetermined frame image as the target frame image after. Identify as. The predetermined frame image is, for example, the first frame image among a plurality of frame images. The predetermined frame image may be a frame image specified by the user. Hereinafter, the feature point associated with the earlier target frame image may be referred to as the earlier feature point, and the feature point associated with the later target frame image may be referred to as the later feature point.

The creation unit 353 determines the later feature points corresponding to the earlier feature points. For example, the creation unit 353 calculates the brightness of each of the first feature point and the second feature point, and can be regarded as the same as the previous feature point by using the Lucas-Kanade method or the like. Determine the feature points.

In the Lucas-Kanade method, the creation unit 353 creates an optical flow (motion vector) starting from the previous feature point and ending at the later feature point that can be regarded as being the same as the previous feature point. calculate. When the first feature point and the second feature point have the same brightness, the first feature point and the second feature point are considered to be the same. In this case, the above feature points and the optical flow can be expressed as in the equation (1).

In the formula (1), I is the brightness of the previous feature point in the first image information, x and y are the two-dimensional coordinates of the previous feature point, and v _x and v _y are optical flows.

Next, the creating unit 353 obtains the equation (2) by Taylor-expanding the equation (1).

Next, the creating unit 353 obtains the following optical flow constraint equation (3) based on the equation (2).

Next, the creation unit 353 sets an image area (virtual window) substantially centered on the above feature points, and satisfies the condition that all optical flows in the image area are constant. Calculate the optical flow starting from the feature point of. The creation unit 353 calculates the optical flow by the following equation (4).

Then, the creating unit 353 determines that the feature point corresponding to the end point of the optical flow when the previous feature point is the start point is the later feature point corresponding to the previous feature point. Hereinafter, the earlier feature point and the later feature point corresponding to the previous feature point (that is, the feature point corresponding to the start point and the feature point corresponding to the end point in the optical flow) may be referred to as the corresponding two feature points.

Regarding the calculation process of the optical flow by the Lucas-Kanade method, for example, B.I. D. Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision", in Proc Imaging Understancing Workshop, pp. See 121-130, 1981.

FIG. 6A is a diagram showing an example of the data structure of the corresponding feature point table T2. The corresponding feature point table T2 is a table for managing the corresponding 2 feature points determined by the creating unit 353. The creation unit 353 stores the corresponding 2 feature points determined for each set of the 2 frame images after or during the determination process of the corresponding 2 feature points.

In the corresponding feature point table T2, a frame set ID (identifier), a frame set, a corresponding feature point group, and the like are stored in association with each other for each set of the target frame images of 2. The frame set ID is an example of identification information for uniquely identifying a set of two target frame images. The frame set is the frame number of each of the two target frame images. The corresponding feature point group is the position in the target frame image of each of the two corresponding feature points.

In the example of the corresponding feature point table T2 shown in FIG. 6A, the frame set ID “B000001” is stored in association with the frame numbers “000001” and “000002”. The frame number "000001" is a frame number indicating the first target frame image, and the frame number "000002" is a frame number indicating the later target frame image. Further, the positions "(132,528), (133,528)", "(142,511), (143,510) of the corresponding two feature points indicating the corresponding feature point group in the target frame image". , ... Are stored in association with the frame set ID “B000001”. The position “(132,528)” in the frame image is the position of the feature point in the frame image after the frame number “000001”. The position "(133,528)" in the frame image is the position of the later feature point in the frame image after the frame number "000002".

The corresponding feature point group may be any information as long as it can identify each of the two feature points included in the corresponding feature point group.

After that, the creation unit 353, the calculation unit 354, the specific unit 355, and the image processing unit 356 execute the exaggerated expression processing of the moving image using the known Eulerian Motion Magnification. The outline of Eulerian Motion Magnification will be described below. For details on Eulerian Motion Magnification, see, for example, "Eulerian Video Magnification for Revealing Subtle Changes in the World", HY Wu et al., ACM Transactions on Graphics (TOG), SIGGRAPH 2012, Conference Proceedings, Volume 31 Issue 4, Article No. Please refer to 65, July 2012, etc.

The brightness corresponding to a specific place x in the frame image at a specific time t can be indicated as I (x, t). Hereinafter, a case where I (x, t) can be expressed as the following equation (5) will be described.

δ (t) is a displacement function.

Assuming that the frame image can be represented by a first-order Taylor series, I (x, t) can be represented by the following equation (6).

Next, when a bandpass filter is applied to I (x, t), f (x) is removed and B (x, t) shown in the following equation (7) is generated.

B (x, t) is amplified by a predetermined amplification factor α, and the amplified B (x, t) is added to the original I (x, t) as shown in the following equation (8). As a result, an amplified signal is generated.

Then, one or a plurality of the amplified signals are combined with the moving image to complete the exaggerated expression processing.

Hereinafter, an example of the exaggerated expression processing of the moving image executed by the creation unit 353, the calculation unit 354, the specific unit 355, and the image processing unit 356 will be described.

The creation unit 353 reads out the corresponding feature point group associated with the frame set ID stored in the corresponding feature point table T2, and among the read corresponding feature point groups, a plurality of corresponding feature point groups including common feature points are selected. Integrate to create a new feature point group.

In the example of the corresponding feature point table T2 shown in FIG. 6A, the corresponding feature point groups “(132,528), (133,528)” associated with the frame set ID “B000001” and the frame set ID “B000002” The corresponding feature point groups "(133,528), (133,529)" associated with "" have the same feature point position "(133,528)" in the frame number "000002". The creation unit 353 integrates the corresponding feature point groups "(132,528), (133,528)" and the corresponding feature point groups "(133,528), (133,529)". , 528), (133,528), (133,529) ”.

After that, the creation unit 353 includes a case where the corresponding feature point group of 2 includes a common feature point, a case where the new feature point group and the corresponding feature point group include a common feature point, and a case where the feature point group of 2 includes a common feature point. When a common feature point is included, a new feature point group is created by integrating the groups having the common feature point. The creation unit 353 repeatedly executes the group integration process until there are no groups having common feature points.

When the group integration process by the creating unit 353 is completed, the calculation unit 354 calculates the vibration frequency of each feature point group for each feature point group based on the displacement between the feature points included in each feature point group. For example, the calculation unit 354 calculates the directions of all or part of the displacements between the feature points, and calculates the average direction of the calculated directions. Next, the calculation unit 354 sets the average direction as the X-axis direction and the orthogonal direction of the average direction as the Y-axis direction. Next, the calculation unit 354 decomposes the displacement vector (motion vector) of all or part of the displacement between the feature points into the set components in the X-axis direction and the components in the Y-axis direction. Then, the calculation unit 354 sets the number of times per second that the direction of the component of the motion vector in the X-axis direction is in the minus direction as the vibration frequency.

FIG. 6B is a diagram showing an example of the data structure of the feature point group table T3. The feature point group table T3 is a table for managing the feature point group created by the creation unit 353 and the vibration frequency calculated by the calculation unit 354. The creation unit 353 stores the created feature point group after or during the creation process of the feature point group, and stores the vibration frequency corresponding to the stored feature point group calculated by the calculation unit 354. ..

In the feature point group table T3, the feature point group ID, the feature points in the group, the vibration frequency, and the like are stored in association with each other for each feature point group. The feature point group I is an example of identification information for uniquely identifying each feature point group. The feature point in the group is a combination of the position of the feature point included in the feature point group created by the creation unit 353 and the frame number of the frame image from which the feature point is extracted. The vibration frequency is a vibration frequency calculated by the calculation unit 354.

In the example of the feature point group table T3 shown in FIG. 6B, the position in the frame image of the frame number “000001” as the feature point in the group included in the feature point group indicated by the feature point group ID “G0001” ( 132,528) feature points, position (133,529) feature points in the frame image of frame number "000001", and position "(133,530) feature points" in the frame image of frame number "000001". ... Etc. are stored. Further, in the feature point group table T3, "2.2 Hz" is stored as the vibration frequency of the feature point group indicated by the feature point group ID "G0001".

FIG. 7A is a diagram showing an example of the data structure of the vibration frequency band table T4. The vibration frequency band table T4 is a table for managing a frequency band peculiar to a vehicle or the like. In the vibration frequency band table T4, a plurality of frequency bands are stored, and the amplification factor (1 + α) for each frequency band is stored in association with each other. In the example of the vibration frequency band table T4 shown in FIG. 7A, the frequency band of 0.4 Hz to 3.0 Hz is a frequency band peculiar to a vehicle or the like, and a characteristic point group of vibration frequencies included in this frequency band. The displacement between the feature points included in is amplified.

The specific unit 355 reads the frequency band “0.4 Hz to 3.0 Hz” from the vibration frequency band table T4, and among the plurality of vibration frequencies stored in the feature point group table T3, the read frequency band “0. The vibration frequency included in "4Hz to 3.0Hz" is specified. Next, the specifying unit 355 identifies the feature point group ID and the feature point in the group associated with the specified vibration frequency, and passes the specified feature point group ID and the feature point in the group to the image processing unit 356.

The image processing unit 356 amplifies the displacement between the feature points based on the feature points in the group received from the specific unit 355 for each of the plurality of frame images, and the specified feature points based on the amplified displacement. Move the feature points included in the group. The image processing unit 356 moves other pixels for each of the plurality of frame images so as to match the feature points after the movement.

The image processing unit 356 generates a moving image including the plurality of changed frame images by synthesizing the plurality of changed frame images with the original moving image, and the output processing unit 358 generates a moving image including the plurality of changed frame images, as shown in FIG. , The generated moving image is displayed and output to the display unit 33.

The processing executed by the creation unit 353, the calculation unit 354, the specific unit 355, and the image processing unit 356 is not limited to the above-mentioned processing, but is a processing that realizes an exaggerated expression processing of a moving image using Eulerian Motion Magnification. If so, any process may be used.

FIG. 7B is a diagram showing an example of the data structure of the amplification feature point table T5. The amplified feature point table T5 is a table for managing the feature points after amplification created by the image processing unit 356. The image processing unit 356 stores the feature points after amplification after the amplification processing or during the amplification processing.

In the amplified feature point table T5, the feature point group ID, the amplified feature point, the vibration frequency, and the like are stored in association with each other for each feature point group. The amplified feature point is a combination of the position of the feature point after amplification included in the feature point group created by the image processing unit 356 and the frame number of the frame image from which the feature point is extracted.

The specific unit 355 includes a feature point group ID and a feature point in the group associated with the vibration frequency included in the frequency band "0.4 Hz to 3.0 Hz" among the plurality of vibration frequencies stored in the feature point group table T3. When is specified, a short-distance group that integrates adjacent feature point groups is created.

For example, the specific unit 355 calculates the average position or the position of the center of gravity of the specified feature points in the group, and associates the calculated position with the specified feature point group ID as the feature point group position. Next, the specific unit 355 integrates the feature point groups having the feature point group positions within the first distance to create a short distance group.

Then, the specifying unit 355 specifies the feature point having the largest corresponding displacement in each of the plurality of frame images among the feature points included in the short-distance group as the wheel position of the vehicle or the like corresponding to the short-distance group. ..

FIG. 8A is a diagram showing an example of the data structure of the wheel position table T6. The wheel position table T6 is a table for managing the wheel positions of the vehicle or the like specified by the specific unit 355. The specifying unit 355 stores the wheel position of the specified vehicle or the like after the wheel position specifying process or during the wheel position specifying process.

In the wheel position table T6, the frame number, the wheel position information, the shooting time, and the like of each frame image are stored in association with each other for each of the plurality of frame images. The wheel position information is information indicating the wheel positions of the wheels and the like corresponding to each short-distance group, which are specified in each of the plurality of frame images. For example, as the ring position, a two-dimensional coordinate value represented by the number of pixels from the upper side of the frame image and the number of pixels from the left side of the frame image is stored.

The image processing unit 356 creates a moving image including a plurality of frame images after the change or a moving image in which a locus image indicating a specified ring position is superimposed on the original moving image, and the output processing unit 358 displays FIG. As in b), the generated moving image is displayed and output to the display unit 33. FIG. 8B is a diagram showing an example of a frame image on which a locus image is superimposed. The locus image shown in FIG. 8B is a line segment image in which a plurality of ring positions are connected.

The determination unit 357 determines the deformation mode of various members of the steel bridge in the lower part of the road surface on which the vehicle or the like travels, based on the frame image after the change. The determination unit 357 is based on a known image matching method using, for example, a plurality of sample images showing the in-plane deformation of the main girder web and a plurality of sample images showing the out-of-plane deformation of the deck plate. It is determined whether the frame image is in-plane deformation of the main girder web or out-of-plane deformation of the deck plate.

FIG. 9A is a schematic diagram for explaining an example of a frame image after the change in which the in-plane deformation of the main girder web is photographed. Since the example of the frame image shown in FIG. 9A is a frame image after the change, the degree of in-plane deformation of the main girder web is displayed larger than that of the original frame image.

FIG. 9B is a diagram showing an example of a frame image after the change in which the out-of-plane deformation of the deck plate is photographed. Since the example of the frame image shown in FIG. 9B is the frame image after the change, the degree of out-of-plane deformation of the deck plate is displayed larger than that of the original frame image.

In the original frame image (before change), the degree of in-plane deformation of the main girder web and out-of-plane deformation of the deck plate is extremely small, so deformation was often not determined even by using a known image matching method. .. Since the determination unit 357 of the information processing apparatus 3 applies an image matching method using the frame image after the change in which the displacement is amplified (the degree of deformation is large), it is possible to perform image matching with high determination accuracy. Become.

FIG. 10 is a diagram showing an example of the data structure of the repair and reinforcement method table T7. The repair and reinforcement method table T7 is a table for managing the deformation mode and the repair and reinforcement method corresponding to the deformation mode.

In the repair and reinforcement method table T7, a sample image, a repair and reinforcement method, and the like are stored in association with each other for each deformation mode (out-of-plane deformation mode or in-plane deformation mode for each member). The sample image is an image including each transformation mode among the frame images after the change created in the past. For example, the relative position of the photographing device from the floor slab and the inclination of the photographing device when the sample image is taken are abbreviated as the relative position of the photographing device from the floor slab and the inclination of the photographing device when the moving image is taken. It is the same.

The repair and reinforcement method is information on the repair and reinforcement method for dealing with the deformation-related damage shown by the sample image. For example, the repair and reinforcement method includes a backing plate reinforcement using L-shaped steel, a backing plate reinforcement using flat steel, and shape improvement repair. The information regarding the repair and reinforcement method includes character information indicating the name of the repair and reinforcement method, character information explaining the repair and reinforcement method, and / or image (moving image and / or still image) information indicating the repair and reinforcement method.

The image matching process executed by the determination unit 357 may be executed by the AI (Artificial Intelligence) program stored in the storage unit 31. For example, the determination unit 357 is a neural network learned based on the teacher data including the sample image (input data) stored in the repair reinforcement method table T7 and the deformation mode (output data) stored in the repair reinforcement method table T7. Using a model or the like, numerical values indicating the possibility of deformation modes of various members included in the deformed frame image may be calculated. Then, the determination unit 357 determines that the transformation mode of the maximum numerical value among the calculated numerical values is the transformation mode of the frame image after the transformation.

As the image matching process executed by the determination unit 357, even when the image matching method using the AI program is applied, the frame image after the change in which the displacement is amplified (the degree of deformation is large) is used, so that the determination is made. The unit 357 can perform image matching with high determination accuracy.

When the determination unit 357 determines the deformation mode of the changed frame image, the determination unit 357 refers to the repair reinforcement method table T7 and determines the repair reinforcement method corresponding to the deformation mode.

(Operation flow of monitoring method by monitoring system 1)
FIG. 11 is a diagram showing an example of an operation flow of the monitoring method by the monitoring system 1.

First, in the acquisition unit 351, a portable recording medium M for recording a moving image obtained by photographing a part (the back surface of the floor slab under the road surface on which a vehicle or the like travels) or the whole of the steel bridge is attached to the attachment unit 34. If this is the case, a moving image is acquired from the portable recording medium M according to the acquisition instruction input in response to the operation of the operation unit 32 by the user (step S101).

Next, the extraction unit 352 extracts a plurality of feature points from each of the plurality of frame images included in the moving image acquired by the acquisition unit 351 (step S102).

Next, the creation unit 353 creates a corresponding feature point group including the corresponding 2 feature points from the target frame image of 2 out of the plurality of frame images from the extraction feature point table T1 (step S103).

Next, the creation unit 353 creates a feature point group based on a plurality of corresponding feature point groups (step S104).

Next, the calculation unit 354 calculates the vibration frequency of each feature point group for each feature point group based on the displacement between the feature points included in each feature point group (step S105).

Next, the specific unit 355 reads out a frequency band peculiar to the vehicle or the like from the vibration frequency band table T4, and among the calculated vibration frequencies of each feature point group, the vibration frequency included in the frequency band peculiar to the vehicle or the like is selected. Identify. Next, the identification unit 355 identifies a feature point group associated with the specified vibration frequency (step S106).

Next, the image processing unit 356 amplifies the displacement between the feature points in the feature point group of the vibration frequency included in the frequency band peculiar to the vehicle or the like for each frame image (step S107).

Next, the image processing unit 356 moves the feature points included in the feature point group of the vibration frequency included in the frequency band peculiar to the vehicle or the like based on the amplified displacement for each frame image, and after the movement, Other pixels are also moved so as to match the feature points of. Then, the image processing unit 356 generates a moving image including the plurality of changed frame images by synthesizing the plurality of changed frame images with the original moving image (step S108).

Next, the determination unit 357 determines the deformation mode of the floor slab and the main girder web under the road surface on which the vehicle or the like travels, based on the changed frame image (step S109).

Then, the determination unit 357 determines the repair / reinforcement method corresponding to the deformation mode determined in step S109 with reference to the repair / reinforcement method table T7 (step S110).

As described in detail above, in the monitoring system 1 of the present embodiment, the steel bridge is configured based only on the moving image obtained by photographing a part or all of the steel bridge (the back surface of the floor slab under the road surface of the road bridge, etc.). It is possible to grasp the deformed state of the member.

(Modification example 1)
The present invention is not limited to the present embodiment. For example, in the process of specifying the wheel position by the specific unit 355, the damage occurrence position of the steel bridge member (for example, the back surface of the floor slab) input by the user may be used.

For example, the acquisition unit 351 acquires the damage occurrence position on the back surface of the floor slab included in the moving image, which is input by the user. For example, the acquisition unit 351 acquires the screen position input in response to the operation of the operation unit 32 by the user as the damage occurrence position when the moving image is displayed on the display unit 33. In each of the plurality of frame images, the specific unit 355 damages the feature points having the largest corresponding displacement among the feature points included in the specified short-distance group and within the second distance from the acquired damage occurrence position. It is specified as the wheel position of the vehicle etc. that affected.

In this case, the image processing unit 356 creates a moving image including a plurality of frame images after the change or the original moving image, and the locus image indicating the specified ring position is superimposed, and the output processing unit 358 creates a moving image. The generated moving image is displayed and output to the display unit 33.

In this way, the monitoring system 1 presents the user with a wheel position that is presumed to affect the already-occurred damage occurrence position, based only on the moving image of the back surface of the floor slab under the road surface of the road bridge. Make it possible.

(Modification 2)
The acquisition unit 351 acquires the strain response data output from the strain measuring device installed near the damage occurrence position together with the measurement time, and the output processing unit 358 acquires the shooting time and acquisition of each frame image included in the moving image. Based on the measured measurement time, the strain response data synchronized with the moving image including a plurality of changed frame images or the original moving image may be displayed and output to the display unit 33.

In this way, the monitoring system 1 makes it possible to present the strain response data regarding the damage occurrence position to the user together with the moving image including the plurality of frame images after the change or the original moving image.

It will be appreciated by those skilled in the art that various changes, substitutions, and modifications can be made to this without departing from the spirit and scope of the invention.

1 Monitoring system 2 Imaging device 3 Information processing device 31 Storage unit 32 Operation unit 33 Display unit 34 Mounting unit 35 Processing unit 351 Acquisition unit 352 Extraction unit 353 Creation unit 354 Calculation unit 355 Specific unit 356 Image processing unit 357 Judgment unit 358 Output processing Department

Claims (6)

An acquisition unit that acquires a moving image of a steel bridge on which a vehicle or the like is traveling from a photographing device,
An extraction unit that extracts a plurality of feature points from each of the plurality of frame images included in the moving image,
A feature including a plurality of feature points that are considered to be the same as one feature point included in a predetermined frame image is determined, and one feature point included in another frame is determined to be the same. A creation unit that creates a point group, and
For each feature point group, a calculation unit that calculates the vibration frequency of each feature point group based on the displacement between the feature points included in each feature point group.
Among the calculated vibration frequencies, a specific unit that specifies the vibration frequency included in the frequency band peculiar to the vehicle or the like and specifies the feature point group of the specified vibration frequency, and
By amplifying the displacement between the feature points included in the specified feature point group and moving the feature points included in the specified feature point group based on the amplified displacement, the plurality of said. The image processing unit that changes each of the frame images of
An output unit that outputs the moving image including the plurality of frame images after the change,
Surveillance system with. A storage unit that stores information on the deformation mode of the member of the steel bridge due to the traveling of the vehicle or the like and information on the repair and reinforcement method of the member corresponding to the deformation mode.
Further, it has a mode determination unit for determining the deformation mode of the member of the steel bridge based on the frame image after the change.
The monitoring system according to claim 1, wherein the output unit extracts information on the repair and reinforcement method corresponding to the determined deformation mode from the storage unit and outputs the extracted information on the repair and reinforcement method. For each of the plurality of frame images
The position of each feature point group is calculated, and the feature point groups whose distances are within the first distance are integrated to create a short-distance group.
Among the feature points included in the short-distance group, the feature point having the largest displacement corresponds to the feature point having a position specifying portion for designating the wheel position of the vehicle or the like.
The monitoring according to claim 2, wherein the output unit outputs a moving image in which the wheel position of a vehicle or the like is specified by superimposing a locus image indicating the specified wheel position on the corresponding frame image. system. The acquisition unit acquires the damage occurrence position of the steel bridge member included in the moving image input by the user.
The position specifying portion is a feature point having the largest displacement among the feature points included in the short-distance group, and the feature point within a second distance from the damage occurrence position affects the damage occurrence. The monitoring system according to claim 3, which specifies the wheel position of the vehicle or the like given the above. The acquisition unit acquires strain response data output from a strain measuring device installed near the damage occurrence position together with the measurement time.
The monitoring system according to claim 4, wherein the output unit outputs strain response data synchronized with the moving image based on the shooting time of the moving image and the measurement time. It is a monitoring method executed by the monitoring system.
Acquire a moving image of a steel bridge on which a vehicle or the like is traveling from the photographing device,
A plurality of feature points are extracted from each of the plurality of frame images included in the moving image, and a plurality of feature points are extracted.
A feature including a plurality of feature points that are considered to be the same as one feature point included in a predetermined frame image is determined, and one feature point included in another frame is determined to be the same. Create a point group and
For each feature point group, the vibration frequency of each feature point group is calculated based on the displacement between the feature points included in each feature point group.
Among the calculated vibration frequencies, the vibration frequency included in the frequency band peculiar to the vehicle or the like is specified, and the feature point group of the specified vibration frequency is specified.
By amplifying the displacement between the feature points included in the specified feature point group and moving the feature points included in the specified feature point group based on the amplified displacement, the plurality of said. Change each of the frame images of
Output the moving image including the plurality of frame images after the change.
Monitoring methods including that.

Images