KR102588436B1 - Method and system for real-time pedestrian dynamic-load localization using vision-based motion sensing - Google Patents

Abstract

A method and system for real-time dynamic walking load location recognition using image-based motion sensing according to an embodiment of the present invention includes the steps of (a) receiving image data, and (b) analyzing the image data to detect object information. (c) calculating image coordinates of an object load reference point from the object information, (d) calculating a scale factor that is the ratio of the length of a unit pixel of the image data to the corresponding actual length; And (e) calculating the actual coordinates of the object load reference point using the image coordinates and the scale factor, so that the position of the dynamic load can be conveniently and economically detected only through image analysis.

Description

Method and system for real-time dynamic walking load location recognition using video-based motion sensing {METHOD AND SYSTEM FOR REAL-TIME PEDESTRIAN DYNAMIC-LOAD LOCALIZATION USING VISION-BASED MOTION SENSING}

The present invention relates to a method and system for recognizing the position of a dynamic walking load in real time. More specifically, the location of a dynamic walking load in real time using image-based motion sensing that can recognize the position of a dynamic walking load in real time by analyzing image data. It relates to recognition methods and systems.

If social infrastructure such as buildings, bridges, etc. are more than 30 years old, there is a high possibility that problems due to deterioration will occur. Accordingly, the need for inspection and maintenance is increasing to minimize damage caused by the aging of structures. Previously, to evaluate or inspect the deterioration of a structure, experts used a method of directly checking and evaluating the condition of the structure. However, this method has limitations in that it not only involves subjective factors, but also requires a significant amount of manpower, time, and cost. There was.

Vibration-based Structure Health Monitoring (SHM) technology is mainly used to identify the deterioration of structures. This is a technology that measures the response by attaching a sensor such as an accelerometer to the structure and determines the state of the structure from this information. SHM technology is divided into an output-only method that utilizes only the response data of the structure and an input-output method that utilizes both input and output. The output-based method is Since the input is assumed to be white noise and only the dynamic behavior information is used as the output, the accuracy is somewhat low, but it is widely used due to the difficulty of measuring the input.

Korean Patent Publication No. 10-2017-0017659 (2017.02.15)

The present invention is intended to solve the above problems, and specifically, a method and system for real-time dynamic walking load location recognition using image-based motion sensing that can accurately and easily recognize the location of real-time dynamic walking load by analyzing image data. The purpose is to provide.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A real-time dynamic location recognition method of walking load using image-based motion sensing according to an embodiment of the present invention to solve the above problem includes the steps of: (a) receiving image data; (b) analyzing the image data to detect object information; (c) calculating image coordinates of the object load reference point from the object information; (d) calculating a scale factor that is a ratio of the length of a unit pixel of the image data and the corresponding actual length; and (e) calculating actual coordinates of the object load reference point using the image coordinates and the scale factor.

Step (b) may be performed using the CRST algorithm.

In step (d), the scale factor can be calculated using Equation 1 below.

<Equation 1>

Scale factor = Y/X

(In Equation 1, X is the pixel length of a specific object in the image data, and Y is the actual length of the specific object)

In step (e), the actual coordinates can be calculated by multiplying the image coordinates by the scale factor.

The step of correcting distortion of the received image data may be performed by the Zhang algorithm.

In addition, the real-time dynamic location recognition method of walking load using image-based motion sensing according to an embodiment of the present invention may further include analyzing the image data to calculate a predicted load value of the object load reference point.

In addition, the position recognition system for dynamic walking load using image-based motion sensing according to an embodiment of the present invention includes an image data receiver that receives image data; an object information detection unit that analyzes the image data to detect object information and calculates image coordinates of an object load reference point from the object information; a scale coefficient calculation unit that calculates a scale coefficient that is a ratio of the length of a unit pixel of the image data and the corresponding actual length; and

It may include an object position calculation unit that calculates actual coordinates of the object load reference point using the image coordinates and the scale factor.

The object information detector may detect the object information using the CRST algorithm.

The scale factor can be calculated using Equation 1 below.

<Equation 1>

Scale factor = Y/X

(In Equation 1, X is the pixel length of a specific object in the image data, and Y is the actual length of the specific object)

The actual coordinates can be calculated by multiplying the image coordinates by the scale factor.

In addition, the real-time dynamic walking load position recognition system using image-based motion sensing according to an embodiment of the present invention may further include an image data correction unit that corrects distortion of the received image data.

The image data correction unit may correct distortion of the image data using the Zhang algorithm.

In addition, the real-time dynamic walking load position recognition system using image-based motion sensing according to an embodiment of the present invention may further include a load prediction unit that analyzes the image data and calculates a load prediction value of the object load reference point.

The method and system for recognizing the position of a real-time dynamic walking load using image-based motion sensing according to an embodiment of the present invention allows the location of a real-time dynamic walking load to be easily recognized by simply analyzing the image data without a separate sensor.

The effects according to the present invention are not limited to the contents exemplified above, and further various effects are included in the present specification.

Figure 1 is a schematic configuration diagram of a real-time dynamic walking load position recognition system using image-based motion sensing according to an embodiment of the present invention.
Figure 2 is a schematic flowchart explaining a method for recognizing the position of a real-time dynamic walking load using image-based motion sensing of the system of Figure 1.
FIG. 3 is a diagram illustrating an example result of a performance evaluation experiment of an object information detection algorithm applicable to the object information detection unit of FIG. 1.
FIG. 4 is a diagram showing positioning results when original image data and data corrected by the image data correction unit of FIG. 1 are input.
FIG. 5 is a diagram showing location information of an object detected in real time by the object information detection unit of FIG. 1.
FIG. 6 is a diagram showing an example of image coordinates of an object load reference point calculated by the object information detection unit of FIG. 1.
FIG. 7 is a conceptual diagram illustrating a method of calculating the actual coordinates of the object load reference point calculated by the object position calculation unit of FIG. 1.
FIG. 8 is a graph showing the results of comparing the location of the object calculated by the object location calculation unit of FIG. 1 and the actual location.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to be understood by those skilled in the art. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Hereinafter, a method and system for real-time dynamic walking load location recognition using image-based motion sensing according to embodiments of the present invention will be described with reference to the drawings.

Figure 1 is a schematic configuration diagram of a real-time dynamic walking load position recognition system using image-based motion sensing according to an embodiment of the present invention.

Referring to Figure 1, the real-time dynamic walking load position recognition system 100 using image-based motion sensing according to an embodiment of the present invention includes an image data reception unit 110, an image data correction unit 120, and an object information detection unit ( 130), a scale factor calculation unit 140, an object position calculation unit 150, and a load prediction unit 160.

In the present invention, the system 100 may be hardware such as a computer including the configuration of FIG. 1. However, this is only an example, and it may be an algorithmic computer program that implements the same operation as the configuration in FIG. 1.

The image data receiving unit 110 serves to receive image data about the structure. At this time, the video data may be a device that can take photos or real-time video, such as a camera or smartphone.

The image data correction unit 120 serves to correct distortion of received image data. When an object such as a moving pedestrian is detected in video data, distortion may occur in the conversion between pixel length and actual length depending on the location of the object due to the perspective effect of the 3D image.

The image data correction unit 120 may perform camera calibration using, for example, the algorithm presented by Zhang (2000) to correct the above distortion.

FIG. 4 exemplarily shows location results when original image data and data corrected by the image data correction unit of FIG. 1 are input.

The results in Figure 4 show that when the image consists of 720 × 404 pixels, the root mean square error (RMSE) occurs about 5.4 pixels. In addition, because the distortion of the image occurs relatively large at the edges, it was confirmed that the error increases as the object moves to the left and right ends of the image.

To check the effect of image correction, we compared the cases where the original image and the corrected image were input.

The height of the pedestrian object to be tracked was already known to be 1,816 mm, and the number of pixels corresponding to the height of the pedestrian in the corrected image data was 163. The total number of horizontal pixels in the corrected image was 720, and the actual width of the floor was 8,150 mm.

The horizontal width of the floor calculated using the formula described later was approximately 8,022 mm, which resulted in an error of approximately 1.57% when compared to the actual measured value of 8,150 mm.

In the case of the image before correction, the number of pixels corresponding to the height of the pedestrian was 168, and the calculated floor width was 7,783 mm, showing an error of about 4.51%. This shows that the error reduction effect of about 2.94% was achieved through image correction. .

The object information detection unit 130 analyzes image data to detect object information and calculates image coordinates of the object load reference point from the detected object information.

The object information detection unit 130 uses the user-specified range as the region of interest as an object tracking technique to track how feature points found within the region of interest of the current frame change in the next frame.

The object information detection unit 130 determines the position change by obtaining a transformation matrix representing the movement of the tracked feature points and calculating the movement to a new frame through geometric transformation using a transformation command. Use it.

In the present invention, the performance of four object tracking algorithms was evaluated: CSRT (Channel and Spatial Reliability), KCF (Kernelized Correlation Filters), MIL (Multiple Instance Learning), and MOSSE (Minimum Output Sum of Squared Error). The results are shown in Figure 3.

The KCF algorithm using the characteristics of a circular matrix showed high processing speed, but frequently lost the object being tracked when the background of the object being tracked changed rapidly or the object was obscured.

The MIL algorithm, which tracks objects through multi-instance learning by considering the object's current location and surrounding locations as positive examples, showed irregular performance deviations depending on resolution and low processing speed.

The MOSSE algorithm using an adaptive correlation filter showed very fast processing speed and a decent tracking success rate, but performance showed a sharp decline when the object being tracked was obscured.

The CSRT algorithm, which operates by mapping the weights of objects set as regions of interest (ROI), operates through three-step operations: building a spatial reliability map, learning a restricted correlation filter, and estimating channel reliability.

The first step is to improve the search range and performance for non-rectangular, irregularly shaped objects by proposing that the range in which the filter operates within the area set as ROI is suitable for the object to be tracked.

In the second step, the influence of the background is reduced for pixels within the limited area through the first step and weight is given to the part corresponding to the object.

Finally, the position of the object is estimated by multiplying the filter's response for the entire unrestricted area by the weight calculated in the previous step.

This CSRT algorithm showed the best tracking success rate, and although the processing speed was on the low side, it was judged to be acceptable for actual use.

Summarizing the performance evaluation results, the CSRT algorithm was found to be suitable for cases where the shape and size of objects, such as pedestrians, change, and the KCF algorithm was also found to be applicable for static objects such as vehicles.

For example, by applying the CSRT algorithm, it is possible to determine the location of pedestrians in real time, as shown in Figure 5.

FIG. 5 is a diagram showing location information of an object detected in real time by the object information detection unit of FIG. 1.

In Figure 5, since the load value cannot be accurately predicted with a fixed floor structure, the shear force diagram and bending moment diagram are shown for each pedestrian's position based on an arbitrary load value.

At this time, the object information detector 130 may calculate the image coordinates of the object load reference point from the object area. The object load reference point refers to the central area that can be the reference point for the load of the object and, for example, can be determined as the left, right, and top and bottom center points of the object area.

In Figure 6, it can be seen that the central part of the object, a pedestrian, is indicated as the object load reference point and the coordinates are indicated as (A, B).

The coordinates (A, B) in FIG. 6 are coordinates on the image pixel reference on the image data. A refers to the pixel distance from the left side of the image to the object load reference point, and B refers to the pixel distance from the top of the image to the object load reference point.

The coordinate setting method in FIG. 6 is only an example, and the pixel length of B may be set in other ways, such as setting the length from the bottom of the image.

The scale coefficient calculation unit 140 serves to calculate a scale coefficient that is the ratio of the unit pixel length of the image data and the corresponding actual length.

FIG. 7 is a conceptual diagram illustrating a method of calculating the actual coordinates of the object load reference point calculated by the object position calculation unit of FIG. 1.

Referring to FIG. 7, the vertical height corresponding to the height of the pedestrian corresponds to the X pixel in the image data, and the actual height of the pedestrian is indicated as the Y value. In this case, the scale factor C, which is the ratio of the pixel to the actual length, can be calculated as in Equation 1 below.

[Equation 1]

In Equation 1 above,

In FIG. 7, the actual width length Z of the image can be obtained by multiplying W, the number of horizontal pixels in the frame, by the scale factor C.

The scale factor calculation unit 140 can calculate the scale factor in this manner, and the object for calculating the scale factor can be set in various ways.

Figure 7 shows an example of a situation in which the pedestrian's height is known in advance, the area of the pedestrian object is detected, pixel data corresponding to the height is extracted, and the scale factor is naturally calculated accordingly.

However, unlike this, it is also possible to detect a characteristic object other than a pedestrian and calculate a scale factor based on the left and right or top and bottom lengths of the object. For example, if the structure has a distinctive structure area that stands out and information on the horizontal length of the area is obtained in advance, the scale factor may be calculated by detecting the number of horizontal pixels in the image of the structure.

However, in this case, in addition to detecting pedestrians, which are dynamic objects, it is necessary to detect separate objects to calculate the scale factor.

The object position calculation unit 150 serves to calculate the actual coordinates of the object load reference point using image coordinates and a scale factor.

In the case of Figure 7, in the case of structural analysis, among the coordinates (A, B) on the image, A, which is the coordinate based on the left side of the image, may be an important value.

The object location calculation unit 150 can calculate the actual corresponding length of the pedestrian object as shown in Equation 2 below.

[Equation 2]

In Equation 2 above, A is the horizontal movement length of the dynamic object on the image, and A' means the actual horizontal movement length of the dynamic object.

FIG. 8 shows a graph showing the results of comparing the location of the object calculated by the object location calculation unit of FIG. 1 and the actual location.

As shown in Figure 8, the RMSE was approximately 2.47 pixels, indicating that the pedestrian's location could be estimated very accurately.

The load prediction unit 160 analyzes image data and calculates the load prediction value of the object load reference point.

The load prediction unit 160 may predict the load generated by the dynamic object by analyzing image data and detecting a change in the shape of the support structure of the dynamic object detected when the dynamic object moves.

For example, if the structure is a sidewalk suspension bridge, the vertical displacement of the bottom surface of the sidewalk suspension bridge may be detected when a pedestrian passes by. Since this vertical displacement may vary depending on the weight and position of the dynamic object, the load prediction unit 160 can predict the load in real time using a pre-built database.

At this time, the load prediction unit 160 can predict the load with a more accurate load prediction algorithm while updating the database according to the similarity of structures, similarity of objects, etc. based on deep learning techniques.

Figure 2 is a schematic flowchart explaining a method for recognizing the position of a real-time dynamic walking load using image-based motion sensing of the system of Figure 1.

The real-time dynamic walking load location recognition method using motion sensing according to an embodiment of the present invention first receives image data captured from a camera, etc. (S210).

In order to correct distortion of image data, image correction is performed using the Zhang algorithm, etc. (S220).

When the corrected image is calculated, moving object information is detected using the CSRT algorithm based on the corrected image data (S230).

When object information is detected, the coordinates of the object load reference point image are calculated (S240).

Additionally, the scale factor is calculated using the number of pixels of a specific object whose actual length is known in advance (S250).

Calculate the actual coordinates of the object load reference point from which the scale factor is calculated (S260). If the structure is a bridge, the actual moving distance or horizontal axis position of the pedestrian can be calculated using the above method.

If the structure is a structure in which vertical displacement occurs due to load, such as a sidewalk suspension bridge, the predicted load value is calculated using a preset algorithm (S270).

At this time, the method of calculating the predicted load value may be to analyze image data to calculate the position of the object and the vertical displacement of the structure at a specific point in time, and then calculate the predicted load value using a pre-built database and algorithm.

At this time, the load prediction method can adopt an accurate prediction method using big data through learning using a deep learning algorithm.

The method and system for real-time dynamic location recognition of a walking load using image-based motion sensing according to an embodiment of the present invention enables the location information of the load to be obtained conveniently and economically by analyzing image data in the same manner as described above.

A person skilled in the art to which the present invention pertains will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: Location recognition system 110: Image data receiving unit
120: Image data correction unit 130: Object information detection unit
140: Scale factor calculation unit 150: Object position calculation unit
160: Load prediction unit

Claims (14)

(a) receiving image data captured from a camera;
(b) analyzing the image data to detect object information;
(c) calculating image coordinates (A) consisting of pixels with respect to the object load reference point from the detected object information;
(d) calculating a scale factor (C), which is the ratio of the length of the unit pixel of the image data and the corresponding actual length, as shown in Equation 1 below;
<Equation 1>
Scale factor C = Y/X
(However, X is the pixel length of a specific object in the image data, and Y is the actual length of the specific object)
(e) calculating the actual coordinates (A') of the object load reference point using the image coordinates (A) and the scale factor (C),
In step (b),
(f) an object tracking step of tracking how feature points found within the region of interest of the current frame in the image data change in the next frame in a preset Region of Interests; and
(g) estimating the position change by calculating the movement to the next frame using a transformation matrix representing the movement of the tracked feature points and a geometric transformation using the transformation matrix; A real-time dynamic walking load location recognition method using image-based motion sensing.
According to paragraph 1,
Step (b) above is
(h) setting the range in which the filter operates within the area set as the area of interest to be limited to a search range for objects of an irregular shape other than a rectangle to suit the object to be tracked;
(i) assigning weights to parts corresponding to the background and object to pixels within the limited search range through step (h);
(j) estimating the location of the object by multiplying the response value of the filter for the entire range that is not limited in step (h) by the weight calculated in the previous step; video-based motion sensing comprising a. A real-time dynamic walking load location recognition method using.
delete According to paragraph 1,
In step (e), the actual coordinates are calculated by multiplying the image coordinates by the scale factor. A real-time dynamic walking load location recognition method using image-based motion sensing.
According to paragraph 1,
(k) A real-time dynamic location recognition method of walking load using image-based motion sensing, further comprising correcting distortion of the received image data between steps (a) and (b).
delete According to paragraph 1,
(l) further comprising calculating a predicted load value of the object load reference point by analyzing the image data,
In step (l), the image data is analyzed to detect changes in the shape of the support structure of the dynamic object detected when the object moves, and the load generated by the dynamic object is predicted in real time,
A real-time dynamic walking load location recognition method using image-based motion sensing, characterized in that the load prediction value is calculated while updating a database according to the similarity of the structure or the similarity of the object based on deep learning techniques.
A video data receiving unit that receives video data;
an object information detection unit that analyzes the image data to detect object information and calculates image coordinates (A) consisting of pixels with respect to the object load reference point from the object information;
A scale coefficient calculation unit that calculates a scale coefficient (C), which is the ratio of the length of the unit pixel of the image data and the corresponding actual length, as shown in Equation 1 below;
<Equation 1>
Scale factor C = Y/X
(However, X is the pixel length of a specific object in the image data, and Y is the actual length of the specific object)
An object position calculation unit that calculates actual coordinates (A') of the object load reference point using the image coordinates (A) and the scale factor (C),
The object information detection unit,
A transformation matrix that tracks how feature points found within the region of interest of the current frame in the image data change in the next frame in a preset Region of Interests, and represents the movement of the tracked feature points. And a position recognition system for dynamic walking load using image-based motion sensing, characterized in that the position change is estimated by calculating movement to the next frame using a geometric transformation using the transformation matrix.
According to clause 8,
The object information detection unit
The range in which the filter operates within the area set as the area of interest is limited to the search range for objects of irregular shape rather than rectangles to suit the object to be tracked,
Each weight is assigned to the background and object portions for pixels within the limited search range,
Position recognition of dynamic walking load using image-based motion sensing, characterized in that the object information is detected by estimating the position of the object by multiplying the response value of the filter for the entire unrestricted range by the weight calculated in the previous step. system.
delete According to clause 8,
A real-time dynamic walking load position recognition system using image-based motion sensing where the actual coordinates are calculated by multiplying the image coordinates by the scale factor.
According to clause 8,
A real-time dynamic walking load position recognition system using image-based motion sensing, further comprising an image data correction unit that corrects distortion of the received image data.
delete According to clause 8,
It further includes a load prediction unit that analyzes the image data and calculates a load prediction value of the object load reference point,
The load prediction unit,
Analyzing the image data to calculate the load prediction value of the object load reference point, analyzing the image data to detect changes in the shape of the support structure of the dynamic object detected when the object moves, predicting the load generated by the dynamic object in real time And, real-time dynamic location recognition of walking load using image-based motion sensing, characterized in that the predicted load value is calculated while updating the database according to the similarity of the structure or the similarity of the object based on deep learning techniques. system.