KR20220095359A - Method and system for real-time pedestrian dynamic-load localization using vision-based motion sensing - Google Patents

Abstract

A method and system for recognizing the location of a real-time dynamic walking load using image-based motion sensing according to one embodiment of the present invention may include the steps of: (a) receiving image data; (b) detecting object information by analyzing the image data; (c) calculating image coordinates of an object load reference point from the object information; (d) calculating a scale factor, which is a ratio of a length of a unit pixel of the image data and a corresponding actual length; and (e) calculating actual coordinates of the object load reference point using the image coordinates and the scale factor. The location of the dynamic load can be detected conveniently and economically only by image analysis.

Description

Method and system for real-time dynamic gait load position recognition using image-based motion sensing

The present invention relates to a real-time dynamic gait load position recognition method and system, and more particularly, to a real-time dynamic gait load position using image-based motion sensing capable of recognizing the position of a dynamic gait load in real time by analyzing image data. It relates to a recognition method and system.

If social infrastructures such as buildings and bridges are more than 30 years old, there is a high possibility of problems due to aging. Accordingly, the need for inspection and maintenance is increasing in order to minimize damage caused by deterioration of structures. In the past, to evaluate or check the deterioration of a structure, a method of directly checking and evaluating the condition of a structure was used by an expert, but this method is limited in that it requires not only subjective factors but also considerable manpower, time, and cost. there was

Vibration-based Structure Health Monitoring (SHM) technology is mainly used as a technology to detect deterioration of structures. This is a technology that measures the response by attaching a sensor such as an accelerometer to the structure, and grasps the state of the structure from this information. SHM technology is divided into an output-only method that utilizes only the response data of the structure and an input-output method that utilizes both inputs and outputs. Assuming that the input is white noise and using only the dynamic behavior information that is the output, the accuracy is somewhat low, but it is widely used due to the difficulty of measuring the input.

Korean Patent Publication No. 10-2017-0017659 (2017.02.15)

The present invention is to solve the above problems, and in detail, a method and system for real-time dynamic gait load position recognition using image-based motion sensing that can accurately and conveniently recognize the position of real-time dynamic gait load by analyzing image data is intended to provide

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In accordance with an embodiment of the present invention for solving the above problems, a method for recognizing a location of a real-time dynamic gait load using an image-based motion sensing includes the steps of: (a) receiving image data; (b) analyzing the image data to detect object information; (c) calculating image coordinates of an object load reference point from the object information; (d) calculating a scale factor that is a ratio of a length of a unit pixel of the image data and a corresponding actual length; and (e) calculating the actual coordinates of the object load reference point using the image coordinates and the scale factor.

Step (b) may be performed using a CRST algorithm.

In step (d), the scale factor may be calculated by Equation 1 below.

<Equation 1>

Scale factor = Y/X

(In Equation 1, X is the pixel length of the specific object in the image data, and Y is the actual length of the specific object)

In step (e), the actual coordinates may be calculated by multiplying the image coordinates by the scale factor.

Correcting the distortion of the received image data may be performed by a Zhang algorithm.

In addition, the real-time dynamic gait load position recognition method using image-based motion sensing according to an embodiment of the present invention may further include the step of analyzing the image data and calculating a load prediction value of the object load reference point.

In addition, the position recognition system of the dynamic walking load using the image-based motion sensing according to an embodiment of the present invention, the image data receiving unit for receiving the image data; an object information detection unit that analyzes the image data to detect object information, and calculates image coordinates of an object load reference point from the object information; an accumulation coefficient calculator configured to calculate a scale coefficient that is a ratio of a length of a unit pixel of the image data to an actual length corresponding to the image data; and

and an object position calculator for calculating the actual coordinates of the object load reference point by using the image coordinates and the scale factor.

The object information detector may detect the object information using a CRST algorithm.

The scale factor may be calculated by Equation 1 below.

<Equation 1>

Scale factor = Y/X

(In Equation 1, X is the pixel length of the specific object in the image data, and Y is the actual length of the specific object)

The actual coordinates may be calculated by multiplying the image coordinates by the scale factor.

In addition, the real-time dynamic gait load position recognition system using image-based motion sensing according to an embodiment of the present invention may further include an image data correction unit for correcting distortion of the received image data.

The image data corrector may correct distortion of the image data by using a Zhang algorithm.

In addition, the real-time dynamic gait load position recognition system using image-based motion sensing according to an embodiment of the present invention may further include a load predictor that analyzes the image data and calculates a load prediction value of the object load reference point.

The method and system for recognizing the location of real-time dynamic gait load using image-based motion sensing according to an embodiment of the present invention enable simple real-time recognition of the location of dynamic gait load simply by analyzing image data without a separate sensor.

The effect according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a schematic configuration diagram of a real-time dynamic gait load position recognition system using image-based motion sensing according to an embodiment of the present invention.
FIG. 2 is a schematic flowchart illustrating a real-time dynamic gait load position recognition method using image-based motion sensing of the system of FIG. 1 .
FIG. 3 is an exemplary result diagram of a performance evaluation experiment of an object information detection algorithm applicable to the object information detection unit of FIG. 1 .
FIG. 4 is a view showing a position result when original image data and data corrected by the image data correction unit of FIG. 1 are input.
5 is a diagram illustrating location information of an object detected in real time by the object information detection unit of FIG. 1 .
6 is a view showing an example of the object load reference point image coordinates calculated by the object information detection unit of FIG. 1 .
7 is a conceptual diagram for explaining a method of calculating the actual coordinates of the object load reference point calculated by the object position calculation unit of FIG. 1 .
8 is a graph showing a result of comparing the actual position of the object calculated by the object position calculator of FIG. 1 .

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Hereinafter, a method and system for real-time dynamic gait load position recognition using image-based motion sensing according to embodiments of the present invention will be described with reference to the drawings.

1 is a schematic configuration diagram of a real-time dynamic gait load position recognition system using image-based motion sensing according to an embodiment of the present invention.

1, the real-time dynamic gait load position recognition system 100 using image-based motion sensing according to an embodiment of the present invention includes an image data receiving unit 110, an image data correcting unit 120, an object information detecting unit ( 130 ), a scale factor calculating unit 140 , an object position calculating unit 150 , and a load prediction unit 160 .

In the present invention, the system 100 may be hardware such as a computer including the configuration of FIG. 1 . However, this is only an embodiment, and it may be an algorithmized computer program to implement the operation as in the configuration of FIG. 1 .

The image data receiving unit 110 serves to receive image data for the structure. In this case, the image data may be a device capable of taking a picture or a real-time video, such as a camera or a smart phone.

The image data correction unit 120 serves to correct distortion of the received image data. When an object such as a moving pedestrian in the image data is detected, distortion may occur in the conversion between the pixel length and the actual length depending on the location of the object according to the perspective effect of the 3D image.

The image data corrector 120 may perform camera calibration using, for example, an algorithm proposed by Zhang (2000) in order to correct the above distortion.

4 exemplarily shows the original image data and the position result when the data corrected by the image data correction unit of FIG. 1 are input.

The result of FIG. 4 shows that when the image is composed of 720 × 404 pixels, the root mean square error (RMSE) occurs about 5.4 pixels. In addition, since the distortion of the image occurs relatively large at the edge, it was confirmed that the error increases as the object moves to the left and right ends of the image.

In order to confirm the effect of image correction, the case in which the original image and the corrected image are input was compared.

The height of the pedestrian object to be tracked was already known to be 1,816 mm, and the number of pixels corresponding to the height of the pedestrian in the corrected image data was 163. The total number of horizontal pixels of the corrected image was 720, and the actual width of the bottom was 8,150 mm.

The width of the floor calculated by the formula to be described later was about 8,022 mm, and an error of about 1.57% occurred when compared with the actual measured value of 8,150 mm.

In the case of the image before correction, the number of pixels corresponding to the height of the pedestrian was 168, and the calculated floor width was 7,783 mm, showing an error of about 4.51%, which shows that the error reduction effect of about 2.94% was shown through image correction. .

The object information detection unit 130 analyzes image data to detect object information, and calculates image coordinates of an object load reference point from the detected object information.

The object information detector 130 uses a range designated by the user as a region of interest as an object tracking technique for tracking how feature points found in the region of interest of the current frame change in the next frame.

The object information detection unit 130 obtains a transformation matrix representing the movement of the tracked feature points, calculates the movement to a new frame through a geometric transformation using a transformation command, and determines the position change. use it

In the present invention, as an object tracking algorithm, performance evaluation was performed on four algorithms: Channel and Spatial Reliability (CSRT), Kernelized Correlation Filters (KCF), Multiple Instance Learning (MIL), and Minimum Output Sum of Squared Error (MOSSE). The result is shown in FIG. 3 .

In the case of the KCF algorithm using the characteristics of the cyclic matrix, the processing speed was high, but the object being tracked was frequently lost when the background of the object being tracked changed abruptly or when the object was covered.

The MIL algorithm that tracks the object through multi-instance learning, considering the current position of the object and the surrounding position as a positive example, showed irregular performance deviations according to resolution and low processing speed.

The MOSSE algorithm using the adaptive correlation filter showed a very fast processing speed and a satisfactory tracking success rate, but the performance dropped sharply when the object being tracked was obscured.

CSRT Algorithm, which operates by mapping the weight of an object set as a region of interest (ROI), operates through three-step operations: building a spatial reliability map, learning a limited correlation filter, and estimating the channel reliability.

The first step is to improve the search range and performance for non-rectangular irregular-shaped objects by proposing suitable for the object to be tracked within the area set by the ROI, where the filter operates.

In the second step, the influence of the background is reduced on the pixels within the limited area through the first step, and a weight is given to the part corresponding to the object.

Finally, the position of the object is estimated by multiplying the response of the filter for all unconstrained areas by the weight calculated in the previous step.

This CSRT algorithm showed the best tracking success rate, and although the processing speed was on the low side, it was judged to be at a level that is not unreasonable for practical use.

Combining the performance evaluation results, the CSRT algorithm was found to be suitable when the shape and size of an object change, such as a pedestrian, and the KCF algorithm was also found to be applicable to a static object such as a vehicle.

For example, if the CSRT algorithm algorithm is applied, the location of the pedestrian can be grasped in real time as shown in FIG. 5 .

5 is a diagram illustrating location information of an object detected in real time by the object information detection unit of FIG. 1 .

In FIG. 5, since the load value cannot be accurately predicted with a fixed floor structure, a shear force diagram and a bending moment diagram are shown for each position of a pedestrian based on an arbitrary load value.

In this case, the object information detection unit 130 may calculate the image coordinates of the object load reference point from the object area. The object load reference point refers to a central area that can be a reference point for the load of an object, and may be determined as, for example, left and right, upper and lower center points of the object area.

In FIG. 6 , it can be seen that the central part of the pedestrian, which is an object, is indicated as an object load reference point, and the coordinates are indicated by (A, B).

Coordinates (A, B) in Fig. 6 are coordinates on the reference image pixel on the image data. A is the pixel distance from the left side of the image to the object load reference point, and B is the pixel distance from the upper side of the image to the object load reference point.

The method of setting the coordinates of FIG. 6 is only one embodiment, and may be set in other ways, such as setting the pixel length of B to the length from the lower side of the image.

The scale factor calculator 140 calculates a scale factor that is a ratio of a unit pixel length of image data to a corresponding actual length.

7 is a conceptual diagram for explaining a method of calculating the actual coordinates of the object load reference point calculated by the object position calculation unit of FIG. 1 .

Referring to FIG. 7 , the vertical height corresponding to the pedestrian's height corresponds to X pixels on the image data, and the actual pedestrian's height is indicated by the Y value. In this case, a scale factor C, which is a ratio between a pixel and an actual length, may be calculated as in Equation 1 below.

[Equation 1]

In Equation 1, X is the length (number) of pixels in the left and right or vertical directions of a specific object in the image data, and Y is the actual length of the corresponding part.

In FIG. 7 , the actual width and length Z of the image may be obtained by multiplying W, which is the number of horizontal pixels of a frame, by the scale factor C. As shown in FIG.

The scale factor calculator 140 may calculate the scale factor in this way, and various objects for calculating the scale factor may be set.

7 exemplifies the case in which the area of the pedestrian object is detected and pixel data corresponding to the height is extracted in a situation in which the height of the pedestrian is known in advance, and the scale factor is naturally calculated accordingly.

However, unlike this, a method of detecting a separate characteristic object other than a pedestrian and calculating a scale factor based on the left and right or vertical lengths of the object is also possible. For example, when there is a distinctive structure area in a structure and horizontal length information of the area is previously obtained, the scale factor may be calculated by detecting the number of horizontal pixels in the image of the structure.

However, in this case, it is necessary to detect a separate object for calculating the scale factor in addition to detecting the pedestrian, which is a dynamic object.

The object position calculation unit 150 serves to calculate the actual coordinates of the object load reference point by using the image coordinates and the scale factor.

In the case of FIG. 7 , in the case of structure analysis, among the coordinates (A, B) on the image, A, which is a coordinate based on the left side of the image, may be an important numerical value.

The object position calculator 150 may calculate the actual corresponding length of the pedestrian object as shown in Equation 2 below.

[Equation 2]

In Equation 2, A is the horizontal movement length on the image of the dynamic object, and A' indicates the actual horizontal movement length of the dynamic object.

8 is a graph showing a result of comparing the actual position of the object calculated by the object position calculating unit of FIG. 1 .

As shown in FIG. 8 , the RMSE is about 2.47 pixels, indicating that the pedestrian's location can be estimated very accurately.

The load prediction unit 160 analyzes image data to calculate a load prediction value of an object load reference point.

The load prediction unit 160 may analyze the image data to detect a change in the shape of the support structure of the dynamic object detected when the dynamic object moves, and predict the load generated by the dynamic object.

For example, when the structure is a sidewalk suspension bridge, the vertical displacement movement of the floor surface of the sidewalk suspension bridge may be detected when a pedestrian passes. Since the vertical displacement may vary depending on the weight and location of the dynamic object, the load predictor 160 may predict the load in real time using a pre-built database.

At this time, the load prediction unit 160 may predict the load with a more accurate load prediction algorithm while updating the database according to the similarity of structures and objects based on the deep learning technique.

FIG. 2 is a schematic flowchart illustrating a real-time dynamic gait load position recognition method using image-based motion sensing of the system of FIG. 1 .

In the real-time dynamic gait load position recognition method using motion sensing according to an embodiment of the present invention, image data captured from a camera or the like is first received (S210).

In order to correct the distortion of the image data, an image correction operation is performed using the Zhang algorithm (S220).

When the corrected image is calculated, moving object information is detected using the CSRT algorithm based on the corrected image data (S230).

When the object information is detected, the coordinates of the object load reference point image are calculated (S240).

In addition, a scale factor is calculated using the number of pixels of a specific object whose actual length is known in advance (S250).

The actual coordinates of the object load reference point from which the scale factor is calculated are calculated (S260). When the structure is a bridge, it is possible to calculate the actual moving distance or the horizontal axis position of the pedestrian in the same way as described above.

If the structure is a structure in which vertical displacement occurs due to a load, such as a sidewalk suspension bridge, a predicted load value is calculated according to a preset algorithm (S270).

In this case, the method of calculating the predicted load value may be a method of calculating the predicted load value by using a pre-built database and algorithm by analyzing the image data to calculate the position of the object and the vertical displacement of the structure at a specific point in time.

In this case, the load prediction method may adopt an accurate prediction method using big data through learning using a deep learning algorithm.

The real-time dynamic gait load position recognition method and system using image-based motion sensing according to an embodiment of the present invention analyzes image data in the same manner as described above to obtain position information of the load conveniently and economically.

Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention. do.

100: location recognition system 110: image data receiving unit
120: image data correction unit 130: object information detection unit
140: scale factor calculation unit 150: object position calculation unit
160: load prediction unit

Claims (14)

(a) receiving image data;
(b) analyzing the image data to detect object information;
(c) calculating image coordinates of an object load reference point from the object information;
(d) calculating a scale factor that is a ratio of a length of a unit pixel of the image data and a corresponding actual length; and
(e) using the image coordinates and the scale factor to calculate the actual coordinates of the reference point of the object load, a real-time dynamic gait load position recognition method using image-based motion sensing comprising the step of.
According to claim 1,
The step (b) is a real-time dynamic gait load position recognition method using image-based motion sensing, characterized in that it is performed using a CSRT algorithm.
According to claim 1,
In step (d), the scale factor is a real-time dynamic gait load position recognition method using image-based motion sensing calculated by Equation 1 below.
<Equation 1>
Scale factor = Y/X
(In Equation 1, X is the pixel length of a specific object in the image data, and Y is the actual length of the specific object)
4. The method of claim 3,
In the step (e), the actual coordinates are a real-time dynamic gait load position recognition method using image-based motion sensing that is calculated by multiplying the image coordinates by the scale factor.
According to claim 1,
The real-time dynamic gait load position recognition method using image-based motion sensing further comprising correcting the distortion of the received image data between the steps (a) and (b).
6. The method of claim 5,
Correcting the distortion of the received image data is a real-time dynamic gait load position recognition method using image-based motion sensing, characterized in that performed by the Zhang algorithm.
The method of claim 1,
The method of real-time dynamic gait load position recognition using image-based motion sensing further comprising the step of analyzing the image data and calculating a load prediction value of the object load reference point.
an image data receiving unit for receiving image data;
an object information detection unit that analyzes the image data to detect object information, and calculates image coordinates of an object load reference point from the object information;
an accumulation coefficient calculator configured to calculate a scale coefficient that is a ratio of a length of a unit pixel of the image data to an actual length corresponding to the image data; and
Position recognition system of dynamic walking load using image-based motion sensing including an object position calculator for calculating actual coordinates of the reference point of the object load by using the image coordinates and the scale factor.
9. The method of claim 8,
Wherein the object information detection unit detects the object information by using a CSRT algorithm, a dynamic gait load position recognition system using image-based motion sensing.
9. The method of claim 8,
The scale factor is a real-time dynamic gait load position recognition system using image-based motion sensing calculated by Equation 1 below.
<Equation 1>
Scale factor = Y/X
(In Equation 1, X is the pixel length of a specific object in the image data, and Y is the actual length of the specific object)
10. The method of claim 9,
The real coordinate is a real-time dynamic gait load position recognition system using image-based motion sensing that is calculated by multiplying the image coordinates by the scale factor.
9. The method of claim 8,
A real-time dynamic gait load position recognition system using image-based motion sensing further comprising an image data correction unit for correcting the distortion of the received image data.
13. The method of claim 12,
The image data correction unit uses a Zhang algorithm to correct the distortion of the image data. A real-time dynamic gait load position recognition system using image-based motion sensing.
9. The method of claim 8,
A real-time dynamic gait load position recognition system using image-based motion sensing further comprising a load predictor that analyzes the image data and calculates a load prediction value of the object load reference point.

Images