KR20200062065A - Method and apparatus for prodicting moving path - Google Patents

Abstract

The present invention relates to a method and a device for accurately predicting a moving path by combining image data and sensor data. The device for predicting a moving path extracts an object from image data to identify an object of interest, extracts behavior information of the object of interest from the sensor data obtained from a personal portable terminal of the object of interest, synchronizes the behavior information of the object of interest obtained from the sensor data with that of the object of interest identified from the image data according to time information to generate training data for an analysis model, and predicts a moving path and behavior information of the object of interest through the output of the analysis model when input data is input to the analysis model.

Description

METHOD AND APPARATUS FOR PRODICTING MOVING PATH}

The present invention relates to a method and apparatus for predicting a movement path, and in particular, a movement path prediction method for predicting a movement path and a behavior of a user by fusing CCTV (Closed Circuit Television) image data and sensor data acquired through a user's personal mobile terminal. And devices.

Existing technologies for predicting the moving path of an object of interest through image data recorded in CCTV include factors such as the angle of light, reflection of light, and other factors in the outdoor environment, and the overlap of non-interested objects in the image to obtain the object of interest. It was not easy to identify the object of interest. Therefore, since the accuracy of detailed information (metadata) of the object extracted through the image data is not high, it is not easy to accurately obtain a prediction of the moving path of the object of interest.

In order to solve this problem, learning data is required to accurately predict a moving path of an object of interest by combining image data and sensor data. Since the prediction result value varies depending on how the training data is constructed and used in the learning model, the construction of the training data combining image data and sensor data acts as an important technical factor for obtaining the prediction result value.

The problem to be solved by the present invention is to provide a moving path prediction method and apparatus capable of accurately predicting a moving path of an object of interest by combining image data and sensor data.

According to an embodiment of the present invention, a method for predicting a moving path of an object of interest in a moving path prediction apparatus is provided. The moving path prediction method includes extracting an object from image data to identify the object of interest, extracting action information of the object of interest from sensor data obtained from a personal portable terminal of the object of interest, and obtained from the sensor data Synchronizing the behavior information of the object of interest according to time information with the object of interest identified from the image data to generate learning data of the analysis model, and when input data is input to the analysis model, through output of the analysis model And predicting a movement path and behavior information of the object of interest.

According to an embodiment of the present invention, by using the recognition and tracking technology of the object of interest (person) used in the field of video security surveillance technology, predict the value of the moving path of the object of interest (person) in the near future, and It can be used in a variety of service models that can predict behavior and proactively respond to dangerous situations.

1 is a view for explaining an example of acquiring a user's movement path and behavior pattern using CCTV image data and sensor data according to an embodiment of the present invention.
2 is a diagram illustrating a method of obtaining image metadata of an object of interest according to an embodiment of the present invention.
3 is a diagram illustrating a method of obtaining sensor metadata of an object of interest according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a method of temporally mapping image metadata and sensor metadata according to an embodiment of the present invention and using it as learning data.
5 is a flowchart illustrating a method of synchronizing image metadata and sensor metadata according to time information according to an embodiment of the present invention.
6 is a diagram illustrating an apparatus for predicting a moving path according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification and claims, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless specifically stated to the contrary.

Now, a method and apparatus for predicting a moving path according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a view for explaining an example of acquiring a user's movement path and behavior pattern using CCTV image data and sensor data according to an embodiment of the present invention.

Referring to Figure 1, CCTV (Closed Circuit Television) (CAM1, CAM2, CAM3, CAM4) is installed in different places. A user having a personal portable terminal moves to a place where CCTV (CAM1, CAM2, CAM3, CAM4) is installed according to the time (T1, T2, T3, T4).

CCTV (CAM1, CAM2, CAM3, CAM4) photographs the user's image at the installed location, records and records the captured image data, and transmits the recorded image data to the movement path prediction apparatus 100.

The movement path prediction apparatus 100 collects image data from CCTV (CAM1, CAM2, CAM3, CAM4), and identifies and tracks a user from the image data recorded through CCTV (CAM1, CAM2, CAM3, CAM4). The motion path prediction apparatus 100 performs a re-identification process to check whether the same user is the same by comparing the image data continuously collected according to the times T1, T2, T3, and T4 with the previously collected image data. In the process of identifying or re-identifying the user, the user may acquire a size recognized in the image data and coordinate values located in the image data.

Specifically, at time T1, the CCTV (CAM1) captures the image of the user whose user identifier is Person A, and transmits the captured image data to the motion path prediction apparatus 100. The movement path prediction apparatus 100 acquires the image data captured at time T1 and the acceleration sensor data from the user's personal portable terminal, analyzes them using these data, and analyzes the result, and corresponds to Person A on the movement path The user can know that he is currently walking.

Then, at time T2, CCTV (CAM2) transmits the CCTV image data of the user photographed at time T2 to the movement path prediction apparatus 100. The movement path prediction apparatus 100 identifies a user corresponding to Person A from the image data captured at time T2, and analyzes acceleration sensor data of the user's personal portable terminal, and corresponds to Person A on the movement path A user who can do this can know that he is in a jogging state.

Subsequently, at time T3, the movement path prediction apparatus 100 analyzes by using the video data captured through CCTV (CAM3) at time T3 and the acceleration sensor data of the user's (Person A) personal mobile terminal. , As a result of analysis, it can be seen that the user corresponding to Person A on the corresponding movement path is in an ascending state.

Subsequently, at time T4, the movement path prediction apparatus 100 analyzes by using the video data captured through CCTV (CAM4) at time T4 and the acceleration sensor data of the personal portable terminal of the user (Person A). As a result, it can be seen that the user corresponding to Person A on the corresponding movement path is in a descending state of stairs.

The moving path prediction apparatus 100 accumulates and analyzes the information analyzed according to the flow of time (T1, T2, T3, T4) as metadata, and uses the information of the user (Person A) at the time to come (for example, T10). Predict the user's behavioral patterns (eg, walking, running, climbing stairs, descending stairs, stopping, falling, etc.) on the moving path and on the moving path. This prediction enables users to avoid the dangerous situations (for example, entering a dangerous place where a past crime occurred, encountering a dangerous vehicle, encountering a dangerous person) in the time to come (for example, T10). Enable public safety assurance services to be realized.

In particular, the movement path prediction apparatus 100 according to an embodiment of the present invention includes image metadata obtained through video data analysis of CCTV (CAM1, CAM2, CAM3, CAM4) and acceleration sensor data of the personal portable terminal of the object of interest. The analysis model is trained using the behavior pattern classification values obtained through feature analysis as training data, and the user's movement path and behavior are predicted for the time to come using the learned analysis model. As for the analysis model, various techniques such as Recurrent Neural Network (RNN) and Long-Short Term Memory (LSTM), which are deep learning techniques used for time series data analysis, can be used.

2 is a diagram for explaining a method of obtaining image metadata of an object of interest according to an embodiment of the present invention.

Referring to FIG. 2, the movement path prediction apparatus 100 acquires image data from CCTV (S210).

The motion path prediction apparatus 100 detects an object for each frame constituting image data. The motion path prediction apparatus 100 may extract a feature using a feature extraction algorithm from frame-by-frame image data to detect the object (S220), and detect the object using the extracted feature (S230). As an example, existing image processing algorithms such as Histogram of Oriented Gradients (HOG) and Har-like features can be used, or the motion path prediction apparatus 100 is an object using deep learning to detect an object. You can use an extraction analysis technique: You can use YOLO (You only Look Once) or Fast-RNN as an object extraction analysis technique using deep learning.

The motion path predicting apparatus 100 classifies the objects detected in the current frame (S240), then compares the classified objects with similarities to the objects detected in the previous frame, and performs a re-identification process for the object of interest (S250). ), the change of the movement of the object of interest with time is confirmed through re-identification.

The motion path prediction apparatus 100 calculates a similarity score by comparing color characteristics and/or contour characteristics of a garment by classifying upper and lower objects of interest to re-identify the object of interest, and if the similarity score is equal to or greater than a certain score It can be re-identified with the same object. For example, the motion path prediction apparatus 100 calculates a color similarity score by comparing the color characteristics of the clothes by classifying the top and bottom of the object of interest as shown in Equation 1 to re-identify the object of interest. If it is above a certain score, it can be re-identified with the same object.

Specifically, the similarity score can be calculated as in Equation 2.

In Equation 2, B, G, and R represent blue, green, and red, respectively. B ₁₁ , G ₁₁ , and R ₁₁ each represent blue, green, and red for the user 1's top, and B ₁₂ , G ₁₂ , and R ₁ 2 respectively represent the blue, green, and red for the top of the user 2. B ₂₁ , G ₂₁ , and R ₂₁ each represent blue, green, and red colors for User 1's bottoms, and B ₂₂ , G ₂₂ , and R ₂₂ respectively represent Blue, green, and red colors for User 2's bottoms.

On the other hand, there is no temporal synchronization information between the video data generated in CCTV and the sensor data in the personal portable terminal because the CCTV for generating video data and the personal portable terminal for generating sensor data are different devices and operate independently. If the image data and the sensor data are asynchronously trained, it is necessary to solve the problem of fusion of features in the learning process, and even if the analysis model is trained, an error occurs in accuracy over time in the process of obtaining prediction results. . Therefore, a method of synchronizing image data and sensor data to solve this problem will be described with reference to FIGS. 3 to 5.

3 is a diagram illustrating a method of obtaining sensor metadata of an object of interest according to an embodiment of the present invention.

Referring to FIG. 3, user A and user B have different behavior patterns according to time T1, T2, and T3. The motion path prediction apparatus 100 predicts the behavior patterns of user A and user B that may occur at time T3 by using the metadata analyzed through time T1 and T2 as past data.

That is, at time T1, when user A and user B are judged to be one of the following actions: fall, stop, and run according to the sensor data analysis model, the analyzed result value is the sensor meta. Used as data. This result value is combined with the image metadata obtained by analyzing in FIG. 2 and mapping the relationship between the image metadata and the sensor metadata based on the time information, thereby predicting the behavior pattern according to the movement paths of the user A and the user B. It can be used as learning data for.

FIG. 4 is a diagram illustrating a method of temporally mapping image metadata and sensor metadata according to an embodiment of the present invention and using it as learning data.

Referring to FIG. 4, input data 401 for learning is input to the analysis model 405.

The input data 401 for learning includes the frame number (frame_number), user identifier (uer_id), time information (timestamp), x coordinate value (x_value) of the object in the image from the time information, and the object in the image from the time information. y coordinate values (y_value) and behavior pattern classification values (activity) may be included.

The learning data 401 configured as described above is input to the analysis model 405 to train the analysis model 401. The learned analysis model 405 outputs a prediction result value 403 in the near future when inputting test data similar or identical to the learning data.

The motion path prediction apparatus operates as shown in FIG. 5 to generate learning data in a fused form of image metadata and sensor metadata based on time information.

5 is a flowchart illustrating a method of synchronizing image metadata and sensor metadata according to time information according to an embodiment of the present invention.

Referring to FIG. 5, the motion path prediction apparatus 100 generates matching information for a user identifier and characteristics of a user image (S502). For example, the movement path prediction apparatus 100 generates and stores matching information characterized by the user's personal mobile terminal number and the user's clothing information (color, shape), and the user's personal mobile terminal number as a user identifier. By using, it is possible to store matching information.

Thereafter, the movement route prediction apparatus 100 acquires CCTV image data recorded from CCTV and sensor data from the user's personal portable terminal (S504). The motion path prediction device 100 collects image data and sensor data at the same place from CCTV, which is a physically independent device, and a personal portable terminal, respectively.

The motion path predicting apparatus 100 operates a system timer to grasp time relationship information between CCTV image data and sensor data. The motion path prediction apparatus 100 sets a CCTV video analysis timer (timer1) to check the time required to extract the object of interest from CCTV video data (S506), and sequentially from the first frame to the last frame of CCTV video data. Objects are detected by analyzing image data.

The movement path prediction apparatus 100 checks whether there is sensor data generated in a time interval of the CCTV video analysis timer (timer1) (S508).

When the sensor data of the related user is collected in the process of analyzing the image from the first frame to the last frame of the CCTV image data during the time interval of the CCTV image analysis timer (timer1), the movement path prediction apparatus 100 is used to collect sensor data. Set a sensor timer (timer2) for checking the time for (S510).

If there is no sensor data input during the time interval of the CCTV video analysis timer (timer1), and the object analysis is completed until the last frame of the input video data (S512), the motion path prediction device 100 is displayed between the analyzed objects by frame. The similarity is calculated (S514). The similarity score may be calculated based on the feature of the contour between the object extracted from the previous frame and the object extracted from the current frame, and/or the color value for the top and bottom clothes.

When the similarity degree exceeds a certain value (for example, 80%), the motion path prediction apparatus 100 determines that the corresponding object is the same user, and gives the same user identifier to the corresponding object (S516).

Next, the motion path prediction apparatus 100 tags the object similar to the feature information of the user image matched with the user identifier as the user ID of the matched user (S518).

The moving path prediction apparatus 100 recognizes an object from the CCTV image data and re-identifies the object through these processes (S502 to S508, S512 to S518).

Meanwhile, if the sensor data collected during the time interval of the CCTV image analysis timer (timer1) exists (S508), the movement path prediction apparatus 100 analyzes the behavior pattern through the sensor data and uses it as learning data To perform. Specifically, the movement path prediction apparatus 100 records the time information (timestamp) of the sensor timer (timer2) and separates the values of the x, y, and z axes of the sensor data based on the time information (timestamp), and behavior patterns The behavior pattern values analyzed through the analysis model for classifying are recorded along with timestamp for each user identifier (S520, S522).

The motion path prediction apparatus 100 generates learning data using coordinate information and behavior pattern information of an object in the image data according to the user identifier (S524).

As described above, the movement path prediction apparatus 100 according to an embodiment of the present invention recognizes an object from CCTV image data, re-identifies the object, and analyzes the behavior pattern of the object through sensor data obtained from the personal portable terminal. , Secure learning data.

The moving path prediction apparatus 100 trains the analysis model by inputting the training data into the analysis model.

When an analysis model learned in this way receives data in a form similar to the learning data, it outputs coordinate information and behavior pattern information of an object corresponding to the near future, thereby predicting the movement path and behavior of the user.

6 is a diagram illustrating an apparatus for predicting a moving path according to an embodiment of the present invention.

Referring to FIG. 6, the motion path prediction device 600 includes a processor 610, a memory 620, a storage device 630, and an input/output (I/O) interface 640.

The processor 610 may be implemented as a central processing unit (CPU) or other chipset, microprocessor, or the like.

The memory 620 is a medium such as RAM, such as dynamic random access memory (DRAM), rambus DRAM (RDRAM), synchronous DRAM (SDRAM), static RAM (SRAM), etc. Can be implemented as

The storage device 630 is a hard disk, compact disk read only memory (CD-ROM), CD rewritable (CD-RW), digital video disk ROM (DVD-ROM), DVD-RAM, DVD-RW disk , Blu-ray disks, optical disks, flash memory, and various types of RAM such as permanent or volatile storage devices.

I/O interface 640 allows processor 610 and/or memory 620 to access storage 630. Also, the I/O interface 640 may provide an interface with a user.

The processor 610 may perform training data generation, learning, and movement path prediction functions for movement path prediction described with reference to FIGS. 1 to 5, and execute program instructions for implementing training data generation, learning, and movement path prediction functions. Loaded into the memory 620, it can be controlled to perform the operation described with reference to Figures 1 to 5. In addition, these program commands may be stored in the storage device 630 or may be stored in another system connected through a network.

Although the embodiments of the present invention have been described in detail above, the scope of rights of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (1)

A method for predicting a moving path of an object of interest in the moving path prediction apparatus,
Identifying the object of interest by extracting an object from image data,
Extracting behavior information of the object of interest from sensor data obtained from the personal portable terminal of the object of interest;
Synchronizing action information of the object of interest obtained from the sensor data according to time information with the object of interest identified from the image data to generate learning data of the analysis model; and
When input data is input to the analysis model, predicting a movement path and behavior information of the object of interest through the output of the analysis model
Moving path prediction method comprising a.

Images