KR102656084B1 - Method and apparatus for mapping objects besed on movement path - Google Patents

Abstract

A movement path-based object mapping device acquires movement path information of an object of interest, extracts an object area from at least one frame image each captured by a plurality of cameras on the movement path, and then selects feature points of the object area extracted from each camera. Based on this, identical objects are classified into one cluster. And, among the images of the object area within the cluster for each object, the image of the object area mapped at the most points on the movement path is output.

Description

Movement path-based object mapping method and device {METHOD AND APPARATUS FOR MAPPING OBJECTS BESED ON MOVEMENT PATH}

The present invention relates to a movement path-based object mapping method and device, and more specifically, to a movement path-based object mapping method and device for finding an object matching a given movement path from a closed-circuit television (CCTV) image when the movement path of an object is given. It relates to methods and devices.

The technology to find the same object (for example, a person or a vehicle) in a multiple closed-circuit television (CCTV) environment through video analysis technology is called Re-ID (Identification) technology. At this time, recognizing the same object in different CCTV screens is not easy even for human abilities because the resolution, lens distortion, and color output for each CCTV are different.

Recently, as Re-ID technology, a part of deep learning technology, has developed, re-recognition performance is increasing. However, there are many cases where it is not easy to distinguish even with deep learning technology. For example, in some CCTVs, only the front of a person is filmed, while in other CCTVs, only the back of a person can be filmed, even though they are the same person. At this time, if this person is wearing clothes that are similar on the front and back, it is relatively easy to distinguish them, but if they are wearing clothes that are completely different on the front and back, there is a high probability that they will be recognized as different people.

Re-ID technology is a problem of finding the same object among gallery images by entering a query image corresponding to the object of interest. However, there are cases where only the movement path information of a specific object is given without a query image, and the object must be recognized through CCTV. For example, location tracking electronic devices are worn by people who require protective surveillance. When a person wearing a location tracking electronic device moves, the person's movement path can be known through real-time GPS information. However, it is impossible to know what a person is doing simply by looking at his/her movement path. This can only be detected through CCTV. If the same object can be identified through CCTV with only information on the movement path of the object requiring observation, damage can be reduced by taking action before a crime occurs, or suspects can be quickly identified through CCTV when the criminal's escape route is expected. It may be limited.

The problem to be solved by the present invention is to provide a movement path-based object mapping method and device that can recognize objects matching the movement path in a multi-CCTV environment based on the movement path information of the object.

According to one embodiment of the present invention, a movement path-based object mapping method of a movement path-based object mapping device is provided. The movement path-based object mapping method includes obtaining movement path information of an object of interest, extracting an object area from at least one frame image each captured by a plurality of cameras on the movement path, and object areas extracted from each camera. It includes the step of classifying the same object into one cluster based on the feature points of , and outputting the image of the object area mapped at the most points on the movement path among the images of the object area within the cluster for each object.

According to an embodiment of the present invention, by mapping the object in the image to the movement path information based on the object's movement path information, an object matching the movement path can be found quickly and tracking of the object is possible.

Figure 1 is a diagram showing the concept of object mapping based on a movement path according to an embodiment of the present invention.
Figure 2 is a diagram showing a movement path-based object mapping device according to an embodiment of the present invention.
FIG. 3 is a diagram showing an example of a clustering result for the same person extracted from one CCTV by the intra-cluster generator shown in FIG. 2.
FIG. 4 is a diagram showing an example of clustering results for the same person extracted from different CCTVs by the inter-cluster generator shown in FIG. 2.
Figure 5 is a flowchart showing a movement path-based object mapping method according to an embodiment of the present invention.
Figure 6 is a schematic diagram of a movement path-based object mapping device according to another embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification and claims, when a part is said to “include” a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Now, the movement path-based object mapping method and device according to an embodiment of the present invention will be described in detail with reference to the drawings.

Figure 1 is a diagram showing the concept of object mapping based on a movement path according to an embodiment of the present invention.

Referring to FIG. 1, the movement path-based object mapping device 100 acquires movement path information of a person or vehicle (hereinafter referred to as “object of interest”). A movement path-based object mapping device can obtain movement path information from location information from a location tracking electronic device possessed by the object of interest (or mounted on the object).

The movement path-based object mapping device 100 receives images captured from a plurality of CCTVs (10a, 10b, 10c, 10d) located on the movement path of the object of interest, and analyzes the input images to determine the number of CCTVs (10a). , 10b, 10c, and 10d). The movement path-based object mapping device 100 maps objects appearing in each CCTV (10a, 10b, 10c, 10d) on the movement path through image analysis, and detects the most mapped object on the given movement path. can do. In Figure 1, for convenience, different objects appearing in each CCTV (10a, 10b, 10c, 10d) on the movement path are expressed in different shapes.

Figure 2 is a diagram showing a movement path-based object mapping device according to an embodiment of the present invention.

Referring to FIG. 2, the movement path-based object mapping device 100 includes a frame extractor 110, an object detector 120, an object recognition feature point extractor 130, an object image storage unit 140, and an intra cluster generator 150. , It includes an intra-cluster result storage unit 160, an inter-cluster generator 170, an inter-cluster result storage unit 180, and a movement path mapping unit 190.

Videos from multiple CCTVs are input to the frame extractor 110.

The frame extractor 110 extracts a frame image by sampling the video of each CCTV. Typically, video uses 30 frames per second. The frame extractor 110 may extract all 30 frames per second of images, but may be limited to 1 to 2 frames per second in consideration of the overhead of image analysis. The extracted image is transmitted to the object detector 120.

The object detector 120 extracts an object area from the frame image extracted from the frame extractor 110 and transmits the extracted object area to the object recognition feature point extractor 130. The image of the object area may include location information of the CCTV that acquired the image. The object detector 120 can extract the object area using a deep learning model that repeatedly learns the location information of objects from a large amount of images containing objects. The object detector 120 may use a YOLO (You only Look Once) or SSD (Single Shot Detector) model, but is not limited thereto.

The object recognition feature point extractor 130 detects whether two objects detected from object areas of different frame images in each CCTV are the same object. The object recognition feature point extractor 130 can check whether different frame images of the same CCTV are obtained from CCTV location information included in the image of the object area of each frame image. The object recognition feature point extractor 130 detects whether the object extracted from the consecutive frames is the same object based on the feature points of the objects extracted from the consecutive frames recorded in one CCTV. Images of the object area for each CCTV are stored in the object image storage unit 140. The object recognition feature point extractor 130 may use a deep learning model to detect whether the object is the same. The deep learning model of the object recognition feature point extractor 130 uses multiple images of the same object and multiple images of different objects to learn whether it is the same object or different objects, so that when two object images are input, it determines whether they are the same object or different objects. Predict and output. The object recognition feature point extractor 130 may use the ResNet model, but is not limited to this.

According to an embodiment of the present invention, the object detector 120 may be composed of a vehicle detector 122 and a person detector 124. The object recognition feature point extractor 130 may be composed of a vehicle recognition feature point extractor 132 and a person recognition feature point extractor 134. Additionally, the object image storage unit 140 may be comprised of a vehicle image storage unit 142 and a person image storage unit 144. In general, most objects of interest to be found are vehicles or people, so in the embodiment of the present invention, the description is limited to the object to be found as a vehicle or person. However, if the object of interest is not a vehicle or a person, an object detector is used according to the object of interest. (120), the configuration of the object recognition feature point extractor 130 and the object image storage unit 140 may vary.

The vehicle detector 122 extracts the vehicle area from the frame image. The person detector 124 extracts the person area from the frame image. The vehicle area information extracted for each CCTV or the person area information extracted for each CCTV is transmitted to the vehicle recognition feature point extractor 132 and the person recognition feature point extractor 134, respectively. The vehicle re-recognition feature point extractor 132 detects whether the vehicle is the same from vehicle areas of different frame images. The person recognition feature point extractor 134 detects whether the person is the same person from the person area of different frame images. Vehicle area information extracted for each CCTV is stored in the vehicle image storage unit 142. The person area information extracted for each CCTV is stored in the person image storage unit 144.

The intra-cluster generator 150 classifies identical objects appearing on consecutive frames for each CCTV into one cluster. The classification results are stored as cluster representative images for each CCTV in the intra-cluster result storage unit 160. When clustering for each CCTV is performed, identical objects for each CCTV form a group in the form of a cluster. The center point is determined based on the Euclidean distance between the objects forming the group, and the image closest to the center point can be selected as the representative image of the cluster in the corresponding CCTV. This method is an example, and the center point may be determined through other methods. Cluster representative images for each CCTV are transmitted to the inter-cluster generator 170. According to one embodiment, the intra cluster generator 150 may use the K-mean model, but is not limited to this.

FIG. 3 is a diagram showing an example of a clustering result for the same person extracted from one CCTV by the intra-cluster generator shown in FIG. 2.

As shown in Figure 3, the same people appearing in consecutive frames in one CCTV are divided into one cluster (C1, C2, C3). At this time, the clustering results for the same people appearing in consecutive frames in one CCTV tend to cluster objects with very similar shapes. This is because there is often little difference in the movement of the same object in consecutive frames.

Again, looking at FIG. 2, the inter-cluster generator 170 classifies the same object into one cluster for all cameras based on the feature points of representative images of each cluster classified by the intra-cluster generator 150. The classification results are stored as cluster representative images for each object in the inter-cluster result storage unit 180. According to one embodiment, the inter-cluster generator 170 may use a K-mean model, but is not limited thereto.

FIG. 4 is a diagram showing an example of clustering results for the same person extracted from different CCTVs by the inter-cluster generator shown in FIG. 2.

Referring to FIG. 4, the inter-cluster generator 170 classifies the same person in different CCTV images into one cluster (C11, C22, C33, C44). However, due to differences in lens distortion, resolution, shooting angle, color, etc. between cameras, people are often classified into the same cluster even though they are not the same person, or conversely, they are classified into different clusters even though they are the same person. For example, like the image 400 of the object area of the cluster C44, the images 400 may be classified into the same cluster C44 even though they are not the same person. The inter-cluster generator 170 can output clustering results for each object, and the user checks misclassified objects from the clustering results for each object. The inter-cluster generator 170 can correct objects misclassified by the user in the object-specific clustering results. Specifically, the inter-cluster generator 170 outputs the results classified by cluster as an image and provides it to the user. The user can finally select the object that appears most frequently in each cluster. Then, the inter-cluster generator 170 can correct misclassified objects by constructing a corresponding cluster only from objects selected by the user.

Next, looking at FIG. 2, the movement path mapping unit 190 acquires the movement path information of the object of interest, and the movement path information and the most Determine a cluster representative image that is mapped at many points. The movement path mapping unit 190 maps the images within each cluster to the movement path information based on the location information of the CCTV from which the corresponding image was extracted from the object cluster classified for all cameras, and maps the images within the movement path information at the most points on the movement path information. The cluster representative image of the mapped cluster, that is, the image of the object area, is detected. The movement path mapping unit 190 can detect an image of an object area that matches the movement path.

In addition, the movement path mapping unit 190 can determine the movement path of the object by checking the location information of the CCTV that captured the image from the images in the cluster for each object.

Figure 5 is a flowchart showing a movement path-based object mapping method according to an embodiment of the present invention.

Referring to FIG. 5, the movement path-based object mapping device 100 acquires movement path information of the object of interest (S500).

A frame image is extracted by sampling video input from multiple CCTVs (S510).

The movement path-based object mapping device 100 extracts an object area from at least one frame image extracted from each CCTV (S520).

The movement path-based object mapping device 100 primarily classifies the same object into one cluster for each CCTV based on feature points of the object area extracted from at least one frame image within the same CCTV (S530).

Next, the movement path-based object mapping device 100 reclassifies the overall identical objects into one cluster for the object areas within each CCTV cluster (S540).

The movement path-based object mapping device 100 maps images of the object area within a cluster for each object to movement path information based on the corresponding CCTV location information (S550), and images of the object area mapped at the most points on the movement path are Print out (S560).

At least some functions of the movement path-based object mapping method and device according to the embodiment of the present invention described above may be implemented as hardware or as software combined with hardware. Below, an embodiment in which a movement path-based object mapping method and device are combined with a computer system will be described in detail with reference to FIG. 6.

Figure 6 is a schematic diagram of a movement path-based object mapping device according to another embodiment of the present invention, which can be used to perform at least some of the movement path-based object mapping method and device described with reference to Figures 1 to 5 represents the system.

Referring to FIG. 6 , the movement path-based object mapping device 600 includes a processor 610, a memory 620, a storage device 630, and an input/output (I/O) interface 640.

The processor 610 may be implemented as a central processing unit (CPU), other chipset, microprocessor, etc.

The memory 620 is a RAM-like medium such as dynamic random access memory (DRAM), rambus DRAM (RDRAM), synchronous DRAM (SDRAM), and static RAM (static RAM, SRAM). It can be implemented as:

The storage device 630 may be a hard disk, compact disk read only memory (CD-ROM), CD rewritable (CD-RW), digital video disk ROM (DVD-ROM), DVD-RAM, or DVD-RW disk. , may be implemented as permanent or volatile storage devices such as optical disks such as Blu-ray disks, flash memory, and various types of RAM.

I/O interface 640 allows processor 610 and/or memory 620 to access storage device 630.

The processor 610 can perform the movement path-based object mapping method described in FIGS. 1 to 5, and includes a frame extractor 110, an object detector 120, an object recognition feature point extractor 130, and an intra cluster generator 150. , program instructions for implementing at least some functions of the inter-cluster generator 170 and the movement path mapping unit 190 are loaded into the memory 620 and controlled so that the operations described with reference to FIGS. 1 to 5 are performed. You can. These program commands may be stored in the storage device 630, or may be stored in another system connected to a network.

The memory 620 or storage device 630 may include an object image storage unit 140, an intra-cluster result storage unit 160, and an inter-cluster result storage unit 180.

I/O interface 640 may provide a user interface. Through the I/O interface 640, information on the movement path of the object of interest and images from CCTV can be input, and an image of the object mapped at the most points on the movement path can be output.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements can be made by those skilled in the art using the basic concept of the present invention defined in the following claims. It falls within the scope of rights.

Claims (9)

A movement path-based object mapping method of a movement path-based object mapping device,
Obtaining information on the movement path of the object of interest,
Extracting an object area from a plurality of frame images each taken from a plurality of cameras on the movement path,
Classifying the same object into one cluster based on feature points of the object area each extracted from the plurality of cameras, and
Outputting the image of the object area mapped at the most points on the movement path among the images of the object area within the cluster for each object.
Including,
The classifying step includes receiving the user's selection, and
A movement path-based object mapping method comprising correcting misclassified objects by forming a cluster only with objects corresponding to the user's selection. In paragraph 1:
The output step is
mapping images of the object area within the cluster for each object to the movement path based on the location information of the corresponding camera; and
A movement path-based object mapping method comprising detecting an image of an object area mapped at the most points on the movement path. In paragraph 1:
The classifying step is performed before the user's selection,
Classifying identical objects into one cluster for each camera,
Storing a cluster representative image for each cluster classified by the camera,
Classifying the same object into one cluster for all cameras using the cluster representative image of each cluster for each camera, and
A movement path-based object mapping method further comprising outputting results classified by object for the entire camera. In paragraph 1:
The classifying step further includes outputting the classified results for each cluster as an image. As a movement path-based object mapping device,
An object detector that extracts object areas from multiple frame images each taken from multiple cameras on the movement path,
A movement path mapping unit that obtains movement path information of the object of interest and outputs an image of the object area mapped at the most points on the movement path among the images of the object area extracted from the plurality of cameras, and
an inter-cluster generator that classifies the same object into one cluster for the plurality of cameras and transmits the classification result for each object to the movement path mapping unit; Including,
The inter-cluster generator outputs the classification results for each cluster as an image, receives the user's selection, and configures a cluster with only objects corresponding to the user's selection. A movement path-based object mapping device that corrects misclassified objects. In paragraph 5,
A movement path-based object mapping device further comprising an object recognition feature point extractor that detects the same objects based on feature points of object areas each extracted from the plurality of cameras. In paragraph 6:
The movement path mapping unit maps images of the object area within the cluster for each object to the movement path based on the location information of the corresponding camera, and detects images of the object area mapped at the most points on the movement path. Mapping device. delete In paragraph 6:
In the plurality of frame images, it is determined whether each frame image is a different frame image from the same camera based on the position information of the camera included in each frame image, and detects whether objects detected from the object areas of the different frame images of the same camera are the same object. A path-based object mapping device further comprising a cluster generator.

Images