KR102594694B1 - Method and apparatus of the same person identification among video sequences from multiple cameras, recording medium and device for performing the same - Google Patents

Abstract

The same person recognition method performed by the same person recognition device of the present invention includes receiving a video input from at least one camera; An image sequence acquisition step in which an image sequence is acquired based on the input video; In the image sequence, the image sequence of the reference person for recognizing the same person is a query image sequence, and a preset number of representative images are selected from the query image sequence based on the pose of the image included in the query image sequence. Image selection step; In the image sequence, the image sequence of the person that is the subject of comparison for recognizing the same person is set as the comparison image sequence, and the comparison target pair is selected by matching the comparison image included in the comparison image sequence with the representative image. Selection stage; A pose correction step of correcting the pose of at least one image among the representative image or comparison image in the pair of comparison objects to a transformation target pose through a transformation algorithm; a feature extraction step of extracting features of the comparison image and the representative image that have undergone the pose correction step; and a calculation step of calculating the similarity and distance between the extracted features. As a result, the accuracy of recognizing the same person can be improved by increasing the performance and reliability of person re-detection in a multi-camera system.

Description

Method for recognizing the same person in images taken by multiple cameras and recording media and devices for performing the same

The present invention relates to a method for recognizing the same person in images captured by multiple cameras and a recording medium and device for performing the same. More specifically, the present invention relates to a method for recognizing a person found in a specific camera as the person in an image captured in another camera. It relates to a method for recognizing the same person from images captured by multiple cameras that can confirm recognition, and recording media and devices for performing the same.

In industrial fields such as building disaster prevention camera (CCTV) systems or autonomous driving, when automatically detecting objects, it is necessary to check whether images captured by multiple cameras or people photographed from different perspectives by one camera are the same person. Technology is used to search for the same person in a video. At this time, a method is generally used to extract the area considered to be a person, extract the similarity between photos of the person, extract features using an algorithm or artificial intelligence learning method, and compare them to determine the similarity.

However, it is not easy to identify the same person because the information in the extracted image is very different depending on the person's pose, camera position, and lighting.

In order to solve this problem, an algorithm is being developed that divides the person into each part rather than the whole person and uses a method of comparing each part. Although this method can take into account the characteristics of each part, the overall body proportions and structure, etc. It has the disadvantage of not being able to take into account.

In addition, since the methods that can be obtained from a single image are limited, matching methods using multiple images are also being developed. However, since the number of images extracted from videos is very large, a method to effectively reduce this is needed. However, the currently developed method is subject to comparison. There is still a problem that it is not effective in detecting the same person because it uses a method of independently selecting images without considering the relationship with the video. In particular, in the case of matching methods using multiple images, when calculating the similarity of people included in two images, the parts of the person are detected and compared for each part, so the calculation amount is excessively large, making it difficult to use in practice. there is a problem.

Republic of Korea Patent Publication No. 10-2012-0108319

The present invention was created to solve the above problems, and the purpose of the present invention is to improve the accuracy of recognizing the same person by increasing the performance and reliability of person re-detection in a multi-camera system. The aim is to provide a method for recognizing the same person in an image and a recording medium and device for performing the same.

In addition, another object of the present invention is to provide a method for recognizing the same person in images captured by multiple cameras that can improve search speed over conventional image comparison operations by selecting and using representative images from the video, and a recording medium for performing the same. and providing a device.

In order to achieve the above object, a method of recognizing a same person performed by a same person recognition device according to an embodiment of the present invention includes receiving a video input from at least one camera; An image sequence acquisition step in which an image sequence is acquired based on the input video; In the image sequence, the image sequence of the reference person for recognizing the same person is a query image sequence, and a preset number of representative images are selected from the query image sequence based on the pose of the image included in the query image sequence. Image selection step; In the image sequence, the image sequence of the person that is the subject of comparison for recognizing the same person is set as the comparison image sequence, and the comparison target pair is selected by matching the comparison image included in the comparison image sequence with the representative image. Selection stage; A pose correction step of correcting the pose of at least one image among the representative image or comparison image in the pair of comparison objects to a transformation target pose through a transformation algorithm; a feature extraction step of extracting features of the comparison image and the representative image that have undergone the pose correction step; and a calculation step of calculating the similarity and distance between the extracted features.

Here, the image sequence acquisition step is a step of detecting and tracking a person area in the input video to obtain an image sequence including a bounding box for the person, and the image sequence acquisition step includes each image included in the image sequence. It may also include a pose estimation step of estimating the pose of the image by extracting joints from .

In the representative image selection step, the pose vector of each image included in the query image sequence is found through a clustering algorithm based on the pose estimated through the pose estimation step, clustered, and then the center of gravity of each cluster is found. , among the images included in the query image sequence, the image with the pose closest to the center of gravity of the cluster may be selected as the representative image.

In addition, in the step of selecting a pair of comparison objects, an image having a pose most similar to the pose of the representative image among the comparison image sequences may be matched with the representative image as a comparison image so that the representative image and the comparison image are paired. there is.

In the pose correction step, the pose of one image among the pose of the representative image or the pose of the comparison image forming a pair in the comparison object pair is set as the target transformation pose, and the pose of the other image is set as the target transformation pose. It can also be corrected.

In addition, in the pose correction step, the intermediate pose of the pose of the representative image and the pose of the comparison image forming a pair in the comparison object pair is set as the transformation target pose, and the pose of the representative image and the pose of the comparison image are set as the target pose. It can also be corrected using the modified target pose.

Meanwhile, a computer program for performing the method for recognizing the same person is recorded on a computer-readable recording medium according to an embodiment for realizing another object of the present invention described above.

A same person recognition device according to an embodiment for realizing another object of the present invention described above includes an input unit that receives a moving image from at least one camera; a pre-processing unit that obtains an image sequence based on the input video; In the image sequence, the image sequence of the reference person for recognizing the same person is a query image sequence, and a representative image sequence is selected from the query image sequence as many as a preset number based on the pose of the image included in the query image sequence. image selection unit; In the image sequence, the image sequence of the person that is the object of comparison for recognizing the same person is set as the comparison image sequence, and the comparison object pair is selected by matching the comparison image included in the comparison image sequence with the representative image. government; an image correction unit that corrects the pose of at least one of the representative images or comparison images in the pair of comparison objects to a transformation target pose through a transformation algorithm; an extraction unit that extracts features of the representative image and the comparison image that have undergone pose correction through the pose correction unit; and a calculation unit that calculates the similarity and distance between the extracted features.

Here, the image correction unit sets the pose of one image of the pose of the representative image or the pose of the comparison image that forms a pair in the pair of comparison objects as the deformation target pose, and corrects the pose of the other image to the deformation target pose. You may.

And the image correction unit sets an intermediate pose of the pose of the representative image and the pose of the comparison image that are paired in the pair of comparison objects as the transformation target pose, and modifies the pose of the representative image and the pose of the comparison image. You can also correct it to your target pose.

According to one aspect of the present invention described above, the performance and reliability of person re-detection in a multi-camera system are improved by providing a method for recognizing the same person in images captured by multiple cameras of the present invention and a recording medium and device for performing the same. The accuracy of recognizing the same person can be improved.

Additionally, by selecting and using representative images from the video, search speed can be improved compared to conventional image comparison operations. In particular, it is possible to advance the conventional technology of searching for the same person with one or two representative images by basing it on video tracking rather than matching each image to recognize the same person.

1 is a block diagram illustrating the configuration of a same-person recognition device according to an embodiment of the present invention;
Figure 2 is a diagram illustrating detection of an object in an input video according to an embodiment of the present invention;
3 is a diagram for explaining an image sequence according to an embodiment of the present invention;
Figure 4 is a diagram illustrating estimating the pose of a person from an image included in an image sequence according to an embodiment of the present invention;
Figure 5 is a diagram for explaining the pose change of an image included in an image sequence according to an embodiment of the present invention;
Figure 6 is a diagram illustrating clustering for selecting a representative image from a query image sequence and selecting a representative image according to an embodiment of the present invention;
Figure 7 is a diagram illustrating selecting a comparison target pair by matching a representative image and a comparison image according to an embodiment of the present invention;
Figure 8 illustrates how, according to an embodiment of the present invention, the pose of the representative image is set as the transformation target pose, the pose of the comparison image is matched to the pose of the representative image, the features of the image are extracted, and the similarity and distance of the features are calculated. Drawings to do,
Figure 9 is a flowchart illustrating a method for recognizing the same person according to an embodiment of the present invention, and
Figure 10 is a diagram for explaining the results of recognizing the same person according to the same person recognition method according to an embodiment of the present invention.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein may be implemented in one embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

Figure 1 is a block diagram for explaining the configuration of the same person recognition device 100 according to an embodiment of the present invention, and Figure 2 is a block diagram showing object detection (O1, O2, O3, O4), Figure 3 is a diagram for explaining an image sequence 10 according to an embodiment of the present invention, and Figure 4 is a diagram showing an image sequence (O3, O4) according to an embodiment of the present invention. This is a diagram to explain how to estimate a person's pose (10-1', 10-2') from the images (10-1, 10-2) included in 10).

The same person recognition device 100 according to this embodiment is provided to recognize the same person from images obtained from video captured by at least one camera. To this end, the same person recognition device 100 of the present invention may be provided including a communication unit 110, an input unit 130, a memory 150, an output unit 170, and a processor 190 as shown in FIG. 1. . In addition, the same person recognition method of the present invention can be executed by installing software (application) for performing the same person recognition method, and includes a communication unit 110, an input unit 130, a memory 150, an output unit 170, and a processor. The configuration of 190 may be controlled by software running on the same person recognition device 100.

The communication unit 110 is provided to transmit and receive necessary information from an external device or an external network, and can also receive video captured by at least one camera through the network.

The input unit 130 is an input means for receiving a user command and can receive an image captured by a camera or information about an image that is a reference object or comparison object for a search to find the same person.

The memory 150 records a program for performing the same person recognition method, provides storage space necessary for the operation of the processor 190, temporarily or permanently stores data processed by the processor 190, and provides volatile storage. It may include media or non-volatile storage media, but the scope of the present invention is not limited thereto. Additionally, the memory 150 may store data accumulated while performing the same person recognition method.

And the output unit 170 may include a display to display the process and results of the same person recognition method.

Meanwhile, the processor 190 includes a pre-processing unit 191, a representative image selection unit 192, a comparison object selection unit 193, an image correction unit 194, and an extraction unit 195 to perform the above-described same person recognition method. and a calculation unit 196.

The pre-processing unit 191 may obtain an image sequence based on a video input through at least one camera. Specifically, as shown in FIG. 2, the pre-processing unit 191 directly detects (O1, O2, O3, O4) and tracks a person, that is, an object, in a video captured by at least one camera, or performs external tracking. It is transmitted from the device, and through this, the pre-processing unit 191 can secure the image sequence 10 including the location information about the person, that is, the bounding box, as shown in FIG. 3. When video is input from multiple cameras, the pre-processing unit 191 can secure an image sequence 10 from c cameras. Since such object detection or tracking uses or can be inferred from deep learning-based conventional object detection technology, detailed descriptions thereof will be omitted.

And the pre-processing unit 191 uses the images 10-1 to 10-Tc of each person included in the image sequence 10 obtained from the video to identify landmarks or joints of the body using a 2D or 3D posture prediction technique. can be extracted. This pre-processing unit 191 can extract landmarks or joints by estimating the skeleton of the person image through analysis of each image 10-1 to 10-Tc of the image sequence 10 including the bounding box, The locations of the extracted landmarks or joints may include the center of the face (nose), left and right shoulders, left and right elbows, left and right wrists, left and right pelvic centers, left and right knees, and left and right ankles. Taking the drawing shown in FIG. 4 as an example, the pre-processing unit 191 divides the first image 10-1 of FIG. 4 (a) corresponding to a two-dimensional pose among the images included in the image sequence 10 into 2 The pose estimated based on the 3D pose prediction technique is the same as the first pose estimate (10-1'), and the second image (10-2) of Figure 4 (b), which corresponds to the 3D pose, is used by the 3D pose prediction technique. The pose estimated based on may be the same as the second pose estimate (10-2'). Already generalized technology can be applied to estimate the pose of a person in an image by extracting these landmarks or joints.

And since the image sequence 10 shown in FIG. 3 is based on a video captured by one camera, the images are all shown to have the same size. However, in the case of a video captured by multiple cameras, the size of each person image (10-1 to 10-Tc) included in the image sequence 10 is different depending on the angle of the person, the position between cameras, or resolution. Accordingly, in the present invention, when the pre-processing unit 191 extracts landmarks or joints to estimate the pose of a person, the coordinate values of the landmarks or joints are normalized by dividing them by the size of each image rather than using pixel value units. Alternatively, if the distance between cameras is measured or known in advance, a standard unit such as the meter, which is an actual physical unit, can be used.

In this way, the pre-processing unit 191 obtains the image sequence 10 from the video, extracts landmarks or joints based on each image of the person included in the image sequence 10, and estimates the person's pose. In this embodiment, the pre-processing unit 191 includes an image sequence 10 including a bounding box, and extracts landmarks or joints of each image included in the image sequence 10 to estimate the pose of the person. This can be performed on all videos input through the communication unit 110 or the input unit 130.

FIG. 5 is a diagram illustrating a change in the pose of an image included in the query image sequence Q10 according to an embodiment of the present invention. As shown in FIG. 5, the pose of a person in a video changes over time, and the change in the person's pose can be estimated through the pre-processing unit 191. Accordingly, as shown in FIG. 5, the pose change of the reference person can be estimated from the query image sequence Q10, which is the image sequence of the reference person among the image sequences 10.

Meanwhile, the representative image selection unit 192 is provided to select a preset number of representative images (Q) from the query image sequence (Q10) for the reference person for recognizing the same person in the image sequence (10). Hereinafter, the image sequence corresponding to the reference person for recognizing the same person among the C image sequences 10 obtained from videos shot by C cameras will be defined and explained as the query image sequence Q10, where the reference image sequence Q10 is defined as the query image sequence Q10. Person refers to the person the user is looking for from videos captured by multiple cameras.

Additionally, the query image sequence Q10 used by the representative image selection unit 192 to select the representative image Q may have pose estimation completed through the pre-processing unit 191, as shown in FIG. 5 .

The representative image selection unit 192 is provided by the user when information about a reference person for recognizing the same person, that is, the query image sequence Q10, is separately input or selected from the image sequences 10. Clustering is performed by finding the pose vector of each image included in the query image sequence (Q10) through a clustering algorithm.

Figure 6 shows clustering (101 to 105) and representative images (101-Q to 105-Q) for selecting representative images (101-Q to 105-Q) from the query image sequence (Q10) according to an embodiment of the present invention. This is a drawing to explain how to select .

As shown, the representative image selection unit 192 selects at least one representative image 101 based on the pose of the query image sequence Q10 for which pose estimation has been completed through the pre-processing unit 191 as shown in FIG. 5. -Q to 105-Q) are selected. The number of representative images selected is a system setting variable input by the user, and K representative images are used to compare and match the comparison images (101-S, 102-S, 103-S) of FIG. 7, which will be described later. can be set.

More specifically, when the setting value input through the communication unit 110 or the input unit 130 is 5, the representative image selection unit 192 selects the representative image based on the pose of the query image sequence Q10 as shown in FIG. 6. Five representative images (101-Q to 105-Q) are selected. For this purpose, the representative image selection unit 192 may use the K-mean algorithm as a clustering algorithm.

The representative image selection unit 192 finds the corresponding pose vector using a cluster algorithm based on the pose and view (angle) information of each image included in the query image sequence Q10, and selects each cluster among them ((101 to 105), In other words, find the center of gravity of the cluster.

Additionally, the representative image selection unit 192 may select the pose closest to the center of gravity of the clusters 101 to 105 as the representative pose and select the image of the pose as the representative image 101-Q to 105-Q.

The representative image selection unit 192 according to this embodiment selects the representative images 101-Q to 105-Q to solve the problem of requiring excessive calculation in the conventional image matching method, and is described in detail. Assuming that the images in one minute of video are shot at 30 frames per second, a total of 1,800 images of 30X60 are created, but comparing and matching all of the images requires an excessive amount of calculation. Accordingly, in the present invention, a representative image is selected from among the images, and the selected representative image is compared with the comparative image to analyze whether the person is the same person, so that the same person can be recognized with less calculation than the conventional image comparison method. Let it happen.

Meanwhile, Figure 7 shows a pair of comparison objects ( This is a drawing to explain how to select (101-P to 103-P), and in the drawing, three representative images are selected.The comparison target selection unit 193 is a comparison included in the comparison image sequence (S10). It can be prepared to select a pair of comparison objects by matching the image and the selected representative images (101-Q to 103-Q), where the comparison image sequence (S10) is a comparison object for recognizing the same person in the image sequence (10). This refers to an image sequence of a person, and like the query image sequence Q10, the comparison image sequence S10 may also be an image sequence in which the pose estimation of the person included in the image has been completed through the pre-processing unit 191.

When the representative images (101-Q, 102-Q, 103-Q) are selected through the representative image selection unit 192, the comparison object selection unit 193 selects the representative image (101-Q) selected from the comparison image sequence (S10). , 102-Q, 103-Q), the image with the most similar pose is selected as the comparison image (101-S, 102-S, 103-S).

Thereafter, the comparison object selection unit 193 matches the selected representative images (101-Q, 102-Q, 103-Q) and the selected comparison images (101-S, 102-S, 103-S) to form a pair. The comparison target pair (101-P, 102-P, 103-P) can be selected by forming . Here, the comparison target pair is selected as a number corresponding to the number of representative images (101-Q, 102-Q, 103-Q), and Figure 7 shows three representative images (101-Q, 102-Q, 103-Q). This is an example of a case where it is set to be selected. As shown in FIG. 7, among the multiple images included in the comparison image sequence (S10), the image most similar to the pose of the three representative images (101-Q, 102-Q, 103-Q) is selected as the comparison image (101-Q). S, 102-S, 103-S), and images with the most similar poses can be matched to select a comparison target pair (101-P, 102-P, 103-P).

And in this embodiment, the comparison object selection unit 193 uses the weighted Euclidean distance to select the comparison object pairs (101-P, 102-P, 103-P) to select images with the most similar poses. You can select a pair by matching. In this embodiment, it is assumed that the weighted Euclidean distance, which can be estimated relatively easily and stably, is used when matching two images corresponding to the most similar views and poses, but it is not necessarily limited to this method and can be used in other ways. Of course, the distance between two poses can also be obtained.

Meanwhile, the image correction unit 194 is provided to correct the pose of at least one image among the representative images or comparison images in the comparison target pair to the transformation target pose through a transformation algorithm.

In selecting a comparison target pair, even if the representative image 102-Q matches the comparison image 102-S with the most similar pose and view, as shown in FIG. 7, the two included in the pair 102-P The pose and view of the images 102-Q and 102-S may not exactly match. To this end, in this embodiment, a process of matching the pose 102-S' of the comparison image 102-S and the pose 102-Q' of the representative image 102-Q is performed through the image correction unit 194. It can be done.

Specifically, the image correction unit 194 operates on the pose 102-Q' of the representative image 102-Q included in a pair of comparison objects 102-P and the pose 102 of the comparison image 102-S. -S'), when correcting the pose of at least one image to the transformation target pose, one of two methods can be selected for correction. One method is to set the pose of one of the representative image poses (102-Q') or the comparison image pose (102-S') as the transformation target pose, and only the pose of the other image can be corrected to the transformation target pose. there is. The remaining method sets the intermediate pose between the pose of the representative image (102-Q') and the pose of the comparison image (102-S') as the transformation target pose, and sets the pose of the two images ((102-Q', 102-S) ') can all be corrected to the modified target pose.

Figure 8 shows that according to an embodiment of the present invention, the pose 102-Q' of the representative image 102-Q is set as the transformation target pose, and the pose 102-S' of the comparison image is changed to the pose 102-Q' of the representative image 102-Q. This is a diagram to explain extracting the features of an image by matching them with Q') and calculating the similarity and distance of the features.

If the transformation target pose is set to the pose 102-Q' of the representative image, the image correction unit 194 determines the pose of the comparison image based on the pose estimated through the pre-processing unit 191, as shown in FIG. 8. (102-S') is corrected to be the same as the pose (102-Q') of the reference image, which is the transformation target pose. Through this, the image correction unit 194 can generate a correction image 102-SC corrected so that the comparison image 102-S becomes the same as the transformation target pose.

On the other hand, unlike FIG. 8, if the modified target pose of the present invention is set to an intermediate pose between the pose (102-Q') of the representative image and the pose (102-S') of the comparison image, the image correction unit 194 Both the representative image (102-Q) and the comparison image (102-S) can be corrected to have the same pose as the middle pose to generate a corrected representative image (not shown) and a corrected comparison image (not shown). .

Through this image correction unit 194, the present invention not only reflects the entire structure through pose matching rather than comparing each body part like the conventional DPM (Deformable Part Model), but also allows many calculations like DPM. It has the advantage of not requiring large amounts.

In determining the transformation target pose, the image correction unit 194 may use a non-rigid geometric transformation method using joint information or a neural network method, whichever method provides the same function. It is possible to utilize it. An example of a non-rigid geometric transformation method may be the Thin plate Spline method or the ARAP (As Rigid as possible) method.

Meanwhile, the extractor 195 may extract features of the corrected image 102-SC and the representative image 102-Q, which are comparative images that have undergone pose correction, through the image corrector 194.

As shown in FIG. 8, when the transformation target pose is set as the pose 102-Q' of the representative image and only the pose 102-S' of the comparison image is corrected to generate the corrected image 102-SC, the extraction unit ( 195) extracts the features of the corrected image (102-SC) and the representative image (102-Q). At this time, the features can be extracted according to a convolutional neural network or a defined function.

If the transformation target pose is an intermediate pose, the extractor 195 may extract features of the corrected representative image and the corrected comparison image.

The calculation unit 196 is a function that calculates the distance between the feature values of the representative image 102-Q and the feature values of the corrected image 102-SC, which is a corrected comparison image, based on the features extracted by the extractor 195. Based on , the distance between two feature values can be calculated. Here, the algorithm for calculating the distance between feature values can be obtained by using invers cosine similarity or learning through a neural network.

In addition, the calculation unit 196 can output K search results with the closest similarity through a Rank-K algorithm that selects up to K pairs of the closest comparison objects according to the similarity. Here, the number of K may be a value set according to the user's selection, and the K search results may be output through the output unit 170 so that the user can check them. Then, the user can visually check the results output through the output unit 170 to determine whether they are the same person or not.

Meanwhile, FIG. 9 is a flowchart for explaining a method for recognizing the same person according to an embodiment of the present invention, and FIG. 10 illustrates the results of recognizing the same person according to the method for recognizing the same person according to an embodiment of the present invention. This is a drawing for this purpose.

The same person recognition method of the present invention is provided to search or confirm whether images captured by multiple cameras or people photographed at different angles from one camera are the same person. The same person recognition device of the present invention described above is provided. It is performed by.

First, in the same person recognition method according to the present invention, the same person recognition device 100 receives video from at least one camera (S110).

Then, the same person recognition device 100 can detect and track the person area based on the input video and obtain an image sequence 10 including a bounding box for the person (S120).

Specifically, as shown in FIG. 2, the same person recognition device 100 directly detects (O1, O2, O3, O4) and tracks a person, that is, an object, in an input video or transmits it from an external device. You can receive it. Through this, the same person recognition device 100 can secure location information about the person, that is, an image sequence 10 including a bounding box, as shown in FIG. 3 . Since object detection or tracking uses or can be inferred from deep learning-based conventional object detection technology, detailed descriptions regarding this will be omitted.

At this time, C image sequences (10) are It can be expressed as here is an ordered set of all frame images of image sequence 10 c, means the number of frames of the image sequence 10 c. And c refers to the number of person image sequences (10) being considered, that is, the number of cameras through which the person is photographed, and t refers to the temporal order. At this time, the resolution and size of each c image sequence 10 may be different.

Thereafter, the same person recognition device 100 may estimate the pose of each image by extracting a joint from each image included in the image sequence 10 (S130).

The same person recognition device 100 of the present invention extracts important body landmarks or joints from the images of each person included in the image sequence 10 obtained as described above using a 2D or 3D posture prediction technique. You can. Here, the location of the landmark or joint may include the center of the face (nose), left and right shoulders, left and right elbows, left and right wrists, left and right pelvic centers, left and right knees, and left and right ankles. And the landmarks or joints extracted in this way are It can be displayed as . Additionally, when extracting a landmark or joint, among the images of a person included in the image sequence 10, if the first image 10-1 corresponds to a two-dimensional pose as shown in FIG. 4 (a), The first pose estimate (10-1') can be made by calculating using Equation 2 below, and the second image (10-2) corresponding to the three-dimensional pose as shown in Figure 4 (b) is calculated using the following Equation: The second pose estimate (10-2') can be performed by calculating by 1.

[Equation 1]

[Equation 2]

In the above equation means the three-dimensional or two-dimensional coordinate value or normalized coordinate value of J landmarks or joints of the person image corresponding to the time point t of the image sequence (10) c, and can be expressed in two or three dimensions through the above equation. The extracted joint is as shown in Figure 4. Already generalized technology can be applied to estimate the pose of a person in an image by extracting these landmarks or joints.

And since the image sequence 10 shown in FIG. 3 is based on a video captured by one camera, the images are all shown to have the same size. However, in the case of a video shot with multiple cameras, the size of each person image (10-1 to 10-Tc) included in the image sequence 10 is c image sequence depending on the angle of the person, the position between cameras, or resolution. (10) may be different. Accordingly, this same person recognition method extracts landmarks or joints to estimate the pose of the person, and normalizes the coordinate values of the landmarks or joints by dividing them by the size of each image rather than using pixel value units, or uses camera When measuring the distance between objects or knowing it in advance, you can use a standard unit such as the meter, which is an actual physical unit.

In this way, the image sequence 10 can be obtained from the video, and the pose of the person can be estimated by extracting landmarks or joints based on each image of the person included in the image sequence 10. And in this embodiment, the image sequence 10 including the bounding box is acquired (S120), and landmarks or joints of each image included in the image sequence 10 are extracted to estimate the pose of the person (S130). Can be performed on all videos input through the video input step (S110).

Thereafter, the same person recognition device 100 selects a preset number of representative images 101-Q to 105-Q from the query image sequence Q10 based on the pose of the image included in the query image sequence Q10. (S140). At this time, the image sequence corresponding to the reference person for recognizing the same person among the C image sequences 10 acquired from the video captured by the C cameras will be defined and explained as the query image sequence Q10, where the reference Person refers to the person the user is looking for from videos captured by multiple cameras.

The preset number of representative images selected here is a variable set by the user in the system, and if the setting value set by the user is 5, five representative images (101-Q to 105-Q) can be selected.

And the selected representative images (101-Q to 105-Q) may be images with the largest independent information among the query image sequence (Q10). In the present invention, in the step of estimating the pose of each image (S130) Representative images (101-Q to 105-Q) can be selected based on the estimated pose of the person and the view (angle) of the camera.

Looking at the query image sequence (Q10) for which pose estimation has been completed, as shown in Figure 5, the pose of the person changes over time. Accordingly, in this embodiment, clustering is performed to select a representative image from the query image sequence (Q10), and one of the images included in each cluster (101 to 105) is selected as a representative image (101-Q to 105-Q). The selection is made as shown in Figure 6.

In the step of selecting a representative image (S140), the same person recognition device 100 estimates each image included in the query image sequence (Q10) through a clustering algorithm based on the pose estimated through the step of estimating the pose of each image (S120). Find the pose vectors of the images, perform clustering, and then find the center of gravity of each cluster.

To this end, the same person recognition device 100 can find the center of gravity of each cluster 101 to 105, that is, the cluster, based on Equation 3 below. Here, the K-mean algorithm can be used as the cluster algorithm.

[Equation 3]

In Equation 3, q refers to the query image sequence, means the center of gravity value of cluster k obtained through clustering from the pose vectors of the query image sequence q.

And based on Equation 4 below, the image with the pose closest to the center of gravity of each cluster (101 to 105) found through the clustering algorithm can be selected as the representative image (101-Q to 105-Q).

[Equation 4]

here is the centroid value of cluster k in the query image sequence q. It means the viewpoint of the image with a pose vector representing .

The purpose of selecting representative images (101-Q to 105-Q) in the same person recognition method of the present invention is to solve the problem of requiring an excessive amount of calculation in the conventional image matching method. To explain specifically, assuming that images in a video shot per minute are shot at 30 frames per second, a total of 1,800 images of 30 This means selecting a representative image that is suitable for use, and using the selected representative image as a reference image.

Once the representative images (101-Q to 105-Q) are selected in this way, the same person recognition device 100 selects a comparison target pair by matching the representative image with the comparison image included in the comparison image sequence (S10) ( S150). Here, the comparison image sequence (S10) refers to the image sequence of the person that is the object of comparison for recognizing the same person among C image sequences.

In the comparison target pair selection step (S150), as shown in FIG. 7, images with poses most similar to the poses of representative images (101-Q, 102-Q, 103-Q) selected from the comparison image sequence (S10) are selected as comparison images. You can select (101-S, 102-S, 103-S). Then, the same person recognition device 100 matches the selected comparison images (101-S, 102-S, 103-S) with the representative images (101-Q, 102-Q, 103-Q) to form a pair. (101-P, 102-P, 103P) can be achieved.

And the same person recognition device 100 can select a comparison target pair based on Equation 5 below.

[Equation 5]

In the above equation is the viewpoint of the query image sequence It refers to the viewpoint in the comparison image sequence that is taking the pose most similar to the pose corresponding to, is a set of pairs of comparison images that have the most similar pose to that of the representative image, that is, a set of comparison target pairs.

Here, the distance between the pose of the representative image and the pose of the comparison image can be calculated through various methods of measuring the difference between the pose and the view. In this embodiment, the weighted Euclidean distance is used as in Equation 6 below. Calculate and at this time the weight can be obtained experimentally or determined by user selection.

[Equation 6]

In Equation 6 above, and means the normalized two-dimensional or three-dimensional coordinate value for the jth joint in the pose of the person at any point in the query image sequence q and the comparison image sequence s.

As above, using the weighted Euclidean distance is relatively easy and can be estimated stably, so in this embodiment, it is assumed to use the weighted Euclidean, but it is not limited to this method and can be used in other ways. You can also find the distance between two poses.

Afterwards, the same person recognition device 100 selects at least one of the representative image 102-Q or the comparison image 102-S from the comparison target pair 102-P through a transformation algorithm as shown in FIG. 8. The pose of one image can be corrected to the modified target pose (S160).

After the step of selecting a pair of comparison objects (S150), a correction image is generated by correcting at least one pose of the representative image (102-Q) or the comparison image (102-S) to the transformation target pose through a transformation algorithm. , As described above, even if a pair of comparison objects (102-P) is selected by matching the representative image (102-Q) and the comparison image (102-S) having similar poses and views in S150, the paired representative image (102-P) This step is performed because the pose and view between the comparison image (102-Q) and the comparison image (102-S) do not exactly match.

Conventionally, when matching images, a method of comparing each body part, such as DPM (deformable part model), is used, but in this case, a large amount of calculation is required to execute the algorithm for DPM, and the comparison of each part is performed as a whole. The problem is that it does not reflect the structure.

Accordingly, the present invention may include a step of matching the poses of the representative image and the comparison image to reflect the overall structure without requiring such a large amount of calculation.

In addition, the method of determining the transformation target pose and transforming the pose of the image to be the same as the transformation target pose can be performed using a non-rigid geometric transformation method using joint information or a method using a neural network, and this method performs the same function. Any method can be used. An example of a non-rigid geometric transformation method of a non-neural network method may be the Thin plate Spline method or the ARAP (As Rigid as possible) method. This modified target pose can be generated based on Equation 7 and Equation 8 below.

[Equation 7]

[Equation 8]

here is a representative image according to the transformation algorithm. It is a corrected image of is a comparison image according to the transformation algorithm. This is a corrected image.

And in the above equation is the current pose A representative image or comparison image included in a comparison target pair. Transform the target pose It refers to a transformation algorithm that transforms into, is the time of the query image sequence This means the modified target pose in . and are images after correction, respectively. and This is the pose of the person in .

In the pose correction step (S160) in the present invention, a corrected image can be generated by performing pose correction by selecting one of two methods. Among these two methods, the first method is one of the poses of the representative image (102-Q) or the pose of the comparison image (102-S) forming a pair in the comparison target pair (102-P), as shown in Equation 9 below. This may be a method of generating a corrected image by using the pose of one image as the target transformation pose and correcting the pose of the other image as the transformation target pose.

[Equation 9]

here is the time of the query image sequence The time of the comparison image sequence to be compared with This means the modified target pose in .

8 shows that, as described above, the pose of the representative image 102-Q is set as the target transformation pose, and the pose of the comparison image 102-S is corrected to be the same as the pose of the representative image 102-Q, resulting in a corrected image ( This is a diagram for matching the poses of two images by generating (102-SC). On the contrary, the pose of the comparison image (102-S) is set as the transformation target pose, and the pose of the representative image (102-Q) is corrected to be the same as the pose of the comparison image (102-S), which is the transformation target pose, to generate a corrected image. Of course, the poses of the two images can be matched.

Meanwhile, the remaining method uses Equation 10 below to transform the pose of the representative image (102-Q) that forms a pair in the comparison target pair (102-P) and the intermediate pose of the pose of the comparison image (102-S). pose, and correct both the pose of the representative image (102-Q) and the pose of the comparison image (102-S) to the deformed target pose to generate two corrected images, that is, the corrected representative image and the corrected comparison image. This may be a method of matching the pose of the image.

[Equation 10]

Therefore, the same person recognition device 100 of the present invention generates a modified target pose through the above equations, and based on the modified target pose thus generated, the pose of the comparison image 102-S or the representative image 102-Q is changed. By modifying the pose, the poses match each other.

And the same person recognition device 100 can extract features of the representative image and comparison image that have gone through the pose correction step (S160) (S170). Here, for convenience of explanation, the description is based on the drawing shown in FIG. 8. As shown in the drawing, in the step of extracting features, the features of the representative image 102-Q and the corrected comparison image, that is, the corrected image 102 -SC) features can be extracted.

In order to calculate the similarity between the representative image (102-Q) and the corrected image (102-SC), feature values can be extracted using a convolutional neural network (CNN) method or a conventional classical method. This can be formalized as Equation 11 below: Same as

[Equation 11]

From above is a neural network or defined function for extracting features from images.

In the above, it was explained that the transformation target pose is the pose of the representative image 102-Q, so the features of the corrected image 102-SC, which is a corrected comparison image, and the features of the representative image 102- are extracted. However, it is limited to this. That is not the case.

Specifically, if the transformation target pose is set to an intermediate pose, the features of the representative image corrected to the transformation target pose and the features of the corrected comparison image are extracted.

Once the features are extracted in this way, the same person recognition device 100 can calculate the similarity and distance between the extracted features (S180).

As shown in the figure, the distance between the feature values of the representative image 102-Q and the feature value of the corrected image 102-SC, which is a corrected comparison image, is calculated as Equation 12 or Equation 13 below. Function to calculate distance Based on , the sum of the distances of the comparison target pair (102-P) can be used as the similarity. The algorithm for calculating the distance between feature values can be obtained by using invers cosine similarity or learning through a neural network.

[Equation 12]

[Equation 13]

here is the feature vector value of the representative image at the viewpoint t of the person in the query image sequence, and is the feature vector value of the corrected comparison image corresponding to that time point, is the distance (difference) between these feature vectors. That is, a set of representative images and set of comparison images The distance between images can be defined as the sum of the distances between images that match each other and form a pair, or the smallest distance among these distances.

The results of the similarity calculated in this way are output through an algorithm that selects the closest Rank-K, that is, the K closest comparison target pairs according to the similarity, and outputs the K search results with the closest similarity so that the user can check them. Allows you to determine whether it is the same person. Here, the number of K may vary depending on the user's selection.

Meanwhile, Figure 10 is a diagram for explaining the results of recognizing the same person according to the same person recognition method according to an embodiment of the present invention. The image sequence shown in Figure 10 (a) is a query image sequence (Q20, Q30, Q40) is an image sequence of a reference person for recognizing the same person, and the image sequence shown in Figure 10 (b) is the output result.

As shown, if the query image sequence (Q20, Q30, Q40), which is a reference image sequence of a reference person for recognizing the same person, is selected or input among the image sequences acquired based on videos shot from a plurality of cameras, the same person is recognized. If the device 100 is set to perform the same person recognition method to calculate the similarity and output 10 image sequences with the highest similarity, 10 results can be output as shown. Then, the user can check the output result and determine with the naked eye whether or not the video contains the same person as the reference person included in the representative image sequence.

In the drawing, the person image sequence shown in red is an image determined to be a different person from the person being searched, and the person image sequence shown in green is an image recognized as the same person. In this way, instead of comparing the conventional person images as a whole or dividing them into parts, the images are matched by pose according to the same person recognition method of the present invention and then compared, so that the view of the person, that is, the same person even if the shooting angle and pose are different, is shown. It can be determined that it is a person. As a result, it is possible to improve the performance and reliability of re-detection of people in a multiple disaster prevention camera system, and of course, increase the search speed by selecting representative images from the video.

The same person recognition method of the present invention can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination.

The program instructions recorded on the computer-readable recording medium may not only be designed and configured specifically for the present invention, but may also be known and usable by those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc.

Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

Although various embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and may be used in the technical field to which the invention pertains without departing from the gist of the invention as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

100: Same person recognition device 110: Communication department
130: input unit 150: memory
170: output unit 190: processor
191: Pre-processing unit 192: Representative image selection unit
193: Comparison target selection unit 194: Image correction unit
195: extraction unit 196: calculation unit

Claims (10)

In the same person recognition method performed by the same person recognition device,
Receiving video input from at least one camera;
An image sequence acquisition step in which an image sequence is acquired based on the input video;
In the image sequence, the image sequence of the reference person for recognizing the same person is a query image sequence, and a preset number of representative images are selected from the query image sequence based on the pose of the image included in the query image sequence. Image selection step;
In the image sequence, the image sequence of the person that is the subject of comparison for recognizing the same person is set as the comparison image sequence, and the comparison target pair is selected by matching the comparison image included in the comparison image sequence with the representative image. Selection stage;
A pose correction step of correcting the pose of at least one image among the representative image or comparison image in the pair of comparison objects to a transformation target pose through a transformation algorithm;
a feature extraction step of extracting features of the comparison image and the representative image that have undergone the pose correction step; and
It includes a calculation step of calculating the similarity and distance between the extracted features,
The image sequence acquisition step is,
A step of detecting and tracking a person's area in the input video to obtain an image sequence including a bounding box for the person.
The image sequence acquisition step is,
A pose estimation step of estimating the pose of the image by extracting a joint from each image included in the image sequence,
The representative image selection step is,
Based on the pose estimated through the pose estimation step, find the pose vector of each image included in the query image sequence through a clustering algorithm, perform clustering, find the center of gravity of each cluster, and find the center of gravity of each cluster. Among the images, the image with the pose closest to the center of gravity of the cluster is selected as the representative image,
The representative image selected in the representative image selection step is,
A method for recognizing the same person, characterized in that the image is selected for each pose of the reference person that changes over time from the query image sequence for which the pose estimation step has been completed.
delete delete According to paragraph 1,
The step of selecting the comparison target pair is,
A method for recognizing the same person, characterized in that, among the comparison image sequences, an image having a pose most similar to the pose of the representative image is matched with the representative image as a comparison image so that the representative image and the comparison image are paired.
According to paragraph 1,
In the pose correction step,
The same, characterized in that the pose of one of the poses of the representative image or the pose of the comparison image forming a pair in the pair of comparison objects is set as the transformation target pose, and the pose of the other image is corrected to the transformation target pose. Person recognition method.
According to paragraph 1,
In the pose correction step,
Setting an intermediate pose of the pose of the representative image and the pose of the comparison image that forms a pair in the pair of comparison objects as the transformation target pose, and correcting the pose of the representative image and the pose of the comparison image to the transformation target pose. Characteristics of the same person recognition method.
A computer-readable recording medium on which a computer program is recorded for performing the same person recognition method according to claim 1.
An input unit that receives video from at least one camera;
a pre-processing unit that obtains an image sequence based on the input video;
In the image sequence, the image sequence of the reference person for recognizing the same person is a query image sequence, and a representative image sequence is selected from the query image sequence as many as a preset number based on the pose of the image included in the query image sequence. image selection unit;
In the image sequence, the image sequence of the person that is the object of comparison for recognizing the same person is set as the comparison image sequence, and the comparison object pair is selected by matching the comparison image included in the comparison image sequence with the representative image. government;
an image correction unit that corrects the pose of at least one of the representative images or comparison images in the pair of comparison objects to a transformation target pose through a transformation algorithm;
an extraction unit that extracts features of the comparison image and the representative comparison image that have undergone pose correction through the image correction unit; and
It includes a calculation unit that calculates the similarity and distance between the extracted features,
The pre-processing unit,
Obtaining an image sequence including a bounding box for the person by detecting and tracking the person area in the input video,
Extract joints from each image included in the image sequence to estimate the pose of the image,
The representative image selection section,
Based on the estimated pose, find the pose vector of each image included in the query image sequence through a clustering algorithm, perform clustering, find the center of gravity of each cluster, and find the center of gravity of each cluster. Select the image with the closest pose to the center of gravity as the representative image,
The representative image selected by the representative image selection unit is,
A same-person recognition device that is an image selected for each pose of the reference person that changes over time from the query image sequence for which the pose estimation has been completed. According to clause 8,
The image correction unit,
The same, characterized in that the pose of one of the poses of the representative image or the pose of the comparison image forming a pair in the pair of comparison objects is set as the transformation target pose, and the pose of the other image is corrected to the transformation target pose. Person recognition device.
According to clause 8,
The image correction unit,
Setting an intermediate pose of the pose of the representative image and the pose of the comparison image that forms a pair in the pair of comparison objects as the transformation target pose, and correcting the pose of the representative image and the pose of the comparison image to the transformation target pose. Characterized by a same-person recognition device.