KR20220104426A - Apparatus and Method for Person Re-Identification based on Heterogeneous Sensor Camera - Google Patents

Abstract

The present invention may provide an apparatus and method for person re-identification based on a heterogeneous sensor camera. The apparatus can perform learning based on a feature map extracted from images acquired from the heterogeneous sensor camera and a warping feature map that warps pixels corresponding to each other in the feature map and, especially, calculate dense triple loss for the feature map and the warping feature map to be reflected on learning, so it is possible to extract subdivided local information even from non-sorted images and re-identify a person even in images acquired in very different conditions in various cameras.

Description

Apparatus and Method for Person Re-Identification based on Heterogeneous Sensor Camera

The present invention relates to an apparatus and method for re-identifying a person, and to an apparatus and method for re-identifying a person based on a heterogeneous sensor camera.

Research on a Person Re-identification (reID) technology that can search for the same person photographed in different environments is being actively conducted.

1 shows the concept of human re-identification, and FIG. 2 shows an example of a person tracking technology as one of the fields using human re-identification.

1, person re-identification is a technique for identifying and detecting the same person photographed under different conditions, and various environmental conditions such as change of posture, change of background, change of lighting, change of shooting distance and angle It aims to accurately detect an image in which the same person is included even if this changes.

As shown in FIG. 2 , this person re-identification technology may be used in various fields of searching and tracking a specific person in a plurality of images, such as searching for missing persons or tracking criminals.

This person re-identification technology extracts a feature representing an identity rather than a specific part of a person to identify an individual using an artificial neural network mainly from an image, and allows the same person to be identified based on the extracted feature. That is, it is not limited to a person's face shape, specific posture, background, shooting angle, lighting, etc., but by extracting features representing each individual identity and re-identifying the person, the same person can be detected from a plurality of different images. However, the existing human re-identification was mainly focused on re-identifying the same person in images acquired from the same type of sensor camera.

3 shows an example of an image acquired by a heterogeneous sensor camera, and FIG. 4 shows a concept of human re-identification based on a heterogeneous sensor camera.

3 and the top three images, the left two images are images acquired during the day and night by a general visible camera (RGB camera or Visible camera), respectively, and the right image is an image acquired at night by an infrared (Infrared: IR) camera indicates

The middle left image represents a human region extracted from an image acquired by a visible camera during the day, and a plurality of right images represents an image to be compared for re-identification. And the lower right image represents the human region extracted from the image acquired by the IR camera at night, and the plurality of images on the left represents the image to be compared for re-identification.

As shown in FIG. 3 , when the visible camera acquires an image during the day, it may acquire an image of a level capable of identifying a person, but at night, it is not distinguished from the surrounding environment, so that it is impossible to identify a person. In contrast, when an IR camera using an IR sensor different from a visible camera is used, it is possible to accurately identify a person at night.

Therefore, as shown in FIG. 4 , in order to identify and track a person regardless of environmental changes, it is necessary to re-identify a person from images acquired by different heterogeneous sensor cameras.

5 is a diagram for explaining a misalignment problem in human re-identification.

In FIG. 5 , images located in the same row at the top and bottom are images of the same object, that is, the same person. As shown in FIG. 5 , even in images acquired with the same visible camera, a person may appear in very different forms depending on various conditions, and this may make it difficult to identify the same person in each image. However, in the case of an image obtained from the same visible camera, a person may be re-identified based on a characteristic according to a color distribution or the like.

However, unlike a visible camera, an IR camera does not include color information, so it is not easy to compare the characteristics of the color image acquired by the visible camera and the characteristics of the IR image acquired by the IR camera. That is, features that can be compared with each other for re-identification are limited. In order to overcome such a difference in modality, conventionally, both RGB and IR images are embedded in a virtual feature space (common feature space), and features embedded in the feature space are extracted and compared at the image or part level. perform learning with However, in such a method of embedding in the common feature space, a problem may occur due to object misalignment due to a dense matching error for feature points.

In the past, when people were re-identified from images acquired from different heterogeneous sensor cameras, it was assumed that the people included in each image were aligned with each other and focused on learning features at the image or part level. That is, learning was performed assuming that the two images were aligned with each other at a certain level or more, even if they were rough. For example, an image is divided into a predetermined horizontal grid, and body parts (eg, shoulder, knee, etc.) included in each grid in the two images are learned assuming that they are aligned to correspond to each other.

However, as shown in FIG. 5, even if the image is of the same object, the position, size, pose, etc. of the object in the image are often very different. Since it is not performed normally, a problem in which a human re-identification error occurs may occur. As described above, such a misalignment problem may be particularly problematic in object re-identification of an image between two modalities having different characteristics.

In addition, in the method of embedding in the common feature space, there is a tendency to overlook detailed local information that can greatly help object re-identification, so there is a limit to the re-identification performance.

Korean Patent Publication No. 10-2019-0078270 (published on July 4, 2019)

It is an object of the present invention to provide an apparatus and method for re-identifying a person capable of accurately re-identifying a person from an image obtained by a heterogeneous sensor camera.

Another object of the present invention is to provide an apparatus and method for re-identifying a person that can accurately re-identify a person even in an unaligned image by using dense matching between cross-modality object images.

A person re-identification apparatus according to an embodiment of the present invention for achieving the above object extracts features according to a learning method by receiving the first and second images obtained using different heterogeneous sensors and learning the first and second images. a feature pooling unit that obtains a second feature map and pools them in a predetermined manner to obtain first and second presenters; It is provided during learning, receives the first and second feature maps to estimate pixels corresponding to each other, and performs mutual warping based on the estimated positions of the corresponding pixels, so that the second feature map becomes the first feature A first warping feature map warped to correspond to the map and a second warping feature map warped so that the first feature map corresponds to the second feature map are obtained, and the first and second warping feature maps are pooled to obtain a first and a cross-modal aligner for obtaining a second warping presenter; first and second scoremaps representing probabilities that the first and second descriptors and each of the first and second warping descriptors respectively correspond to a plurality of predefined classes, respectively, according to the manner in which they are learned; a re-identification unit for obtaining a warping scoremap and obtaining an identifier of a class having the highest probability from the obtained first and second scoremaps; and a positive identifier different from the positive in case the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map is the same identifier, provided at the time of learning. It includes a loss calculator for calculating the dense triple loss by dividing the case into negatives.

The cross-modal alignment unit may include: a similarity measurer configured to obtain a similarity map by calculating a similarity between pixels between the first and second feature maps according to a predetermined method; a matching probability estimator for obtaining a matching probability map by calculating a matching probability for each pixel of the first and second feature maps based on the similarity map; The first and second warping feature maps are obtained by weighting the probability that pixels of each of the first and second feature maps are matched with each pixel of the relative feature map using the matching probability map and warping the relative feature map. a soft warping unit; and a warping pooling unit configured to pool the first and second warping feature maps in a predetermined manner to obtain first and second warping presenters.

The cross modal aligner may further include a mask providing unit configured to obtain first and second masks by normalizing sizes of pixels for each position of the first and second feature maps.

The soft warping unit receives the first and second masks, weights and aggregates the probability that the pixels of each of the first and second feature maps match each pixel of the relative feature map, and adds the mask to the weighted aggregated warping pixel value. You can warp by additionally weighting the pixel value.

The loss calculator acquires first and second common attention maps that emphasize human regions that can be identified in both the first and second images, and includes an identifier of the first feature map that is an anchor in the first common attention map and The difference between the positive of the first warping feature map estimated with the same identifier and the negative of the first warping feature map estimated with a different identifier is the same as the identifier of the second feature map that is the anchor in the second common attention map. The dense triple loss may be calculated by accumulating differences between positives estimated as identifiers and negatives as second warping feature maps estimated as different identifiers.

The loss calculator includes an identification loss according to a difference between each of the first and second scoremaps and an identifier applied by labeling the first and second images during training, and the first and second scoremaps and the first and further calculating a consistency loss according to a difference between the second warping scoremaps, and performing learning by backpropagating a total loss calculated by weighting the identification loss, the consistency loss, and the dense triple loss.

The loss calculator obtains the identification loss by calculating a negative cross entropy between the first and second scoremaps and the identifiers labeled and applied to the first and second images, and the first and second scoremaps The coherence loss may be calculated by calculating a negative cross entropy between each and a corresponding one of the first and second warping scoremaps.

Person re-identification method according to another embodiment of the present invention for achieving the above object is a learning step; and a person re-identification step of using an artificial neural network to obtain an identifier for a person included in each image by receiving images obtained using different heterogeneous sensors, wherein the learning step is different from each other as learning data. First and second images obtained using heterogeneous sensors and previously labeled with identifiers are obtained by extracting features according to a learning method, obtaining first and second feature maps, and pooling the first and second images in a predetermined manner and obtaining a second presenter; The first and second feature maps are applied, pixels corresponding to each other are estimated, and mutual warping is performed based on the estimated positions of the corresponding pixels, so that the second feature map is warped to correspond to the first feature map. A first warping feature map and a second warping feature map warped so that the first feature map corresponds to the second feature map are obtained, and first and second warping presenters are generated by pooling the first and second warping feature maps. obtaining; obtaining first and second scoremaps and first and second warping scoremaps representing probabilities of the first and second presenters and each of the first and second warping presenters corresponding to each of a plurality of predefined classes and obtaining the identifier of the class with the highest probability from the obtained first and second scoremaps; and dividing the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map into a positive in the case of the same identifier and a negative in the case of a different identifier. calculating the loss.

Therefore, the person re-identification apparatus and method according to an embodiment of the present invention perform learning based on a warping feature map obtained by warping pixels corresponding to each other in a feature map together with a feature map extracted from images obtained from heterogeneous sensor cameras. In particular, by calculating the dense triple loss for the feature map and the warping feature map and reflecting it in learning, segmented local information can be extracted even from unaligned images, so it is possible to extract human information from images acquired under very different conditions from various cameras. can be accurately re-identified.

1 shows the concept of human re-identification.
2 shows an example of a person tracking technology as one of the fields using human re-identification.
3 shows an example of an image obtained by a heterogeneous sensor camera.
4 shows a heterogeneous sensor camera-based human re-identification concept.
5 is a diagram for explaining a misalignment problem in human re-identification.
6 shows a schematic structure of a heterogeneous sensor camera-based human re-identification apparatus according to an embodiment of the present invention.
7 is a view for explaining the operation of the person re-identification apparatus of FIG.
8 is a diagram for explaining a common attention map.
9 shows the performance change according to the loss setting of the human re-identification apparatus.
10 shows a simulation result of the performance of the human re-identification apparatus according to the present embodiment.
11 illustrates a method for re-identifying a person based on a heterogeneous sensor camera according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in several different forms, and is not limited to the described embodiments. In addition, in order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it does not exclude other components unless otherwise stated, meaning that other components may be further included. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

6 shows a schematic structure of a heterogeneous sensor camera-based human re-identification apparatus according to an embodiment of the present invention, FIG. 7 is a diagram for explaining the operation of the person re-identification apparatus of FIG. 6, and FIG. 8 is a common attention map It is a drawing for explaining.

6 and 7, the heterogeneous sensor camera-based human re-identification apparatus of this embodiment includes a heterogeneous image acquisition unit 100, a feature extraction unit 200, a pooling unit 300, a re-identification unit 400, and an intersection It may include a modal alignment unit 500 and a loss calculation unit 600 .

The heterogeneous image acquisition unit 100 acquires heterogeneous images obtained from different sensor cameras. The heterogeneous image acquisition unit 100 may acquire a first image and a second image, wherein, for example, the first image is an RGB image acquired using a visible camera, and the second image is an IR camera acquired using an IR camera. It is assumed that it is an IR image and will be described. The heterogeneous image obtaining unit 100 may include a first image obtaining unit 110 obtaining a first image and a second image obtaining unit 120 obtaining a second image. However, the apparatus for re-identifying a person based on a heterogeneous sensor camera of the present embodiment may acquire an image using a sensor camera other than a visible camera or an IR camera. In this case, each of the first image and the second image may be an image of a single frame, but may be an image composed of multiple frames, and may be an image in which the same person is included or a different person is included.

The first image acquisition unit 110 and the second image acquisition unit 120 are implemented as heterogeneous sensor cameras including different sensors, respectively, or a storage device for storing images acquired using the heterogeneous sensor cameras or from the outside. It may be implemented as a communication module that receives an image obtained by using a sensor camera, or the like.

The feature extraction unit 200 obtains a feature map by extracting features from each of the heterogeneous images obtained by the heterogeneous image acquisition unit 100 . The feature extractor 200 may include a first feature extractor 210 and a second feature extractor 220 each implemented as an artificial neural network such as a convolutional neural network (CNN). The first feature extraction unit 210 receives the first image obtained from the first image acquisition unit 110 and extracts the features according to a pre-learned method to obtain a first feature map f _RGB , and a second The feature extracting unit 220 receives the second image obtained from the second image obtaining unit 120 and extracts the features according to a pre-learned method to obtain a second feature map f _IR .

)과 제2 특징맵(f_IR ∈

)은 동일한 높이(h)와 폭(w) 및 채널(d)을 갖는 h × w × d 크기로 추출될 수 있다.At this time, the first feature map (f _RGB ∈

) and the second feature map (f _IR ∈

) can be extracted in the size h × w × d with the same height (h), width (w), and channel (d).

The pooling unit 300 receives the first and second feature maps f _RGB and f _IR obtained from the feature extraction unit 200 and pulls them in a predetermined manner, respectively, to obtain first and second presenters. The pulling unit 300 may include a first pulling unit 310 and a second pulling unit 320 . The first pooling unit 310 receives and pulls the first feature map f _{RGB to obtain a first presenter ø(f RGB )} , and the second pooling unit 320 receives the second feature map f _IR ) may be applied and pooled to obtain a second descriptor ø(f _IR ). Here, each of the first and second descriptors (ø(f _RGB ), ø(f _IR )) has a one-dimensional vector form corresponding to the length of the channel (d), and a feature that enables identification of a person included in the image can be viewed as an expression.

Here, the feature extracting unit 200 and the pulling unit 300 may be integrated into the feature pooling unit.

The re-identification unit 400 receives the first and second descriptors (ø(f _RGB ), ø(f _IR )), which are characteristic descriptors of a person included in the image, and receives the person included in the first and second images. to re-identify For example, the re-identification unit 400 measures the similarity between the first and second descriptors ø(f _RGB ) and ø(f _IR ) to determine whether the person included in the first and second images is the same person. can be identified. However, the re-identification unit 400 identifies the class corresponding to the first and second presenters (ø(f _RGB ), ø(f _IR )) among a plurality of classes set to correspond to a plurality of people designated in advance to obtain an identifier. may be configured to obtain. That is, the re-identification unit 400 may obtain an identifier of a person included in each of the heterogeneous images to determine whether they are the same person.

Here, the re-identification unit 400 is implemented as an artificial neural network including, for example, an activation function such as soft max and a fully connected layer, the first and second presenters ø(f _RGB ), An identifier may be selected by obtaining a scoremap in which each ø(f _IR )) represents a probability corresponding to each of a plurality of predetermined classes, and identifying a class with the highest probability in the obtained scoremap.

In FIG. 6 , the feature extracting unit 200, the pulling unit 300, and the re-identification unit 400 are illustrated as separate components for convenience of explanation, but the feature extracting unit 200, the pulling unit 300 and the re-identification unit are illustrated. The unit 400 may be composed of layers that perform different functions in the re-identification artificial neural network for re-identifying a person.

The apparatus for re-identifying a person may be composed of only the heterogeneous image acquisition unit 100 , the feature extraction unit 200 , the pooling unit 300 , and the re-identification unit 400 in actual application. However, in the human re-identification apparatus, the feature extracting unit 200, the pooling unit 300, and the re-identification unit 400, etc. composed of an artificial neural network, must be learned before being actually applied. Accordingly, the apparatus for re-identifying a person of the present embodiment may further include a cross-modal alignment unit 500 and a loss calculation unit 600 while learning is performed, and may be excluded when learning is completed.

,

)을 획득한다. 여기서 제1 와핑 특징맵(

)은 제2 이미지에서 추출된 제2 특징맵(f_IR)을 와핑하여 획득되는 특징맵이고, 제2 와핑 특징맵(

)은 제1 이미지에서 추출된 제1 특징맵(f_RGB)을 와핑하여 획득되는 특징맵이다. 즉 제1 및 제2 와핑 특징맵(

,

)은 서로 상대의 이종 이미지로부터 추출된 특징맵을 와핑하여 모달리티를 교차하여 획득된다.The cross-modal alignment unit 500 receives the first and second feature maps f _RGB , f _IR extracted from different heterogeneous images, and receives the applied first feature map f _RGB and the second feature map f The pixels corresponding to each other are estimated based on the inter-pixel similarity of _IR ), and mutual warping is performed based on the estimated positions of the corresponding pixels, so that the first and second warping feature maps (

,

) is obtained. Here, the first warping feature map (

) is a feature map obtained by warping the second feature map (f _IR ) extracted from the second image, and the second warping feature map (

) is a feature map obtained by warping the first feature map f _RGB extracted from the first image. That is, the first and second warping feature maps (

,

) is obtained by intersecting the modality by warping the feature maps extracted from heterogeneous images of each other.

Specifically, the cross-modal alignment unit 500 may include a similarity measuring unit 510 , a matching probability estimation unit 520 , a mask providing unit 530 , a soft warping unit 540 , and a warping pooling unit 550 . can

The similarity measuring unit 510 receives the first and second feature maps f _RGB , f _IR , and calculates the similarity between pixels between the applied first and second feature maps f _RGB , f _IR to obtain similarity. Obtain a map C(p,q). The similarity measuring unit 510 is configured to include a pixel vector and a second feature map f _IR , which are one-dimensional vectors having a channel direction length d for each position p on the h × w plane of the first feature map f _RGB . A similarity map (C(p,q) by calculating the similarity between pixel vectors, which is a one-dimensional vector having a channel direction length (d) for each position (q), based on the cosine similarity as in Equation 1 on the h × w plane of )) can be obtained.

Here, f _RGB (p) denotes a pixel vector for each position (p) of the first feature map (f _RGB ), f _IR (q) denotes a pixel vector for each position (q) of the second feature map (f _IR ), , |||| ₂ represents the L2 norm function, and T represents the transpose matrix.

The matching probability estimation unit 520 receives the similarity map C(p, q) obtained from the similarity measurement unit 510 and calculates a matching probability for each pixel. The matching probability estimator 520 is configured to calculate the first feature map f _RGB and the second feature map f based on the similarity between each pixel vector of the first feature map f _RGB and the second feature map f _IR . A matching probability map (P(p, q)) having a size of h × w × h × w indicating the matching probability for each pixel position of _IR ) is obtained by calculating according to Equation 2;

where β is the temperature parameter.

The mask providing unit 530 receives the first and second feature maps f _RGB , f _IR , and receives a first mask M _RGB and a second feature map f corresponding to the first feature map f _RGB . A second mask M _IR corresponding to _IR is obtained and output to the soft warping unit 540 .

As learning is performed, the mask providing unit 530 performs the first image without a label on the real human area on the assumption that the area corresponding to the person, that is, the foreground, is more prominent than the other areas in the first and second images as learning is performed. A first mask M _RGB in which the human region is emphasized may be obtained from the first feature map f _RGB obtained in Equation (3).

Here, f is the maximum-minimum normalization function defined by Equation (4).

Similarly, the mask providing unit 530 may also obtain a second mask M _IR corresponding to the human region from the second feature map f _IR obtained from the second image.

That is, the mask providing unit 530 obtains a mask by normalizing the size of the pixel vector for each position of the first and second feature maps f _RGB and f _IR . If the mask providing unit 530 uses a binary mask, it is effective when the foreground and the background can be clearly distinguished, but a large error can be caused when the mask providing unit 530 is not clearly distinguished. In order to prevent such an error from occurring, the mask providing unit 530 of the present embodiment normalizes the size of the pixel vector for each position of the first and second feature maps f _RGB , f _IR so that the foreground region becomes the background region. The first and second masks M _RGB and M _IR that are emphasized compared to are obtained.

,

)을 획득한다.On the other hand, the soft warping unit 540 compares each pixel of the first and second feature maps f _RGB , f _IR to each other based on the pixel-to-pixel matching probability map P(p,q) obtained according to Equation 2 The first and second warping feature maps (

,

) is obtained.

In this case, the soft warping unit 540 uses the first and second masks M _RGB , M _IR provided from the mask providing unit 530 , and uses the first and second feature maps f _RGB , f _IR . Warping may be intensively performed on the foreground area corresponding to .

)을 획득할 수 있다.The soft warping unit 540 is configured to _warp each pixel of the second feature map f _IR by highlighting the foreground area corresponding to a person in the first feature map f _RGB according to Equation 5. ) by performing soft warping using the first warping feature map (

) can be obtained.

Here, W is a soft warping function defined by Equation (6).

)을 획득한다.According to Equations 5 and 6, the soft warping unit 540 weights and aggregates the pixel vectors for each position q of the second feature map f _IR according to the matching probability by using the soft warping function W to calculate the second 1 warps to the corresponding pixel position (p) of the feature map (f _RGB ), wherein the matching probability weighted aggregated pixel vector of the second feature map (f _IR ) in the human region specified by the first mask (M _RGB ) is It is emphasized and warped, and in the remaining area, the pixel vector of the first feature map f _RGB is maintained as much as possible. That is, in the first feature map f _RGB , the foreground region corresponding to a person is weighted and aggregated according to the matching probability in the second feature map f _IR , and the warped pixel vector is emphasized, while the remaining background region is the first feature map. The pixel vector of (f _RGB ) is emphasized so that the first warping feature map (

) is obtained.

)을 획득한다.In addition, the soft warping unit 540 uses the soft warping function and the second mask M _IR , similarly to Equations 5 and 6 , so that the foreground area corresponding to the person in the second feature map f _IR is the first feature. The warped pixel vector is highlighted in the map f _RGB , and the pixel vector of the second feature map f _IR is emphasized in the remaining background area, so that the second warped feature map (

) is obtained.

The reason that the cross-modal aligner 500 uses the soft warping function W in this embodiment is that when using the argmax function that is generally used for pixel-to-pixel matching in different images, pixel-to-pixel matching is performed very hard. This is because, while it is possible to set an explicit corresponding pixel, a case in which the corresponding points are not densely distributed or the corresponding points do not exist may frequently occur due to the texture or occlusion of different images. In particular, in the case of detecting a corresponding pixel between the first and second feature maps (f _RGB , f _IR ) extracted from heterogeneous images as in the present embodiment, the image quality is more important than detailed features such as color or texture of the image. More attention should be paid to the rough shape change. Accordingly, in the present embodiment, warping is performed to positions of pixels corresponding to each other in heterogeneous images using a soft warping function (W) rather than an argmax function.

)과 제2 와핑 특징맵(

) 각각을 풀링하여 제1 와핑 표현자(ø(

))과 제2 와핑 표현자(ø(

))를 획득한다.The warping pooling unit 550 includes a first warping feature map (

) and the second warping feature map (

) by pooling each of the first warping descriptors (ø(

)) and the second warping descriptor (ø(

)) is obtained.

), ø(

))을 인가받아 제1 및 제2 표현자(ø(f_RGB), ø(f_IR))에 대한 식별자를 추출할 수 있다.And the re-identification unit 400 is similar to the first and second descriptors ø(f _RGB ), ø(f _IR ), the first and second warping descriptors ø(

), ø(

)), identifiers for the first and second descriptors ø(f _RGB ), ø(f _IR ) may be extracted.

The loss calculation unit 600 calculates a loss occurring in the learning process in a predetermined manner, and backpropagates the calculated loss, so that the feature extraction unit 200, the pooling unit 300, the re-identification unit 400 and the intersection The modal alignment unit 500 is trained.

In this embodiment, the loss calculator 600 calculates an identification loss (L _ID ) and a consistency loss (L _IC ) and a dense triplet loss (L _DT ), respectively, and , learning can be performed by backpropagating the total loss (L) computed by combining the computed identification loss (L _ID ) with the coherence loss (L _IC ) and dense triplet loss (L _DT ).

The loss calculator 600 may first obtain an identification loss L _ID based on the first and second descriptors ø(f _RGB ) and ø(f _IR ). Here, the identification loss (L _ID ) is the first and second scoremaps obtained from the first and second presenters (ø(f _RGB ), ø(f _IR )) and the first and second images input during training. It may be calculated and obtained based on the difference between the identifiers for the persons pre-labeled in . That is, the apparatus for re-identifying a person according to the present embodiment may receive a heterogeneous image with a pre-labeled identifier for a person as training data during learning, so that learning may be performed in a supervised learning method.

The loss calculator 600 may calculate the identification loss L _ID by calculating a negative cross-entropy between the first and second scoremaps and the identifier applied as training data. Here, a method of calculating an identification loss (L _ID ) based on negative cross entropy is a known technique and will not be described in detail here.

On the other hand, the loss calculator 600 determines that if the foreground regions of the first and second feature maps f _RGB , f _IR obtained from heterogeneous images by the cross modal aligner 500 are normally warped, the first and second A loss of consistency (L _IC ) that the identifier extracted from the scoremap and the identifier extracted from the first and second warping scoremaps must be the same is calculated.

In this case, the loss calculator 600 determines that the identifiers finally extracted from the first and second scoremaps by learning are the same as the identifiers labeled in the training data, and thus the identification loss (L _ID ) from the first and second scoremaps. ), the consistency loss (L _IC ) may be calculated based on the first and second scoremaps and the identifiers labeled in the training data.

However, since the loss of consistency (L _IC ) represents a difference between the identifier extracted from the first and second scoremaps and the identifier extracted from the first and second warping scoremaps rather than the identifier for the actual image, the loss calculator 600 ) may compute the coherence loss (L _IC ) as the negative cross entropy between the first and second scoremaps and the first and second warping scoremaps.

,

)을 인가받고, 이중 서로 대응하는 특징맵과 와핑 특징맵을 기반으로 제1 및 제2 공통 주의맵(co-attention map)을 획득하고, 획득된 제1 및 제2 공통 주의맵을 이용하여 밀집 삼중 손실(L_DT)을 계산한다.Meanwhile, the first and second feature maps f _RGB , f _IR obtained by the feature extraction unit 200 and the first and second warping feature maps obtained from the soft warping unit 540 (

,

), acquire first and second co-attention maps based on the feature maps and warping feature maps corresponding to each other, and cluster using the obtained first and second common attention maps Calculate the triple loss (L _DT ).

,

) 사이의 오차를 계산하는 가장 직접적인 방식은 L₂ 놈 함수를 적용하여 대응하는 픽셀간 L₂ 거리를 계산하는 것이다. 그러나, 이는 폐색 영역 등을 고려하지 않는다는 한계가 있다. 이에 본 실시예에서 손실 계산부(600)는 이종의 이미지 양측에서 모두 확인 가능한 사람 영역을 강조하는 공통 주의맵을 획득하고, 공통 주의맵 상에서 특징맵과 와핑 특징맵 사이의 오차를 계산하여 밀집 삼중 손실(L_DT)을 획득한다.The first and second feature maps (f _RGB , f _IR ) and the soft-warped reconstructed first and second warping feature maps (

,

), the most direct way to calculate the error between the pixels is to calculate the L ₂ distance between the corresponding pixels by applying the L ₂ norm function. However, this has a limitation in that it does not consider an occluded area or the like. Accordingly, in the present embodiment, the loss calculator 600 acquires a common attention map emphasizing the human region that can be identified on both sides of the heterogeneous image, and calculates the error between the feature map and the warping feature map on the common attention map to densely triple A loss (L _DT ) is obtained.

First, the first common attention map (A _RGB ) for the first image may be obtained according to Equation (7).

Referring to Equation 7, the first common attention map A _RGB (p) represents a map composed of intersections between the first mask M _RGB and the warped second mask M _IR .

Similarly, the second common attention map (A _IR ) for the second image may also be obtained in the same manner.

In FIG. 8 , the left figure shows the second mask (M _IR ), the right figure shows the first mask (M _RGB ), and the middle figure shows the second common attention map (A _IR ). As shown in FIG. 8 , the second common attention map A _IR representing the intersection between the second mask M _IR and the warped first mask M _RGB is a portion of a person included in the first image. Even though the region is covered, only a region commonly displayed in the first and second images may be emphasized, and an unmarked region (eg, a knee) in at least one image may be suppressed.

When the first and second common attention maps (A _RGB , A _IR ) are obtained, the loss calculation unit 600 calculates a positive indicating a feature map for the same identifier with respect to an anchor and a negative indicating a feature map for a different identifier. The loss is calculated using a triplet learning method of comparison. Here, the positive and negative are obtained from a heterogeneous image different from the anchor (eg, when the anchor is a first image, a positive and negative pair is a heterogeneous second image).

)는 앵커(f^a _RGB)와 동일한 식별자를 갖는 포지티브 제2 특징맵(f^p _IR)을 와핑하여 재구성된 제1 와핑 특징맵(

)을 나타내고, 네거티브(

)는 앵커(f^a _RGB)와 다른 식별자를 갖는 네거티브 제2 특징맵(fⁿ _IR)을 와핑하여 재구성된 제1 와핑 특징맵(

)을 나타낸다.For example, if the first feature map (f _RGB ) is an anchor (f ^a _RGB ), positive (

) is the first warping feature map reconstructed by warping the second positive feature map f ^p _IR having the same identifier as the anchor f ^a _RGB (

), and negative (

^{) is} the first _{warping feature} map (

) is indicated.

)는 앵커(f^a _IR)와 동일한 식별자를 갖는 포지티브 제1 특징맵(f^p _RGB)을 와핑하여 재구성된 제2 와핑 특징맵(

)을 나타내고, 네거티브(

)는 앵커(f^a _IR)와 다른 식별자를 갖는 네거티브 제1 특징맵(fⁿ _RGB)을 와핑하여 재구성된 제2 와핑 특징맵(

)을 나타낸다.Similarly, when the second feature map (f _IR ) is an anchor (f ^a _IR ), positive (

) is the second warping feature map reconstructed by warping the positive first feature map (f ^p _RGB ) having the same identifier as the anchor (f ^a _IR )

), and negative (

^{) is} the second _{warping feature} map (

) is indicated.

Accordingly, the loss calculator 600 may calculate the dense triple loss L _DT based on each of the first and second common attention maps A _i , i ∈ {RGB, IR} according to Equation 8.

Here, α represents a predefined margin, and d _i ⁺ (p) and d _i ^- (p) represent the error between the anchor and the positive and the error between the anchor and the negative, respectively, calculated according to Equation 9, respectively.

When the identification loss (L _IC ), the coherence loss (L _IC ), and the dense triple loss (L _DT ) are calculated, the loss calculator 600 calculates the total loss (L) according to Equation 10 and backpropagates it, thereby learning can be performed.

where λ _IC and λ _DT are the coherence loss weight and the dense triple loss weight, respectively.

9 shows the performance change according to the loss setting of the human re-identification apparatus.

In FIG. 9, (a) shows pixels that match each other in heterogeneous images when learning is performed by applying only identification loss (L _IC ), and (b) shows identification loss (L _IC ) and coherence loss (L _IC ) When learning is performed by applying, it indicates pixels that are mutually matched in heterogeneous images. And (c) shows pixels that match each other when learning is performed by applying identification loss (L _IC ), coherence loss (L _IC ), and dense triple loss (L _DT ).

As shown in Fig. 9(c), when learning is performed by applying the identification loss (L _IC ), the coherence loss (L _IC ), and the dense triple loss (L _DT ), each pixel of the human region in the heterogeneous image It can be seen that these are precisely matched to positions corresponding to each other. And this indicates that the person re-identification apparatus of the present embodiment can re-identify a person with very high accuracy.

10 shows a simulation result of the performance of the human re-identification apparatus according to the present embodiment.

As described above, in heterogeneous images, each pixel of the human region can be precisely matched to a position corresponding to each other, and as shown in FIG. Even in an image containing multiple people, such as , it is possible to accurately match the corresponding pixels, so that people can be re-identified.

11 illustrates a method for re-identifying a person based on a heterogeneous sensor camera according to an embodiment of the present invention.

Referring to FIG. 11 , the heterogeneous sensor camera-based person re-identification method according to the present embodiment can be largely divided into a learning step ( S10 ) and a heterogeneous image person re-identification step ( S20 ). Referring to FIGS. 6 and 7 , first, the learning step S10 is described. First and second images, which are heterogeneous images obtained from a heterogeneous sensor camera and labeled with an identifier of a person included in each, are acquired as learning data. (S11). In this case, each of the first image and the second image may be an image of a single frame or an image composed of a plurality of frames. Also, it may be an image containing the same person or an image containing different people.

Then, the first and second learning feature maps are obtained by extracting features of each of the first and second images according to the learning method (S12). Thereafter, the similarity map C(p, q) is obtained by calculating the similarity between pixels in a predetermined manner between the obtained first and second learning feature maps (S14). Here, the similarity map C(p, q) may be calculated according to Equation 1 based on, for example, cosine similarity.

When the similarity map C(p,q) is obtained, the matching probability for each pixel position in the first and second learning feature maps is calculated according to Equation 2 based on the obtained similarity map C(p,q)) to obtain a matching probability map (P(p, q)) (S15). Then, using the matching probability map (P(p, q)), pixels corresponding to each other in the first and second learning feature maps are soft-warped to obtain a warping feature map (S15). In this case, in the soft warping, first and second masks in which the human region is emphasized in advance in each of the first and second learning feature maps are obtained according to Equations 3 and 4, and as shown in Equations 5 and 6, the obtained first masks are obtained. and using a corresponding mask among the second masks to concentrate and warp the human area of the warped feature map, while maintaining the pixels of the existing feature map in the remaining areas. That is, the pixels of the first learning feature map are warped to positions corresponding to the human region of the second feature map using the second mask, and the pixels of the second learning feature map are maintained in the remaining regions to obtain the second warped feature map. do. The pixels of the second learning feature map are warped to a position corresponding to the human region of the first feature map using the first mask, and the pixels of the second learning feature map are maintained in the remaining regions to obtain the first warped feature map. do. When the first and second warping feature maps are obtained, the first and second learning feature maps and the first and second warping feature maps are respectively pooled to obtain the first and second presenters and the first and second warping presenters. Acquire (S16).

Thereafter, based on the obtained first and second descriptors and each of the first and second warping descriptors, classes representing predetermined corresponding probabilities for each class are classified to obtain an identifier corresponding to each (S17). As an example, the identifier includes first and second scoremaps and first and second warping scoremaps indicating probabilities that the first and second presenters and the first and second warping presenters respectively correspond to a plurality of predetermined classes, respectively. The identifier of the class representing the highest probability may be extracted and obtained based on the obtained first and second scoremaps and the first and second warping scoremaps, respectively.

And when the identifier is obtained, the identification loss (L _ID ) and the consistency loss (L _IC ) is calculated, and the total loss is calculated and backpropagated by calculating the dense triple loss (L _DT ) using the first and second feature maps and the first and second warping feature maps ( S18 ).

where the identification loss (L _ID ) is calculated as the difference between the first and second scoremaps obtained from the first and second presenters and the identifiers labeled in the first and second training images, for example, the first and second 2 It can be computed as the negative cross entropy between the scoremap and the labeled identifiers.

And the loss of consistency (L _IC ) may be obtained by calculating a negative cross entropy between the first and second scoremaps and the first and second warping scoremaps.

On the other hand, the dense triple loss (L _DT ) can be confirmed from both sides of the heterogeneous learning image based on the learning feature map and the warping feature map corresponding to each other in the first and second learning feature maps and the first and second warping feature maps. First and second common attention maps emphasizing the human region are obtained, and on the common attention map, between the anchor, which is a learning feature map, and the positive, which is a warping feature map for the same identifier as the anchor's identifier, and the negative, which is a warping feature map for another identifier, on the common attention map. Calculate the error of to obtain the dense triple loss (L _DT ).

Then, the total loss is calculated by weighted summing the calculated identification loss (L _IC ), coherence loss (L _IC ), and dense triple loss (L _DT ), and learning is performed by backpropagating it. Learning may be performed a predetermined number of times or may be repeatedly performed until the total loss becomes less than or equal to a predetermined reference loss.

On the other hand, when the learning is completed, it is applied for real person re-identification to perform the heterogeneous image person re-identification step ( S20 ).

In the heterogeneous image person re-identification step, first and second heterogeneous images for which person re-identification is to be performed are first acquired ( S21 ). Then, first and second feature maps are obtained by extracting features from each of the first and second images obtained according to the method learned in the learning step (S22). Thereafter, first and second presenters are obtained by pooling the first and second feature maps in a predetermined manner (S23). When the first and second presenters are obtained, first and second scoremaps representing the probability that the obtained first and second presenters correspond to each of a plurality of predefined classes corresponding to a plurality of persons to be re-identified are generated. and re-identify the person by extracting the identifier corresponding to the class with the highest probability based on the obtained score map.

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is only exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: heterogeneous image acquisition unit 110: first image acquisition unit
120: second image acquisition unit 200: feature extraction unit
210: first feature extraction unit 220: second feature extraction unit
300: pulling unit 310: first pulling unit
320: second pulling unit 400: re-identification unit
500: cross modal alignment unit 510: similarity measurement unit
520: matching probability estimation unit 530: mask providing unit
540: soft warping unit 550: warping pulling unit
600: loss calculator

Claims (20)

The first and second images obtained using different heterogeneous sensors are obtained by extracting features according to a learning method and first and second feature maps are obtained, and the first and second features are pooled by a predetermined method. a feature pooling unit for obtaining a presenter;
It is provided during learning, receives the first and second feature maps to estimate pixels corresponding to each other, and performs mutual warping based on the estimated positions of the corresponding pixels, so that the second feature map becomes the first feature A first warping feature map warped to correspond to the map and a second warping feature map warped so that the first feature map corresponds to the second feature map are obtained, and the first and second warping feature maps are pooled to obtain a first and a cross-modal aligner for obtaining a second warping presenter;
first and second scoremaps representing probabilities that the first and second descriptors and each of the first and second warping descriptors respectively correspond to a plurality of predefined classes, respectively, according to the manner in which they are learned; a re-identification unit for obtaining a warping scoremap and obtaining an identifier of a class having the highest probability from the obtained first and second scoremaps; and
It is provided at the time of learning, and the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map are positive when the same identifier and different identifiers Person re-identification device including a loss calculator for calculating the dense triple loss by dividing the negative of. According to claim 1, wherein the cross-modal alignment unit
a similarity measuring unit for obtaining a similarity map by calculating a similarity between pixels between the first and second feature maps according to a predetermined method;
a matching probability estimator for obtaining a matching probability map by calculating a matching probability for each pixel of the first and second feature maps based on the similarity map;
The first and second warping feature maps are obtained by weighting the probability that pixels of each of the first and second feature maps are matched with each pixel of the relative feature map using the matching probability map and warping the relative feature map. a soft warping unit; and
and a warping pooling unit configured to pool the first and second warping feature maps in a predetermined manner to obtain first and second warping presenters. According to claim 2, wherein the cross-modal alignment unit
The apparatus for re-identifying a person further comprising a mask providing unit for obtaining first and second masks by normalizing the sizes of pixels for each location of the first and second feature maps. According to claim 3, wherein the soft warping portion
By receiving the first and second masks, the probability that the pixels of each of the first and second feature maps match each pixel of the relative feature map is weighted and aggregated, and the pixel value of the mask is added to the weighted aggregated warping pixel value. A warping person re-identification device weighted by The method of claim 4, wherein the mask providing unit
The first mask (M _RGB ) from the first feature map (f _RGB ) is calculated

(Where ||||||] ₂ is an L ₂ norm function, and f is a max-min regularization function.)
obtained according to
The soft warping part
From the first feature map f _RGB , the second feature map f _IR , and the first mask M _RGB , the first warping feature map (

) to the formula

(where W is according to the matching probability map (P(p,q))

It is a soft warping function represented by .)
A person re-identification device that is acquired according to The method of claim 4, wherein the loss calculator
First and second common attention maps that emphasize human regions that can be identified in both the first and second images are acquired, and estimated as the same identifier as the identifier of the first feature map that is an anchor in the first common attention map The difference between the positive, which is the first warping feature map, and the negative, which is the first warping feature map estimated with a different identifier, and the identifier that is the same as the identifier of the second feature map that is the anchor in the second common attention map. A person re-identification apparatus for calculating the dense triple loss by accumulating differences between positive and negative second warping feature maps estimated with different identifiers. The method of claim 6, wherein the loss calculator
Based on the first and second common attention map (A _i , i ∈ {RGB, IR}), the dense triple loss (L _DT ) is calculated by the equation

(where α denotes a predefined margin, and d _i ⁺ (p) and d _i ^- (p) are respectively

represents the error between the anchor (f _i ^a ) and the positive (f _i ^p ) and the error between the anchor (f _i ^a ) and the negative (f _i ⁿ ), respectively, calculated as .
A person re-identification device that counts according to. The method of claim 7, wherein the loss calculator
Identification loss due to a difference between each of the first and second scoremaps and identifiers that are labeled and applied to the first and second images during training, and the first and second scoremaps and the first and second warping A person re-identification apparatus for performing learning by further calculating a coherence loss according to a difference between the scoremaps, and backpropagating a total loss calculated by weighting the identification loss, the coherence loss, and the dense triple loss. The method of claim 8, wherein the loss calculator
The identification loss is obtained by calculating a negative cross entropy between the first and second scoremaps and the identifiers labeled and applied to the first and second images, and each of the first and second scoremaps and the second image A person re-identification device for calculating the coherence loss by calculating a negative cross entropy between the corresponding warping scoremaps among the first and second warping scoremaps. The apparatus of claim 1, wherein the first image is an RGB image obtained from a visible camera and the second image is an IR image obtained from an IR camera. learning stage; and
A person re-identification step of using an artificial neural network to obtain an identifier for a person included in each image by receiving images obtained using different heterogeneous sensors,
The learning step is
The first and second images acquired using different heterogeneous sensors as learning data and the identifiers are pre-labeled are applied and the features are extracted according to the learning method to obtain the first and second feature maps, and a predetermined method pooling to obtain first and second presenters;
The first and second feature maps are applied, pixels corresponding to each other are estimated, and mutual warping is performed based on the estimated positions of the corresponding pixels, so that the second feature map is warped to correspond to the first feature map. A first warping feature map and a second warping feature map warped so that the first feature map corresponds to the second feature map are obtained, and first and second warping presenters are generated by pooling the first and second warping feature maps. obtaining;
obtaining first and second scoremaps and first and second warping scoremaps representing probabilities of the first and second presenters and each of the first and second warping presenters corresponding to each of a plurality of predefined classes and obtaining the identifier of the class with the highest probability from the obtained first and second scoremaps; and
Loss by dividing the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map into a positive in case of the same identifier and a negative in the case of different identifiers A method of re-identification of a person comprising the step of calculating 12. The method of claim 11, wherein obtaining the first and second warping descriptors comprises:
obtaining a similarity map by calculating a similarity between pixels between the first and second feature maps according to a predetermined method;
obtaining a matching probability map by calculating a matching probability for each pixel of the first and second feature maps based on the similarity map;
The first and second warping feature maps are obtained by weighting the probability that pixels of each of the first and second feature maps are matched with each pixel of the relative feature map using the matching probability map and warping the relative feature map. to do; and
and pooling the first and second warping feature maps in a predefined manner to obtain first and second warping descriptors. 13. The method of claim 12, wherein obtaining the first and second warping descriptors comprises:
and obtaining first and second masks by normalizing the sizes of pixels for each location of the first and second feature maps. 14. The method of claim 13, wherein the obtaining of the first and second warping feature maps comprises:
receiving the first and second masks and weighting and counting probabilities that pixels of each of the first and second feature maps match each pixel of a relative feature map; and
A method of re-identifying a person, comprising the step of warping by additionally weighting the pixel values of the mask to the weighted aggregated warping pixel values. 15. The method of claim 14, wherein obtaining the mask comprises:
The first mask (M _RGB ) from the first feature map (f _RGB ) is calculated

(Where ||||||] ₂ is an L ₂ norm function, and f is a max-min regularization function.)
obtained according to
The step of warping by adding weight is
From the first feature map f _RGB , the second feature map f _IR , and the first mask M _RGB , the first warping feature map (

) to the formula

(where W is according to the matching probability map (P(p,q))

It is a soft warping function represented by .)
How to re-identify the person you acquire according to. 15. The method of claim 14, wherein calculating the loss comprises:
obtaining first and second common attention maps emphasizing a human region that can be identified in both the first and second images; and
In the first common attention map, the difference between the positive, which is the first warping feature map estimated with the same identifier as the identifier of the first feature map, which is the anchor, and the negative, which is the first warping feature map, estimated with different identifiers, respectively, and the In the second common attention map, the density triple loss calculated by accumulating the difference between the positives estimated with the same identifier as the identifier of the second feature map, which is the anchor, and the negative, the second warping feature map, estimated with different identifiers. A method of re-identification of a person comprising the step of obtaining. 17. The method of claim 16, wherein obtaining the dense triple loss comprises:
Based on the first and second common attention map (A _i , i ∈ {RGB, IR}), the dense triple loss (L _DT ) is calculated by the equation

(where α denotes a predefined margin, and d _i ⁺ (p) and d _i ^- (p) are respectively

represents the error between the anchor (f _i ^a ) and the positive (f _i ^p ) and the error between the anchor (f _i ^a ) and the negative (f _i ⁿ ), respectively, calculated as .
The method of re-identification of the person counting according to. 18. The method of claim 17, wherein calculating the loss comprises:
calculating an identification loss according to a difference between each of the first and second scoremaps and an identifier that is labeled and applied to the first and second images during learning;
calculating a consistency loss according to a difference between the first and second scoremaps and the first and second warping scoremaps; and
and backpropagating a total loss calculated by weighting the identification loss, the coherence loss, and the dense triple loss. 19. The method of claim 18, wherein calculating the identification loss comprises:
obtaining the identification loss by calculating a negative cross entropy between the first and second scoremaps and the identifiers labeled and applied to the first and second images;
Calculating the consistency loss comprises:
and calculating the coherence loss by calculating a negative cross entropy between each of the first and second scoremaps and a corresponding one of the first and second warping scoremaps. The method of claim 11, wherein the step of re-identifying the person
obtaining a feature map by extracting features of each of the heterogeneous images applied according to the method previously learned in the learning step;
pooling a feature map obtained from each of the heterogeneous images in a predetermined manner to obtain a corresponding presenter; and
A method of re-identifying a person, comprising: obtaining a scoremap corresponding to each obtained presenter according to a pre-learned method; .

Images