KR102540290B1 - Apparatus and Method for Person Re-Identification based on Heterogeneous Sensor Camera - Google Patents

Abstract

The present invention performs learning based on warping feature maps obtained by warping pixels corresponding to each other in the feature maps together with feature maps extracted from images acquired from heterogeneous sensor cameras. Apparatus and method for re-identifying a person accurately even from images acquired under very different conditions from various cameras by extracting segmented local information even from unaligned images by calculating loss and reflecting it in learning. can provide.

Description

Apparatus and Method for Person Re-Identification based on Heterogeneous Sensor Camera}

The present invention relates to an apparatus and method for human re-identification, and relates to a heterogeneous sensor camera-based apparatus and method for human re-identification.

Research on Person Re-identification (referred to as reID) technology capable of searching for the same person photographed in different environments is being actively conducted.

1 shows the concept of person re-identification, and FIG. 2 shows an example of a person tracking technology as one of the fields using person re-identification.

As shown in FIG. 1, person re-identification is a technique for identifying and detecting the same person photographed under different conditions, and various environmental conditions such as change in posture, change in background, change in lighting, and change in shooting distance and angle. It is aimed at accurately detecting an image including the same person even if the image is changed.

As shown in FIG. 2 , such person re-identification technology can be used in various fields of searching for and tracking a specific person in multiple images, such as searching for a missing person or tracking a criminal.

Such a person re-identification technology extracts features representing an identity rather than a specific part of a person in order to identify an individual from an image, mainly using an artificial neural network, and allows the same person to be identified based on the extracted features. That is, it is possible to detect the same person in a plurality of different images by re-identifying a person by extracting features representing each individual identity, not limited to a person's face shape, specific posture, background, shooting angle, lighting, etc. However, existing human re-identification was mainly focused on re-identifying the same person in images obtained from the same type of sensor camera.

3 shows an example of an image obtained from a heterogeneous sensor camera, and FIG. 4 illustrates a concept of re-identifying a person based on a heterogeneous sensor camera.

3 and the upper three images, the left two images are images acquired during the day and night from a general visible camera (RGB camera or visible camera), respectively, and the right image is an image acquired at night from an infrared (hereinafter referred to as IR) camera. indicates

The image on the left in the middle represents a human region extracted from an image acquired by a visible camera during the daytime, and multiple images on the right represent images to be compared for re-identification. In addition, the lower right image represents a human area extracted from an image acquired by an IR camera at night, and a plurality of images on the left represent images to be compared for re-identification.

As shown in FIG. 3 , the visible camera can acquire an image at a level at which a person can be identified when acquiring an image during the day, but cannot identify a person at night because it is not distinguished from the surrounding environment. In contrast, when an IR camera using an IR sensor different from a visible camera is used, a person can be accurately identified at night.

Therefore, as shown in FIG. 4 , in order to identify and track a person regardless of environmental changes, it is necessary to re-identify a person from images obtained from different types of sensor cameras.

5 is a diagram for explaining a misalignment problem in person re-identification.

In FIG. 5 , images located in the same row at the top and bottom are images of the same object, that is, the same person. As shown in FIG. 5 , even in images obtained by the same visible camera, people may appear in very different shapes depending on various conditions, and as a result, it may be difficult to identify the same person in each image. However, in the case of an image obtained from the same visible camera, a person may be re-identified based on characteristics according to color distribution and the like.

However, unlike a visible camera, an IR camera does not include color information, so it is not easy to compare the characteristics of a color image obtained from a visible camera and an IR image obtained from an IR camera. That is, features that can be compared with each other for re-identification are limited. In order to overcome this difference in modality, conventionally, both the RGB image and the IR image are embedded in a virtual feature space (common feature space), and features embedded in the feature space are extracted and compared at the image or part level. carry out learning with However, in such a method of embedding in a common feature space, a problem may occur due to object misalignment due to an error in dense matching of feature points.

In the past, when re-identifying a person in images obtained from different heterogeneous sensor cameras, it was assumed that the people included in each image were aligned with each other and focused on learning features at the image or part level. That is, learning was performed assuming that the two images were aligned at a certain level or higher, even if they were approximate. For example, it is learned by assuming that images are divided into predetermined horizontal grids, and body parts (eg, shoulders, knees, etc.) included in each grid in the two images are aligned to correspond to each other.

However, as shown in FIG. 5, even though they are images of the same object, the position, size, and pose of the object in the image are often very different. This may not be performed normally, resulting in a large human re-identification error. As described above, such a misalignment problem may appear more serious in object re-identification of an image between two modalities having different characteristics.

In addition, in the method of embedding in a common feature space, there is a limit to re-identification performance because segmented local information that can be greatly helpful in object re-identification tends to be overlooked.

Korean Patent Publication No. 10-2019-0078270 (published on July 4, 2019)

An object of the present invention is to provide a person re-identification device and method capable of accurately re-identifying a person from an image acquired by a heterogeneous sensor camera.

Another object of the present invention is to provide a person re-identification device and method capable of accurately re-identifying a person using dense matching between cross-modality object images even in unaligned images.

In order to achieve the above object, an apparatus for re-identifying a person according to an embodiment of the present invention receives first and second images acquired using different types of sensors and extracts features according to a learning method to obtain first and second images. a feature pooling unit that obtains a second feature map and performs pooling in a predetermined manner to obtain first and second descriptors; Provided during learning, receiving the first and second feature maps, estimating pixels corresponding to each other, and performing mutual warping based on the positions of the estimated corresponding pixels, so that the second feature map corresponds to the first feature Obtaining a first warped feature map warped to correspond to the map and a second warped feature map warped so that the first feature map corresponds to the second feature map, and pooling the first and second warped feature maps to obtain a first and a cross-modal arranging unit obtaining a second warping descriptor; First and second score maps representing probabilities corresponding to each of a plurality of classes in which the first and second descriptors and the first and second warping descriptors are pre-specified according to the learned method, and the first and second descriptors a re-identification unit that obtains a warping score map and obtains an identifier of a class having the highest probability in the obtained first and second score maps; And provided during learning, the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map is a positive identifier in the case of the same identifier and a different identifier and a loss calculation unit that calculates the dense triple loss by dividing the case into the negative.

The cross-modal arranging unit may include a similarity measurement unit calculating a similarity between pixels between the first and second feature maps according to a predetermined method and obtaining a similarity map; a matching probability estimator calculating a matching probability for each pixel of the first and second feature maps based on the similarity map to obtain a matching probability map; Obtaining the first and second warping feature maps by weighting probabilities of matching pixels of the first and second feature maps using the matching probability map to each pixel of the relative feature map and warping the relative feature map a soft warping unit; and a warping pooling unit that obtains first and second warping descriptors by pooling the first and second warping feature maps in a predetermined manner.

The cross-modal arranging unit may further include a mask providing unit for obtaining first and second masks by normalizing the sizes of pixels for each position of the first and second feature maps.

The soft warping unit receives the first and second masks, weights and aggregates probabilities of matching pixels of each of the first and second feature maps to each pixel of the relative feature map, and calculates the value of the mask based on the weighted and aggregated warped pixel values. You can warp by additionally weighting the pixel values.

The loss calculation unit obtains first and second common attentional maps emphasizing a human region identifiable in both the first and second images, and an identifier of the first feature map that is an anchor in the first common attentional map. The difference between each positive, which is the first warping feature map estimated with the same identifier and the negative, which is the first warping feature map estimated with different identifiers, and the same as the identifier of the second feature map, which is an anchor in the second common attention map The dense triple loss may be calculated by accumulating differences between positive values estimated as identifiers and negative values, which are second warping feature maps estimated using different identifiers.

The loss calculation unit identifies an identification loss according to a difference between each of the first and second scoremaps and identifiers that are labeled and applied to the first and second images during training, and the first and second scoremaps and the first and second scoremaps. and further calculating a coherence loss according to the difference between the second warping scoremaps, and performing learning by backpropagating a total loss calculated by weighting the identification loss, the coherence loss, and the dense triplet loss.

The loss calculation unit obtains the identification loss by calculating a negative cross entropy between the first and second scoremaps and identifiers labeled and applied to the first and second images, and the first and second scoremaps. The coherence loss may be calculated by calculating a negative cross entropy between each of the warping score maps and a corresponding warping score map of the first and second warping score maps.

A method for re-identifying a person according to another embodiment of the present invention for achieving the above object includes a learning step; and a person re-identification step of obtaining an identifier for a person included in each image by receiving images acquired using different types of sensors using an artificial neural network, wherein the learning step includes different types of learning data. First and second feature maps are obtained by extracting features according to a method in which first and second images obtained using heterogeneous sensors and pre-labeled with identifiers are received and learned, and first and second feature maps are obtained by pooling in a predetermined manner to obtain first and second images. and obtaining a second presenter; The first and second feature maps are received, pixels corresponding to each other are estimated, and mutual warping is performed based on the positions of the estimated corresponding pixels, so that the second feature map is warped to correspond to the first feature map. Obtaining a first warped feature map and a second warped feature map warped so that the first feature map corresponds to the second feature map, and pooling the first and second warped feature maps to obtain first and second warped descriptors obtaining; Obtaining first and second scoremaps and first and second warping scoremaps indicating probabilities corresponding to each of a plurality of classes in which the first and second descriptors and the first and second warping descriptors are pre-specified, respectively and obtaining an identifier of a class having the highest probability in the obtained first and second score maps; And dividing the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map into a positive in the case of the same identifier and a negative in the case of different identifiers It involves calculating the loss.

Therefore, the apparatus and method for re-identification of a person according to an embodiment of the present invention performs learning based on a feature map extracted from images acquired from a heterogeneous sensor camera and a warped feature map obtained by warping pixels corresponding to each other in the feature map. In particular, by calculating the dense triple loss for the feature map and the warping feature map and applying it to learning, it is possible to extract segmented local information even from unaligned images. can be accurately re-identified.

1 illustrates the concept of person re-identification.
2 shows an example of a person tracking technology as one of the fields using person re-identification.
3 shows an example of an image acquired by a heterogeneous sensor camera.
4 shows a concept of human re-identification based on heterogeneous sensor cameras.
5 is a diagram for explaining a misalignment problem in person re-identification.
6 shows a schematic structure of a device for re-identifying a person based on a heterogeneous sensor camera according to an embodiment of the present invention.
FIG. 7 is a diagram for explaining the operation of the apparatus for re-identifying a person of FIG. 6 .
8 is a diagram for explaining a common attention map.
9 shows the performance change according to the loss setting of the person re-identification device.
10 shows simulation results of the performance of the apparatus for re-identification of a person according to the present embodiment.
11 illustrates a method for re-identifying a person based on a heterogeneous sensor camera according to an embodiment of the present invention.

In order to fully understand the present invention and its operational advantages and objectives achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the described embodiments. And, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that it may further include other components, not excluding other components unless otherwise stated. In addition, terms such as "... unit", "... unit", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. And it can be implemented as a combination of software.

6 shows a schematic structure of a device for re-identifying a person based on a heterogeneous sensor camera according to an embodiment of the present invention, FIG. 7 is a diagram for explaining the operation of the device for re-identifying a person in FIG. 6, and FIG. 8 is a common attention map. It is a drawing for explaining.

Referring to FIGS. 6 and 7 , the heterogeneous sensor camera-based human re-identification device according to the present embodiment includes a heterogeneous image acquisition unit 100, a feature extraction unit 200, a pooling unit 300, a re-identification unit 400, an intersection It may include a modal arranging unit 500 and a loss calculating unit 600 .

The heterogeneous image acquisition unit 100 acquires heterogeneous images obtained from different sensor cameras. The heterogeneous image acquisition unit 100 may acquire a first image and a second image, where, for example, the first image is an RGB image obtained using a visible camera, and the second image is obtained using an IR camera. It is assumed that it is an IR image and explained. The heterogeneous image acquisition unit 100 may include a first image acquisition unit 110 that acquires a first image and a second image acquisition unit 120 that obtains a second image. However, the heterogeneous sensor camera-based human re-identification apparatus of the present embodiment may acquire images using a sensor camera of a different type in addition to a visible camera or an IR camera. In this case, the first image and the second image may each be a single frame image, or may be images composed of multiple frames, and may include the same person or different people.

The first image acquisition unit 110 and the second image acquisition unit 120 are implemented as heterogeneous sensor cameras each including different sensors, or in a storage device for storing images acquired using heterogeneous sensor cameras, or in heterogeneous external sources. It may also be implemented as a communication module that receives an image obtained using a sensor camera.

The feature extraction unit 200 obtains a feature map by extracting features from each of the heterogeneous images acquired by the heterogeneous image obtaining unit 100 . The feature extractor 200 may include a first feature extractor 210 and a second feature extractor 220 each implemented as an artificial neural network such as a convolutional neural network (CNN). The first feature extractor 210 receives the first image acquired by the first image acquirer 110 and extracts features according to a pre-learned method to obtain a first feature map f _RGB , The feature extractor 220 receives the second image acquired by the second image acquirer 120 and extracts features according to a pre-learned method to obtain a second feature map f _IR .

)과 제2 특징맵(f_IR ∈

)은 동일한 높이(h)와 폭(w) 및 채널(d)을 갖는 h × w × d 크기로 추출될 수 있다.At this time, the first feature map (f _RGB ∈

) and the second feature map (f _IR ∈

) can be extracted in the size of h × w × d with the same height (h), width (w) and channel (d).

The pooling unit 300 receives the first and second feature maps f _RGB and f _IR obtained from the feature extraction unit 200 and pools them in a predetermined manner, respectively, to obtain first and second descriptors. The pulling unit 300 may include a first pulling unit 310 and a second pulling unit 320 . The first pooling unit 310 receives and pools the first feature map f _RGB to obtain a first descriptor ø(f _RGB ), and the second pooling unit 320 obtains the second feature map f _IR ) may be applied and pooled to obtain the second descriptor ø(f _IR ). Here, each of the first and second descriptors ø(f _RGB ) and ø(f _IR ) has a form of a 1-dimensional vector corresponding to the length of the channel d, and is a feature enabling identification of a person included in the image. can be seen as expressive.

Here, the feature extraction unit 200 and the pooling unit 300 may be integrated into a feature pooling unit.

The re-identification unit 400 receives the first and second expressers ø(f _RGB ) and ø(f _IR ), which are feature expressors of the person included in the image, and identifies the person included in the first and second images. re-identify For example, the re-identification unit 400 determines whether the people included in the first and second images are the same person by measuring the degree of similarity between the first and second presenters ø(f _RGB ) and ø(f _IR ). can be identified. However, the re-identification unit 400 checks the classes corresponding to the first and second descriptors ø(f _RGB ) and ø(f _IR ) among a plurality of classes set to correspond to a plurality of pre-specified persons, and determines the identifier. may be configured to obtain That is, the re-identification unit 400 may obtain an identifier of a person included in each of the different types of images to determine whether they are the same person.

Here, the re-identification unit 400 is implemented as an artificial neural network including, for example, an activation function such as soft max and a fully connected layer, and the first and second expressors ø(f _RGB ), An identifier may be selected by obtaining a scoremap representing a probability corresponding to each of a plurality of predefined classes, respectively ø(f _IR ), and checking a class with the highest probability in the obtained scoremap.

In FIG. 6, for convenience of explanation, the feature extraction unit 200, the pooling unit 300, and the re-identification unit 400 are shown as separate configurations, but the feature extraction unit 200, the pooling unit 300, and the re-identification unit 400 are shown. Unit 400 may be composed of layers that perform different functions in a re-identification artificial neural network for re-identification of a person.

The human re-identification device may be composed of only the heterogeneous image acquisition unit 100, the feature extraction unit 200, the pooling unit 300, and the re-identification unit 400 when applied in practice. However, in the human re-identification device, the feature extraction unit 200, the pooling unit 300, and the re-identification unit 400 composed of artificial neural networks must be learned before they are actually applied. Accordingly, the apparatus for re-identification of a person according to the present embodiment may further include a cross-modal arranging unit 500 and a loss calculating unit 600 during learning, and may be excluded when learning is completed.

,

)을 획득한다. 여기서 제1 와핑 특징맵(

)은 제2 이미지에서 추출된 제2 특징맵(f_IR)을 와핑하여 획득되는 특징맵이고, 제2 와핑 특징맵(

)은 제1 이미지에서 추출된 제1 특징맵(f_RGB)을 와핑하여 획득되는 특징맵이다. 즉 제1 및 제2 와핑 특징맵(

,

)은 서로 상대의 이종 이미지로부터 추출된 특징맵을 와핑하여 모달리티를 교차하여 획득된다.The cross-modal arranging unit 500 receives first and second feature maps (f _RGB , f _IR ) extracted from different heterogeneous images, and receives the applied first feature map (f _RGB ) and the second feature map (f _IR ), by estimating pixels corresponding to each other based on the similarity between pixels, and performing mutual warping based on the positions of the estimated corresponding pixels, to obtain first and second warping feature maps (

,

) to obtain Here, the first warping feature map (

) is a feature map obtained by warping the second feature map (f _IR ) extracted from the second image, and the second warped feature map (

) is a feature map obtained by warping the first feature map (f _RGB ) extracted from the first image. That is, the first and second warping feature maps (

,

) is obtained by crossing modalities by warping the feature maps extracted from each other's heterogeneous images.

Specifically, the cross-modal arranging unit 500 may include a similarity measuring unit 510, a matching probability estimating unit 520, a mask providing unit 530, a soft warping unit 540, and a warping pulling unit 550. can

The similarity measurer 510 receives the first and second feature maps f _RGB and f _IR , and calculates the pixel-to-pixel similarity between the applied first and second feature maps f _RGB and f _IR to determine the degree of similarity. Obtain map C(p,q). The similarity measurement unit 510 calculates a pixel vector, which is a one-dimensional vector having a channel direction length (d) for each position (p) on the h × w plane of the first feature map (f _RGB ), and the second feature map (f _IR ). A similarity map (C(p,q )) can be obtained.

Here, f _RGB (p) represents a pixel vector for each position (p) of the first feature map (f _RGB ), and f _IR (q) represents a pixel vector for each position (q) of the second feature map (f _IR ). , Α Π ₂ denotes the L2 norm function, and T denotes the transposition matrix.

The matching probability estimator 520 receives the similarity map C(p,q) obtained from the similarity measurement unit 510 and calculates a matching probability for each pixel. The matching probability estimator 520 determines the first feature map (f _RGB ) and the second feature map (f RGB ) based on the similarity between pixel vectors of the second feature map (f _RGB ) and the second feature map (f _IR ). A matching probability map (P(p,q)) having a size of h × w × h × w representing a matching probability for each pixel location of _IR is calculated and obtained according to Equation 2.

where β is the temperature parameter.

The mask providing unit 530 receives the first and second feature maps f _RGB and f _IR , and receives the first mask M _RGB corresponding to the first feature map f _RGB and the second feature map f A second mask (M _IR ) corresponding to _IR ) is obtained and output to the soft warping unit 540 .

As learning is performed, the mask providing unit 530 assumes that the region corresponding to a person in the first and second images, that is, the foreground has a prominent feature compared to the remaining regions, and the first image without labels for the real human region. A first mask (M _RGB ) in which the human region is emphasized may be obtained from the first feature map (f _RGB ) obtained in Equation 3.

where f is the maximum-minimum normalization function defined by Equation 4.

Similarly, the mask providing unit 530 may also obtain a second mask M _IR corresponding to the human region from the second feature map f _IR obtained from the second image.

That is, the mask providing unit 530 obtains a mask by normalizing the size of the pixel vector for each position of the first and second feature maps f _RGB and f _IR . If the mask providing unit 530 uses a binary mask, it is effective when the foreground and background can be clearly distinguished, but may cause a large error when not clearly distinguished. In order to prevent such an error from occurring, the mask provider 530 of the present embodiment normalizes the size of the pixel vector for each position of the first and second feature maps f _RGB and f _IR so that the foreground area is the background area. First and second masks (M _RGB , M _IR ) that are emphasized compared to M are obtained.

,

)을 획득한다.Meanwhile, the soft warping unit 540 compares each pixel of the first and second feature maps f _RGB and f _IR based on the inter-pixel matching probability map P(p,q) obtained according to Equation 2. First and second warped feature maps by soft warping to corresponding pixel locations (

,

) to obtain

At this time, the soft warping unit 540 uses the first and second masks M _RGB and M _IR provided from the mask providing unit 530 to determine the human face in the first and second feature maps f _RGB and f _IR . Warping may be performed intensively on a foreground region corresponding to .

)을 획득할 수 있다.The soft warping unit 540 generates a first mask (M _RGB ) according to Equation 5 so that each pixel of the second feature map (f _IR ) emphasizes and warps the foreground region corresponding to the person of the first feature map (f _RGB ). ) by performing soft warping using the first warping feature map (

) can be obtained.

Here, W is a soft warping function defined by Equation 6.

)을 획득한다.According to Equations 5 and 6, the soft warping unit 540 weights the pixel vectors for each position q of the second feature map f _IR using the soft warping function W according to the matching probability. 1 warp to the corresponding pixel position (p) of the feature map (f _RGB ), but the matching probability weighted pixel vector of the second feature map (f _IR ) is aggregated in the human area designated by the first mask (M _RGB ) It is emphasized and warped, and the pixel vector of the first feature map (f _RGB ) is maintained as much as possible in the remaining area. That is, the foreground area corresponding to a person in the first feature map f _RGB is weighted and aggregated according to the matching probability in the second feature map f _IR , and the warped pixel vector is emphasized, while the remaining background area is the first feature map The first warping feature map (f _RGB ) is emphasized so that the pixel vector of

) to obtain

)을 획득한다.Also, the soft warping unit 540 uses a soft warping function and a second mask M _IR similar to Equations 5 and 6, so that the foreground region corresponding to the person in the second feature map f _IR is the first feature The pixel vector warped in the map f _RGB is emphasized, and the pixel vector of the second feature map f _IR is emphasized in the remaining background area, so that the second warped feature map (

) to obtain

In this embodiment, the cross-modal aligning unit 500 uses the soft warping function (W) because when using the argmax function generally used for matching between pixels in different images, matching between pixels is performed very hard. This is because while it is possible to explicitly set corresponding pixels, cases in which corresponding points are not dense and scattered or do not exist frequently occur due to textures or occlusion of different images. In particular, as in this embodiment, when detecting corresponding pixels between the first and second feature maps (f _RGB , f _IR ) extracted from heterogeneous images, rather than detailed features such as color or texture of the image, More emphasis should be placed on the rough shape change. Therefore, in this embodiment, warping is performed on the positions of pixels corresponding to each other in heterogeneous images by using the soft warping function (W) instead of the argmax function.

)과 제2 와핑 특징맵(

) 각각을 풀링하여 제1 와핑 표현자(ø(

))과 제2 와핑 표현자(ø(

))를 획득한다.The warping pooling unit 550 generates a first warping feature map obtained from the soft warping unit 540 in the same manner as the first and second pooling units 310 and 320 (

) and the second warping feature map (

) by pooling each of the first warped descriptors (ø(

)) and the second warping descriptor (ø(

)) to obtain

), ø(

))을 인가받아 제1 및 제2 표현자(ø(f_RGB), ø(f_IR))에 대한 식별자를 추출할 수 있다.And, like the first and second descriptors ø(f _RGB ) and ø(f _IR ), the re-identification unit 400 generates first and second warping descriptors ø(

), ø(

)), identifiers for the first and second descriptors ø(f _RGB ) and ø(f _IR ) may be extracted.

The loss calculation unit 600 calculates the loss generated in the learning process in a predetermined manner and back-propagates the calculated loss, so that the feature extraction unit 200, the pooling unit 300, the re-identification unit 400 and the intersection The modal alignment unit 500 is trained.

In this embodiment, the loss calculator 600 calculates identity loss (L _ID ), identity consistency loss (L _IC ), and dense triplet loss (L _DT ), respectively. , learning can be performed by backpropagating the total loss (L), which is calculated by combining the calculated identification loss (L _ID ) with the coherence loss (L _IC ) and the dense triple loss (L _DT ).

The loss calculation unit 600 may first obtain an identification loss L _ID based on the first and second descriptors ø(f _RGB ) and ø(f _IR ). Here, the identification loss (L _ID ) is the first and second score maps obtained from the first and second presenters (ø(f _RGB ) and ø(f _IR ) and the first and second images input during training. may be calculated and obtained based on a difference between identifiers for persons pre-labeled in . That is, the apparatus for re-identifying a person according to the present embodiment may perform learning in a supervised learning method by receiving a heterogeneous image pre-labeled with an identifier for a person as training data during learning.

The loss calculation unit 600 may calculate an identification loss (L _ID ) by calculating a negative cross-entropy between the first and second score maps and an identifier applied as training data. Here, a method of calculating the identification loss (L _ID ) based on negative cross entropy is a well-known technique and will not be described in detail here.

On the other hand, the loss calculation unit 600 determines if the foreground regions of the first and second feature maps f _RGB and f _IR obtained from the heterogeneous image by the cross-modal arranging unit 500 are normally warped, the first and second Consistency loss (L _IC ) is calculated in which an identifier extracted from the scoremap and an identifier extracted from the first and second warping scoremaps must be the same.

At this time, the loss calculation unit 600 finally extracts the identifiers from the first and second scoremaps by learning to be the same as the identifiers labeled in the training data, so the identification loss (L _ID) from the first and second scoremaps. ), the consistency loss (L _IC ) may be calculated based on the identifiers labeled in the first and second score maps and the training data.

However, since the consistency loss (L _IC ) represents the difference between an identifier extracted from the first and second scoremaps and an identifier extracted from the first and second warping scoremaps, rather than an identifier for an actual image, the loss calculation unit 600 may calculate a coherence loss (L _IC ) with a negative cross entropy between the first and second scoremaps and the first and second warping scoremaps.

,

)을 인가받고, 이중 서로 대응하는 특징맵과 와핑 특징맵을 기반으로 제1 및 제2 공통 주의맵(co-attention map)을 획득하고, 획득된 제1 및 제2 공통 주의맵을 이용하여 밀집 삼중 손실(L_DT)을 계산한다.Meanwhile, the first and second feature maps f _RGB and f _IR obtained by the feature extractor 200 and the first and second warped feature maps obtained by the soft warping unit 540 (

,

), obtaining first and second co-attention maps based on the feature maps and warping feature maps corresponding to each other, and concentrating using the obtained first and second co-attention maps Calculate the triple loss (L _DT ).

,

) 사이의 오차를 계산하는 가장 직접적인 방식은 L₂ 놈 함수를 적용하여 대응하는 픽셀간 L₂ 거리를 계산하는 것이다. 그러나, 이는 폐색 영역 등을 고려하지 않는다는 한계가 있다. 이에 본 실시예에서 손실 계산부(600)는 이종의 이미지 양측에서 모두 확인 가능한 사람 영역을 강조하는 공통 주의맵을 획득하고, 공통 주의맵 상에서 특징맵과 와핑 특징맵 사이의 오차를 계산하여 밀집 삼중 손실(L_DT)을 획득한다.First and second warped feature maps reconstructed by soft warping with the first and second feature maps (f _RGB , f _IR ) (

,

) is to calculate the L ₂ distance between corresponding pixels by applying the L ₂ norm function. However, this has a limitation in that it does not consider the occluded area or the like. Therefore, in this embodiment, the loss calculation unit 600 obtains a common attention map emphasizing the human region identifiable on both sides of the heterogeneous images, calculates the error between the feature map and the warping feature map on the common attention map, and calculates the dense triplet Loss (L _DT ) is obtained.

First, the first common attentional map A _RGB for the first image may be obtained according to Equation 7.

Referring to Equation 7, the first common attentional map A _RGB (p) represents a map composed of intersections between the first mask M _RGB and the warped second mask M _IR .

Likewise, the second common attentional map (A _IR ) for the second image may also be obtained in the same manner.

In FIG. 8 , the left figure represents the second mask (M _IR ), the right figure represents the first mask (M _RGB ), and the middle figure represents the second common attentional map (A _IR ). As shown in FIG. 8 , the second common attentional map A _IR representing the intersection between the second mask M _IR and the warped first mask M _RGB is part of the person included in the first image. Even though the region is covered, only the region commonly displayed in the first and second images is emphasized, and an unmarked region (eg, a knee) in at least one image may be suppressed.

When the first and second common attentional maps (A _RGB , A _IR ) are acquired, the loss calculation unit 600 generates a positive representing a feature map for the same identifier and a negative representing a feature map for a different identifier based on the anchor. The loss is calculated using a triplet learning method that compares. Here, the positive and negative pairs are acquired from a heterogeneous image different from the anchor (eg, when the anchor is a first image, a positive and negative pair is a heterogeneous second image).

)는 앵커(f^a _RGB)와 동일한 식별자를 갖는 포지티브 제2 특징맵(f^p _IR)을 와핑하여 재구성된 제1 와핑 특징맵(

)을 나타내고, 네거티브(

)는 앵커(f^a _RGB)와 다른 식별자를 갖는 네거티브 제2 특징맵(fⁿ _IR)을 와핑하여 재구성된 제1 와핑 특징맵(

)을 나타낸다.For example, when the first feature map (f _RGB ) is an anchor (f ^a _RGB ), a positive (

) is a first warped feature map reconstructed by warping the second positive feature map (f ^p _IR ) having the same identifier as the anchor (f ^a _RGB ) (

), negative (

) is a first warped feature map reconstructed by warping the second negative feature map (f ⁿ _IR ) having an identifier different from the anchor (f ^a _RGB ) (

).

)는 앵커(f^a _IR)와 동일한 식별자를 갖는 포지티브 제1 특징맵(f^p _RGB)을 와핑하여 재구성된 제2 와핑 특징맵(

)을 나타내고, 네거티브(

)는 앵커(f^a _IR)와 다른 식별자를 갖는 네거티브 제1 특징맵(fⁿ _RGB)을 와핑하여 재구성된 제2 와핑 특징맵(

)을 나타낸다.Similarly, when the second feature map (f _IR ) is an anchor (f ^a _IR ), a positive (

) is a second warped feature map (reconstructed by warping the positive first feature map (f ^p _RGB ) having the same identifier as the anchor (f ^a _IR ) (

), negative (

) is a second warped feature map reconstructed by warping the first negative feature map (f ⁿ _RGB ) having an identifier different from that of the anchor (f ^a _IR ) (

).

Accordingly, the loss calculation unit 600 may calculate the dense triple loss (L _DT ) according to Equation 8 based on each of the first and second common attentional maps (A _i , i ∈ {RGB, IR}).

Here, α represents a predefined margin, and di ₊ ⁽ p) and _di ^- (p) represent the error between the anchor and the positive and the error between the anchor and the negative, respectively, calculated according to Equation 9.

When the identification loss (L _IC ), coherence loss (L _IC ), and dense triple loss (L _DT ) are calculated, the loss calculation unit 600 calculates the total loss (L) according to Equation 10 and backpropagates, thereby learning can be performed.

where λ _IC and λ _DT are the coherence loss weight and the dense triple loss weight, respectively.

9 shows the performance change according to the loss setting of the person re-identification device.

In FIG. 9, (a) represents mutually matched pixels in heterogeneous images when learning is performed by applying only the identification loss (L _IC ), and (b) represents the identification loss (L _IC ) and coherence loss (L _IC ). When learning is performed by applying, it indicates pixels that are mutually matched in heterogeneous images. And (c) shows mutually matched pixels when learning is performed by applying identification loss (L _IC ), coherence loss (L _IC ), and dense triple loss (L _DT ).

As shown in (c) of FIG. 9, when learning is performed by applying identification loss (L _IC ), coherence loss (L _IC ), and dense triple loss (L _DT ), each pixel of the human area in a heterogeneous image It can be seen that these are exactly matched to the positions corresponding to each other. And this indicates that the device for re-identifying a person in this embodiment can re-identify a person with very high accuracy.

10 shows simulation results of the performance of the apparatus for re-identification of a person according to the present embodiment.

As described above, each pixel of the human area in a heterogeneous image can be accurately matched to a position corresponding to each other, and as shown in (a) of FIG. Even in an image that includes many people, it is possible to accurately match corresponding pixels, so that people can be re-identified.

11 illustrates a method for re-identifying a person based on a heterogeneous sensor camera according to an embodiment of the present invention.

Referring to FIG. 11 , the method for re-identifying a person based on a heterogeneous sensor camera according to the present embodiment can be largely divided into a learning step S10 and a step S20 for re-identifying a person in a heterogeneous image. Referring to FIGS. 6 and 7 , the learning step (S10) is first described. First and second images, which are heterogeneous images obtained from a heterogeneous sensor camera and labeled with an identifier of a person included, are obtained as training data. (S11). In this case, each of the first image and the second image may be a single frame image or may be an image composed of multiple frames. Also, images containing the same person or images containing different people may be used.

In addition, first and second learning feature maps are obtained by extracting features of each of the first and second images according to the learned method (S12). Thereafter, a similarity map (C(p,q)) is obtained by calculating a similarity between pixels between the obtained first and second learning feature maps in a predetermined manner (S14). Here, the similarity map C(p,q) may be calculated according to Equation 1 based on, for example, cosine similarity.

When the similarity map (C(p,q)) is obtained, the matching probability for each pixel position in the first and second learning feature maps is calculated according to Equation 2 based on the obtained similarity map (C(p,q)). to obtain a matching probability map (P(p,q)) (S15). Then, pixels corresponding to each other in the first and second learning feature maps are soft-warped using the matching probability map (P(p,q)) to obtain a warped feature map (S15). At this time, soft warping obtains the first and second masks in which the human region is emphasized in advance in each of the first and second learning feature maps according to Equations 3 and 4, and as shown in Equations 5 and 6, the obtained first and using a corresponding mask among the second masks so that the pixels are warped in a concentrated manner in the human area of the feature map to be warped, and pixels of the existing feature map are maintained in the remaining areas. That is, the pixels of the first learning feature map are warped to the position corresponding to the human area of the second feature map using the second mask, and the pixels of the second learning feature map are maintained in the remaining area to obtain the second warped feature map. do. Pixels of the second learning feature map are warped to positions corresponding to the human region of the first feature map using the first mask, and pixels of the second learning feature map are maintained in the remaining regions to obtain the first warped feature map. do. When the first and second warping feature maps are obtained, the first and second learning feature maps and the first and second warping feature maps are pooled, respectively, to obtain the first and second descriptors and the first and second warping feature maps. Obtain (S16).

Thereafter, based on the obtained first and second descriptors and the first and second warping descriptors, respectively, a class representing a corresponding probability for each predefined class is classified, and an identifier corresponding to each is obtained (S17). As an example, the identifier may include first and second score maps and first and second warping score maps indicating probabilities that each of the first and second descriptors and the first and second warping descriptors correspond to each of a plurality of predetermined classes. It may be obtained by extracting and obtaining an identifier of a class showing the highest probability based on the obtained first and second scoremaps and the first and second warping scoremaps, respectively.

When the identifier is obtained, identification loss (L _ID ) and consistency loss (L _IC ) is calculated, and a dense triple loss (L _DT ) is calculated using the first and second feature maps and the first and second warping feature maps, and the total loss is calculated and back-propagated (S18).

Here, the identification loss (L _ID ) is calculated as the difference between the first and second scoremaps obtained from the first and second presenters and the identifiers labeled in the first and second training images, for example, the first and second scoremaps. 2 can be computed as a negative cross-entropy between the scoremap and the labeled identifiers.

In addition, the coherence loss (L _IC ) may be obtained by calculating a negative cross entropy between the first and second scoremaps and the first and second warping scoremaps.

On the other hand, dense triple loss (L _DT ) is based on the learning feature maps and warping feature maps corresponding to each other in the first and second learning feature maps and the first and second warping feature maps. Obtain first and second common attention maps emphasizing the human area, and on the common attention map, between the anchor, which is a learning feature map, and the positive, which is a warping feature map for the same identifier as the identifier of the anchor, and the negative, which is a warping feature map for other identifiers Calculate the error of to obtain the dense triple loss (L _DT ).

Then, the total loss is calculated by weighting the calculated identification loss (L _IC ), coherence loss (L _IC ), and dense triple loss (L _DT ), and learning is performed by backpropagation. Learning may be performed a predetermined number of times or repeatedly until the total loss becomes equal to or less than a predetermined reference loss.

On the other hand, when the learning is completed, it is applied for real person re-identification, and a heterogeneous image person re-identification step (S20) is performed.

In the heterogeneous image person re-identification step, first and second heterogeneous images for which person re-identification is to be performed are acquired (S21). In addition, first and second feature maps are obtained by extracting features from each of the obtained first and second images according to the learned method in the learning step (S22). Thereafter, first and second descriptors are acquired by pooling the first and second feature maps in a predetermined manner (S23). When the first and second presenters are obtained, first and second scoremaps indicating probabilities corresponding to each of a plurality of classes predefined so that the obtained first and second presenters correspond to a plurality of persons to be re-identified are obtained. and re-identify the person by extracting an identifier corresponding to the class with the highest probability based on the obtained score map.

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Here, computer readable media may be any available media that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, including read-only memory (ROM) dedicated memory), random access memory (RAM), compact disk (CD)-ROM, digital video disk (DVD)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is only exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: heterogeneous image acquisition unit 110: first image acquisition unit
120: second image acquisition unit 200: feature extraction unit
210: first feature extraction unit 220: second feature extraction unit
300: pulling unit 310: first pulling unit
320: second pooling unit 400: re-identification unit
500: cross-modal sorting unit 510: similarity measuring unit
520: Matching probability estimation unit 530: Mask providing unit
540: soft warping unit 550: warping pulling unit
600: loss calculator

Claims (20)

The first and second images acquired using different types of sensors are applied and features are extracted according to a learning method to obtain first and second feature maps, and the first and second feature maps are obtained by pooling in a predetermined method. a feature pooling unit that obtains presenters;
Provided during learning, receiving the first and second feature maps, estimating pixels corresponding to each other, and performing mutual warping based on the positions of the estimated corresponding pixels, so that the second feature map corresponds to the first feature Obtaining a first warped feature map warped to correspond to the map and a second warped feature map warped so that the first feature map corresponds to the second feature map, and pooling the first and second warped feature maps to obtain a first and a cross-modal arranging unit obtaining a second warping descriptor;
First and second score maps representing probabilities corresponding to each of a plurality of classes in which the first and second descriptors and the first and second warping descriptors are pre-specified according to the learned method, and the first and second descriptors a re-identification unit that obtains a warping score map and acquires an identifier of a class having the highest probability in the obtained first and second score maps; and
Provided during learning, the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map is positive in the case of the same identifier and different identifiers Including a loss calculation unit that calculates a dense triple loss by dividing by the negative of
The cross-modal alignment unit
a similarity measurement unit calculating a similarity between pixels between the first and second feature maps according to a predetermined method and obtaining a similarity map;
a matching probability estimator calculating a matching probability for each pixel of the first and second feature maps based on the similarity map to obtain a matching probability map;
Obtaining the first and second warping feature maps by weighting probabilities of matching pixels of the first and second feature maps using the matching probability map to each pixel of the relative feature map and warping the relative feature map a soft warping unit; and
and a warping pooling unit configured to acquire first and second warping descriptors by pooling the first and second warping feature maps in a predetermined manner.
delete The method of claim 1, wherein the cross-modal alignment unit
and a mask providing unit configured to obtain first and second masks by normalizing a size of a pixel at each position of the first and second feature maps. The method of claim 3, wherein the soft warping unit
When the first and second masks are applied, probabilities of matching the pixels of each of the first and second feature maps to each pixel of the relative feature map are weighted and aggregated, and the pixel value of the mask is added to the weighted aggregated warping pixel value. A device for re-identifying people who warp by weighting with The method of claim 4, wherein the mask providing unit
The first mask M _RGB is obtained from the first feature map f _RGB by Equation

(Where │ mer ₂ is the L ₂ norm function, and f is the maximum-minimum regularization function.)
obtained according to
The soft warping part
The first warping feature map (from the first feature map f _RGB ), the second feature map f _IR , and the first mask M _RGB

) to the equation

(Where W is according to the matching probability map (P(p,q))

It is a soft warping function represented by .)
Acquiring Person Re-Identification Devices. The method of claim 4, wherein the loss calculator
Obtain first and second common attentional maps emphasizing human regions identifiable in both the first and second images, and estimate the same identifier as the identifier of the first feature map, which is an anchor in the first common attentional map. The difference between the positive, which is the first warping feature map, and the negative, which is the first warping feature map estimated with different identifiers, and the identifier of the second feature map, which is the anchor in the second common attention map, Estimated to be the same identifier Apparatus for calculating the dense triple loss by accumulating differences between positive and negative values that are second warping feature maps estimated with different identifiers. The method of claim 6, wherein the loss calculator
Equation for the dense triple loss (L _DT ) based on the first and second common attentional maps (A _i , i ∈ {RGB, IR})

(where α represents a predefined margin, and di ₊ ⁽ p) and _di ^- (p) are respectively

represents the error between the anchor ( _fi ^a ) and the positive ( _fi ^p ) and the error between the anchor ( _fi ^a ) and the negative (f _i ⁿ ), respectively, which is calculated as
Person re-identification device counting according to. The method of claim 7, wherein the loss calculator
Identification loss according to a difference between each of the first and second scoremaps and identifiers that are labeled and applied to the first and second images during learning and the first and second scoremaps and the first and second warping A person re-identification apparatus for performing learning by further calculating a coherence loss according to a difference between scoremaps, and backpropagating a total loss calculated by weighting the identification loss, the coherence loss, and the dense triplet loss. The method of claim 8, wherein the loss calculator
The identification loss is obtained by calculating a negative cross entropy between the first and second scoremaps and an identifier labeled and applied to the first and second images, and An apparatus for calculating the coherence loss by calculating a negative cross entropy between corresponding warping score maps of the first and second warping score maps. The apparatus of claim 1, wherein the first image is an RGB image obtained from a visible camera and the second image is an IR image obtained from an IR camera. learning phase; and
A person re-identification step of acquiring an identifier for a person included in each image by receiving images acquired using different types of sensors using an artificial neural network,
The learning phase is
As learning data, first and second images obtained by using different types of sensors and pre-labeled identifiers are applied, and features are extracted according to a learning method to obtain first and second feature maps, and pre-specified methods are used. obtaining first and second presenters by pooling with ;
The first and second feature maps are received, pixels corresponding to each other are estimated, and mutual warping is performed based on the positions of the estimated corresponding pixels, so that the second feature map is warped to correspond to the first feature map. Obtaining a first warped feature map and a second warped feature map warped so that the first feature map corresponds to the second feature map, and pooling the first and second warped feature maps to obtain first and second warped descriptors obtaining;
Obtaining first and second scoremaps and first and second warping scoremaps indicating probabilities corresponding to each of a plurality of classes in which the first and second descriptors and the first and second warping descriptors are pre-specified, respectively and obtaining an identifier of a class having the highest probability in the obtained first and second score maps; and
Loss by dividing the difference between the first feature map and the first warping feature map and the difference between the second feature map and the second warping feature map into a positive in the case of the same identifier and a negative in the case of different identifiers Including the step of calculating
The step of obtaining the first and second warping descriptors is
obtaining a similarity map by calculating a similarity between pixels between the first and second feature maps according to a predetermined method;
obtaining a matching probability map by calculating a matching probability for each pixel of the first and second feature maps based on the similarity map;
Obtaining the first and second warping feature maps by weighting probabilities of matching pixels of the first and second feature maps using the matching probability map to each pixel of the relative feature map and warping the relative feature map doing; and
and pooling the first and second warped feature maps in a predetermined manner to obtain first and second warped descriptors.
delete 12. The method of claim 11, wherein obtaining the first and second warped descriptors comprises:
The method of re-identifying a person further comprising obtaining first and second masks by normalizing a pixel size for each position of the first and second feature maps. 14. The method of claim 13, wherein obtaining the first and second warping feature maps comprises:
weighting a probability that pixels of each of the first and second feature maps match each pixel of a relative feature map by receiving the first and second masks; and
A method for re-identifying a person comprising the step of additionally weighting the pixel values of a mask on the weighted aggregated warping pixel values and performing warping. 15. The method of claim 14, wherein obtaining the mask comprises:
The first mask M _RGB is obtained from the first feature map f _RGB by Equation

(Where │ mer ₂ is the L ₂ norm function, and f is the maximum-minimum regularization function.)
obtained according to
The step of warping with additional weighting
The first warping feature map (from the first feature map f _RGB ), the second feature map f _IR , and the first mask M _RGB

) to the equation

(Where W is according to the matching probability map (P(p,q))

It is a soft warping function represented by .)
How to Re-Identify Persons Obtaining Under. 15. The method of claim 14, wherein calculating the loss comprises
obtaining first and second common attentional maps emphasizing a human region identifiable in both the first and second images; and
The difference between each positive, which is a first warping feature map estimated with the same identifier as the identifier of the first feature map, which is an anchor in the first common attention map, and each negative, which is a first warping feature map estimated with different identifiers, and the Dense triple loss calculated by accumulating the difference between each positive estimated with the same identifier as the identifier of the second feature map, which is an anchor in the second common attention map, and each negative, the second warping feature map estimated with different identifiers A person re-identification method comprising the step of obtaining. 17. The method of claim 16, wherein obtaining the dense triple loss comprises:
Equation for the dense triple loss (L _DT ) based on the first and second common attentional maps (A _i , i ∈ {RGB, IR})

(where α represents a predefined margin, and di ₊ ⁽ p) and _di ^- (p) are respectively

represents the error between the anchor ( _fi ^a ) and the positive ( _fi ^p ) and the error between the anchor ( _fi ^a ) and the negative (f _i ⁿ ), respectively, which is calculated as
Person re-identification method counting according to. 18. The method of claim 17, wherein calculating the loss comprises
calculating an identification loss according to a difference between each of the first and second score maps and identifiers that are labeled and applied to the first and second images during training;
calculating a consistency loss according to a difference between the first and second scoremaps and the first and second warping scoremaps; and
and backpropagating a total loss calculated by a weighted sum of the identification loss, the coherence loss, and the dense triple loss. 19. The method of claim 18, wherein calculating the identification loss comprises:
Obtaining the identification loss by calculating a negative cross entropy between the first and second scoremaps and identifiers labeled and applied to the first and second images;
The step of calculating the coherence loss is
wherein the loss of consistency is calculated by calculating a negative cross entropy between each of the first and second scoremaps and a corresponding warping scoremap of the first and second warping scoremaps. 12. The method of claim 11, wherein the person re-identification step
obtaining a feature map by extracting features of each of the heterogeneous images applied according to a method pre-learned in the learning step;
acquiring a corresponding descriptor by pooling feature maps obtained from each of the heterogeneous images in a predetermined manner; and
A method for re-identifying a person, comprising acquiring a scoremap corresponding to each of the acquired expressers according to a pre-learned method, and acquiring an identifier for a person included in each of the authorized heterogeneous images based on the acquired scoremap. .

Images