JP2021039740A - Pedestrian re-identification device and method - Google Patents

Abstract

To provide a pedestrian re-identification device, method, and computer storage medium, which improve accuracy of pedestrian re-identification in scenes where a pedestrian changed his/her posture significantly or is identified by different cameras.SOLUTION: A pedestrian re-identification device 100 is provided, comprising: a pedestrian detection unit configured to detect pedestrians from each video frame of a video sequence; a feature extraction unit configured to extract appearance features of each pedestrian detected in the video frames; and a pedestrian identification unit configured to match extracted appearance features of each pedestrian with target appearance features of a target pedestrian, and identify the target pedestrian in the video frames based on a matching result. The target appearance features of the target pedestrian are generated based on initial appearance features and most recent appearance features of the target pedestrian.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to image processing, specifically to pedestrian re-identification devices, methods and computer storage media.

Pedestrian re-identification is a technique for determining whether or not a specific target pedestrian exists by analyzing an image sequence or a video sequence from a camera in which multiple fields of view overlap or do not overlap. Is. Unlike normal pedestrian tracking, pedestrian re-identification involves specific in image or video sequences taken by different cameras to allow long-term tracking and surveillance of a particular target pedestrian. Since it can identify the target pedestrian, it has a very good prospect of application in the field of surveillance and crime detection work.

Currently, pedestrian re-identification is detected by matching the characteristics of each pedestrian detected from image sequences and video sequences with the characteristics of known target pedestrians in a pre-built target pedestrian library. It is common to determine whether a pedestrian is a known target pedestrian, but the effect is often poor, and the main causes are the shooting angle between different cameras, the viewing range, and so on. Due to differences in lighting conditions, the posture and appearance of the same pedestrian in images taken by different cameras may differ significantly, and pedestrians are often highly mobile, so the posture and appearance also change slightly over time. The pre-constructed target pedestrian has a single characteristic, and misidentification is likely to occur when the posture or appearance of the same pedestrian changes.

According to one aspect of the present disclosure, a pedestrian detection unit configured to detect a pedestrian in each video frame of a video sequence and an appearance feature extracted from each pedestrian detected in the video frame. The configured feature extraction unit matches the extracted appearance features of each pedestrian with the target appearance features of the target pedestrian, and the pedestrian is configured to identify the target pedestrian from the video frame based on the matching result. A pedestrian re-identification device is provided that includes an identification unit and the target pedestrian's target appearance features are generated based on the target pedestrian's initial appearance features and recent appearance features.

According to another aspect of the present disclosure, it comprises a processor and a memory in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor walks to each video frame of the video sequence. A step of detecting a person, a step of extracting appearance features for each pedestrian detected from a video frame, and a step of matching the extracted appearance features of each pedestrian with the target appearance features of the target pedestrian, and the matching result is obtained. A step of identifying a target pedestrian from a video frame based on is performed, and the target pedestrian target appearance feature is a pedestrian reidentification generated based on the target pedestrian's initial appearance feature and recent appearance feature. The device is provided.

According to another aspect of the present disclosure, a step of detecting a pedestrian in each video frame of the video sequence, a step of extracting appearance features for each pedestrian detected from the video frame, and each extracted pedestrian. The target pedestrian's target appearance feature is the target pedestrian's target appearance feature, including a step of matching the appearance feature of the target pedestrian with the target pedestrian's target appearance feature and identifying the target pedestrian from the video frame based on the matching result. Provided are pedestrian re-identification methods generated based on initial appearance features and recent appearance features.

According to another aspect of the present disclosure, a computer-readable storage medium is provided in which a computer program instruction is stored and when the computer program instruction is executed by a processor, the computer performing the pedestrian reidentification method described above is performed. ..

The above-mentioned devices, methods and computer-readable storage media of the present disclosure improve the accuracy of re-identification for a pedestrian when the pedestrian's posture changes significantly or is identified by a different camera.

These and / or other aspects and advantages of the present disclosure, together with the accompanying drawings, will become clearer and easier to understand from the following detailed description of the embodiments of the present disclosure.
An example of misidentification when reidentifying a pedestrian based on the initial appearance characteristics of the target pedestrian is shown. An example of misidentification when reidentifying a pedestrian based on the recent appearance characteristics of the target pedestrian is shown. A block diagram of a configuration example of a pedestrian re-identification device according to an embodiment of the present disclosure is shown. The schematic diagram which re-identifies a pedestrian based on the target appearance feature of the target pedestrian in the pedestrian re-identification device which concerns on embodiment of this disclosure is shown. A hardware block diagram of an example of a computing device for realizing the pedestrian re-identification device according to the embodiment of the present disclosure is shown. A schematic flowchart of the pedestrian re-identification method according to the embodiment of the present disclosure is shown. The schematic diagram of the verification result of the pedestrian re-identification method which concerns on the prior art and the Example of this disclosure is shown.

In order for those skilled in the art to better understand the disclosure, the disclosure will be described in more detail below with reference to the accompanying drawings and detailed embodiments.

The technical proposals in the examples of the present disclosure are clearly and completely described below with reference to the accompanying drawings in the examples of the present disclosure, and the examples described are only a part of the examples of the present disclosure. , It is clear that not all examples. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present disclosure.

As described above, the re-identification of existing pedestrians matches the characteristics of each pedestrian detected from the image sequence or video sequence with the characteristics of known target pedestrians in the pre-constructed target pedestrian library. Therefore, it is common to determine whether or not the detected pedestrian corresponds to a known target pedestrian, but for various reasons such as changes in the pedestrian's posture, the conventional pedestrian re-identification method is used. , It becomes easy to misidentify pedestrians. In particular, when the target pedestrian library for feature matching has a single feature, if the posture or appearance of the same pedestrian changes significantly, it may be misidentified as another pedestrian, and further. , Misidentification may continue. An example of pedestrian misidentification that may occur when reidentifying a pedestrian will be described below with reference to FIGS. 1 and 2.

FIG. 1 shows an example of pedestrian misidentification that occurs when feature matching is performed based on the initial appearance features of a target pedestrian. In this case, for the target pedestrian in the target pedestrian library, the feature at the time of the first appearance is stored in the target pedestrian library as the initial appearance feature of the target pedestrian ID, and thereafter, matching with the initial appearance feature of this target pedestrian. I do. As shown in FIG. 1, among three (continuous or discontinuous) images of the same pedestrian taken by cameras with overlapping or non-overlapping visual fields, the target pedestrian is designated as ID1 in the first image. The features in the first image are identified and used for subsequent matching as the initial appearance features of this target pedestrian ID1, but since the pedestrian's posture, shooting angle, etc. have changed significantly since then, in the second image. This target pedestrian is erroneously identified as ID2, and this target pedestrian is erroneously identified as ID3 in the third image. In this case, since the initial appearance characteristics of the target pedestrian used for matching are constant, it can be seen that it is not possible to respond well to a situation in which the pedestrian posture and the imaging angle change significantly.

FIG. 2 is a diagram showing an example of pedestrian misidentification that occurs when feature matching is performed based on the recent appearance features of a target pedestrian. In this case, the feature of the target pedestrian when it appeared most recently is stored in the target pedestrian library as the recent appearance feature of the target pedestrian ID, and the feature of the target pedestrian at the time of the previous appearance is deleted. Match the recent appearance features of the target pedestrian. As shown in FIG. 2, of the three images of the same pedestrian photographed by cameras with overlapping or non-overlapping visual fields, the target pedestrian is identified as ID1 in the first image and appears this time. The feature of time may be the recent appearance feature of the target pedestrian ID1, and then, for various reasons such as overlapping pedestrians, this target pedestrian is misidentified as ID2 in the second image, and the book The feature at the time of appearance will be the recent appearance feature of the target pedestrian ID2, replacing the previously stored appearance feature and maintaining the erroneous result of identifying the target pedestrian ID1 as ID2 in the third image. .. In this case, recent matching based on appearance features can respond to changes in the posture and imaging angle of pedestrians to some extent, but if misidentification occurs due to overlapping of pedestrians, the result of the misidentification is not corrected. It tends to be maintained.

Regarding such circumstances, in the present disclosure, the characteristics of the target pedestrian are constructed by comprehensively considering the initial appearance characteristics of the target pedestrian and the recent appearance characteristics, and the characteristics are matched with the detected characteristics of the pedestrian. By using it, it is possible to provide an identification result with sufficient accuracy even in a scene in which the posture of a pedestrian changes significantly or a scene in which the pedestrian is identified by a different camera.

Hereinafter, the pedestrian re-identification device according to the embodiment of the present disclosure will be described with reference to FIG. FIG. 3 shows a block diagram of a configuration example of the pedestrian re-identification device 100 according to the embodiment of the present disclosure. As shown in FIG. 3, the pedestrian re-identification device 100 may include a pedestrian detection unit 110, a feature extraction unit 120, and a pedestrian identification unit 130. Hereinafter, the main functions of each unit of the pedestrian re-identification device 100 will be described.

The pedestrian detection unit 110 can detect a pedestrian from each video frame of the video sequence. As described above, in the pedestrian re-identification technique according to the present disclosure, unlike ordinary pedestrian tracking, a specific target pedestrian is identified from an image sequence or a video sequence taken by a different camera, and a specific target pedestrian walks. Allows long-term tracking and monitoring of individuals. More specifically, the video sequence is a video sequence to be analyzed that requires the presence or absence of a target pedestrian to be identified, from which the video sequence and the previously captured target pedestrian were captured. The video frame from which the initial appearance feature or the latest appearance feature has been extracted may be shot by a different camera, or may be shot by the same camera at different times.

Specifically, the video sequence analyzed by the pedestrian detection unit 110 may include one or more video frames, each video frame being a single camera or multiple overlapping or non-overlapping visual fields. It may be a continuous or discontinuous video frame taken by the camera. The pedestrian detection unit 110 can detect a pedestrian from each video frame of a video sequence using any suitable image detection technique in the art, which is not limited herein. For example, the pedestrian detection unit 110 performs processing such as foreground segmentation, edge extraction, and motion detection for each video frame, and identifies each sub-image area corresponding to each pedestrian appearing in this video frame. For example, the sub-image area of this video frame may be represented by a rectangular frame circumscribing the detected contour of the pedestrian's body. Further, for example, the pedestrian detection unit 110 detects a pedestrian for each video frame by a pre-trained pedestrian detection classifier using a machine learning method such as a neural network or a support vector machine. , The position of each pedestrian appearing in the video frame may be specified.

The feature extraction unit 120 may extract appearance features for each pedestrian detected from the video frame. In the present disclosure, the appearance features may include features that reflect the appearance morphology of the pedestrian, such as color features, texture features, shape features, and face features, and the present disclosure does not limit this. The above methods for extracting various features are well known in the art, and description thereof will be omitted here.

The pedestrian identification unit 130 may match the detected appearance features of each pedestrian with the target appearance features of the target pedestrian, and identify the target pedestrian from the video frame based on the matching result. Specifically, the pedestrian identification unit 130 matches the detected appearance characteristics of each pedestrian with the appearance characteristics of the target pedestrian, and the pedestrian detected from the video frame based on the similarity between the two is the pedestrian. The pedestrian may be re-identified by determining whether or not it corresponds to the target pedestrian. Optionally, the number of target pedestrians may be plural, and correspondingly, the pedestrian identification unit 130 sets the appearance characteristics of each detected pedestrian to the appearance of each target pedestrian constructed. Pedestrians are re-identified by matching each feature and determining which of the target pedestrians the pedestrian detected from the video frame corresponds to based on the similarity between the two. May be good. The target pedestrian's target appearance features are included in a pre-built pedestrian appearance feature library, which is accessible locally or by the pedestrian reidentification device. It can be understood that it can be stored in various servers. Alternatively, the target appearance feature of the target pedestrian may be generated from a photograph and input from the outside, if necessary, without being stored in the pedestrian appearance feature library in advance. In the present disclosure, the method of storing or acquiring the target appearance feature of the target pedestrian is not limited.

As described above, when identification is performed based only on the initial appearance characteristics of the target pedestrian, it is not possible to respond well to the situation where the posture and imaging angle of the pedestrian have changed, and only the recent appearance characteristics of the target pedestrian are used. If misidentification occurs, it becomes difficult to correct the identification. In the present disclosure, by generating this target pedestrian target appearance feature based on both the target pedestrian's initial appearance feature and the recent appearance feature, the generated feature is the unique appearance characteristic of the target pedestrian. , The change in appearance and posture can be sufficiently reflected, and the accuracy of pedestrian re-identification can be ensured. The initial appearance features include appearance features pre-extracted from an initial video frame containing the target pedestrian, and the recent appearance features may include appearance features pre-extracted from a recent video frame containing the target pedestrian. Good. In the present disclosure, the initial appearance feature of the target pedestrian and the recent appearance feature may be fused to generate the target appearance feature of the target pedestrian, and the detected pedestrian feature may be matched with the target appearance feature. Hereinafter, the target appearance features of the target pedestrian, which the pedestrian identification unit 130 is based on when performing feature matching, will be described.

According to one embodiment of the present disclosure, the initial appearance feature of the target pedestrian and the recent appearance feature can be weighted and fused to obtain the target appearance feature of the target pedestrian based on a preset weight. .. In the embodiments of the present disclosure, the initial appearance features of the target pedestrian and the recent appearance features can be weighted and fused in various ways based on preset weights, but for completeness of description. Two exemplary methods will be described below.

In one example, first, the number of initial video frames and the number of recent video frames of the target pedestrian used for weighting fusion may be determined based on preset weights. Specifically, the number of initial video frames of the target pedestrian is determined by this preset weight, taking into account the specific gravity of both the initial appearance feature and the recent appearance feature of the target pedestrian at the time of weighting fusion. You may determine the relative number of recent video frames to. This preset weight can be obtained by various methods, for example, using empirical values or by machine learning, and this is not limited in the present disclosure. Then, the weighted average value of each appearance feature of this target pedestrian in the determined number of initial video frames and the determined number of recent video frames can be calculated. Specifically, using the above-mentioned feature extraction method such as color feature, texture feature, shape feature, face feature, etc., the appearance of the sub-image area including the target pedestrian out of the determined number of video frames. The features may be extracted and the average value of each extracted appearance feature may be calculated as the target appearance feature of the target pedestrian. As an example, if the weights of the initial appearance feature and the recent appearance feature are 0.3: 0.7, then the number of initial video frames and the number of recent video frames required for weighted fusion are 3 and 7 frames, respectively. Further, the average value of the appearance features of the target pedestrian in a total of 10 video frames may be used as the target appearance features of the target pedestrian.

In another example, first, a predetermined number of initial video frames and recent video frames of the target pedestrian may be acquired, for example, 5 initial video frames and 5 recent video frames of the target pedestrian. You may get it. Then, at the time of weighting fusion, different weights are given to the initial video frame and the appearance features extracted from the recently video frames. For example, when the weights of the initial appearance features and the recent appearance features are 0.8: 0.2. The sum of 0.8 times the average value of the initial appearance feature of the target pedestrian in the initial video frame of 5 frames and 0.2 times the average value of the recent appearance feature of the target pedestrian in the recent video frame of 5 frames. May be calculated as the target appearance feature of the target pedestrian.

Note that the weights and the number of video frames in the above examples are both exemplary, and in the present disclosure, based on preset weights, depending on the actual calculation speed and accuracy requirements, The initial appearance features of the target pedestrian in any number of initial video frames may be weighted and fused with the recent appearance features in any number of recent video frames.

According to another embodiment of the present disclosure, the initial appearance feature and the recent appearance feature of the target pedestrian are cascaded, and the initial appearance feature and the recent appearance feature are convoluted by performing a convolution operation on the cascaded appearance feature. The appearance features can be cascaded and fused to form the target appearance features of this target pedestrian. Various methods can be used to cascade and fuse the initial appearance features of the target pedestrian with the recent appearance features, but to illustrate the integrity, one exemplary method is described below. To do. First, the initial appearance feature and the recent appearance feature are cascaded in the channel dimension. Then, a convolution operation, for example, a 1 × 1 convolution layer, is applied to the cascaded features to fuse the cascaded features and to make the dimensions of the cascaded features at least the same as the dimensions before the cascaded features. It can be compressed, which allows the features of each channel to be fused while reducing the amount of computation by reducing the dimensions.

The example of the target appearance feature of the target pedestrian based on the re-identification of the pedestrian by the pedestrian identification unit 130 according to the embodiment of the present disclosure has been described above. The pedestrian identification operation will be specifically described. As described above, the pedestrian identification unit 130 matches the detected appearance features of each pedestrian with the target appearance features of the target pedestrian, and identifies the target pedestrian from the video frame based on the similarity between the two. You may. Specifically, the pedestrian identification unit 130 is a feature distance between the detected appearance feature of each pedestrian and the target appearance feature of the target pedestrian, for example, Manhattan distance or Euclidean distance. ), Bhattacharyya distance, etc. may be calculated and compared with a preset threshold value to determine whether or not the detected pedestrian corresponds to the target pedestrian. Optionally, the number of target pedestrians may be plural, in which case the pedestrian identification unit 130 calculates the characteristic distances of each of the detected pedestrians with each target pedestrian, and each of the detected pedestrians. A target that compares the feature distance with a preset threshold, identifies the closest feature distance from the feature distances lower than the preset threshold, and assigns the detected pedestrian to the closest feature distance. It may be identified as a pedestrian. The pedestrian identification unit 130 may output a pedestrian re-identification result based on the matching result. For example, when the detected pedestrian corresponds to a target pedestrian, the detected pedestrian and the target. It associates with the pedestrian's identity and assigns a new identity to the detected pedestrian if the detected pedestrian cannot match the target pedestrian.

Hereinafter, an example in which the pedestrian identification unit 130 according to the embodiment of the present disclosure reidentifies a pedestrian based on the target appearance feature of the target pedestrian will be described with reference to FIG. As shown in FIG. 4, the first image on the left side is the initial video frame of the target pedestrian ID1, and the second image in the center is the recent video frame of the target pedestrian ID1, the initial video frame and The initial appearance feature and the recent appearance feature of the target pedestrian ID1 are extracted from the recent video frame, and the initial appearance feature and the recent appearance feature are fused to obtain the target appearance feature of the target pedestrian ID1. Is used not only to track the latest posture and appearance changes of this target pedestrian, but may also retain the pedestrian's initial appearance characteristics. After that, as shown in the third image on the right side of FIG. 4, even when the posture and appearance of the pedestrian change significantly, the accuracy of the pedestrian is set as ID1 based on the target appearance feature. Can be identified well.

Optionally, the pedestrian re-identification device 100 may further include a feature update unit (not shown). The feature updating unit may update the target appearance feature of the target pedestrian based on the identification result by the pedestrian identification unit 130. For example, if the pedestrian identification unit 130 determines that a pedestrian detected from a video frame corresponds to a target pedestrian and associates the detected pedestrian with the identity of the target pedestrian, the feature update unit , The detected pedestrian appearance feature in the video frame is taken as the newest recent appearance feature of the target pedestrian, and the initial appearance feature of the target pedestrian and the newest recent appearance feature are fused to form the target pedestrian. By generating a new target appearance feature or sending the newest recent appearance feature to a server that maintains the target pedestrian's target appearance feature, the target walk is based on the latest recent appearance feature received by the server. By updating the target appearance feature of the person, the updated feature may be able to follow the recent changes in the posture and appearance of the target pedestrian in real time. Further, when the pedestrian identification unit 130 determines that the pedestrian detected from the video frame cannot be matched with the target pedestrian and assigns a new identity to the detected pedestrian, the feature update unit is the pedestrian. May be set as a new target pedestrian, and the target appearance feature may be constructed for subsequent feature matching.

As described above, the pedestrian re-identification device 100 according to the embodiment of the present disclosure identifies the target pedestrian using the initial appearance feature and the recent appearance feature of the target pedestrian, thereby identifying the target pedestrian recently. It can effectively follow changes in posture and appearance, effectively correct errors based on the initial characteristics of the target pedestrian, and in scenes where the pedestrian's posture has changed significantly or identified by different cameras. The accuracy of pedestrian re-identification can be improved.

The block diagram used in the description of the above embodiment shows a block of functional units. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly by two or more physically and / or logically separated devices. (For example, wired and / or wireless) may be connected and realized by these a plurality of devices.

FIG. 5 is a hardware flock diagram showing an example of a computer device of the pedestrian re-identification device according to the embodiment of the present disclosure. The pedestrian re-identification device 100 may be physically configured as a computer device including a processor 501, a memory 502, a storage 503, a communication device 504, an input device 505, an output device 506, a bus 507, and the like.

In the following description, the word "device" can be read as a circuit, a device, a unit, or the like. The hardware configuration of the pedestrian re-identification device 100 may be configured to include one or more of the devices shown in the figure, or may be configured not to include some of the devices.

For example, although only one processor 501 is shown, it may be a plurality of processors. Further, the processing may be executed by one processor, or the processing may be executed simultaneously, sequentially, or by another method by one or more processors. Further, the processor 501 may be mounted by one or more chips.

For each function of the pedestrian re-identification device 100, by loading predetermined software (program) on hardware such as the processor 501 and the memory 502, the processor 501 performs an calculation, and communication by the communication device 504 and the memory 502 And by controlling the read and / or write of data in the storage 503.

Processor 501, for example, operates an operating system to control the entire computer. The processor 501 may be composed of a central processing unit (CPU: Central Processing Unit) including an interface with peripheral devices, a control device, an arithmetic unit, a register, and the like. For example, the pedestrian detection unit 110, the feature extraction unit 120, the pedestrian identification unit, and the like may be realized by the processor 501.

Further, the processor 501 reads a program (program code), a software module, and data from the storage 503 and / or the communication device 504 into the memory 502, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described in the above-described embodiment is used.

The memory 502 is a computer-readable recording medium, and is, for example, a ROM (Read Only Memory), an EPROM (Erasable Program ROM), an EPROM (Electrically Erasable Program ROM), a RAM (Random Access Memory), or a RAM (Random Access Memory). It may be composed of at least one of. The memory 502 may be called a register, a cache, a main memory (main storage device), or the like. The memory 502 can store a program (program code), a software module, or the like that can be executed to carry out the pedestrian re-identification method according to the embodiment of the present disclosure.

The storage 503 is a computer-readable recording medium, such as a flexible disk, a floppy disk, an optical magnetic disk (for example, a CD-ROM (Compact Disk ROM), etc.), and the like. Digital versatile disks, Blu-ray® disks, removable disks, optical disc drives, smart cards, flash memory devices (eg cards, sticks, key drivers), magnetic strips, databases, servers, etc. It may be composed of at least one of suitable storage media. The storage 503 may be referred to as an auxiliary storage device.

The communication device 504 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like. The communication device 504 includes, for example, a high frequency switch, a duplexer, a filter, a frequency synthesizer, etc. in order to realize frequency division duplex (FDD, Frequency Division Duplex) and / or time division duplex (TDD, Time Division Duplex). obtain. For example, the video sequence to be analyzed may be received via the communication device 504.

The input device 505 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside. The output device 506 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 505 and the output device 506 may have an integrated configuration (for example, a touch panel).

Further, each device such as the processor 501 and the memory 502 is connected by a bus 507 for communicating information. The bus 507 may be composed of a single bus or may be composed of different buses between the devices.

Further, the pedestrian re-identification device 100 includes a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device) hardware, an FPGA (FGA), and the like. It may be configured to include, and a part or all of each functional block may be realized by the hardware. For example, processor 501 may be implemented on at least one of these hardware.

Hereinafter, a flowchart of the pedestrian re-identification method according to the embodiment of the present disclosure will be described with reference to FIG.

As shown in FIG. 6, in step S601, a pedestrian is detected from each video frame of the video sequence. As described above, in the pedestrian re-identification technique according to the present disclosure, unlike ordinary pedestrian tracking, a specific target pedestrian is identified from an image sequence or a video sequence taken by a different camera, and a specific target pedestrian walks. Allows long-term tracking and monitoring of individuals. The video sequence is a target video sequence to be analyzed that requires the presence or absence of a target pedestrian to be identified, and the video sequence and its initial or recent appearance from which the target pedestrian was previously photographed. The feature-extracted video frame may be shot by different cameras, or may be shot by the same camera at different times. The video sequence may include one or more video frames, where each video frame is a continuous or discontinuous video taken by a single camera or multiple cameras with overlapping or non-overlapping fields of view. It may be a frame.

In that step, pedestrians can be detected from each video frame of the video sequence using any suitable image detection technique in the art, which is not limited herein. For example, for each video frame, processing such as foreground segmentation, edge extraction, and motion detection may be performed to specify each sub-image area corresponding to each pedestrian appearing in this video frame. The sub-image area of this video frame may be represented by a rectangular frame circumscribing the detected contour of the pedestrian's body. Further, for example, by using a machine learning method such as a neural network or a support vector machine to detect a pedestrian for each video frame by a pedestrian detection classifier trained in advance, the pedestrian appears in the video frame. The position of each pedestrian may be specified.

In step S602, appearance features may be extracted for each pedestrian detected from the video frame. Appearance features may include features that reflect the pedestrian's appearance morphology, such as color features, texture features, shape features, and face features, and are not limited to this in the present disclosure. The step may extract the appearance features of each pedestrian detected by adopting a feature extraction method well known in the art, and description thereof will be omitted here.

In step S603, the detected appearance features of each pedestrian and the target appearance features of the target pedestrian are matched, and the target pedestrian is identified from the video frame based on the matching result. In this step, the appearance characteristics of each detected pedestrian are matched with the appearance characteristics of the target pedestrian, and whether or not the pedestrian detected from the video frame corresponds to the target pedestrian based on the similarity between the two. The pedestrian may be re-identified by determining the above. Optionally, the number of target pedestrians may be plural, and the appearance features of each pedestrian detected in the step correspond to the appearance features of each target pedestrian constructed. Matching may be performed, and based on the similarity between the two, it may be determined whether or not the pedestrian detected from the video frame corresponds to each target pedestrian, and the pedestrian may be re-identified. The target pedestrian's target appearance features are included in a pre-built pedestrian appearance feature library, which is accessible locally or by the pedestrian reidentification device. It can be understood that it can be stored in various servers. Alternatively, the target appearance feature of the target pedestrian may be generated from a photograph and input from the outside, if necessary, without being stored in the pedestrian appearance feature library in advance.

In this step, the target pedestrian's target appearance feature, which is the basis for feature matching, is generated based on both the initial appearance feature and the recent appearance feature of this target pedestrian, so that the generated feature is the target. It is possible to sufficiently reflect the pedestrian's unique appearance characteristics and changes in the appearance and posture, and to ensure the accuracy of pedestrian re-identification. The initial appearance features include appearance features pre-extracted from an initial video frame containing the target pedestrian, and the recent appearance features may include appearance features pre-extracted from a recent video frame containing the target pedestrian. Good. The initial appearance feature of the target pedestrian may be fused with the recent appearance feature to generate the target appearance feature of the target pedestrian and match the detected pedestrian feature with it. The target appearance characteristics of the target pedestrian based on the feature matching in the step are similar to the contents described for the pedestrian identification unit 130 of the pedestrian re-identification device 100, and will be briefly described below. ..

According to one embodiment of the present disclosure, the initial appearance feature of the target pedestrian and the recent appearance feature can be weighted and fused to obtain the target appearance feature of the target pedestrian based on a preset weight. .. In the embodiments of the present disclosure, the initial appearance features of the target pedestrian and the recent appearance features can be weighted and fused in various ways based on preset weights. In one example, first, the number of initial video frames and the number of recent video frames of the target pedestrian used for weighting fusion may be determined based on preset weights. Then, the weighted average value of each appearance feature of the target pedestrian in the determined number of initial video frames and the determined number of recent video frames may be calculated. In another example, first, a predetermined number of initial video frames and recent video frames of the target pedestrian may be acquired. Then, at the time of weighting fusion, different weights may be given to the appearance features extracted from the initial video frame and the recently video frame, and the target appearance feature of the target pedestrian may be obtained by the weighted average.

According to another embodiment of the present disclosure, the initial appearance feature and the recent appearance feature of the target pedestrian are cascaded, and the initial appearance feature and the recent appearance feature are convoluted by performing a convolution operation on the cascaded appearance feature. The appearance features can be cascaded and fused to form the target appearance features of this target pedestrian.

The specific operation of returning to step S603 and re-identifying the pedestrian based on the above-mentioned target appearance feature will be described in detail below. For example, the feature distance between the detected appearance feature of each pedestrian and the target appearance feature of the target pedestrian, such as Manhattan distance, Euclidean distance, Bhattacharyya distance, etc. By calculating and comparing with a preset threshold, it may be determined whether or not the detected pedestrian corresponds to the target pedestrian. Optionally, the number of target pedestrians may be plural, in which case, for the detected pedestrians, the characteristic distances of each target pedestrian are calculated, and the characteristic distances and the respective characteristic distances are preset. Even if the pedestrian is compared with the threshold value, the closest feature distance is specified from the feature distances lower than the preset threshold value, and the detected pedestrian is identified as the target pedestrian corresponding to the closest feature distance. Good. Next, the pedestrian re-identification may be output based on this matching result. For example, when the detected pedestrian corresponds to a target pedestrian, the detected pedestrian and the idea of the target pedestrian Associate with the entity and assign a new identity to the detected pedestrian if the detected pedestrian cannot match the target pedestrian.

Optionally, the pedestrian re-identification method further includes the step of updating the target appearance feature of the target pedestrian based on the identification result. For example, when determining that a pedestrian detected from a video frame corresponds to a target pedestrian as described above and associating the detected pedestrian with the identity of the target pedestrian, the detected pedestrian in the video frame. The appearance feature of the target pedestrian is set as the newest recent appearance feature of the target pedestrian, and the initial appearance feature of the target pedestrian is fused with the newest recent appearance feature to generate a new target appearance feature of the target pedestrian. By doing so, the updated feature may be able to follow the recent changes in the posture and appearance of the target pedestrian in real time. In addition, as described above, when it is determined that the pedestrian detected from the video frame cannot be matched with the target pedestrian and a new identity is assigned to the detected pedestrian, this new feature matching is performed. The target appearance feature of the target pedestrian may be constructed.

As described above, in the pedestrian re-identification method according to the embodiment of the present disclosure, the recent posture of the target pedestrian is identified by identifying the target pedestrian using the initial appearance feature and the recent appearance feature of the target pedestrian. And can effectively follow changes in appearance, effectively correct errors based on the initial characteristics of the target pedestrian, and walk in scenes where the pedestrian's posture has changed significantly or identified by different cameras. The accuracy of re-identification of a person can be improved.

In addition to the methods and devices described above, the embodiments of the present disclosure also include a computer-readable storage medium in which computer program instructions that can be executed by the processor to cause the processor to perform the pedestrian reidentification method are stored. It may be included.

Result verification In the following, (1) re-identification of pedestrians based on the initial appearance characteristics of the target pedestrian, (2) re-identification of pedestrians based on the recent appearance characteristics of the target pedestrian, (3) target appearance of the target pedestrian. The results of verifying three types of pedestrian re-identification methods based on characteristics are shown. Specifically, in this exemplary validation, a target pedestrian library containing the appearance features of multiple target pedestrians was pre-built to identify pedestrians for one or more video frames in the video sequence to be analyzed. The target pedestrian is identified from the video frame by calculating the feature distance between the appearance feature of each pedestrian detected and the appearance feature of the constructed target pedestrian and the appearance feature of the constructed target pedestrian. In this verification, the accuracy of pedestrian re-identification by the above three types of pedestrian re-identification methods is evaluated by the following two indexes.

Rank1: This index matches the detected appearance characteristics of the pedestrian with the appearance characteristics of each target pedestrian, and then, among the target pedestrians, the target pedestrian with the best identification result (that is, the feature distance is the best). It shows the probability that (close target pedestrian) is just the correct answer result, and is expressed by the following formula.

(Equation 1)

Correct answer rate: This index indicates the average value of the probabilities that the same pedestrian holds the same ID in the image sequence, and is expressed by the following formula.

(Equation 2)

Table 1 shows the results of verifying three types of pedestrian re-identification methods based on the above two indexes.
(Table 1)

As shown in Table 1, the pedestrian re-identification method based on the target appearance features according to the present disclosure gives the best results in the two indicators.

Further, FIGS. 7 (a) to 7 (c) schematically show the results of re-identifying pedestrians using the above-mentioned three types of pedestrian re-identification methods. Specifically, FIG. 7A shows the result of re-identifying the pedestrian based on the initial appearance characteristics of the target pedestrian, and the image angle and posture of the same pedestrian to be rounded are taken as an example. Due to the change, pedestrians are identified as P12 and P7 in different video frames, respectively. FIG. 7B shows the result of re-identifying the pedestrian based on the recent appearance characteristics of the target pedestrian. Taking the same pedestrian to be rounded as an example, this pedestrian is mistaken for P17 to P15. After being identified, it becomes impossible to correct while retaining the misidentification result of P15. FIG. 7 (c) shows the result of pedestrian re-identification based on the target appearance feature of the target pedestrian, and the target appearance feature of the target pedestrian is generated based on the initial appearance feature and the recent appearance feature. Therefore, correct identification results are obtained at any of the different imaging angles and pedestrian postures.

In the pedestrian re-identification method based on the target appearance feature of the present invention, the posture of the pedestrian has changed significantly as compared with the case where the pedestrian is re-identified based only on the initial appearance feature of the target pedestrian or the recent appearance feature. It can be seen that it improves the accuracy of pedestrian re-identification in scenes or scenes identified by different cameras.

Although the basic principles of the present disclosure have been described in the context of specific embodiments, the advantages, advantages, effects, etc. referred to in the present disclosure are merely examples, not limitations, and each embodiment of the present disclosure. It should not be considered that it must be prepared. In addition, the specific details disclosed above are merely exemplary actions and actions that facilitate understanding, and are not limiting. It is not limited to be realized by using.

The phrase "based on" as used herein does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

Any reference to elements using designations such as "first", "second" as used herein does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

The term "determining" as used herein may include a wide variety of actions. A "decision" is, for example, a calculation, a computing, a processing, a deriving, an investigating, a searching up (eg, a table, a database or another data). It can be regarded as "judgment (decision)" that the search in the structure), the confirmation (ascertaining), and the like. In addition, "judgment (decision)" includes receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access (for example). It can be regarded as "judgment (decision)" that the input) (for example, accessing the data in the memory) is performed. Further, the "judgment (decision)" can be regarded as "judgment (decision)" such as solving, selecting, selecting, establishing, and comparing. That is, "judgment (decision)" can be regarded as "judgment (decision)" of some action.

The terms "connected", "coupled", or any variation thereof, mean any direct or indirect connection or connection between two or more elements, and each other. It can include the presence of one or more intermediate elements between two "connected" or "combined" elements. The connections or connections between the elements may be physical, logical, or a combination of both. For example, "connection" may be read as "access." As used herein, the two elements are by using one or more wires, cables and / or printed electrical connections, and, as some non-limiting and non-comprehensive examples, radio frequencies. By using electromagnetic energies such as electromagnetic energies with wavelengths in the region, microwave region and / or light (both visible and invisible) regions, they can be considered to be "connected" or "coupled" to each other.

As long as "inclusion," "comprising," and variations thereof are used herein or in the claims, these terms are as comprehensive as the term "comprising." Intended to be targeted. Furthermore, the term "or" as used herein or in the claims is intended not to be an exclusive OR.

Each of the embodiments / embodiments described herein may be used alone, in combination, or switched during execution. Further, the order of the processing procedures, sequences, flowcharts, etc. of each embodiment / embodiment described in the present specification may be changed as long as there is no contradiction. For example, for the methods described herein, various step elements are presented in an exemplary order and are not limited to the particular order presented.

It should be noted that in the devices and methods of the present disclosure, software is a command, instruction set, code, code segment, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name. , Program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc. should be broadly interpreted to mean.

Further, software, instructions, information and the like may be transmitted and received via a transmission medium. For example, the software uses wired technology (coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL, Digital Subscriber Line), etc.) and / or wireless technology (infrared, microwave, etc.) to website, When transmitted from a server, or other remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

Although the present invention has been described in detail above, it is clear to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as a modified or modified mode without departing from the spirit and scope of the present invention determined by the description of the claims. Therefore, the description of the present specification is for the purpose of exemplification and does not have any limiting meaning to the present invention.

Claims (10)

With a pedestrian detection unit configured to detect pedestrians from each video frame in the video sequence,
A feature extraction unit configured to extract appearance features for each pedestrian detected from a video frame,
Includes a pedestrian identification unit configured to match the extracted appearance features of each pedestrian with the target appearance features of the target pedestrian and identify the target pedestrian from the video frame based on the matching results.
The target appearance feature of the target pedestrian is a pedestrian re-identification device generated based on the initial appearance feature and the recent appearance feature of the target pedestrian. The initial appearance feature includes appearance features pre-extracted from an initial video frame containing the target pedestrian, and the recent appearance feature includes appearance features pre-extracted from a recent video frame containing the target pedestrian. The pedestrian re-identification device according to 1. The pedestrian re-identification device according to claim 2, wherein the initial appearance feature of the target pedestrian and the recent appearance feature are fused to generate the target appearance feature of the target pedestrian. The pedestrian re-identification device according to claim 3, wherein the initial appearance feature of the target pedestrian and the recent appearance feature are weighted and fused based on a preset weight. Weighting and fusing the initial appearance feature of the target pedestrian with the recent appearance feature based on a preset weight is an initial weighting fusion of the target pedestrian for weighting fusion based on a preset weight. Claiming, including determining the number of video frames and the number of recent video frames and calculating the weighted average of the determined number of initial video frames and the determined number of recent video frames of the target pedestrian appearance feature. Item 4. The pedestrian re-identification device according to item 4. By cascade-connecting the initial appearance feature and the recent appearance feature of the target pedestrian and performing a convolution operation on the cascade-connected appearance feature, the initial appearance feature and the recent appearance feature are cascade-connected and fused. , The pedestrian re-identification device according to claim 3. The pedestrian identification unit determines whether or not the pedestrian detected from the video frame corresponds to the target pedestrian based on the similarity between the extracted pedestrian appearance feature and the target appearance feature. The pedestrian re-identification device according to claim 2, which is further configured as described above. When the pedestrian detected from the video frame corresponds to the target pedestrian, the feature update configured to make the detected pedestrian appearance feature in the video frame the newest recently appearance feature of the target pedestrian. The pedestrian re-identification device according to claim 7, further comprising a unit. The video sequence and the initial video frame and the recent video frame from which the target appearance feature of the target pedestrian is extracted are those taken by different cameras or taken by the same camera at different times. , The pedestrian re-identification device according to claim 2. It has a processor and a memory in which computer program instructions are stored.
When the computer program instruction is executed by the processor, the processor detects a pedestrian from each video frame of the video sequence.
For each pedestrian detected from the video frame, the steps to extract the appearance features,
The step of matching the extracted appearance features of each pedestrian with the target appearance features of the target pedestrian and identifying the target pedestrian from the video frame based on the matching result is executed.
The target appearance feature of the target pedestrian is a pedestrian re-identification device generated based on the initial appearance feature and the recent appearance feature of the target pedestrian.