CN104616032A - Multi-camera system target matching method based on deep-convolution neural network - Google Patents

Abstract

Disclosed is a multi-camera system target matching method based on a deep-convolution neural network. The multi-camera system target matching method based on the deep-convolution neural network comprises initializing multiple convolution kernels on the basis of a local protective projection method, performing downsampling on images through a maximum value pooling method, and through layer-by-layer feature transformation, extracting histogram features higher in robustness and representativeness; performing classification and identification through a multi-category support vector machine (SVM) classifier; when a target enters one camera field of view from another camera field of view, performing feature extraction on the target and marking a corresponding target tag. The multi-camera system target matching method based on the deep-convolution neural network achieves accurate identification of the target in a multi-camera cooperative monitoring area and can be used for target handoff, tracking and the like.

Description

Based on the multi-camera system target matching method of degree of depth convolutional neural networks

Technical field

The invention belongs to the field of intelligent video surveillance in computer vision, be applicable to the target matching method based on degree of depth convolutional neural networks of multiple-camera cooperation video monitoring system.

Background technology

In large-scale video monitoring place, as airport, subway station, square etc., follow the tracks of for the target in multiple-camera cooperation supervisory system, the object matching between multiple-camera is a committed step.For large-scale video monitoring scene, the demarcation of video camera is difficult, complicated, spatial relationship to each other, time relationship and mistiming are difficult to carry out reasoning, the object matching of the mainly feature based of the target matching method between the multiple-camera that therefore at present application is wider, the validity of Feature Selection directly affects the accuracy of matching result.But can be effectively a difficult problem to the extraction of the robust features that target characterizes.Feature conventional at present comprises color, texture etc., and these features are difficult to all keep good robustness in all monitoring scenes.Therefore, we have proposed a kind of target matching method based on degree of depth study, from sequence of frames of video, feature can be learnt out adaptively to realize object matching accurately.Compared with traditional neural network, deep neural network overcomes the less problem of the network number of plies, by successively converting feature, obtains more abstract feature representation, and the last output layer of target classification as network is realized, substantially increases speed and the efficiency of object matching.

Summary of the invention

The present invention will overcome the above-mentioned deficiency of prior art, provide a kind of based on the degree of depth study multiple-camera between target matching method, the concrete steps of the method comprise:

(1) pre-service of target image: the n width target image extracting multiple-camera territory, is divided into m label; Utilize bicubic interpolation algorithm (bicubic interpolation) to be adjusted to h × w by unified for picture size, wherein h is the height of image, and w is the wide of image; Simple scalability is carried out to the pixel value of image pattern, makes final pixel value all drop between [0,1]; The tag storage of n width image is the data of n × 1, the span of each label be [1 ..., m];

(2) feature extraction is carried out based on degree of depth convolutional neural networks:

A () is extracted in the target image obtained and is selected n from step (1) _tindividual training sample, as the sensing node of convolutional neural networks ground floor input layer wherein, X _t, i=1,2 ..., n _trepresent the i-th width image;

B wave filter that () is applied to target image characteristics extraction is a kind of convolution kernel built based on localised protection projecting method, and its concrete construction method is as follows:

To image X _tcarry out piecemeal process, a setting point block size is p ₁× p ₂, then X _iwhole piecemeals be: wherein, x _i,j, j=1 ..., hw represents X _ijth piecemeal vector; Then deduct piecemeal average from each piecemeal, obtain: wherein, j=1 ..., hw represents the piecemeal after removing average; Identical process is done to all input picture X, obtains:

Proper vector is calculated as follows: XLX ^ta=λ XDX ^ta, wherein, a is proper vector, and λ is a characteristic of correspondence value, and D is diagonal matrix, its element value be weight matrix W row and or row and; Weight matrix W is dimension is n _t× n _tsparse matrix, W _ijrepresent sample with between connection weight, calculate the Euclidean distance between all samples, for each sample, find the k nearest with it _nearestindividual sample, if i.e.: sample at sample k _nearestin individual nearest-neighbors, or sample at sample k _nearestin individual nearest-neighbors, then otherwise, W _ij=0; D _ii=∑ _jw _ji, L=W-D is Laplacian matrix; The proper vector calculated is sorted by its eigenwert size, gets front k ¹individual proper vector make V _i ¹=a _i-1, i=1,2 ..., k ¹, then be the convolution kernel of extraction;

By convolution kernel V ¹with every two field picture carry out convolution, namely i=1 ..., n _t, j=1 ..., k ¹, then n is produced at this convolutional layer _tk ¹width output characteristic mapping graph, is expressed as:

C (), for Feature Mapping figure Y obtained above, carries out unique point down-sampling to it based on maximal value pond (maxpooling); If sample window size is s ¹× s ¹, then n is obtained _tk ¹width output characteristic mapping graph:

wherein, the i-th width output characteristic mapping graph $Z_i={\mathrm{\&Sigma;}}_{j=1}^{\frac{h}{s^1}}{\mathrm{\&Sigma;}}_{k=1}^{\frac{w}{s^1}}{Z}_{j,k}^i,Z_{j,k}^i={\max }_{0\&le;u,v<s^1}\left\{Y_{j\&CenterDot;s^1+u,k\&CenterDot;s^1+v}^i\right\}$ Represent that the row of the i-th width output characteristic mapping graph arranges, i=1 ..., n _tk ¹, u, v represent sampling step length, Y ⁱrepresent the i-th width input picture, max{.} represents and gets max function; In addition, this algorithm adopts zero lap sampling, namely gets u=v=s ¹;

D () adopts the step similar with (b), to image X _icarry out piecemeal process, a setting point block size is p ₁× p ₂, Feature Mapping figure Z step (c) obtained, as the input of this convolutional layer, goes average to the block data of every two field picture, obtains input picture:

wherein, the i-th width input feature vector mapping graph i=1 ..., n _tk ¹represent the i-th width piecemeal go average after image, j=1 ..., hw represents in the i-th width image the piecemeal vector of the jth after removing average; Structure weight matrix W, and according to ZLZ ^ta=λ ZDZ ^ta calculates proper vector, after sorting, gets front k according to eigenwert size ²individual proper vector is as the convolution kernel chosen wherein, V _i ², i=1 ..., k ²represent V ²in i-th convolution kernel; Then the convolution kernel V obtained is utilized ²to every two field picture carry out convolution, then produce n at this convolutional layer _tk ¹k ²width output characteristic mapping graph:

wherein, i=1 ..., n _tk ¹, j=1 ..., k ²;

E (), for Feature Mapping figure U obtained above, adopts the step similar with (c), carries out unique point down-sampling to it based on maximal value pond method; If sample window size is s ²× s ², then n is obtained _tk ¹k ²width output characteristic mapping graph:

wherein, the i-th width output characteristic mapping graph represent that the row of the i-th width output characteristic mapping graph arranges, i=1 ..., n _tk ¹k ², $j=1,...,\frac{(\frac{h}{s^1}-s^2)}{u}+1,k=1,...,\frac{(\frac{w}{s^1}-s^2)}{v}+1,$

, v represents sampling step length, U ⁱrepresent the i-th width input picture, max{.} represents and gets max function; In addition, this algorithm adopts zero lap sampling, namely gets u=v=s ²;

F () makes $P_i=(O_{(i-1)k^2+1},...,O_{(i-1)k^2+k^2}),$ I=1 ..., n _tk ¹, namely get the every k in O ²width image is one group, and being carried out Heaviside two-value quantification aftertreatment is decimal value, then every k ²width image converts piece image to i=1 ..., n _tk ¹, wherein, H () represents Heaviside function, P _i ^jrepresent P _iin jth width image, T _irepresent decimal system result, span is then often k is got ¹width T _iimage is one group, first every width image is divided into B block, then calculates the histogram feature in every block region, then the B block histogram feature obtained is connected into row vector, is defined as wherein, l=1 ..., n _t, s=1 ..., k ¹; Then for the every width image X in (a) _l, finally extract based on convolutional neural networks and obtain eigenvector l=1 ..., n _t;

(3) Classification and Identification: the feature that said extracted is obtained as input, target labels corresponding to each eigenvector, as output, builds the sorter model obtaining target by multi-class support vector machine SVM.Based on this sorter model, can realize marking the target in the different cameras visual field and classifying, for target handoff and tracking etc.

The invention has the beneficial effects as follows:

This invention is owing to adopting localised protection projecting method initialization convolution kernel; but not random initializtion convolution kernel; accurately can retain the notable feature of target image; thus the histogram feature extracted can be maintained the invariance to the dimensional variation of target and rotation; to the illumination variation of scene, there is stronger adaptability, greatly increase the discrimination of target.This invention carries out down-sampling to the image after use localised protection projecting method convolution; effectively reduce intrinsic dimensionality; avoid dimension disaster; substantially reduce the recognition time of target; and adopt multireel to amass collecting image and carry out convolution and Superposition Characteristics, effectively eliminate the discrimination caused because of dimensionality reduction and decline.

Accompanying drawing explanation

Fig. 1 is process flow diagram of the present invention.

Embodiment

The inventive method comprises the extraction of target signature and Classification and Identification two parts of target.Wherein, clarification of objective is extracted and is adopted degree of depth learning method, by building the neural network of many hidden layers, sample characteristics is successively converted, sample is transformed to a new feature space at the character representation in former space, learns more useful feature, then use this feature as the input feature vector of Multi-class SVM classifier, carry out the Classification and Identification of target, thus the final accuracy promoting classification or prediction.Figure 1 show the enforcement block diagram of this algorithm, concrete steps are as follows:

(1) pre-service of target image: the n width target image extracting multiple-camera territory, is divided into m label; Utilize bicubic interpolation algorithm (bicubic interpolation) to be adjusted to h × w by unified for picture size, wherein h is the height of image, and w is the wide of image; Simple scalability is carried out to the pixel value of image pattern, makes final pixel value all drop between [0,1]; The tag storage of n width image is the data of n × 1, the span of each label be [1 ..., m]; Namely 001 represent pretreated image;

(2) feature extraction is carried out based on degree of depth convolutional neural networks:

A () selects n from 001 _tindividual training sample, as the sensing node of convolutional neural networks ground floor input layer namely shown in 002, wherein, X _i, i=1,2 ..., n _trepresent the i-th width image;

B wave filter that () is applied to target image characteristics extraction is a kind of convolution kernel built based on localised protection projecting method, and its concrete construction method is as follows:

To image X in 002 _icarry out piecemeal process, a setting point block size is p ₁× p ₂, then X _iwhole piecemeals be: wherein, x _i,j, j=1 ..., hw represents X _ijth piecemeal vector; Then deduct piecemeal average from each piecemeal, obtain: wherein, j=1 ..., hw represents the piecemeal after removing average; Identical process is done to all input picture X, obtains: namely shown in 003;

Proper vector is calculated as follows: XLX ^ta=λ XDX ^ta, wherein, a is proper vector, and λ is a characteristic of correspondence value, and D is diagonal matrix, its element value be weight matrix W row and or row and; Weight matrix W is dimension is n _t× n _tsparse matrix, W _ijrepresent sample with between connection weight, calculate the Euclidean distance between all samples, for each sample, find the k nearest with it _nearestindividual sample, if i.e.: sample at sample k _nearestin individual nearest-neighbors, or sample at sample k _nearestin individual nearest-neighbors, then otherwise, W _ij=0; D _ii=∑ _jw _ji, L=W-D is Laplacian matrix; The proper vector calculated is sorted by its eigenwert size, gets front k ¹individual proper vector make V _i ¹=a _i-1, i=1,2 ..., k ¹, then be the convolution kernel of extraction;

By convolution kernel V ¹with 003 in every two field picture carry out convolution, i=1 ..., n _t, j=1 ..., k ¹, then n is produced at this convolutional layer _tk ¹width output characteristic mapping graph, is expressed as: namely shown in 004;

C (), for the Feature Mapping figure Y shown in 004, carries out unique point down-sampling to it based on maximal value pond (maxpooling); If sample window size is s ¹× s ¹, then n is obtained _tk ¹width output characteristic mapping graph:

namely shown in 005, wherein, the i-th width output characteristic mapping graph represent the jth row kth row of the i-th width output characteristic mapping graph, i=1 ..., n _tk ¹, $j=1,...,\frac{(h-s^1)}{u}+1,,k=1,...,\frac{(w-s^1)}{v}+1,$ U, v represent sampling step length, Y ⁱrepresent the i-th width input picture, max{.} represents and gets max function; In addition, this algorithm adopts zero lap sampling, namely gets u=v=s ¹;

D () adopts and (b) similar step, to image X in 005 _icarry out piecemeal process, a setting point block size is p ₁× p ₂, Feature Mapping figure Z step (c) obtained, as the input of this convolutional layer, goes average to the block data of every two field picture, obtains input picture:

namely shown in 006, wherein, the i-th width input feature vector mapping graph i=1 ..., n _tk ¹represent the i-th width piecemeal go average after image, j=1 ..., hw represents in the i-th width image the piecemeal vector of the jth after removing average; Structure weight matrix W, and according to ZLZ ^ta=λ ZDZ ^ta calculates proper vector, after sorting, gets front k according to eigenwert size ²individual proper vector is as the convolution kernel chosen wherein, V _i ², i=1 ..., k ²represent V ²in i-th convolution kernel; Then the convolution kernel V obtained is utilized ²to every two field picture carry out convolution, then produce n at this convolutional layer _tk ¹k ²width output characteristic mapping graph:

namely shown in 007, wherein, i=1 ..., n _tk ¹, j=1 ..., k ²;

E (), for the Feature Mapping figure U shown in 007, adopts the step similar with (c), carries out unique point down-sampling to it based on maximal value pond; If sample window size is s ²× s ², then n is obtained _tk ¹k ²width output characteristic mapping graph:

namely shown in 008, wherein, the i-th width output characteristic mapping graph represent the jth row kth row of the i-th width output characteristic mapping graph, i=1 ..., n _tk ¹k ², $j=1,...,\frac{(\frac{h}{s^1}-s^2)}{u}+1,k=1,...,\frac{(\frac{w}{s^1}-s^2)}{v}+1,$

, v represents sampling step length, U ⁱrepresent the i-th width input picture, max{.} represents and gets max function; In addition, this algorithm adopts zero lap sampling, namely gets u=v=s ²;

F () makes $P_i=(O_{(i-1)k^2+1},...,O_{(i-1)k^2+k^2}),$ I=1 ..., n _tk ¹, namely get every k in the O shown in 008 ²width image is one group, and being carried out Heaviside two-value quantification aftertreatment is decimal value, then every k ²width image converts piece image to i=1 ..., n _tk ¹, wherein, H () represents Heaviside function, P _i ^jrepresent P _iin jth width image, T _irepresent decimal system result, span is then often k is got ¹width T _iimage is one group, first every width image is divided into B block, then calculates the histogram feature in every block region, then the B block histogram feature obtained is connected into row vector, is defined as wherein, l=1 ..., n _t, s=1 ..., k ¹; Then for the every width image X in 002 _l, finally extract based on convolutional neural networks and obtain eigenvector l=1 ..., n _t;

(3) Classification and Identification: the feature that said extracted is obtained as input, target labels corresponding to each eigenvector, as output, builds the sorter model obtaining target by multi-class support vector machine SVM.Based on this sorter model, can realize marking the target in the different cameras visual field and classifying, for target handoff and tracking etc.

Content described in this instructions embodiment is only enumerating the way of realization of inventive concept; protection scope of the present invention should not be regarded as being only limitted to the concrete form that embodiment is stated, protection scope of the present invention also and conceive the equivalent technologies means that can expect according to the present invention in those skilled in the art.

Claims (1)

1., based on a multi-camera system target matching method for degree of depth convolutional neural networks, its feature comprises:

(1) pre-service of target image: the n width target image extracting multiple-camera territory, is divided into m label; Utilize bicubic interpolation algorithm (bicubic interpolation) to be adjusted to h × w by unified for picture size, wherein h is the height of image, and w is the wide of image; Simple scalability is carried out to the pixel value of image pattern, makes final pixel value all drop between [0,1]; The tag storage of n width image is the data of n × 1, the span of each label be [1 ..., m];

(2) feature extraction is carried out based on degree of depth convolutional neural networks:

A () is extracted in the target image obtained and is selected n from step (1) _tindividual training sample, as the sensing node of convolutional neural networks ground floor input layer wherein, X _i, i=1,2 ..., n _trepresent the i-th width image;

B wave filter that () is applied to target image characteristics extraction is a kind of convolution kernel built based on localised protection projecting method, and its concrete construction method is as follows:

To image X _icarry out piecemeal process, a setting point block size is p ₁× p ₂, then X _iwhole piecemeals be: wherein, x _i,j, j=1 ..., hw represents X _ijth piecemeal vector; Then deduct piecemeal average from each piecemeal, obtain: wherein, j=1 ..., hw represents the piecemeal after removing average; Identical process is done to all input picture X, obtains:

Proper vector is calculated as follows: XLX ^ta=λ XDX ^ta, wherein, a is proper vector, and λ is a characteristic of correspondence value, and D is diagonal matrix, its element value be weight matrix W row and or row and; Weight matrix W is dimension is n _t× n _tsparse matrix, W _ijrepresent sample with between connection weight, calculate the Euclidean distance between all samples, for each sample, find the k nearest with it _nearestindividual sample, if i.e.: sample at sample k _nearestin individual nearest-neighbors, or sample at sample k _nearestin individual nearest-neighbors, then otherwise, W _ij=0; D _ii=∑ _jw _ji, L=W-D is Laplacian matrix; The proper vector calculated is sorted by its eigenwert size, gets front k ¹individual proper vector order i=1,2 ..., k ¹, then be the convolution kernel of extraction;

By convolution kernel V ¹with every two field picture carry out convolution, namely i=1 ..., n _t, j=1 ..., k ¹, then n is produced at this convolutional layer _tk ¹width output characteristic mapping graph, is expressed as:

C (), for Feature Mapping figure Y obtained above, carries out unique point down-sampling to it based on maximal value pond (maxpooling); If sample window size is s ¹× s ¹, then n is obtained _tk ¹width output characteristic mapping graph: wherein, the i-th width output characteristic mapping graph $Z_i={\mathrm{\&Sigma;}}_{j=1}^{\frac{h}{s^1}}{\mathrm{\&Sigma;}}_{k=1}^{\frac{w}{s^1}}{Z}_{j,k}^i,Z_{j,k}^i={\max }_{0\&le;u,v<s^1}\left\{Y_{j\&CenterDot;s^1+u,k\&CenterDot;s^1+v}^i\right\}$ Represent the jth row kth row of the i-th width output characteristic mapping graph, i=1 ..., n _tk ¹, u, v represent sampling step length, Y ⁱrepresent the i-th width input picture, max{.} represents and gets max function; In addition, this algorithm adopts zero lap sampling, namely gets u=v=s ¹;

D () adopts the step similar with (b), to image X _icarry out piecemeal process, a setting point block size is p ₁× p ₂, Feature Mapping figure Z step (c) obtained, as the input of this convolutional layer, goes average to the block data of every two field picture, obtains input picture: wherein, the i-th width input feature vector mapping graph i=1 ..., n _tk ¹represent the i-th width piecemeal go average after image, j=1 ..., hw represents in the i-th width image the piecemeal vector of the jth after removing average; Structure weight matrix W, and according to ZLZ ^ta=λ ZDZ ^ta calculates proper vector, after sorting, gets front k according to eigenwert size ²individual proper vector is as the convolution kernel chosen wherein, i=1 ..., k ²represent V ²in i-th convolution kernel; Then the convolution kernel V obtained is utilized ²to every two field picture carry out convolution, then produce n at this convolutional layer _tk ¹k ²width output characteristic mapping graph: wherein, i=1 ..., n _tk ¹, j=1 ..., k ²;

E (), for Feature Mapping figure U obtained above, adopts the step similar with (c), carries out unique point down-sampling to it based on maximal value pond; If sample window size is s ²× s ², then n is obtained _tk ¹k ²width output characteristic mapping graph: wherein, the i-th width output characteristic mapping graph $O_{j,k}^i={\max }_{0\&le;u,v<s^2}\left\{U_{j\&CenterDot;s^2+u,k\&CenterDot;s^2+v}^i\right\}$ Represent the jth row kth row of the i-th width output characteristic mapping graph, i=1 ..., n _tk ¹k ², $j=1,...,\frac{(\frac{h}{s^1}-s^2)}{u}+1,k=1,...,\frac{(\frac{w}{s^1}-s^2)}{v}+1,$ U, v represent sampling step length, U ⁱrepresent the i-th width input picture, max{.} represents and gets max function; In addition, this algorithm adopts zero lap sampling, namely gets u=v=s ²;

F () makes $P_i=(O_{(i-1)k^2+1},...,O_{(i-1)k^2+k^2}),$ I=1 ..., n _tk ¹, namely get the every k in O ²width image is one group, and being carried out Heaviside two-value quantification aftertreatment is decimal value, then every k ²width image converts piece image to i=1 ..., n _tk ¹, wherein, H () represents Heaviside function, represent P _iin jth width image, T _irepresent decimal system result, span is then often k is got ¹width T _iimage is one group, first every width image is divided into B block, then calculates the histogram feature in every block region, then the B block histogram feature obtained is connected into row vector, is defined as wherein, l=1 ..., n _t, s=1 ..., k ¹; Then for the every width image X in (a) _l, finally extract based on convolutional neural networks and obtain eigenvector l=1 ..., n _t;

(3) Classification and Identification: the feature that said extracted is obtained as input, target labels corresponding to each eigenvector, as output, builds the sorter model obtaining target by multi-class support vector machine SVM.Based on this sorter model, can realize marking the target in the different cameras visual field and classifying, for target handoff and tracking etc.