CN109829405A

CN109829405A - Data correlation method, device and the storage medium of video object

Info

Publication number: CN109829405A
Application number: CN201910059911.4A
Authority: CN
Inventors: 李良群; 湛西羊; 谢维信; 刘宗香
Original assignee: Shenzhen University
Current assignee: Shenzhen University
Priority date: 2019-01-22
Filing date: 2019-01-22
Publication date: 2019-05-31

Abstract

A kind of data correlation method of video object, device and storage medium.Wherein, the data correlation method of the video object includes: the second characteristics of image for obtaining the first characteristics of image of target object and observation object respectively；The characteristic similarity for carrying out N kind feature classification to the first image feature and second characteristics of image calculates；N group characteristic similarity result is screened based on the characteristic similarity of rough set, and the result after screening is merged, obtains Fusion Features result；Based on maximum entropy Intuitionistic Fuzzy Clustering, cost matrix is associated according to the Fusion Features result and is calculated, obtains association cost matrix calculated result；Judge whether the target object is associated with the observation object according to the association cost matrix calculated result；If it is not, then carrying out target trajectory management to the target object, the newly track and termination track of target object in video are counted, the termination track collection and the new track collection are obtained；Blurring trajectorie association is carried out according to the termination track collection and the new track collection.

Description

Data association method and device of video target and storage medium

Technical Field

The present application relates to the field of electronic technologies, and in particular, to a method and an apparatus for associating data of a video object, and a storage medium.

Background

Video multi-target tracking is one of important research contents of machine vision, and is mainly used for acquiring basic motion information such as the position, the posture and the track of a moving target in a video. With the development of digital computing technology, video multi-target tracking has opened up a plurality of research fields and application fields, and relates to the fields of intelligent video monitoring, virtual reality, human-computer interaction, automatic driving, traffic control, oceanography, intelligent robots, remote control sensing, biomedicine and the like, so that more and more students and researchers are attracted to participate actively, and a great deal of research results are obtained. However, under the environment of a complex background and dense shielding of targets, a lot of uncertain and incomplete information exists in the multi-target tracking process, such as deformation of pedestrians, illumination change, false observation, mutual shielding between targets and the like, so that it is difficult to accurately describe the relationship between the targets and observation by using a traditional probability statistical method.

At present, due to the development of detection technology, a video target tracking algorithm based on target detection is rapidly developed and widely applied. The method estimates the current state of the target according to the position, size, label, characteristics and the like of the detected target without manual marking, and can update the state of the target in real time according to the detection result in the tracking process. In recent years, with the application of a deep learning method in video target detection, a target detection technology is continuously advanced, a video multi-target detection technology is mature, a more accurate detection result can be obtained even in a clutter environment, and the accuracy of data association becomes a main factor influencing the accuracy of tracking and sending, so that the multi-target data association method based on detection is widely researched.

The key problem of the multi-target tracking method based on detection is the data correlation between the online detection result received by the detector and the existing target. In order to solve the data association problem, in recent thirty years, scholars and researchers at home and abroad propose many types of data association methods. Most tracking algorithms can be classified into the following two categories according to the data association mode: a generative method and a discriminant method. A target model is constructed by utilizing a plurality of characteristics of appearance, motion and the like of a target based on a tracking algorithm under a generating method, and tracking association is carried out by considering the similarity between the target and observation in the tracking process. Ross et al can effectively adapt to changes in the appearance of a target by gradually learning low-dimensional subspace feature representation of the target, Babu et al propose tracking according to reliable feature points of the target, Azab et al use fuzzy integration to fuse a plurality of features in a particle filter frame for tracking, Zhang et al use a sparse representation target model based on a particle filter to select a candidate template with the minimum reconstruction error of the target model for association, and Zhang and Liu et al use a double-layer convolutional neural network to train robust features for association. However, the correlation methods based on the generation formula sometimes cannot distinguish the target from the background well, because the background pixels and other target pixels in the bounding box of the target are inevitably considered as part of the target, so that the target appearance model has inaccurate information, which cannot correctly distinguish the object from the background, and the tracking may fail. The discriminant method aims at better separating the target from the background, for example, Jang et al proposes a projection network model with the minimum error of a fusion particle filter observation model, and Mei et al proposes to correlate the target and the observation by using a least square method and Bayesian state estimation. However, the discrimination method cannot adapt to the appearance change of the target, so Babenko et al propose to train a target model by using multi-instance learning, Kalal et al propose to monitor coding rules based on optical flow and dynamic appearance model, Tuzel et al solve the problem of shielding and tracking of multiple targets and take the problem as a binary pattern classification problem, Wu et al propose a region deep learning tracker, observe the targets through multiple sub-regions and observe each region through the deep learning model, and compared with most existing trackers which only use two-dimensional color or gray images to learn the appearance model of the tracked object on line, the method obtains better effect. In addition, the correlation filter-based method and the deep learning-based method have achieved excellent performance and attracted much attention.

In recent years, as the application of the fuzzy set theory in multi-target tracking is more and more extensive, it is pointed out that modeling uncertain information in a tracking system by using fuzzy mathematics is beneficial to improving the tracking performance, and the associated algorithm based on the intuition fuzzy set theory has more and more achievements. Such as fuzzy orthogonal particle filtering, intuitive fuzzy joint probability data association filtering, fuzzy kalman filtering and fuzzy code histograms. The literature proposes an active contour-based object tracking algorithm for modeling a neural-fuzzy network. The contour-based model is used to extract the feature vectors of objects, and in order to train and identify moving objects, the self-built neuro-fuzzy inference network is used to simultaneously perform Discrete Fourier Transform (DFT) on the human contour histogram in horizontal and vertical projection. The advantage of uncertain information processing by an intuitionistic fuzzy set theory is more obvious, Li et al propose an on-line video multi-target tracking algorithm based on an intuitionistic fuzzy set, the obtained intuitionistic fuzzy membership replaces the relevance between a target and observation to realize the relevance between the target and the observation, ZHao et al propose an inference algorithm based on intuitionistic logic, and Revathi et al propose a histogram fuzzy logic block matching algorithm to track video multi-targets.

Disclosure of Invention

The embodiment of the application provides a data association method, a data association device and a storage medium of a video target, which are used for data association of video multi-target tracking.

A first aspect of an embodiment of the present application provides a data association method for a video object, including:

acquiring a first image characteristic of a target object and a second image characteristic of an observation object respectively;

performing feature similarity calculation of N feature categories on the first image features and the second image features to obtain N groups of feature similarity results, wherein N is an integer greater than 1;

screening the N groups of feature similarity results based on the feature similarity of the rough set, and fusing the screened results to obtain a feature fusion result;

based on the maximum entropy intuitive fuzzy clustering, performing association cost matrix calculation according to the feature fusion result to obtain an association cost matrix calculation result;

judging whether the target object is associated with the observation object or not according to the association cost matrix calculation result, if not, performing target track management on the target object, and counting new start tracks and end tracks of the target object in a video to obtain an end track set and a new track set;

and performing fuzzy track association according to the terminal track set and the new track set, wherein the fuzzy track association comprises the following steps: s1, acquiring a terminal track set and a new track set in a video, and determining a fuzzy credibility matrix of the terminal track set and the new track set; s2 finding the ending trace i in the ending trace set and the new trace j at the nth time in the new trace set^*: determining the ending track i and the new track j according to the fuzzy credibility matrix^*The maximum degree of identity between; if the maximum degree of identity is greater than or equal to the reliability threshold value, the terminal track i and the new track j^*Is associated with and the new track j^*Not associated with other terminal tracks; s3 when the ending track i and the new track j^*When in association, if the terminal track i is associated with the new track j^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Are the same track; if the ending track i and the new track j are^*If the track quality between the end track i and the new track j is less than the track quality threshold value, the end track i and the new track j are determined^*Are not the same trajectory; repeating the steps S2 and S3 until all the tracks in the video target data are completely associated.

Optionally, the determining the fuzzy reliability matrix of the terminal track set and the new track set includes:

at the t-th frame of the video,a set of already-terminated tracks is represented,a new set of trajectories is represented in which,andrespectively representing the number of terminated tracks and the number of new tracks, l_0,iIndicating the time at which the track i terminates, l_1,jIndicates the start time of the new track j, if l_1,jIf t, then new trackIs a new observation track which is associated; thenThe confidence matrix of (2) is: u shape_t＝{μ_ij}; wherein, mu_ijIndicating the fuzzy integrated similarity of the track i and the new track j.

Optionally, the final track i and the new track j are determined according to the fuzzy credibility matrix^*The maximum degree of identity between, including:

function(s)Respectively representing the average similarity of the characteristics of the distance, the color, the edge, the language, the shape, the motion direction, the overlapping area and the like of the terminal track i and the new track j at the t-th moment, wherein the average similarity specifically comprises the following steps:

wherein f is₁ ^m,n(i,j),f₂ ^m,n(i,j),f₃ ^m,n(i,j),f₄ ^m,n(i,j),f₅ ^m,n(i,j),f₆ ^m,n(i, j) and f₇ ^m,n(i, j) represents the similarity of the characteristics of the distance, color, edge, language, shape, moving direction, overlapping area, etc. between the ending trajectory i and the new trajectory j at the m-th time, and k is t-l_1,j+3；

Determining fuzzy comprehensive similarity mu of the track i and the new track j according to the following formula_ij；

Determining the ending track i and the new track j according to the following formula^*Maximum degree of identity between:

optionally, the ending track i and the new track j are determined^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Before the same track, the method comprises the following steps:

determining the ending track i and the new track j according to the following formula^*Track quality between

Optionally, if the maximum degree of identity is greater than or equal to a reliability threshold, the terminating track i and the new track j are determined^*Associating, comprising:

if it is notThe ending track i and the new track j^*And (3) associating, wherein epsilon represents a reliability threshold value, and the value range of epsilon is as follows: epsilon is more than or equal to 0.5 and less than or equal to 1.

Optionally, if the ending track i and the new track j are provided^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Is the same track, comprising:

if it is notThe terminating track i and the new track j are considered^*The same track is adopted, and simultaneously, a linear interpolation method is adopted to connect the terminal track i with the new track j^*And changing the track number of the new track into the track number of the terminal track, and the terminal tracki is restored to the existing track i.

A second aspect of the embodiments of the present application provides another electronic apparatus, including:

an acquisition unit configured to acquire a first image feature of a target object and a second image feature of an observation object, respectively;

the similarity calculation unit is used for performing feature similarity calculation of N feature categories on the first image features and the second image features to obtain N groups of feature similarity results, wherein N is an integer greater than 1;

the characteristic fusion unit is used for screening the N groups of characteristic similarity results based on the characteristic similarity of the rough set and fusing the screened results to obtain a characteristic fusion result;

the correlation calculation unit is used for performing correlation cost matrix calculation according to the feature fusion result based on the maximum entropy intuitive fuzzy clustering to obtain a correlation cost matrix calculation result;

a track association unit, configured to determine whether the target object is associated with the observed object according to the association cost matrix calculation result, if not, perform target track management on the target object, and count new start tracks and end tracks of the target object in a video to obtain the end track set and the new track set; and performing fuzzy track association according to the terminal track set and the new track set, wherein the fuzzy track association comprises the following steps: s1, acquiring a terminal track set and a new track set in a video, and determining a fuzzy credibility matrix of the terminal track set and the new track set; s2 finding the ending trace i in the ending trace set and the new trace j at the nth time in the new trace set^*: determining the ending track i and the new track j according to the fuzzy credibility matrix^*The maximum degree of identity between; if the maximum degree of identity is greater than or equal to the reliability threshold value, the terminal track i and the new track j^*Is associated with and the new track j^*Not associated with other terminal tracks; s3 when the ending track i is equal toThe new track j^*When in association, if the terminal track i is associated with the new track j^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Are the same track; if the ending track i and the new track j are^*If the track quality between the end track i and the new track j is less than the track quality threshold value, the end track i and the new track j are determined^*Are not the same trajectory; repeating the steps S2 and S3 until all the tracks in the video target data are completely associated.

A third aspect of the embodiments of the present application provides another electronic apparatus, including: the data association method for the video object provided by the first aspect of the embodiment of the present application is implemented by a memory, a processor, and a computer program stored in the memory and executable on the processor.

A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data association method for a video object provided in the first aspect of the embodiments of the present application.

Due to the existence of missed detection or false alarms, fuzzy data association can only deal with the tracking problem of no observation condition in a short time. When the target track is blocked for a long time or the track is not updated for a long time, the target track can generate batch breaking. Therefore, in order to solve the problem of continuous tracking of the target track under the condition of long-time occlusion, the track management method provided by the embodiment of the application can realize connection of different sub-tracks of the same target object at different moments.

Drawings

Fig. 1-a is a schematic flow chart illustrating an implementation of a data association method for a video object according to an embodiment of the present application;

1-b is a schematic diagram of an occlusion between a target object and a target object provided in the present application;

1-c are schematic diagrams of the occlusion between a target object A and a target object B provided in the present application;

FIG. 1-d is a block diagram of a fuzzy track piece fusion provided by an embodiment of the present application;

fig. 2 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 3 is a schematic diagram of a hardware structure of an electronic device according to another embodiment of the present disclosure.

Detailed Description

In order to make the objects, features and advantages of the present invention more apparent and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Suffixes such as "module", "part", or "unit" used to denote elements are used herein only for the convenience of description of the present invention, and have no specific meaning in themselves.

Example one

Referring to fig. 1-a, the data association method of the video object mainly includes the following steps:

101. acquiring a first image characteristic of a target object and a second image characteristic of an observation object respectively;

the target object is a target object tracked in a video subjected to target tracking. The first image feature is an image feature corresponding to the target object.

The observation object is an environmental object except the target object in the video for target tracking and is used for carrying out comparison analysis with the target object. The second image feature is an image feature corresponding to the observation object.

Specifically, in practical applications, there may be a plurality of observation objects, for example, j observation objects are defined, where j is an integer greater than 1.

102. Carrying out feature similarity calculation of N feature categories on the first image features and the second image features;

and performing feature similarity calculation of N feature categories on the first image features and the second image features to obtain N groups of feature similarity results, wherein N is an integer greater than 1.

In the embodiment of the application, feature similarity between the target object and the observed object is comprehensively defined by N feature categories. The feature categories may be features with different dimensions such as distance, color, shape, and the like.

Further, in practical applications, N may be equal to 7, where the 7 feature categories are: distance, color, edge, grammar, shape, direction of motion, and overlap area.

103. Screening and fusing the feature similarity results based on the feature similarity of the rough set;

and screening the N groups of feature similarity results based on the feature similarity of the rough set, and fusing the screened results to obtain a feature fusion result.

In the embodiment of the application, a concept of a neighborhood rough set is introduced, wherein the neighborhood rough set is a model provided for solving the problem that a classical rough set cannot process continuous data. The neighborhood of the model is defined according to the maximum distance from the center point of the neighborhood to the boundary on a certain metric, and the dependency of the decision attribute on the condition attribute is used for reducing the attribute.

Because the interference between adjacent targets is one of the factors influencing the correct association of the targets, because the adjacent targets are very close in position, and also because the factors such as occlusion and the like have relatively similar appearance characteristics, the probability of mismatching in the association process is high. Therefore, how to select the characteristics which can distinguish the target from the target and the target from the false observation is the key of the multi-target tracking of the video. In the embodiment of the present application, the features are selected and fused by using the neighborhood rough set.

In the embodiment of the application, N feature similarity measures the similarity relation between the target and the observation, and if only linear accumulation is carried out on the features during fusion, the reliability of the fusion result is influenced. Since not all features are well differentiated, objects can be mapped to observations on a one-to-one basis. In fact, some features may also affect the accuracy of the correlation, for example, in the case that the targets are mutually occluded, the visual features of the targets are affected, but the features such as the motion direction and the overlapping area are more robust, and if the features are directly linearly fused at this time, a wrong correlation result may be obtained. In contrast, the target features are selected by utilizing the attribute reduction capability of the neighborhood rough set, the number of the features is reduced according to the dependence of decision attributes on the domain of interest, and the selection of useful features is realized.

104. Performing correlation cost matrix calculation based on the maximum entropy intuitive fuzzy clustering and according to the feature fusion result;

and based on the maximum entropy intuitive fuzzy clustering, performing association cost matrix calculation according to the feature fusion result to obtain an association cost matrix calculation result.

In the embodiment of the application, maximum entropy intuitive fuzzy clustering of local information is introduced.

The local information is the local information of the target object, and the OSPA distance is used for defining the local information distance measure in the embodiment of the application, so that the accuracy of the target distance measure is further improved. OSPA is a consistency measurement method for overall performance evaluation of a target tracking system, a measurement distance is defined on a system state space, and the measurement distance can be used for measuring the error between a real track and an estimated track.

Conventional clustering is a hard classification that strictly classifies each object to be identified into a class that is well-defined. Fuzzy clustering, as used in the examples of the present application, is the degree of uncertainty in confirming that the sample to be classified belongs to each class, and expresses the fuzzy concept of sample category being also. The set obtained by fuzzy clustering is an intuitionistic fuzzy set, and the intuitionistic fuzzy entropy is used for expressing the uncertainty of the intuitionistic fuzzy set.

The maximum entropy principle used in the embodiments of the present application is a criterion for selecting the statistical characteristics of random variables to best meet objective conditions, and the probability distribution of random variables is difficult to determine, and generally, only various mean values (such as mathematical expectation, variance, etc.) or values under certain known limiting conditions (such as peak value, number of values, etc.) can be measured, and the distribution meeting the measured values can be various, or even infinite, and generally, the entropy of one distribution is maximum. The distribution with the maximum entropy is selected as the distribution of the random variable, which is an effective processing method and criterion.

In the embodiment of the application, a novel maximum entropy intuitive fuzzy clustering method is provided by utilizing a maximum entropy principle, and the association probability is replaced by the intuitive fuzzy membership.

105. Judging whether the target object is associated with the observation object or not according to the calculation result of the association cost matrix;

judging whether the target object is associated with the observation object or not according to the association cost matrix calculation result, and if so, updating a target model; and if not, carrying out target track management on the target object. And counting new start tracks and end tracks of the target objects in the video to obtain the end track set and the new track set. In practical application, multi-target tracking needs to be performed on a target object in the video, and in the process of performing multi-target tracking, a target tracking system can obtain multiple sections of ending tracks and multiple sections of new starting tracks of the target object.

The target model is a tracking model corresponding to the target object, and the tracking state of the target object is recorded in real time. The target model can update and maintain the information of the position, the color, the texture feature and the like of the target object.

106. And carrying out fuzzy track association according to the terminal track set and the new track set.

S1, acquiring a terminal track set and a new track set in a video, and determining a fuzzy credibility matrix of the terminal track set and the new track set; s2 finding the ending trace i in the ending trace set and the new trace j at the nth time in the new trace set^*: determining the ending track i and the new track j according to the fuzzy credibility matrix^*The maximum degree of identity between; if the maximum degree of identity is greater than or equal to the reliability threshold value, the terminal track i and the new track j^*Is associated with and the new track j^*Not associated with other terminal tracks; s3 when the ending track i and the new track j^*When in association, if the terminal track i is associated with the new track j^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Are the same track; if the ending track i and the new track j are^*If the track quality between the end track i and the new track j is less than the track quality threshold value, the end track i and the new track j are determined^*Are not the same trajectory; repeating the steps S2 and S3 until all the tracks in the video target data are completely associated.

In the scheme, N groups of feature similarity results are obtained through similarity calculation of image features between the target object and the observation object; then screening the N groups of feature similarity results, and fusing the screened results to obtain a feature fusion result; based on the maximum entropy intuitive fuzzy clustering, performing association cost matrix calculation according to the feature fusion result to obtain an association cost matrix calculation result; finally, judging whether the target object is associated with the observation object or not according to the calculation result of the association cost matrix; the scheme has the following advantages: firstly, the video target has various characteristic attributes, and the importance of the target characteristics is different for different places at different moments, so that the scheme of the application provides that the characteristics are selected by utilizing a neighborhood rough set, the characteristics with low discrimination are removed, and the accuracy of the characteristic similarity is improved; secondly, introducing local information of the target, and defining a local information distance measure by using (OSPA, Optimal Sub-pattern Assignment) distance, thereby further improving the accuracy of the target distance measure; thirdly, by utilizing the maximum entropy principle, a novel maximum entropy intuitive fuzzy clustering method is provided, and the association probability is replaced by intuitive fuzzy membership, so that the constraint of the traditional probability statistical method is avoided, and the quick and accurate calculation of the association probability is realized.

Example two

The data association method for introducing the video target through the specific formula and the algorithm process specifically comprises the following steps:

some parameter concepts used in the embodiments of the present application are defined, including:

field roughness set

The neighborhood rough set is a model provided for solving the problem that the classical rough set cannot process continuous data. The neighborhood of the model is defined according to the maximum distance from the center point of the neighborhood to the boundary on a certain metric, and the dependency of the decision attribute on the condition attribute is used for reducing the attribute.

Assume a finite non-empty set U ═ x defined over a real number space₁,x₂,…,x_nD is decision attribute, for any x_iE.u, its δ neighborhood is defined as:

δ(x_i)＝{x|x∈U,Δ(x,x_i)≤δ} (1)

wherein, Delta (x, x)_i) For distance function, the distance function herein is chosen to be an infinite norm distance, defined as:

Δ(x,x_i)＝|x-x_i| (2)

decision attribute D sets domain of discourse partitions into D_UThen D is_UUpper and lower approximation ofMeaning as follows:

given an information system IS ═ (U, D, V), where U IS the domain of discourse, V IS the set of attribute values, D IS the decision attribute,decision attribute D sets domain of discourse partitions into D_UDefining the dependency of the decision attribute D on the domain of discourse U as follows:

where | represents a number. When the dependency of the decision attribute D on the domain of interest is larger, the smaller the boundary domain of the decision attribute is, the more the decision attribute can distinguish the domain of interest, otherwise, the roughness and uncertainty of the conditional attribute with low dependency are larger during classification, and the decision is not facilitated.

Degree of similarity of features

In the embodiment of the present application, the target object o is comprehensively calculated by 7 features of distance, color, edge, grammar, shape, moving direction, and overlapping area_iAnd an observation object z_kThe similarity between them, the 7 feature similarity functions are defined as follows:

wherein f is₁(o_i,z_k) As a function of spatial distance feature similarity measure, f₂(o_i,z_k) As a geometric feature similarity measure function, f₃(o_i,z_k) As a function of the similarity measure of the direction of motion, f₄(o_i,z_k) As a function of a color feature similarity measure, f₅(o_i,z_k) As a gradient direction feature similarity measure function, f₆(o_i,z_k) As a similarity measure function of the literal features, f₇(o_i,z_k) A similarity measure function for the overlapping areas; (x)_o,y_o) Is a target object o_i(x) of (a)_z,y_z) For observation of object z_kCentral coordinate of (a), h_oIs a target object o_iThe height of the image of (a) is,is a space distance variance constant, h_zFor observation of object z_kThe height of the image of (a) is,is a geometric size variance constant, (x'_o,y'_o) Is the target object o at the last moment_iThe center coordinates of the center of the optical fiber,is the target object o at the last moment_iIs projected on the image coordinate axis,is a motion direction variance constant, rho (-) represents the solving of the Babbitt coefficient, H_r(. cndot.) represents a color histogram,is a constant of variance of the color histogram, H_g(. cndot.) represents a block gradient direction histogram feature,is a gradient direction variance constant, H_l(. cndot.) represents a histogram of textural features,is a texture feature variance constant, w (o)_i,z_k) Representing a target object o_iAnd an observation object z_kThe degree of shielding between the two parts is,is the overlap area variance constant.

For w (o)_i,z_k) Is defined as follows: FIG. 1-b is a schematic diagram of occlusion between prediction results of different target objects according to the present invention. As shown in fig. 1-B, in the current video frame, the shapes of the tracking icons of prediction result a and prediction result B are both rectangles, and there is an overlap between the two, and the parameters of prediction result a are expressed as: [ x, y, w, d]Wherein x, y represent the coordinates of the rectangular frame, w represents the width of the rectangular frame, d represents the height of the rectangular frame, and the parameters of the prediction result B are expressed as: [ x ', y', w ', h']Wherein, x', y' denotes coordinates of a rectangular frame, w ' denotes a width of the rectangular frame, h ' denotes a height of the rectangular frame, and a hatched portion between the prediction result a and the prediction result B is expressed as: [ x ] of_o,y_o,w_o,h_o]And the overlapping part thereof is represented as:

from this, it can be seen that the area of the overlap between prediction result a and prediction result B is represented as w_o*h_o. If above-mentioned w_o、h_oDoes not satisfy w_o> 0 or h_oIf the area is more than 0, an overlapping rectangle is not formed between the two tracking rectangle frames, namely the area of the overlapping rectangle is 0.

Assuming that the predicted result a and the predicted result B are occluded as shown in fig. 2, and the overlapped shadow part between the two tracking rectangular frames represents an occlusion area, the occlusion degree between the two tracking rectangular frames is defined as:

wherein s (·) represents the area of the region, and the shielding degree satisfies that w (A, B) is more than or equal to 0 and less than or equal to 1. When w (A, B) is larger than 0, the occlusion between the prediction result A and the prediction result B is shown. And further by the vertical image coordinate value y of the bottom of two tracking rectangular frames respectively representing the prediction result A and the prediction result B_AAnd y_BIt can be seen that if y_A＞y_BIf yes, the prediction result B is shielded by the prediction result A, otherwise, the prediction result A is shielded by the prediction result B.

Feature screening

The embodiment of the invention selects the image characteristics of the target object by utilizing the attribute reduction capability of the neighborhood rough set, reduces the number of the characteristics according to the dependence of the decision attribute on the domain of discourse, and realizes the selection of useful characteristics. For feature f_n(n ═ 1,2,. 7) selectionThe method comprises the following specific steps:

1) calculating a target object o_iAnd all observed objects z for feature f_nThe similarity of (2) is taken as a domain of discourse U_n。

2) Decision attribute D_nIs defined asFor all the degrees of similarity f_nGreater than τ_cfOf the observation objectThe feature f is then calculated using equation (5)_nDecision attribute of (1) to domain of discourse U_nDegree of dependence of (gamma)_n。

3) Dependency of current characteristics gamma_n≥τ_cfWhen the feature is a valid feature, the latter fusion can be made.

The greater the dependency of the decision attribute on the domain of discourse is, the more favorable the decision attribute is for the decision by the conditional attribute, the greater the lower approximation of the decision D is, the smaller the boundary domain is, and the better the feature can distinguish the observation object corresponding to the target object from other observation objects. Threshold τ in this context_cfAnd δ were set experimentally to 0.7 and 0.1, respectively.

And after some characteristics without distinctiveness are removed through the neighborhood rough set, the remaining characteristics are fused. The method of similarity fusion is adopted for multi-feature fusion. Suppose a target object o_iAnd an observation object z_kDescribed by n characteristics, the similarity of the ith characteristic is f_i(o_i,z_k) Then, the weight and fusion mode of the feature are as follows:

maximum entropy intuitive fuzzy clustering introducing local information

Assume that the set of target objects at time t-1 isThe set of target objects at the time t can be predicted according to the Kalman filter asthe observed object at time t isThe predicted position of the target object is used as a clustering center, and the target function of the maximum entropy intuitive fuzzy clustering method added with local information is as follows:

wherein, mu_ikRepresenting an object of observationAnd a cluster centerThe degree of fuzzy membership between them,representing an object of observationAnd a cluster centerThe distance between them.Representing a local information distance measure, omega, between an observed object and a cluster center_ikIs the weight of the local information. Wherein,

the objective optimization function is:

α therein_k∈[0.40.6]Commonly referred to as a difference factor.

The fuzzy membership degree mu can be obtained by optimizing by using a Lagrange multiplier method_ikComprises the following steps:

in order to combine the intuition fuzzy characteristic, the membership function is expanded from the traditional fuzzy set to the intuition fuzzy set, and the membership function is modified by introducing an intuition index to be defined as a new intuition fuzzy membership:

wherein,for a new degree of intuitive fuzzy membership,indicating an intuitive index pi_ikUpper implicit membership proportion, pi_ikIs given by equation (14). The corresponding intuitive index is expressed as:

where φ is a normal number. The value range is [0.40.8 ].

Normalizing the intuition fuzzy membership to obtain the final intuition fuzzy membership:

approximate computation of local information distance measure

According to the proposed fuzzy clustering method, the main problem is to solve the distanceAnd local information distance measureAnd (4) calculating. According to the feature similarity defined by the above formula (6), the distance between the target object and the observed object is measured by the similarity of feature fusion. Observed object at time tAnd a cluster centerThe distance betweenCan be defined as:

to introduce local information of the target object, a target object o is defined as follows_iReference topology set of (2):

wherein w (o)_i,o_j) Is a target object o_iWith the target object o_jThe overlap ratio therebetween. Referencing a topology set to a target object o_iAnd taking the adjacent target object with the occlusion as a query, and retrieving the reference topological set which forms the target object with the adjacent target object meeting a certain occlusion degree. Similarly, an observed object z may be defined_kReference topology set of (2):

wherein,andthe number of elements of the reference topology set of the target object and the observation object is respectively represented.

Thus, the OSPA distance may be used to calculate the distance of the target object from the reference topological set of observed objectsOSPA distance considers two factors of the number of sets and the distance between the sets, and is used for a target object o_iAnd an observation object z_kThe OSPA distance of the reference topology set of (1) is as follows:

where c is a penalty on unpaired, q_hThe number represents the number of successfully paired target object reference set and reference topology, and is defined as:

wherein h is_klIs defined as:

when two reference topographies are in focusAndthe distance of the center of mass is minimum and conforms to (x)_o,y_o)-(x_z,y_z)||₂And when the T is less than or equal to T, the elements in the two topological sets are successfully matched.

Therefore, it isIndicates the sum of the number of pairings and the number of unpaired pairings,indicating the number of unsuccessful pairings and,for reference on a topological setThe distance between the centers of mass of the elements is defined as:

whereinAndrespectively represent the target object o_iReference topology set RET of_i ^oCentroid coordinates of the first element and the observed object z_kReference topology set ofThe coordinates of the centroid of the upper r-th element,is a constant value of the variance of the spatial distance,is the overlap area variance constant.

As shown in fig. 1-c, the reference topology set of the target object 1 is the target object 2, and at the 33 rd frame, the target object 1 loses most of the appearance information because it is heavily shielded by the target object 2, but the information of the reference topology set is completely stored, so the distance between the target object and the reference topology set of the observation object can be usedAnd performing association.

Consider neighboring target to target object o_iLocal information weight omega_ikInfluence of, ω_ikIs defined as:

wherein N is_iIndicates the number of target objects in the vicinity of the target object, which means o_iIs greater than a specified threshold value w (i, j) > tau₁Of a neighboring target object o_jNumber, N_i(j) The influence factor to which the neighboring target object representing the target object i influences is defined as:

wherein w (i, j) represents the degree of occlusion of the target object with other target objects when the target object and other neighboring target objects exceed the degree of occlusion by a certain threshold τ₂(0<τ₁<τ₂) When it is said that the target object is affected by more severe occlusion by a neighboring target object, τ will be used herein₁、τ₂Is set to tau₁＝0.1，τ₂＝0.6。

Fuzzy trajectory correlation

When the target track is blocked for a long time or the track is not updated for a long time, the target track can generate batch breaking. Therefore, in order to solve the problem of continuous tracking of a target track under a long-time shielding condition, the embodiment of the application provides a track association method based on a fuzzy comprehensive function, which is used for connecting different sub-tracks of the same target at different moments. As shown in particular in fig. 1-d.

Firstly, constructing a credibility matrix based on a fuzzy comprehensive function

It is assumed that in the t-th frame,a set of already-terminated tracks is represented,a new set of trajectories is represented in which,andrespectively representing the number of terminated tracks and the number of new tracks, l_0,iIndicating the time at which the track i terminates, l_1,jIndicates the start time of the new track j, if l_1,jIf t, then new trackIs a new observation object to be associated. Is provided with a U_t＝{μ_ijIs asConfidence matrix, mu_ijIndicating the fuzzy integrated similarity of the track i and the new track j.

To calculate the similarity mu_ij7 features of distance, color, edge, grammar, shape, motion direction and overlapping area are introduced to comprehensively calculate the similarity of the trajectory and the target object. Distance feature similarity f₁(i, j), color feature similarity f₂(i, j), edge feature similarity f₃(i, j) similarity of grammatical features f₄(i, j), shape feature similarity f₅(i, j), motion direction feature similarity f₆(i, j) and overlap area feature similarity f₇(i, j) are defined as follows:

wherein (x)_i,y_i) Is the center coordinate of the target object i, (x)_j,y_j) Is the center coordinate of the target object j, h_iIs the image height of the target object j,is a space distance variance constant, h_jIs the image height of the target object j,is a geometric size variance constant, (x'_i,y′_i) The center coordinates of the target object i at the last time,the projection of the velocity of the target object i on the image coordinate axis at the previous moment,is a motion direction variance constant, rho (-) represents the solving of the Babbitt coefficient, H_r(. cndot.) represents a color histogram,is a motion direction variance constant, H_g(. cndot.) represents a block gradient direction histogram feature,is a gradient direction variance constant, H_l(. cndot.) represents a histogram of textural features,is a constant variance of the texture features, w (i, j) represents the degree of occlusion between the target object i and the observation object j,is the overlap area variance constant.

Similarity vector Lambda according to the similarity defined by equation (17-21)_t(i, j) is defined as follows:

wherein, Λ_t(i,j)∈[0,1]⁷,Respectively representing the average similarity of the characteristics of the distance, the color, the edge, the grammar, the shape, the motion direction, the overlapping area and the like of the terminal track i and the new track j at the t-th moment, and respectively defined as follows:

κ＝t-l_1,j+3

wherein f is₁ ^m,n(i,j),f₂ ^m,n(i,j),f₃ ^m,n(i,j),f₄ ^m,n(i,j),f₅ ^m,n(i,j),f₆ ^m,n(i, j) and f₇ ^m,n(i, j) represents the similarity of the characteristics of the distance, color, edge, grammar, shape, movement direction, and overlapping area between the ending trajectory i at the m-th time and the new trajectory j at the n-th time.

Based on the similarity definition, the similarity model based on the fuzzy synthesis function is defined as follows:

wherein ^ represents the large operation and the small operation. Then the fuzzy similarity mu_ijCan be defined as:

second, association rules

In order to solve multi-target association under the complex environment, the maximum similarity and the track quality between the terminal track i and the new track j at the nth moment are definedThe following were used:

if it is notThe ending track i and the new track j^*Association while a new track j^*No correlation is carried out with other terminal tracks, wherein epsilon represents a credibility threshold, and epsilon is more than or equal to 0.5 and less than or equal to 1 in the patent.

If it is notThe terminating track i and the new track j are considered^*The same track is adopted, and simultaneously, a linear interpolation method is adopted to connect the terminal track i with the new track j^*And changes the track number (ID) of the new track to the track number (ID) of the terminating track. Finally, the terminating track i is restored to the existing track i.

The following describes a data association method of a video object in an embodiment of the present application through a flow of steps, including:

step one, detecting a t frame image by using a Gaussian mixture detection method;

using mixed Gaussian detection method to process t frame imageDetecting the image to obtain an observation object setAnd assume that there is a current set of target objects

The embodiment of the application needs to predict the state of the target object according to the previous frame state of the target objectPredicting the position of the target object of the next frame by using a Kalman filtering method to obtain the predicted state of the target object

Acquiring image characteristics of a target object and an observation object in the t frame of image, and screening according to a neighborhood rough set;

the screening according to the neighborhood rough set specifically comprises the following steps:

a) computing a target objectAnd all the observed objectsFor feature f_nThe similarity of (2) is taken as a domain of discourse U_n；

b) Decision attribute D_nIs defined asFor all the degrees of similarity f_nGreater than τ_cfOf the observation objectThe feature f is then calculated using equation (5)_nDecision attribute of (1) to domain of discourse U_nDegree of dependence of (gamma)_n；

c) When feature f_nDegree of dependence of (gamma)_n≥τ_cfTime, characteristic f_nFor a valid feature, a later fusion may be performed.

Step three, calculating the multi-feature fusion distance of the screened features;

according to the feature similarity f calculated in the previous step_nComputing a set of target objects using equation (10)And a set of observation objectsSimilarity of multi-feature fusion ofMethod for calculating multi-feature fusion distance between target object and observation object by using formula (17)

Step four, calculating the distance of a reference topological set between the target object and the observation object;

the multi-feature fusion distance obtained according to the aboveSeparately computing a set of target objectsAnd a set of observed objectsUsing the formula (20) to find the distance between the reference topological set of the target object and the reference topological set of the observed object

Step five, calculating the intuitive fuzzy membership degree of the target object and the observation object;

calculating the intuitionistic fuzzy membership degree of the normalized target object and the observation object according to the reference topological set distance between the target object and the observation object and the formula (16)

And step six, constructing a correlation cost matrix according to the intuitive fuzzy membership degree of the target object and the observation object to correlate the target object.

And constructing a correlation cost matrix according to the intuitive fuzzy membership degree of the target object and the observation object.

For target object O at time t^tAnd an observation object Z^tCalculating each pairIntuitive fuzzy degree of membershipTo construct an association cost matrix S:

in this context τ_aIs set to tau_a0.6. And associating the target object and the observation object of the video frame according to the maximum association degree by using the obtained association cost matrix S, wherein the method specifically comprises the following steps:

a) finding the maximum membership value from the correlation cost matrix SWhen s is_pq≥τ_aWhen the target object p corresponding to the row and the column where the target object p is located is associated with the observation object q, other elements in the row and the column are set to 0.

b) Repeating the previous step until s_pq<τ_aAnd when all the association pairs meeting the conditions are found, the association is stopped.

If the target object is successfully associated, performing state updating on the associated target object by using Kalman filtering; carrying out extrapolation prediction on the unassociated target object so as to maintain the track of the target object; the unassociated observed object can be directly used as a new track starting target point.

And seventhly, carrying out fuzzy track association on the track in the video.

If the target object association is unsuccessful, performing target track management on the target object, which specifically comprises the following steps:

a) it is assumed that the k-th frame,a set of already-terminated tracks is represented,representing a new set of tracks, using (26-36) to calculate a fuzzy confidence matrix between the terminated tracks and the new tracks

b) According to the confidence matrix U_kObtaining the final track i and the new track j with the maximum reliability by using the formula (37)^*If, ifThe ending track i and the new track j^*Association while a new track j^*No longer associated with other terminal tracks;

c) calculating the ending track i and the new track j by using the formula (38)^*Track ofQuality ofIf it is notThe terminating track i and the new track j are considered^*The same track is adopted, and simultaneously, a linear interpolation method is adopted to connect the terminal track i with the new track j^*And changes the track number (ID) of the new track to the track number (ID) of the terminating track. Finally, the terminating track i is restored to the existing track i.

d) Repeating steps b) -c) until the association is completed.

At the end of the track setIn the method, the terminal track which is not updated and exceeds 70 frames is removed, and the updating is not performed any more.

EXAMPLE III

Referring to fig. 2, an electronic device is provided according to an embodiment of the present application. The electronic device can be used to implement the data association method of the video object provided by the embodiment shown in fig. 1-a. As shown in fig. 2, the electronic device mainly includes:

an acquisition unit 210 configured to acquire a first image feature of a target object and a second image feature of an observation object, respectively;

a similarity calculation unit 220, configured to perform feature similarity calculation for N types of feature categories on the first image feature and the second image feature to obtain N groups of feature similarity results, where N is an integer greater than 1;

a feature fusion unit 230, configured to screen the N groups of feature similarity results based on the feature similarity of the rough set, and fuse the screened results to obtain a feature fusion result;

the correlation calculation unit 240 is configured to perform correlation cost matrix calculation according to the feature fusion result based on the maximum entropy intuitive fuzzy clustering to obtain a correlation cost matrix calculation result;

a track association unit 250, configured to determine whether the target object is associated with the observed object according to the association cost matrix calculation result, if not, perform target track management on the target object, and count new start tracks and end tracks of the target object in the video to obtain the end track set and the new track set; and performing fuzzy track association according to the terminal track set and the new track set, wherein the fuzzy track association comprises the following steps: s1, acquiring a terminal track set and a new track set in a video, and determining a fuzzy credibility matrix of the terminal track set and the new track set; s2 finding the ending trace i in the ending trace set and the new trace j at the nth time in the new trace set^*: determining the ending track i and the new track j according to the fuzzy credibility matrix^*The maximum degree of identity between; if the maximum degree of identity is greater than or equal to the reliability threshold value, the terminal track i and the new track j^*Is associated with and the new track j^*Not associated with other terminal tracks; s3 when the ending track i and the new track j^*When in association, if the terminal track i is associated with the new track j^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Are the same track; if the ending track i and the new track j are^*If the track quality between the end track i and the new track j is less than the track quality threshold value, the end track i and the new track j are determined^*Are not the same trajectory; repeating the steps S2 and S3 until all the tracks in the video target data are completely associated.

It should be noted that, in the embodiment of the electronic device illustrated in fig. 2, the division of the functional modules is only an example, and in practical applications, the above functions may be distributed by different functional modules according to needs, for example, configuration requirements of corresponding hardware or convenience of implementation of software, that is, the internal structure of the electronic device is divided into different functional modules to complete all or part of the functions described above. In practical applications, the corresponding functional modules in this embodiment may be implemented by corresponding hardware, or may be implemented by corresponding hardware executing corresponding software. The above description principles can be applied to various embodiments provided in the present specification, and are not described in detail below.

For a specific process of each function module in the electronic device provided in this embodiment to implement each function, please refer to the specific content described in the embodiment shown in fig. 1-a, which is not described herein again.

Example four

An embodiment of the present application provides an electronic device, please refer to fig. 3, which includes:

a memory 301, a processor 302 and a computer program stored on the memory 301 and executable on the processor 302, which when executed by the processor 302, implement the data association method for video objects described in the embodiment of fig. 1-a.

Further, the electronic device further includes:

at least one input device 303 and at least one output device 304.

The memory 301, the processor 302, the input device 303, and the output device 304 are connected via a bus 305.

The input device 303 may be a camera, a touch panel, a physical button, a mouse, or the like. The output device 304 may specifically be a display screen.

The Memory 301 may be a Random Access Memory (RAM) Memory or a non-volatile Memory (non-volatile Memory), such as a magnetic disk Memory. The memory 301 is used to store a set of executable program code, and the processor 302 is coupled to the memory 301.

Further, an embodiment of the present application also provides a computer-readable storage medium, where the computer-readable storage medium may be provided in an electronic device in the foregoing embodiments, and the computer-readable storage medium may be the memory in the foregoing embodiment shown in fig. 3. The computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements the data association method for video objects described in the foregoing embodiment shown in fig. 1-a. Further, the computer-readable storage medium may be various media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a RAM, a magnetic disk, or an optical disk.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a readable storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned readable storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

It should be noted that, for the sake of simplicity, the above-mentioned method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present application is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

The above description is provided for the data association method, apparatus and storage medium of video object, and for those skilled in the art, there may be variations in the embodiments and application ranges according to the ideas of the embodiments of the present application.

Claims

1. A method for data association of a video object, comprising:

2. The method of claim 1,

the determining the fuzzy credibility matrix of the terminal track set and the new track set comprises:

3. The method of claim 2,

determining the ending track i and the new track j according to the fuzzy credibility matrix^*The maximum degree of identity between, including:

whereinAndand (d) similarity of features such as distance, color, edge, texture, shape, moving direction and overlapping area between the terminal track i at the m-th moment and the new track j at the n-th moment is represented, and k is t-l_1,j+3；

Determining the ending trace i andthe new trajectory j^*Maximum degree of identity between:

4. the method of claim 3,

if the ending track i and the new track j are^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Before the same track, the method comprises the following steps:

5. The method of claim 3,

if the maximum degree of identity is greater than or equal to a reliability threshold value, the terminal track i and the new track j^*Associating, comprising:

6. The method of claim 4,

if the ending track i and the new track j are^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Are in the same trackThe method comprises the following steps:

if it is notThe terminating track i and the new track j are considered^*The same track is adopted, and simultaneously, a linear interpolation method is adopted to connect the terminal track i with the new track j^*And the track number of the new track is changed into the track number of the terminal track, and the terminal track i is restored to the existing track i.

7. An electronic device, comprising:

a track association unit, configured to determine whether the target object is associated with the observed object according to the association cost matrix calculation result, if not, perform target track management on the target object, and count new start tracks and end tracks of the target object in a video to obtain the end track set and the new track set; and performing fuzzy track association according to the terminal track set and the new track set, wherein the fuzzy track association comprises the following steps: s1, acquiring a terminal track set and a new track set in a video, and determining a fuzzy credibility matrix of the terminal track set and the new track set; s2 finding the ending trace i in the ending trace set and the new trace j at the nth time in the new trace set^*: according to the fuzzy credibilityDetermining the ending track i and the new track j by the degree matrix^*The maximum degree of identity between; if the maximum degree of identity is greater than or equal to the reliability threshold value, the terminal track i and the new track j^*Is associated with and the new track j^*Not associated with other terminal tracks; s3 when the ending track i and the new track j^*When in association, if the terminal track i is associated with the new track j^*If the track quality between the two is greater than or equal to the track quality threshold, determining the ending track i and the new track j^*Are the same track; if the ending track i and the new track j are^*If the track quality between the end track i and the new track j is less than the track quality threshold value, the end track i and the new track j are determined^*Are not the same trajectory; repeating the steps S2 and S3 until all the tracks in the video target data are completely associated.

8. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 6.