WO2021027344A1

WO2021027344A1 - Image processing method and device, electronic apparatus, and storage medium

Info

Publication number: WO2021027344A1
Application number: PCT/CN2020/089402
Authority: WO
Inventors: 黄垂碧; 莫涛; 杨川; 秦晨翀; 陈宇恒
Original assignee: 深圳市商汤科技有限公司
Priority date: 2019-08-15
Filing date: 2020-05-09
Publication date: 2021-02-18
Also published as: KR20220025052A; CN110502651B; CN110502651A; JP2022542127A; TW202109514A; TWI761851B

Abstract

An image processing method and device, an electronic apparatus, and a storage medium. The method comprises: acquiring an image dataset comprising multiple images and first indices respectively associated with the multiple images, the first indices being used to determine spatio-temporal data of objects in the images (S10); performing distributed clustering of the images in the image dataset, and obtaining at least one cluster (S20); and determining, on the basis of the first indices associated with the images in the obtained cluster, spatio-temporal trajectory information of the objects corresponding to the cluster (S30). The method can realize quick and efficient acquisition of a spatio-temporal trajectory of an object.

Description

Image processing method and device, electronic equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910755628.5, and the invention title is "Image processing method and device, electronic equipment and storage medium" on August 15, 2019, the entire content of which is incorporated by reference In this application.

Technical field

The present disclosure relates to the field of computer vision technology, and in particular to an image processing method and device, electronic equipment, and storage medium.

Background technique

With the construction of a safe city, the city-level surveillance system is producing massive snapshots of human faces every day. These face data have the characteristics of large scale, wide time and regional distribution.

Summary of the invention

The present disclosure provides a technical solution for image processing.

According to an aspect of the present disclosure, there is provided an image processing method, including: acquiring an image data set, the image data set including a plurality of images and first indexes respectively associated with the plurality of images, the first The index is used to determine the spatio-temporal data of the object in the image; perform distributed clustering processing on the images in the image data set to obtain at least one cluster; based on the obtained first cluster associated with the images in the cluster The index determines the spatiotemporal trajectory information of the object corresponding to the cluster.

In some possible implementation manners, the method further includes: acquiring image features of the input image; performing quantization processing on the image features of the input image to obtain the quantized features of the input image; based on the quantized features of the input image And the cluster center of the at least one cluster obtained by the distributed clustering process to determine the cluster where the input image is located.

In some possible implementation manners, the determining the cluster where the input image is located based on the quantized feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering processing includes: Obtain the third degree of similarity between the quantized feature of the input image and the quantized feature of the cluster center of the at least one cluster obtained by the distributed clustering process; determine the degree of similarity between the quantized feature of the input image The K3 class centers with the third highest similarity, where K3 is an integer greater than or equal to 1, obtain the fourth similarity between the image features of the input image and the image features of the K3 class centers; in response to the The fourth similarity between the image feature of any one of the K3 class centers and the image feature of the input image is the highest, and the fourth similarity is greater than the third threshold, the input image is added to any one of the The cluster corresponding to the cluster center.

In some possible implementation manners, the determining the cluster where the input image is located based on the quantized feature of the input image and the cluster center obtained by the distributed clustering processing further includes responding to There is no class center whose fourth similarity with the image features of the input image is greater than the third threshold, and the distributed clustering is performed based on the quantized features of the input image and the quantized features of the images in the image data set. Class processing, at least one new cluster is obtained.

In some possible implementation manners, the first index includes at least one of the following information: the collection time of the image, the collection location, the identification of the image collection device that collected the image, and the installation of the image collection device s position.

In some possible implementation manners, the performing distributed clustering processing on the images in the image data set to obtain at least one cluster includes: acquiring image features of the images in the image data set in a distributed and parallel manner Distributed parallel quantization processing on the image features to obtain the quantized features corresponding to the image features; based on the quantized features corresponding to the image in the image data set, the distributed clustering processing is performed to obtain the At least one cluster.

In some possible implementation manners, the distributed and parallel acquisition of the image features of the images in the image data set includes: grouping the multiple images in the image data set to obtain multiple image groups Input the multiple image groups into multiple feature extraction models, and use the multiple feature extraction models to execute feature extraction processing of images in the image group corresponding to the feature extraction model in a distributed and parallel manner to obtain the multiple The image features of each image, where each feature extraction model inputs different image groups.

In some possible implementation manners, the distributed and parallel quantization processing on the image features to obtain the quantized features corresponding to the image features includes: grouping the image features of the multiple images to obtain multiple The first grouping, the first grouping includes image features of at least one image; the quantization processing of the image features of the plurality of first groups is executed in parallel in a distributed manner to obtain the quantized features corresponding to the image features.

In some possible implementation manners, before the distributed and parallel execution of the quantization processing of the image features of the multiple first groups to obtain the quantized features corresponding to the image features, the method further includes: Configure second indexes for each of the first groups to obtain multiple second indexes; the distributed and parallel execution of the quantization processing of the image features of the multiple first groups to obtain the quantized features corresponding to the image features includes: The plurality of second indexes are respectively allocated to a plurality of quantizers, and each of the plurality of quantizers is allocated a different second index; the plurality of quantizers are used to execute the allocated second indexes respectively in parallel Quantization processing of image features in the first group corresponding to the second index.

In some possible implementation manners, the quantization processing includes product quantization encoding processing.

In some possible implementation manners, the performing the distributed clustering process based on the quantitative features corresponding to the images in the image data set to obtain the at least one cluster includes: obtaining the image data set The first degree of similarity between the quantized feature of any image and the quantized features of the rest of the image; based on the first degree of similarity, determine the K1 neighbor image of the any image, and the quantized feature of the K1 neighbor image is The first K1 quantized features with the highest similarity of the quantized features of any image, where K1 is an integer greater than or equal to 1, and the distribution is determined by using the any image and K1 neighboring images of the any image The clustering result of the formula clustering process.

In some possible implementation manners, the determining the clustering result of the distributed clustering process by using any image and K1 neighbor images of the any image includes: selecting from the K1 neighbor images A first image set whose first similarity with the quantized feature of any image is greater than a first threshold; mark all images in the first image set and any image as the first state, and based on Each image marked as a first state forms a cluster, and the first state is a state including the same object in the image.

In some possible implementation manners, the determining the clustering result of the distributed clustering processing by using the any image and the K1 neighbor images of the any image includes: acquiring image features of the any image The second degree of similarity with the image features of the K1 neighbor image of any image; based on the second degree of similarity, the K2 neighbor image of the any image is determined, and the image feature of the K2 neighbor image is The K2 image features with the second highest similarity to the image features of any of the image features in the K1 neighbor images, K2 is an integer greater than or equal to 1 and less than or equal to K1; and the K2 neighbor images are selected from the K2 neighbor images. The second image set in which the second similarity of the image feature of the any image is greater than the second threshold; all the images in the second image set and the any image are marked as the first state, and based on being Each image marked as a first state forms a cluster, and the first state is a state including the same object in the image.

In some possible implementation manners, before the acquiring the first similarity between the quantized features of any image in the image data set and the quantized features of other images, the method further includes: Grouping the quantized features of the multiple images to obtain multiple second groups, where the second grouping includes the quantized features of at least one image; and, obtaining the quantized features of any image in the image data set and The first degree of similarity between the quantized features of the remaining images includes: obtaining the first degree of similarity between the quantized features of the images in the second group and the quantized features of the remaining images in a distributed and parallel manner.

In some possible implementation manners, before the distributed and parallel acquisition of the first similarity between the quantized features of the images in the second group and the quantized features of the remaining images, the method further includes: The plurality of second groups respectively configure third indexes to obtain a plurality of third indexes; and, the distributed and parallel acquisition of the quantized features of the images in the second group and the quantized features of the remaining images The first similarity includes: establishing a similarity calculation task corresponding to the third index based on the third index, and the similarity calculation task is to obtain the target image in the second group corresponding to the third index The first similarity between the quantized feature and the quantized features of all images other than the target image; the similarity acquisition task corresponding to each third index of the plurality of third indexes is executed in a distributed and parallel manner.

In some possible implementation manners, the method further includes: determining the cluster center of the cluster obtained by the distributed clustering processing; configuring a fourth index for the cluster center, and storing the fourth index in association with each other. Index and corresponding class center.

In some possible implementation manners, the determining the cluster center of the cluster obtained by the distributed clustering processing includes: determining the cluster center based on the average value of the image features of the images in the at least one cluster. State the cluster center.

In some possible implementation manners, the determining the spatiotemporal trajectory information of the object corresponding to the cluster based on the obtained first index associated with the images in the cluster includes: based on each image in the cluster The associated first index determines the time information and location information of the object corresponding to the cluster; and determines the spatiotemporal trajectory information of the object based on the time information and location information.

In some possible implementation manners, the method further includes: determining an object identity corresponding to each of the clusters based on the identity feature of at least one object in the identity feature library.

In some possible implementation manners, the determining the object identity corresponding to each of the clusters based on the identity feature of at least one object in the identity feature library includes: obtaining the quantitative feature of the known object in the identity feature library ; Determine the fifth similarity between the quantitative feature of the known object and the quantitative feature of the at least one cluster center, and determine the K4 with the fifth highest similarity to the quantitative feature of the cluster center Quantified features of known objects; acquiring the sixth similarity between the image features of the cluster center and the corresponding image features of K4 known objects; responding to the K4 known object’s The sixth similarity between the image feature and the image feature of the class center is the highest and the sixth similarity is greater than a fourth threshold, and it is determined that the known object with the highest sixth similarity corresponds to the class center The cluster matches.

In some possible implementation manners, the determining the object identity corresponding to each of the clusters based on the identity feature of at least one object in the identity feature library further includes: responding to the image features of the K4 known objects The sixth similarity with the image feature of the corresponding cluster center is all less than the fourth threshold, and it is determined that there is no cluster matching the known object.

According to a second aspect of the present disclosure, there is provided an image processing device, which includes: an acquisition module for acquiring an image data set, the image data set including a plurality of images and a first image associated with the plurality of images. An index, the first index is used to determine the spatiotemporal data of the object in the image; a clustering module is used to perform distributed clustering processing on the images in the image data set to obtain at least one cluster; determine The module is used to determine the spatiotemporal trajectory information of the object corresponding to the cluster based on the obtained first index associated with the image in the cluster.

In some possible implementation manners, the device further includes an incremental clustering module, which is used to obtain image features of the input image; perform quantization processing on the image features of the input image to obtain the quantized features of the input image; Determine the cluster where the input image is located based on the quantitative feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering process.

In some possible implementations, the incremental clustering module is further configured to obtain the difference between the quantitative feature of the input image and the quantitative feature of the cluster center of the at least one cluster obtained by the distributed clustering process. Determine the K3 class centers with the highest third similarity between the quantized features of the input image; obtain the difference between the image features of the input image and the image features of the K3 class centers The fourth degree of similarity; in the case that the fourth degree of similarity between the image feature of any one of the K3 class centers and the image feature of the input image is the highest and the fourth degree of similarity is greater than the third threshold, The input image is added to the cluster corresponding to the center of any type, and K3 is an integer greater than or equal to 1.

In some possible implementation manners, the incremental clustering module is further configured to, in the case that there is no cluster center with a fourth similarity greater than a third threshold between the image features of the input image, based on the The quantized features of the input image and the quantized features of the images in the image data set execute the distributed clustering process to obtain at least one new cluster.

In some possible implementation manners, the clustering module includes: a first distribution processing unit, which is used to obtain image features of the images in the image data set in a distributed and parallel manner; a second distribution processing unit, which uses Distributedly perform quantization processing on the image features in parallel to obtain the quantized features corresponding to the image features; a clustering unit is used to execute the distributed based on the quantized features corresponding to the image in the image data set Clustering processing to obtain the at least one cluster.

In some possible implementation manners, the first distribution processing unit is further configured to group the multiple images in the image data set to obtain multiple image groups; and input the multiple image groups into multiple image groups respectively. A feature extraction model, which uses the multiple feature extraction models to execute feature extraction processing of images in an image group corresponding to the feature extraction model in a distributed and parallel manner to obtain image features of the multiple images, wherein each feature extraction model The input image group is different.

In some possible implementation manners, the second distribution processing unit is further configured to perform grouping processing on the image features of the multiple images to obtain multiple first groups, and the first group includes the image features of at least one image ; Distributed and parallel execution of the quantization processing of the image features of the plurality of first groups, to obtain the quantized feature corresponding to the image feature.

In some possible implementation manners, the second distribution processing unit is further configured to execute the quantization processing of the image features of the plurality of first groups in the distributed parallel to obtain the quantized feature corresponding to the image feature, Respectively configure second indexes for the plurality of first groups to obtain a plurality of second indexes; and are used to allocate the plurality of second indexes to a plurality of quantizers, each of the plurality of quantizers The allocated second indexes are different; and the multiple quantizers are used to respectively execute quantization processing of image features in the first group corresponding to the allocated second indexes in parallel.

In some possible implementation manners, the clustering unit is further configured to obtain a first degree of similarity between a quantized feature of any image in the image data set and quantized features of other images; based on the first degree of similarity, Determine the K1 neighbor image of any image, the quantized feature of the K1 neighbor image is the K1 quantized feature with the highest first similarity to the quantized feature of the any image, and the K1 is greater than or equal to 1 Integer; the clustering result of the distributed clustering process is determined by using the any image and the K1 neighbor image of the any image.

In some possible implementation manners, the clustering unit is further configured to select, from the K1 neighboring images, a first image set whose first similarity with the quantized feature of any image is greater than a first threshold. ; Mark all the images in the first image set and any one of the images as a first state, and form a cluster based on each image marked as the first state, the first state is that the images include the same object status.

In some possible implementation manners, the clustering unit is further configured to obtain a second degree of similarity between the image feature of any image and the image feature of the K1 neighbor image of the any image; based on the first Second similarity, determining the K2 neighbor image of any image, and the image feature of the K2 neighbor image is the K2 image feature with the second highest similarity between the K1 neighbor image and the image feature of the any image , K2 is an integer greater than or equal to 1 and less than or equal to K1; selecting a second image set whose second similarity with the image feature of any image is greater than a second threshold from the K2 neighboring images; All the images in the second image set and any one of the images are marked as the first state, and a cluster is formed based on each image marked as the first state, and the first state is the images that include the same object status.

In some possible implementation manners, the clustering unit is further configured to compare the image data with the first similarity between the quantized features of any image in the image dataset and the quantized features of other images. The quantized features of the multiple images in the data set are grouped to obtain multiple second groups, where the second group includes the quantized features of at least one image; and the quantization of any image in the image data set is obtained The first degree of similarity between the features and the quantized features of the remaining images includes: obtaining the first degree of similarity between the quantized features of the images in the second group and the quantized features of the remaining images in a distributed and parallel manner.

In some possible implementation manners, the clustering unit is further configured to obtain the first similarity between the quantized features of the images in the second group and the quantized features of the remaining images in the distributed and parallel manner. , Configure a third index for each of the plurality of second groups to obtain a plurality of third indexes; and, the distributed and parallel acquisition of the quantized features of the images in the second group and the quantized features of the remaining images The first similarity between the two includes: establishing a similarity calculation task corresponding to the third index based on the third index, and the similarity calculation task is to obtain a target in the second group corresponding to the third index The first similarity between the quantized feature of the image and the quantized features of all the images except the target image; the similarity acquisition task corresponding to each third index of the plurality of third indexes is executed in a distributed and parallel manner.

In some possible implementation manners, the class center determining module is used to determine the class center of the cluster obtained by the distributed clustering process; configure a fourth index for the class center and store it in association The fourth index and the corresponding class center.

In some possible implementation manners, the cluster center determining module is further configured to determine the cluster center based on the average value of the image features of each image in the at least one cluster.

In some possible implementation manners, the determining module is further configured to determine the time information and location information of the object corresponding to the cluster based on the first index associated with each image in the cluster; based on the time information and The location information determines the spatiotemporal trajectory information of the object.

In some possible implementation manners, the device further includes an identity determining module, which is configured to determine the identity of the object corresponding to each cluster based on the identity feature of at least one object in the identity feature library.

In some possible implementations, the identity determination module is further configured to obtain the quantitative characteristics of the known objects in the identity feature library; determine the quantitative characteristics of the known objects and the cluster center of the at least one cluster Quantify the fifth similarity between the quantized features, and determine the quantized features of the K4 known objects with the fifth highest similarity to the quantized features of the class center; obtain the image features of the class center and the corresponding K4 The sixth degree of similarity between the image features of known objects; the sixth degree of similarity between the image feature of a known object in the K4 known objects and the image feature of the class center is the highest and the sixth degree of similarity When the similarity is greater than the fourth threshold, it is determined that the known object with the sixth highest similarity matches the cluster corresponding to the cluster center.

In some possible implementation manners, the identity determination module is further configured to: when the sixth similarity between the image features of the K4 known objects and the image features of the corresponding class center is less than the fourth threshold , It is determined that there is no cluster matching the known object.

According to a third aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to Perform the method described in any one of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions, when executed by a processor, implement the method described in any one of the first aspects.

According to a fifth aspect of the present disclosure, there is provided a computer program, the computer program comprising computer readable code, when the computer readable code runs in an electronic device, the processor in the electronic device executes Implement the method described in any one of the first aspect.

In the embodiments of the present disclosure, each image can be configured with corresponding index information to determine the spatiotemporal data of objects in the image. Based on this configuration, the spatiotemporal trajectories of different objects can be analyzed. After performing distributed clustering, the image set corresponding to each object is obtained (a cluster is equivalent to the image set of an object), and the index information (first index) associated with each image in the cluster can be obtained. The spatiotemporal trajectory information of the objects corresponding to the clustering can realize the trajectory analysis of different objects. At the same time, the embodiments of the present disclosure adopt a distributed clustering method, which can improve the efficiency of clustering, so that the spatiotemporal trajectory of the object can be obtained quickly and effectively.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure.

According to the following detailed description of exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present disclosure will become clear.

Description of the drawings

The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments that conform to the disclosure and are used together with the specification to explain the technical solutions of the disclosure.

Fig. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure;

Fig. 2 shows a flowchart of step S20 in an image processing method according to an embodiment of the present disclosure;

Fig. 3 shows a flowchart of step S21 in an image processing method according to an embodiment of the present disclosure;

Fig. 4 shows a flowchart of step S22 in an image processing method according to an embodiment of the present disclosure;

Fig. 5 shows a flowchart of step S23 in an image processing method according to an embodiment of the present disclosure;

Fig. 6 shows a flowchart of step S233 in an image processing method according to an embodiment of the present disclosure;

Fig. 7 shows another flowchart of step S233 in an image processing method according to an embodiment of the present disclosure;

Fig. 8 shows a flowchart of an image processing method for performing clustering increment processing according to an embodiment of the present disclosure;

Fig. 9 shows a flowchart of step S43 in an image processing method according to an embodiment of the present disclosure;

FIG. 10 shows a flowchart of determining the identity of objects matched by clusters in an image processing method according to an embodiment of the present disclosure;

Fig. 11 shows a block diagram of an image processing device according to an embodiment of the present disclosure;

Fig. 12 shows a block diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 13 shows a block diagram of another electronic device according to an embodiment of the present disclosure.

detailed description

Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the drawings. The same reference numerals in the drawings indicate elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, unless otherwise noted, the drawings are not necessarily drawn to scale.

The dedicated word "exemplary" here means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" need not be construed as being superior or better than other embodiments.

The term "and/or" in this article is only an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, exist alone B these three situations. In addition, the term "at least one" in this document means any one or any combination of at least two of the multiple, for example, including at least one of A, B, and C, may mean including A, Any one or more elements selected in the set formed by B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some instances, the methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present disclosure.

It can be understood that, without violating the principle logic, the various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment, which is limited in length and will not be repeated in this disclosure.

In addition, the present disclosure also provides image processing devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any image processing method provided in the present disclosure. For the corresponding technical solutions and descriptions, refer to the corresponding records in the method section. ,No longer.

The image processing method provided by the embodiments of the present disclosure can be applied to any image processing device. For example, the image processing method can be executed by a terminal device or a server or other processing device. The terminal device can be a User Equipment (UE), Mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc., are not illustrated in this disclosure. In addition, in some possible implementation manners, the image processing method may be implemented by a processor calling computer-readable instructions stored in the memory.

The embodiments of the present disclosure are described in detail below. Fig. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure. As shown in Fig. 1, the image processing method may include:

S10: Obtain an image data set, the image data set including a plurality of images and a first index respectively associated with the plurality of images, the first index being used to determine spatiotemporal data of objects in the images;

In some possible implementations, the image data set may include multiple images, and the multiple images may be acquired by an image acquisition device, and each image may be acquired by the same image acquisition device, or may also be acquired by different image devices, This disclosure does not specifically limit this. For example, image capture devices can be deployed in streets, shopping malls, security areas, homes, communities, or other areas, and images in corresponding places can be captured by deploying image capture devices. The image obtained by the embodiment of the present disclosure may be an image captured by at least one image capture device, and the image capture device may include a mobile phone, a camera, or other devices capable of capturing images, and the present disclosure will not illustrate them one by one.

In some possible implementation manners, the images in the image data set of the embodiments of the present disclosure may include objects of the same type, for example, they may include person objects. Correspondingly, the time and space of the same person object can be obtained through the image processing method of the embodiments of the present disclosure. Track information. Or, in other embodiments, the images in the image data set may also include other types of objects, such as animals, so that the spatiotemporal trajectory of the same animal can be determined. The present disclosure does not specifically limit the type of object in the image.

In some possible implementations, the method of obtaining the image data set may include directly connecting with an image capturing device to receive the captured images, or may also connect with a server or other electronic devices to receive images transmitted by the server or other electronic devices. . In addition, the images in the image data set in the embodiments of the present disclosure may also be preprocessed images. For example, the preprocessing may intercept images including human faces (face images) from the collected images, or delete collected images. The image has a low signal-to-noise ratio, is blurry or does not include an image of human objects. The foregoing is only an exemplary description, and the present disclosure does not limit the specific manner of obtaining the image data set.

In some possible implementation manners, the image data set further includes a first index associated with each image, where the first index is used to determine spatiotemporal data corresponding to the image, and the spatiotemporal data includes at least one of time data and spatial location data, for example The first index may include at least one of the following information: at least one of the collection time of the image, the collection location, the identification of the image collection device that collected the image, and the location where the image collection device is installed. Therefore, the temporal and spatial data information such as the appearance time and location of the object in the image can be determined through the first index associated with the image.

In some possible implementation manners, when the image capture device is capturing an image and sending the captured image, it may also send the first index of the image, for example, the time when the image was captured, the location where the image was captured, and the image capture device that captured the image ( Such as camera identification and other information. After the image and the first index are received, the image can be stored in association with the corresponding first index, such as stored in a database, which may be a local database or a cloud database.

S20: Perform distributed clustering processing on the images in the image data set to obtain at least one cluster;

In some possible implementations, after the image data set is obtained, distributed clustering processing may be performed on multiple images in the image data set. Among them, the images in the image data set can be images of the same object or different objects. The embodiments of the present disclosure can perform distributed clustering processing on the images to obtain multiple clusters, wherein the obtained images in each cluster include the same The image of the object. Among them, the distributed clustering process can perform the clustering process in parallel at the same time, and the clustering efficiency can be improved under the premise of ensuring the clustering accuracy.

In some possible implementation manners, it may be determined whether two images include the same object based on the similarity between the feature information corresponding to the images in the image data set. For example, the facial features of the human object in the image can be extracted to determine the similarity between the facial features of any two images, and the two images with the similarity greater than the threshold are determined to include the same object. The two images are Can be clustered together, and then get the clustering result. Or, in other embodiments, it is also possible to determine the face features of the K neighbors of the face features of each image (the K face features with the highest similarity, K is an integer greater than or equal to 1), and from the Among the facial features of K-nearest neighbors, the facial features whose similarity is greater than the threshold are determined. Alternatively, the clustering process can also be performed in other ways.

S30: Based on the obtained first index associated with each image in the cluster, determine spatiotemporal trajectory information of the object corresponding to the cluster.

In some possible implementations, the obtained images included in each cluster are images of the same object. Therefore, the object corresponding to the cluster can be determined through the first index associated with the images in the cluster. The appearance time, location and other information of. Through the time information and location information of each object, the spatiotemporal trajectory information about the object can be formed. For example, a time and location coordinate system can be established, and the appearance time and location of the object can be marked in the coordinate system through the first index of each image in a cluster, so that the spatiotemporal trajectory of the object can be displayed intuitively.

Based on the above configuration, the embodiment of the present disclosure can obtain the spatiotemporal trajectory information of the object corresponding to the cluster according to the first index associated with each image in each cluster based on the clustering result of the distributed clustering. The embodiment of the present disclosure effectively Mining potential trajectory information in the data, making full use of the value of the data and the resource input behind the data, and the embodiment of the present disclosure can accelerate the clustering processing speed through the distributed clustering clustering method.

The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Among them, after the image data set is obtained, clustering processing can be performed on the images in the image data set. Fig. 2 shows a flowchart of step S20 in an image processing method according to an embodiment of the present disclosure. Wherein, performing distributed clustering processing on the images in the image data set to obtain at least one cluster (step S20) may include:

S21: Acquire image features of the images in the image data set in a distributed and parallel manner;

S22: Distributed and parallelly perform quantization processing on the image feature to obtain the quantized feature corresponding to the image feature;

S23: Perform the distributed clustering process based on the quantified feature corresponding to the image in the image data set to obtain the at least one cluster.

In some possible implementation manners, the image may be a face image, and the corresponding image feature is the corresponding face feature. In step S21, the image feature of the image can be extracted by a feature extraction algorithm when acquiring the image feature of the image, or the image feature can be extracted by a neural network trained to perform feature extraction. Among them, the feature extraction algorithm may include at least one of Principal Components Analysis (PCA), Linear Discriminant Analysis (LDA), Independent Component Analysis (ICA) and other algorithms. Alternatively, other algorithms that can recognize the face area and obtain the characteristics of the face area can also be used. The neural network can be a convolutional neural network, such as the VGG network (Visual Geometry Group Network), which convolutes the image through the convolutional neural network. , And get the features of the face area of the image, that is, face features. The embodiment of the present disclosure does not specifically limit the feature extraction algorithm and the neural network for feature extraction, as long as the extraction of facial features (image features) can be realized, it can be used as an embodiment of the present disclosure.

In addition, in some possible implementation manners, in order to speed up the extraction of image features, the embodiments of the present disclosure may extract image features of each image in a distributed and parallel manner.

Fig. 3 shows a flowchart of step S21 in an image processing method according to an embodiment of the present disclosure, wherein the distributed and parallel acquisition of the image characteristics of the image in the image data set (step S21) may include:

S211: Group the multiple images in the image data set to obtain multiple image groups;

In some possible implementations, multiple images in the image data set may be grouped to obtain multiple image groups, and each image group may include at least one image. The method for grouping images may include average grouping or random grouping. The number of image groups obtained can be a pre-configured number of groups, and the number of groups can be less than or equal to the number of feature extraction models described below.

S212: Input the multiple image groups into multiple feature extraction models respectively, and use the multiple feature extraction models to execute feature extraction processing of images in the image groups corresponding to the feature extraction models in a distributed and parallel manner, to obtain the multiple feature extraction models. The image features of each image, where each feature extraction model inputs different image groups.

In some possible implementations, based on the obtained multiple image groups, a distributed parallel processing process of feature extraction may be performed. Each of the obtained multiple image groups can be assigned to one of the feature extraction models, and the feature extraction model is used to perform feature extraction processing of the images in the assigned image groups to obtain the image features of the corresponding images.

In some possible implementations, the feature extraction model can use the feature extraction algorithm described above to perform feature extraction processing, or the feature extraction model can be constructed as the feature extraction neural network described above to obtain image features, which is not specifically limited in the present disclosure.

In some possible implementations, multiple feature extraction models can be used to perform feature extraction of each image group in a distributed and parallel manner. For example, each feature extraction model can perform image feature extraction of one image group or multiple image groups at the same time, thereby Speed up feature extraction.

In some possible implementations, after the image feature of the image is obtained, the first index of the image and the image feature can be stored in association, the mapping relationship between the first index and the image feature can be established, and the mapping can be stored in the database. relationship. For example, the monitored real-time image stream can be input to the front-end distributed feature extraction module (feature extraction model). After the image features are extracted by the distributed feature extraction module, the image features are stored in the form of persistent features based on spatio-temporal information. The feature database, that is, the first index and image features are stored in the feature database in the form of persistent features. In the database, the persistence feature is stored in an index structure, and the first index key of the persistence feature in the database may include Region id, Camera idx, Captured time, and Sequence id. Among them, Region id is the camera area identifier, Camera idx is the camera id in the area, Captured time is the image capture time, and Sequence id is the self-increasing sequence identifier (such as the identifier of the number in sequence), which can be used for deduplication. The first index can constitute a unique identification of each image feature and can include the spatiotemporal information of the image feature. After the first index is stored in association with the corresponding image feature, the image feature (persistent feature) of each image can be easily obtained, and the spatiotemporal data information (time and location) of the object in the image can be obtained at the same time.

In some possible implementation manners, the embodiment of the present disclosure may perform quantization processing on the image after obtaining the image feature of the image to obtain the quantized feature corresponding to each image, that is, step S22 may be performed. The embodiments of the present disclosure may adopt product quantization (PQ) coding to obtain quantized features corresponding to image features of each image in the image data set. This quantization process is performed by a PQ quantizer, for example. The process of performing the quantization process through the PQ quantizer can include decomposing the vector space of the image feature into the Cartesian product of multiple low-dimensional vector spaces, and quantizing the low-dimensional vector space obtained by the decomposition, so that each image feature is There can be multiple quantized combination representations of low-dimensional spaces to obtain quantized features. Regarding the specific process of PQ encoding, this disclosure does not specifically describe this, and those skilled in the art can implement the quantization process through existing technical means. Image feature data compression can be achieved through quantization processing. For example, the image feature dimension of the image in the embodiment of the present disclosure can be N, and each dimension data is a float32 floating point number (ie, a 32-bit floating point number). The quantization feature obtained after the quantization process is The dimension can be N, and the data of each dimension is a half floating point number (that is, half-precision floating point number), that is, the amount of feature data can be reduced through quantization.

In some possible implementations, the quantization process of all image features can be performed by one quantizer, or the quantization process of image features can be performed by multiple quantizers, that is, the image features of all images can be performed by at least one quantizer. The quantization process obtains the quantized features corresponding to all images. Among them, when the image feature quantization process is executed by multiple quantizers, a distributed parallel execution method can be adopted to improve the processing speed.

The process of quantization and clustering will be described in detail below. As described in the above embodiments, in order to speed up the process of acquiring quantized features, the embodiments of the present disclosure may use distributed parallel execution to execute the quantization process, where 4 shows a flowchart of step S22 in an image processing method according to an embodiment of the present disclosure, wherein the distributed and parallel quantization processing on the image features to obtain the quantized features corresponding to the image features may include :

S221: Perform grouping processing on the image features of the multiple images to obtain multiple first groups, where the first group includes the image features of at least one image;

The embodiments of the present disclosure can group image features, and perform quantization processing on image features of each group in a distributed and parallel manner to obtain corresponding quantized features. When multiple quantizers are used to perform quantization of image features of an image data set, the multiple quantizers can be distributed and executed in parallel to quantize image features of different images, thereby reducing the time required for quantization and increasing the speed of calculation.

When performing the quantization process of each image feature in parallel, the image features can be divided into multiple groups (multiple first groups), and the first group can also be the same as the above grouping of images (image groups), that is, grouped according to images The image features are divided into corresponding number of groups, that is, the image feature of the image group can be directly obtained to determine the group of the image feature, or multiple first groups can be re-formed, which is not specifically limited in the present disclosure. Each first group includes at least one image feature of the image. The present disclosure does not specifically limit the number of the first group, which can be determined comprehensively according to the number of quantizers, processing capabilities, and the number of images, and can be determined by a person skilled in the art or a neural network according to actual needs.

In addition, in the embodiment of the present disclosure, the manner of grouping the image features of the multiple images may include: performing average grouping on the image features of the multiple images, or grouping the multiple images in a random manner. Perform grouping of image features. That is, the embodiment of the present disclosure can group the image features of each image in the image data set equally according to the number of groups, or can also group randomly to obtain multiple first groups. As long as the image features of multiple images can be divided into multiple first groups, it can be used as an embodiment of the present disclosure.

In some possible implementation manners, in the case of grouping image features to obtain multiple first groups, an identifier (such as a second index) may also be assigned to each first group, and the second index and the first group may be associated storage. For example, each image feature of the image data set can be formed into an image feature library T (feature database), and the image features in the image feature library T are grouped (sliced) to obtain n first groups {S ₁ , S ₂ ,. ..S _n }, where S _i represents the i-th first group, i is an integer greater than or equal to 1 and less than or equal to n, n represents the number of first groups, and n is an integer greater than or equal to 1. Each of the first groups may include image characteristics of at least one image. In order to easily distinguish each of the first packet and convenient quantization process may correspond to a first packet for the second index distribution _{_{{I 11, I 12, ...}} I 1n}, wherein the first index of the first packet may be S _i I _1i .

S222: Distributedly execute quantization processing of the image features of the multiple first groups in parallel to obtain quantized features corresponding to the image features.

In some possible implementation manners, after the image features are grouped to obtain multiple (at least two) first groups, the quantization processing of the image features in each first group may be performed in parallel respectively. For example, the quantization process may be performed by multiple quantizers, and each quantizer may perform quantization process of one or more image features of the first group, thereby speeding up the processing.

In some possible implementation manners, each quantizer may also be assigned a corresponding quantization processing task according to the second index of each first group. That is, the second index of each first group can be assigned to multiple quantizers, where each quantizer is assigned a different second index, and the quantizers respectively execute the quantization processing tasks corresponding to the assigned second indexes in parallel. , That is, perform the quantization process of the image features in the corresponding first group.

In addition, in order to further improve the quantization processing speed, the number of quantizers can be greater than or equal to the number of second groups, and each quantizer can be assigned at most one second index, that is, each quantizer can only execute one second index. The quantization processing of the image features in the corresponding first group. However, the foregoing is not a specific limitation of the embodiments of the present disclosure. The number of groups and the number of quantizers, and the number of first indexes allocated to each quantizer can be set according to different requirements.

As described in the foregoing embodiment, the quantization process can reduce the data amount of image features. The quantization processing method in the embodiment of the present disclosure may be product quantization (PQ) coding, for example, the quantization processing is performed by a PQ quantizer. Image feature data compression can be achieved through quantization processing. For example, the dimension of the image feature of the image in the embodiment of the present disclosure can be N, and the data of each dimension can be a float32 floating point number. The dimension of the quantization feature obtained after quantization can be N, and each The dimensional data is a half floating point number, that is, the amount of feature data can be reduced through quantization.

Through the above-mentioned embodiments, distributed parallel execution of quantization processing can be realized, and the speed of quantization processing can be improved.

After obtaining the quantized features of the images in the image data set, the quantized features can also be stored in association with the first index, so that the associated storage of the first index, second index, image, image feature, and quantized feature can be established to facilitate data retrieval Read and call.

In addition, when the quantized feature of the image is obtained, the quantized feature of each image can be used to perform clustering processing on the image data set, that is, step S23 can be performed. Among them, the images in the image data set may be images of the same object or different objects. The embodiments of the present disclosure may perform clustering processing on the images to obtain multiple clusters, and the obtained images in each cluster are images of the same object.

Fig. 5 shows a flowchart of step S23 in an image processing method according to an embodiment of the present disclosure, in which the distributed clustering process is executed based on the quantized feature corresponding to the image in the image data set, Obtaining the at least one cluster (step S23) may include:

S231: Acquire a first degree of similarity between a quantized feature of any image in the image data set and quantized features of other images;

In some possible implementations, after the quantitative features corresponding to the image features of the image are obtained, clustering of the image can be performed based on the quantitative features, that is, clusters of the same objects (clusters of objects with the same identity) are obtained. Among them, the embodiment of the present disclosure may first obtain the first similarity between any two quantized features, where the first similarity may be the cosine similarity. In other embodiments, other methods may be used to determine the difference between the quantized features. The first degree of similarity is not specifically limited in this disclosure.

In some possible implementation manners, one arithmetic unit can be used to calculate the first similarity between any two quantized features, or multiple arithmetic units can be used to calculate the first similarity between each quantized feature in a distributed and parallel manner. Parallel execution of calculations by multiple operators can speed up calculations.

Similarly, the embodiment of the present disclosure may also perform the first similarity between the quantitative features of each group and the remaining quantitative features based on the group distribution of the quantitative features. Wherein, the quantized features of each image can be grouped to obtain multiple second groups, and each second group includes at least one quantized feature of the image. Wherein, the second group can be determined directly based on the first group, that is, the corresponding quantized feature is determined according to the image feature of the first group, and the second group is directly formed according to the quantized feature corresponding to the image feature in the first group. Alternatively, it is also possible to regroup according to the quantized characteristics of each image to obtain multiple second groups. Similarly, the grouping method may be average grouping or random grouping, which is not specifically limited in the present disclosure.

After obtaining multiple second groups, you can also configure a third index for each second group to obtain multiple third indexes. The third index can distinguish each second group, and you can also associate the third index with the second group. storage. For example, the quantitative features of each image in the image data set can be formed into a quantitative feature library L, or the quantitative features can also be stored in the aforementioned image feature library T in association with each other. The quantitative features are related to the image, image feature, first index, second The index and the third index can be stored correspondingly. By grouping (slicing) the quantized features in the quantized feature library L, m second groups {L ₁ , L ₂ ,...L _m } can be obtained, where L _j represents the jth second group, and j is An integer greater than or equal to 1 and less than or equal to m, where m represents the number of second groups, and m is an integer greater than or equal to 1. In order to distinguish each second group conveniently and facilitate clustering processing, each second group can be assigned a corresponding third index {I ₂₁ , I ₂₂ ,...I _2m }, where the third index of the second group L _j can be Is I _2j .

After multiple second groups are obtained, multiple operators may be used to respectively execute the first similarity between the quantized features in the multiple second groups and the remaining quantized features. Since the data volume of the image data set may be very large, multiple operations can be used to execute the first similarity between any one of the quantized features in each second group and all the remaining quantized features in parallel.

In some possible implementation manners, multiple arithmetic units may be included, and the arithmetic units may be any electronic device with arithmetic processing function, such as a CPU, a processor, a single-chip computer, etc., which is not specifically limited in the present disclosure. Among them, each arithmetic unit can calculate the first degree of similarity between each quantized feature in one or more second groups and the quantized features of all other images, thereby speeding up the processing.

In some possible implementation manners, each arithmetic unit may also be assigned a corresponding similarity calculation task according to the third index of each second group. That is, the third index of each second group can be assigned to multiple arithmetic units, where each operation is assigned a different third index, and the similarity calculation tasks corresponding to the assigned third index can be executed in parallel through the arithmetic units. , The similarity calculation task is to obtain the first similarity between the quantized feature of the image in the second group corresponding to the third index and the quantized feature of all images except the image. Therefore, through the parallel execution of multiple arithmetic units, the first degree of similarity between the quantized features of any two images can be quickly obtained.

In addition, in order to further improve the similarity calculation speed, the number of operators can be greater than or equal to the number of the second group. At the same time, each operator can be assigned at most one third index, and each operator can execute only one third index. The first similarity calculation between the quantized features in the corresponding second group and the remaining quantized features. However, the foregoing is not a specific limitation of the embodiments of the present disclosure. The number of groups and the number of operators, and the number of third indexes allocated to each operator can be set according to different requirements.

S232: Determine the K1 neighbor image of any image based on the first similarity, where the quantized feature of the K1 neighbor image is the K1 quantized feature with the highest first similarity to the quantized feature of the any image , The K1 is an integer greater than or equal to 1;

After obtaining the first similarity between any two quantized features, the K1 neighbor image of any image can be obtained, that is, the image corresponding to the K1 quantized feature with the highest first similarity of the quantized feature of any image. Any image corresponding to the first K1 quantized feature with the highest similarity is a neighboring image, which represents an image that may include the same object. The first similarity sequence for any quantized feature can be obtained. The first similarity sequence is a sequence of quantized features sorted from high to low or from low to high with the any quantized feature, and the first similarity sequence is obtained After that, it is convenient to determine the K1 quantized features with the highest first similarity to any quantized feature, and then determine the K1 neighbors of any image. The number of K1 can be determined according to the number in the image data set, such as 20, 30, or other values in other embodiments, which is not specifically limited in the present disclosure.

S233: Determine a clustering result of the distributed clustering processing by using any image and the K1 neighbor image of the any image.

In some possible implementation manners, after the K1 neighbor image of each image is obtained, subsequent clustering processing may be performed. For example, K1 neighbors can be directly used to obtain clusters, or clusters can also be obtained based on image features of K1 neighbors. Fig. 6 shows a flowchart of step S233 in an image processing method according to an embodiment of the present disclosure. Wherein, the determining the clustering result of the distributed clustering process by using the any image and the K1 neighbor image of the any image (step S233) may include:

S23301: Select, from the K1 neighboring images, a first image set that has a first similarity with the quantized feature of any image that is greater than a first threshold;

S23302: Mark all the images and any one of the images in the first image set as a first state, and form a cluster based on each image marked as the first state, and the first state is that the images include the same The state of the object.

In some possible implementations, after obtaining the K1 neighbor images of each image (the K1 images with the highest first similarity of quantized features), the first similarity can be directly selected from the K1 neighbor images of each image For the images larger than the first threshold, the first image set is formed by selecting the images with the first similarity larger than the first threshold. The first threshold may be a set value, such as 90%, but it is not a specific limitation of the present disclosure. The image that is closest to any image can be selected by setting the first threshold.

After selecting a first image set with a first similarity greater than the first threshold from K1 neighbor images of any image, any image and all images in the selected first image set can be marked as the first state, And form a cluster according to the image in the first state. For example, if the image with the first similarity greater than the first threshold is selected from the K1 neighbor images of image A as the first image set including A1 and A2, then A and A1 and A2 can be marked as the first state respectively, from and The image with the first similarity greater than the first threshold is selected from the K1 neighbor images of A1 to include the first image set of B1. At this time, A1 and B1 can be marked as the first state, and there is no first image in the K1 neighbor images of A2. For images with similarity greater than the first threshold, A2 is no longer labeled in the first state. Through the above, A, A1, A2, and B1 can be classified into one cluster. That is, images A, A1, A2, and B1 include the same object.

The clustering results can be easily obtained in the above manner. Since the quantized feature reduces the image feature amount, the clustering speed can be accelerated, and the clustering accuracy can be improved by setting the first threshold.

In other possible embodiments, the similarity of image features can be further combined to improve the clustering accuracy. FIG. 7 shows another flowchart of step S233 in an image processing method according to an embodiment of the present disclosure, wherein the use of any image and K1 neighbor images of any image determines the distributed cluster The clustering result of the class processing (step S233) may also include:

S23311: Acquire a second degree of similarity between the image feature of any image and the image feature of the K1 neighbor image of the any image;

In some possible implementation manners, after obtaining the K1 neighbor images of each image (the K1 images with the highest quantization feature first similarity), the image features of any image and the corresponding K1 neighbor images can be further calculated The second degree of similarity between features. That is, after obtaining the K1 neighboring image of any image, the second similarity between the image feature of the any image and the image features of the K1 neighboring images can be further calculated. The second degree of similarity may also be a cosine degree of similarity, or in other embodiments, the degree of similarity may also be determined in other ways, which is not specifically limited in the present disclosure.

S23312: Determine the K2 neighbor image of any image based on the second similarity, and the image feature of the K2 neighbor image is the second similarity between the K1 neighbor image and the image feature of any image The highest K2 image features, K2 is an integer greater than or equal to 1 and less than or equal to K1;

In some possible implementations, the second similarity between the image feature of any image and the image feature of the corresponding K1 neighbor image can be obtained, and the K2 image features with the second highest similarity are further selected. The image corresponding to the K2 image features is determined as the K2 neighbor image of any image. Among them, the value of K2 can be set according to requirements.

S23313: Select, from the K2 neighbor images, a second image set whose second similarity with the image feature of any image is greater than a second threshold;

In some possible implementations, after obtaining the K2 neighbor images of each image (the K2 images with the highest second similarity of image features), the second similarity can be directly selected from the K2 neighbor images of each image For images larger than the second threshold, the selected images can form a second image set. The second threshold may be a set value, such as 90%, but it is not a specific limitation of the present disclosure. The image closest to any image can be selected by setting the second threshold.

S23314: Mark all the images and any one of the images in the second image set as the first state, and form a cluster based on each image marked as the first state, the first state is that the images include the same The state of the object.

In some possible implementations, after selecting a second image set whose second similarity between image features is greater than the first threshold from the K2 neighbor images of any image, the selected image may be All the images in the second image set are marked as the first state, and a cluster is formed according to the images in the first state. For example, from the K2 neighbor images of image A, the images with the second similarity greater than the second threshold are selected as images A3 and A4, then A, A3, and A4 can be marked as the first state, from the K2 neighbor images of A3 The image with the second similarity greater than the second threshold is selected as image B2. At this time, A3 and B2 can be marked as the first state, and there is no image with the second similarity greater than the second threshold in the K2 neighbor images of A4. Then mark A4 in the first state. Through the above, A, A3, A4, and B2 can be classified into one cluster. That is, images A, A3, A4, and B2 include the same object.

The clustering results can be easily obtained by the above method. Since the quantized feature reduces the image feature amount, and the K1 neighbor obtained based on the quantized feature further determines the closest K2 neighbor of the image feature, thereby speeding up the clustering speed and further improving the clustering. Class precision. In addition, in the process of performing the calculation of the similarity between quantized features and image features, a distributed parallel operation can also be used to speed up the clustering speed.

In the embodiments of the present disclosure, since the feature amount of the quantized feature is reduced relative to the image feature, the computational cost is reduced, and the parallel processing of multiple arithmetic units can further increase the computational speed.

After at least one cluster of the image is obtained, the images in the same cluster can be considered as a collection of images of the same object (such as a person object), and the first index associated with the images in the cluster can be used to obtain the time when the object appears Information and corresponding location information, based on the time information and location information, the spatiotemporal trajectory information of the object can be further obtained.

As described above, after performing the clustering process, at least one cluster may be obtained, wherein each cluster may include at least one image, and the images in the same cluster may be regarded as including the same object. Among them, after performing the clustering process, the cluster center of each cluster can be further determined. In some possible implementations, the average value of the image features of each image in the cluster may be used as the cluster center. After the class center is obtained, a fourth index can be assigned to the class center to distinguish the clusters corresponding to each class center. That is, each image in the embodiment of the present disclosure includes a third index as an image identifier, a first index as an identifier of the first group of image features, a second index as an identifier of the second group where the quantized feature is located, and as The fourth index of the cluster identification, the above-mentioned index and the corresponding feature, image and other data can be stored in association. In other embodiments, there may be indexes of other feature data, which are not specifically limited in the present disclosure. In addition, the third index of the image, the first index of the first group of image features, the second index of the second group of quantized features, and the fourth index of the cluster are all different, and can be represented by different symbols.

In addition, after multiple clusters are obtained through the embodiments of the present disclosure, the received image can also be clustered to determine the cluster to which the received image belongs, that is, to perform clustering increment processing, where After the received image matches the cluster, the received image can be assigned to the corresponding cluster. If the current cluster does not match the received image, the received image can be used as a cluster alone. Or merge with an existing image data set to perform clustering again.

FIG. 8 shows a flowchart of performing clustering increment processing by an image processing method according to an embodiment of the present disclosure, where the clustering increment processing may include:

S41: Obtain image features of the input image;

In some possible implementation manners, the input image may be an image captured by an image capture device in real time, or may also be an image transmitted through other devices, or an image stored locally. This disclosure does not specifically limit this. After the input image is obtained, the image features of the input image can be obtained. As in the foregoing embodiment, the image features can be obtained through a feature collection algorithm, or through at least one layer of convolution processing of a convolutional neural network. Among them, the image may be a face image, and the corresponding image feature is a face feature.

S42: Perform quantization processing on the image feature of the input image to obtain the quantized feature of the input image;

After the image feature is obtained, quantization processing can be further performed on the image feature to obtain the corresponding quantized feature. Among them, the input images acquired by the embodiments of the present disclosure may be one or more. When performing image feature acquisition and image feature quantization processing, they can all be acquired through distributed parallel execution. The specific parallel execution process is the same as the above-mentioned embodiment. The described process is the same, so it will not be repeated here.

S43: Determine the cluster in which the input image is located based on the quantized feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering process.

After the quantitative feature of the image is obtained, the cluster in which the input image is located can be determined according to the quantitative feature and the cluster center of each cluster. The specific clustering method can also refer to the above process.

Fig. 9 shows a flowchart of step S43 in an image processing method according to an embodiment of the present disclosure. Wherein, the determining the cluster in which the input image is located based on the quantitative feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering processing (S43) may include:

S4301: Acquire a third degree of similarity between the quantitative feature of the input image and the quantitative feature of the cluster center of the at least one cluster obtained by the distributed clustering process;

As mentioned above, the cluster center (the image feature of the cluster center) can be determined according to the average value of the image features of each image in the cluster, and the corresponding quantitative features of the cluster center can also be obtained. The feature performs quantization processing to obtain the quantized feature of the center of the class, or the quantized feature of each image in the cluster can be averaged to obtain the quantized feature of the center of the class.

Further, a third degree of similarity between the input image and the quantized feature of the cluster center of each cluster can be obtained. Similarly, the third degree of similarity may be cosine similarity, but it is not a specific limitation of the present disclosure.

In some possible implementations, multiple class centers can be grouped to obtain multiple class center groups, and the multiple class center groups are allocated to multiple operators, each of which is assigned a different class center group , The third degree of similarity between the class centers in the various center groups and the quantized features of the input image is executed in parallel through multiple arithmetic units, thereby speeding up the processing.

S4302: Determine the K3 class centers with the third highest similarity between the quantized features of the input image, and K3 is an integer greater than or equal to 1;

After the third degree of similarity between the quantized feature of the input image and the quantized feature of the clustered cluster center is obtained, K3 cluster centers with the highest similarity can be obtained. Among them, the number of K3 is less than the number of clusters. The obtained K3 cluster centers can be expressed as K3 clusters that best match the input object.

In some possible implementation manners, the third degree of similarity between the input image and the quantized features of the cluster centers of each cluster can be obtained by means of distributed parallel execution. That is, the centers can be grouped, and the similarity between the quantized features of the corresponding grouped class centers and the quantized features of the input image can be calculated through different arithmetic units, thereby increasing the calculation speed.

S4303: Acquire a fourth degree of similarity between the image features of the input image and the image features of the K3 class centers;

In some possible implementation manners, when the K3 class centers with the fourth highest similarity to the quantized features of the input image are obtained, the difference between the image features of the input image and the corresponding K3 class centers can be further obtained. Similarly, the fourth similarity degree may be a cosine similarity degree, but it is not a specific limitation of the present disclosure.

Similarly, when calculating the fourth similarity between the image features of the input image and the image features of the corresponding K3 class centers, a distributed parallel execution method can also be used. For example, the K3 class centers are divided into multiple groups, and The K3 class centers are assigned to multiple arithmetic units, and the arithmetic unit can perform the fourth similarity between the image features of the assigned class centers and the image features of the input image, thereby speeding up the calculation.

S4304: In the case that the fourth similarity between the image feature of any one of the K3 class centers and the image feature of the input image is the highest and the highest fourth similarity is greater than the third threshold, change The input image is added to the cluster corresponding to the center of any type.

S4305: In the case that there is no class center whose fourth similarity with the image feature of the input feature is greater than the third threshold, perform the calculation based on the quantized feature of the input image and the quantized feature of the image in the image data set. The clustering process is described to obtain at least one new cluster.

In some possible implementations, if the fourth similarity between the image features of the input image and the image features of the K3 class centers has a fourth similarity greater than the third threshold, it can be determined that the input image and the first The cluster matching corresponding to the center of the four clusters with the highest similarity, that is, the object included in the input image and the object corresponding to the fourth cluster with the highest similarity are the same objects. At this time, the input image can be added to the cluster. For example, the cluster identifier can be assigned to the input image for associated storage, so that the cluster to which the input image belongs can be determined.

In some possible implementations, if the fourth similarity between the image features of the input image and the image features of the K3 cluster centers is less than the third threshold, then it can be determined that the input image does not match all the clusters. . At this time, the input image can be used as a separate cluster, or the input image can be fused with the existing image data set to obtain a new image data set, and step S20 can be performed again on the new image data set, that is, for all images The distributed clustering is performed again to obtain at least one new cluster, and the image data can be accurately clustered in this way.

In some possible implementations, if the images included in a cluster change, for example, a new input image is newly added, or the clustering process is re-executed, the cluster center can be re-determined, thereby improving the cluster center Accurately, it is convenient for accurate clustering in the subsequent process.

After the images are clustered, the identity of the object matched by the image in each cluster can also be determined, that is, the identity of the object corresponding to each cluster can be determined based on the identity feature of at least one object in the identity feature library. FIG. 10 shows a flow chart of determining the identity of objects matched by clusters in an image processing method according to an embodiment of the present disclosure, wherein the determination is based on the identity feature of at least one object in the identity feature library with each of the clusters. The object identity corresponding to the class, including:

S51: Obtain quantitative features of known objects in the identity feature library;

In some possible implementations, the identity feature database includes multiple known identities of object information, for example, it may include the face image of the object with known identities and the identity information of the object, and the identity information may include name, age, work, etc. Basic Information.

Correspondingly, the identity feature library can also include the image features and quantified features of each known object. The corresponding image features can be obtained from the face image of each known object, and the quantized features can be obtained by quantizing the image features. .

S52: Determine the fifth similarity between the quantitative feature of the known object and the quantitative feature of the class center of the at least one cluster, and determine K4 with the fifth highest similarity to the quantitative feature of the class center The quantitative characteristics of a known object, K4 is an integer greater than or equal to 1;

In some possible implementation manners, after the quantitative feature of each known object is obtained, the fifth similarity between the quantitative feature of the known object and the obtained quantitative feature of the cluster center can be further obtained. The fifth degree of similarity may be cosine similarity, but it is not a specific limitation of the present disclosure. Further, the quantitative features of the K4 known objects with the fifth highest similarity to the quantitative features of each class center can be determined. That is, the K4 known objects with the fifth highest similarity to the quantitative feature of the class center can be found from the identity database, and the K4 known objects may be the K4 identities with the highest matching pair with the class center.

In other possible implementations, the K4 class centers with the fifth highest similarity to the quantitative feature of each known object can also be obtained. The corresponding correspondences of the K4 class centers are the K4 class centers with the highest matching degree with the identity of the known object.

Similarly, the quantized features of known objects can be grouped, and the fifth similarity between the quantized features of the known objects and the quantized features of the cluster centers obtained by at least one quantizer can be used to improve the processing speed. .

S53: Acquire a sixth degree of similarity between the image feature of the cluster center and the corresponding image features of K4 known objects;

In some possible implementations, after obtaining the K4 known objects corresponding to each class center, the sixth similarity between the image features of each class center and the corresponding K4 known objects can be further determined, The sixth degree of similarity may be cosine similarity, but it is not a specific limitation of the present disclosure.

In some possible implementations, in the case where it is determined that the K4 class centers corresponding to the known object are determined, after the K4 class centers corresponding to the known object are obtained, the image characteristics of the known object can be further determined. The sixth similarity between the image features of the K4 class centers, where the sixth similarity may be a cosine similarity, but it is not a specific limitation of the present disclosure.

S54: Determine if the sixth similarity between the image feature of a known object among the K4 known objects and the image feature of the cluster center is the highest and the sixth similarity is greater than the fourth threshold. The known object with the sixth highest similarity is matched with the cluster corresponding to the cluster center.

S55: In the case where the sixth similarity between the image features of the K4 known objects and the image features of the corresponding cluster centers is less than the fourth threshold, it is determined that there is no cluster matching the known object .

In some possible implementations, if it is determined that there are K4 known objects matching the class center, at this time, if there is at least one of the image features of the known object and the corresponding class center among the image features of the K4 known objects. The sixth similarity between the two is greater than the fourth threshold. At this time, the image feature of the known object with the highest sixth similarity can be determined as the image feature that best matches the cluster center. The identity of the known object is determined as the identity that matches the center of the class, that is, the identity of each image in the cluster corresponding to the center of the class is the identity of the known object with the sixth highest similarity. Or, in the case of determining K4 class centers corresponding to the known object, if the K4 class centers corresponding to the known object have the sixth similarity with the image feature of the known object greater than the fourth threshold The cluster center of the sixth highest similarity can be matched with the known object, that is, the cluster corresponding to the sixth highest similarity center is matched with the identity of the known object, thereby determining the corresponding cluster The identity of the object of the class.

In some possible implementations, in the case where it is determined that K4 known objects matching the class center, at this time, if the sixth similarity between the K4 known objects and the image features of the corresponding class center All are less than the fourth threshold, indicating that there is no identity object that matches the center of the class. Or in the case where it is determined that the K4 class centers that match the known object, if the sixth similarity between the image features of the K4 class centers and the image feature of the known object is less than the fourth threshold, then It indicates that there is no identity matching the known object in the obtained cluster.

In summary, in the embodiments of the present disclosure, corresponding index information can be configured for each image to determine the spatio-temporal data of the object in the image. Based on this configuration, the analysis of the spatio-temporal trajectory of different objects can be realized. After the images in the image data set are clustered, the image set corresponding to each object is obtained (a cluster is equivalent to the image set of an object), and the index information (first index) associated with each image in the cluster is The spatiotemporal trajectory information of the object corresponding to the cluster can be obtained, so that the trajectory analysis of different objects can be realized. At the same time, the embodiments of the present disclosure adopt a distributed clustering method, which can improve clustering efficiency.

Those skilled in the art can understand that in the above methods of the specific implementation, the writing order of the steps does not mean a strict execution order but constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possibility. The inner logic is determined.

FIG. 11 shows a block diagram of an image processing device according to an embodiment of the present disclosure. As shown in FIG. 11, the image processing device includes:

The acquiring module 10 is configured to acquire an image data set, the image data set including a plurality of images and a first index respectively associated with the plurality of images, and the first index is used to determine an object in the image Spatiotemporal data

The clustering module 20 is configured to perform distributed clustering processing on the images in the image data set to obtain at least one cluster;

The determining module 30 is configured to determine the spatiotemporal trajectory information of the object corresponding to the cluster based on the obtained first index associated with the image in the cluster.

In some possible implementation manners, the device further includes an identity determining module, which is configured to determine the identity of the object corresponding to each of the clusters based on the identity feature of at least one object in the identity feature library.

In some embodiments, the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. For specific implementation, refer to the description of the above method embodiments. For brevity, here No longer.

The embodiments of the present disclosure also provide a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above-mentioned method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium.

An embodiment of the present disclosure also provides an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured as the above method.

The embodiment of the present disclosure also proposes a computer program product, the computer program includes computer readable code, and when the computer readable code runs in an electronic device, the processor in the electronic device executes to realize the above An image processing method provided by any embodiment.

The electronic device can be provided as a terminal, server or other form of device.

Fig. 12 shows a block diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and other terminals.

12, the electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814 , And communication component 816.

The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations in the electronic device 800. Examples of these data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk.

The power supply component 806 provides power for various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC). When the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

The sensor component 814 includes one or more sensors for providing the electronic device 800 with various aspects of state evaluation. For example, the sensor component 814 can detect the on/off status of the electronic device 800 and the relative positioning of the components. For example, the component is the display and the keypad of the electronic device 800. The sensor component 814 can also detect the electronic device 800 or the electronic device 800. The position of the component changes, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact. The sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the electronic device 800 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field A programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the foregoing method.

FIG. 13 shows a block diagram of another electronic device according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 13, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application program stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958 . The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the foregoing method.

The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon The protruding structure in the hole card or the groove, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission of electrical signals.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device .

The computer program instructions used to perform the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or in one or more programming languages. Source code or object code written in any combination, the programming language includes object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on a remote computer, or entirely on the remote computer or server carried out. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to access the Internet connection). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer-readable program instructions. The computer-readable program instructions are executed to realize various aspects of the present disclosure.

Herein, various aspects of the present disclosure are described with reference to flowcharts and/or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and/or block diagrams and combinations of blocks in the flowcharts and/or block diagrams can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing device , A device that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make computers, programmable data processing apparatuses, and/or other devices work in a specific manner, so that the computer-readable medium storing instructions includes An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.

It is also possible to load computer-readable program instructions onto a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , So that the instructions executed on the computer, other programmable data processing apparatus, or other equipment realize the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the drawings show the possible implementation architecture, functions, and operations of the system, method, and computer program product according to multiple embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction contains one or more functions for implementing the specified logical function. Executable instructions. In some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions.

Without violating logic, different embodiments of the present disclosure can be combined with each other, and the description of different embodiments is emphasized. For the part of the description, reference may be made to the records of other embodiments.

The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the described embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or improvements to technologies in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

An image processing method, characterized by comprising:

Acquiring an image data set, the image data set including a plurality of images and first indexes respectively associated with the plurality of images, the first indexes being used to determine spatiotemporal data of objects in the images;

Perform distributed clustering processing on the images in the image data set to obtain at least one cluster;

Based on the obtained first index associated with the images in the cluster, the spatiotemporal trajectory information of the object corresponding to the cluster is determined.
The method of claim 1, wherein the method further comprises:

Obtain the image characteristics of the input image;

Performing quantization processing on the image feature of the input image to obtain the quantization feature of the input image;

Determine the cluster where the input image is located based on the quantitative feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering process.
The method according to claim 2, characterized in that, based on the quantized feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering process, determine where the input image is located Clustering, including:

Acquiring a third degree of similarity between the quantitative feature of the input image and the quantitative feature of the cluster center of the at least one cluster obtained by the distributed clustering process;

Determine the K3 class centers with the third highest similarity between the quantized features of the input image, and K3 is an integer greater than or equal to 1;

Acquiring a fourth degree of similarity between the image features of the input image and the image features of the K3 class centers;

In response to the fourth similarity between the image feature of any one of the K3 class centers and the image feature of the input image is the highest and the fourth similarity is greater than the third threshold, the input image is added to The cluster corresponding to any type of center.
The method according to claim 3, wherein the determining the cluster where the input image is located is based on the quantized feature of the input image and the cluster center obtained by the distributed clustering process ,Also includes

In response to the absence of a class center having a fourth similarity greater than a third threshold with the image features of the input image, the distribution is performed based on the quantized features of the input image and the quantized features of the images in the image data set In order to obtain at least one new cluster.
The method according to any one of claims 1 to 4, wherein the first index includes at least one of the following information: the collection time of the image, the collection location, and the image collection used to collect the image The identification of the device and the location where the image capture device is installed.
The method according to any one of claims 1-5, wherein the performing distributed clustering processing on the images in the image data set to obtain at least one cluster comprises:

Acquiring image features of the images in the image data set in a distributed and parallel manner;

Distributed and parallelly perform quantization processing on the image feature to obtain the quantized feature corresponding to the image feature;

The distributed clustering process is executed based on the quantified feature corresponding to the image in the image data set to obtain the at least one cluster.
The method according to claim 6, wherein the distributed and parallel acquisition of the image features of the images in the image data set comprises:

Grouping the multiple images in the image data set to obtain multiple image groups;

The multiple image groups are respectively input to multiple feature extraction models, and the multiple feature extraction models are used to execute feature extraction processing of images in the image group corresponding to the feature extraction models in a distributed and parallel manner to obtain the multiple feature extraction models. The image features of the image, where each feature extraction model inputs different image groups.
The method according to claim 6 or 7, wherein the distributed and parallel quantization process on the image feature to obtain the quantized feature corresponding to the image feature comprises:

Grouping the image features of the multiple images to obtain multiple first groups, where the first group includes the image features of at least one image;

The quantization processing of the image features of the plurality of first groups is executed in a distributed and parallel manner to obtain the quantized feature corresponding to the image feature.
The method according to claim 8, characterized in that, before the quantization processing of the image features of the plurality of first groups is executed in parallel in the distributed manner to obtain the quantized feature corresponding to the image feature, the method further comprises :

Respectively configuring second indexes for the plurality of first groups to obtain a plurality of second indexes;

The distributed and parallel execution of the quantization processing of the image features of the plurality of first groups to obtain the quantized features corresponding to the image features includes:

Allocating the plurality of second indexes to a plurality of quantizers, each of the plurality of quantizers is allocated a different second index;

The multiple quantizers are used to perform quantization processing of the image features in the first group corresponding to the assigned second index respectively in parallel.
The method according to any one of claims 6-9, wherein the quantization processing comprises product quantization coding processing.
The method according to any one of claims 6-10, wherein the distributed clustering process is executed based on the quantized feature corresponding to the image in the image data set to obtain the at least one Clustering, including:

Acquiring the first degree of similarity between the quantized features of any image in the image data set and the quantized features of other images;

Based on the first similarity, determine the K1 neighbor image of any image, and the quantized feature of the K1 neighbor image is the K1 quantized feature with the highest first similarity to the quantized feature of any image, so Said K1 is an integer greater than or equal to 1;

The clustering result of the distributed clustering process is determined by using the any image and the K1 neighbor image of the any image.
The method according to claim 11, wherein the determining the clustering result of the distributed clustering process by using the any image and the K1 neighbor image of the any image comprises:

Selecting, from the K1 neighboring images, a first image set that has a first similarity with a quantized feature of any image greater than a first threshold;

All the images in the first image set and any one of the images are marked as a first state, and a cluster is formed based on each image marked as the first state, and the first state is the images that include the same object status.
The method according to claim 11, wherein the determining the clustering result of the distributed clustering process by using the any image and the K1 neighbor image of the any image comprises:

Acquiring a second degree of similarity between the image feature of the any image and the image feature of the K1 neighbor image of the any image;

Based on the second similarity, determine the K2 neighbor image of any image, and the image feature of the K2 neighbor image is the second highest similarity between the K1 neighbor image and the image feature of any image K2 image features, K2 is an integer greater than or equal to 1 and less than or equal to K1;

Selecting, from the K2 neighboring images, a second image set whose second similarity to the image feature of any image is greater than a second threshold;

All the images in the second image set and any one of the images are marked as the first state, and a cluster is formed based on each image marked as the first state, and the first state is the images that include the same object status.
The method according to any one of claims 11-13, characterized in that before said acquiring the first degree of similarity between the quantized features of any image in the image data set and the quantized features of other images, The method also includes:

Grouping the quantized features of the multiple images in the image data set to obtain multiple second groups, the second groupings including the quantized features of at least one image;

And, the acquiring the first degree of similarity between the quantized features of any image in the image data set and the quantized features of other images includes:

The first degree of similarity between the quantized features of the images in the second group and the quantized features of the remaining images is acquired in a distributed and parallel manner.
The method according to claim 14, characterized in that, before the distributed and parallel acquisition of the first similarity between the quantized features of the images in the second group and the quantized features of the remaining images, the Methods also include:

Configure third indexes for the plurality of second groups respectively to obtain a plurality of third indexes;

In addition, the distributed and parallel acquisition of the first similarity between the quantized features of the images in the second group and the quantized features of the remaining images includes:

Based on the third index, a similarity calculation task corresponding to the third index is established, and the similarity calculation task is to obtain the quantized feature of the target image in the second group corresponding to the third index and the target image The first degree of similarity between the quantized features of all images other than those;

Distributed and parallel execution of the similarity acquisition task corresponding to each third index of the plurality of third indexes.
The method according to any one of claims 1-15, wherein the method further comprises:

Determining the cluster center of the cluster obtained by the distributed clustering process;

A fourth index is configured for the class center, and the fourth index and the corresponding class center are stored in association.
The method according to claim 16, wherein the determining the cluster center obtained by the distributed clustering processing comprises:

Based on the average value of the image features of the images in the at least one cluster, the cluster center is determined.
The method according to any one of claims 1-17, characterized in that, based on the obtained first index associated with the images in the cluster, the spatiotemporal trajectory information of the object corresponding to the cluster is determined ,include:

Determining, based on the first index associated with each image in the cluster, the time information and location information of the object corresponding to the cluster;

The spatiotemporal trajectory information of the object is determined based on the time information and the position information.
The method according to any one of claims 1-18, wherein the method further comprises:

Based on the identity feature of at least one object in the identity feature library, the object identity corresponding to each cluster is determined.
The method according to claim 19, wherein the determining the identity of the object corresponding to each of the clusters based on the identity feature of at least one object in the identity feature library comprises:

Obtaining quantitative features of known objects in the identity feature library;

Determine the fifth degree of similarity between the quantitative feature of the known object and the quantitative feature of the cluster center of the at least one cluster, and determine the K4 has the fifth highest degree of similarity with the quantitative feature of the cluster center Know the quantitative characteristics of the object;

Acquiring the sixth similarity between the image feature of the cluster center and the image features of the corresponding K4 known objects;

In response to the sixth similarity between the image feature of a known object among the K4 known objects and the image feature of the cluster center being the highest and the sixth similarity is greater than a fourth threshold, determining the sixth The known object with the highest similarity is matched with the cluster corresponding to the cluster center.
The method according to claim 20, wherein the determining the identity of the object corresponding to each of the clusters based on the identity feature of at least one object in the identity feature library further comprises:

In response to the sixth similarity between the image features of the K4 known objects and the image features of the corresponding cluster centers are all less than the fourth threshold, it is determined that there is no cluster matching the known object.
An image processing device, characterized by comprising:

An acquisition module for acquiring an image data set, the image data set including a plurality of images and a first index respectively associated with the plurality of images, the first index is used to determine the time and space of the object in the image data;

A clustering module, configured to perform distributed clustering processing on the images in the image data set to obtain at least one cluster;

The determining module is configured to determine the spatiotemporal trajectory information of the object corresponding to the cluster based on the obtained first index associated with the image in the cluster.
The device according to claim 22, wherein the device further comprises an incremental clustering module, which is used to obtain image features of the input image; perform quantization processing on the image features of the input image to obtain the input The quantified feature of the image; based on the quantized feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering process, the cluster where the input image is located is determined.
The device according to claim 23, wherein the incremental clustering module is further configured to obtain the quantized feature of the input image and the cluster center of the at least one cluster obtained by the distributed clustering process. The third degree of similarity between the quantized features of the input image; determine the K3 class centers with the third highest degree of similarity between the quantized features of the input image; obtain the image features of the input image and the K3 class centers The fourth degree of similarity between image features; the fourth degree of similarity between the image feature of any one of the K3 class centers and the image feature of the input image is the highest, and the fourth degree of similarity is greater than the third In the case of a threshold, the input image is added to the cluster corresponding to the center of any type, and K3 is an integer greater than or equal to 1.
The device according to claim 24, wherein the incremental clustering module is further configured to: when there is no cluster center whose fourth similarity is greater than the third threshold between the image features of the input image Next, perform the distributed clustering process based on the quantized features of the input image and the quantized features of the images in the image data set to obtain at least one new cluster.
The device according to any one of claims 22-25, wherein the first index includes at least one of the following information: the collection time of the image, the collection location, and the image collection used to collect the image The identification of the device and the location where the image capture device is installed.
The device according to any one of claims 22-26, wherein the clustering module comprises:

A first distributed processing unit, configured to acquire image features of the images in the image data set in a distributed and parallel manner;

A second distribution processing unit, configured to perform quantization processing on the image features in a distributed and parallel manner to obtain the quantized features corresponding to the image features;

The clustering unit is configured to perform the distributed clustering process based on the quantified feature corresponding to the image in the image data set to obtain the at least one cluster.
The device according to claim 27, wherein the first distribution processing unit is further configured to group a plurality of the images in the image data set to obtain a plurality of image groups;

The multiple image groups are respectively input to multiple feature extraction models, and the multiple feature extraction models are used to execute feature extraction processing of images in the image group corresponding to the feature extraction models in a distributed and parallel manner to obtain the multiple feature extraction models. The image features of the image, where each feature extraction model inputs different image groups.
The device according to claim 27 or 28, wherein the second distribution processing unit is further configured to perform grouping processing on the image features of the multiple images to obtain multiple first groups, and the first group Including image features of at least one image;

The quantization processing of the image features of the plurality of first groups is executed in a distributed and parallel manner to obtain the quantized feature corresponding to the image feature.
The apparatus according to claim 29, wherein the second distribution processing unit is further configured to perform quantization processing of the image features of the plurality of first groups in the distributed parallel to obtain the image feature corresponding Before the quantization feature of, configure second indexes for the plurality of first groups respectively to obtain a plurality of second indexes;

And used to allocate the plurality of second indexes to a plurality of quantizers, each of the plurality of quantizers is allocated a different second index;

The multiple quantizers are used to perform quantization processing of the image features in the first group corresponding to the assigned second index respectively in parallel.
The device according to any one of claims 27-30, wherein the quantization processing comprises product quantization coding processing.
The device according to any one of claims 27-31, wherein the clustering unit is further configured to obtain the first difference between the quantized features of any image in the image data set and the quantized features of other images. Similarity

Based on the first similarity, determine the K1 neighbor image of any image, and the quantized feature of the K1 neighbor image is the K1 quantized feature with the highest first similarity to the quantized feature of any image, so Said K1 is an integer greater than or equal to 1;

The clustering result of the distributed clustering process is determined by using the any image and the K1 neighbor image of the any image.
The device according to claim 32, wherein the clustering unit is further configured to select from the K1 neighboring images that the first similarity with the quantized feature of any image is greater than a first threshold The first collection of images;

All the images in the first image set and any one of the images are marked as a first state, and a cluster is formed based on each image marked as the first state, and the first state is the images that include the same object status.
The device according to claim 32, wherein the clustering unit is further configured to obtain a second degree of similarity between the image feature of the any image and the image feature of the K1 neighbor image of the any image ；

Based on the second similarity, determine the K2 neighbor image of any image, and the image feature of the K2 neighbor image is the second highest similarity between the K1 neighbor image and the image feature of any image K2 image features, K2 is an integer greater than or equal to 1 and less than or equal to K1;

Selecting, from the K2 neighboring images, a second image set whose second similarity to the image feature of any image is greater than a second threshold;

All the images in the second image set and any one of the images are marked as the first state, and a cluster is formed based on each image marked as the first state, and the first state is the images that include the same object status.
The device according to any one of claims 32-34, wherein the clustering unit is further configured to determine between the quantized features of any image in the acquired image data set and the quantized features of other images Before the first degree of similarity, grouping the quantized features of the multiple images in the image data set to obtain multiple second groups, where the second grouping includes the quantized features of at least one image;

And, the acquiring the first degree of similarity between the quantized features of any image in the image data set and the quantized features of other images includes:

The first similarity between the quantized features of the images in the second group and the quantized features of the remaining images is acquired in a distributed and parallel manner.
The device according to claim 35, wherein the clustering unit is further configured to obtain the difference between the quantized features of the images in the second group and the quantized features of the remaining images in the distributed and parallel manner. Before the first degree of similarity, configure third indexes for the plurality of second groups respectively to obtain a plurality of third indexes;

In addition, the distributed and parallel acquisition of the first similarity between the quantized features of the images in the second group and the quantized features of the remaining images includes:

Based on the third index, a similarity calculation task corresponding to the third index is established, and the similarity calculation task is to obtain the quantized feature of the target image in the second group corresponding to the third index and the target image The first degree of similarity between the quantized features of all images other than those;

Distributed and parallel execution of the similarity acquisition task corresponding to each third index of the plurality of third indexes.
The device according to any one of claims 22-36, wherein the class center determination module is configured to determine the class center of the cluster obtained by the distributed clustering process;

A fourth index is configured for the class center, and the fourth index and the corresponding class center are stored in association.
The device according to claim 37, wherein the cluster center determining module is further configured to determine the cluster center based on the average value of the image features of each image in the at least one cluster.
The device according to claim 37, wherein the determining module is further configured to determine the time information and location information of the object corresponding to the cluster based on the first index associated with each image in the cluster;

The spatiotemporal trajectory information of the object is determined based on the time information and the position information.
The device according to any one of claims 22-39, wherein the device further comprises an identity determining module, which is configured to determine the identity of at least one object in the identity feature library to determine the identity of The object identity corresponding to the class.
The device according to claim 40, wherein the identity determination module is further configured to obtain quantitative characteristics of known objects in the identity characteristic library;

Determine the fifth degree of similarity between the quantitative feature of the known object and the quantitative feature of the cluster center of the at least one cluster, and determine the K4 has the fifth highest degree of similarity with the quantitative feature of the cluster center Know the quantitative characteristics of the object;

Acquiring the sixth similarity between the image feature of the cluster center and the image features of the corresponding K4 known objects;

In the case that the sixth similarity between the image feature of a known object among the K4 known objects and the image feature of the cluster center is the highest and the sixth similarity is greater than the fourth threshold, it is determined that the The known object with the sixth highest similarity is matched with the cluster corresponding to the cluster center.
The device according to claim 41, wherein the identity determination module is further configured to determine that the sixth similarity between the image features of the K4 known objects and the image features of the corresponding cluster centers is less than that of the first In the case of four thresholds, it is determined that there is no cluster matching the known object.
An electronic device, characterized in that it comprises:

processor;

A memory for storing processor executable instructions;

Wherein, the processor is configured to call instructions stored in the memory to execute the method according to any one of claims 1-21.
A computer-readable storage medium with computer program instructions stored thereon, wherein the computer program instructions implement the method of any one of claims 1-21 when executed by a processor.
A computer program, characterized in that the computer program includes computer readable code, and when the computer readable code is executed in an electronic device, the processor in the electronic device executes for implementing claims 1-21 The method described in any one of.