CN112632300A

CN112632300A - Image retrieval method and device, electronic device and storage medium

Info

Publication number: CN112632300A
Application number: CN202011050459.4A
Authority: CN
Inventors: 郑波; 熊鹏; 李蔚琳; 张贵明; 李嘉宾
Original assignee: Shenzhen Sensetime Technology Co Ltd
Current assignee: Shenzhen Sensetime Technology Co Ltd
Priority date: 2020-09-29
Filing date: 2020-09-29
Publication date: 2021-04-09
Also published as: TW202213123A; WO2022068024A1

Abstract

The application discloses an image retrieval method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining information to be retrieved; using the information to be retrieved to retrieve a database to obtain at least one first image matched with the information to be retrieved; the at least one first image comprises at least one second image; and in the case of receiving a retrieval instruction for the at least one second image, using the at least one second image to retrieve the database, and obtaining at least one image matched with the at least one second image as at least one result image.

Description

Image retrieval method and device, electronic device and storage medium

Technical Field

The application relates to the technical field of security protection, in particular to an image retrieval method and device, electronic equipment and a storage medium.

Background

In order to improve the social security control capability and maintain a good social security environment, monitoring cameras are arranged in more and more places, and when related personnel need to find a target object, the related personnel can use the target information of the target object to retrieve images acquired by the monitoring cameras to obtain the images including the target object. Therefore, how to acquire more information about the target object from the image acquired by the monitoring camera has a very important meaning.

Disclosure of Invention

The application provides an image retrieval method and device, an electronic device and a storage medium.

In a first aspect, an image retrieval method is provided, and the method includes:

acquiring information to be retrieved;

using the information to be retrieved to retrieve a database to obtain at least one first image matched with the information to be retrieved; the at least one first image comprises at least one second image;

and in the case of receiving a retrieval instruction for the at least one second image, using the at least one second image to retrieve the database, and obtaining at least one image matched with the at least one second image as at least one result image.

In combination with any embodiment of the present application, the at least one second image includes a third image; before said retrieving said database using said at least one second image, resulting in at least one image matching said at least one second image, said method further comprises;

performing retrieval object detection processing on the third image to obtain at least one first retrieval object contained in the third image;

the retrieving the database using the at least one second image to obtain at least one image matching the at least one second image comprises:

and using the third image to search the database to obtain at least one fourth image containing the first search object.

With reference to any embodiment of the present application, after the database is retrieved by using the information to be retrieved to obtain at least one first image matching the information to be retrieved, before a retrieval instruction for the at least one second image is received, the method further includes:

displaying the at least one first image.

In combination with any embodiment of the present application, the method further comprises:

displaying a first incidence relation between the information to be retrieved and the third image and a second incidence relation between the third image and the fourth image in a display page; the first association relationship includes: the third image is a child node of the information to be retrieved, and the second association relationship includes: the fourth image is a child node of the third image.

and displaying a third association relation between the fourth image and the first image in the display page, wherein the third association relation comprises that the fourth image is obtained by searching by taking the first search object as a search basis.

under the condition that an instruction of deleting the fourth image is received, deleting the fourth image from the child node of the third image, and carrying out special effect deleting processing on the fourth image to obtain a fifth image;

displaying a fourth association relationship between the third image and the fifth image in the display page; the fourth association relationship includes: the fifth image is a child node of the third image.

and deleting the fifth image from the child node of the third image and displaying the fourth image and the second association relation in the display page if a recovery instruction for the fifth image is received.

With reference to any embodiment of the present application, in a case that the at least one first search object includes a second search object and a third search object, and the search instruction includes a search criterion for the second search object, the searching the database using the third image to obtain at least one fourth image including the first search object includes:

and using the third image retrieval database to obtain a sixth image containing the second retrieval object.

carrying out close object detection processing on the sixth image to obtain a close object of the second retrieval object;

and searching the database by using the sixth image to obtain an image of an intimate object containing the second search object as the result image.

With reference to any one of the embodiments of the present application, the information to be retrieved includes an image to be retrieved.

In a second aspect, there is provided an image retrieval apparatus comprising:

the acquisition unit is used for acquiring information to be retrieved;

the retrieval unit is used for retrieving a database by using the information to be retrieved to obtain at least one first image matched with the information to be retrieved; the at least one first image comprises at least one second image;

the retrieval unit is used for retrieving the database by using the at least one second image under the condition that a retrieval instruction aiming at the at least one second image is received, and obtaining at least one image matched with the at least one second image as at least one result image.

In combination with any embodiment of the present application, the at least one second image includes a third image;

the image retrieval apparatus further includes:

a detection unit, configured to perform search object detection processing on the third image to obtain at least one first search object included in the third image before the database is searched by using the at least one second image to obtain at least one image that matches the at least one second image;

the retrieval unit is configured to:

With reference to any one of the embodiments of the present application, the image retrieval apparatus further includes:

and the display unit is used for displaying at least one first image matched with the information to be retrieved after the database is retrieved by using the information to be retrieved and before the retrieval instruction for the at least one second image is received.

In combination with any embodiment of the present application, the display unit is further configured to:

the processing unit is used for deleting the fourth image from the child node of the third image and carrying out special effect deleting processing on the fourth image to obtain a fifth image under the condition that an instruction for deleting the fourth image is received;

the display unit is further configured to display a fourth association relationship between the third image and the fifth image in the display page; the fourth association relationship includes: the fifth image is a child node of the third image.

With reference to any embodiment of the present application, in a case that the at least one first retrieval object includes a second retrieval object and a third retrieval object, and the retrieval instruction includes that the second retrieval object is used as a retrieval basis, the retrieval unit is further configured to:

With reference to any embodiment of the present application, the retrieval unit is further configured to perform an intimate object detection process on the sixth image to obtain an intimate object of the second retrieval object;

the retrieval unit is further configured to retrieve the database using the sixth image, and obtain an image of an intimate object including the second retrieval object as the result image.

In a third aspect, an electronic device is provided, which includes: a processor and a memory for storing computer program code comprising computer instructions, the electronic device performing the method of the first aspect and any one of its possible implementations as described above, if the processor executes the computer instructions.

In a fourth aspect, another electronic device is provided, including: a processor, transmitting means, input means, output means, and a memory for storing computer program code comprising computer instructions, which, when executed by the processor, cause the electronic device to perform the method of the first aspect and any one of its possible implementations.

In a fifth aspect, there is provided a computer-readable storage medium having stored therein a computer program comprising program instructions which, if executed by a processor, cause the processor to perform the method of the first aspect and any one of its possible implementations.

A sixth aspect provides a computer program product comprising a computer program or instructions which, when run on a computer, causes the computer to perform the method of the first aspect and any of its possible implementations.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments or the background art of the present application, the drawings required to be used in the embodiments or the background art of the present application will be described below.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

Fig. 1 is a schematic flowchart of an image retrieval method according to an embodiment of the present application;

fig. 2 is a schematic diagram of a relationship between information to be retrieved and a first image according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of an image retrieval apparatus according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a hardware structure of an image retrieval apparatus according to an embodiment of the present disclosure.

Detailed Description

In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to improve the social security control capability and maintain a good social security environment, monitoring cameras are arranged in more and more places, and when related personnel need to find a target object, the related personnel can use target information of the target object to retrieve video streams and/or images acquired by the monitoring cameras to obtain the images including the target object.

For example, the target object is a person, and the target information of the target object may be an image containing the person; the target information of the target object may also be a person attribute of the person (e.g., what color of clothing to wear). For another example, the target object is a vehicle, and the target information of the target object may be an image including the vehicle; the target information of the target object may also be vehicle attributes of the vehicle (e.g., what color of vehicle, what make of vehicle, what model of vehicle to wear). For another example, the target object is an event, and the target information of the target object may be the number of people participating in the event (e.g., the event is fighting, and the target information may be the number of people participating in fighting).

Since the target information of the target object has limitations, the number of images containing the target object retrieved using the target information is limited. In the case where the number of images including the target object is limited, the related person may know less information about the target object from the images including the target object.

Based on this, the embodiment of the present application provides an image retrieval method, which can retrieve an image containing a target object by using target information of the target object, and then retrieve a video stream and/or an image collected by a monitoring camera again by using the image containing the target object as a retrieval basis to obtain more information about the target object.

The embodiments of the present application will be described below with reference to the drawings. The execution subject of the embodiment of the application is an image retrieval device. The optional image retrieval means may be one of the following: server, computer.

Referring to fig. 1, fig. 1 is a schematic flow chart of an image retrieval method according to an embodiment of the present disclosure.

101. And acquiring information to be retrieved.

In the embodiment of the application, the information to be retrieved comprises image data and non-image data. The image data includes pictures and video streams. The non-image data includes data other than the image data, the non-image data including at least one of: person attributes, vehicle attributes, object attributes, phone number, certificate number, name, age.

In one implementation of obtaining information to be retrieved, an image retrieval device receives information to be retrieved input by a user through an input component. The above-mentioned input assembly includes: keyboard, mouse, touch screen, touch pad, audio input device, etc.

In another implementation manner of acquiring the information to be retrieved, the image processing apparatus receives the information to be retrieved sent by the first terminal. Optionally, the first terminal may be any one of the following: cell-phone, computer, panel computer, server, wearable equipment.

In another implementation manner of obtaining the information to be retrieved, a communication connection is provided between the image processing apparatus and the monitoring camera, and the image retrieval apparatus can receive at least one image sent by the monitoring camera as the information to be retrieved through the communication connection. Optionally, the surveillance camera is deployed on a road or indoors.

In another implementation manner of obtaining the information to be retrieved, a communication connection is provided between the image processing apparatus and the monitoring camera, and the image retrieval apparatus may receive a video stream sent by the monitoring camera through the communication connection, and use at least one image in the video stream as the information to be retrieved. Optionally, the surveillance camera is deployed on a road or indoors.

In another implementation manner of obtaining the information to be retrieved, the image processing apparatus may obtain at least one image through its own image collecting component, as the information to be retrieved, for example, a camera, and directly collect the information to be retrieved to obtain the image to be processed.

102. And using the information to be retrieved to retrieve the database to obtain at least one first image matched with the information to be retrieved.

In an embodiment of the present application, the database includes image data and non-image data. Wherein the non-image data includes at least one of: person attributes, vehicle attributes, object attributes, phone number, certificate number, name, age.

In the case where the information to be retrieved includes image data (hereinafter, the image data in the information to be retrieved is referred to as an image to be retrieved), the image retrieval apparatus retrieves the database using the image to be retrieved, and obtains at least one image that matches the image to be retrieved as at least one first image.

In one possible implementation, the database further includes feature data of the image data. The image retrieval device obtains the feature data of the image to be retrieved by carrying out feature extraction processing on the image to be retrieved. And comparing the characteristic data of the image to be retrieved with the characteristic data in the database to obtain characteristic data matched with the characteristic data of the image to be retrieved, wherein the characteristic data is used as first intermediate characteristic data. And taking the image corresponding to the first intermediate characteristic data as a first image.

Optionally, there is an index between the image data and the non-image data. In this way, in the case where the information to be retrieved includes non-image data (hereinafter, the non-image data in the information to be retrieved is referred to as data to be retrieved), the image retrieval apparatus uses the retrieval image retrieval database to obtain non-image data matching the data to be retrieved as first intermediate non-image data. The image retrieval device further obtains an image having an index relationship with the first intermediate non-image data as a first image according to the index between the image data and the non-image data.

For example, the database contains image a, which contains person b. At this time, the database also includes the person attribute of person b. The electronic device may establish an index between the person attribute of the person b and the image a in the process of establishing the database. In a case where the image retrieval apparatus retrieves the database using the information to be retrieved to obtain the first intermediate non-image data matching the information to be retrieved, the image retrieval apparatus may determine, as the first image, image data having an index relationship with the first intermediate non-image data from the database according to an index between the image data and the non-image data.

103. And under the condition that a retrieval instruction for at least one second image is received, using the at least one second image to retrieve the database, and obtaining at least one image matched with the at least one second image as at least one result image.

In a case where it is determined that an image including the target object exists in the at least one first image, the user may input a retrieval instruction, which takes the image including the target object as retrieval information, to the image retrieval apparatus, so that the image retrieval apparatus retrieves an image matching the image including the target object from the database to obtain more information related to the target object.

In an embodiment of the application, the at least one second image is at least one image containing the target object in the at least one first image. The image retrieval device uses the at least one second image retrieval database to obtain at least one image matched with the at least one second image as at least one result image when receiving a retrieval instruction for the at least one second image.

For example, the at least one second image displayed by the image retrieval device includes: image a, image b, and image c. The searching party determines at least one second image after studying and judging, wherein the image a comprises a searching object A, and the image B comprises a searching object B. The retrieval party can input a retrieval instruction to the image retrieval device to enable the image retrieval device to retrieve the database by taking the image a as a retrieval basis to obtain an image (namely, an image containing the retrieval object A) matched with the image a as a result image; the retrieval party may also input a retrieval instruction to the image retrieval device to cause the image retrieval device to retrieve the database with the image B as a retrieval basis to obtain an image (i.e., an image including the retrieval object B) matching the image B as a result image; the search direction may also input a search instruction to the image search device, so that the image search device searches the database using the image a and the image B as search bases, and obtains an image matching the image a (i.e., an image including the search object a) and an image matching the image B (i.e., an image including the search object B). An image containing the retrieval object a and an image containing the retrieval object B are taken as result images.

In the embodiment of the application, the image retrieval device firstly uses the information to be retrieved to retrieve the database to obtain at least one first image, and displays the at least one first image so that a user can determine whether the at least one first image contains an image of a target object. In the case where it is determined that an image containing the target object exists in the at least one first image, the user may input to the image retrieval apparatus to further retrieve the database using the image containing the target object as retrieval information to obtain more information about the target object. When receiving a search instruction for an image including a target object, an image search device obtains at least one result image using an image search database including the target object.

As an alternative implementation, the at least one second image includes a third image, and before the image retrieving device performs the step of retrieving the database using the at least one second image to obtain the at least one image matching the at least one second image, the image retrieving device further performs the following steps:

1. and performing search object detection processing on the third image to obtain at least one first search object contained in the third image.

In the embodiment of the present application, the retrieval object (including the first retrieval object described above and a second retrieval object to be mentioned below) includes at least one of: person, vehicle, object, event. For example, the third image includes a person, and the first retrieval object may be a person in the third image; the third image comprises a bus, and the first retrieval object can be the bus in the third image; the third image includes an umbrella, and the first search object may be an umbrella in the third image; the third image includes a fighting event, and the first retrieval object may be the fighting event.

In the embodiment of the present application, by performing search object detection processing on the third image, it is possible to determine whether the third image includes a search object. When the search target is included in the third image, the search target included in the third image is set as the first search target. For example, the image retrieval apparatus determines that the third image contains one person, two vehicles, and a rear-end collision event by performing retrieval object detection processing on the third image. In this case, the image search device may set the person in the third image as the first search target, may set any one vehicle in the third image as the first search target, and may set the rear-end collision event in the third image as the first search target.

In one possible implementation, the search object detection process may be implemented by a convolutional neural network. The convolutional neural network is trained by taking a plurality of images with the labeling information as training data, so that the trained convolutional neural network can finish the detection processing of the retrieval object of the images. The labeling information of the image in the training data is a retrieval object. In the process of training the convolutional neural network by using the training data, the convolutional neural network extracts the characteristic data of the image from the image and determines a retrieval object in the image according to the characteristic data. And monitoring the result obtained by the convolutional neural network in the training process by taking the marking information as the monitoring information, updating the parameters of the convolutional neural network, and finishing the training of the convolutional neural network. In this way, the image search device can process the third image using the trained convolutional neural network to obtain the first search target.

In another possible implementation manner, the search object detection processing may be implemented by a search object detection processing algorithm. Optionally, the search object detection processing may be implemented by one of the following algorithms: only one-eye algorithm (you only look once, YOLO), target detection algorithm (DMP), single-image multi-target detection algorithm (SSD), fast-RCNN algorithm, etc. are required, and the search object detection algorithm for realizing the search object detection processing is not specifically limited in the present application.

After obtaining the first retrieval object, the image retrieval apparatus executes the following steps in the process of retrieving the database by using the at least one second image to obtain at least one image matched with the at least one second image:

2. and searching the database by using the third image to obtain at least one fourth image containing the first search.

For example, the third image includes a person and a vehicle. In the case where the first retrieval object is a person in the third image (hereinafter referred to as a person to be retrieved), the image retrieval apparatus retrieves the database using the third image, and obtains an image containing the person to be retrieved as a fourth image. In the case where the first retrieval object is a vehicle (hereinafter referred to as a vehicle to be retrieved) in the third image, the image retrieval apparatus retrieves an image containing the vehicle to be retrieved as a fourth image using the third image retrieval database.

It should be understood that, in the case where the third image includes at least two retrieval objects, the image retrieval apparatus may use each retrieval object in the third image separately in retrieving data using the third image, resulting in at least one fourth image.

For example, the third image includes a person and a vehicle. In the case where a person in the third image (hereinafter referred to as a person to be retrieved) is taken as a first retrieval object, the image retrieval apparatus retrieves the database using the third image, and obtains an image containing the person to be retrieved as a fourth image. In the case where a vehicle in the third image (hereinafter referred to as a vehicle to be retrieved) is taken as a first retrieval target, the image retrieval apparatus retrieves an image containing a person to be retrieved as a fourth image using the third image retrieval database.

As an alternative embodiment, after executing step 102 and before executing step 103, the image retrieval apparatus further executes the following steps:

3. and displaying the at least one first image.

After obtaining the at least one first image, the image retrieval device displays the at least one first image so that a user can confirm whether the at least one first image contains an image of the target object. For example, the retrieval party in location a inputs an image containing a retrieval object as information to be retrieved to the server, so that the server retrieves a database, wherein the database contains video streams and/or images acquired by the camera in location a. The server searches the database to obtain two first images, namely an image a and an image b. The server displays image a and image b for confirmation by the retrieving party. After the searching party judges the image a and the image b, the searching party determines that the image a contains the searching object, namely the image a contains the target object.

As an alternative implementation manner, the image retrieval apparatus may display the at least one first image as a child node of the information to be retrieved in the process of displaying the at least one first image. For example, as shown in fig. 2, the information to be retrieved is images, and the first image has two pieces. And the two first images are child nodes of the information to be retrieved.

As an alternative implementation, after obtaining the fourth image, the image retrieval apparatus further performs the following steps:

4. and displaying a first incidence relation between the information to be retrieved and the third image and a second incidence relation between the third image and the fourth image in a display page.

In the embodiment of the application, the first association relationship includes that the third image is a child node of the information to be retrieved, and the second association relationship includes that the fourth image is a child node of the third image.

In this step, the image search device uses the searched image as a child node of the search basis. Specifically, the third image is an image obtained by the image retrieval device retrieving the database by using the information to be retrieved, that is, the information to be retrieved is a retrieval basis, and the third image is an image obtained by retrieval. The fourth image is an image obtained by the image retrieval device retrieving the database by using the information to be retrieved, namely the third image is a retrieval basis, and the fourth image is an image obtained by retrieval.

In the embodiment of the application, the image retrieval device can display the retrieval records in a visual form by taking the retrieved images as child nodes of the retrieval basis, so that the retrieval records can be conveniently and rapidly acquired by a user, and when the target object is searched, the searching threads are connected in series in the visual form, so that the target object can be searched by the user.

As an alternative embodiment, the image retrieval apparatus further performs the steps of:

5. and displaying a third association relationship between the fourth image and the first image on the display page.

In this embodiment of the application, the third association relationship includes that the fourth image is obtained by searching based on the first search object.

The image retrieval device may further display a fourth image in the display page retrieved according to the first retrieval object based on the step 4. For example, when the first search target is a person, the image search device may obtain a face similarity set by comparing the fourth image with the images in the database during the process of searching the database using the third image. And taking the image with the similarity exceeding the face similarity threshold in the face similarity set as a fourth image. At this time, the image retrieval device may display in a display page, and the fourth image is obtained by comparing the third image with the images in the database.

For another example, when the first search target is a vehicle, the image search device may obtain the vehicle similarity set by performing a vehicle alignment between the fourth image and the images in the database in the process of searching the database using the third image. And taking the image with the similarity exceeding the vehicle similarity threshold value in the vehicle similarity set as a fourth image. At this time, the image retrieval device may display in a display page, and the fourth image is obtained by comparing the third image with the images in the database.

In the embodiment of the application, the image retrieval device obtains the retrieval basis of the fourth image by displaying in the display page, which is beneficial for a user to study and judge the retrieval record.

As an alternative embodiment, on the premise of executing the foregoing steps, the image retrieval apparatus further executes the following steps:

6. and when receiving an instruction of deleting the fourth image, deleting the fourth image from the child node of the third image, and performing special effect deleting processing on the fourth image to obtain a fifth image.

In the embodiment of the present application, the delete special effect processing includes at least one of: converting the color image into a gray image; adding a dotted line frame around the image to enable the dotted line frame to surround the image; delete information (e.g., crosses, deleted) is added to the image.

When receiving an instruction to delete the fourth image, the image retrieval device performs special effect deletion processing on the fourth image to obtain a fifth image.

7. Displaying a fourth association relationship between the third image and the fifth image on the display page.

In this embodiment of the application, the fourth association relationship includes that the fifth image is a child node of the third image. The image retrieval device, upon receiving an instruction to delete the retrieved image, deletes the fourth image from the child node of the third image, and sets the fifth image as the child node of the third image. In one possible implementation, the image retrieval apparatus replaces the fourth image with the fifth image in the display page.

In the process of viewing the retrieval record and the retrieved image, the user may perform an operation of deleting a certain image by mistake and/or an operation of deleting a certain image which is regarded as unimportant. Thus, when the retrieval records are rearranged and the retrieved images are analyzed, the information may be lost, and the clue for searching the target object may be interrupted.

Therefore, in the embodiment of the present application, when the image retrieval apparatus receives an instruction to delete an image obtained by retrieval, the image retrieval apparatus leaves the image from which the special effect processing has been deleted in the display page. Therefore, when the user rearranges the retrieval records and analyzes the retrieved images in the subsequent process, the retrieved images can still be retrieved by deleting the images after the special effect processing.

As an alternative embodiment, the image retrieval apparatus further performs the following steps on the basis of performing step 7:

8. when a restoration instruction for the fifth image is received, the fifth image is deleted from the child node of the third image, and the fourth image and the second association relation are displayed on the display page.

In the embodiment of the application, the restoring instruction is used for instructing the image retrieval device to restore the image subjected to the special effect deleting processing to the state before the special effect deleting processing. Therefore, the image retrieval apparatus deletes the fifth image from the child node of the third image and newly designates the fourth image as the child node of the third image, i.e., displays the second association in the display page, in the case of receiving the restoration instruction for the fifth image.

In one possible implementation manner, the image retrieval apparatus replaces the fifth image with the fourth image in a case where a restoration instruction for the fifth image is received.

In the embodiment of the present application, when it is determined that an image in a search record is deleted by mistake, the user can restore the image from which the special effect processing has been deleted to the state before the special effect processing has been deleted by inputting a restoration instruction for the image from which the special effect processing has been deleted to the image search device.

As an optional implementation manner, the at least one first retrieval object includes a second retrieval object and a third retrieval object, and the retrieval instruction includes taking the second retrieval object as a retrieval basis. For example, the third image includes two first search objects of zhang san and lie san, and the search party can input an instruction carrying zhang san as a search basis as a search instruction to the image search device when confirming zhang san as a search object. For another example, the third image includes two first retrieval objects, i.e., zhangsan and the vehicle a, and the retrieval party may input an instruction carrying the vehicle a as a retrieval basis to the image retrieval device as a retrieval instruction when confirming that the vehicle a is the troubled vehicle.

In a case where the retrieval instruction includes an object in the second image as the second retrieval object, the image retrieval apparatus performs the following steps in executing step 103:

9. and obtaining a sixth image including the second search target by using the third image search database.

In a possible implementation manner, the image retrieval device performs feature extraction processing on the second image to obtain a feature vector of the second retrieval object. The image retrieval apparatus performs feature extraction processing on all images in the database to obtain a feature vector (hereinafter referred to as a data feature vector) of an object in each image. And comparing the characteristic vector of the second retrieval object with each data characteristic vector to obtain a similarity set. And taking the image corresponding to the similarity exceeding the first threshold in the similarity set as a sixth image.

For example, assume that the second search object is one third in the second image. And the image retrieval device obtains the feature vector of Zhang III by performing feature extraction processing on the second image, wherein the feature vector of Zhang III carries identity information of Zhang III. The image retrieval device performs feature extraction processing on each image in the database to obtain a feature vector of a person in each image. The image retrieval device compares the characteristic vector of Zhang III with the characteristic vector of the person in each image to determine that the similarity between the characteristic vector of the person A and the characteristic vector of Zhang III exceeds a first threshold value. Since the person a belongs to the image b, the image retrieval means determines the image b as the sixth image.

In the embodiment of the application, the user may determine the second retrieval object from the second image, so that the image retrieval device performs targeted retrieval on the second retrieval object to obtain the sixth image as the result image.

10. and performing close object detection processing on the sixth image to obtain a close object of the second search object.

In the embodiment of the present application, the close object refers to an object associated with the search object. For example, in the case where the second search object is a person, the close object may be a fellow person of the second search object; the close object may also be a person in physical contact with the second search object; the close object may also be a person who has eye contact with the second retrieval object; the close object may also be a vehicle in which the second retrieval object rides; the close object may also be an object (e.g., umbrella, backpack, luggage) used by the second search object.

For another example, in the case where the second search object is a vehicle, the close object may be a person riding in the vehicle; the close object may also be a person making physical contact with the vehicle.

For another example, in the case where the second retrieval object is an event, the close object may be a person who participates in the event; the close object may also be a person who views the event.

The image retrieval device performs the close object detection processing using the sixth image, and can specify the close object of the second retrieval object from the sixth image. In one possible implementation, the close object detection processing on the sixth image may be implemented by a convolutional neural network. The image with the labeling information is used as training data to train the convolutional neural network, so that the trained convolutional neural network can complete object detection processing on the image. The labeling information of the image in the training data is a retrieval object and an intimate object of the retrieval object.

For example, the training data includes an image a including a person a, a vehicle B, and a person C. In the labeling information, the retrieval object can be a person A, and the vehicle B is an intimate object of the person A; the retrieval object may be a vehicle B, and the person a is an intimate object of the vehicle B; the search object may be person a, and person C is an intimate object of person a.

11. Searching the database using the sixth image to obtain an image of an intimate object including the search object as the result image.

In this step, the image search device selects an image including the close object of the second search target from the database based on the close object of the second search target in searching the database using the sixth image, and uses the partial image as a result image. In this way, more information related to the second retrieval object, i.e., more information related to the target object, can be obtained by retrieval.

For example, when the second search object is zhang and the close object of zhang is the vehicle a, the image search device may select not only the image including zhang but also the image including the vehicle a from the database. In this way, the user can obtain the information (such as the track) of the page three not only from the image including the page three, but also from the image including the vehicle a.

For example, when the second search object is zhang and the intimacy object of zhang is zhang (the third party is lie), the image search device may select not only the image including zhang but also the image including zhang from the database. Thus, the user can obtain the information (such as the track) of the third page from the image containing the third page, and can also obtain the information (such as the track) of the third page from the image containing the fourth page.

As an optional implementation manner, in the case that the information to be retrieved includes a mobile phone number to be retrieved, the image retrieval device may retrieve the database by using the mobile phone number to be retrieved, so as to obtain an intimate number of the mobile phone number to be retrieved. The image search device searches the database using the intimacy number, and obtains an image matching the intimacy number as a result image.

For example, the image retrieval device retrieves the database by using the mobile phone number to be retrieved, and obtains the mobile phone number with the frequency exceeding the frequency threshold value in contact with the mobile phone number to be retrieved as the close number of the mobile phone number to be retrieved. The image retrieval device uses the intimacy number to retrieve the database, obtains the identity card information bound by the intimacy number, and takes the face image in the identity card information as a result image. The image retrieval device or the database is retrieved by using the close number to obtain the identity card information bound by the close number, and the database is retrieved by taking the face image in the identity card information as a retrieval basis to obtain an image containing the person in the face image as a result image.

As an optional implementation manner, in the case that the information to be retrieved includes a bank account to be retrieved, the image retrieval apparatus may retrieve the database by using the bank account to be retrieved, so as to obtain the intimate number of the bank account to be retrieved. The image search device searches the database using the intimacy number, and obtains an image matching the intimacy number as a result image.

For example, the image retrieval device retrieves the database by using the bank account to be retrieved, and obtains the bank account with the number of times of transfer exceeding a threshold value as the close account of the bank account to be retrieved. The image retrieval device uses the close account number to retrieve the database, obtains the identity card information bound by the close account number, and takes the face image in the identity card information as a result image. The image retrieval device or the close account number is used for retrieving the database to obtain the identity card information bound by the close account number, and the face image in the identity card information is used as a retrieval basis to retrieve the database to obtain the image containing the person in the face image as a result image.

In order to improve the social security control capability and maintain a good social security environment, monitoring cameras are arranged in more and more places, and when related personnel need to find a target person, the track of the target person can be determined from video streams collected by the cameras arranged at different positions according to the human body images, the human face images, the wearing characteristics, the decoration characteristics and the like of the target person. Based on the technical scheme provided by the embodiment of the application, the embodiment of the application also provides several possible application scenarios.

Scene 1: the character attributes of the retrieval object are as follows: a short-haired woman wearing white jacket and black trousers and wearing glasses. And the monitoring camera of the A place is in communication connection with the server of the monitoring center. The server can acquire video streams and/or images from the monitoring camera of the A place through the communication connection, and further can construct a database based on the acquired video streams and/or images. When searching for a search target, the search party in location a may input the person attribute of the search target as information to be searched for to the server, so that the server may select an image related to the search target (e.g., an image including the search target, an image including a pedestrian who is the search target, and an image including a vehicle in which the search target is seated) from the database by searching the database. The server selects an image related to the search target from the database and displays the partial image. The searching party can further know information such as the track of the searched object according to the partial image.

Scene 2: and B, causing a hit-and-run case to occur, and acquiring images of hit-and-run vehicles by monitoring a hit-and-run scene. And the monitoring camera of the B place is in communication connection with the server of the monitoring center. The server can acquire video streams and/or images from the monitoring camera of the B place through the communication connection, and further can construct a database based on the acquired video streams and/or images. When the searching party in the place B searches for the hit vehicle, the image of the hit vehicle may be input to the server as the information to be searched, so that the server retrieves the image related to the hit vehicle (e.g., the image including the hit vehicle, the image including the person riding the hit vehicle) from the database by searching the database. After the server selects the image related to the hit vehicle from the database and displays the partial image. The searching party can obtain the information such as the track of the troublemaking vehicle according to the partial image.

Scene 3: the searcher in place C wants to search for the object to be searched. The mobile phone number used by the search target (hereinafter referred to as a target mobile phone number) can be known from the information provided by the operator. The database in the server of the supervision centre of C contains all telephone numbers provisioned by the operator, as well as data relating to the telephone numbers. In addition, communication connection is arranged between the monitoring camera in the C place and a server of the monitoring center, and the database further comprises video streams and/or images acquired by the monitoring camera in the C place.

When searching for the search object, the searching party in the place C can input the target mobile phone number as the information to be searched to the server, so that the server searches the database to obtain the data related to the target mobile phone number. If the image retrieval device uses the target mobile phone number to retrieve the database, the identity card information bound with the target mobile phone number is obtained, wherein the face image in the identity card information comprises the object to be confirmed. And then, taking the face image in the identity card information as a retrieval basis, retrieving the database to obtain an image containing the object to be confirmed as data related to the target mobile phone number. In another example, the image search device first identifies a cell phone number having a frequency exceeding a frequency threshold value as an intimate number of a fraudulent cell phone number. And using the intimacy number to search the database to obtain the identity card information bound with the intimacy number, wherein the face image in the identity card information contains an intimacy object to be confirmed. And the image retrieval device takes the face image in the identity card information as a retrieval basis, and retrieves the database to obtain an image containing the close object to be confirmed as data related to the target mobile phone number.

The searching party can obtain the related information of the searched object according to the data related to the target mobile phone number displayed by the server, and then find the searched object according to the information.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

The method of the embodiments of the present application is set forth above in detail and the apparatus of the embodiments of the present application is provided below.

Referring to fig. 3, fig. 3 is a schematic structural diagram of an image retrieval apparatus 1 according to an embodiment of the present application, including: an acquisition unit 11, a retrieval unit 12, a detection unit 13, a display unit 14, and a processing unit 15, wherein:

an obtaining unit 11, configured to obtain information to be retrieved;

the retrieval unit 12 is configured to retrieve a database using the information to be retrieved to obtain at least one first image matching the information to be retrieved; the at least one first image comprises at least one second image;

the retrieving unit 12 is configured to, if a retrieval instruction for the at least one second image is received, retrieve the database using the at least one second image, and obtain at least one image matching the at least one second image as at least one result image.

the image retrieval apparatus 1 further includes:

a detecting unit 13, configured to perform search object detection processing on the third image to obtain at least one first search object included in the third image before the database is searched by using the at least one second image to obtain at least one image matched with the at least one second image;

the retrieving unit 12 is configured to:

With reference to any one of the embodiments of the present application, the image search device 1 further includes:

a display unit 14, configured to, after obtaining at least one first image matching the information to be retrieved by using the information to be retrieved to retrieve the database, display the at least one first image before receiving a retrieval instruction for the at least one second image.

In combination with any embodiment of the present application, the display unit 14 is further configured to:

the processing unit 15 is configured to delete the fourth image from a child node of the third image and perform special-effect deletion processing on the fourth image to obtain a fifth image when an instruction to delete the fourth image is received;

the display unit 14 is further configured to display a fourth association relationship between the third image and the fifth image in the display page; the fourth association relationship includes: the fifth image is a child node of the third image.

With reference to any embodiment of the present application, in a case that the at least one first retrieval object includes a second retrieval object and a third retrieval object, and the retrieval instruction includes that the second retrieval object is used as a retrieval basis, the retrieval unit 12 is further configured to:

With reference to any embodiment of the present application, the retrieving unit 12 is further configured to perform an intimate object detection process on the sixth image to obtain an intimate object of the second retrieval object;

the retrieving unit 12 is further configured to retrieve the database using the sixth image, and obtain an image of an intimate object including the second retrieval object as the result image.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present application may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

Fig. 4 is a schematic diagram of a hardware structure of an image retrieval apparatus according to an embodiment of the present disclosure. The image retrieval apparatus 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled by a connector, which includes various interfaces, transmission lines or buses, etc., and the embodiment of the present application is not limited thereto. It should be appreciated that in various embodiments of the present application, coupled refers to being interconnected in a particular manner, including being directly connected or indirectly connected through other devices, such as through various interfaces, transmission lines, buses, and the like.

The processor 21 may be one or more Graphics Processing Units (GPUs), and in the case that the processor 21 is one GPU, the GPU may be a single-core GPU or a multi-core GPU. Alternatively, the processor 21 may be a processor group composed of a plurality of GPUs, and the plurality of processors are coupled to each other through one or more buses. Alternatively, the processor may be other types of processors, and the like, and the embodiments of the present application are not limited.

Memory 22 may be used to store computer program instructions, as well as various types of computer program code for executing the program code of aspects of the present application. Alternatively, the memory includes, but is not limited to, Random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for associated instructions and data.

The input means 23 are for inputting data and/or signals and the output means 24 are for outputting data and/or signals. The input device 23 and the output device 24 may be separate devices or may be an integral device.

It is understood that, in the embodiment of the present application, the memory 22 may be used to store not only the related instructions, but also the related data, for example, the memory 22 may be used to store the information to be retrieved acquired through the input device 23, or the memory 22 may also be used to store at least one result image obtained by searching through the processor 21, and the like, and the embodiment of the present application is not limited to the data specifically stored in the memory.

It will be appreciated that fig. 4 only shows a simplified design of an image retrieval apparatus. In practical applications, the image retrieval devices may further include other necessary components, including but not limited to any number of input/output devices, processors, memories, etc., and all image retrieval devices that can implement the embodiments of the present application are within the scope of the present application.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It is also clear to those skilled in the art that the descriptions of the various embodiments of the present application have different emphasis, and for convenience and brevity of description, the same or similar parts may not be repeated in different embodiments, so that the parts that are not described or not described in detail in a certain embodiment may refer to the descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in or transmitted over a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)), or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., Digital Versatile Disk (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

One of ordinary skill in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by hardware related to instructions of a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the above method embodiments. And the aforementioned storage medium includes: various media that can store program codes, such as a read-only memory (ROM) or a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Claims

1. An image retrieval method, characterized in that the method comprises:

acquiring information to be retrieved;

2. The method of claim 1, wherein the at least one second image comprises a third image;

before said retrieving said database using said at least one second image, resulting in at least one image matching said at least one second image, said method further comprises;

3. The method according to claim 2, wherein after the database is retrieved using the information to be retrieved, at least one first image matching the information to be retrieved is obtained, and before the retrieval instruction for the at least one second image is received, the method further comprises:

displaying the at least one first image.

4. The method of claim 3, further comprising:

5. The method of claim 4, further comprising:

6. The method according to claim 4 or 5, characterized in that the method further comprises:

7. The method of claim 6, further comprising:

8. The method according to any one of claims 2 to 7,

in a case that the at least one first retrieval object includes a second retrieval object and a third retrieval object, and the retrieval instruction includes that the second retrieval object is used as a retrieval basis, the retrieving the database by using the third image to obtain at least one fourth image including the first retrieval object includes:

9. The method of claim 8, further comprising:

10. The method according to any one of claims 1 to 9, wherein the information to be retrieved comprises an image to be retrieved.

11. An image retrieval apparatus characterized by comprising:

the acquisition unit is used for acquiring information to be retrieved;

12. An electronic device, comprising: a processor and a memory for storing computer program code comprising computer instructions which, if executed by the processor, the electronic device performs the method of any of claims 1 to 10.

13. A computer-readable storage medium, in which a computer program is stored, which computer program comprises program instructions which, if executed by a processor, cause the processor to carry out the method of any one of claims 1 to 10.