CN117112821A

CN117112821A - Image retrieval system and method, storage medium, and electronic device

Info

Publication number: CN117112821A
Application number: CN202310692982.4A
Authority: CN
Inventors: 刘世章; 王全宁; 陈楚怡; 吴润泽; 汪昭辰
Original assignee: Qingdao Chenyuan Technology Information Co ltd
Current assignee: Qingdao Chenyuan Technology Information Co ltd
Priority date: 2023-06-09
Filing date: 2023-06-09
Publication date: 2023-11-24

Abstract

The application discloses an image retrieval system and method, a storage medium and electronic equipment. The system comprises: the device comprises an image retrieval module, a resource image processing module and a data display module; the image retrieval module performs image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set; the resource image processing module establishes an image information network and an image resource library according to the received original image file; and the data display module displays the search result according to the search result set. According to the application, the user can search the similar images in the resource image library at any time through the image information network according to the set images, and view the search result at the client, thereby providing an effective method for image similarity analysis.

Description

Image retrieval system and method, storage medium, and electronic device

Technical Field

The present application relates to the field of video and image processing technologies, and in particular, to an image retrieval system and method, a storage medium, and an electronic device.

Background

With the rapid development of information technology and the internet, images on the network are generated and spread at a remarkable speed, and the mass spread of some images may have a great influence on social public opinion. The massive spread of images such as false news and rumors can negatively affect social public opinion, which is not beneficial to the stability of society; as another example, people comment on the national policy, which can cause concerns of related departments and even influence the adjustment of the national policy in the next step; under the background, how to quickly search out images similar to the image events focused by inspectors from a large number of images, and timely discover and limit the transmission of negative public opinion and clear the negative public opinion, so that the correct guidance of the image public opinion is ensured, which is the important issue of public opinion monitoring.

In the prior art, images existing in a period of time are generally traversed for analysis, and a large number of images with the same content exist in the period of time, so that the number of traversed images is huge, meanwhile, a mode of manually checking the images by a inspector is time-consuming and labor-consuming, all resource images similar to specified images of the images cannot be found out rapidly in time from the massive images, and accordingly, the propagation of negative public opinion guiding images cannot be found out and limited in time and are cleared, and the correct guiding and stability of public opinion are not guaranteed.

Disclosure of Invention

The embodiment of the application provides an image retrieval system and method, a storage medium and electronic equipment, which at least solve the technical problem that all images similar to a designated image cannot be quickly searched from a large number of images in time.

According to one aspect of an embodiment of the present application, an image retrieval system is disclosed, for use in an image information network, the system comprising: the device comprises an image retrieval module, a resource image processing module and a data display module; the image retrieval module is used for performing image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set; the resource image processing module is used for establishing an image information network and an image resource library according to the received original image file; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to the original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and sub-nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the sub-node of each root node and the image corresponding to the root node is less than or equal to the preset threshold, and the target node is the root node or the sub-node; and the data display module is used for displaying the search result according to the search result set.

Optionally, the image retrieval module includes: the method comprises the steps of obtaining a sub-module, a first adding sub-module, a second adding sub-module and a determining sub-module; the acquisition sub-module is used for acquiring a target image to be searched and a target search condition for searching the node; the first adding sub-module is used for traversing the root node meeting the target retrieval condition in the image information network according to the target retrieval condition, and judging whether to add the image corresponding to the root node to the result set according to the target image when traversing to the root node meeting the target retrieval condition; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to an original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and child nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the child nodes of each root node and the images corresponding to the root nodes of the child nodes is smaller than or equal to the preset threshold, and the target node is the root node or the child node; the second adding sub-module is used for traversing all sub-nodes under the root node when the image corresponding to the root node is added to the result set, calculating a first similarity between the target image and the image corresponding to each sub-node, and associating the first similarity to the image corresponding to each sub-node and adding the first similarity to the result set; and the determining submodule is used for determining all results in the result set as a retrieval result set of the target image under the condition that all root nodes meeting the target retrieval conditions in the image information network are traversed.

Optionally, before traversing the root node meeting the target retrieval condition in the image information network according to the target retrieval condition, the method further comprises: preprocessing a target image to obtain a preprocessed image; the preprocessing at least comprises image color space normalization processing; calculating image features of the preprocessed image; the image features of the preprocessed image comprise at least a first feature vector and a first feature matrix.

Optionally, determining whether to add the first image corresponding to the root node to the result set according to the target image includes: acquiring image information of a root node, wherein the image information of the root node at least comprises a second feature vector and a second feature matrix; calculating a modulus of the first feature matrix according to the first feature matrix, and calculating a modulus of the second feature matrix according to the second feature matrix; judging whether the image corresponding to the root node is an alternative image or not according to the absolute value of the difference value of the modulus of the first feature matrix and the modulus of the second feature matrix and the difference rate of the first feature vector and the second feature vector; if the image corresponding to the root node is an alternative image, calculating the characteristic difference rate between the target image and the image according to the first characteristic matrix and the second characteristic matrix; subtracting the first subtracting characteristic difference rate to determine a second similarity rate between the target image and the image corresponding to the root node; and when the second similarity is greater than or equal to a preset similarity threshold, associating the second similarity to the image corresponding to the root node, and adding the second similarity to the result set.

Optionally, establishing an image information network according to the received original image file, including: receiving and preprocessing an original video file in real time to obtain a resource video file; performing video granulation processing on the resource video file, and acquiring a content frame of the resource video file from the granulated video data; the content frame is a frame representing the content of a shot and comprises a first frame, a last frame and N intermediate frames, wherein N is a natural number, the intermediate frames are obtained when the difference rate is larger than a preset threshold value by calculating the difference rate of all sub-frames of a shot except the first frame and the last frame and the previous content frame; calculating characteristic data of each content frame; according to the characteristic data of each content frame, effectively verifying each content frame, and determining the content frames meeting the preset conditions as resource images; taking the resource image as a root node or a child node, and constructing an image information network in an image information space; and storing the original video file, the resource video file, the content frame data and the image information network into an image resource library.

Optionally, establishing an image information network according to the received original image file, including: receiving and preprocessing an original image file in real time to obtain a resource image file; calculating image characteristic data of the resource image file; according to the image characteristic data of the resource image file, effectively verifying the resource image file, and determining the resource image file meeting the preset condition as a resource image; taking the resource image as a root node or a child node, and constructing an image information network in an image information space; and storing the original image file, the resource image data and the image information network into an image resource library.

Optionally, the feature data of each content frame comprises a modulus of the first feature matrix; the image feature data includes a modulus of the second feature matrix; according to the characteristic data of each content frame, effectively verifying each content frame, determining the content frame meeting the preset condition as a resource image, and comprising the following steps: under the condition that the modulus of the first feature matrix is larger than a first threshold value and smaller than a second threshold value, determining that each content frame meets the preset condition, and taking each content frame as a resource image, wherein the first threshold value is smaller than the second threshold value; according to the image characteristic data of the resource image file, effectively verifying the resource image file, determining the resource image file meeting the preset condition as a resource image, including: and under the condition that the modulus of the second feature matrix is larger than a first threshold value and smaller than a second threshold value, determining that the resource image file meets the preset condition, taking the resource image file as a resource image, wherein the first threshold value is smaller than the second threshold value.

Optionally, displaying the search result according to the search result set includes: filtering the search result set of the target image according to the search result filtering condition to obtain a filtering result set; acquiring the image type of each search result in the search result set of the target image; sorting all results in the filtering result set according to the image type of each search result, and returning the sorted results to the client for display; the retrieval results are displayed in different modes according to different image types; when the image type corresponding to the display result is an image, acquiring an image original file corresponding to the display result for display; or when the image type corresponding to the display result is the content frame, acquiring the video original file corresponding to the content frame for display.

According to another aspect of the embodiment of the present application, there is also provided an image retrieval method applied to an image information network, the method including: the image retrieval module performs image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set; the resource image processing module establishes an image information network and an image resource library according to the received original image file; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to the original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and sub-nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the sub-node of each root node and the image corresponding to the root node is less than or equal to the preset threshold, and the target node is the root node or the sub-node; and the data display module displays the search result according to the search result set.

According to still another aspect of the embodiments of the present application, there is also provided an electronic apparatus including a memory in which a computer program is stored, and a processor configured to execute the above-described image retrieval method by the above-described computer program.

According to a further aspect of embodiments of the present application, there is also provided a computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to perform the above-described image retrieval method when run.

In the embodiment of the application, an image retrieval module is used for performing image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set; the resource image processing module is used for establishing an image information network and an image resource library according to the received original image file; and the data display module is used for displaying the search result according to the search result set. According to the application, the user can search the similar images in the resource image library at any time through the image information network according to the target images set by the user, and the search result is checked at the client, so that an effective method is provided for image similarity analysis.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1 is a schematic illustration of an alternative image retrieval system application environment in accordance with an embodiment of the present application;

FIG. 2 is a schematic illustration of an application environment of another alternative image retrieval system in accordance with an embodiment of the present application;

FIG. 3 is a schematic diagram of an alternative image retrieval system according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an image retrieval module according to an embodiment of the present application;

FIG. 5 is a flowchart of an embodiment of an image retrieval module according to the present application;

FIG. 6 is a schematic diagram of content frame selection according to an embodiment of the application;

FIG. 7 is a schematic diagram of a hierarchical extraction process of video to content frames according to an embodiment of the present application;

FIG. 8 is a schematic block diagram of a construction flow of an image information network according to an embodiment of the present application;

FIG. 9 is a schematic block diagram of a flow chart of a query result based on an image information network in accordance with an embodiment of the present application;

FIG. 10 is a diagram showing the results of image retrieval according to an embodiment of the present application;

FIG. 11 is a process schematic block diagram of an image retrieval process provided by the present application;

FIG. 12 is a process schematic block diagram of another image retrieval process provided by the present application;

FIG. 13 is a flow chart of an image retrieval method according to an embodiment of the present application;

fig. 14 is a schematic structural view of an alternative electronic device according to an embodiment of the present application.

Detailed Description

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an aspect of the embodiment of the present application, there is provided an image retrieval method, which may be applied, but not limited to, in an application environment as shown in fig. 1, as an alternative implementation manner. The application environment comprises the following steps: a terminal device 102, a network 104 and a server 106 which interact with a user in a man-machine manner. Human-machine interaction can be performed between the user 108 and the terminal device 102, and an image retrieval application program runs in the terminal device 102. The terminal device 102 includes a man-machine interaction screen 1022, a processor 1024 and a memory 1026. The man-machine interaction screen 1022 is used for displaying a search result set; the processor 1024 is used to generate a target search condition and a set of search results. The memory 1026 is used to store the target search condition and the search result set described above.

The server 106 includes a database 1062 and a processing engine 1064, and the database 1062 stores the target search condition and the search result set. The processing engine 1064 is configured to: acquiring a target image to be searched and a target search condition for searching nodes; traversing the root node meeting the target retrieval conditions in the image information network according to the target retrieval conditions, and judging whether to add the image corresponding to the root node into a result set according to the target image when traversing to the root node meeting the target retrieval conditions; when the image corresponding to the root node is added to the result set, traversing all the sub-nodes under the root node, calculating a first similarity between the target image and the image corresponding to each sub-node, associating the first similarity to each image, and adding the first similarity to the result set; and under the condition that all root nodes meeting the target retrieval conditions in the image information network are traversed, determining all results in the result set as a retrieval result set of the target image.

In one or more embodiments, the above image retrieval method of the present application may be applied to the application environment shown in fig. 2. As shown in fig. 2, a human-machine interaction may be performed between a user 202 and a user device 204. The user device 204 includes a memory 206 and a processor 208. The user equipment 204 in this embodiment may, but is not limited to, refer to performing the operations performed by the terminal equipment 102 to retrieve the retrieval result set similar to the target image.

Optionally, the terminal device 102 and the user device 204 include, but are not limited to, a mobile phone, a tablet computer, a notebook computer, a PC, a vehicle-mounted electronic device, a wearable device, and the like, and the network 104 may include, but is not limited to, a wireless network or a wired network. Wherein the wireless network comprises: WIFI and other networks that enable wireless communications. The wired network may include, but is not limited to: wide area network, metropolitan area network, local area network. The server 106 may include, but is not limited to, any hardware device that may perform calculations. The server may be a single server, a server cluster composed of a plurality of servers, or a cloud server. The above is merely an example, and is not limited in any way in the present embodiment.

In the related art, images existing in a period of time are generally traversed for analysis, a large number of images with the same content exist in the period of time, so that the number of traversed images is huge, meanwhile, a mode of manually checking the images by an inspector is time-consuming and labor-consuming, all resource images similar to a designated image cannot be quickly found out from the massive images in time, and accordingly, the propagation of negative public opinion guiding images cannot be timely found out and limited and eliminated, and the correct guiding and stabilization of public opinion distinguishing are not facilitated.

In order to solve the above technical problem, as an alternative implementation manner, as shown in fig. 3, a schematic structural diagram of an image retrieval system provided by an exemplary embodiment of the present application is shown. The image retrieval system may be implemented as all or part of the terminal by software, hardware, or a combination of both. The system comprises: the device comprises an image retrieval module, a resource image processing module and a data display module; the image retrieval module is used for performing image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set; the resource image processing module is used for establishing an image information network and an image resource library according to the received original image file; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to the original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and sub-nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the sub-node of each root node and the image corresponding to the root node is less than or equal to the preset threshold, and the target node is the root node or the sub-node; and the data display module is used for displaying the search result according to the search result set.

Further, as shown in fig. 4, for example, the image retrieval module includes: the system comprises an acquisition sub-module 10, a first adding sub-module 20, a second adding sub-module 30 and a determining sub-module 40.

An acquisition sub-module 10 for acquiring a target image to be retrieved and a target retrieval condition for retrieving a node;

a first adding sub-module 20, configured to traverse a root node that meets the target search condition in the image information network according to the target search condition, and when traversing to the root node that meets the target search condition, determine whether to add an image corresponding to the root node to the result set according to the target image; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to an original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and child nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the child nodes of each root node and the images corresponding to the root nodes of the child nodes is smaller than or equal to the preset threshold, and the target node is the root node or the child node;

The second adding sub-module 30 is configured to traverse all sub-nodes under the root node when the image corresponding to the root node is added to the result set, calculate a first similarity between the target image and the image corresponding to each sub-node, and associate the first similarity to the image corresponding to each sub-node and add the first similarity to the result set;

a determining sub-module 40, configured to determine all the results in the result set as a search result set of the target image when all the root nodes in the image information network that meet the target search condition have traversed.

It should be noted that, in the image retrieval module of the image retrieval system provided in the above embodiment, only the division of the above functional modules is used for illustration, in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the image retrieval system and the image retrieval method provided in the foregoing embodiments belong to the same concept, which embody detailed implementation procedures in the method embodiments, and are not described herein again.

The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

As shown in fig. 5, the image retrieval module specifically performs the following steps:

s1, acquiring a target image to be searched and a target search condition for searching nodes;

wherein the target image comprises a base image and a content frame; the basic image is any image determined by a user and uploaded to the system, and the image can be stored in a local storage space, can be received in real time on line, can be from a cloud, can be determined according to actual conditions, and is not limited; the content frame is obtained by preprocessing the received original video file; the target search condition is a condition set by a user in the system and used for screening the root node or the child node, and the search condition can be one condition or a combination of a plurality of conditions.

Specifically, the content frame refers to a frame representing the shot content, including a first frame, a last frame, and N intermediate frames, where N is a natural number, and the intermediate frames are obtained when the difference rate is greater than a preset threshold by performing difference rate calculation on all subframes of a shot except for the first frame and the last frame in sequence and the previous content frame.

For example, by analyzing the difference of the content in the shot, a small number of frames can be selected from the continuous frame sequence to represent the content of the shot, and the frames are content frames. The content frames at least comprise the first and last two frames (shot frames) of the shot, so that the number of the shot content frames is more than or equal to 2.

For example, as shown in fig. 6, the first frame is the first content frame, and then the 2 nd and 3 rd frames are calculated. And then calculating the difference rates of the 5 th, 6 th and 4 th frames until the preset threshold is exceeded, and if the difference rates of the 5 th, 6 th and 7 th frames and the first frame are smaller than the preset threshold and the 8 th frame is larger than the preset threshold, the 8 th frame is the third content frame. And by analogy, calculating the content frames in all subframes between all the first frames and all the tail frames. The end frame is selected directly as the last content frame without having to calculate the rate of difference with its previous content frame.

For example, a surveillance video, with few people and few cars during the night, the video frame changes little, and the content frames will be few, for example, only a single number of content frames are extracted within 10 hours. The number of people and vehicles in the daytime is large, the change of people and objects in the video picture is frequent, and the content frames calculated according to the method are much more than those in the evening. Thus, the content frames are guaranteed not to lose all of the content information of the shot video relative to the key frames, as the key frames may lose part of the shot content. Compared with the scheme that each frame of the video is calculated and considered, the selection of the content frames is that only partial video image frames are selected, so that the image calculation amount is greatly reduced on the premise of not losing the content.

For example, as shown in fig. 7, the video content of the target video is composed of a sequence of consecutive frames, and the sequence of consecutive frames can be divided into multiple groups according to the continuity of the video content, and each group of consecutive frame sequence is a shot. By analyzing the difference of the content in the video shots, a small number of frames are selected from the sequence of consecutive frames to represent the content of the shots, i.e. the frames of the content. The content frames at least comprise the first and last two frames (shot frames) of the shot, so that the number of the shot content frames is more than or equal to 2.

In the embodiment of the application, when the received original video file is preprocessed to obtain the content frame, firstly, the uploaded target video file is received, then the target video file is preprocessed to obtain the video, then the video is subjected to shot segmentation to obtain at least one shot, then the shot data is generated according to the at least one shot, and the content frame of each shot is extracted to obtain the content frame of each shot. The lens segmentation may be performed by any one of the prior art, and will not be described herein. Video preprocessing comprises operations such as frame decomposition, normalization and the like, and video attribute information comprises size, duration, resolution and the like.

In one possible implementation manner, when performing image retrieval, a basic image uploaded by a user is firstly acquired to obtain a target image to be retrieved, and then a retrieval condition selected by the user is received to obtain a target retrieval condition for retrieving a node.

In another possible implementation manner, when image retrieval is performed, an original video file is received first, then a frame decomposition and parameter normalization operation of image frames are performed on the original video file to obtain a video, then shot segmentation is performed on the video, a content frame is extracted, finally the content frame is determined to be a target image to be retrieved, and then retrieval conditions selected by a user are received to obtain target retrieval conditions for retrieval nodes. Parameters of the image frames may include, among other things, resolution, aspect ratio, color space, etc.

Further, after the target image is obtained, if the target image is a basic image, preprocessing the target image to obtain a preprocessed image; the preprocessing at least comprises image color space normalization processing, and then image characteristics of the preprocessed image are calculated; the image features of the preprocessed image comprise at least a first feature vector and a first feature matrix. If the target image is a content frame, the content frame characteristic data of the content frame may be directly calculated.

S2, traversing root nodes meeting the target retrieval conditions in the image information network according to the target retrieval conditions, and judging whether to add images corresponding to the root nodes into a result set according to the target images when traversing the root nodes meeting the target retrieval conditions;

The image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to the original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and sub-nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the sub-node of each root node and the image corresponding to the root node is less than or equal to the preset threshold, and the target node is the root node or the sub-node;

it should be noted that, the multi-level tree in the present application may be a 2-level tree structure, each child node at least belongs to 1 root node, and no child node may exist under the root node.

In the embodiment of the application, a user manually sets one or more screening conditions in a displayed node screening condition set, then a terminal traverses root nodes in an image information network to traverse to the root nodes meeting the selected one or more screening conditions, and if the root nodes meeting the selection are present, whether the images corresponding to the root nodes are added to a result set is judged according to a target image.

It should be noted that, in the process of traversing the root node, when the traversed root node does not meet the target screening condition, whether the child node under the root node meets the target screening condition can be judged, if the child node meets the target screening condition, a first similarity between the target image and the image corresponding to the child node is calculated, and the first similarity is associated to each image and added to the result set.

In the embodiment of the application, when judging whether to add a first image corresponding to a root node to a result set according to a target image, firstly acquiring image information of the root node, wherein the image information of the root node at least comprises a second feature vector and a second feature matrix; calculating a modulus of the first feature matrix according to the first feature matrix, and calculating a modulus of the second feature matrix according to the second feature matrix; then, according to the absolute value of the difference value of the modulus of the first feature matrix and the modulus of the second feature matrix, the difference rate of the first feature vector and the second feature vector jointly judges whether the image corresponding to the root node is an alternative image or not; if the image corresponding to the root node is an alternative image, calculating the difference rate between the target image and the image according to the first feature matrix and the second feature matrix; secondly, determining a subtraction difference rate as a second similarity rate between the target image and the image; and finally, when the second similarity is greater than or equal to a preset similarity threshold, the second similarity is related to the image and added to the result set.

In the embodiment of the application, when an image information network is generated, an original video file is received and preprocessed in real time to obtain a target video file; then video granulating is carried out on the target video file, and the content frame of the target video file is obtained from the granulated video data; the content frame is a frame representing the content of a shot and comprises a first frame, a last frame and N intermediate frames, wherein N is a natural number, the intermediate frames are obtained when the difference rate is larger than a preset threshold value by calculating the difference rate of all sub-frames of a shot except the first frame and the last frame and the previous content frame; calculating the characteristic data of each content frame; secondly, according to the characteristic data of each content frame, effectively verifying each content frame, and determining the content frame meeting the preset condition as a target image; then taking the target image as a root node or a child node, and constructing an image information network in an image information space; and finally, storing the original video file, the resource video file, the content frame data and the image information network into an image resource library.

In the embodiment of the application, when an image information network is generated, an original image file is received and preprocessed in real time to obtain a target image file; then calculating image characteristic data of the target image file; secondly, according to the image characteristic data of the target image file, effectively verifying the target image file, and determining the target image file meeting the preset condition as a target image; then taking the target image as a root node or a child node, and constructing an image information network in an image information space; and finally, storing the original image file, the resource image data and the image information network into an image resource library.

Wherein the feature data for each content frame comprises a modulus of the first feature matrix; the image feature data includes a modulus of the second feature matrix.

Specifically, when each content frame is effectively verified according to the feature data of each content frame and the content frame meeting the preset condition is determined to be the target image, under the condition that the modulus of the first feature matrix is larger than a first threshold and smaller than a second threshold, each content frame is determined to meet the preset condition, and each content frame is taken as the target image, wherein the first threshold is smaller than the second threshold.

Specifically, according to the image feature data of the target image file, the target image file is effectively verified, when the target image file meeting the preset condition is determined to be the target image, and when the modulus of the second feature matrix is larger than the first threshold and smaller than the second threshold, the target image file is determined to meet the preset condition, the target image file is taken as the target image, and the first threshold is smaller than the second threshold.

For example, as shown in fig. 8, fig. 8 is a schematic block diagram of a construction flow of an image information network according to the present application, and when constructing the image information network, the construction may be performed based on an image file or a video file.

For a video file, firstly, the video file is acquired, then the video file is preprocessed to obtain video information, the video information can be stored in a video resource library, then the preprocessed video is subjected to video granulation processing, namely, the preprocessed video is subjected to shot segmentation, the content frames are extracted from the segmented shots, the event characteristics of the content frames are calculated, the content frame information can be obtained, and finally, a network is constructed based on the content frame information to obtain an image information network.

For an image file, firstly, the image file is acquired, image preprocessing is carried out, the image preprocessing comprises image data reading and normalization processing of image resolution, shape-amplitude ratio, color space and the like, then, characteristic data of the image is calculated to obtain image information, the characteristic data of the image comprises a characteristic matrix, a mode of the characteristic matrix, a characteristic vector and a mode of the characteristic vector, and finally, the image information can be stored in an image resource library to be stored in an image information network based on the image information.

S3, traversing all the sub-nodes under the root node when the image corresponding to the root node is added to the result set, calculating a first similarity between the target image and the image corresponding to each sub-node, and associating the first similarity to the image corresponding to each sub-node and adding the first similarity to the result set;

In the embodiment of the application, when the image corresponding to the root node is added to the result set, the child node under the root node is in accordance with the target retrieval condition, and therefore, the child node under the root node is in accordance with the target retrieval condition.

Further, if the child node meeting the target retrieval condition does not exist, traversing the next root node.

And S4, when all root nodes meeting the target retrieval conditions in the image information network are traversed, determining all results in the result set as a retrieval result set of the target image.

In the embodiment of the application, when the images corresponding to all the child nodes are added into the result set, continuously acquiring the next root node meeting the target screening condition, and if the next root node meeting the target screening condition is not available, indicating that all the child nodes under the root node are traversed and all the root nodes meeting the target screening condition are traversed and ended, and determining all the results in the result set as a retrieval result set of the target image.

Further, if there is a next root node meeting the target screening condition, whether to add the next root node to the result set is continuously judged according to the target image until all the sub-nodes under the root node are traversed and all the root nodes meeting the root node screening condition are traversed.

For example, as shown in fig. 9, fig. 9 is a schematic block diagram of a flow chart based on an image information network query result, which specifically includes the following steps: step 1: generating image information F according to a target image to be searched, traversing root nodes meeting target searching conditions in an image information network, and setting the traversed root node image information as F _r The method comprises the steps of carrying out a first treatment on the surface of the Step 2: calculating F and F _r Is a similarity ratio Sim of (2); step 3: if meeting the condition that Sim is not less than Sim _min (Sim _min Preset threshold value) then F and F _r Similarly, F is _r Adding to a result set; otherwise, turning to step 8; step 4: traversal F _r Setting the image information of each sub-node as F for all the sub-nodes _c The method comprises the steps of carrying out a first treatment on the surface of the Step 5: calculating F and F _c Similarity Sim of (a); step 6: correlating Sim to F _c Adding the result to a result set; step 7: repeating the steps 4-6 until the sub-node traversal is finished; step 8: repeating the steps 1-7 until the root node traversal is finished; step 9: and returning a result set.

Further, after obtaining a search result set of the target image, filtering the search result set of the target image according to a search result filtering condition to obtain a filtering result set; then obtaining the image type of each search result in the search result set of the target image; secondly, sorting all results in the filtering result set according to the image type of each search result, and returning the sorted results to the client for display; the retrieval results are displayed in different modes according to different image types; finally, when the image type corresponding to the display result is an image, acquiring an image original file corresponding to the display result for display; or when the image type corresponding to the display result is the content frame, acquiring the video original file corresponding to the content frame for display, for example, as shown in fig. 10.

For example, as shown in fig. 11, fig. 11 is a schematic block diagram of a process of an image retrieval process provided in the present application, when a target image to be retrieved is an image file, the image file is first obtained, then the image file is subjected to image preprocessing, image characteristics of the preprocessed image are calculated, image information is obtained, the image information is retrieved in an image information network according to the image information, a retrieval result is obtained, a filtering result is obtained after conditional filtering is performed on the retrieval result, and finally the filtering result is ranked according to an image attribute, so that a ranked result is obtained and fed back to a client.

For example, as shown in fig. 12, fig. 12 is a schematic block diagram of another image retrieval process provided in the present application, when a target image to be retrieved is a video file, video preprocessing is performed on the video file, where the preprocessing includes video de-framing and normalization processing of resolution, aspect ratio, color space, and the like, and then video granulation processing is performed on the preprocessed video, where the video granulation processing includes shot segmentation, content frame extraction, and calculation of content frame feature data. The characteristic data of the content frame comprises a characteristic matrix, a module of the characteristic matrix, a characteristic vector and a module of the characteristic vector; and obtaining content frame information after granulating, searching in an image information network according to the content frame information to obtain a search result, performing conditional filtering on the search result to obtain a filtering result, and finally sorting according to attributes to obtain a final sorting result and feeding back to the client.

The embodiment of the application also has the following beneficial effects:

according to the method and the device, the user can search similar images in the resource image library at any time through the image information network according to the target images set by the user, and search results are searched at the client side, so that an effective method is provided for image similarity analysis.

It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.

The following are examples of the method of the present application.

Referring to fig. 13, fig. 13 is a flowchart of an alternative image retrieval method, which includes the following steps:

s101, a resource image processing module establishes an image information network and an image resource library according to a received original image file; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to the original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and sub-nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the sub-node of each root node and the image corresponding to the root node is less than or equal to the preset threshold, and the target node is the root node or the sub-node;

S102, an image retrieval module performs image retrieval according to an acquired target image to be retrieved to obtain a retrieval result set;

s103, the data display module displays the search result according to the search result set.

In an embodiment of the present application, an image retrieval system is applied to an image information network, the system comprising: the device comprises an image retrieval module, a resource image processing module and a data display module; the image retrieval module is used for performing image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set; the resource image processing module is used for establishing an image information network and an image resource library according to the received original image file; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to the original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by calculating after feature matrixes are extracted from the images under the same coordinate system, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and sub-nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the sub-node of each root node and the image corresponding to the root node is less than or equal to the preset threshold, and the target node is the root node or the sub-node; and the data display module is used for displaying the search result according to the search result set. According to the application, the user can search the similar images in the resource image library at any time through the image information network according to the target images set by the user, and can check the search result at the client, thereby providing an effective method for image analysis.

According to still another aspect of the embodiment of the present application, there is also provided an electronic device for implementing the above-mentioned image retrieval method, which may be a terminal device or a server as shown in fig. 14. The present embodiment is described taking the electronic device as an example. As shown in fig. 14, the electronic device comprises a memory 1802 and a processor 1804, the memory 1802 having stored therein a computer program, the processor 1804 being arranged to perform the steps of any of the method embodiments described above by means of the computer program.

Alternatively, in this embodiment, the electronic device may be located in at least one network device of a plurality of network devices of the computer network.

Alternatively, in the present embodiment, the above processor may be configured to execute the above steps S101 to S103 by a computer program.

Alternatively, it will be understood by those skilled in the art that the structure shown in fig. 14 is only schematic, and the electronic device of the electronic system may also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, a palm computer, and a terminal device such as a mobile internet device (Mobile Internet Devices, MID), a PAD, etc. Fig. 14 is not limited to the structure of the electronic device of the electronic system. For example, the electronic system electronics may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 14, or have a different configuration than shown in FIG. 14.

The memory 1802 may be used for storing software programs and modules, such as program instructions/modules corresponding to the image retrieval system and method in the embodiments of the present application, and the processor 1804 executes the software programs and modules stored in the memory 1802, thereby performing various functional applications and data processing, that is, implementing the image retrieval method described above. The memory 1802 may include high-speed random access memory, but may also include non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1802 may further include memory that is remotely located relative to the processor 1804, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 1802 may be used for storing, but is not limited to, image information, content frames, and the like. As an example, as shown in fig. 14, the memory 1802 may include, but is not limited to, the dividing unit 1702, the acquiring unit 1704, and the first determining unit 1706 in the image retrieval system. In addition, other module units in the image retrieval system may be included, but are not limited to, and are not described in detail in this example.

Optionally, the transmission system 1806 is used to receive or transmit data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission system 1806 includes a network adapter (Network Interface Controller, NIC) that may be connected to other network devices and routers via a network cable to communicate with the internet or a local area network. In one example, the transmission system 1806 is a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.

In addition, the electronic device further includes: a display 1808, configured to display a processing result of the above-mentioned billing subtask; and a connection bus 1810 for connecting the various module components in the electronic device described above.

According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions are read from a computer readable storage medium by a processor of a computer device, which computer instructions are executed by the processor, causing the computer device to perform the above-described image retrieval method, wherein the computer program is arranged to execute the steps of any of the method embodiments described above when run.

Alternatively, in the present embodiment, the above-described computer-readable storage medium may be configured to store a computer program for executing the above steps S101 to S103.

Alternatively, in this embodiment, it will be understood by those skilled in the art that all or part of the steps in the methods of the above embodiments may be performed by a program for instructing a terminal device to execute the steps, where the program may be stored in a computer readable storage medium, and the storage medium may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like.

The integrated units in the above embodiments may be stored in the above-described computer-readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing one or more computer devices (which may be personal computers, servers or network devices, etc.) to perform all or part of the steps of the method of the various embodiments of the present application.

In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In several embodiments provided by the present application, it should be understood that the disclosed client may be implemented in other manners. The system embodiments described above are merely exemplary, such as division of units, merely a logic function division, and other division manners may be implemented in practice, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.

Claims

1. An image retrieval system for use in an image information network, the system comprising:

the device comprises an image retrieval module, a resource image processing module and a data display module; wherein,

the image retrieval module is used for performing image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set;

the resource image processing module is used for establishing an image information network and an image resource library according to the received original image file; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to an original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by extracting feature matrixes from images under the same coordinate system and then calculating, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and child nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the child node of each root node and the image corresponding to the root node of each root node is smaller than or equal to the preset threshold, and the target node is a root node or a child node;

And the data display module is used for displaying the search result according to the search result set.

2. The image retrieval system of claim 1, wherein the image retrieval module comprises:

the method comprises the steps of obtaining a sub-module, a first adding sub-module, a second adding sub-module and a determining sub-module; wherein,

the acquisition sub-module is used for acquiring a target image to be searched and a target search condition for searching the node;

the first adding sub-module is used for traversing the root node meeting the target retrieval condition in the image information network according to the target retrieval condition, and judging whether to add the image corresponding to the root node into a result set according to the target image when traversing to the root node meeting the target retrieval condition;

the second adding sub-module is used for traversing all sub-nodes under the root node when the image corresponding to the root node is added to a result set, calculating a first similarity between the target image and the image corresponding to each sub-node, associating the first similarity to the image corresponding to each sub-node and adding the first similarity to the result set;

and the determining submodule is used for determining all results in the result set as a retrieval result set of the target image under the condition that all root node traversal meeting the target retrieval condition in the image information network is finished.

3. The system of claim 2, wherein the traversing the root node in the image information network that meets the target retrieval condition according to the target retrieval condition further comprises:

preprocessing the target image to obtain a preprocessed image; wherein the preprocessing at least comprises image color space normalization processing;

calculating image features of the preprocessed image; the image features of the preprocessed image comprise at least a first feature vector and a first feature matrix.

4. A system according to claim 3, wherein said determining whether to add the first image corresponding to the root node to the result set based on the target image comprises:

acquiring image information of the root node, wherein the image information of the root node at least comprises a second feature vector and a second feature matrix;

calculating a modulus of the first feature matrix according to the first feature matrix, and calculating a modulus of the second feature matrix according to the second feature matrix;

judging whether the image corresponding to the root node is an alternative image or not according to the absolute value of the difference value of the modulus of the first feature matrix and the modulus of the second feature matrix and the difference rate of the first feature vector and the second feature vector;

If the image corresponding to the root node is an alternative image, calculating a characteristic difference rate between the target image and the image according to the first characteristic matrix and the second characteristic matrix;

subtracting the characteristic difference rate from one to determine a second similarity rate between the target image and the image corresponding to the root node;

and when the second similarity is greater than or equal to a preset similarity threshold, associating the second similarity to the image corresponding to the root node and adding the second similarity to a result set.

5. The system of claim 1, wherein the establishing an image information network and an image repository from the received original image file comprises:

receiving and preprocessing an original video file in real time to obtain a resource video file;

performing video granulation processing on the resource video file, and acquiring a content frame of the resource video file from the granulated video data; the content frame is a frame representing the content of a shot and comprises a first frame, a last frame and N intermediate frames, wherein N is a natural number, the intermediate frames are obtained when the difference rate is larger than a preset threshold value by calculating the difference rate of all sub-frames of a shot except the first frame and the last frame and the previous content frame;

Calculating characteristic data of each content frame;

according to the characteristic data of each content frame, effectively verifying each content frame, and determining the content frames meeting the preset conditions as resource images;

taking the resource image as a root node or a child node, and constructing an image information network in an image information space;

and storing the original video file, the resource video file, the content frame data and the image information network into an image resource library.

6. The system of claim 1, wherein said establishing an image information network from the received original image file comprises:

receiving and preprocessing an original image file in real time to obtain a resource image file;

calculating image characteristic data of the resource image file;

according to the image characteristic data of the resource image file, effectively verifying the resource image file, and determining the resource image file meeting the preset condition as a resource image;

and storing the original image file, the resource image data and the image information network into an image resource library.

7. The system of claim 5 or 6, wherein the feature data for each content frame comprises a modulus of a first feature matrix; the image feature data comprises a modulus of a second feature matrix;

and according to the characteristic data of each content frame, effectively verifying each content frame, and determining the content frame meeting the preset condition as a resource image, wherein the method comprises the following steps:

under the condition that the modulus of the first feature matrix is larger than a first threshold value and smaller than a second threshold value, determining that each content frame meets a preset condition, and taking each content frame as a resource image, wherein the first threshold value is smaller than the second threshold value;

the effective verification of the resource image file according to the image feature data of the resource image file, and determining the resource image file meeting the preset condition as the resource image, includes:

and under the condition that the modulus of the second feature matrix is larger than a first threshold value and smaller than a second threshold value, determining that the resource image file meets a preset condition, and taking the resource image file as a resource image, wherein the first threshold value is smaller than the second threshold value.

8. The system of claim 1, wherein the displaying of the search results according to the set of search results comprises:

Filtering the search result set of the target image according to the search result filtering condition to obtain a filtering result set;

acquiring the image type of each search result in the search result set of the target image;

sorting all results in the filtering result set according to the image type of each search result, and returning the sorted results to the client for display;

the retrieval results are displayed in different modes according to different image types;

when the image type corresponding to the display result is an image, acquiring an image original file corresponding to the display result for display; or,

and when the image type corresponding to the display result is a content frame, acquiring a video original file corresponding to the content frame for display.

9. An image retrieval method applied to an image information network, the method comprising:

the image retrieval module performs image retrieval according to the acquired target image to be retrieved to obtain a retrieval result set;

the resource image processing module establishes an image information network and an image resource library according to the received original image file; the image information network is a forest structure constructed by taking a target image as a target node in an image information space, wherein the target image is determined by obtaining a frame image or an original image file according to an original video file, the image information space is a multidimensional vector space in which image feature vectors are located, the image feature vectors are obtained by extracting feature matrixes from images under the same coordinate system and then calculating, the forest structure is constructed based on the image information space by taking a multi-level tree set as a basis, the multi-level tree comprises root nodes and child nodes, the difference rate between images corresponding to any two root nodes is greater than a preset threshold, the difference rate between the child node of each root node and the image corresponding to the root node of each root node is smaller than or equal to the preset threshold, and the target node is a root node or a child node;

And the data display module displays the search result according to the search result set.

10. A computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the method of claim 9.

11. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method of claim 9.