CN114661936B

CN114661936B - Image retrieval method applied to industrial vision and electronic equipment

Info

Publication number: CN114661936B
Application number: CN202210543251.9A
Authority: CN
Inventors: 周凡; 刘海亮; 郑贵锋; 苏航; 汤武惊; 张怡; 李泽原
Original assignee: Sun Yat Sen University; Shenzhen Research Institute of Sun Yat Sen University
Current assignee: Sun Yat Sen University; Shenzhen Research Institute of Sun Yat Sen University
Priority date: 2022-05-19
Filing date: 2022-05-19
Publication date: 2022-10-14
Anticipated expiration: 2042-05-19
Also published as: CN114661936A

Abstract

The application is applicable to the technical field of data processing, and provides an image retrieval method and electronic equipment applied to industrial vision, wherein the method comprises the following steps: responding to a retrieval instruction initiated by a user, and generating an image retrieval task according to a reference image and self-defined information in the retrieval instruction; generating modal characteristics of a plurality of modalities based on the image retrieval task; the plurality of modalities includes: a profile modality, a structural modality, and a text feature modality; configuring corresponding retrieval subtasks for the modal characteristics of each mode respectively, and sending each retrieval subtask to a distributed system associated with the mode so as to retrieve and process the retrieval subtasks through distributed nodes in the distributed system; and receiving a retrieval list fed back by the logic server, and generating a retrieval result corresponding to the retrieval instruction based on the retrieval list. By adopting the method, the image content can be understood from multiple dimensions, and the accuracy of the retrieval process is improved.

Description

Image retrieval method applied to industrial vision and electronic equipment

Technical Field

The application belongs to the technical field of multimedia, and particularly relates to an image retrieval method and electronic equipment applied to industrial vision.

Background

With the continuous development of information technology, the amount of multimedia data such as electronic documents, images, and videos increases at a geometric rate. Along with the increase of the number of multimedia data, search technologies have appeared for users to quickly find corresponding data, and particularly in industrial production, a large amount of image data such as drawings, design drawings and drafts exist, and industrial vision technologies are required to be used when searching is required. Since people often perform semantic expression by characters, searching by keywords is the mainstream search method in searching. However, when searching for a non-text electronic document, such as an image type, since the expression of image data is often expressed by a combination of an outline and a color, rather than by text, a keyword search cannot accurately search for a desired image.

In the existing image searching technology, corresponding labels are generally added to images, when a user needs to perform image retrieval, a retrieval keyword can be input, and the retrieval keyword is matched with each candidate image added with the label, so that the purpose of image retrieval is achieved. However, as the precision requirement of image retrieval is continuously increased and the accuracy of the label on the image expression is low, when the image is retrieved through the keyword, only coarse-grained search is often performed, so that the difficulty of screening the target image by the user is greatly increased, and the retrieval efficiency is reduced.

Disclosure of Invention

The embodiment of the application provides an image retrieval method, an image retrieval device, electronic equipment and a storage medium, which are applied to industrial vision, and can solve the problems that in the existing image retrieval technology, when images are retrieved through keywords, only coarse-grained search is often performed, so that the difficulty of screening target images by users is greatly increased, and the retrieval efficiency is reduced.

In a first aspect, an embodiment of the present application provides a method for image retrieval applied in industrial vision, including:

responding to a retrieval instruction initiated by a user, and generating an image retrieval task according to a reference image and custom information in the retrieval instruction;

generating modality features of a plurality of modalities based on the image retrieval task; the plurality of modalities includes: a profile modality, a structural modality, and a text feature modality;

configuring corresponding retrieval subtasks for the modal characteristics of each modal respectively, and sending each retrieval subtask to a distributed system associated with the modal so as to retrieve the retrieval subtasks through distributed nodes in the distributed system;

receiving a retrieval list fed back by a logic server, and generating a retrieval result corresponding to the retrieval instruction based on the retrieval list; the retrieval list is obtained by the logic server after receiving the task processing results fed back by each distributed system and performing logic combination processing on all the task processing results; the retrieval result comprises a target image which accords with the retrieval instruction.

In a possible implementation manner of the first aspect, the configuring, for the modal features of each of the modalities, a corresponding retrieval subtask and sending each of the retrieval subtasks to a distributed system associated with the modality, so as to perform retrieval processing on the retrieval subtasks through a distributed node in the distributed system includes:

acquiring a task template associated with the modality, and adding the modality characteristics into the task template to generate the retrieval subtask;

determining a distributed system associated with the mode according to the corresponding relation between the mode and a service system, and sending the retrieval subtask to the distributed system; the distributed system comprises a plurality of map nodes and reduction nodes; the map node is used for acquiring data blocks related to the same candidate image from a distributed system and restoring the candidate image based on all the data blocks; the reduction node is used for receiving the candidate images fed back by the map node, generating candidate features corresponding to the modalities according to the candidate images, calculating the similarity between the candidate features and the modality features, comparing the similarity with a preset similarity threshold value, generating matching results about the candidate images, generating the task processing results from all the matched candidate images, and sending the task processing results to the logic server; wherein the map nodes are multiplexed to the distributed system of each of the modalities.

In a possible implementation manner of the first aspect, when the reduction node calculates the similarity between the candidate feature and the modal feature, the similarity may be expressed as:

wherein SimiarLv is the similarity; SMod (i) is the ith characteristic value in the modal characteristic; BMod (i) is the ith feature value in the candidate feature; and N is the total number of characteristic values contained in the modality.

In a possible implementation manner of the first aspect, the configuring, for each modal feature of each modal, a corresponding retrieval subtask, and sending each retrieval subtask to a distributed system associated with the modal, so as to perform retrieval processing on the retrieval subtask through a distributed node in the distributed system, includes:

sending the retrieval subtask corresponding to the Mth-level modality to the distributed system corresponding to the Mth-level modality based on a retrieval order preset by each modality; the initial value of M is 1;

if a retrieval completion instruction fed back by the distributed system corresponding to the M-level mode is received, judging whether the value of the M is larger than the maximum value of the retrieval sequence; the retrieval completion instruction is generated when the distributed system corresponding to the Mth-level mode obtains the task processing result;

if the value of M is smaller than the maximum value of the retrieval order, sending a first result forwarding instruction to the distributed system corresponding to the Mth-level modality, increasing the value of M, returning to execute the retrieval order preset based on each modality, and sending the retrieval subtask corresponding to the Mth-level modality to the distributed system corresponding to the Mth-level modality; the first result forwarding instruction comprises a communication address of a distributed system corresponding to the M + 1-level mode, so that the distributed system corresponding to the M + 1-level mode sends the task processing result to the distributed system corresponding to the M + 1-level mode;

and if the value of the M is greater than or equal to the maximum value of the retrieval order, sending a second result forwarding instruction to the distributed system corresponding to the Mth-level mode, so that the task processing result of the distributed system corresponding to the Mth-level mode is sent to the logic server.

In a possible implementation manner of the first aspect, the generating, in response to a retrieval instruction initiated by a user, an image retrieval task according to a reference image and custom information in the retrieval instruction includes:

extracting the custom information of the retrieval instruction, and determining the retrieval type specified by the custom information;

if the retrieval type is an appearance patent retrieval type, adjusting a semantic recognition algorithm according to patent keywords;

importing the custom information into the semantic recognition algorithm to generate dimensional characteristic values in a plurality of patent dimensions;

and generating the image retrieval task according to the reference image and the dimension characteristic value.

In one possible implementation manner of the first aspect, the generating modality features of a plurality of modalities based on the image retrieval task includes:

generating a contour division algorithm of the reference image according to the image visual angle corresponding to the reference image and the product type specified by the custom information; the contour dividing algorithm is determined according to product components contained in the product type;

importing the reference image into the contour division algorithm to obtain a plurality of image areas, and respectively determining the contour characteristics of each image area;

generating the modal characteristics related to the contour modal according to the association relationship between each contour characteristic and the product component corresponding to the contour characteristic.

In a possible implementation manner of the first aspect, the receiving a search list fed back by a logical server, and generating a search result corresponding to the search instruction based on the search list includes:

sending a search instruction of a file identifier contained in the retrieval list to the distributed system so as to acquire a plurality of target data blocks corresponding to the file identifier in a plurality of search dimensions through map nodes in the distributed system; the search dimension is determined based on a search type of the search instruction;

receiving each target data block fed back by the distributed system, and generating preview data corresponding to the file identification;

and generating the retrieval result based on the preview data corresponding to all the file identifications.

In a second aspect, an embodiment of the present application provides an apparatus for image retrieval applied in industrial vision, including:

the image retrieval task generating unit is used for responding to a retrieval instruction initiated by a user and generating an image retrieval task according to a reference image and custom information in the retrieval instruction;

a modal feature generation unit configured to generate modal features of a plurality of modalities based on the image retrieval task; the plurality of modalities includes: a profile modality, a structural modality, and a text feature modality;

the retrieval subtask issuing unit is used for configuring corresponding retrieval subtasks for the modal characteristics of each modal respectively and sending each retrieval subtask to a distributed system associated with the modal so as to retrieve and process the retrieval subtasks through distributed nodes in the distributed system;

the retrieval result generating unit is used for receiving a retrieval list fed back by the logic server and generating a retrieval result corresponding to the retrieval instruction based on the retrieval list; the retrieval list is obtained by the logic server after receiving the task processing results fed back by each distributed system and performing logic combination processing on all the task processing results; the retrieval result comprises a target image which accords with the retrieval instruction.

In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the method according to any one of the first aspect is implemented.

In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the method according to any one of the above first aspects.

In a fifth aspect, embodiments of the present application provide a computer program product, which, when run on a server, causes the server to perform the method of any one of the first aspect.

Compared with the prior art, the embodiment of the application has the beneficial effects that: the user can generate a corresponding retrieval instruction from a reference image to be retrieved and self-defined information associated with retrieval, generate a corresponding image retrieval task by processing the reference image and the self-defined information in the retrieval instruction, extract modal characteristics of the image retrieval task, determine the modal characteristics corresponding to the image retrieval task under a plurality of different modalities, and express the image content of the reference image from multiple dimensions so as to improve the accuracy of image retrieval; and then configuring corresponding retrieval subtasks for different modal characteristics, distributing each retrieval subtask to a corresponding distributed system for retrieval processing to obtain a task processing result corresponding to each modal, summarizing all task processing results into a logic server for logic combination, so that a target image matched with the modal characteristics in each modal can be screened out, feeding a retrieval list back to the electronic equipment by the logic server, and generating a corresponding retrieval result by the electronic equipment according to the retrieval list, thereby achieving the purpose of accurately searching the image. Compared with the prior image searching technology, the image searching method has the advantages that image searching is not carried out in a keyword mode, but the related reference images and the custom information can be generated into the searching instruction, and then the modal characteristics of a plurality of different modals, such as the characteristics of three modals including outline, structure and text, can be generated, the image content can be understood from multiple dimensions, and the accuracy of the searching process is improved; and corresponding retrieval subtasks are generated according to different modal characteristics and distributed to different distributed systems for retrieval processing, so that the retrieval speed can be greatly improved, the response efficiency of retrieval instructions is improved, and the waiting time of users is reduced.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

FIG. 1 is a schematic structural diagram of an image retrieval system according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an implementation of a method for image retrieval according to an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating initiation of a retrieve instruction provided by an embodiment of the present application;

fig. 4 is a schematic diagram illustrating an implementation manner of S203 of a method for image retrieval according to an embodiment of the present application;

FIG. 5 is a diagram illustrating the results of a distributed system provided by an embodiment of the present application;

fig. 6 is a flowchart illustrating a specific implementation of the method S203 for image retrieval according to a third embodiment of the present application;

FIG. 7 is a schematic diagram of a retrieval system provided by an embodiment of the present application;

fig. 8 is a flowchart of a specific implementation of the image retrieval methods S201 and S202 according to the fourth embodiment of the present application;

fig. 9 is a flowchart illustrating a detailed implementation of the method S204 for image retrieval according to a fifth embodiment of the present application;

FIG. 10 is a schematic structural diagram of an apparatus for image retrieval according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.

The image retrieval method provided by the embodiment of the application can be applied to electronic equipment which can realize video data processing, such as a smart phone, a server, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a server and the like. The embodiment of the present application does not set any limit to the specific type of the electronic device. In particular, the image retrieval method is applied to an image retrieval system, and the electronic device provided by the embodiment of the application is specifically a retrieval server in the image retrieval system. Fig. 1 schematically shows a structure of an image retrieval system provided in an embodiment of the present application. Referring to fig. 1, the image retrieval system includes a retrieval server 11, a distributed system 12 for processing retrieval subtasks of different modalities, a logic server 13 for logically summarizing task processing results, and a user terminal 14 for initiating a retrieval instruction. Wherein, the user terminal 14 may establish a communication connection with the retrieval server 11 to send a retrieval instruction and receive a retrieval result fed back by the retrieval server 11; the retrieval server 11 establishes communication connection with the distributed system 12, and sends a retrieval subtask; the distributed system 12 establishes a communication connection with the logic server 13 to feed back the corresponding task processing result, and then the logic server 13 may generate a corresponding retrieval list to feed back to the retrieval server 11 to generate a corresponding retrieval result.

Referring to fig. 2, fig. 2 is a schematic diagram illustrating an implementation of a method for image retrieval according to an embodiment of the present application, where the method includes the following steps:

in S201, in response to a retrieval instruction initiated by a user, an image retrieval task is generated according to a reference image and custom information in the retrieval instruction.

In this embodiment, when a user needs to perform a search, the user may send a search instruction to the search server, where the search instruction may carry a reference image to be searched and custom information related to the search. It should be noted that the above-mentioned custom information may be null, that is, when the custom search is not needed, the user may not fill in the content of the custom information. Exemplarily, if the user needs to retrieve other images similar to the reference image, the user only needs to upload the reference image to be retrieved and click a retrieval button, at this time, the user terminal may package the reference image into a retrieval instruction and send the retrieval instruction to the retrieval server, and the corresponding custom information is default to null because the user does not fill in the corresponding content.

Exemplarily, fig. 3 illustrates an initiation schematic diagram of a retrieval instruction provided in an embodiment of the present application. Referring to fig. 3, a search page may be generated in the user terminal, where the search page includes an image selection control 31 for uploading an image and an information filling area 32 for filling in custom information, the information filling area 32 further includes a plurality of information items, which are a search type item 321, a keyword item 322, an outline description item 323, and a structure item 324, respectively, the user may fill in corresponding information in a required information item, and the user terminal may generate the custom information based on the content of each information item in the information filling area. For example, the search page is specifically a patent search page, the search type may specify a patent type to be searched, such as patent search, utility model patent search, or appearance patent search, and the keyword may input a keyword related to the content of the image to be searched, such as a mobile phone, a display screen, and the like, the outline description item may input information related to an outline, such as a cylinder, a cube, and the like, the structure item may input information related to a structure included in the image, such as a base, a circular support, and the like, and of course, the user may also describe the image to be searched through a natural language, that is, the custom information includes a sentence based on natural language description for describing the search image, and the search server may perform semantic understanding on the natural language, so as to determine modal features of different modalities in subsequent S202.

In this embodiment, if the electronic device is a search server, the search server may receive a search instruction sent by each user terminal, in this case, the user terminal may be installed with a corresponding search client, generate a corresponding search interface through the search client, generate a corresponding search instruction in the search interface, and send the search instruction to the search server through the search client; if the electronic device is a terminal operable by a user, such as a computer or a smart phone, the electronic device may locally upload a corresponding search page, and when receiving a search control clicked by the user in the search page, may generate a corresponding search instruction based on information input by the user in the search page.

In this embodiment, the search server may analyze the search command, extract the reference image and the custom information included in the search command, and package the two pieces of information, thereby generating the image search task.

In S202, generating modality features of a plurality of modalities based on the image retrieval task; the plurality of modalities includes: an outline modality, a structural modality, and a text feature modality.

In this embodiment, the search server is configured with feature extraction algorithms of different modalities, and is capable of determining corresponding modality features in different modalities. The above-mentioned multiple different modalities include an outline modality, a structural modality and a text feature modality, and correspondingly, the extracted modality features include: three types of outline features, structural features and text features.

In this embodiment, the data required to be input by the feature extraction algorithms of different modalities may be different or the same, and are specifically configured according to actual situations. For example, when the contour features of the contour modality are output, the corresponding input data may be a reference image, that is, it is not necessary to input custom information to the feature extraction algorithm of the contour modality, and the retrieval server may input the reference image carried in the image retrieval task as the feature extraction algorithm of the contour modality, may extract the contour features of the reference image, and may represent the extracted contour in the form of a vector or a keyword based on the extracted contour, thereby generating the corresponding contour features in the contour modality.

In a possible implementation manner, the retrieval server may determine a modality to be extracted according to a retrieval type corresponding to the retrieval instruction, extract a feature extraction algorithm corresponding to each modality associated with the retrieval type from the database, and output a modality feature corresponding to each modality through each feature extraction algorithm.

In S203, corresponding retrieval subtasks are configured for the modal features of each of the modalities, and each of the retrieval subtasks is sent to the distributed system associated with the modality, so that the retrieval subtasks are retrieved by the distributed nodes in the distributed system.

In this embodiment, in the image retrieval system, in order to improve processing efficiency of different modalities, corresponding distributed systems may be configured for different modalities, a plurality of distributed nodes may be configured in the distributed systems corresponding to different modalities, and different distributed nodes may be configured with a feature extraction algorithm corresponding to the modality, that is, specifically, the feature extraction algorithm is used to process a certain type of modality feature extraction operation, so that time required for configuration of the feature extraction algorithm can be reduced, and thus, retrieval efficiency can be improved.

In a possible implementation manner, a large number of candidate images are stored in the image retrieval system, different candidate images can be divided into a plurality of data blocks to be stored in each distributed node, after the retrieval server sends the corresponding retrieval subtasks to the distributed system, the distributed system can acquire the data blocks stored in each distributed node in the distributed system by the distributed nodes, so that all the data blocks are integrated to restore the candidate images, image features corresponding to the candidate images are generated by a feature extraction algorithm associated with the distributed system, the image features of the candidate images are matched with modal features of a reference image, if the image features are matched with the modal features of the reference image, the candidate images are images associated with the reference image under the modal, and the candidate images are added into the task processing results; on the other hand, if the two images do not match, it indicates that the candidate image is an image not related to the reference image in the modality, and the candidate image is not added to the task processing result. The distributed system may be configured with a corresponding matching threshold, and may compare the matching degree between the modal feature of the reference image and the image feature of the candidate image with the matching threshold, and identify a corresponding matching result. Optionally, the matching threshold may be determined according to the retrieval precision selected by the user, and if the retrieval precision selected by the user is higher, the corresponding matching threshold is higher; otherwise, if the retrieval precision selected by the user is low, the corresponding matching threshold is small.

In this embodiment, the search server may generate corresponding search subtasks according to different modal features, that is, the search subtasks include the modal feature of the reference image, and the modal feature may be a feature value, or may be data representations in various forms such as an array, a matrix, or a vector, and is specifically set according to the actual situation. And the retrieval server sends each retrieval subtask to a distributed system corresponding to the modality, and each distributed system generates a task processing result corresponding to the modality. All candidate images associated with the reference image in the modality are recorded in the task processing result.

In S204, receiving a search list fed back by a logic server, and generating a search result corresponding to the search instruction based on the search list; the retrieval list is obtained by the logic server after receiving the task processing results fed back by each distributed system and performing logic combination processing on all the task processing results; the retrieval result comprises a target image which accords with the retrieval instruction.

In this embodiment, since the distributed system is specifically configured to screen out candidate images associated with the reference image in the modal dimension thereof, for example, for the distributed system of the profile feature modality, the task processing result includes candidate images whose profile features are similar to the reference image; for the distributed system of the structural feature modality, the task processing result comprises a candidate image with the structural feature similar to the reference image. In practice, when a relevant image is detected, a candidate image with similar contour features, text features and structural features to those of a reference image may need to be screened, and at this time, a logic server needs to perform logic integration on corresponding task processing results in each modality, so that each distributed system can send the task processing results corresponding to the modality to the logic server, and the logic server can perform logic operation based on the task processing results fed back by all the distributed systems, so as to obtain a retrieval list. The logical relationship among the modalities can be specifically determined according to the retrieval instruction, and the retrieval server can extract the logical relationship among the modalities in the retrieval instruction and send the logical relationship to the logical server, so that the logical server can generate the retrieval list according to the logical relationship.

In a possible implementation manner, the retrieval instruction defines images to be retrieved, which are matched with features of each modality, and the logic server may perform intersection identification on candidate images included in task processing results of all modalities, identify the candidate images intersected in the task processing results as target images, and generate the retrieval list based on all the target images.

In this embodiment, after receiving the search list fed back by the logic server, the search server may acquire preview images corresponding to target images included in each search list, and generate a search result based on the preview images of all the target images.

As can be seen from the above, in the image retrieval method provided by the embodiment of the application, a user can generate a corresponding retrieval instruction from a reference image to be retrieved and self-defined information associated with retrieval, generate a corresponding image retrieval task by processing the reference image and the self-defined information in the retrieval instruction, and perform modal feature extraction on the image retrieval task, so that modal features corresponding to the image retrieval task in a plurality of different modalities can be determined, and image content of the reference image can be expressed from multiple dimensions, so as to improve accuracy of image retrieval; and then configuring corresponding retrieval subtasks for different modal characteristics, distributing each retrieval subtask to a corresponding distributed system for retrieval processing to obtain a task processing result corresponding to each modal, summarizing all task processing results into a logic server for logic combination, so that a target image matched with the modal characteristics in each modal can be screened out, feeding a retrieval list back to the electronic equipment by the logic server, and generating a corresponding retrieval result by the electronic equipment according to the retrieval list, thereby achieving the purpose of accurately searching the image. Compared with the prior image searching technology, the image searching method has the advantages that image searching is not carried out in a keyword mode, but the related reference images and the custom information can be generated into the searching instruction, and then the modal characteristics of a plurality of different modals, such as the characteristics of three modals including outline, structure and text, can be generated, the image content can be understood from multiple dimensions, and the accuracy of the searching process is improved; and corresponding retrieval subtasks are generated according to different modal characteristics and distributed to different distributed systems for retrieval processing, so that the retrieval speed can be greatly improved, the response efficiency of retrieval instructions is improved, and the waiting time of users is reduced.

Fig. 4 shows a flowchart of a specific implementation of the method S203 for image retrieval according to the second embodiment of the present invention. Referring to fig. 4, with respect to the embodiment described in fig. 1, in the method for retrieving an image provided by this embodiment, S203 includes: S2031-S2032, which is detailed as follows:

further, the configuring, for the modal features of each of the modalities, a corresponding retrieval subtask, and sending each retrieval subtask to a distributed system associated with the modality to perform retrieval processing on the retrieval subtask through a distributed node in the distributed system includes:

in S2031, a task template associated with the modality is obtained, and the modality feature is added to the task template to generate the retrieval subtask.

In this embodiment, the retrieval server may be configured with a corresponding task template, where content to be filled in by the task template is associated with a corresponding modality of the task template, for example, a corresponding modality feature, a retrieval type, a search range, and the like may be imported into each task template, and the retrieval server may be configured according to the modality feature and a retrieval instruction to align the task template to generate a retrieval subtask corresponding to the modality.

In S2032, according to a correspondence between a modality and a service system, determining a distributed system associated with the modality, and sending the retrieval subtask to the distributed system; the distributed system comprises a plurality of map nodes and reduction nodes; the map node is used for acquiring data blocks related to the same candidate image from a distributed system and restoring the candidate image based on all the data blocks; the reduction node is used for receiving the candidate images fed back by the map node, generating candidate features corresponding to the modalities according to the candidate images, calculating the similarity between the candidate features and the modality features, comparing the similarity with a preset similarity threshold value, generating matching results about the candidate images, generating the task processing results from all the matched candidate images, and sending the task processing results to the logic server; wherein the map nodes are multiplexed to the distributed system of each of the modalities.

In this embodiment, the electronic device may determine, according to a preset correspondence between a modality and a service system, a communication address of the distributed system corresponding to the modality, and then send the retrieval subtask to the distributed system according to the communication address, so that the distributed system processes the retrieval subtask. The distributed subtask includes a map node and a reduction node. All candidate images in the distributed system are stored in a plurality of different storage nodes in the distributed system through distributed storage, namely, one candidate image is divided into a plurality of different data blocks. The map node is used for acquiring data blocks of the candidate images from different storage nodes, merging all the data blocks to restore the candidate images, and sending the restored candidate images to the reduction node. After receiving the restored candidate image, the reduction node may extract a candidate feature corresponding to the modality through a feature extraction algorithm corresponding to the modality of the candidate image, perform similarity calculation on the candidate feature and a modality feature corresponding to the reference image in the modality, and compare the calculated similarity with a similarity threshold to obtain a matching result on the candidate image. And adding the image identifications of the candidate images with the matching results being successful into the task processing results, and sending the task processing results to the logic server.

Illustratively, fig. 5 shows a schematic diagram of the result of the distributed system provided by an embodiment of the present application. Referring to fig. 5, the retrieval system includes at least two distributed systems, and the distributed systems include at least two types of nodes, which are map nodes and reduction nodes. The number of map nodes included in one distributed system may be multiple, the number of reduction nodes may be multiple, and the number is specifically set according to actual situations, and is not limited again. Since the whole retrieval system shares the candidate images of the same database, that is, the corresponding image libraries are the same, in order to reduce unnecessary reduction operations of the candidate images, the distributed systems of all the modalities can share the same map node, and the map node can feed back the candidate images obtained by reduction to the reduction nodes corresponding to different modalities, so as to generate the candidate features corresponding to the modalities thereof through a feature extraction algorithm configured in the reduction nodes.

Further, as another embodiment of the present application, when the reduction node calculates the similarity between the candidate feature and the modal feature, the similarity may be expressed as:

In this embodiment, one modal feature may include a plurality of different feature values, and when calculating the similarity between the modal features of the candidate feature and the reference image, the reduction node first calculates a difference value between corresponding feature values in the same dimension, normalizes the difference value by using the softmax function, and finally calculates a corresponding distance mean value, calculates a standard deviation between the modal feature of the reference image and the candidate feature of the candidate image by using the distance mean value, and thereby takes a value of the standard deviation as the similarity between the modal feature of the reference image and the candidate feature of the candidate image. Through normalization processing, the influence of dimensions of different dimensions can be reduced, and therefore the accuracy of similarity calculation can be improved.

In the embodiment of the application, the reusable map nodes and the reduction nodes for logical operation are configured in the distributed system, so that the reduction efficiency of the candidate images can be improved, and the number of the nodes can be reduced, thereby reducing the manufacturing cost of the system and improving the utilization rate of the nodes.

Fig. 6 shows a flowchart of a specific implementation of the method S203 for image retrieval according to the third embodiment of the present invention. Referring to fig. 6, with respect to the embodiment described in fig. 1, the method S203 for image retrieval provided by this embodiment includes: S601-S604, the details are as follows:

in S601, based on a preset retrieval order of each modality, sending the retrieval subtask corresponding to the mth-level modality to the distributed system corresponding to the mth-level modality; the initial value of M is 1.

In this embodiment, the image retrieval system includes distributed systems of different modalities, and the processing order of different distributed systems is determined according to the retrieval order corresponding to the modality. Because a large number of images need to be screened in the retrieval process, the target images which meet the modal characteristics of each modality are selected, and if all the modalities perform matching operation on a large number of candidate images, the execution times of the matching operation may be increased, so that the processing pressure of the distributed nodes is increased. Based on this, the retrieval system can divide different modalities into different retrieval orders according to the operation amount of modality processing, and the number of modalities in the same retrieval order may be one or multiple. For example, the modality corresponding to a certain search instruction includes an issue time modality, an issue object modality, a profile feature modality, and a structure modality, and since the computation amount for performing matching of the time dimension and matching of the issue object is small, and the computation amount for performing feature extraction of the profile feature modality and feature extraction of the structure modality is large, the search order of the time dimension and the issue object dimension can be set to 1, and the search order of the profile feature modality and the structure modality is set to 2, so that a large number of invalid candidate images can be screened by the modality in the previous search order, and for the modality in the large computation amount, the screened candidate images can be processed to reduce unnecessary feature extraction operations, thereby reducing the computation amount in the search process.

In this embodiment, the number of stages in which the distributed systems are located is determined according to the retrieval order corresponding to the modality of the distributed systems, for example, the retrieval order is 1, the corresponding distributed system is located at level 1, the retrieval order is 2, and the corresponding distributed system is located at level 2. Exemplarily, fig. 7 shows a schematic diagram of a retrieval system provided by an embodiment of the present application. The retrieval server may sequentially send the retrieval subtasks to each distributed system according to the retrieval order, for example, to the first distributed system and the second distributed system in the retrieval order of 1, and the first distributed system and the second distributed system may send the corresponding task processing results to other distributed systems in the retrieval order of 2, such as the third distributed system and the fourth distributed system, and perform further screening according to the corresponding task processing results, thereby reducing unnecessary processing. Optionally, the distributed system at the M-th level may also send the task processing result to the logic server, and the logic server performs logic combination according to the task processing results of all the distributed systems at the M-th level, for example, taking an intersection of all the task processing results, and sends a result after the logic combination to the distributed system at the M + 1-th level when a first result forwarding instruction is subsequently received.

In S602, if a retrieval completion instruction fed back by the distributed system corresponding to the mth-level modality is received, determining whether the value of M is greater than the maximum value of the retrieval order; and the retrieval completion instruction is generated when the distributed system corresponding to the M-level mode obtains the task processing result.

In this embodiment, after the distributed system completes the search task of the current level, a search completion instruction may be sent to the search server, and at this time, the search server may determine whether the search processing of all levels has been completed, so that the value of M may be compared with the maximum value of the search order.

In S603, if the value of M is smaller than the maximum value of the retrieval order, sending a first result forwarding instruction to the distributed system corresponding to the mth level modality, increasing the value of M, returning to execute the retrieval order preset based on each modality, and sending the retrieval subtask corresponding to the mth level modality to the distributed system corresponding to the mth level modality; the first result forwarding instruction comprises a communication address of the distributed system corresponding to the M + 1-th level mode, so that the distributed system corresponding to the M + 1-th level mode sends the task processing result to the distributed system corresponding to the M + 1-th level mode.

In this embodiment, if the value of M is smaller than the maximum value of the retrieval order, it indicates that the retrieval of all the hierarchies is not completed, and at this time, a first result forwarding instruction may be sent to the distributed system corresponding to the M-th hierarchy modality, so that the distributed system corresponding to the M-th hierarchy modality may forward the task processing result to the distributed system corresponding to the next hierarchy modality, so as to implement the screening operation on the data, and issue the corresponding retrieval subtask to the distributed system corresponding to the next hierarchy modality.

In S604, if the value of M is greater than or equal to the maximum value of the retrieval order, a second result forwarding instruction is sent to the distributed system corresponding to the mth level modality, so that the task processing result of the distributed system corresponding to the mth level modality is sent to the logic server.

In this embodiment, if the value of M is greater than or equal to the maximum value of the retrieval order, indicating that the retrieval of all the hierarchies has been completed, the corresponding task processing result may be sent to the logical server to generate the retrieval list.

In the embodiment of the application, by carrying out hierarchical division on the modes, unnecessary feature extraction operation can be reduced, so that the retrieval efficiency can be improved, and the calculation amount of a distributed system can be reduced.

Fig. 8 shows a flowchart of a specific implementation of the image retrieval methods S201 and S202 according to the fourth embodiment of the present invention. Referring to fig. 8, with respect to the embodiment described in any one of fig. 1 to 7, in the method for image retrieval provided by this embodiment, S201 includes: S2011-S2014, S202 includes: s2021 to S2023 are described in detail as follows:

further, the generating an image retrieval task according to the reference image and the custom information in the retrieval instruction in response to the retrieval instruction initiated by the user includes:

in S2011, the custom information of the search instruction is extracted, and the search type specified by the custom information is determined.

In S2012, if the search type is an appearance patent search type, the semantic recognition algorithm is adjusted according to the patent keywords.

In S2013, the customization information is imported to the semantic recognition algorithm, and dimension feature values in a plurality of patent dimensions are generated.

In S2014, the image search task is generated based on the reference image and the dimension feature value.

In this embodiment, the user may set a corresponding search type in the custom information, and the search server may analyze the search type field in the custom information to determine whether the search type is an appearance patent search type. If yes, the semantic recognition algorithm can be adjusted through keywords related to the patent, and therefore the accuracy of semantic understanding can be improved. The search server can import the custom information into the adjusted semantic recognition algorithm, so as to extract and obtain a plurality of eigenvalues related to the search dimension of the appearance patent, for example, to extract and obtain an eigenvalue corresponding to the applicant, an eigenvalue corresponding to the invention name, and an eigenvalue corresponding to the classification number, and package the eigenvalues of the plurality of patent dimensions and the reference image required to perform the appearance patent search, so as to obtain the image search task.

In the embodiment of the application, when appearance patent retrieval is required, the semantic extraction algorithm can be adjusted through the corresponding patent keywords, so that the semantic understanding accuracy of the user-defined information can be improved, and the accuracy of subsequent retrieval operation is improved.

Further, the generating modality features of a plurality of modalities based on the image retrieval task includes:

in S2021, a contour partition algorithm of the reference image is generated according to the image view angle corresponding to the reference image and the product type specified by the customization information; the contour dividing algorithm is determined according to product components contained in the product type.

In S2022, the reference image is imported into the contour segmentation algorithm to obtain a plurality of image regions, and contour features of each image region are determined respectively;

in S2023, the modal characteristics relating to the contour modality are generated according to the association relationship between each of the contour characteristics and the product component corresponding to the contour characteristics.

In this embodiment, the structural partitions corresponding to different product types are different, so that during contour recognition, a plurality of different image regions can be obtained by partitioning according to product components, and contour features can be obtained by contour recognition on the different image regions, so that the accuracy of the modal features corresponding to contour modalities can be improved.

Fig. 9 shows a flowchart of a specific implementation of the method S204 for image retrieval according to a fifth embodiment of the present invention. Referring to fig. 9, with respect to the embodiment described in any one of fig. 1 to 7, the method S204 for image retrieval provided by this embodiment includes: s2041 to S2043, the details are as follows:

in S2041, sending a search instruction of a file identifier included in the retrieval list to the distributed system, so as to obtain, through a map node in the distributed system, a plurality of target data blocks corresponding to the file identifier in a plurality of search dimensions; the search dimension is determined based on a search type of the search instruction.

In S2042, receiving each target data block fed back by the distributed system, and generating preview data corresponding to the file identifier;

in S2043, the search result is generated based on the preview data corresponding to all the file identifiers.

In this embodiment, the search list fed back by the logic server includes file identifiers of the target image, and in order to improve the readability of the search result, the search server may send each file identifier to a map node in the distributed system, so as to perform restoration of preview data corresponding to the target image through the map node, since different search types of preview images may have differences, for example, for the search of an appearance patent, the corresponding preview data needs to include information such as an image, a patent name, and an inventor of the appearance patent, the map node in the distributed system may determine a corresponding search dimension according to the search type corresponding to the search instruction, acquire a target data block associated with the search dimension, thereby generate preview data corresponding to the target image, and then generate the search result according to all preview data.

Fig. 10 is a block diagram illustrating an image retrieval apparatus according to an embodiment of the present invention, where the image retrieval apparatus includes units for executing steps implemented by the encryption apparatus in the corresponding embodiment of fig. 2. Please refer to fig. 2 and fig. 2 for the corresponding description of the embodiment. For convenience of explanation, only the portions related to the present embodiment are shown.

Referring to fig. 10, the image retrieval apparatus includes:

the image retrieval task generating unit 101 is configured to generate an image retrieval task according to a reference image and custom information in a retrieval instruction, in response to the retrieval instruction initiated by a user;

a modality feature generation unit 102 configured to generate modality features of a plurality of modalities based on the image retrieval task; the plurality of modalities includes: a profile modality, a structural modality, and a text feature modality;

a retrieval subtask issuing unit 103, configured to configure corresponding retrieval subtasks for the modal features of each modal, and send each retrieval subtask to a distributed system associated with the modal, so as to perform retrieval processing on the retrieval subtask through a distributed node in the distributed system;

a retrieval result generation unit 104, configured to receive a retrieval list fed back by the logic server, and generate a retrieval result corresponding to the retrieval instruction based on the retrieval list; the retrieval list is obtained by the logic server after receiving the task processing results fed back by each distributed system and performing logic combination processing on all the task processing results; the retrieval result comprises a target image which accords with the retrieval instruction.

Optionally, the search subtask issuing unit 103 includes:

the retrieval subtask generating unit is used for acquiring a task template associated with the modality and adding the modality characteristics into the task template to generate the retrieval subtask;

the retrieval subtask sending unit is used for determining a distributed system associated with the mode according to the corresponding relation between the mode and the service system and sending the retrieval subtask to the distributed system; the distributed system comprises a plurality of map nodes and reduction nodes; the map node is used for acquiring data blocks related to the same candidate image from a distributed system and restoring the candidate image based on all the data blocks; the reduction node is used for receiving the candidate images fed back by the map node, generating candidate features corresponding to the modalities according to the candidate images, calculating the similarity between the candidate features and the modality features, comparing the similarity with a preset similarity threshold value, generating matching results about the candidate images, generating the task processing results from all the matched candidate images, and sending the task processing results to the logic server; wherein the map nodes are multiplexed to the distributed system of each of the modalities.

Optionally, when the reduction node calculates the similarity between the candidate feature and the modal feature, the similarity may be expressed as:

Optionally, the search subtask issuing unit 103 includes:

the retrieval order determining unit is used for sending the retrieval subtask corresponding to the M-th level modality to the distributed system corresponding to the M-th level modality based on the retrieval order preset by each modality; the initial value of M is 1;

a maximum value comparison unit, configured to, if a retrieval completion instruction fed back by the distributed system corresponding to the mth-level modality is received, determine whether the value of M is greater than the maximum value of the retrieval order; the retrieval completion instruction is generated when the distributed system corresponding to the Mth-level mode obtains the task processing result;

a cycle unit, configured to send a first result forwarding instruction to the distributed system corresponding to the M-th-level modality if the value of M is smaller than the maximum value of the retrieval order, increase the value of M, return to execute the retrieval order preset based on each modality, and send the retrieval subtask corresponding to the M-th-level modality to the distributed system corresponding to the M-th-level modality; the first result forwarding instruction comprises a communication address of a distributed system corresponding to the M + 1-level mode, so that the distributed system corresponding to the M + 1-level mode sends the task processing result to the distributed system corresponding to the M + 1-level mode;

and a loop ending unit, configured to send a second result forwarding instruction to the distributed system corresponding to the M-th level modality if the value of M is greater than or equal to the maximum value of the retrieval order, so that a task processing result of the distributed system corresponding to the M-th level modality is sent to the logic server.

Optionally, the image retrieval task generating unit 101 includes:

the retrieval type determining unit is used for extracting the custom information of the retrieval instruction and determining the retrieval type specified by the custom information;

the semantic recognition algorithm adjusting unit is used for adjusting a semantic recognition algorithm according to patent keywords if the retrieval type is an appearance patent retrieval type;

the dimension characteristic value determining unit is used for importing the user-defined information into the semantic recognition algorithm to generate dimension characteristic values in multiple patent dimensions;

and the dimension characteristic value packaging unit is used for generating the image retrieval task according to the reference image and the dimension characteristic value.

Optionally, the modal characteristics generating unit 102 includes:

the outline division algorithm determining unit is used for generating an outline division algorithm of the reference image according to the image visual angle corresponding to the reference image and the product type specified by the custom information; the contour dividing algorithm is determined according to product components contained in the product type;

the contour feature determination unit is used for importing the reference image into the contour division algorithm to obtain a plurality of image areas and respectively determining the contour feature of each image area;

and the modal characteristic determining unit is used for generating the modal characteristics related to the profile modal according to the profile characteristic and the association relationship between the product components corresponding to the profile characteristic.

Optionally, the search result generating unit 104 includes:

a search instruction sending unit, configured to send a search instruction of a file identifier included in the retrieval list to the distributed system, so as to obtain, through a map node in the distributed system, a plurality of target data blocks corresponding to the file identifier in a plurality of search dimensions; the search dimension is determined based on a search type of the search instruction;

a target data block receiving unit, configured to receive each target data block fed back by the distributed system, and generate preview data corresponding to the file identifier;

and the preview data generating unit is used for generating the retrieval result based on the preview data corresponding to all the file identifications.

Therefore, the image retrieval device provided by the embodiment of the invention can also generate a corresponding retrieval instruction by using the reference image to be retrieved and the customized information associated with the retrieval, generate a corresponding image retrieval task by processing the reference image and the customized information in the retrieval instruction, and extract the modal characteristics of the image retrieval task, so that the modal characteristics corresponding to the image retrieval task in a plurality of different modalities can be determined, and the image content of the reference image can be expressed from multiple dimensions, thereby improving the accuracy of the image retrieval; and then configuring corresponding retrieval subtasks for different modal characteristics, distributing each retrieval subtask to a corresponding distributed system for retrieval processing to obtain a task processing result corresponding to each modal, summarizing all task processing results into a logic server for logic combination, so that a target image matched with the modal characteristics in each modal can be screened out, feeding a retrieval list back to the electronic equipment by the logic server, and generating a corresponding retrieval result by the electronic equipment according to the retrieval list, thereby achieving the purpose of accurately searching the image. Compared with the existing image searching technology, the image searching is not carried out in a keyword mode, but the associated reference image and the custom information can be generated into a searching instruction, and then a plurality of modal characteristics of different modalities, such as characteristics of three modalities of an outline, a structure and a text, are generated, so that the image content can be understood from multiple dimensions, and the accuracy of the searching process is improved; and corresponding retrieval subtasks are generated according to different modal characteristics and distributed to different distributed systems for retrieval processing, so that the retrieval speed can be greatly improved, the response efficiency of retrieval instructions is improved, and the waiting time of users is reduced.

It should be understood that, in the structural block diagram of the image retrieval apparatus shown in fig. 10, each module is used to execute each step in the embodiment corresponding to fig. 1 to 9, and each step in the embodiment corresponding to fig. 1 to 9 has been explained in detail in the above embodiment, and specific reference is made to the relevant description in the embodiment corresponding to fig. 1 to 9 and fig. 1 to 9, which is not repeated herein.

Fig. 11 is a block diagram of an electronic device according to another embodiment of the present application. As shown in fig. 11, the electronic apparatus 1100 of this embodiment includes: a processor 1110, a memory 1120, and a computer program 1130, such as a program of a method of image retrieval, stored in the memory 1120 and executable on the processor 1110. Processor 1110, when executing computer program 1130, performs the steps of the various embodiments of the method for image retrieval described above, such as S201 through S204 shown in fig. 2. Alternatively, when the processor 1110 executes the computer program 1130, the functions of the modules in the embodiment corresponding to fig. 10, for example, the functions of the units 101 to 104 shown in fig. 10, are implemented, and refer to the related description in the embodiment corresponding to fig. 10 specifically.

Illustratively, the computer program 1130 may be divided into one or more modules, which are stored in the memory 1120 and executed by the processor 1110 to accomplish the present application. One or more of the modules may be a series of computer program instruction segments that can perform particular functions, and that describe the execution of the computer program 1130 in the electronic device 1100. For example, the computer program 1130 may be divided into unit modules, and the specific functions of the modules are as described above.

The electronic device 1100 may include, but is not limited to, a processor 1110, a memory 1120. Those skilled in the art will appreciate that fig. 11 is merely an example of the electronic device 1100 and does not constitute a limitation of the electronic device 1100 and may include more or fewer components than illustrated, or combine certain components, or different components, e.g., the electronic device may also include input-output devices, network access devices, buses, etc.

The processor 1110 may be a central processing unit, but may also be other general purpose processors, digital signal processors, application specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete hardware components, and the like. The general purpose processor may be a microprocessor or any conventional processor or the like.

The memory 1120 may be an internal storage unit of the electronic device 1100, such as a hard disk or a memory of the electronic device 1100. The memory 1120 may also be an external storage device of the electronic device 1100, such as a plug-in hard disk, a smart card, a flash memory card, etc. provided on the electronic device 1100. Further, the memory 1120 may also include both internal and external storage units of the electronic device 1100.

The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the embodiments of the present application, and they should be construed as being included in the present application.

Claims

1. A method for image retrieval for use in industrial vision, comprising:

responding to a retrieval instruction initiated by a user, and generating an image retrieval task according to a reference image and self-defined information in the retrieval instruction;

receiving a retrieval list fed back by a logic server, and generating a retrieval result corresponding to the retrieval instruction based on the retrieval list; the retrieval list is obtained by the logic server after receiving the task processing results fed back by each distributed system and performing logic combination processing on all the task processing results; the retrieval result comprises a target image which accords with the retrieval instruction;

the method for retrieving the modal characteristics of the modalities includes the steps of configuring corresponding retrieval subtasks for the modal characteristics of the modalities respectively, sending the retrieval subtasks to a distributed system associated with the modalities to retrieve and process the retrieval subtasks through distributed nodes in the distributed system, and includes the following steps:

2. A method according to claim 1, wherein the reduction node, when calculating the similarity between the candidate feature and the modal feature, the similarity may be expressed as:

wherein SimiarLv is the similarity; SMod (i) is the ith characteristic value in the modal characteristic; BMod (i) is the ith feature value in the candidate feature; n is the total number of characteristic values contained in the modality;

the difference value between the feature values corresponding to the modal feature and the candidate feature in the same dimension after normalization;

is composed of

Is measured.

3. The method according to claim 1, wherein the configuring a corresponding retrieval sub-task for each modal feature of each modal, and sending each retrieval sub-task to a distributed system associated with the modal, so as to perform retrieval processing on the retrieval sub-tasks through distributed nodes in the distributed system, comprises:

sending the retrieval subtask corresponding to the M-th level modality to the distributed system corresponding to the M-th level modality based on a retrieval sequence preset by each modality; the initial value of M is 1;

if a retrieval completion instruction fed back by the distributed system corresponding to the M-th level mode is received, judging whether the value of the M is larger than the maximum value of the retrieval sequence; the retrieval completion instruction is generated when the distributed system corresponding to the Mth-level mode obtains the task processing result;

if the value of M is smaller than the maximum value of the retrieval order, sending a first result forwarding instruction to the distributed system corresponding to the M-th level mode, increasing the value of M, returning to execute the retrieval order preset based on each mode, and sending the retrieval subtask corresponding to the M-th level mode to the distributed system corresponding to the M-th level mode; the first result forwarding instruction comprises a communication address of a distributed system corresponding to the M + 1-level mode, so that the distributed system corresponding to the M + 1-level mode sends the task processing result to the distributed system corresponding to the M + 1-level mode;

4. The method according to any one of claims 1-3, wherein the generating an image retrieval task according to the reference image and the custom information in the retrieval instruction in response to the retrieval instruction initiated by the user comprises:

importing the custom information into the semantic recognition algorithm to generate dimension characteristic values in a plurality of patent dimensions;

5. The method of claim 4, wherein generating modal features for a plurality of modalities based on the image retrieval task comprises:

6. The method according to any one of claims 1 to 3, wherein the receiving a search list fed back by a logical server and generating a search result corresponding to the search instruction based on the search list comprises:

7. An apparatus for image retrieval for use in industrial vision, comprising:

the retrieval result generating unit is used for receiving a retrieval list fed back by the logic server and generating a retrieval result corresponding to the retrieval instruction based on the retrieval list; the retrieval list is obtained by the logic server after receiving the task processing results fed back by each distributed system and performing logic combination processing on all the task processing results; the retrieval result comprises a target image which accords with the retrieval instruction;

the retrieval subtask issuing unit comprises:

the retrieval subtask generating unit is used for acquiring a task template associated with the modality, and adding the modality characteristics into the task template to generate the retrieval subtask;

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 6.