CN117093628A

CN117093628A - Vector retrieval method, device, equipment and storage medium

Info

Publication number: CN117093628A
Application number: CN202311070242.3A
Authority: CN
Inventors: 王頔; 韦振中
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2023-08-23
Filing date: 2023-08-23
Publication date: 2023-11-21

Abstract

The disclosure provides a vector retrieval method, a vector retrieval device, vector retrieval equipment and a vector retrieval storage medium, relates to the technical field of data processing, in particular to an intelligent search technology, and can be used for technical scenes such as big data or artificial intelligence. The specific implementation scheme is as follows: obtaining scalar retrieval results; wherein the scalar search result is generated from candidate data retrieved from the data to be searched according to the reference scalar data; in the process of carrying out vector retrieval from the data to be retrieved based on reference vector data, judging whether the current data to be retrieved belongs to candidate data corresponding to the scalar retrieval result or not according to the current data to be retrieved in any vector retrieval process; and determining a vector retrieval result of the current data to be retrieved according to the determination result. According to the technology disclosed by the disclosure, a new vector retrieval mode is provided, and the vector retrieval efficiency is improved.

Description

Vector retrieval method, device, equipment and storage medium

Technical Field

The disclosure relates to the technical field of data processing, in particular to an intelligent search technology, which can be used for technical scenes such as big data or artificial intelligence.

Background

Vector retrieval is a key technology for deep learning, generating AI (Artificial Intelligence ) to solve the dependence of multi-modal data analysis. For unstructured data or multi-modal data, the multi-modal data can be converted into feature vectors by Embedding (Embedding) and then vector retrieval is performed. Meanwhile, since vector search is approximate search, there is also a need to optimize vector search results in combination with scalar data filtering (i.e., scalar search).

Disclosure of Invention

The present disclosure provides a vector retrieval method, apparatus, device, and storage medium.

According to an aspect of the present disclosure, there is provided a vector retrieval method including:

obtaining scalar retrieval results; wherein the scalar search result is generated from candidate data retrieved from the data to be searched according to the reference scalar data;

in the process of carrying out vector retrieval from the data to be retrieved based on reference vector data, judging whether the current data to be retrieved belongs to candidate data corresponding to the scalar retrieval result or not according to the current data to be retrieved in any vector retrieval process;

and determining a vector retrieval result of the current data to be retrieved according to the determination result.

According to another aspect of the present disclosure, there is also provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the vector retrieval methods provided by the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any one of the vector retrieval methods provided according to the embodiments of the present disclosure.

According to the technology disclosed by the invention, a new vector retrieval mode is provided, and the retrieval efficiency is improved.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a flow chart of a vector retrieval method provided by an embodiment of the present disclosure;

FIG. 2 is a flow chart of a scalar search result generation method provided by an embodiment of the present disclosure;

FIG. 3A is a flow chart of a method of data retrieval provided by an embodiment of the present disclosure;

FIG. 3B is a schematic diagram of a data conversion process provided by an embodiment of the present disclosure;

FIG. 4 is a block diagram of a vector retrieval apparatus provided by an embodiment of the present disclosure;

fig. 5 is a block diagram of an electronic device used to implement the vector retrieval method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The vector retrieval method and the vector retrieval device provided by the embodiment of the disclosure are suitable for application scenes for vector retrieval in data to be retrieved comprising scalar data and vector data. The method for searching the vectors provided by the embodiment of the disclosure can be executed by a vector searching device, and the device can be realized by software and/or hardware and is specifically configured in electronic equipment.

For ease of understanding, a detailed description will be given first of all of the vector search method.

Referring to the vector retrieval method shown in fig. 1, the method includes:

s101, acquiring a scalar retrieval result; wherein the scalar search result is generated from candidate data retrieved from the data to be searched in accordance with the reference scalar data.

The reference scalar data is understood to be scalar data input by a search requester, and is used as an important basis for scalar search.

The data to be searched is a searching range for searching the data by the searching demand party. Alternatively, the data retrieval may include scalar retrieval based on scalar data, and accordingly, scalar data having only a numerical value and no direction may be included in the data to be retrieved; or alternatively, the data retrieval may include vector retrieval based on vector data, and accordingly, the data to be retrieved may include vector data having a data size and direction. In one specific example, the name of the trained model is scalar data; the structured network parameters corresponding to the trained model are vector data.

Wherein scalar retrieval and vector retrieval may be implemented in at least one manner of the prior art, which is not limited in any way by this disclosure.

In one embodiment, candidate data associated with the reference scalar data may be retrieved from the data to be retrieved by way of an inverted index or a forward index, or the like.

Illustratively, the scalar search result may include the candidate data itself. In order to reduce the amount of data of the scalar search result, which may include a data identification of the candidate data, in order to perform a localization of the corresponding candidate data, the data storage space is saved.

The data identifier is used for uniquely characterizing the corresponding candidate data, and can be presented in a data number or a sequencing number mode and the like. In order to increase the information amount carried by the data identifier, in a specific implementation manner, the data identifier of the data to be searched may be a sequence number of the data to be searched, so that the sequence condition of different data to be searched can be obtained according to the data identifier. It should be noted that the present disclosure is not limited in any way to the specific ordering adopted.

Alternatively, the scalar search result may be generated in real time during the data search process, and vector search may be performed in real time after the scalar search result is generated. Or alternatively, scalar search may be performed in advance, scalar search results may be generated and stored, and then, when vector search is required, scalar search results may be directly obtained.

The search device that performs scalar search and the search device that performs vector search candidate may be the same or different, and the present disclosure is not limited in this regard. To avoid time delays due to data transmission, scalar search and vector search can be performed using the same search device. Further, in order to ensure the immediacy of data retrieval, scalar retrieval and vector retrieval are generally performed in real time.

S102, in the process of carrying out vector retrieval from the data to be retrieved based on the reference vector data, judging whether the current data to be retrieved belongs to candidate data corresponding to scalar retrieval results or not according to the current data to be retrieved in any vector retrieval process.

The reference vector data can be understood as the vector data input by the retrieval requirement party and is used as an important basis for vector retrieval.

In a specific implementation manner, the vector retrieval can be performed in the data to be retrieved through a preset vector retrieval algorithm. The preset vector search algorithm may be implemented by at least one vector search algorithm in the prior art, which is not limited in this disclosure. For example, the preset vector search algorithm may be a hnsw (Hierarchical Navigable Small World ) algorithm (a graph-based near nearest neighbor search algorithm).

For the current data to be searched in any vector searching process, the similarity between the current data to be searched and the reference vector data can be determined, and if the similarity is larger than a preset similarity threshold value, the current data to be searched and the reference vector data are determined to be related; and if the similarity is not greater than the preset similarity threshold, determining that the current data to be retrieved is irrelevant to the reference vector data. The preset similarity threshold may be set by a skilled person according to needs or experience, or may be determined through a plurality of experiments, which is not limited in any way by the present disclosure.

In the prior art, scalar search and vector search are generally performed separately, and the scalar search result and the vector search result are combined by taking an intersection set, so as to obtain a final data search result. However, when the intersection is empty, a case of search failure will occur. To avoid this, the search range of scalar search and vector search is generally widened, which affects the search efficiency.

The method is different from the prior art, aiming at any vector retrieval process, the data to be retrieved used in the vector retrieval process can be used as current data to be retrieved, and whether the current data to be retrieved belongs to candidate data corresponding to scalar retrieval results or not is judged, so that the data merging process is directly fused into the vector retrieval process, screening of candidate data corresponding to scalar retrieval results or rejection of data to be retrieved corresponding to non-scalar retrieval results in the vector retrieval process is realized, the vector retrieval range is reduced to a certain extent, and the retrieval efficiency is improved.

It should be noted that, the determining process of whether the current data to be retrieved belongs to the candidate data and the similarity matching process of the reference vector data and the current data to be retrieved may be implemented in series or in parallel, which is not limited in this disclosure. In a specific implementation manner, a parallel processing manner may be adopted to improve the vector retrieval efficiency.

S103, determining a vector retrieval result of the current data to be retrieved according to the determination result.

For example, if the current data to be searched does not belong to the candidate data, prohibiting the current data to be searched from being used as a vector search result; if the current data to be searched belongs to the candidate data, determining whether the current data to be searched belongs to a vector search result according to the similarity matching condition of the current data to be searched and the reference vector data. Specifically, if the reference vector data is similar to the current data to be retrieved (i.e., the similarity between the reference vector data and the current data to be retrieved is greater than a preset similarity threshold), determining that the current data to be retrieved belongs to a vector retrieval result; if the reference vector data and the current data to be searched are dissimilar (namely, the similarity between the reference vector data and the current data to be searched is not greater than a preset similarity threshold value), determining that the current search data does not belong to a vector search result.

The embodiment of the disclosure provides a new vector retrieval mode, wherein scalar retrieval results are introduced in a vector retrieval process, and whether the current data to be retrieved belongs to candidate data corresponding to the scalar retrieval results is judged according to the current data to be retrieved in any vector retrieval process, so that a data merging process is directly fused into the vector retrieval process, and screening of candidate data corresponding to the scalar retrieval results or rejection of candidate data corresponding to non-scalar retrieval results in the vector retrieval process is realized. Therefore, whether the vector search range is enlarged or not, the introduction of scalar search results can reduce vector search of the current data to be searched which does not belong to candidate data, reduce the vector search range and improve the search efficiency.

Based on the technical schemes, the disclosure also provides an optional embodiment, in which the generation process of the scalar search result is optimized and improved. It should be noted that, in the embodiments of the present disclosure, parts are not described in detail, and reference may be made to related expressions of other embodiments, which are not limited in any way by the present disclosure.

Referring to fig. 2, a scalar search result generation method includes:

S201, acquiring a data identification of candidate data retrieved from the data to be retrieved based on the reference scalar data.

Wherein the number of candidate data may be at least one; accordingly, the number of data identifications is at least one.

S202, converting the data identification of the candidate data into binary numbers stored in the bitmap structure.

The data identifier of the different data to be retrieved is converted into a state value of the corresponding bit in the bitmap, so that whether the corresponding data to be retrieved belongs to the candidate data is represented by the state value. Since the data to be retrieved has only two states: belonging to candidate data or not, therefore, the representation of the two states of "not belonging to" and "belonging to" can be respectively carried out through "0" and "1" to obtain a binary bitmap structure.

Optionally, the bit positions of the different data to be retrieved in the bitmap can be determined directly according to the preset corresponding relation between the data identifier of the different data to be retrieved and the bit times of the bit positions in the bitmap; and respectively determining the state value of each bit in the bitmap according to the candidate data in the data to be retrieved and other data except the candidate data.

Or alternatively, if the data identifier of the data to be retrieved corresponds to the ordering order of the data to be retrieved, the ordering order may be directly used as the bit order of the corresponding data to be retrieved, and the state value of the corresponding bit may be set according to whether the data to be retrieved belongs to the candidate data.

S203, generating a scalar retrieval result according to the bitmap conversion result.

In an alternative embodiment, the bitmap conversion results may be directly used as scalar retrieval results for subsequent use.

In order to overcome the above problem, in another alternative embodiment, the bitmap conversion result may be converted into the target number again, and a scalar search result including the target number may be generated; wherein the radix of the target number is greater than 2.

It can be understood that, because the radix of the target number is greater than 2 (the radix of the binary system), the memory space occupation amount of the target number is smaller than the memory space occupation amount of the bitmap, so that the memory space occupation amount of the scalar search result is reduced and the use of the memory space is saved on the premise that the information integrity carried in the scalar search result is ensured.

To enhance the readability of scalar search results, in one particular implementation, the target number may be a decimal number, i.e., the base of the target number may be 10.

Correspondingly, the determining whether the current data to be searched belongs to candidate data corresponding to the scalar search result may be: determining a bit to be judged of current data to be searched in a bitmap corresponding to a scalar search result; and judging whether the current data to be retrieved belongs to candidate data or not according to the bits to be judged.

Because the bitmap carries different data to be retrieved, whether the corresponding bit belongs to the state value of the candidate data or not can be directly determined, and the bit of the current data to be retrieved in the bitmap can be used as the bit to be determined; and determining whether the current data to be retrieved belongs to the candidate data or not according to the state value of the bit to be determined, or performing shift processing on the bitmap to shift the bit to be determined to the last bit of the bitmap, and determining whether the current data to be retrieved belongs to the candidate data or not according to the state value of the last bit.

Optionally, the corresponding bit to be determined of the current data to be retrieved in the bitmap may be determined according to a preset corresponding relationship between the data identifier of different data to be retrieved and the bit order of the bit in the bitmap.

Or alternatively, if the data of the data to be retrieved is identified as the ordering order of the data to be retrieved, the ordering order of the current data to be retrieved may be directly used as the bit number of the bit to be determined corresponding to the current data to be retrieved.

It can be understood that by introducing the bitmap to generate the scalar retrieval result, whether the current data to be retrieved belongs to candidate data can be judged by a simple state value comparison or shift processing mode, the operation is convenient, the data operation amount is small, and the accuracy and the retrieval efficiency of vector retrieval are improved.

According to the embodiment of the disclosure, the data identification of the candidate data is converted into the binary number stored in the bitmap structure, and the binary number is used as the generation basis of the scalar retrieval result, and the bitmap carries the data identification of the candidate data in a simpler mode, so that in the subsequent vector retrieval process, when judging whether the current data to be retrieved belongs to the candidate data or not based on the scalar retrieval result, the judgment process is more convenient and rapid, the judgment efficiency is improved, and the vector retrieval efficiency is improved.

When the data identification of the candidate data is large, the data identification of the whole candidate data is directly converted into binary numbers stored in a bitmap structure, and the situation that data overflows or data is omitted possibly affects the accuracy and the effectiveness of judging whether the current data to be searched belongs to the candidate data or not based on a scalar search result. To overcome the above problem, in an alternative embodiment, candidate data may be grouped so that the conversion of bitmaps and the subsequent determination of whether or not it belongs to candidate data is implemented by group.

Illustratively, according to a preset number, grouping the data identification sequences of the candidate data to obtain at least one identification group; for each identification group, the data identification in the identification group is converted into binary numbers stored in a bitmap structure.

The preset number can be set by a technician according to the needs or experience values or can be repeatedly determined through a large number of experiments. Furthermore, if the bitmap conversion result is converted into the target number, the preset number can be determined according to the byte memory occupation condition of the target number.

And converting the data identification in the identification group into a state value of a corresponding bit in the bitmap aiming at each identification group, thereby representing whether the corresponding data to be searched belongs to candidate data or not through the state value. Since the data to be retrieved has only two states: the binary bitmap structure corresponding to the identification group can be obtained by respectively carrying out characterization of the two states of "not belonging to" and "belonging to" through "0" and "1" belonging to the candidate data or not belonging to the candidate data.

By sequentially grouping the data identifications, different identification groups can jointly carry the data identifications of different candidate data, and the situations of omission, confusion and the like of the data identifications are avoided. In addition, the bitmap data is converted for each identification group, so that the huge bitmap structure corresponding to the whole candidate data is disassembled into a plurality of bitmap structures with smaller data volume, the subsequent rapid judgment of the condition of the candidate data is facilitated, and the accuracy of the judgment result is improved.

In an alternative embodiment, the scalar search result including the bitmap conversion result of each identification group may be generated directly according to the generation order of each identification group.

In order to overcome the above problem, in another alternative embodiment, the bitmap conversion result of the identifier array may be converted into the target number for each identifier group, because the memory space occupation of the bitmap is large; generating scalar retrieval results comprising different target system numbers according to the generation sequence of each identification group; wherein the radix of the target number is greater than 2.

It will be appreciated that since the radix of the target number is greater than 2 (the radix of the binary), the memory space footprint of the target number is less than the memory space footprint of the bitmap. Therefore, the generated scalar search result comprising the target system numbers corresponding to different identification groups also has the memory space occupation amount smaller than the memory space occupation amount comprising the bitmap conversion results corresponding to different identification groups, and the memory space is saved. Meanwhile, as each target number is obtained by converting the bitmap conversion result of the corresponding identification group, whether different data to be searched carried in the bitmap conversion result belongs to the state information of the candidate data is not omitted, and therefore the integrity of the information carried in the scalar search result is ensured by the mode.

Because the scalar search result carries bitmap conversion results corresponding to different identification groups, when the judgment of whether the current data to be searched belongs to candidate data is carried out later, the determination of the bit to be judged is carried out, and the determination of the bitmap to which the current data to be searched belongs is carried out. That is, according to the preset number, determining bitmaps to be determined of the current data to be retrieved in different bitmaps corresponding to scalar retrieval results and bits to be determined in the bitmaps to be determined; and judging whether the current data to be retrieved belongs to the candidate data or not according to the bit to be judged in the bit map to be judged. The method has the advantages that accurate positioning of the current data to be retrieved corresponding to the position to be judged can be achieved, influence of positioning errors on judging results of whether the data belong to candidate data is avoided, and therefore accuracy of vector retrieval results is improved.

It can be understood that, in the scalar search result generation stage, a preset number is introduced to sequentially group candidate data, so that a bitmap of an identification group to which the candidate data belongs (i.e., a bitmap to be determined) and a corresponding bit (i.e., a bit to be determined) can be restored by the preset number, so as to determine whether the current data to be searched belongs to the candidate data according to a state value of the corresponding bit.

For example, the data identifier of the current data to be retrieved can be determined, and the quotient value and the remainder value of the preset number of the operators are calculated; determining bitmaps to be judged of the current data to be searched in different bitmaps corresponding to scalar search results according to the quotient; and determining the to-be-determined bit in the to-be-determined bit map according to the residual value.

Since each identification group is obtained by grouping the data identification sequences of the candidate data according to the preset number, the number of bits in the other identification groups is equal to the preset number except that the number of bits in the last generated identification group may be smaller than the preset number. And bitmap conversion results or target system numbers corresponding to different identification groups are arranged according to the generation sequence of each identification group in the scalar retrieval result, so that the bitmap and bit can be accurately positioned in a mode of obtaining quotient values and residual values by using the data identification of the current data to be retrieved as a preset number of quotients.

Specifically, the bitmap corresponding to the identification group generated by the quotient corresponding sequence is used as a bitmap to be judged; and taking the corresponding bit number of the remainder value in the bitmap to be determined as the bit to be determined.

It can be understood that the method of introducing the provider assists in determining the bitmap to be determined and the bit to be determined, the operation method is convenient and quick, the data operation amount is small, the positioning efficiency is high, the accuracy is good, and the method is beneficial to improving the determination efficiency of whether the candidate data belongs to, thereby being beneficial to improving the vector retrieval efficiency.

On the basis of the above technical solutions, the present disclosure also provides a preferred embodiment in which scalar search and vector search are unified into the same search device to provide a service of scalar and vector joint data search through the search device.

Referring to the data retrieval method shown in fig. 3A, the method includes:

s301, acquiring reference scalar data and reference vector data input by a retrieval demand party.

S302, scalar retrieval is carried out in the data to be retrieved, and data identification of candidate data associated with the reference scalar data is obtained.

Wherein the candidate data may be data to be retrieved including reference scalar data.

Illustratively, scalar retrieval may be performed using a forward index or an inverted index, and the like, which is not intended to be limiting in any way in this disclosure.

For example, candidate data with a field value of "test" corresponding to a field of "field" may be retrieved from the data to be retrieved, to obtain an identification set {1,12,13,64,65,70} including the identification of the candidate data corresponding to the data.

S303, grouping the data identifiers of the candidate data according to the preset number to obtain different identifier groups.

S304, aiming at each identification group, converting the data identification of the candidate data in the identification group into binary numbers stored in a bitmap structure, and converting the binary bitmap into decimal numbers.

S305, sequentially adding decimal numbers corresponding to different identification groups into a preset integer array according to the generation sequence of the identification groups.

Taking the preset number as 64 as an example, when the maximum data identifier of the candidate data is smaller than 64, only one identifier group is constructed; when the maximum data identification of the candidate data is not less than 64, constructing an identification group every 64 bits, and sequentially generating different identification groups. If the identification set of the identification of the corresponding data of the candidate data is {1,12,13,64,65,70}, the constructed identification group comprises {1,12,13} and {64,65,70}.

Continuing with the previous example, the identification group {1,12,13} is first converted to a bitmap corresponding to binary number "11000000000010" according to the conversion method shown in fig. 3B, and then the bitmap is converted to decimal number "12290". The identification group {64,65,70}, is first converted to a binary number "1000011", and then the binary bitmap is converted to a decimal number "67". Accordingly, "12290" and "67" are added sequentially to the integer array.

Because the decimal integer occupies 8 bytes of memory, namely 64 bits, one decimal integer can have 64 status bits of data to be retrieved, and memory use and transmission efficiency can be greatly saved.

S306, in the vector retrieval process based on the reference vector data, for any vector retrieval process, the current data to be retrieved in the process is subjected to quotient of a preset number to obtain a quotient value and a remainder value.

For example, vector retrieval may be based on the hnsw algorithm, based on reference vector data.

S307, selecting decimal numbers from the integer array according to the quotient value, and right-shifting the digit number of the remainder value for the decimal numbers.

S308, determining whether the current data to be retrieved is candidate data according to whether the last bit is 1.

S309, if so, determining whether the current data to be searched is a vector search result according to the similarity condition of the reference vector data and the current data to be searched; otherwise, determining that the current data to be retrieved is not a vector retrieval result.

As an implementation of the above-described vector search methods, the present disclosure also provides an optional embodiment of an execution apparatus that implements the above-described vector search methods.

Referring to the vector retrieving apparatus 400 shown in fig. 4, it includes: a scalar search result acquisition module 401, a candidate data decision module 402, and a vector search result determination module 403. Wherein,

a scalar search result acquisition module 401, configured to acquire a scalar search result; wherein the scalar search result is generated from candidate data retrieved from the data to be searched according to the reference scalar data;

A candidate data determining module 402, configured to determine, in a process of performing vector retrieval from the data to be retrieved based on reference vector data, whether the current data to be retrieved belongs to candidate data corresponding to the scalar retrieval result, with respect to the current data to be retrieved in any vector retrieval process;

the vector search result determining module 403 is configured to determine a vector search result of the current data to be searched according to the determination result.

In an alternative embodiment, the device further comprises a scalar search result generation module, specifically comprising:

a candidate data identification acquisition unit configured to acquire a data identification of candidate data retrieved from the data to be retrieved based on the reference scalar data;

the bitmap conversion unit is used for converting the data identifier of the candidate data into binary numbers stored in a bitmap structure;

and the scalar search result generation unit is used for generating the scalar search result according to the bitmap conversion result.

In an alternative embodiment, the scalar search result generation unit is specifically configured to:

converting the bitmap conversion result into a target number and generating a scalar retrieval result comprising the target number;

wherein the cardinality of the target number is greater than 2.

In an alternative embodiment, the bitmap conversion unit includes:

a data identification grouping subunit, configured to group the data identifications of the candidate data according to a preset number, so as to obtain at least one identification group;

and the bitmap conversion subunit is used for converting the data identifications in the identification groups into binary numbers stored in the bitmap structure aiming at the identification groups.

In an alternative embodiment, the scalar search result generation unit includes:

a target system number conversion subunit, configured to convert, for each identifier group, a bitmap conversion result of the identifier group into a target system number;

a scalar search result generation subunit, configured to generate a scalar search result including different target system numbers according to a generation order of each identification group;

wherein the cardinality of the target number is greater than 2.

In an alternative embodiment, the candidate data determination module 402 includes:

the first positioning unit is used for determining the to-be-determined position of the current to-be-retrieved data in the bitmap corresponding to the scalar retrieval result;

and the first judging unit is used for judging whether the current data to be retrieved belongs to the candidate data or not according to the to-be-judged bit.

the second positioning unit is used for determining bitmaps to be judged of the current data to be searched in different bitmaps corresponding to the scalar search result and bits to be judged in the bitmaps to be judged according to the preset quantity;

and the second judging unit is used for judging whether the current data to be retrieved belongs to the candidate data or not according to the bit to be judged in the bit map to be judged.

In an alternative embodiment, the second positioning unit includes:

a Shang Zi unit, configured to determine a data identifier of the current data to be retrieved, and quotient and remainder values of the preset number of quotients;

a map positioning subunit, configured to determine, according to the quotient, a bitmap to be determined of the current data to be retrieved in different bitmaps corresponding to the scalar retrieval result;

and the bit positioning subunit is used for determining the bit to be determined in the bitmap to be determined according to the remainder value.

The vector retrieval device can execute the vector retrieval method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of executing each vector retrieval method.

In the technical scheme of the disclosure, the related processes such as collection, storage, use, processing, transmission, provision, disclosure and the like of scalar search results, reference vector data, reference scalar data and the like all conform to the regulations of related laws and regulations, and the public order is not violated.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 5 illustrates a schematic block diagram of an example electronic device 500 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 5, the apparatus 500 includes a computing unit 501 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 502 or a computer program loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, ROM 502, and RAM 503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

Various components in the device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, etc.; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508 such as a magnetic disk, an optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 501 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the respective methods and processes described above, such as a vector retrieval method. For example, in some embodiments, the vector retrieval method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the vector retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the vector retrieval method by any other suitable means (e.g. by means of firmware).

Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.

Artificial intelligence is the discipline of studying the process of making a computer mimic certain mental processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.) of a person, both hardware-level and software-level techniques. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like; the artificial intelligent software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, a machine learning/deep learning technology, a big data processing technology, a knowledge graph technology and the like.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions provided by the present disclosure are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A vector retrieval method comprising:

2. The method of claim 1, wherein the scalar search result is generated by:

acquiring a data identifier of candidate data retrieved from the data to be retrieved based on the reference scalar data;

converting the data identification of the candidate data into binary numbers stored in a bitmap structure;

and generating the scalar retrieval result according to the bitmap conversion result.

3. The method of claim 2, wherein the generating the scalar search result from bitmap conversion results comprises:

Wherein the cardinality of the target number is greater than 2.

4. The method of claim 2, wherein said converting the data identification of the candidate data into a bitmap structure stored binary number comprises:

grouping the data identification sequences of the candidate data according to a preset quantity to obtain at least one identification group;

for each identification group, the data identification in the identification group is converted into binary numbers stored in a bitmap structure.

5. The method of claim 4, wherein the generating the scalar search result from bitmap conversion results comprises:

converting bitmap conversion results of the identification groups into target system numbers aiming at the identification groups;

generating scalar retrieval results comprising different target system numbers according to the generation sequence of each identification group;

wherein the cardinality of the target number is greater than 2.

6. The method of claim 2, wherein the determining whether the current data to be retrieved belongs to candidate data corresponding to the scalar search result comprises:

determining a bit to be determined of the current data to be retrieved in a bitmap corresponding to the scalar retrieval result;

and judging whether the current data to be retrieved belongs to the candidate data or not according to the to-be-judged bit.

7. The method of claim 4, wherein the determining whether the current data to be retrieved belongs to candidate data corresponding to the scalar search result comprises:

determining bitmaps to be judged in different bitmaps corresponding to the scalar retrieval result of the current data to be retrieved and bits to be judged in the bitmaps to be judged according to the preset quantity;

and judging whether the current data to be retrieved belongs to the candidate data or not according to the bit to be judged in the bitmap to be judged.

8. The method of claim 7, wherein the determining, according to the preset number, a bitmap to be determined of the current data to be retrieved in a different bitmap corresponding to the scalar search result, and a bit to be determined in the bitmap to be determined, includes:

determining the data identification of the current data to be retrieved, and carrying out quotient value and remainder value on the preset number of the operators;

determining bitmaps to be judged of the current data to be searched in different bitmaps corresponding to the scalar search result according to the quotient;

and determining the bit to be judged in the bit map to be judged according to the remainder value.

9. A vector retrieval apparatus comprising:

The scalar search result acquisition module is used for acquiring scalar search results; wherein the scalar search result is generated from candidate data retrieved from the data to be searched according to the reference scalar data;

a candidate data judging module, configured to judge, in a process of vector retrieval from the data to be retrieved based on reference vector data, whether the current data to be retrieved belongs to candidate data corresponding to the scalar retrieval result, with respect to the current data to be retrieved in any vector retrieval process;

and the vector retrieval result determining module is used for determining the vector retrieval result of the current data to be retrieved according to the judging result.

10. The apparatus of claim 9, wherein the apparatus further comprises a scalar search result generation module, comprising:

11. The apparatus according to claim 10, wherein the scalar search result generation unit is specifically configured to:

wherein the cardinality of the target number is greater than 2.

12. The apparatus of claim 10, wherein the bitmap conversion unit comprises:

13. The apparatus according to claim 12, wherein the scalar search result generation unit includes:

wherein the cardinality of the target number is greater than 2.

14. The apparatus of claim 10, wherein the candidate data determination module comprises:

15. The apparatus of claim 12, wherein the candidate data determination module comprises:

16. The apparatus of claim 15, wherein the second positioning unit comprises:

17. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the vector retrieval method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the vector retrieval method according to any one of claims 1-8.

19. A computer program product comprising computer programs/instructions which when executed by a processor implement the steps of the vector retrieval method of any of claims 1-8.