CN117251641A - Vector database retrieval method, system, electronic device and storage medium - Google Patents
Vector database retrieval method, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN117251641A CN117251641A CN202311541835.3A CN202311541835A CN117251641A CN 117251641 A CN117251641 A CN 117251641A CN 202311541835 A CN202311541835 A CN 202311541835A CN 117251641 A CN117251641 A CN 117251641A
- Authority
- CN
- China
- Prior art keywords
- vector
- hash
- original
- selecting
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000013598 vector Substances 0.000 title claims abstract description 223
- 238000000034 method Methods 0.000 title claims abstract description 93
- 238000013507 mapping Methods 0.000 claims abstract description 24
- 238000004422 calculation algorithm Methods 0.000 claims description 24
- 238000004891 communication Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000002452 interceptive effect Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 13
- 238000010845 search algorithm Methods 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 5
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- BASFCYQUMIYNBI-UHFFFAOYSA-N platinum Chemical compound [Pt] BASFCYQUMIYNBI-UHFFFAOYSA-N 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 229910052697 platinum Inorganic materials 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9014—Indexing; Data structures therefor; Storage structures hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/906—Clustering; Classification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a vector database retrieval method, a vector database retrieval system, electronic equipment and a storage medium, wherein the vector database retrieval method comprises the following steps: acquiring a query vector; mapping each original vector in the vector database into a plurality of hash buckets; and selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor searching method. The invention reconstructs two single search algorithms, namely a hash and a graph-based approximate nearest neighbor search method, and integrates the two-stage search to be applied to large-scale high-dimensional data. The method can not only perform hash processing on high-dimensional data to reduce the range so as to improve the efficiency, but also use a graph-based approximate nearest neighbor search method to accurately range so as to increase the accuracy of the obtained result. The scheme can control the problem of recall rate and accuracy rate. The method ensures the speed and the recall rate.
Description
Technical Field
The present invention relates to the field of information technologies, and in particular, to a vector database retrieval method, a vector database retrieval system, an electronic device, and a storage medium.
Background
When the method is oriented to a large amount of data, a quick and accurate retrieval method of information is particularly important. Among them, the approximate nearest neighbor search (ANN) is a core technology that has been paid attention to in information retrieval. ANN aims to efficiently and quickly find an approximate solution to the nearest neighbor search problem in high-dimensional space, at the expense of some accuracy. The currently prevailing search techniques on the market are tree-based, graph, quantization and hash. However, in the face of large data sets in high dimensions, ANNs present certain limitations, including:
(1) As the data dimension increases, many algorithms have difficulty providing accurate results. This is called a dimension disaster. In high-dimensional space, the data becomes more sparse, resulting in less accurate final results.
(2) The algorithm itself also requires a tradeoff between recall and accuracy. Increasing accuracy while decreasing recall and vice versa. Sometimes some programs will require very high recall rates, which means that the algorithm must retrieve most of the nearest neighbors, which can greatly impact efficiency gains. Therefore, the ANN algorithm cannot achieve such high recall without affecting efficiency or a large number of parameters.
Currently there is a lack of a simple and effective solution to this problem.
Disclosure of Invention
The invention aims to provide a vector database retrieval method, a vector database retrieval system, an electronic device and a storage medium, wherein retrieval optimization is realized in two stages by utilizing a hash-based algorithm and a graph-based approximate nearest neighbor search method.
In order to achieve the above object, the present invention provides a vector database retrieval method, comprising the steps of: acquiring a query vector; mapping each original vector in the vector database into a plurality of hash buckets; and selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor searching method.
Optionally, the mapping each original vector in the vector database to a plurality of hash buckets specifically includes: mapping each original vector in the vector database into a plurality of hash buckets based on a locality sensitive hashing algorithm.
Optionally, the selecting, according to the original vectors mapped to the hash buckets, the original vectors with a preset number of digits before the query vector is ranked from small to large by using a graph-based approximate nearest neighbor searching method as an output result, specifically includes: and selecting the original vector with the preset bit number from small to large in the distance from the query vector based on a navigable small world network algorithm in the range of each hash bucket.
Optionally, the selecting, according to the original vectors mapped to the hash buckets, the original vectors with a preset number of digits before the query vector is ranked from small to large by using a graph-based approximate nearest neighbor searching method as an output result, specifically includes: and acquiring the hash bucket corresponding to the query vector, and selecting the original vector with the preset bit number before the ranking from small to large from the distance between the original vector and the query vector in the hash bucket and/or the hash bucket adjacent to the hash bucket.
Optionally, the selecting, according to the original vectors mapped to the hash buckets, the original vectors with a preset number of digits before the query vector is ranked from small to large by using a graph-based approximate nearest neighbor searching method as an output result, specifically includes: selecting a plurality of hash buckets from all the hash buckets, and selecting the original vectors with preset digits from small to large according to the distances between the original vectors in the selected hash buckets and the query vectors.
Optionally, selecting a plurality of hash buckets from all the hash buckets specifically includes: for each hash bucket, obtaining average vectors of all original vectors in the hash bucket; and calculating the distance between each average vector and the query vector, and selecting the hash bucket corresponding to the average vector with the preset bit number before the distance between the average vector and the query vector is ranked from small to large.
Optionally, the method further comprises: converting the query vector into a first hash vector; mapping each original vector in a vector database into a plurality of hash buckets to obtain second hash vectors corresponding to the original vectors one by one; and selecting the original vector with the preset bit number before the ranking from small to large according to the distance between the original vector and the query vector as an output result by a graph-based approximate nearest neighbor searching method according to the first hash vector and the second hash vector.
The invention also provides a retrieval system, comprising: the acquisition module is used for acquiring the query vector; the mapping module is used for mapping each original vector in the vector database into a plurality of hash buckets; and the calculation module is used for selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor search method.
The invention also provides an electronic device comprising: a memory storing a computer program; a processor, communicatively connected to the memory, for executing the vector database retrieval method of any one of the above when the computer program is called; and the display is in communication connection with the processor and the memory and is used for displaying a GUI interactive interface related to the vector database retrieval method.
The present invention also provides a storage medium storing a computer program which, when executed by a processor, implements the vector database retrieval method according to any one of the above.
The vector database retrieval method, the vector database retrieval system, the electronic equipment and the storage medium provided by the invention have the following beneficial effects:
the invention provides a vector database retrieval method, which comprises the following steps: acquiring a query vector; mapping each original vector in the vector database into a plurality of hash buckets; and selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor searching method. So configured, the present invention reconstructs two single search algorithms, hash and graph-based approximate nearest neighbor search methods, and integrates two-stage searches for application on large-scale high-dimensional data. The method can not only perform hash processing on high-dimensional data to reduce the range so as to improve the efficiency, but also use a graph-based approximate nearest neighbor search method to accurately range so as to increase the accuracy of the obtained result. The scheme can control the problem of recall rate and accuracy rate. The method ensures the speed and the recall rate.
The invention also provides a retrieval system, because the retrieval system and the vector database retrieval method belong to the same invention conception, the retrieval system can not only carry out hash processing on high-dimensional data so as to reduce the range and improve the efficiency, but also use an approximate nearest neighbor search method based on a graph to accurately range and increase the accuracy of the obtained result.
The invention also provides an electronic device, which belongs to the same invention conception as the vector database retrieval method, so that the electronic device can not only perform hash processing on high-dimensional data to reduce the range so as to improve the efficiency, but also use an approximate nearest neighbor search method based on a graph to accurately range so as to increase the accuracy of the obtained result.
The invention also provides a storage medium, which belongs to the same inventive concept as the vector database retrieval method, so that the storage medium can not only perform hash processing on high-dimensional data to reduce the range so as to improve the efficiency, but also use an approximate nearest neighbor search method based on a graph to accurately range so as to increase the accuracy of the obtained result.
Drawings
Fig. 1 is a flowchart of a vector database retrieval method according to a first embodiment of the present invention.
Fig. 2 is a flowchart of a vector database retrieval method according to a second embodiment of the present invention.
Fig. 3 is a schematic block diagram of an electronic device according to an embodiment of the invention.
Wherein the reference numerals are as follows:
a 101-processor; 102-a communication interface; 103-memory; 104-a communication bus; 105-display.
Detailed Description
The invention will be described in further detail with reference to the drawings and the specific embodiments thereof in order to make the objects, advantages and features of the invention more apparent. It should be noted that the drawings are in a very simplified form and are not drawn to scale, merely for convenience and clarity in aiding in the description of embodiments of the invention. Furthermore, the structures shown in the drawings are often part of actual structures. In particular, the drawings are shown with different emphasis instead being placed upon illustrating the various embodiments.
It will be understood that when an element or layer is referred to as being "on" or "connected to" another element or layer, it can be directly on, connected to, or comprise the intervening element or layer. In contrast, when an element is referred to as being "directly on" …, "directly connected to" another element or layer, there are no intervening elements or layers involved. Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the present invention. Spatially relative terms, such as "under … …," "below," "lower," "above … …," "upper," and the like, may be used herein for convenience of description to describe one element or feature's relationship to another element or feature as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use and operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements or features described as "under" … … "," below "and" beneath "would then be oriented" on "other elements or features. The device may be otherwise oriented (rotated 90 degrees or other orientations) and the spatially relative descriptors used herein interpreted accordingly. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups. As used herein, the term "and/or" includes any and all combinations of the associated listed items.
The invention aims to provide a vector database retrieval method, a vector database retrieval system, an electronic device and a storage medium, wherein retrieval optimization is realized in two stages by utilizing a hash-based algorithm and a graph-based approximate nearest neighbor search method.
In order to achieve the above objective, the present invention provides a vector database searching method, please refer to fig. 1, fig. 1 is a flow chart of a vector database searching method according to a first embodiment of the present invention. As shown in fig. 1, the method comprises the steps of:
acquiring a query vector;
mapping each original vector in the vector database into a plurality of hash buckets;
and selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor searching method.
So configured, the present invention reconstructs two single search algorithms, hash and graph-based approximate nearest neighbor search methods, and integrates two-stage searches for application on large-scale high-dimensional data. The method can not only perform hash processing on high-dimensional data to reduce the range so as to improve the efficiency, but also use a graph-based approximate nearest neighbor search method to accurately range so as to increase the accuracy of the obtained result. The scheme can control the problem of recall rate and accuracy rate. The method ensures the speed and the recall rate. The invention also increases the parallelism of the algorithm, can be applied to a distributed system, and can reduce the time delay of an index constructing stage and a query stage by improving the parallelism of the algorithm. Compared with the scheme of Kmeans clustering+graph indexing, the method guarantees feasibility under large data volume. For example, in a billion-scale high-dimensional vector scene, the whole data cannot be imported into the memory, a high-quality clustering algorithm is difficult to complete, and the hash algorithm is utilized to divide the buckets, so that the whole data is not required to be stored into the memory at the same time.
It should be understood that mapping each original vector in the vector database into a plurality of hash buckets mainly uses the related principles of the hash algorithm, and specifically includes: each of the original vectors in the vector database is mapped into a plurality of the hash buckets based on a locality sensitive hashing algorithm (LSH). High-dimensional data suffers from dimensional-suffering problems, and traditional distance metrics lose effectiveness due to increasing sparsity and computational complexity of the data. The hashing algorithm helps solve this problem. The hash algorithm aims to preserve the similarity between data while reducing the computational burden by mapping high-dimensional data points to low-dimensional data. One popular hashing technique in an ANN search is Locality Sensitive Hashing (LSH). The basic idea of LSH is that if two texts are similar in the original data space, they still have similarity after conversion by a hash function, and if they are not, they still have no similarity after conversion. The conventional hash algorithm maps the original data into the corresponding bucket, and during the hash processing of the data set, different data are mapped into the same bucket/collision is avoided as much as possible, and the problem can be solved by mapping the data into other empty buckets through re-hash. After the LSH transforms two adjacent data points in the original data space by the same mapping or projection, the probability that the two data points remain adjacent in the new data space is large, while the probability that non-adjacent data points are mapped to the same bucket is small. That is, if we do some hash mapping on the original data, we want the two originally adjacent data to be hashed into the same bucket, with the same bucket number.
It should be understood that, in the course of the hash process, the query vector and the original vector should be converted into corresponding hash vectors, specifically, the query vector is converted into a first hash vector; mapping each original vector in a vector database into a plurality of hash buckets to obtain second hash vectors corresponding to the original vectors one by one; and selecting the original vector with the preset bit number before the ranking from small to large according to the distance between the original vector and the query vector as an output result by a graph-based approximate nearest neighbor searching method according to the first hash vector and the second hash vector.
Hashing algorithms such as LSH may perform poorly in terms of accuracy. The LSH algorithm may also filter out some real neighbors during the dimension reduction process, ultimately resulting in low recall. While the graph-based approximate nearest neighbor searching method, namely graph index, has shown good effects in the field of approximate nearest neighbor searching, the graph index structure still has some problems such as greatly increased memory consumption caused by the graph index structure, insufficient searching parallelism, possibly occurrence of a problem of local optimal solution or non-monotonous searching path, high composition cost under hundreds of millions of data volume, incapacity of becoming a good distributed solution, and the like. Thus, constructing a two-stage search technique can optimize existing single ANN search techniques in these respects, and in one exemplary embodiment, better retrieval results can be achieved using a graph index such as the navigable small world network algorithm (HNSW). The invention further provides the following technical scheme based on the invention: the method specifically includes the steps of selecting, as an output result, the original vectors with a preset number of digits before ranking from small to large according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor search method, wherein the original vectors are the same as the query vectors in distance from small to large:
and selecting the original vector with the preset bit number from small to large in the distance from the query vector based on a navigable small world network algorithm in the range of each hash bucket.
In LSH, it can not be guaranteed that all data in a sub-barrel are similar, but the invention uses HNSW as an efficient search algorithm, and HNSW search can be performed in parallel between different barrels, so in the search stage, the invention can select to use a larger number of hash barrels and finally merge the results to ensure the recall rate.
Preferably, the selecting, according to the original vectors mapped to the hash buckets, the original vectors with a preset number of digits before ranking from small to large from the distance of the query vector as output results by using a graph-based approximate nearest neighbor searching method specifically includes:
and acquiring the hash bucket corresponding to the query vector, and selecting the original vector with the preset bit number before the ranking from small to large from the distance between the original vector and the query vector in the hash bucket and/or the hash bucket adjacent to the hash bucket.
By the arrangement, the adjacent hash buckets can be screened more accurately, and the vectors closer to the query vector can be prevented from being missed because of the retrieval of the single hash bucket.
Preferably, the selecting, according to the original vectors mapped to the hash buckets, the original vectors with a preset number of digits before ranking from small to large from the distance of the query vector as output results by using a graph-based approximate nearest neighbor searching method specifically includes:
selecting a plurality of hash buckets from all the hash buckets, and selecting the original vectors with preset digits from small to large according to the distances between the original vectors in the selected hash buckets and the query vectors. In an exemplary embodiment, the selecting a plurality of hash buckets from all the hash buckets specifically includes: for each hash bucket, obtaining average vectors of all original vectors in the hash bucket; and calculating the distance between each average vector and the query vector, and selecting the hash bucket corresponding to the average vector with the preset bit number before the distance between the average vector and the query vector is ranked from small to large. By the arrangement, the hash buckets for retrieval can be screened out more scientifically, and the retrieval efficiency and accuracy are improved.
Referring to fig. 2, fig. 2 is a flowchart of a vector database searching method according to a second embodiment of the present invention. Fig. 2 shows a specific scheme in an exemplary embodiment of the invention, comprising the steps of:
LSH processing is carried out on the original vectors, and the original vectors are distributed into corresponding hash buckets;
simultaneously constructing HNSW indexes for data in each bucket;
mapping the query vector to the hash bucket and selecting a current hash bucket and a plurality of nearby hash buckets for a total of m buckets;
simultaneously inquiring m HNSW graph structures corresponding to m barrels of data;
the merging result generates k-nearest neighbor.
The invention also provides a retrieval system, comprising:
the acquisition module is used for acquiring the query vector;
the mapping module is used for mapping each original vector in the vector database into a plurality of hash buckets;
and the calculation module is used for selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor search method.
Because the search system and the vector database search method belong to the same invention conception, the search system can not only perform hash processing on high-dimensional data to reduce the range so as to improve the efficiency, but also use an approximate nearest neighbor search method based on a graph to accurately range so as to increase the accuracy of the obtained result.
Referring to fig. 3, fig. 3 is a block diagram of an electronic device according to an embodiment of the invention. As shown in fig. 3, the present invention further provides an electronic device, including:
a memory 103 storing a computer program;
a processor 101, communicatively connected to the memory, for executing the vector database retrieval method according to any one of the above-described methods when the computer program is called;
and a display 105, communicatively coupled to the processor and the memory, for displaying a GUI interactive interface associated with the vector database retrieval method.
Because the electronic equipment and the vector database retrieval method belong to the same invention conception, the electronic equipment can not only carry out hash processing on high-dimensional data so as to reduce the range and improve the efficiency, but also use an approximate nearest neighbor search method based on a graph to accurately range and increase the accuracy of the obtained result.
As shown in fig. 3, the electronic device further comprises a communication interface 102 and a communication bus 104, wherein the processor 101, the communication interface 102, and the memory 103 communicate with each other via the communication bus 104. The communication bus 104 may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The communication bus 104 may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus. The communication interface 102 is used for communication between the electronic device and other devices.
The processor 101 of the present invention may be a central processing unit (Central Processing Unit, CPU), other general purpose processor, digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, and the processor 101 is a control center of the electronic device, and connects various parts of the entire electronic device using various interfaces and lines.
The memory 103 may be used to store the computer program, and the processor 101 may implement various functions of the electronic device by running or executing the computer program stored in the memory 103 and invoking data stored in the memory 103.
The memory 103 may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The present invention also provides a storage medium storing a computer program which, when executed by a processor, implements the vector database retrieval method according to any one of the above.
Because the storage medium and the vector database retrieval method belong to the same invention conception, the storage medium can not only carry out hash processing on high-dimensional data so as to reduce the range and improve the efficiency, but also use a graph-based approximate nearest neighbor search method to accurately range so as to increase the accuracy of the obtained result.
Because the storage medium provided by the invention and the vector database searching method described above belong to the same inventive concept, the storage medium provided by the invention has all the advantages of the vector database searching method described above, so the beneficial effects of the storage medium provided by the invention are not repeated here.
The storage media of embodiments of the present invention may take the form of any combination of one or more computer-readable media. The readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer hard disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Based on the technical principle of the invention, the inventor performs a comparison experiment with the prior art, and experimental data are as follows:
experimental environment:
operating system: ubuntu 20.04.5 LTS
CPU：Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz
Memory: 256G
Data set: SIFT1M
In addition to the above resources, other resources are not limited. GPU implementations version refer to the SONG paper.
Experiment: single concurrency test
Experimental parameters:
number of barrels: 100
M=31
K=100
Compared with a single HNSW algorithm, the invention has the advantages of higher recall rate, shorter time and larger data volume capable of being processed.
It should be further noted that although the present invention has been disclosed in the preferred embodiments, the above embodiments are not intended to limit the present invention. Many possible variations and modifications of the disclosed technology can be made by anyone skilled in the art without departing from the scope of the technology, or the technology can be modified to be equivalent. Therefore, any simple modification, equivalent variation and modification of the above embodiments according to the technical substance of the present invention still fall within the scope of the technical solution of the present invention.
It should be further understood that the terms "first," "second," "third," and the like in this specification are used merely for distinguishing between various components, elements, steps, etc. in the specification and not for indicating a logical or sequential relationship between the various components, elements, steps, etc., unless otherwise indicated.
It should also be understood that the terminology described herein is for the purpose of describing particular embodiments only, and is not intended to limit the scope of the present invention. It must be noted that, as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. For example, reference to "a step" or "an apparatus" means a reference to one or more steps or apparatuses, and may include sub-steps as well as sub-apparatuses. All conjunctions used should be understood in the broadest sense. And, the word "or" should be understood as having the definition of a logical "or" rather than a logical "exclusive or" unless the context clearly indicates the contrary. Further, implementation of embodiments of the present invention may include performing selected tasks manually, automatically, or in combination.
Claims (10)
1. A method for retrieving a vector database, comprising the steps of:
acquiring a query vector;
mapping each original vector in the vector database into a plurality of hash buckets;
and selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor searching method.
2. The method for searching the vector database according to claim 1, wherein the mapping each original vector in the vector database into a plurality of hash buckets specifically comprises:
mapping each original vector in the vector database into a plurality of hash buckets based on a locality sensitive hashing algorithm.
3. The vector database retrieval method according to claim 1, wherein said selecting as an output result said original vector having a distance from said query vector as a preset number of digits from a small to a large rank by a graph-based approximate nearest neighbor search method based on each of said original vectors mapped to a plurality of said hash buckets, specifically comprises:
and selecting the original vector with the preset bit number from small to large in the distance from the query vector based on a navigable small world network algorithm in the range of each hash bucket.
4. A vector database retrieval method according to any one of claims 1 to 3, wherein said selecting as an output result said original vector having a distance from said query vector as a preset number of digits from a small to a large rank by a graph-based approximate nearest neighbor search method based on each of said original vectors mapped to a plurality of said hash buckets, specifically comprises:
and acquiring the hash bucket corresponding to the query vector, and selecting the original vector with the preset bit number before the ranking from small to large from the distance between the original vector and the query vector in the hash bucket and/or the hash bucket adjacent to the hash bucket.
5. A vector database retrieval method according to any one of claims 1 to 3, wherein said selecting as an output result said original vector having a distance from said query vector as a preset number of digits from a small to a large rank by a graph-based approximate nearest neighbor search method based on each of said original vectors mapped to a plurality of said hash buckets, specifically comprises:
selecting a plurality of hash buckets from all the hash buckets, and selecting the original vectors with preset digits from small to large according to the distances between the original vectors in the selected hash buckets and the query vectors.
6. The method for retrieving a vector database according to claim 5, wherein selecting a plurality of hash buckets from all the hash buckets comprises:
for each hash bucket, obtaining average vectors of all original vectors in the hash bucket;
and calculating the distance between each average vector and the query vector, and selecting the hash bucket corresponding to the average vector with the preset bit number before the distance between the average vector and the query vector is ranked from small to large.
7. A vector database retrieval method as recited in any one of claims 1 to 3, further comprising:
converting the query vector into a first hash vector;
mapping each original vector in a vector database into a plurality of hash buckets to obtain second hash vectors corresponding to the original vectors one by one;
and selecting the original vector with the preset bit number before the ranking from small to large according to the distance between the original vector and the query vector as an output result by a graph-based approximate nearest neighbor searching method according to the first hash vector and the second hash vector.
8. A retrieval system applied to the vector database retrieval method according to any one of claims 1 to 7, comprising:
the acquisition module is used for acquiring the query vector;
the mapping module is used for mapping each original vector in the vector database into a plurality of hash buckets;
and the calculation module is used for selecting the original vectors with the preset bit number before the ranking from small to large from the distance of the query vector as an output result according to the original vectors mapped to the hash buckets and by a graph-based approximate nearest neighbor search method.
9. An electronic device, comprising:
a memory storing a computer program;
a processor in communication with the memory, the processor executing the vector database retrieval method of any one of claims 1-7 when the computer program is invoked;
and the display is in communication connection with the processor and the memory and is used for displaying a GUI interactive interface related to the vector database retrieval method.
10. A storage medium storing a computer program, characterized by: the computer program, when executed by a processor, implements the vector database retrieval method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311541835.3A CN117251641A (en) | 2023-11-20 | 2023-11-20 | Vector database retrieval method, system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311541835.3A CN117251641A (en) | 2023-11-20 | 2023-11-20 | Vector database retrieval method, system, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117251641A true CN117251641A (en) | 2023-12-19 |
Family
ID=89135423
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311541835.3A Pending CN117251641A (en) | 2023-11-20 | 2023-11-20 | Vector database retrieval method, system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117251641A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117873909A (en) * | 2024-03-13 | 2024-04-12 | 上海爱可生信息技术股份有限公司 | Fault diagnosis execution method, fault diagnosis execution system, electronic device, and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012078930A (en) * | 2010-09-30 | 2012-04-19 | Ntt Docomo Inc | Content retrieval device and content retrieval method |
| CN114357220A (en) * | 2022-01-07 | 2022-04-15 | 南京市中西医结合医院 | Similar medical image calculation method based on locality sensitive hashing algorithm |
| CN115408555A (en) * | 2022-08-31 | 2022-11-29 | 中国银行股份有限公司 | Voiceprint retrieval method, system, storage medium and electronic equipment |
-
2023
- 2023-11-20 CN CN202311541835.3A patent/CN117251641A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012078930A (en) * | 2010-09-30 | 2012-04-19 | Ntt Docomo Inc | Content retrieval device and content retrieval method |
| CN114357220A (en) * | 2022-01-07 | 2022-04-15 | 南京市中西医结合医院 | Similar medical image calculation method based on locality sensitive hashing algorithm |
| CN115408555A (en) * | 2022-08-31 | 2022-11-29 | 中国银行股份有限公司 | Voiceprint retrieval method, system, storage medium and electronic equipment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117873909A (en) * | 2024-03-13 | 2024-04-12 | 上海爱可生信息技术股份有限公司 | Fault diagnosis execution method, fault diagnosis execution system, electronic device, and storage medium |
| CN117873909B (en) * | 2024-03-13 | 2024-05-28 | 上海爱可生信息技术股份有限公司 | Fault diagnosis execution method, fault diagnosis execution system, electronic device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10162598B2 (en) | Flash optimized columnar data layout and data access algorithms for big data query engines | |
| US10452691B2 (en) | Method and apparatus for generating search results using inverted index | |
| US8625907B2 (en) | Image clustering | |
| US9442929B2 (en) | Determining documents that match a query | |
| US11604834B2 (en) | Technologies for performing stochastic similarity searches in an online clustering space | |
| US11327985B2 (en) | System and method for subset searching and associated search operators | |
| US20180276507A1 (en) | Machine learning classifiers | |
| CN117251641A (en) | Vector database retrieval method, system, electronic device and storage medium | |
| CN115203455A (en) | Techniques to tune performance and/or accuracy of similarity searches using random associative memories | |
| CN112070550A (en) | Keyword determination method, device and equipment based on search platform and storage medium | |
| WO2022007596A1 (en) | Image retrieval system, method and apparatus | |
| CN111368020A (en) | Feature vector comparison method and device and storage medium | |
| CN110889424B (en) | Vector index establishing method and device and vector retrieving method and device | |
| CN110880005B (en) | Vector index establishing method and device and vector retrieving method and device | |
| Teixeira et al. | Scalable locality-sensitive hashing for similarity search in high-dimensional, large-scale multimedia datasets | |
| CN116610848B (en) | Vector database retrieval method based on NPU optimization and readable storage medium | |
| Feng et al. | Effective optimizations of cluster-based nearest neighbor search in high-dimensional space | |
| CN117540056B (en) | Method, device, computer equipment and storage medium for data query | |
| CN116304258B (en) | Retrieval method, retrieval system and readable storage medium based on vector database | |
| CN116796021B (en) | Image retrieval method, system, electronic device and medium | |
| WO2023222091A1 (en) | Vector retrieval method and apparatus | |
| CN116777727B (en) | Integrated memory chip, image processing method, electronic device and storage medium | |
| CN117271621A (en) | Vector fuzzy retrieval method and NPU | |
| WO2022165797A1 (en) | Data layout optimization for object-oriented storage engine | |
| CN117456210A (en) | Vector processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |