CN110209895A - Vector index method, apparatus and equipment - Google Patents
Vector index method, apparatus and equipment Download PDFInfo
- Publication number
- CN110209895A CN110209895A CN201910490118.XA CN201910490118A CN110209895A CN 110209895 A CN110209895 A CN 110209895A CN 201910490118 A CN201910490118 A CN 201910490118A CN 110209895 A CN110209895 A CN 110209895A
- Authority
- CN
- China
- Prior art keywords
- vector
- similarity
- index
- center
- grouping
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/908—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This specification embodiment provides a kind of vector index method, apparatus and equipment.This method comprises: extracting the feature vector of retrieval object；Vector similar with feature vector is obtained from graphics processor GPU, as first part's similarity vector；Vector similar with feature vector is obtained from central processor CPU, as second part similarity vector；According to first part's similarity vector and second part similarity vector, the search result of retrieval object is determined.
Description
Technical field
This specification one or more embodiment be related to field of computer technology more particularly to a kind of vector index method,
Device and equipment.
Background technique
With the development of internet technology, when retrieving interested content, retrieval object becomes to become increasingly complex people.
For example, gradually development is search pictures, search audio data and video data etc. by search text key word.With search and
Query object becomes more complicated, and the index of difficulty grade of search increases.
Usually complicated retrieval object will use high dimension vector to indicate, therefore it is many that comparison is usually required in retrieving
The distance between high dimension vector or similarity.When needing the data volume retrieved very big, the calculation amount being related to it is huge and expend compared with
Long retrieval time is too long, it is therefore desirable to provide a kind of vector index scheme for improving retrieval performance.
Summary of the invention
This specification embodiment provides a kind of vector index method, apparatus and equipment, can use CPU and GPU to data
Object is retrieved, and recall precision is improved.
In a first aspect, this specification embodiment provides a kind of vector index method, comprising:
Extract the feature vector of retrieval object；Vector similar with feature vector is obtained from graphics processor GPU, as
First part's similarity vector；Similar with feature vector vector is obtained from central processor CPU, as second part it is similar to
Amount；According to first part's similarity vector and second part similarity vector, the search result of retrieval object is determined.
Second aspect, this specification embodiment provide a kind of vector index device, comprising:
Vector extraction module, for extracting the feature vector of retrieval object；First retrieval module is used for from graphics processor
Vector similar with described eigenvector is obtained in GPU, as first part's similarity vector；Second retrieval module, for therefrom
Vector similar with described eigenvector is obtained in central processor CPU, as second part similarity vector；As a result determining module,
For determining the retrieval knot of the retrieval object according to first part's similarity vector and the second part similarity vector
Fruit.
The third aspect, this specification embodiment provide a kind of vector index equipment, comprising: memory and processor；This is deposited
Reservoir is for storing program；The processor is for reading the executable program code stored in memory to execute above-mentioned first party
The vector index method in face.
According to the vector index method, apparatus and equipment in this specification embodiment, opposite direction duration set carries out cutting in advance,
The similarity retrieval that respectively data that cutting obtains are carried out with the feature vector for retrieval object using CPU and GPU, is being utilized
While GPU improves arithmetic speed, the resource of CPU is efficiently used.
Detailed description of the invention
It, below will be to required in this specification embodiment in order to illustrate more clearly of the technical solution of this specification embodiment
Attached drawing to be used is briefly described, for those of ordinary skill in the art, in the premise not made the creative labor
Under, it is also possible to obtain other drawings based on these drawings.
Fig. 1 shows the configuration diagram of the searching system of this specification one embodiment；
Fig. 2 shows the flow diagrams of the vector index method of this specification one embodiment；
Fig. 3 shows the flow diagram of the vector index method of this specification one embodiment；
Fig. 4 shows the flow diagram of the vector index method of this specification another embodiment；
Fig. 5 shows the structural schematic diagram of the vector index device of this specification one embodiment；
Fig. 6 shows the structure chart that can be realized the exemplary hardware architecture of calculating equipment of this specification embodiment.
Specific embodiment
The feature and exemplary embodiment of the various aspects of this specification is described more fully below, in order to make this specification
Objects, technical solutions and advantages are more clearly understood, and with reference to the accompanying drawings and embodiments, carry out to this specification further detailed
Description.It should be understood that specific embodiment described herein is only configured to explain this specification, it is not configured as limiting this theory
Bright book.To those skilled in the art, this specification can be in the feelings for not needing some details in these details
Implement under condition.Below the description of embodiment is used for the purpose of providing to this specification more by showing the example of this specification
Good understanding.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence " including ... ", it is not excluded that including
There is also other identical elements in the process, method, article or equipment of element.
It, can be by extracting the feature vector of retrieval object in extensive vector index, and calculate between feature vector
The method of characteristic similarity, by the highest one or more features vector pair of characteristic similarity of the feature vector with retrieval object
The data object answered, the search result as retrieval object.
That is, the point in high-dimensional vector space is mapped as object is retrieved by feature extraction, it thus will retrieval
The similarity searching problem of object is converted into the arest neighbors in higher dimensional space and searches problem.
With directly retrieve data object data file compared with, based between feature vector characteristic similarity carry out to
Retrieving is measured, has the characteristics that computation complexity is lower and operand is lesser, vector index efficiency can be improved.
In this specification embodiment, data object is according to different data types, such as may include text data, figure
As a variety of different types of data objects such as data, audio data, video data.For the sake of to simplify the description, under this specification
The multiple embodiments stated illustrate vectorization of the image data in vector index by taking image data as an example.But the description is not answered
It is read as the range or operational feasibility of limitation this programme, the retrieval object of other data types other than image retrieval object
Processing method be consistent with the processing method to image retrieval object.
In one embodiment of this specification, feature vector is the characteristic information according to data object, by data object
It is mapped to the vector data that high-dimensional vector space obtains.In picture search, as an example, to search for personage in specified image
For face, retrieve facial image the step of may include:
Firstly, it is necessary to the face feature of personage in the facial image is extracted, ratio, face's colour of skin, head such as face's length and width
Send out the distance between color, the width (nose, mouth etc. in such as human face five-sense-organ) of local detail, eyes；Secondly, by the face of extraction
Feature Mapping obtains the vector data of a specified dimension to high-dimensional vector space, and the vector data of a face picture can be with
For characterizing the feature in the facial image, and it is properly termed as the feature vector of the facial image；Then, by the face figure of extraction
Feature vector compares in the feature vector and image library of picture, and according to the distance between feature vector or similarity, determining should
The search result of facial image.
In this specification one or more embodiment, central processing unit (Central Processing can use
Unit, CPU) or graphics processor (Graphics Processing Unit, GPU) for vector index process provide calculating money
Source.Wherein, CPU can be used for interpretive machine instruction and processing computer software as a kind of super large-scale integration
In data；Processor of the GPU as video card can be used for executing complicated data operation and geometric operation.
This specification one or more embodiment provides a kind of vector index method and vector index method, can be in benefit
During carrying out vector index with feature vector, CPU and GPU synergistic application is in the storage of data object, classification and retrieves
Cheng Zhong makes full use of cpu resource and GPU resource, promotes retrieval performance and recall precision.
Fig. 1 shows the configuration diagram of the searching system of this specification one embodiment.As shown in Figure 1, this specification
The searching system 100 of exemplary embodiment may include: data storage 101, CPU102, GPU103 and proxy server
104。
In one embodiment, the data stored in data storage 101 may include vector data set, and can fit
For such as 10,000,000,000 GB or more scales on a large scale sample storage and calculate demand, can economic and efficient big rule of analysis processing
Apperance notebook data.
As an example, data storage 101 can provide a variety of data processing services, can be inquired with support structureization
The vector index of language (Structured Query Language, SQL), and can support in data storage 101 to
Vector data in amount data acquisition system is read or write operation.
As an example, the vector data set stored in data storage 101 may include the feature of image data
Vector, the feature vector of voice data, the feature vector of video data and natural language processing (Natural Language
Process, NLP) data at least one of feature vector.
In one embodiment, proxy server 104 is for receiving retrieval request, according to the retrieval object in retrieval request
CPU and GPU is called to retrieve retrieval object.
It continues to refer to figure 1, in this specification one embodiment, vector index method may include:
Step S110 tears the vector data set in data storage 101 open with reference to memory parameters and video memory parameter
Point, obtain first part's vector subset of deposit memory and the second part vector subset of deposit video memory.
Step S120 carries out vector index in first part's vector subset of deposit memory by CPU, obtains data object
First part's similar features vector.
Step S130 carries out vector index by second part vector subset of the GPU to deposit video memory, obtains retrieval object
Second part similar features vector.
Step S140 determines retrieval object according to first part's similar features vector sum second part similar features vector
Search result.
In the vector index method of this specification, the vector index of data object is carried out at collaboration using CPU and GPU
Reason.The parallel processing capability and cpu resource for making full use of GPU improve the arithmetic speed and runnability of vector index.
Fig. 2 shows the flow diagrams of the vector index method of this specification one embodiment.As shown in Fig. 2, at this
In one embodiment of specification, vector index method 200 may include:
Step S210, opposite duration set carry out cluster calculation, obtain the class cluster of the specified quantity in vector set and each
The central point feature vector of class cluster.
In this step, the cluster calculation of opposite duration set, it can be understood as be according to the distance between vector or similar
A kind of data processing method being grouped is spent, every group of vector is known as a class cluster.By cluster calculation, available specified number
A class cluster is measured, and obtains the center vector of each class cluster in the specified quantity class cluster.Wherein, the center vector of each class cluster
With following similarity condition: vector similarity in same class cluster is greater than the vector similarity in inhomogeneity cluster, and to span
With a distance from center vector from same class cluster, less than the distance of the center vector of the vector distance inhomogeneity cluster.
In one or more embodiments of this specification, it can be clustered using a variety of clustering method opposite direction duration sets
It calculates.
In one embodiment, quick clustering method k-means algorithm can be used.In this embodiment, cluster calculation can
To include: k vector of random selection as initial center vector, each vector was assigned to apart from nearest center vector institute's generation
In the class cluster of table；After institute's directed quantity is assigned, for all vector datas in same class cluster, pass through the side being averaged
Formula recalculates the center vector of such cluster, to update center vector；According to the center vector of update, each vector is distributed
In class cluster representated by the new center vector nearest to distance.Iteration carries out vector distribution and vector center in vector set
The step of point updates until the variation of the center vector of each class cluster is less than distance threshold, or reaches maximum update times.
In another embodiment, can be taken based on level clustering algorithm carry out cluster calculation, such as BIRCH algorithm,
CURE algorithm etc.；In yet another embodiment, the clustering algorithm of density, such as DBSCAN algorithm, OPTICS can be taken based on
Algorithm etc.；Other clustering methods can also be taken to carry out cluster calculation, this specification embodiment is not specifically limited in this embodiment.
Step S220 obtains the central point feature vector of each class cluster, obtains central point feature vector set.
Step S230 divides central point feature vector set, obtains center point set A and center point set B.Wherein, in
Heart point set A includes a part of central point feature vector in central point feature vector set, and center point set B includes central point
Another part central feature vector in feature vector set.
In the description of the following embodiments of this specification, it can use GPU and determine the phase for retrieving object in the point set A of center
Like feature vector, and the similar features vector that object is retrieved in the point set B of center is determined using CPU.
Step S240 establishes the index of central point feature vector for each central point feature vector in the point set A of center
File, centered on point set A central point index file.
Step S250, for class cluster belonging to each central point feature vector file in the point set A of center, for such cluster
In central point feature vector other than each feature vector, establish the index data of feature vector, centered on point set A
Vector index file.
Step S260 establishes the index of central point feature vector for each central point feature vector in the point set B of center
Data, centered on point set B first order index data.
Step S270, for class cluster belonging to each central point feature vector in the point set B of center, in such cluster
Each feature vector other than central point feature vector, establishes the index data of feature vector, centered on point set B second
Grade index data.
In one or more embodiments of this specification, retrieval is being searched compared to the similarity for directly passing through feature vector
The method of the similar features vector of object, retrieval rate can obviously be accelerated and improve to examine by carrying out vector index by vector index
Rope efficiency.
In one embodiment, the central point index file of center point set A or vector index file, center point set B
Central point index file or vector index file can be direct retrieval file or inverted index file (Inverted File
Index)。
In one embodiment, direct index file can be used for recording the mark ID of feature vector Yu this feature vector
Between corresponding relationship, a feature vector at least has a mark ID, and the mark ID of each feature vector have it is unique
Property.
In one embodiment, inverted index file can be used for recording the center vector in each feature vector grouping
The center vector mark ID between corresponding relationship and each feature vector grouping in center vector other than other
Corresponding relationship between feature vector and the mark ID of this feature vector.
In one or more embodiments of this specification, inverted index is a kind of higher vector index mode of efficiency,
It is particularly suitable for the retrieval of mass data.It is only a kind of implementation that this specification provides using inverted index, in practical application
In other modes that arbitrarily can establish index information can be used as the implementation of this specification embodiment.
Fig. 3 shows the flow diagram of the vector index method of this specification one embodiment.As shown in figure 3, at this
In one embodiment of specification, vector index method 300 may include:
Step S311 using the central point index file (such as central point index A) pre-established, is obtained in GPU
N1 nearest central point vector of distance retrieval object, the N1 specified integer for being more than or equal to 1 in heart point set A.
Step S313, for each of N1 central point vector central point vector, according to the central point pre-established
The vector index file (such as vector index A) of set A in the class cluster belonging to central point vector, passes through reverse index, determines
The central point vector of most like specified number with retrieval object, as the retrieval knot in class cluster belonging to each central point vector
Fruit.
In this step, the search result in class cluster belonging to each central point vector is summarized, obtains GPU processing
Obtained first part's similarity vector search result, according to each of retrieval object and first part similarity vector search result
The order of the similarity of similarity vector from big to small is ranked up, and obtains K1 top ranked similarity vector of similarity, and K1 is
The specified integer for being more than or equal to 1.
S311 to step S313 through the above steps obtains the storage obtained through the vector index method that GPU executes processing
In vector subset in video memory, first part's similar features vector index result of object is retrieved.
Step S321 utilizes central point index file (such as the central point of the center point set B pre-established in CPU
Index B), determine the N2 central point vector that distance retrieval object is nearest in the point set B of center, what N2 was specified is more than or equal to 1
Integer.
Step S323, for each of N2 central point vector central point vector, according to the central point pre-established
The vector index file (such as vector index B) of set B in the class cluster belonging to central point vector, passes through reverse index, determines
The central point vector of most like specified number with retrieval object, as the retrieval knot in class cluster belonging to each central point vector
Fruit.
In this step, the search result in class cluster belonging to each central point vector is summarized, obtains CPU processing
Obtained second part similarity vector search result, according to retrieval each of object and second part similarity vector search result
The order of the similarity of similarity vector from big to small is ranked up, and obtains K2 top ranked similarity vector of similarity, and K2 is
The specified integer for being more than or equal to 1.
S321 to step S323 through the above steps obtains the storage obtained through the vector index method that CPU executes processing
In vector subset in memory, the second part similar features vector index result of object is retrieved.
In the embodiment of this specification, N2 may be the same or different with above-mentioned N1；K2 can phase with above-mentioned K1
Together, it can also be different.
Step S330, from high to low according to the similarity degree with retrieval object, the first part handled GPU is similar
Vector index is as a result, the second part similarity vector search result handled with CPU is ranked up, and extracts similarity degree row
The vector of the preceding specified number of name, the search result as retrieval object.
According to the vector index method of this specification, CPU and GPU synergistic application is in the vector index process of data object
In, to improve using cpu resource and GPU resource and promote retrieval performance and recall precision.
Fig. 4 shows the flow diagram of the vector index method of this specification another embodiment.As shown in figure 4, one
In a embodiment, vector index method 400 may include:
Step S410 extracts the feature vector of retrieval object.
Step S420 obtains similar with feature vector vector from graphics processor GPU, as first part it is similar to
Amount.
Step S430 obtains similar with feature vector vector from central processor CPU, as second part it is similar to
Amount.
Step S440 determines the retrieval knot of retrieval object according to first part's similarity vector and second part similarity vector
Fruit.
In one embodiment, step S420 can specifically include:
Step S422, selects one or more first center vectors from GPU, in one or more first center vectors
Each first center vector and feature vector meet the first default similarity condition.
Step S424, in the vector grouping where each first center vector, selection first part's similarity vector, first
Part similarity vector and feature vector meet the second default similarity condition.
In one embodiment, one or more first center vectors are by retrieving first row's of the falling rope pre-established
For argument according to obtained vector, the first inverted index data include the index address of each first center vector.
In one embodiment, first part's similarity vector is obtained by retrieving the first index data pre-established
Vector, the first index data includes the index address of vector in vector grouping where each first center vector, wherein every
Index address in a vector grouping between vector is different.
In one embodiment, step S430 can specifically include:
Step S432, selects one or more second center vectors from CPU, in one or more second center vectors
Each second center vector meets third and presets similarity condition.
Step S434, in the vector grouping where each second center vector, selection second part similarity vector, second
Part similarity vector and feature vector meet the 4th default similarity condition.
In one embodiment, one or more second center vectors are by retrieving second row's of the falling rope pre-established
For argument according to obtained vector, the second inverted index data include the index address of each second center vector.
In one embodiment, second part similarity vector is obtained by retrieving the second index data pre-established
Vector, the second index data includes the index address of vector in vector grouping where each second center vector, wherein every
Index address in a vector grouping between vector is different.
In one embodiment, the search result of object is retrieved, including from high to low according to similarity, from first part's phase
Like the vector for the specified number that vector sum second part similarity vector selects.
According to the vector index method of this specification, CPU and GPU synergistic application is in the vector index process of data object
In, to improve using cpu resource and GPU resource and promote retrieval performance and recall precision.
In one embodiment, first part's similarity vector is obtained from the pre-loaded primary vector subset of GPU
Vector similar with feature vector；Second part similarity vector, be obtained from the pre-loaded secondary vector subset of CPU with
The similar vector of feature vector；Primary vector subset and secondary vector subset are to carry out cutting to specified feature vector set
It obtains.
In one embodiment, the cutting carried out to specified feature vector set is size and video memory size based on memory
The cutting of progress.
According to the vector index method of this specification, opposite direction duration set carries out cutting in advance, memory and video memory is stored in, rear
It is continuous to retrieve vector stored in memory using CPU, and using the vector stored in GPU retrieval video memory for retrieval object, it can
To utilize cpu resource and GPU resource, improves and promote retrieval performance and recall precision.
Fig. 5 shows the flow chart of the vector index device according to this specification one embodiment.As shown in figure 5, one
In a embodiment, vector index device 500 may include:
Vector extraction module 510, for extracting the feature vector of retrieval object.
First retrieval module 520, for obtaining similar with feature vector vector from graphics processor GPU, as the
A part of similarity vector.
Second retrieval module 530, for obtaining similar with feature vector vector from central processor CPU, as the
Two part similarity vectors.
As a result determining module 540, for determining retrieval pair according to first part's similarity vector and second part similarity vector
The search result of elephant.
In one embodiment, the first retrieval module 520 is specifically used for:
One or more first center vectors, each of one or more first center vectors first are selected from GPU
Center vector and feature vector meet the first default similarity condition；
In the vector grouping where each first center vector, first part's similarity vector is selected, first part is similar
Vector and feature vector meet the second default similarity condition.
In one embodiment, one or more first center vectors are by retrieving first row's of the falling rope pre-established
For argument according to obtained vector, the first inverted index data include the index address of each first center vector.
In one embodiment, first part's similarity vector is obtained by retrieving the first index data pre-established
Vector, the first index data includes the index address of vector in vector grouping where each first center vector, wherein every
Index address in a vector grouping between vector is different.
In one embodiment, the second retrieval module 530 is specifically used for:
One or more second center vectors, each of one or more second center vectors second are selected from CPU
Center vector meets third and presets similarity condition；In the vector grouping where each second center vector, second is selected
Divide similarity vector, second part similarity vector and feature vector meet the 4th default similarity condition.
In one embodiment, one or more second center vectors are by retrieving second row's of the falling rope pre-established
For argument according to obtained vector, the second inverted index data include the index address of each second center vector.
In one embodiment, second part similarity vector is obtained by retrieving the second index data pre-established
Vector, the second index data includes the index address of vector in vector grouping where each second center vector, wherein every
Index address in a vector grouping between vector is different.
In one embodiment, retrieve object search result include according to similarity from high to low, from first part's phase
Like the vector for the specified number that vector sum second part similarity vector selects.
In one embodiment, first part's similarity vector is obtained from the pre-loaded primary vector subset of GPU
Vector similar with feature vector；Second part similarity vector, be obtained from the pre-loaded secondary vector subset of CPU with
The similar vector of feature vector；Primary vector subset and secondary vector subset are to carry out cutting to specified feature vector set
It obtains.
In one embodiment, the cutting carried out to specified feature vector set is size and video memory size based on memory
The cutting of progress.
According to the vector index method of this specification, the vector index process of CPU and GPU collaboration processing data object, benefit
With cpu resource and GPU resource, improves and promote retrieval performance and recall precision.And it is possible to which opposite direction duration set is cut in advance
Point, it is stored in memory and video memory, subsequent for retrieval object, retrieves vector stored in memory using CPU, and utilize GPU
The vector stored in retrieval video memory, can use cpu resource and GPU resource, improves and promotes retrieval performance and recall precision.
Fig. 6 is the structure for showing the exemplary hardware architecture that can be realized the calculating equipment according to this specification embodiment
Figure.As shown in fig. 6, calculate equipment 600 include input equipment 601, input interface 602, central processing unit 603, memory 604,
Output interface 605 and output equipment 606.Wherein, input interface 602, central processing unit 603, memory 604 and output
Interface 605 is connected with each other by bus 610, and input equipment 601 and output equipment 606 pass through input interface 602 and output respectively
Interface 605 is connect with bus 610, and then is connect with the other assemblies for calculating equipment 600.Specifically, input equipment 601, which receives, comes
Central processing unit 603 is transmitted to by information is inputted from external input information, and by input interface 602；Central processing unit 603
Input information is handled to generate output information based on the computer executable instructions stored in memory 604, will be exported
Information temporarily or permanently stores in the memory 604, and output information is then transmitted to output by output interface 605 and is set
Standby 606；Output information is output to the outside of calculating equipment 600 for users to use by output equipment 606.
In one embodiment, calculating equipment 600 shown in fig. 6 may be implemented as a kind of vector index equipment, this to
Measuring retrieval facility may include: memory, be configured as storage program；Processor is configured as storing in run memory
Program, to execute the vector index method of above-described embodiment description.
The apparatus embodiments described above are merely exemplary, wherein described, unit can as illustrated by the separation member
It is physically separated with being or may not be, component shown as a unit may or may not be physics list
Member, it can it is in one place, or may be distributed over multiple network units.It can be selected according to the actual needs
In some or all of the modules achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not paying creativeness
Labour in the case where, it can understand and implement.
It is above-mentioned that this specification specific embodiment is described.Other embodiments are in the scope of the appended claims
It is interior.In some cases, the movement recorded in detail in the claims or step can be come according to the sequence being different from embodiment
It executes and desired result still may be implemented.In addition, process depicted in the drawing not necessarily require show it is specific suitable
Sequence or consecutive order are just able to achieve desired result.In some embodiments, multitasking and parallel processing be also can
With or may be advantageous.
Claims (17)
1. a kind of vector index method, comprising:
Extract the feature vector of retrieval object；
Vector similar with described eigenvector is obtained from graphics processor GPU, as first part's similarity vector；
Vector similar with described eigenvector is obtained from central processor CPU, as second part similarity vector；
According to first part's similarity vector and the second part similarity vector, the retrieval knot of the retrieval object is determined
Fruit.
2. vector index method according to claim 1, wherein described to be obtained and the spy from graphics processor GPU
The similar vector of vector is levied, as first part's similarity vector, comprising:
One or more first center vectors, each of one or more of first center vectors are selected from the GPU
First center vector and described eigenvector meet the first default similarity condition；
In the vector grouping where each first center vector, first part's similarity vector is selected, described first
Part similarity vector and described eigenvector meet the second default similarity condition.
3. vector index method according to claim 2, wherein
One or more of first center vectors, be by retrieve that the first inverted index data for pre-establishing obtain to
Amount, the first inverted index data include the index address of each first center vector；
First part's similarity vector, is the vector obtained by retrieving the first index data for pre-establishing, described first
Index data includes the index address of vector in vector grouping where each first center vector, wherein described each
Index address in vector grouping between vector is different.
4. vector index method according to claim 1, wherein described to be obtained and the spy from central processor CPU
The similar vector of vector is levied, as second part similarity vector, comprising:
One or more second center vectors, each of one or more of second center vectors are selected from the CPU
Second center vector meets third and presets similarity condition；
In the vector grouping where each second center vector, the second part similarity vector is selected, described second
Part similarity vector and described eigenvector meet the 4th default similarity condition.
5. vector index method according to claim 4, wherein
One or more of second center vectors, be by retrieve that the second inverted index data for pre-establishing obtain to
Amount, the second inverted index data include the index address of each second center vector；
The second part similarity vector, is the vector obtained by retrieving the second index data for pre-establishing, described second
Index data includes the index address of vector in vector grouping where each second center vector, wherein described each
Index address in vector grouping between vector is different.
6. vector index method according to claim 1, wherein
The search result of the retrieval object, including from high to low according to similarity, from first part's similarity vector and institute
State the vector of the specified number of second part similarity vector selection.
7. vector index method according to claim 1, wherein
First part's similarity vector is obtaining with the feature from the pre-loaded primary vector subset of the GPU
The similar vector of vector；
The second part similarity vector is obtaining with the feature from the pre-loaded secondary vector subset of the CPU
The similar vector of vector；
The primary vector subset and the secondary vector subset are to carry out cutting to specified feature vector set to obtain.
8. vector index method according to claim 7, wherein
The cutting is the cutting carried out based on the corresponding memory size of the CPU and the corresponding video memory size of the GPU.
9. a kind of vector index device, which is characterized in that the vector index device includes:
Vector extraction module, for extracting the feature vector of retrieval object；
First retrieval module, for obtaining vector similar with described eigenvector from graphics processor GPU, as first
Divide similarity vector；
Second retrieval module, for obtaining vector similar with described eigenvector from central processor CPU, as second
Divide similarity vector；
As a result determining module, described in determining according to first part's similarity vector and the second part similarity vector
Retrieve the search result of object.
10. vector index device according to claim 9, wherein first retrieval module is specifically used for:
One or more first center vectors, each of one or more of first center vectors are selected from the GPU
First center vector and described eigenvector meet the first default similarity condition；
In the vector grouping where each first center vector, first part's similarity vector is selected, described first
Part similarity vector and described eigenvector meet the second default similarity condition.
11. vector index device according to claim 10, wherein
One or more of first center vectors, be by retrieve that the first inverted index data for pre-establishing obtain to
Amount, the first inverted index data include the index address of each first center vector；
First part's similarity vector, is the vector obtained by retrieving the first index data for pre-establishing, described first
Index data includes the index address of vector in vector grouping where each first center vector, wherein described each
Index address in vector grouping between vector is different.
12. vector index device according to claim 9, wherein second retrieval module is specifically used for:
One or more second center vectors, each of one or more of second center vectors are selected from the CPU
Second center vector meets third and presets similarity condition；
In the vector grouping where each second center vector, the second part similarity vector is selected, described second
Part similarity vector and described eigenvector meet the 4th default similarity condition.
13. vector index device according to claim 12, wherein
One or more of second center vectors, be by retrieve that the second inverted index data for pre-establishing obtain to
Amount, the second inverted index data include the index address of each second center vector；
The second part similarity vector, is the vector obtained by retrieving the second index data for pre-establishing, described second
Index data includes the index address of vector in vector grouping where each second center vector, wherein described each
Index address in vector grouping between vector is different.
14. vector index device according to claim 9, wherein
The search result of the retrieval object, including from high to low according to similarity, from first part's similarity vector and institute
State the vector of the specified number of second part similarity vector selection.
15. vector index device according to claim 9, wherein
First part's similarity vector is obtaining with the feature from the pre-loaded primary vector subset of the GPU
The similar vector of vector；
The second part similarity vector is obtaining with the feature from the pre-loaded secondary vector subset of the CPU
The similar vector of vector；
The primary vector subset and the secondary vector subset are to carry out cutting to specified feature vector set to obtain.
16. the vector index device according to weighing and require 15, wherein
The cutting is the cutting carried out based on the corresponding memory size of the CPU and the corresponding video memory size of the GPU.
17. a kind of vector index equipment, which is characterized in that including memory and processor；
The memory is for storing executable program code；
The processor is used to read the executable program code stored in the memory and requires to appoint in 1 to 8 with perform claim
Vector index method described in one.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910490118.XA CN110209895B (en) | 2019-06-06 | 2019-06-06 | Vector retrieval method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910490118.XA CN110209895B (en) | 2019-06-06 | 2019-06-06 | Vector retrieval method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110209895A true CN110209895A (en) | 2019-09-06 |
| CN110209895B CN110209895B (en) | 2023-09-05 |
Family
ID=67791317
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910490118.XA Active CN110209895B (en) | 2019-06-06 | 2019-06-06 | Vector retrieval method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209895B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110781247A (en) * | 2019-09-23 | 2020-02-11 | 华为技术有限公司 | Vector clustering method, device and storage medium |
| CN111309984A (en) * | 2020-03-10 | 2020-06-19 | 支付宝(杭州)信息技术有限公司 | Method and device for searching node vector from database by using index |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017012491A1 (en) * | 2015-07-23 | 2017-01-26 | 北京京东尚科信息技术有限公司 | Similarity comparison method and apparatus for high-dimensional image features |
| CN108229289A (en) * | 2017-06-27 | 2018-06-29 | 北京市商汤科技开发有限公司 | Target retrieval method, apparatus and electronic equipment |
| CN108241745A (en) * | 2018-01-08 | 2018-07-03 | 阿里巴巴集团控股有限公司 | The processing method and processing device of sample set, the querying method of sample and device |
| CN109165307A (en) * | 2018-09-19 | 2019-01-08 | 腾讯科技（深圳）有限公司 | A kind of characteristic key method, apparatus and storage medium |
-
2019
- 2019-06-06 CN CN201910490118.XA patent/CN110209895B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017012491A1 (en) * | 2015-07-23 | 2017-01-26 | 北京京东尚科信息技术有限公司 | Similarity comparison method and apparatus for high-dimensional image features |
| CN108229289A (en) * | 2017-06-27 | 2018-06-29 | 北京市商汤科技开发有限公司 | Target retrieval method, apparatus and electronic equipment |
| CN108241745A (en) * | 2018-01-08 | 2018-07-03 | 阿里巴巴集团控股有限公司 | The processing method and processing device of sample set, the querying method of sample and device |
| CN109165307A (en) * | 2018-09-19 | 2019-01-08 | 腾讯科技（深圳）有限公司 | A kind of characteristic key method, apparatus and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 王立等: "Hadoop分布式的海量图像检索", 《现代电子技术》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110781247A (en) * | 2019-09-23 | 2020-02-11 | 华为技术有限公司 | Vector clustering method, device and storage medium |
| CN110781247B (en) * | 2019-09-23 | 2021-11-26 | 华为技术有限公司 | Vector clustering method, device and storage medium |
| CN111309984A (en) * | 2020-03-10 | 2020-06-19 | 支付宝(杭州)信息技术有限公司 | Method and device for searching node vector from database by using index |
| CN111309984B (en) * | 2020-03-10 | 2023-09-05 | 支付宝(杭州)信息技术有限公司 | Method and device for retrieving node vector from database by index |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110209895B (en) | 2023-09-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108804641B (en) | Text similarity calculation method, device, equipment and storage medium | |
| Pelevina et al. | Making sense of word embeddings | |
| CN107436875B (en) | Text classification method and device | |
| US11899681B2 (en) | Knowledge graph building method, electronic apparatus and non-transitory computer readable storage medium | |
| CN107885874A (en) | Data query method and apparatus, computer equipment and computer-readable recording medium | |
| EP3940555A2 (en) | Method and apparatus of processing information, method and apparatus of recommending information, electronic device, and storage medium | |
| WO2018086401A1 (en) | Cluster processing method and device for questions in automatic question and answering system | |
| CN109101479A (en) | A kind of clustering method and device for Chinese sentence | |
| CN104731861B (en) | Multi-medium data method for pushing and device | |
| CN110442718A (en) | Sentence processing method, device and server and storage medium | |
| CN110929161A (en) | Large-scale user-oriented personalized teaching resource recommendation method | |
| CN109933660A (en) | The API information search method based on handout and Stack Overflow towards natural language form | |
| CN108288208A (en) | The displaying object of image content-based determines method, apparatus, medium and equipment | |
| CN110472043A (en) | A kind of clustering method and device for comment text | |
| CN110795613B (en) | Commodity searching method, device and system and electronic equipment | |
| CN114741581A (en) | Image classification method and device, computer equipment and medium | |
| WO2023020214A1 (en) | Retrieval model training method and apparatus, retrieval method and apparatus, device and medium | |
| CN110209895A (en) | Vector index method, apparatus and equipment | |
| CN111160699A (en) | Expert recommendation method and system | |
| CN114490923A (en) | Training method, device and equipment for similar text matching model and storage medium | |
| CN114328800A (en) | Text processing method and device, electronic equipment and computer readable storage medium | |
| CN110209780A (en) | A kind of question template generation method, device, server and storage medium | |
| CN113761017A (en) | Similarity searching method and device | |
| CN113569018A (en) | Question and answer pair mining method and device | |
| CN117435685A (en) | Document retrieval method, document retrieval device, computer equipment, storage medium and product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200924 Address after: Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200924 Address after: Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: Grand Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| TA01 | Transfer of patent application right | ||
| CB02 | Change of applicant information |
Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Grand Cayman Islands Applicant before: Innovative advanced technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |