CN116523024B

CN116523024B - Training method, device, equipment and storage medium of recall model

Info

Publication number: CN116523024B
Application number: CN202310804526.4A
Authority: CN
Inventors: 姚丽丽
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-07-03
Filing date: 2023-07-03
Publication date: 2023-10-13
Anticipated expiration: 2043-07-03
Also published as: CN116523024A

Abstract

A training method, device, equipment and storage medium for recall model belong to the technical field of artificial intelligence. The method comprises the following steps: acquiring query information and at least one positive case query result corresponding to the query information; extracting feature vectors of query information by adopting a first feature extraction network; according to the feature vector of the query information and the feature vector corresponding to each candidate query result, k query results with similarity with the query information are selected from the candidate query results; selecting at least one query result from k query results as a negative case query result corresponding to the query information; and adjusting parameters of the recall model based on the positive case query result and the negative case query result corresponding to the query information to obtain the recall model after training. The method realizes the automatic determination of the difficult negative case query result overlapped with the query information, and is beneficial to improving the efficiency of acquiring the difficult negative case query result.

Description

Training method, device, equipment and storage medium of recall model

Technical Field

The application relates to the technical field of artificial intelligence, in particular to a training method, device and equipment for a recall model and a storage medium.

Background

The recall model is a widely used machine learning model in the field of search recall, and can find at least one query result matched with query information from a large number of query results.

In the related technology, the training process of the recall model needs to use a positive case query result matched with the query information and a negative case query result not matched with the query information; the negative case query results comprise difficult negative case query results, and the difficult negative case query results and the query information are overlapped, but the contents of the difficult negative case query results do not meet the query information, compared with the simple negative case query results which can be directly generated according to the requirements determined by the click times and the like, the difficult negative case query information is often manually marked from the massive candidate query results.

However, the method has low efficiency of obtaining the difficult negative case query results, and the number of the difficult negative case query results is limited, so that the training effect of the recall model is limited.

Disclosure of Invention

The application provides a training method, device and equipment for a recall model and a storage medium. The technical scheme is as follows:

according to an aspect of an embodiment of the present application, there is provided a training method of a recall model including a first feature extraction network and a second feature extraction network; the method comprises the following steps:

Acquiring query information and at least one positive case query result corresponding to the query information, wherein the positive case query result refers to a query result matched with the query information;

extracting feature vectors of the query information by adopting the first feature extraction network;

according to the feature vector of the query information and the feature vector corresponding to each candidate query result, k query results with similarity with the query information are selected from the candidate query results; the feature vector corresponding to the candidate query result is obtained by adopting the second feature extraction network, and k is a positive integer;

selecting at least one query result from the k query results as a negative case query result corresponding to the query information, wherein the negative case query result refers to a query result which is not matched with the query information;

and adjusting parameters of the recall model based on the positive case query result and the negative case query result corresponding to the query information to obtain the recall model after training.

According to an aspect of an embodiment of the present application, there is provided a training apparatus of a recall model including a first feature extraction network and a second feature extraction network; the device comprises:

The system comprises a positive case acquisition module, a positive case analysis module and a positive case analysis module, wherein the positive case acquisition module is used for acquiring query information and at least one positive case query result corresponding to the query information, and the positive case query result refers to a query result matched with the query information;

the feature extraction module is used for extracting feature vectors of the query information by adopting the first feature extraction network;

the result determining module is used for selecting k query results with similarity with the query information from the query results of each candidate according to the feature vector of the query information and the feature vector corresponding to the query result of each candidate; the feature vector corresponding to the candidate query result is obtained by adopting the second feature extraction network, and k is a positive integer;

the negative example acquisition module is used for selecting at least one query result from the k query results as a negative example query result corresponding to the query information, wherein the negative example query result refers to a query result which is not matched with the query information;

and the model training module is used for adjusting parameters of the recall model based on the positive case query result and the negative case query result corresponding to the query information to obtain a trained recall model.

According to an aspect of an embodiment of the present application, there is provided a computer device including a processor and a memory, the memory having stored therein a computer program that is loaded and executed by the processor to implement the training method of the recall model as described above.

According to an aspect of an embodiment of the present application, there is provided a computer-readable storage medium having stored therein a computer program loaded and executed by a processor to implement the training method of the recall model as described above.

According to an aspect of an embodiment of the present application, there is provided a computer program product including a computer program stored in a computer-readable storage medium, from which a processor reads and executes the computer program to implement the training method of the recall model as described above.

The technical scheme provided by the embodiment of the application has the beneficial effects that at least:

in the training process of the recall model, at least one difficult negative example query result corresponding to the query information is determined according to the similarity between the feature vector of the query information and the feature vector of the candidate query result, and compared with the prior art, the difficult negative example query result corresponding to the query information is required to be manually marked, so that the determination efficiency of the difficult negative example query result is improved. More difficult negative case query results are provided for the training process of the recall model, so that the recall model is helped to learn differences between query information and negative case query results, the identification capability of the recall model on the difficult negative case query results is improved, and the recall accuracy of the recall model after training is improved.

Drawings

FIG. 1 is a schematic illustration of an implementation environment for an embodiment of the present application;

FIG. 2 is a flow chart of a training method for recall models provided by one exemplary embodiment of the present application;

FIG. 3 is a schematic diagram of a search bar provided by an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram of query result selection provided by an exemplary embodiment of the present application;

FIG. 5 is a schematic diagram of a method for determining query results corresponding to query information according to an exemplary embodiment of the present application;

FIG. 6 is a block diagram of a training apparatus for recall models provided in one exemplary embodiment of the present application;

fig. 7 is a block diagram of a computer device according to an exemplary embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.

Artificial intelligence (Artificial Intelligence, AI): the system is a theory, a method, a technology and an application system which simulate, extend and extend human intelligence by using a digital computer or a machine controlled by the digital computer, sense environment, acquire knowledge and acquire an optimal result by using the knowledge.

The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

Natural language processing (Nature Language Processing, NLP): is an important direction in the fields of computer science and artificial intelligence. It is studying various theories and methods that enable effective communication between a person and a computer in natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Thus, the research in this field will involve natural language, i.e. language that people use daily, so it has a close relationship with the research in linguistics. Natural language processing techniques typically include text processing, semantic understanding, machine translation, robotic questions and answers, knowledge graph techniques, and the like.

Bi-directional encoder representation based on transgrames (Bidirectional Encoder Representation from Transformers, BERT): the method is based on the NLP pre-training technology, can represent the input query information into a feature vector with context information, and is better used for calculating the downstream task. BERT is typically composed of a superposition of layers of transducer structures, with the ability to perceive context information through a cross-attention (cross-attention) mechanism.

Computer Vision technology (CV): the method is a science for researching how to make the machine "look at", and further means that a camera and a computer are used to replace human eyes to recognize and measure targets and other machine vision, and further graphic processing is performed, so that the computer is used to process images which are more suitable for human eyes to observe or transmit to an instrument to detect. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, video content/behavior recognition, virtual reality, augmented reality, and the like.

Machine Learning (ML): is a multi-domain interdisciplinary, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like.

Vector recall: refers to the process of recalling a query result having similarity with the feature vector of the query information according to the feature vector of the query information. Taking vector recall in a search engine as an example: the search engine can determine and store the feature vectors corresponding to all displayable query results in advance so as to input the query information in the search box by the user, can quickly calculate the feature vectors of the query information, and returns the first Y query results closest to the query information, wherein Y is a positive integer.

Along with research and progress of artificial intelligence technology, the artificial intelligence technology is developed for research and application in multiple fields, for example, in the field of vector recall, similarity between query information and massive existing query results is measured through a machine learning model, query results which are relatively similar to the query information are found, and recall effects of vector recall are improved. Artificial intelligence technology will find application in more fields and will find increasing importance.

Positive example query results: refers to the query result matching the query information. And determining through clicking conditions which can be respectively corresponding to at least one query result corresponding to certain query information.

Simple negative example: it means that the query information is completely mismatched with the query result, and it is assumed that n training samples, i.e., < q1, t1>, < q2, t2>, …, < qn, tn >, are included in a training round (batch) of a recall model. Because each query information is not matched with the positive case query results corresponding to other query information, that is, the positive case query results corresponding to other query information can be used as the negative case query results of the query information, n-1 negative case query results corresponding to the query information can be constructed based on the positive case query results, for example, for query information q1, one positive case query result t1, and n-1 negative case query results are corresponding to the query information q1, q 1: t2, t3, …, tn; the simple negative case query results corresponding to other query information can also be obtained in this way, and the n-1 negative case query results are simple negative case query results, which are simply referred to as "simple negative case".

Recall accuracy: the method is used for representing the matching degree between the query result determined by the recall model according to the query information and the query information. Recall accuracy may also be referred to as recall accuracy. Optionally, the recall accuracy can be determined according to the click condition of the user on a plurality of historical query results corresponding to the historical query information, if the historical query results with higher similarity of the historical query information are clicked less frequently; or after the user inputs the first query information in the search bar, if the user inputs the second query information similar to the first query information in the search bar in a short time, the matching degree of the historical query result corresponding to the first query information and the first query information is lower. If the server counts and determines that the number of clicks of the historical query result is small and a large number of users input the first query information and the second query information in a short time in unit time, the recall accuracy of the recall model is poor, and the recall accuracy of the recall model may need to be improved.

FIG. 1 is a schematic diagram of an implementation environment for an embodiment of the present application. The implementation environment of the scheme can comprise: computer device 10, terminal device 20, and server 30.

The computer device 10 includes, but is not limited to, a personal computer (Personal Computer, PC), tablet, cell phone, wearable device, smart home appliance, vehicle terminal, etc. electronic devices with computing and memory capabilities. In some embodiments, the computer device is configured to train the recall model to obtain a trained recall model. Illustratively, the recall model includes at least the following two parts: a first feature extraction network and a second feature extraction network. In the training process of the recall model, a first feature extraction network is used for extracting features of query information to generate feature vectors of the query information; the second feature extraction network is used for extracting features of the query result to obtain a feature vector for generating the query result.

In the method, the negative example sample pair (the negative example sample pair comprises a query information and a negative example sample corresponding to the query information) used in the training process of the recall model is generated in real time in the training process, the computer equipment 10 performs at least one round of training on the recall model until the loss function value of the recall model tends to converge after a certain training round, the computer equipment 10 stops training on the recall model to obtain a trained recall model, and reference is made to the following examples for specific steps of the training process of the recall model.

The terminal device 20 may be an electronic device such as a personal computer, tablet computer, cell phone, wearable device, smart home appliance, vehicle terminal, etc. A client with a target application running on the terminal device 20. The target application searches for recall functionality (which may be understood to be query functionality). And the search recall function is used for acquiring the query information input by the user and feeding back a query result corresponding to the query information to the user. For example, the target application is used to provide a search recall function for articles, the target application is used to determine articles corresponding to the query information according to the query information, and the trained recall model obtained by the computer device 10 is needed in determining the search recall process. The trained recall model is used to determine at least one query result that is similar to the query information.

In addition, the target application program may also be a news application program, a shopping application program, a social application program, an interactive entertainment application program, a browser application program, a content sharing application program, a virtual reality application program, an augmented reality application program, and the like, which is not limited by the embodiment of the present application. In addition, for different application programs, the types of the query information and the query result processed by the application programs may be different, and the corresponding functions may also be different, which may be configured in advance according to actual requirements, which is not limited by the embodiment of the present application.

The server 30 is used to provide background services for clients of the target application in the terminal device 20. For example, the server 30 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, secure service content distribution network, (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platform, but is not limited thereto. The server 30 has at least a data receiving function and a calculating function. Alternatively, the computer device 10 is mounted in the server 30, or the computer device 10 is another device different from the server 30.

In one example, computer device 10 sends the trained recall model to server 30, terminal device 20 sends query information to server 30, and server 30 determines a query result corresponding to the query information from the query information based on the trained recall model. The server 30 feeds back the query result corresponding to the query information to the terminal device 20.

FIG. 2 is a flowchart of a training method for a recall model provided by one exemplary embodiment of the present application. The recall model comprises a first feature extraction network and a second feature extraction network; illustratively, the execution subject of the method may be the computer device 10 in fig. 1, and the training method of the recall model will be described below with the computer device as the execution subject. As shown in fig. 2, the method may include the following steps (210-250):

step 210, obtaining query information and at least one positive case query result corresponding to the query information, where the positive case query result is a query result matched with the query information.

In some embodiments, the query information refers to the content to be queried. The query information is used to carry the search intent of the user, and the query information can be understood as a question from the user. Taking recall information corresponding to a search engine as an example, query information refers to information input into a search bar in the search engine by a user, for example, text information in the search bar, or pictures uploaded or shot by the user in an information input interface, and the like.

FIG. 3 is a schematic diagram of a search bar provided by an exemplary embodiment of the present application. Search bar 310 in fig. 3 is capable of receiving query information entered by a user.

Optionally, the form of the query information includes at least one of: text, picture, audio; wherein the text-form query information semantically characterizes the user's query search purpose, and the audio-form query information includes speech for representing the query search purpose.

In some embodiments, query results refer to results from conducting a search query based on query information. The form of the query results includes at least one of: text, pictures, audio, video, etc. The form of the query result is set according to the function of the actual target application, and the present application is not limited herein. In some embodiments, the types of query results include, but are not limited to: articles, web pages, video, audio, merchandise pages, applications, communities, etc. Optionally, the target application supports setting the type of query result. Illustratively, the query result type selection option is included in the user interface of the target application.

FIG. 4 is a schematic diagram of query result selection provided by an exemplary embodiment of the present application. The article option 420 in fig. 4 can filter the type of query results.

In the case that the query result is in the text form, the query result corresponds to web page content, articles, and the like. Optionally, the query result is key information in the article and the webpage content. Illustratively, taking an article as an example, the type of key information includes at least one of: an article title (title), a collection of section titles in the article, and a key position of the article; wherein, the key positions of the article include: the beginning paragraph of the article, the end paragraph of the article, etc.

The key positions are used as the query results, so that the data volume of the query results is reduced, the processing difficulty of the query results is reduced in the training process of the recall model, the speed of determining the similarity between the query information and massive selectable query results is improved, the calculation pressure of the search recall process is reduced, and the recall speed is improved.

The query result matched with the query information means that the content of the query result is related to the intention of the query information, and the query result matched with the query information can be understood as follows: the query results can solve the questioning of the query information. Optionally, the similarity between the positive case query result corresponding to the query information and the query information is higher. The measure of similarity between the positive query results corresponding to the query information includes, but is not limited to, at least one of: the inner product between the feature vectors corresponding to the two features and the distance between the two features in the spatial distribution.

In some embodiments, during the training of the recall model, training data is formed by the query information and the query results, i.e., the training data includes the query information and the query results corresponding to the query information. Optionally, the training data used in the recall model training process belongs to binary data (Pairwise). For example, for query information (query), the training data may be expressed as < query, title >, for the query result (title) to which the query information corresponds.

Optionally, the training data is divided into two types, namely positive training data and negative training data; the positive example training data includes query information and positive example query results corresponding to the query information, the negative example training data includes query information and negative example query results corresponding to the query information, the negative example query results refer to query results not matched with the query information, and specific content of the negative example query results is referred to in the following examples.

For a certain query information, the similarity between the positive case query result and the query information is higher than that between the negative case query result and the query information compared with the negative case query result corresponding to the query information. That is, the matching degree between the positive case query result and the query information is better, the matching degree between the negative case query result and the query information is poorer, or the negative case query result and the query information are not matched.

In some embodiments, for the training process of the recall model provided by the application, the positive case query result corresponding to the query information can be preformed before the training of the recall model is started, and the negative case query result corresponding to the query information can be dynamically generated in the training process of the recall model. Optionally, in different training rounds, the same query information corresponds to different negative case query results which are not identical.

For determining the negative example query result, please refer to the following embodiments, and the method for determining the positive example query result is initially described below.

In some embodiments, the query information used by the training process of the recall model is historical query information. Historical query information may be understood as query information submitted by a user of a target application over a period of time.

Optionally, after the user sends the historical query information to the server through the client, the server acquires the historical query information and carries out search recall according to the historical query information to obtain a query result corresponding to the historical query information; and the server returns the historical query result corresponding to the historical query information to the terminal equipment.

For example, the server determines at least one historical query result corresponding to the historical query information according to the historical query information, feeds back an exposure list corresponding to the historical query information to the terminal equipment, and generates a click log according to the click condition of the at least one historical query result in the exposure list; the computer equipment determines query information which is used in the training process by using at least one piece of historical query information according to the historical query information, the exposure list and the click log, and at least one positive case query result corresponding to the query information.

It should be noted that, the server uses the click log to count the number of clicks of at least one target user on the historical query result, and the click log does not include the target user information. The training data used by the application are all collected and counted according to the data safety regulation after the target user agrees.

Illustratively, the computer device determines whether to use the historical query information as query information used in the recall model training process based on how frequently the server received the historical query information per unit time. If the occurrence frequency of a certain historical query message is higher than the frequency threshold, the computer equipment takes the historical query message as the query message used in the recall model training process; if the occurrence frequency of the historical query information is lower than the frequency threshold value, the historical query information is not selected as the query information used in the recall model training process. For example, the frequency threshold value is set according to actual needs, and the frequency threshold value is set to be 100 times per day.

In some embodiments, the exposure list refers to: responding to search query actions of a user, and returning a result display list to the terminal equipment by the server, wherein the result display list comprises at least one query result corresponding to historical query information; the click log is used for recording the click condition of a target user on at least one historical query result under the same historical query information, and the target user refers to the user who proposes the same historical query information.

Optionally, the click condition is characterized by at least one of: the number of clicks, the click rate, and the click frequency; wherein, the clicking times refer to the total clicking number of at least one target user on the same historical query result; the click rate refers to the ratio between the total number of clicks of at least one target user on a certain historical query result and the total number of clicks of all historical query results corresponding to the historical query information, and the click frequency is used for representing the minimum time interval or the average time interval between two adjacent clicks of the same historical query result. For a certain historical query data, the computer device uses the historical query result with the click condition being greater than or equal to a first threshold value as a positive query result of the historical query data, the first threshold value is a positive integer, the value of the first threshold value is determined according to the actual result, and the application is not limited herein.

In some embodiments, the query information corresponds to a plurality of positive query results, and each positive query result and the query information may form a positive training data. For example, if the query information q1 corresponds to the positive case query results t1, t2, and t3, the positive case training data including the query information q1 includes < q1, t1>, < q1, t2> and < q1, t3>, and if the query information q2 corresponds to the positive case query results t4, t5, the positive case training data including the query information q2 includes < q2, t4> and < q2, t5>, and q1 is different from q 2.

In some embodiments, the recall model belongs to a dual-tower model, the first feature extraction network is used for feature extraction of the query information, and the second feature extraction network is used for feature extraction of the query result respectively. Optionally, the first feature extraction network and the second feature extraction network have similar network structures, and network parameters of the first feature extraction network and the second feature extraction network are different.

Illustratively, the first feature extraction network and the second feature extraction network are both BERT networks, transformer networks designed based on an attention mechanism.

Optionally, before the training method of the recall model provided by the application is performed, the recall model is a recall model in use in a target application program, and a first feature extraction network and a second feature extraction network in the recall model have the capability of extracting features.

Step 220, extracting feature vectors of the query information by using the first feature extraction network.

The following describes a process of determining feature vectors of query information, taking the query information in text form as an example. Optionally, for query information in text form, feature vectors of the query information are used to characterize semantic features of the query information.

In some embodiments, the computer device employs a first feature extraction network to extract feature vectors of query information, comprising: performing word segmentation processing on the query information to obtain at least one word segmentation unit; generating embedded representations of query information according to the embedded representations respectively corresponding to the at least one word segmentation unit; inputting the embedded representation of the query information into a first feature extraction network; and carrying out feature extraction on the embedded representation of the query information through a first feature extraction network, and taking a first vector of a last network layer of the first feature extraction network as a feature vector of the query information.

The word segmentation unit refers to a unit with independent semantics in the query information. The word segmentation unit includes at least one character, e.g., the word segmentation unit is a word, or the word segmentation unit is a word. Alternatively, the computer device may use existing word segmentation tools to determine that the query information includes word segmentation units. Illustratively, the word segmentation tool may be designed based on word vectors (word vectors, word2 vec).

In some embodiments, the embedded token corresponding to the word segmentation unit refers to an embedded vector used to token the word segmentation unit. The embedded representation corresponding to the word segmentation unit can be preset, can be obtained by processing the word segmentation unit through a machine learning model, can be obtained by using any embedded representation generation method in the prior art, and is not set here.

In some embodiments, the computer device generates an embedded representation of the query information from the embedded representations respectively corresponding to the at least one word segmentation unit, including: the computer equipment arranges the embedded characterization corresponding to the word segmentation units according to the arrangement sequence of the word segmentation units in the query information to obtain an embedded characterization sequence; the computer equipment places the classification token before the embedded token sequence to obtain the embedded token of the query information; wherein the classification tokens are used to aggregate individual embedded tokens. The specific values of the classification characterization may vary in different ones of the first feature extraction network layers.

Alternatively, the feature vector of the query information refers to a classification token in the last network layer in the first feature extraction network, the classification token being in the form of a vector, which can be represented using CLS [ ].

The training process of the recall model comprises at least one training round, and for any training round in the at least one training round, the computer equipment performs K query information obtained in the step 210 and at least one positive query result corresponding to the K query information respectively; the computer device calculates feature vectors corresponding to the K query information respectively using the method in step 220, so as to find negative case query results corresponding to the K query information respectively.

Alternatively, the K query information selected by the computer device is not exactly the same in different rounds. For example, in the x-th training round, k=3, the computer device selects query information 1, query information 2, and query information 3; in the x+1th training round, the computer device selects query information 1, query information 5, and query information 6. Illustratively, in each training round, the computer device randomly selects K pieces of query information from the plurality of pieces of query information, and takes the K pieces of query information and query results (the query results include positive case query results and negative case query results) respectively corresponding to the respective pieces of query information as training data used in the training round.

Step 230, selecting k query results with similarity with the query information from the query results of each candidate according to the feature vectors of the query information and the feature vectors corresponding to the query results of each candidate; the feature vector corresponding to the candidate query result is obtained by adopting a second feature extraction network, and k is a positive integer.

In some embodiments, the feature vector corresponding to the candidate query result is referred to as a feature vector of the candidate query result, which is used to characterize the content features of the candidate query result. For example, for candidate query results in textual form, feature vectors of the candidate query results are used to characterize semantic features of the candidate query results.

Alternatively, the dimension of the feature vector of the query information is the same as the dimension of the feature vector of the candidate query result, that is, the query information and the candidate query result are mapped in the same vector space through the recall model.

The candidate query results include historical query results. Optionally, the search recall function of the target application program corresponds to a query result library, the query result library includes at least one query result, the candidate query result belongs to the query result library, and the query result included in the query result library can be updated or deleted so as to adapt to the search recall scene change of the target application program.

Optionally, feature vectors corresponding to the at least one candidate query result respectively are pre-generated before a training process of the recall model is started. After determining the feature vectors to which the at least one candidate query result corresponds, respectively, the computer device stores the feature vectors of the at least one candidate query result. The storage space for storing feature vectors of at least one candidate query result may be referred to as a corpus (coupus).

In the process of using the candidate feature vector of the query result, the computer device may directly read the candidate feature vector of the query result from the storage space. The method is beneficial to avoiding repeated calculation of the feature vector of the candidate query result, reducing the calculation pressure of the computer equipment and improving the training speed of the recall model.

The computer device determines index positions of the feature vectors corresponding to the candidate query results according to the feature vectors corresponding to the candidate query results, wherein the index positions can be hard disk indexes or display card indexes; the computer device can quickly read the feature vector of the candidate query result stored in the index position from the storage space according to the index position.

For example, under the condition that the index position is the display card index, because the training process of the recall model is performed in the display card, under the instruction of the display card index, the computer equipment directly determines the characteristic information of the candidate query result, and data reading and writing operations among different storage media are not needed, so that the time consumption of data reading and writing in the training process of the recall model is reduced, and the time consumption of the training process of the recall model is shortened.

Optionally, feature vectors corresponding to at least one candidate query result respectively may also be generated during the training process of the recall model. For example, during a first training round in a training process of the recall model, the computer device determines feature vectors corresponding to at least one candidate query result, respectively, through the second feature extraction network. Optionally, feature vectors corresponding to the candidate query results respectively form a corpus.

Optionally, the feature vectors corresponding to the at least one candidate query result are determined by a feature extraction network other than the second feature extraction network, where the other feature extraction network may be the first feature extraction network, or may be a feature extraction network in a recall model other than the recall model to be trained. For example, before a training process of the recall model begins, the computer device uses the first feature extraction network to respectively determine feature vectors to which at least one candidate query result respectively corresponds.

The method for determining the feature vector of the candidate query result is the same as the method for determining the feature vector of the query information, and specific reference is made to the above embodiment, and details are not repeated here.

In some embodiments, the computer device selects k query results having similarity with the query information from the candidate query results according to the feature vector of the query information and the feature vector corresponding to each candidate query result, including: the computer equipment takes the top k query results which are most similar to the query information as k query results, wherein k is a positive integer. For details of determining k query results, refer to the following examples.

Optionally, the similarity between the candidate query result and the candidate query information is determined according to the first metric. The first metric includes at least one of: geometric distance between the characteristic vector of the query information and the characteristic vector of the candidate query result, and angle cosine between the characteristic vector of the query information and the characteristic vector of the candidate query result.

Illustratively, the computer device determines a first metric value of a candidate query result based on the feature vector of the candidate query result and the feature vector of the query information, the first metric value being used to measure similarity between the feature vector of the candidate query result and the feature vector of the query information.

In one example, the first metric value is proportional to the similarity, and for a candidate query result, the greater the first metric value between the candidate query result and the query information, the greater the degree of similarity between the candidate query result and the query information; the smaller the first metric value between the candidate query result and the query information, the smaller the degree of similarity between the candidate query result and the query information.

In one example, the first metric value is inversely proportional to the similarity, for a candidate query result, the smaller the first metric value between the candidate query result and the query information, the greater the degree of similarity between the candidate query result and the query information; the greater the first metric value between the candidate query result and the query information, the less similar the candidate query result and the query information.

Optionally, for different query information in one training round, the number of k query results corresponding to any one query information may be the same or different, and the present application is not set here.

Step 240, selecting at least one query result from the k query results as a negative case query result corresponding to the query information, where the negative case query result is a query result that does not match the query information.

In some embodiments, the negative example query result corresponding to the query information refers to a query result that does not match the query information. The negative case query result can be divided into a simple negative case and a difficult negative case; the simple negative examples refer to negative example query results which are completely irrelevant to the query information in terms of content, and the difficult negative examples refer to negative example query results which are partially overlapped with the query information but are different in terms of content.

Optionally, for the query information and the query result in text form, the difficult negative example and the query information have at least one same functional word on the sentence structure, and the functional word includes at least one of the following: subject, object, verb, clause, etc. For example, a certain query information is "xxx evaluation notice", and there is a candidate query result "hot search | xxx urgent apology", and although the query information and the candidate query result both include "xxx", the requirement of the query information is inconsistent with the content of the candidate query result, and it is difficult to reject the query result in the search recall process, so the candidate query result is a difficult negative example of the query information. The difficult negative case query result is also referred to as a difficult negative case.

Alternatively, for the difficult negative example in the form of a picture and the query information, the difficult negative example and the query information have similar partial pictures including the content of the picture center area, the object included in the picture, and the like.

Optionally, the negative example query result determined in step 240 belongs to a difficult negative example query result.

And 250, adjusting parameters of the recall model based on the positive case query result and the negative case query result corresponding to the query information to obtain the trained recall model.

In some embodiments, the parameters of the recall model include: parameters of the first feature extraction network and parameters of the second feature extraction network. Optionally, the computer device adjusts parameters of the recall model, including: and fixing the parameters of the second characteristic extraction network, and adjusting the parameters of the first characteristic network.

Optionally, the recall model further comprises a classification prediction layer for predicting a similarity between the query information and the query result based on the feature vector of the query information and the feature vector of the query result. In this case, the computer device adjusts parameters of the first feature extraction network and parameters of the classification prediction layer in the recall model according to the positive case query result and the negative case query result corresponding to the query information, and obtains the recall model after training.

In one example, the training process of the recall model includes at least one training round, each training round in the at least one training round, the computer device selects K query information, uses a first feature extraction model in the training round, calculates feature vectors corresponding to the K query information respectively, and for any one of the K query information, the computer device uses the first feature extraction model in the training round to perform feature extraction on the query information to obtain feature vectors of the query information; the computer equipment selects k query results according to the feature vector of the query information and the feature vector corresponding to at least one candidate query result respectively, and selects at least one negative case query result corresponding to the query information from the k query results.

The computer equipment calculates the loss function value of the recall model according to the feature vector of the query information and the feature vector of the query result. Under the condition that the loss function value is not converged, the computer equipment adjusts parameters of the recall model to obtain a trained recall model, and carries out the next training round on the trained recall model; under the condition that the loss function value is converged, the computer equipment adjusts parameters of the recall model to obtain a recall model after training, and the training process of the recall model is finished.

In summary, in the training process of the recall model, at least one difficult negative example query result corresponding to the query information is determined according to the similarity between the feature vector of the query information and the feature vector of the candidate query result, and compared with the prior art, the difficult negative example query result corresponding to the query information needs to be manually marked, thereby improving the determination efficiency of the difficult negative example query result. More difficult negative case query results are provided for the training process of the recall model, so that the recall model is helped to learn differences between query information and negative case query results, the identification capability of the recall model on the difficult negative case query results is improved, and the recall accuracy of the recall model after training is improved.

The following describes the process of determining the negative example query results by several embodiments.

In some embodiments, the computer device selects at least one query result from the k query results as a negative example query result corresponding to the query information, including: the computer equipment selects at least one query result from query results except at least one positive instance query result included in the k query results as a negative instance query result corresponding to the query information.

From the above, it can be seen that k query results are the top k query results with the highest similarity with the query information in the at least one candidate query result, where the k query results may include the positive query results matched with the query information. The computer equipment needs to remove positive case query results corresponding to the query information from k query results to avoid mistakes in recognizing the positive case query results as negative case query results, so that positive case training data and negative case training data are the same, interference caused by the training data is introduced in the training process of the recall model, the convergence speed of the recall model in the training process is facilitated to be accelerated, and the recall effect on the recall model after the effect is reduced.

Assuming k is equal to 5, for a certain query information a, the k query results include: query result a, query result b, query result c, query result d, and query result e; if the query result a is a positive case query result of the query information a, the computer device determines at least one query result from the query result b, the query result c, the query result d and the query result e as a negative case query result corresponding to the query information a.

Optionally, when the computer device stores the positive case query result (or the feature vector of the positive case query result), the query information identifiers corresponding to the positive case query result are synchronously stored, and the query information identifiers are used for uniquely identifying the query information. After determining k query results corresponding to a certain query message, the computer equipment respectively judges whether query message identifiers corresponding to the k query results are query message identifiers of the query message; if the query identification information corresponding to a certain query result is the same as the query information identification of the query information in the k query results, indicating that the query result is a positive query result corresponding to the query information, and removing the query result from the k query results by the computer equipment; if the query identification information corresponding to a certain query result is different from the query information identification of the query information in the k query results, the query result is not the positive case query result corresponding to the query information, and the query result is reserved in the k query results.

In some embodiments, the computer device randomly selects at least one query result from among query results included in the k query results except for the at least one positive instance query result as a negative instance query result corresponding to the query information. For example, the computer device randomly selects 10 query results from query results included in the k query results except for at least one positive instance query result as negative instance query results corresponding to the query information. Alternatively, the number of negative case query results corresponding to the unused query information may be the same or different.

Illustratively, the number of negative example query results corresponding to any one of the query information in the same training round is the same. That is, the computer device determines y negative case query results from k query results corresponding to "query information 1", and the computer device determines y negative case query results from k query results corresponding to "query information 2", where y is a positive integer less than k.

By keeping the number of negative case query results corresponding to different query information consistent, the contribution degree of each query information in the process of determining the loss function value is the same in the process of carrying out the recall model training, so that the capability of the recall model after training to the different query information is improved, and the robustness of the recall model after training is improved.

Fig. 5 is a schematic diagram of a method for determining a query result corresponding to query information according to an exemplary embodiment of the present application.

The computer equipment adopts a first feature extraction network to determine a feature vector of query information, selects k query results according to the feature vector of the query information and the feature vector of at least one candidate query result included in the corpus, and can obtain a plurality of negative case query results corresponding to the query information by removing positive case query results and pseudo negative case query results in the k query results, and adjusts model parameters of a recall model by using one positive case query result and a plurality of negative case query results corresponding to the query information to obtain adjusted model parameters.

And selecting the negative example query result from query results except at least one positive example query result included in the k query results, so that the positive example query result is prevented from being determined as the negative example query result, and the reliability of the negative example training data is improved.

Two methods of determining negative example query results are described below in terms of several embodiments.

In the process of determining the positive case query results corresponding to the query information, the number of the positive case query results corresponding to the query information has the maximum limit (the maximum limit can be manually set according to factors such as recall accuracy of a recall model and processing pressure of computer equipment), and the number of the query results matched with the query information can be different for different query information, so that all historical query results matched with the query information cannot be used as the positive case query results corresponding to the query information.

In this case, some pseudo-negative query results may exist among the candidate query results, which are query results that can match the query information, but cannot be determined as positive query results due to the above-described limiting factors.

For example, for a certain query information, the similarity between the query information and the historical query result 1 is 0.935, the similarity between the query information and the historical query result 2 is 0.935, the similarity between the query information and the historical query result 3 is 0.945, and the similarity between the query information and the historical query result 4 is 0.92; it is assumed that the recall model is limited in terms of data storage, time consumption for training and the like in the training process, each query message corresponds to at most two positive query results, and the computer equipment takes the historical query result 1 and the historical query result 3 as the positive query results of the query message.

In this case, although the historical query results 2 and 4 have high similarity with the query information, the historical query results 2 and 4 are not selected to be positive-example query results, that is, the historical query results 2 and 4 belong to pseudo-negative-example query results corresponding to the query information.

In order to improve the confidence of the determined negative case query results, the computer equipment needs to screen query results except at least one positive case query result included in the k query results, remove the false negative case query results, and obtain the negative case query results corresponding to the query information. Specifically, the following two examples may be included:

example 1

In some embodiments, the computer device selects at least one query result from among query results included in the k query results except for the at least one positive instance query result as a negative instance query result corresponding to the query information, including: the computer equipment removes at least one positive case query result from the k query results to obtain at least one residual query result; the computer equipment determines at least one query result with minimum similarity with the query information according to the similarity between the query information and each residual query result; and the computer equipment determines at least one query result with the minimum similarity as a negative example query result corresponding to the query information.

In some embodiments, the query result with the least similarity refers to: and at least one query result with the lowest matching degree with the query information among the rest query results. That is, for any negative example query result, the similarity between the feature vector of the negative example query result and the feature vector of the query information is less than or equal to the similarity between the feature vector of the other query result and the feature vector of the query information, and the other query result refers to: any one of the remaining query results that is not selected.

Optionally, in step 230, after determining k query results, the computer device ranks the k query results according to the similarity between the k query results and the query information, to obtain a query result sequence.

Illustratively, the computer device orders the k query results in order of greater similarity, the more front the query result in the sequence of query results is similar to the query information. In this case, after removing at least one positive case query result from k query results (corresponding to deleting the at least one positive case query result from the query result sequence), the computer device inverts the query result sequence by y query results, where y is a positive integer less than k, as a negative case query result corresponding to the query information.

Illustratively, the computer device orders the k query results in order of decreasing similarity, with the higher the first in the sequence of query results, the greater the deviation between the query results and the query information. In this case, after removing at least one positive instance query result from the k query results, the computer device treats the first y query results of the query result sequence as negative instance query results for the query information.

By selecting the query results which have similarity with the query information and are not high in similarity as negative case query results, the selection of false negative case query results to participate in the training process of the recall model can be avoided, the confidence of the determined negative case query results is improved, the reliability of training data is improved, and the limit between the positive case query results and the negative case query results is clearer; by the method, the recall model is helped to learn and identify the positive case query result and the difficult negative case query result, and the convergence speed of the recall model in the training process is improved.

Example 2

In some embodiments, the computer device selects at least one query result from among query results included in the k query results except for the at least one positive instance query result as a negative instance query result corresponding to the query information, including: according to the similarity between the query information and each residual query result, the computer equipment determines n query results with the minimum similarity with the query information, wherein n is a positive integer smaller than k; the computer equipment removes at least one positive case query result from the n query results, and determines the rest query results as negative case query results corresponding to the query information.

Optionally, a query result sequence can be obtained after the computer device determines k query results (the specific content of the query sequence is the same as that of the previous embodiment, and details are not repeated here).

Illustratively, the computer device sorts the k query results in order of greater similarity to obtain a query result sequence; the computer equipment determines n query results with the smallest similarity with the query information as n query results with the inverse n query results in the query result sequence; the computer device removes the positive case query result from the n query results to obtain at least one negative case query result (please refer to the above embodiments for details regarding the method of removing the positive case query result, and details are not repeated here).

For example, k is equal to 100, n is equal to 50, the computer device takes the 51 st to 100 th query results in the query result sequence as n query results, and removes positive case query results included in the 51 st to 100 th query results to obtain at least one negative case query result.

Illustratively, the computer device orders the k query results according to the order from the smaller similarity to the larger similarity to obtain a query result sequence, determines the first n query results in the query result sequence as n query results with the smallest similarity to the query information, and executes the subsequent steps, which are described in detail herein.

Through the method, the n query results with the minimum similarity are selected firstly, and then the positive query results are removed from the n query results, so that the number of the positive query results to be removed is reduced, the number of times of detecting whether the positive query results are required to be detected by the computer equipment is reduced, and the determination efficiency of the negative query results is improved. The method is also beneficial to avoiding the false negative case query result from participating in the training process of the recall model, and is beneficial to improving the quality of the negative case query result and optimizing the training effect of the recall model.

The training process of the recall model is described in several embodiments below.

In some embodiments, step 250: the computer device adjusts parameters of the recall model based on the positive case query result and the negative case query result corresponding to the query information to obtain a trained recall model, wherein the method comprises the following sub-steps (not shown in the drawing of the specification).

Substep 252: the computer equipment determines a loss function value of the recall model according to the positive case query result and the negative case query result corresponding to the query information, wherein the loss function value is used for representing the prediction accuracy of the recall model on whether the query result is positive or negative.

In some embodiments, the loss function value is calculated based on cross entropy loss (Cross entropy loss). Cross entropy loss is a kind of loss function row adjustment for quantifying the difference between two probability distributions.

Optionally, sub-step 252 includes the steps of: for the target query result, the computer equipment determines a loss function value of the recall model according to the similarity between the feature vector of the query information and the feature vector of the target query result and the label information of the target query result; the target query result is any positive case query result or any negative case query result corresponding to the query information, and the tag information is used for representing that the target query result belongs to the positive case query result or the negative case query result.

In some embodiments, for a certain query information, the query results corresponding to the query information include a positive example query result and at least one negative example query result. The target query result refers to any one of the positive instance query result or at least one negative instance query result.

For example, for any one of the query information in one training round, the query information corresponds to 1 positive instance query result and 49 negative instance query results, i.e., there are a total of 50 query results related to the query information that participate in the loss function value determination process.

Optionally, the one positive case query result refers to a positive case query result with the lowest similarity with the query information in the at least one positive case query result. That is, in one training round, for any query information, the computer device selects a positive query result from at least one positive query result corresponding to the query information, where the selected positive query result participates in the training round, and other positive query results corresponding to the query information do not participate in the training round. By selecting the positive case query result with the lowest similarity with the query information from at least one positive case query result, the capability of identifying the difficult negative case of the recall model is improved, and the recall accuracy of the recall model after training is improved.

In each training round, the computer equipment selects a positive case query result from at least one positive case query result corresponding to the query information, and uses the positive case query result to participate in the loss function value calculation, wherein at least one positive case query result except the positive case query result does not participate in the round training process, and the loss function value calculation is performed.

In some implementations, the computer device determines similarity between the feature vector of the query information and the feature vector of the target query result using the second metric. Optionally, the second metric comprises: the vector inner product between the feature vector of the query information and the feature vector of the target query result.

In some embodiments, the tag information is used to characterize the actual properties of the target query result. Optionally, if the target query result is a positive case query result, the tag information of the target query result is equal to 1, that is, the actual probability that the target query result belongs to the positive case query result is equal to 1, and the actual probability that the target query result belongs to the negative case query result is equal to 0; if the target query result is a negative case query result, the tag information of the target query result is equal to 0, that is, the actual probability that the target query result belongs to a positive case query result is equal to 0, and the actual probability that the target query result belongs to a negative case query result (including difficult negative cases and simple negative cases) is equal to 1.

Alternatively, the calculation formula of the cross entropy may be expressed as the following one.

Wherein N represents the number of training samples, M represents the total amount of query information in one training round, Is a sign function%The value of (1) is 0 or 1), for the positive training sampleTaking negative training samples；The probability that the recall model predicts that a certain query result belongs to a positive instance query result or to a negative instance query result is represented.

Substep 255: the computer equipment adjusts parameters of the first feature extraction network according to the loss function value to obtain a recall model after training; wherein the parameters of the second feature extraction network are not adjusted.

In some embodiments, the computer device adjusts a parameter of the first feature extraction network based on the loss function value, comprising: the computer device calculates a gradient of the parameter of the first feature extraction network from the loss function value and updates the parameter of the first feature extraction network in a counter-propagating manner.

In the training process of the recall model, only the first feature extraction network is adjusted, and the network parameters of the second feature extraction network are unchanged, namely, in any training round of the recall model, the feature vector of the same query result is not changed, and based on the training method, the feature vector of at least one candidate query result is not required to be updated in the training process, and only the feature vector of one candidate query result is required to be determined, so that the calculation pressure in the training process of the recall model is reduced, the convergence speed of the recall model is increased, and the training time consumption of the recall model is shortened.

In some embodiments, the computer device adjusts parameters of the first feature extraction network according to the loss function value, and after obtaining the trained recall model, further comprises: if the recall model does not meet the condition of stopping training, the computer equipment performs the next round of training on the recall model after training, and the step of acquiring the query information is performed again; for the same query information, feature vectors of the query information are different in training processes of different rounds, and negative-example query results corresponding to the query information are different in training processes of different rounds.

In some embodiments, the condition to stop training is used to determine whether a training process of a next round is required after the training of the current round is completed. Optionally, the condition for stopping training is whether the loss function value converges. If the loss function value converges, the computer equipment stops training the trained recall model; if the loss function value is not converged, the computer equipment performs the next round of training on the trained recall model; wherein the convergence of the loss function value means that the loss function value is stabilized in at least one consecutive training round.

The training process of the recall model is described below by way of an example.

The training process A for a certain recall model includes the following steps.

Step A10, acquiring query information and at least one positive case query result corresponding to the query information, wherein the positive case query result refers to a query result matched with the query information. Optionally, the computer device acquires a plurality of query information and at least one positive case query record result corresponding to each query information respectively; wherein the query requirements of at least two of the plurality of query information are different.

For example, the computer device obtains 60000 pieces of query information, and a positive case query result corresponding to each piece of query information in the 60000 pieces of query information.

For the p-th training round (i.e., the p-th training round), p is a positive integer; the computer equipment selects x pieces of inquiry information from the plurality of pieces of inquiry information acquired in the step A10, wherein x is a positive integer. Optionally, the computer device shuffles the plurality of query information and randomly selects x query information from the plurality of query information. The computer device performs p rounds of training based on the x pieces of query information.

For example, for the 2 nd training round, the computer device selects 1024 query information from the plurality of query information, and uses the selected 1024 query information to train the recall model for the 2 nd round.

Optionally, in the training process of the recall model, there are at least two training rounds, where the at least two training rounds have q identical query information, and q is a positive integer.

Step A20, for any one of the x pieces of query information, the computer device adopts a first feature extraction network to extract feature vectors of the query information.

For query information I in the x query information, the computer equipment adopts a first feature extraction network to extract feature vectors of the query information I. For details on this process, reference is made to the above embodiments.

And step A30, the computer equipment selects k query results with similarity with the query information from the query results of each candidate according to the feature vectors of the query information and the feature vectors respectively corresponding to the query results of each candidate.

For example, the computer device uses k candidate query results with highest similarity with the query information as k query results. For details on this part, reference is made to the above examples.

Step A40, the computer equipment removes at least one positive case query result from k query results to obtain at least one residual query result; the computer equipment determines at least one query result with minimum similarity with the query information according to the similarity between the query information and each residual query result; and determining at least one query result with the minimum similarity as a negative example query result corresponding to the query information. For the specific process of this step, please refer to the above embodiment, and a detailed description is omitted here.

And step A50, for the target query result, determining a loss function value of the recall model according to the similarity between the feature vector of the query information and the feature vector of the target query result and the label information of the target query result. Optionally, the computer device calculates a loss function value for the recall model during the p-th round of training using the cross entropy loss function.

Optionally, the computer device determines the similarity between the query information and the target query result based on a vector inner product of the feature vector of the query information and the feature vector of the target query result.

And step A60, adjusting parameters of the first feature extraction network according to the loss function value to obtain a trained recall model. Optionally, for any one training round, the computer device only modifies the parameters of the first feature extraction network and not the parameters of the second feature extraction network.

Step A70, the computer device determines whether the recall model satisfies the condition for stopping training.

Optionally, if the loss function value of the recall model does not converge, the recall model does not meet the condition of stopping training, and the computer equipment performs the training of the next round on the trained recall model obtained in the step A60; specifically, the computer device starts to perform the p+1st training round again in step a 10; and (C) if the loss function value of the recall model is converged, the recall model meets the condition of stopping training, and the recall model after training in the step A60 is obtained.

In the training process of the recall model, network parameters of the first feature extraction network are changed, so that feature vectors of the same query information are changed in different training rounds, the feature vectors of candidate query results are unchanged due to the fact that parameters of the second feature extraction network are unchanged, and the feature vectors of the first feature extraction query information are changed, so that in different training rounds, the similarity between the determined query information and the candidate query results is possibly changed according to the feature vectors of the query information and the feature vectors of the candidate query results, and negative case query results corresponding to the same query information respectively are not identical in different training rounds.

By the method, the negative case query result can be automatically generated in the training process, and the negative case query result corresponding to the query information does not need to be generated in advance. In different training rounds, the negative case query results corresponding to the query information can be dynamically changed, and the richness of the negative case query results corresponding to the query information in the training process is improved.

In the following, a process of finding k query results from the candidate query results is described by several embodiments.

In some embodiments, step 230 selects k query results having similarity with the query information from the candidate query results according to the feature vector of the query information and the feature vector corresponding to each of the candidate query results.

Alternatively, in order to quickly find k query results from a vast number of candidate query results, the computer device uses the index information to determine k query results that are displayed with the query information. That is, in the corpus, feature vectors corresponding to at least one candidate query result are stored according to a certain storage structure, and feature vectors of any two candidate query results correspond to different index information; wherein the index information is used for: the feature vector of the candidate query result corresponding to the feature vector is quickly found in the corpus, and the feature vector of the candidate query result corresponding to the index information is directly obtained at a storage position in the computer equipment.

Illustratively, the index information is associated with at least one other index information (also referred to as a friend index) by which the other index information with which it has an association can be directly determined. For example, there is a correlation between index information corresponding to the feature vector of the candidate query result s and index information corresponding to the feature vector of the candidate query result t, and the similarity between the candidate query result s and the candidate query result is higher than the similarity between the candidate query result s and other candidate query results.

If the similarity between the query information and the candidate query result s is high, the similarity between the query information and the candidate query result t may be high. Then, after determining the similarity between the query information and the candidate query result s, the computer device may preferentially determine the similarity between the query information and the candidate query result t, rather than preferentially determine the similarity between the query information and the other candidate query results.

The index information is used for searching k query results, so that all candidate query results are prevented from being traversed in the process of searching k query information, and the speed of searching k query results can be increased.

Alternatively, step 230 may include the following sub-steps (not shown in the drawings of the specification):

in step 231, the computer device generates a dynamic result list according to feature vectors of candidate query results at a top level of an index structure, where the index structure is configured to store feature vectors corresponding to at least one candidate query result respectively according to similarity between at least one candidate query result, and the dynamic result list is configured to dynamically record index information of candidate query results having similarity with the query information in the index structure in a process of selecting the query results.

Optionally, the feature vector of the candidate query result at the top level of the index structure refers to the feature vector of the first candidate query result in the index structure for calculating similarity with the query information. For any query information, in determining k query results, similarity needs to be calculated with candidate query results at the top level of the index structure.

For example, the dynamic result list in substep 231 includes: index information index1 of candidate query results at the top layer of the index structure, and at least one index information (index 2, index 3) associated with the index information index1 in the index structure, wherein index2 and index3 are friend index of index1 respectively.

At this time, the dynamic result list includes: index1, index2, index3.

In sub-step 233, for the ith index information included in the dynamic result list, the computer device adds at least one friendly point index associated with the ith index information in the index structure to the dynamic result list, where the feature vector of the candidate query result corresponding to the friendly point index is adjacent to the feature vector of the candidate query result corresponding to the ith index information, and i is a positive integer.

Suppose that index 2's friendly index is index4, index6, index 3's friendly index is index5, index6. Then in sub-step 233 the dynamic results list includes: index1, index2, index3, index4, index5, index6.

A sub-step 235, selecting k adjacent indexes from the dynamic result list, and obtaining an updated dynamic result list by the computer equipment; the similarity between the feature vector of the candidate query result corresponding to the adjacent index and the feature vector of the query information is higher than the similarity between the feature vector of the candidate query result corresponding to the unselected other index information in the dynamic result list and the feature vector of the query information.

Optionally, the computer device calculates similarity between the feature vector of the candidate query result and the feature vector of the query information, where the feature vector corresponds to index1, index2, index3, index4, index5, and index6, respectively. Optionally, the computer device determines, using the first metric, a similarity between the feature vector of the candidate query result corresponding to the index information and the feature vector of the query information, respectively. The first metric includes: the distance of the feature vectors in the feature space (such as Euclidean distance), the inner product of the feature vectors, the cosine of the included angle between the feature vectors, etc.

The computer equipment sorts the six index information according to the similarity from high to low to obtain index1, index3, index2, index4, index6 and index5. Assuming k is equal to 4, the computer device selects index1, index3, index2, index4. At this time, the updated dynamic result list includes index1, index3, index2, index4.

In a substep 237, the computer device again starts with the step of adding the i-th index information to the dynamic result list in association with at least one friend index in the index structure, and performs the next round of selection process.

In this example, since index1, index2, index3 always exist in the dynamic result list, only the friendly index of index4 needs to be added to the dynamic result list. Assuming that index 4's friendly index is index1, index7, the dynamic result list includes: index1, index2, index3, index4, and index7.

In the substep 239, if the updated dynamic result list is the same as the updated dynamic result list obtained in the next round of selection process, the computer device uses the candidate query results corresponding to the k index information included in the updated dynamic result list as k query results.

Alternatively, if the updated dynamic result list is different from the updated dynamic result list obtained in the next round of selection process, the computer device starts executing in sub-step 233 until the index information included in the updated dynamic result list obtained in each of the two adjacent rounds of selection process is identical.

Continuing with the example above, the computer device includes, for the dynamic results list: and re-executing the substep 235 by index1, index2, index3, index4 and index7, and performing the next round of selection process, wherein the computer equipment sorts the 5 index information according to the similarity from high to low to obtain index1, index3, index2, index4 and index7.k is equal to 4, then the computer device selects index1, index3, index2, index4. At this time, the updated dynamic result list of the next round includes index1, index3, index2, and index4, that is, the updated dynamic result list (that is, the updated dynamic result list obtained in the z-th round of selection process, z is a positive integer) is the same as the updated dynamic result list obtained in the next round of selection process (the updated dynamic result list obtained in the z+1th round of selection process), and the computer device uses the candidate query results corresponding to index1, index2, index3, and index4 as k query results.

In some embodiments, to avoid computing the similarity between the feature vector of the query result and the feature vector of the query information for the same candidate multiple times in the same training round, the method further includes, when performing the sub-step 235: for any one of the index information included in the dynamic result list, the computer device determines whether the friend index of the index information was added to the dynamic result list or the friend index is being located in the dynamic list.

If a certain friend index of the index information is added into the dynamic result list, the certain friend index is not added into the dynamic result list; if a certain friend index of the index information is positioned in the dynamic result list, the certain friend index is not added into the dynamic result list; if a friend index of the index information is not added to the dynamic result list, and the dynamic result list does not comprise the friend index at present, adding the friend index to the dynamic result list.

The method is beneficial to avoiding repeated calculation caused by adding certain index information into the dynamic result list for multiple times, reduces the calculation amount of the process of determining k query results, and is beneficial to improving the speed of determining k query results.

In some embodiments, the computer device determines a feature vector of a candidate query result corresponding to the index information, comprising: determining the display card index corresponding to the index information according to the mapping relation between the index information and the display card index; and the computer equipment acquires the feature vector of the candidate query result from the display card according to the display card index. The display card index is used for indicating the storage position of the feature vector of the candidate query result in the display card.

Because the training process of the recall model is completed in the display card, the number of data reading and writing times among hardware of the computer equipment in the process of determining the feature vector of the candidate query result is reduced by storing the feature vector of the candidate query result in the display card, thereby being beneficial to reducing the reading and writing pressure of the computer equipment.

The method of creating the index structure is described in the following by way of an exemplary embodiment.

In some embodiments, the training method of the recall model further comprises: for any one candidate query result in the at least one candidate query result, the computer equipment takes the candidate query result as the query result to be stored; the computer equipment adopts a second feature extraction network to determine the feature vector of the query result to be stored; the computer equipment determines first index information of the feature vector of the query result to be stored in the index structure according to the feature vector of the query result to be stored and the feature vector of the query result stored in the index structure, wherein the stored query result belongs to a candidate query result; the method comprises the steps that the computer equipment determines m stored query results with highest similarity with query results to be stored, index information corresponding to the m stored query results respectively is used as friend indexes associated with first index information, and m is a positive integer.

Optionally, the computer device builds the index structure before the recall model training begins. Feature vectors are added to the index structure by similarity between the feature vectors.

It should be noted that, the above construction of the index result according to the feature vector of at least one candidate query structure, and the determination of k query results in the index result may be implemented by a program written by a worker, or may be implemented by a vector indexing tool, where when the index structure is constructed using the vector indexing tool, the computer device transmits the feature vector of the at least one candidate query structure to the vector indexing tool, where the index tool performs the above construction process of the index structure, and when k query results are selected, the computer device transmits the feature vector of the query information to the vector indexing tool, where the vector indexing tool performs the above process of searching for k query results.

The method is helpful to reduce the number of times of calculating the similarity between the feature vector of the candidate query result and the feature vector of the query information in the process of selecting k query results corresponding to the query information.

The method for acquiring the query result of the positive example is described by several embodiments.

The computer equipment obtains the query information and at least one positive case query result corresponding to the query information, and the method comprises the following steps: the method comprises the steps that a computer device obtains query information and a plurality of historical query results corresponding to the query information; the computer equipment screens at least one historical query result with the click times larger than or equal to a first threshold value from the historical query results and uses the at least one historical query result as at least one positive query result corresponding to the query information.

In some embodiments, the computer device determines at least one historical query result corresponding to the query information and a click condition corresponding to each historical query result according to the click log.

In some embodiments, the computer device screens out at least one historical query result from the plurality of historical query results that has a click condition greater than or equal to the first threshold. Optionally, the first threshold is preset.

In some embodiments, the click condition is used to characterize the acceptance of individual historical query results by users in conducting search recalls based on the query information. Optionally, the click condition includes at least one of: the number of clicks, the click rate, the click frequency. For specific description of the clicking situation, please refer to the above embodiments, and details are not described here.

In some embodiments, the computer device screens out at least one historical query result with a click number greater than or equal to a first threshold from the plurality of historical query results as at least one positive query result corresponding to the query information, including: the computer equipment screens at least one historical query result with the click frequency being greater than or equal to a first threshold value from the historical query results; the computer equipment selects at least one positive case query result corresponding to the query information from the at least one historical query result according to the quality information respectively corresponding to the at least one historical query result.

Optionally, the quality information includes quality of the historical query results, and gear of the historical query results. Optionally, the quality information may be manually marked, or may be determined according to the browsing times, the user recommendation times, the praise times, the collection times, and the like of the historical query result.

The historical query information is screened through the clicking condition, at least one positive query result corresponding to the query information is determined, the receiving condition of the user on the historical query information is equivalently utilized, preliminary manual screening is realized, the manpower consumption for screening the positive query result from the massive historical query results is reduced, and the cost for labeling the positive query result is reduced.

The following describes, by way of an example, a training process for a recall model provided by the present application. Taking the query results as article titles (title) as an example (the training process of the recall model is simply denoted as process B for descriptive convenience) may include the following steps.

Step B10, the computer equipment acquires inquiry information used in the training process, and for each inquiry information, the computer equipment screens at least one title with the click times larger than or equal to a first threshold value from a plurality of historical titles; and the computer equipment selects at least one positive title corresponding to the query information from the at least one historical title according to the quality information respectively corresponding to the at least one historical title.

Step B20, the computer equipment uses the second feature extraction network to determine the feature vector of at least one candidate title; the computer device determines index information of the candidate title in the index structure based on distances between the feature vectors of the candidate title and the feature vectors of other candidate titles, and a friendly point index associated with the index information.

Training of the recall model is then initiated: step B30, in any training round, the computer equipment randomly selects a plurality of query information used by the training round from the query information used in the training process. The computer equipment adopts a first feature extraction network to extract feature vectors respectively corresponding to the plurality of inquiry information.

Step B40, for any one of the plurality of query information, the computer device selects k titles having similarity with the query information from the index structure according to the feature vector of the query information. For details of this process, please refer to the above embodiments, and a detailed description is omitted here.

Alternatively, the computer device performs step B30 and step B40 on different query information in the same period of time, respectively. For example, when determining the feature vector of the ith query information, the computer device searches the index structure for k titles corresponding to the f query information, where i is a positive integer, and f is a positive integer smaller than i, and the different parallel steps B30 and B40 help to shorten the time consumption of determining the negative case query result corresponding to each query information used in the training round.

And B50, selecting at least one query result from query results except at least one positive example title contained in the k titles as a negative example title corresponding to the query information by the computer equipment.

And step B60, the computer equipment determines a loss function value of the recall model according to the positive title and the negative title corresponding to the query information, wherein the loss function value is used for representing the prediction accuracy of the recall model on whether the title is positive or negative.

And step B70, adjusting parameters of the first feature extraction network according to the loss function value to obtain a trained recall model.

And step B80, if the recall model does not meet the training stopping condition, performing the next round of training on the recall model after training, and starting to execute the step B30 again.

The method does not need to pre-determine the negative case query result in the process of training the recall model, and is beneficial to reducing the preparation time of training data of the recall model. In addition, in the training process, negative case data corresponding to the same query information in different training rounds are not identical, so that richer negative case query results are provided for the recall model, and the recognition capability of the recall model on difficult negative case query results is facilitated. And improving the recall capability of the trained recall model on the query results related to the query information.

In addition, in the training process of the recall model, only the parameters of the first feature extraction network are adjusted, so that feature vectors corresponding to massive candidate query results do not need to be repeatedly generated, the reasoning speed of the training process of the recall model is reduced, and the linear speed of the recall model after training is carried out under the line is improved.

The training method of the recall model is suitable for a video recall model, an article recall model, an audio recall model, a commodity recall model, an application recall model and the like. For example, in the video recall model: the query information is text information input by a user in a search field, and the query result is a video title. For another example, in the article recall model: the query information is text information input by a user in a search bar, and the query result is an article title or a key paragraph in the article. For another example, in the merchandise recall model: the inquiry information is text information input by a user in a search field or pictures uploaded by the user, and the inquiry result is commodity titles, commodity webpages and the like. For another example, in the application recall model: the query information is text information input by a user in a search field, and the query result is functional introduction, name or download link of the target application program.

The feature extraction networks (comprising a first feature extraction network and a second feature extraction network) corresponding to different recall models are similar in mechanism of feature processing, embedded characterization of query information in the forms of characters, pictures or audio is determined first, and then feature vectors corresponding to the query information are determined based on the embedded characterization by using the feature extraction networks. Optionally, the feature extraction network is designed based on an attention mechanism, and the specific network structure of the feature extraction network is not limited by the present application.

In some embodiments, the recall model training method provided by the application can be performed after fine tuning of the original recall model; the original recall model can be an existing recall model in the Internet, and can also be a recall model with lower recall accuracy.

In the fine tuning process, the computer equipment trains the original recall model by using query information, positive case query results corresponding to the query information and negative case query results corresponding to the query information (mainly simple negative cases and also can comprise negative cases with difficult manual labeling).

In each training round of the fine tuning process, the computer equipment adopts a first feature extraction network to perform feature extraction on the query information to obtain a feature vector of the query information, and adopts a second feature extraction network to perform feature extraction on the query result to obtain a feature vector of the query result; the computer equipment calculates a loss function value of the recall model in the fine tuning process according to the feature vector of the query information and the feature vector of the query result, and adjusts the parameters of the first feature extraction network and the parameters of the second feature extraction network by using the loss function value. Optionally, the calculation method of the feature vector and the loss function value is similar to the training method of the recall model provided by the present application, and will not be described herein.

After the fine tuning is finished, the computer equipment adopts the training method of the recall model provided by the application to train the recall model after the fine tuning is finished so as to improve the recognition capability of the recall model on the difficult negative case query result, and further improve the degree of matching between the query result determined by the recall model and the query information in the actual application process.

In some embodiments, to maintain the performance of the recall model, the recall model in the target application program needs to be updated in training at intervals to obtain a new version of the recall model. Optionally, the foregoing fine-tuning training manner and the training method of the recall model provided by the present application may be alternately performed during the training update process.

For example, in the h training updating process, parameters of the recall model after h-1 training updating are adjusted in a fine adjustment mode; in the h+g training updating process, parameters of the first feature extraction network after h+g-1 training updating are adjusted by adopting the recall model training mode provided by the application, and h and g are positive integers. The value of g can be set according to actual needs, for example, g is equal to 1.

Optionally, in a certain training and updating process, training the recall model in a fine tuning manner to obtain a first recall model, and then adjusting parameters of a first feature extraction network in the first recall model by using the training method of the recall model provided by the application; or, firstly, using the recall model training method provided by the application to adjust the parameters of the first feature extraction network in the recall model to obtain a second recall model; and then the parameters of the second recall model are adjusted in a fine adjustment mode.

Illustratively, the training method used by a training update process is related to the performance of the current recall model, for example, if the recall accuracy of the recall model is below an accuracy threshold, then the current recall model is trained using a fine-tuning approach; if the query result determined by the recall model comprises a large number of negative case query results, training is carried out by using the training method of the recall model. The accuracy threshold is set according to actual needs, and recall accuracy is used for representing the matching degree between the query result and the query information determined by the recall model in the actual use process.

The following describes the actual application process of the recall model after training.

In the training process of the recall model provided by the application, only the parameters in the first feature extraction network are required to be updated, and the parameters in the second feature extraction network are kept unchanged in the training process of the recall model, so that in the process of applying the recall model after training, only the parameters in the first feature extraction network in the recall model corresponding to the target application program are required to be updated, thereby being beneficial to shortening the online process, and the feature vectors respectively corresponding to the query results are not required to be changed.

After the online process of the recall model is completed, the server acquires information to be queried sent by the terminal equipment, the server uses a first feature extraction network to perform feature extraction on the queried information to obtain a feature vector of the information to be queried, and the feature vector is also called a [ cls ] vector under the condition that the first feature extraction network is of a BERT structure.

The server determines W class clusters closest to the feature vector of the information to be queried according to the feature vector of the information to be queried, wherein the same class cluster comprises query results of the same type, and W is a positive integer. For example, W is equal to 50.

Optionally, the server determines the proximity degree of the information to be queried and the class cluster according to the feature vector of the information to be queried and the center point of the class cluster. The center point of the class cluster refers to the center of the feature vector of each query result included in the class cluster in the multidimensional feature space. Optionally, the sum of distances from the center point of the cluster to the feature vectors of the respective query results is minimal.

And the server calculates the similarity between all query results and the information to be queried, which are respectively included by the W class clusters, according to the recall model.

The server selects a front Y query result with highest similarity with the information to be queried from all query results contained in the W class clusters, sorts the Y query results, and returns the sorted Y query results to the terminal equipment in the form of an exposure list.

The following are examples of the apparatus of the present application that may be used to perform the method embodiments of the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the method of the present application.

FIG. 6 illustrates a block diagram of a recall model training apparatus provided in one exemplary embodiment of the present application. The recall model includes a first feature extraction network and a second feature extraction network, and the apparatus 600 may include: a positive example acquisition module 610, a feature extraction module 620, a result determination module 630, a negative example acquisition module 640, and a model training module 650.

The positive example obtaining module 610 is configured to obtain query information and at least one positive example query result corresponding to the query information, where the positive example query result is a query result matched with the query information.

The feature extraction module 620 is configured to extract a feature vector of the query information using the first feature extraction network.

The result determining module 630 is configured to select k query results with similarity to the query information from the query results of each candidate according to the feature vector of the query information and the feature vector corresponding to the query result of each candidate; and the feature vector corresponding to the candidate query result is obtained by adopting the second feature extraction network, and k is a positive integer.

And the negative example obtaining module 640 is configured to select at least one query result from the k query results as a negative example query result corresponding to the query information, where the negative example query result is a query result that does not match the query information.

The model training module 650 is configured to adjust parameters of the recall model based on the positive case query result and the negative case query result corresponding to the query information, so as to obtain a recall model after training.

In some embodiments, the negative example acquisition module 640 includes: a result selection unit configured to: and selecting at least one query result from query results except for the at least one positive instance query result included in the k query results as a negative instance query result corresponding to the query information.

In some embodiments, the result selecting unit is configured to remove the at least one positive case query result from the k query results, to obtain at least one remaining query result; determining at least one query result with minimum similarity with the query information according to the similarity between the query information and each of the rest query results; determining at least one query result with the minimum similarity as a negative example query result corresponding to the query information; or, according to the similarity between the query information and each of the rest query results, determining n query results with the minimum similarity with the query information, wherein n is a positive integer smaller than k; and removing the at least one positive case query result from the n query results, and determining the rest query results as negative case query results corresponding to the query information.

In some embodiments, the model training module 650 includes: the loss calculation unit is used for determining a loss function value of the recall model according to a positive case query result and a negative case query result corresponding to the query information, and the loss function value is used for representing the prediction accuracy of the recall model on whether the query result is positive or negative; the parameter adjustment module is used for adjusting the parameters of the first feature extraction network according to the loss function value to obtain the trained recall model; wherein the parameters of the second feature extraction network are not adjusted.

In some embodiments, the loss calculation unit is configured to determine, for a target query result, a loss function value of the recall model according to similarity between a feature vector of the query information and a feature vector of the target query result, and tag information of the target query result; the target query result is any positive case query result or any negative case query result corresponding to the query information, and the tag information is used for representing that the target query result belongs to the positive case query result or the negative case query result.

In some embodiments, the apparatus 600 further comprises: the round circulation module is used for carrying out next round of training on the trained recall model under the condition that the recall model does not meet the condition of stopping training, and the step of acquiring query information is carried out again; for the same query information, the feature vectors of the query information are different in the training process of different rounds, and the negative case query results corresponding to the query information are different in the training process of different rounds.

In some embodiments, the result determining module 630 is configured to generate a dynamic result list according to feature vectors of candidate query results on a top level of an index structure, where the index structure is configured to store feature vectors corresponding to the at least one candidate query result respectively according to similarity between the at least one candidate query result, and the dynamic result list is configured to dynamically record index information of the candidate query result having similarity with the query information in the index structure in a query result selection process; adding at least one friend index associated with the ith index information in the index structure to the dynamic result list for the ith index information included in the dynamic result list, wherein the feature vector of the candidate query result corresponding to the friend index is adjacent to the feature vector of the candidate query result corresponding to the ith index information, and i is a positive integer; k adjacent indexes are selected from the dynamic result list, and an updated dynamic result list is obtained; the similarity between the feature vector of the candidate query result corresponding to the adjacent index and the feature vector of the query information is higher than the similarity between the feature vector of the candidate query result corresponding to the unselected other index information in the dynamic result list and the feature vector of the query information; starting from the step of adding the i-th index information to the dynamic result list in the index structure with at least one friend index associated with the i-th index information, executing a next round of selection process; and if the updated dynamic result list is the same as the updated dynamic result selection list obtained in the next round of selection process, respectively using the k candidate query results corresponding to the k index information included in the updated dynamic result list as the k query results.

In some embodiments, the apparatus 600 further comprises: the index establishing module is used for regarding any one candidate query result in the at least one candidate query result as a query result to be stored; determining a feature vector of the query result to be stored by adopting the second feature extraction network; determining first index information of the feature vector of the query result to be stored in the index structure according to the feature vector of the query result to be stored and the feature vector of the query result stored in the index structure, wherein the stored query result belongs to the candidate query result; determining m stored query results with highest similarity to the query results to be stored, and taking index information corresponding to the m stored query results as friend indexes associated with the first index information, wherein m is a positive integer.

In some embodiments, the positive example obtaining module 610 is configured to obtain query information, and a plurality of historical query results corresponding to the query information; and screening at least one historical query result with the click frequency being greater than or equal to a first threshold value from the plurality of historical query results, and taking the at least one historical query result as at least one positive query result corresponding to the query information.

It should be noted that, in the apparatus provided in the foregoing embodiment, when implementing the functions thereof, only the division of the foregoing functional modules is used as an example, in practical application, the foregoing functional allocation may be implemented by different functional modules, that is, the content structure of the device is divided into different functional modules, so as to implement all or part of the functions described above. In addition, the apparatus and the method embodiments provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the apparatus and the method embodiments are detailed in the method embodiments and are not repeated herein. The beneficial effects of the device provided in the foregoing embodiments are described with reference to the method side embodiments, and are not repeated herein.

Fig. 7 shows a block diagram of a computer device according to an exemplary embodiment of the present application. The training device 700 of the recall model may be the computer device described above.

In general, the computer device 700 includes: a processor 701 and a memory 702.

Processor 701 may include one or more processing cores, such as a 4-core processor, a 7-core processor, and the like. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 701 may also include a main processor, which is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 may integrate a GPU (Graphics Processing Unit, picture processor) for rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.

Memory 702 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 702 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 702 stores at least one instruction, at least one program, code set, or instruction set that is loaded and executed by processor 701 to implement the training method of the recall model provided by the various method embodiments described above.

The embodiment of the application also provides a computer readable storage medium, and the storage medium stores a computer program, and the computer program is loaded and executed by a processor to realize the training method of the recall model provided by the above method embodiments.

The computer readable medium may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash Memory or other solid state Memory technology, DVD (Digital Video Disc, high density digital video disc) or other optical storage, magnetic tape cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the ones described above.

Embodiments of the present application also provide a computer program product, which includes a computer program stored in a computer readable storage medium, and a processor reads and executes the computer program from the computer readable storage medium to implement the training method of the recall model provided in the above-mentioned method embodiments.

It should be understood that references herein to "a plurality" are to two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

It should be noted that, before and during the process of collecting the relevant data of the user, the present application may display a prompt interface, a popup window or output voice prompt information, where the prompt interface, popup window or voice prompt information is used to prompt the user to collect the relevant data currently, so that the present application only starts to execute the relevant step of obtaining the relevant data of the user after obtaining the confirmation operation of the user to the prompt interface or popup window, otherwise (i.e. when the confirmation operation of the user to the prompt interface or popup window is not obtained), the relevant step of obtaining the relevant data of the user is finished, i.e. the relevant data of the user is not obtained. In other words, the application collects the inquiry information and the clicking times of the inquiry result, which user the inquiry information and the clicking times of the inquiry result come from are not recorded, and the processing strictly meets the requirements of the laws and regulations of the relevant country, the informed consent or the independent consent of the personal information main body is collected under the condition that the user agrees and authorizes, the subsequent data use and processing actions are carried out within the range of the laws and regulations and the authorization of the personal information main body, and the collection, the use and the processing of the relevant user data need to comply with the relevant laws and regulations and standards of the relevant country and region.

The foregoing description of the preferred embodiments of the application is not intended to limit the application to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and principles of the application are intended to be included within the scope of the application.

Claims

1. A training method of a recall model, wherein the recall model comprises a first feature extraction network and a second feature extraction network; the method comprises the following steps:

generating a dynamic result list according to feature vectors corresponding to candidate query results on the top layer of an index structure, wherein the index structure is used for storing feature vectors corresponding to at least one candidate query result respectively according to the similarity between the at least one candidate query result, the dynamic result list is used for dynamically recording index information of the candidate query result with similarity to the query information in the index structure in the process of selecting the query result, and the feature vectors corresponding to the candidate query result are obtained by adopting the second feature extraction network;

Adding at least one friend index associated with the ith index information in the index structure to the dynamic result list for the ith index information included in the dynamic result list, wherein the feature vector of the candidate query result corresponding to the friend index is adjacent to the feature vector of the candidate query result corresponding to the ith index information, and i is a positive integer;

k adjacent indexes are selected from the dynamic result list, and an updated dynamic result list is obtained; the similarity between the feature vector of the candidate query result corresponding to the adjacent index and the feature vector of the query information is higher than the similarity between the feature vector of the candidate query result corresponding to the unselected other index information in the dynamic result list and the feature vector of the query information;

starting from the step of adding the i-th index information to the dynamic result list in the index structure with at least one friend index associated with the i-th index information, executing a next round of selection process;

if the updated dynamic result list is the same as the updated dynamic result list obtained in the next round of selection process, using k candidate query results respectively corresponding to k index information included in the updated dynamic result list as k query results having similarity with the query information, wherein k is a positive integer;

2. The method according to claim 1, wherein selecting at least one query result from the k query results as a negative instance query result corresponding to the query information comprises:

and selecting at least one query result from query results except for the at least one positive instance query result included in the k query results as a negative instance query result corresponding to the query information.

3. The method according to claim 2, wherein selecting at least one query result from among query results included in the k query results except the at least one positive instance query result as the negative instance query result corresponding to the query information includes:

removing the at least one positive case query result from the k query results to obtain at least one residual query result; determining at least one query result with minimum similarity with the query information according to the similarity between the query information and each of the rest query results; determining at least one query result with the minimum similarity as a negative example query result corresponding to the query information;

Or alternatively, the process may be performed,

according to the similarity between the query information and each of the rest query results, determining n query results with the minimum similarity with the query information, wherein n is a positive integer smaller than k; and removing the at least one positive case query result from the n query results, and determining the rest query results as negative case query results corresponding to the query information.

4. The method of claim 1, wherein the adjusting parameters of the recall model based on the positive case query result and the negative case query result corresponding to the query information to obtain the trained recall model comprises:

determining a loss function value of the recall model according to a positive case query result and a negative case query result corresponding to the query information, wherein the loss function value is used for representing the prediction accuracy of the recall model on whether the query result is positive or negative;

according to the loss function value, adjusting parameters of the first feature extraction network to obtain the trained recall model; wherein the parameters of the second feature extraction network are not adjusted.

5. The method of claim 4, wherein the determining the loss function value of the recall model based on the positive case query result and the negative case query result corresponding to the query information comprises:

For a target query result, determining a loss function value of the recall model according to the similarity between the feature vector of the query information and the feature vector of the target query result and the label information of the target query result;

the target query result is any positive case query result or any negative case query result corresponding to the query information, and the tag information is used for representing that the target query result belongs to the positive case query result or the negative case query result.

6. The method of claim 4, wherein adjusting parameters of the first feature extraction network according to the loss function value, after obtaining the trained recall model, further comprises:

if the recall model does not meet the condition of stopping training, performing next round of training on the recall model after training, and executing again from the step of acquiring query information;

for the same query information, the feature vectors of the query information are different in the training process of different rounds, and the negative case query results corresponding to the query information are different in the training process of different rounds.

7. The method according to claim 1, wherein the method further comprises:

for any one candidate query result in the at least one candidate query result, taking the candidate query result as a query result to be stored;

determining a feature vector of the query result to be stored by adopting the second feature extraction network;

determining first index information of the feature vector of the query result to be stored in the index structure according to the feature vector of the query result to be stored and the feature vector of the query result stored in the index structure, wherein the stored query result belongs to the candidate query result;

determining m stored query results with highest similarity to the query results to be stored, and taking index information corresponding to the m stored query results as friend indexes associated with the first index information, wherein m is a positive integer.

8. The method of claim 1, wherein the obtaining query information and at least one positive example query result corresponding to the query information comprises:

acquiring query information and a plurality of historical query results corresponding to the query information;

And screening at least one historical query result with the click frequency being greater than or equal to a first threshold value from the plurality of historical query results, and taking the at least one historical query result as at least one positive query result corresponding to the query information.

9. A training device of a recall model, wherein the recall model comprises a first feature extraction network and a second feature extraction network; the device comprises:

the result determining module is used for generating a dynamic result list according to feature vectors corresponding to candidate query results on the top layer of an index structure, the index structure is used for storing feature vectors corresponding to at least one candidate query result respectively according to similarity among the at least one candidate query result, the dynamic result list is used for dynamically recording index information of the candidate query result with similarity with the query information in the index structure in the process of selecting the query result, and the feature vectors corresponding to the candidate query result are obtained by adopting the second feature extraction network; adding at least one friend index associated with the ith index information in the index structure to the dynamic result list for the ith index information included in the dynamic result list, wherein the feature vector of the candidate query result corresponding to the friend index is adjacent to the feature vector of the candidate query result corresponding to the ith index information, and i is a positive integer; k adjacent indexes are selected from the dynamic result list, and an updated dynamic result list is obtained; the similarity between the feature vector of the candidate query result corresponding to the adjacent index and the feature vector of the query information is higher than the similarity between the feature vector of the candidate query result corresponding to the unselected other index information in the dynamic result list and the feature vector of the query information; starting from the step of adding the i-th index information to the dynamic result list in the index structure with at least one friend index associated with the i-th index information, executing a next round of selection process; if the updated dynamic result list is the same as the updated dynamic result list obtained in the next round of selection process, using k candidate query results respectively corresponding to k index information included in the updated dynamic result list as k query results having similarity with the query information, wherein k is a positive integer;

10. A computer device comprising a processor and a memory, the memory having stored therein a computer program that is loaded and executed by the processor to implement the training method of the recall model of any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that the storage medium has stored therein a computer program, which is loaded and executed by a processor to implement the training method of the recall model according to any one of claims 1 to 8.