CN108388563B - Information output method and device - Google Patents
Information output method and device Download PDFInfo
- Publication number
- CN108388563B CN108388563B CN201710063291.2A CN201710063291A CN108388563B CN 108388563 B CN108388563 B CN 108388563B CN 201710063291 A CN201710063291 A CN 201710063291A CN 108388563 B CN108388563 B CN 108388563B
- Authority
- CN
- China
- Prior art keywords
- sample data
- classification model
- information classification
- information
- category
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
Abstract
The application discloses an information output method and device. One embodiment of the method comprises: receiving information sent by a client; extracting a feature vector from the information, wherein the feature vector is used for representing the content of the information; importing the feature vector into a pre-trained information classification model for classification to obtain the class of the information; and matching target feedback information associated with the information from a pre-stored feedback information set indicated by the category, and outputting the target feedback information to the client. This embodiment improves the information output efficiency.
Description
Technical Field
The application relates to the technical field of computers, in particular to the technical field of internet, and particularly relates to an information output method and device.
Background
In the field of artificial intelligence, many automatic artificial intelligence response systems based on natural language processing are emerging at present. The information output efficiency of the artificial intelligence response system depends on the accuracy of the classification model to a large extent.
However, the accuracy of existing classification models is generally low, resulting in inefficient information output by artificial intelligence response systems.
Disclosure of Invention
It is an object of the present application to provide an improved information output method and apparatus to solve the technical problems mentioned in the background section above.
In a first aspect, the present application provides an information output method, including: receiving information sent by a client; extracting a feature vector from the information, wherein the feature vector is used for representing the content of the information; importing the feature vectors into a pre-trained information classification model for classification to obtain the category of the information; and matching target feedback information associated with the information from a pre-stored feedback information set indicated by the category, and outputting the target feedback information to the client.
In some embodiments, the method further comprises the step of establishing an information classification model, wherein the step of establishing the information classification model comprises the steps of obtaining a sample data set and a category respectively associated with each sample data in the sample data set; for each candidate information classification model in pre-stored candidate information classification models, taking the candidate information classification model as a candidate information classification model to be evaluated, and determining the accuracy of the candidate information classification model to be evaluated based on the sample data set and the class respectively associated with each sample data in the sample data set, wherein the pre-stored candidate information classification model is an untrained model; determining a target information classification model from the pre-stored candidate information classification models based on the determined accuracy; and training the target information classification model to obtain an information classification model by utilizing a machine learning method based on the sample data set and the class associated with each sample data in the sample data set.
In some embodiments, the determining the accuracy of the candidate information classification model to be evaluated includes: and determining the accuracy of the candidate information classification model to be evaluated by adopting a cross validation method.
In some embodiments, the determining the accuracy of the candidate information classification model to be evaluated includes: determining a weight value associated with each class of sample data in the sample data set, wherein the weight value is a ratio of the number of the class of sample data to the total number of the sample data contained in the sample data set; and determining the accuracy of the candidate information classification model to be evaluated according to the determined weight value.
In some embodiments, the determining the accuracy of the candidate information classification model to be evaluated according to the determined weight value includes: the following processing steps are executed in a loop for a predetermined number of times: dividing the sample data set into a training set and a test set, training the candidate information classification model to be evaluated based on the training set and a class respectively associated with each training sample data in the training set by using a machine learning method to obtain a trained candidate information classification model, predicting the class of each test sample data in the test set by using the trained candidate information classification model to obtain a prediction result, determining the prediction accuracy rate associated with the class test sample data based on the prediction result for each class test sample data in the test set, taking the product of a weight value associated with the sample data in the same class as the class test sample data in the sample data set and the prediction accuracy rate as a weighted prediction accuracy rate associated with the class test sample data, and recovering the candidate information classification model to be evaluated to a state without training the training sample data classification, wherein the weighted prediction accuracy rates associated with the class test sample data in the test set and the test sample data in the same class are added to obtain a value as a first weighted prediction accuracy rate associated with the candidate information classification model to be evaluated, and the training sample data classification accuracy rate is a training sample data classification number of the training sample data in the training set; and taking the average value of the obtained predetermined number of first weighted prediction accuracy rates as the accuracy rate of the candidate information classification model to be evaluated.
In some embodiments, the determining, based on the determined accuracy, a target information classification model among the pre-stored candidate information classification models includes: and taking the candidate information classification model with the highest accuracy in the pre-stored candidate information classification models as a target information classification model.
In a second aspect, the present application provides an information output apparatus comprising: the receiving unit is configured to receive information sent by the client; an extracting unit, configured to extract a feature vector from the information, wherein the feature vector is used for representing the content of the information; the classification unit is configured to import the feature vectors into a pre-trained information classification model for classification to obtain the category of the information; and the output unit is configured to match target feedback information associated with the information from a pre-stored feedback information set indicated by the category and output the target feedback information to the client.
In some embodiments, the above apparatus further comprises: the information classification model establishing unit is configured for establishing an information classification model and comprises the following steps: the acquisition subunit is configured to acquire a sample data set and a category associated with each sample data in the sample data set; an accuracy determining subunit, configured to, for each of pre-stored candidate information classification models, use the candidate information classification model as a candidate information classification model to be evaluated, and determine an accuracy of the candidate information classification model to be evaluated based on the sample data set and a category associated with each sample data in the sample data set, where the pre-stored candidate information classification model is an untrained model; a target information classification model determination subunit configured to determine a target information classification model among the prestored candidate information classification models based on the determined accuracy; and the training subunit is configured to train the target information classification model to obtain an information classification model based on the sample data set and the class associated with each sample data in the sample data set by using a machine learning method.
In some embodiments, the accuracy determination subunit includes: and the first accuracy determining module is configured to determine the accuracy of the candidate information classification model to be evaluated by adopting a cross validation method.
In some embodiments, the accuracy determination subunit includes: a weight value determining module configured to determine, for each class of sample data in the sample data set, a weight value associated with the class of sample data, where the weight value is a ratio of the number of the class of sample data to a total number of sample data included in the sample data set; and the second accuracy determining module is configured to determine the accuracy of the candidate information classification model to be evaluated according to the determined weight value.
In some embodiments, the second accuracy determination module comprises: a processing submodule configured to cyclically execute a predetermined number of the following processing steps: dividing the sample data set into a training set and a test set, training the candidate information classification model to be evaluated based on the training set and a class respectively associated with each training sample data in the training set by using a machine learning method to obtain a trained candidate information classification model, predicting the class of each test sample data in the test set by using the trained candidate information classification model to obtain a prediction result, determining the prediction accuracy rate associated with the class test sample data based on the prediction result for each class test sample data in the test set, taking the product of a weight value associated with the sample data in the same class as the class test sample data in the sample data set and the prediction accuracy rate as a weighted prediction accuracy rate associated with the class test sample data, and recovering the candidate information classification model to be evaluated to a state without training the training sample data classification, wherein the weighted prediction accuracy rates associated with the class test sample data in the test set and the test sample data in the same class are added to obtain a value as a first weighted prediction accuracy rate associated with the candidate information classification model to be evaluated, and the training sample data classification accuracy rate is a training sample data classification number of the training sample data in the training set; and the accuracy determining submodule is configured to use an average value of the obtained predetermined number of first weighted prediction accuracy as the accuracy of the candidate information classification model to be evaluated.
In some embodiments, the target information classification model determining subunit includes: and the target information classification model determining module is configured to take the candidate information classification model with the highest accuracy in the pre-stored candidate information classification models as the target information classification model.
In a third aspect, the present application provides a server, comprising: one or more processors; storage means for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation manner of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method as described in any implementation manner of the first aspect.
According to the information output method and the information output device, the feature vector is extracted from the received information, so that the feature vector is led into a pre-trained information classification model to be classified to obtain the category of the information. And then matching target feedback information associated with the information from a pre-stored feedback information set indicated by the category so as to output the target feedback information to the client. Therefore, the information is classified by effectively utilizing the pre-trained information classification model, and the information output efficiency is improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:
FIG. 1 is an exemplary system architecture diagram to which the present application may be applied;
FIG. 2 is a flow diagram of one embodiment of an information output method according to the present application;
FIG. 3 is a schematic diagram of an application scenario corresponding to the embodiment shown in FIG. 2;
FIG. 4 is a flow diagram of one embodiment of determining an accuracy of a candidate information classification model to be evaluated based on determined weight values according to the present application.
FIG. 5 is a schematic block diagram of an embodiment of an information output apparatus according to the present application;
FIG. 6 is a schematic block diagram of a computer system suitable for use in implementing a server according to embodiments of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that, in the present application, the embodiments and features of the embodiments may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 shows an exemplary system architecture 100 to which embodiments of the information output method or information output apparatus of the present application may be applied.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. Various communication client applications, such as an application supporting automatic artificial intelligence response, a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like, may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like.
The server 105 may be a server that provides various services, such as a backend server that performs processing such as analysis on information transmitted by the terminal apparatuses 101, 102, and 103, and may also feed back a processing result (e.g., target feedback information associated with the above information) to the terminal apparatuses.
It should be noted that the information output method provided in the embodiment of the present application is generally executed by the server 105, and accordingly, the information output apparatus is generally disposed in the server 105.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow 200 of one embodiment of an information output method according to the present application is shown. The information output method comprises the following steps:
In this embodiment, the electronic device (for example, the server 105 shown in fig. 1) on which the information output method operates may receive information transmitted by a client (for example, the terminal devices 101, 102, 103 shown in fig. 1) through a wired connection manner or a wireless connection manner. The information may be consultation information sent by the user through the client in a text or voice manner, for example, information used for consulting whether delivery can be performed, whether orders can be modified after orders are placed, what payment methods are available, what the large telephone installation expense is calculated, and the like.
In this embodiment, the electronic device may extract a feature vector from the information. Wherein, the feature vector can be used for characterizing the content of the information. Here, if the information is voice information, the electronic device may convert the information into text information using a voice recognition technology. The electronic device may import the information or text information obtained by converting the information into a feature extraction model trained in advance for extracting feature words to perform feature word extraction, and generate a feature vector based on the obtained feature words. As an example, for the text information "how big electric installation cost is, the feature words of the text information extracted by the above feature extraction model may be" big household appliance "and" installation cost ", and then a two-dimensional feature vector of the text information (" big household appliance "," installation cost ") may be obtained.
In some optional implementation manners of this embodiment, the electronic device may further quantize the feature words extracted from the information to obtain a numerical value used for characterizing the feature words, and generate the feature vector based on the numerical value. Here, the above numerical value may be a weight of the feature word, and the weight may be a number of times that the feature word appears in the information.
And 203, importing the feature vectors into a pre-trained information classification model for classification to obtain the category of the information.
In this embodiment, the electronic device may import a feature vector extracted from the information into the information classification model, and the information classification model may find a category corresponding to the feature vector according to a pre-trained correspondence relationship with the feature vector, and set the category as a category to which the information belongs.
In some optional implementation manners of this embodiment, the information output method may further include a step of establishing an information classification model, which may include: first, the electronic device may obtain a sample data set and a category associated with each sample data in the sample data set, where the category associated with each sample data in the sample data set may be artificially pre-labeled. Then, regarding each candidate information classification model in the pre-stored candidate information classification models, taking the candidate information classification model as a candidate information classification model to be evaluated, and the electronic device may determine the accuracy of the candidate information classification model to be evaluated based on the sample data set and the category associated with each sample data in the sample data set. Then, the electronic device may determine a target information classification model among the pre-stored candidate information classification models based on the determined accuracy. Finally, the electronic device may train the target information classification model to obtain an information classification model based on the sample data set and a category associated with each sample data in the sample data set by using a machine learning method. Here, the electronic device may determine the accuracy of the candidate information classification model to be evaluated by using a cross-validation method. And the electronic device may use a candidate information classification model with the highest accuracy among the pre-stored candidate information classification models as the target information classification model. If the number of the candidate information classification models with the highest accuracy is more than 1, the electronic device can randomly select one candidate information classification model from the candidate information classification models with the highest accuracy as a target information classification model. It should be noted that the cross-validation method is a well-known technique widely studied and applied at present, and is not described herein again.
In some optional implementations of the embodiment, for each class of sample data in the sample data set, the electronic device may determine a weight value associated with the class of sample data, where the weight value may be a ratio of the number of the class of sample data to a total number of sample data included in the sample data set. For example, if the total number of sample data included in the sample data is 1000, and the number of sample data of a certain category included in the sample data set is 100, the weight value associated with the sample data of the category is a ratio of 100 to 1000, that is, 10%. Then, the electronic device may determine an accuracy of the candidate information classification model to be evaluated according to the determined weight value. Here, the electronic device may determine the accuracy of the candidate information classification model to be evaluated according to the determined weight value by executing the flow 400 shown in fig. 4.
And 204, matching target feedback information associated with the information from a pre-stored feedback information set indicated by the category, and outputting the target feedback information to the client.
In this embodiment, for each category that can be identified by the information classification model, the server local to the electronic device or remotely connected to the electronic device may have a set of pre-stored feedback information associated with the category, and the set of pre-stored feedback information may include feedback information related to information of the category. For example, if a certain category identified by the information classification model is "order consultation", the set of pre-stored feedback information associated with the category "order consultation" may include feedback information related to the information of the order consultation category, for example, the feedback information "you can click on a" delete "option behind the corresponding order number viewed here, and delete the order purchase record. Delete orders can be viewed inside my orders page order recycle bin ". Moreover, for each category that can be identified by the information classification model, a server local to the electronic device or remotely connected to the electronic device may pre-store a list of information characterizing the correspondence between the category and a set of pre-stored feedback information associated with the category. Each piece of information in the list of information may include an identification of a category and a set of pre-stored feedback information associated with the category.
In this embodiment, after the electronic device obtains the category to which the information belongs by executing the step 203, the electronic device may determine a pre-stored feedback information set indicated by the category by reading the information list, and then the electronic device may match target feedback information associated with the information in the pre-stored feedback information set and output the target feedback information to the client. Here, the electronic device may select a second predetermined number of pieces of feedback information from the set of pre-stored feedback information as the target feedback information in an order from a larger matching degree to a smaller matching degree by calculating the matching degree of the information and each piece of feedback information in the set of pre-stored feedback information indicated by the category.
In some optional implementation manners of this embodiment, for each piece of feedback information in a set of pre-stored feedback information indicated by a category to which the piece of information belongs, a matching degree between the piece of feedback information and the piece of information may be a ratio of the number of feature words of the piece of information included in the piece of feedback information to a total number of feature words included in the piece of information. As an example, the information is "how big appliance installation cost is calculated", and the characteristic words of the information are "big appliance" and "installation cost"; one piece of feedback information in the pre-stored feedback information set indicated by the category is that 'different manufacturers have different installation fee collection standards, consultation manufacturers are advised to know by customers, and the installation fee of the air conditioner can be checked on a commodity introduction page'. Since the piece of feedback information includes the feature word "installation cost" of the information, the matching degree of the piece of feedback information and the information may be a ratio of 1 to 2, i.e., 50%.
With continued reference to fig. 3, fig. 3 is an application scenario corresponding to the embodiment shown in fig. 2. In the application scenario of fig. 3, the user first sends the text message "how the order was cancelled" through the client, as indicated by reference numeral 301. Thereafter, as indicated by reference numeral 302, the server may extract a feature vector (3, 1) from the text information, where 3 denotes the feature word "order" and 1 denotes the feature word "cancel". Then, as shown by reference numeral 303, the server may import the feature vector into a pre-trained information classification model for classification to obtain a category "order consultation" to which the text information belongs. Finally, as shown by reference numeral 304, the server may match the target feedback information associated with the text information from the pre-stored feedback information set indicated by "order consultation", and "please click here to request cancellation of the corresponding order. If the goods are sent out and the goods are cancelled, refusal (except special goods such as fresh goods, luxury goods, customized goods and the like) is recommended, and the target feedback information is output to the client.
The information output method disclosed by the embodiment effectively utilizes the pre-trained information classification model to classify the information, and improves the information output efficiency.
With further reference to FIG. 4, a flow 400 for determining an accuracy of a candidate information classification model to be evaluated based on determined weight values according to the present application is illustrated. The process 400 includes the following steps:
In this embodiment, the electronic device may divide the sample data set into a training set and a test set, where the training set and the test set include sample data of the same category. Here, the training sample data included in the training set and the test sample data included in the test set are usually different from each other. The number of training sample data included in the training set is usually greater than the number of test sample data included in the test set. As an example, the sample data set includes sample data a1, a2, a3, b1, b2, and b3, where a category to which a1, a2, and a3 belong is denoted as a, and a category to which b1, b2, and b3 belong is denoted as b, the electronic device may randomly select a first predetermined number of sample data from the sample data of the category a as the test sample, and if a1 is used as the test sample, the electronic device may use a2 and a3 as the training samples; similarly, if b2 in the b-type sample data is used as a test sample, the electronic device may use b1 and b3 as training samples; the electronic device may use the test samples a1 and b2 as a test set, and use the training samples a2, a3, b1, and b3 as a training set.
And 402, training the candidate information classification model to be evaluated based on the training set and the category respectively associated with each training sample data in the training set by using a machine learning method to obtain the trained candidate information classification model.
In this embodiment, the electronic device may utilize a machine learning method to train the candidate information classification model to be evaluated through the training set and the category associated with each training sample data in the training set, so as to obtain a trained candidate information classification model capable of establishing an accurate correspondence between each training sample data in the training set and the category associated with the training sample data.
And 403, predicting the category of each test sample data in the test set by using the trained candidate information classification model to obtain a prediction result.
In this embodiment, for each test sample data in the test set, the electronic device may extract a feature vector of the test sample data from the test sample data by the same method as in step 206, and introduce the feature vector into the trained candidate information classification model to perform class prediction, so as to obtain a prediction result.
In this embodiment, for each class of test sample data in the test set, the electronic device may determine a prediction accuracy associated with the class of test sample data based on the prediction result. The prediction accuracy may be a ratio of a number of correct class predictions of the trained candidate information classification model for the class of test sample data to a number of the class of test sample data. Here, the electronic device may compare the prediction result associated with the class of test sample data with the actual class of the class of test sample data, so as to obtain the number of correct class predictions of the trained candidate information classification model for the class of test sample data. Then, the electronic device may use a product of a weight value associated with sample data included in the sample data set and having the same category as the category of the test sample data and the prediction accuracy as a weighted prediction accuracy associated with the category of the test sample data.
As an example, the sample data set contains 1000 sample data, wherein 200 sample data belong to the class denoted by a, 400 sample data belong to the class denoted by b, and the remaining 400 sample data belong to the class denoted by c. The weight value associated with the a-class sample data is 20%. The weight values associated with the b-class sample data and the c-class sample data are both 40%. Assuming that the test set includes 50 test sample data of a, b, and c categories, and the electronic device has determined that the trained candidate information classification model has 45 number of correct predictions for the category of the test sample data of the a category, 46 number of correct predictions for the category of the test sample data of the b category, and 44 number of correct predictions for the category of the test sample data of the c category. Then for each class of test sample data in the test set, the prediction accuracy associated with class a test sample data may be a ratio of 45 to 50, i.e., 90%, and the weighted prediction accuracy associated with class a test sample data may be a product of 20% (the weight value associated with class a sample data included in the sample data set) and 90%, i.e., 16%; the prediction accuracy associated with the b-class test pattern data may be a ratio of 46 to 50, i.e. 92%, and the weighted prediction accuracy associated with the b-class test pattern data may be a product of 40% (the weight value associated with the b-class sample data included in the sample data set) and 92%, i.e. 36.8%; the prediction accuracy associated with the c-class test sample data may be a ratio of 44 to 50, i.e. 88%, and the weighted prediction accuracy associated with the c-class test sample data may be a product of 40% (the weight value associated with the c-class sample data included in the sample data set) and 88%, i.e. 35.2%.
In this embodiment, the electronic device may use a value obtained by adding weighted prediction accuracy rates respectively associated with each category of test sample data in the test set as a first weighted prediction accuracy rate associated with the candidate information classification model to be evaluated. Continuing with the example in step 404, given that the weighted prediction accuracy associated with the class a test sample data in the test set is 16%, the weighted prediction accuracy associated with the class b test sample data in the test set is 36.8%, and the weighted prediction accuracy associated with the class c test sample data in the test set is 35.2%, then the sum of the weighted prediction accuracy associated with the class a test sample data in the test set and the weighted prediction accuracy associated with the class c test sample data in the test set is 88%, which may be used as the first weighted prediction accuracy associated with the candidate information classification model to be evaluated.
In some optional implementation manners of this embodiment, the electronic device may record the first weighted prediction accuracy into a local memory or a hard disk.
And step 406, restoring the candidate information classification model to be evaluated to an untrained state.
In this embodiment, after obtaining the first weighted prediction accuracy rate associated with the candidate information classification model to be evaluated, the electronic device may restore the candidate information classification model to be evaluated to an untrained state for subsequent training.
In some optional implementation manners of this embodiment, after the electronic device restores the candidate information classification model to be evaluated to an untrained state, the number of times of loop execution for the step 401 to the step 406 may be recorded.
Step 407 determines whether the number of times of loop execution for the above step 401 to the above step 406 reaches a predetermined number of times.
In this embodiment, after the electronic device executes the step 406, the recorded number of times of loop execution for the step 401 to the step 406 may be compared with a predetermined number, and if the number of times of loop execution is lower than the predetermined number, the electronic device may go to the step 401; if the loop is executed a predetermined number of times, the electronic device may proceed to step 408.
And step 408, taking the average value of the obtained predetermined number of first weighted prediction accuracy rates as the accuracy rate of the candidate information classification model to be evaluated.
In this embodiment, after the electronic device circularly performs the step 401 to the step 406 for the predetermined number of times, the average value of the obtained predetermined number of first weighted prediction accuracy rates may be used as the accuracy rate of the candidate information classification model to be evaluated.
In the embodiment shown in fig. 4, the accuracy of the candidate information classification model to be evaluated is determined by combining the weight value associated with each class of sample data in the sample data set, so that the accuracy of evaluating the accuracy of the candidate information classification model to be evaluated is improved.
With further reference to fig. 5, as an implementation of the methods shown in the above-mentioned figures, the present application provides an embodiment of an information output apparatus, which corresponds to the embodiment of the method shown in fig. 2, and which is particularly applicable to various electronic devices.
As shown in fig. 5, the information output apparatus 500 shown in the present embodiment includes: a receiving unit 501, an extracting unit 502, a classifying unit 503 and an outputting unit 504. The receiving unit 501 is configured to receive information sent by a client; the extracting unit 502 is configured to extract a feature vector from the information, where the feature vector is used to represent the content of the information; the classification unit 503 is configured to introduce the feature vectors into a pre-trained information classification model for classification to obtain categories to which the information belongs; the output unit 504 is configured to match target feedback information associated with the information from a set of pre-stored feedback information indicated by the category, and output the target feedback information to the client.
In the present embodiment, in the information output apparatus 500: for specific processing of the receiving unit 501, the extracting unit 502, the classifying unit 503 and the outputting unit 504 and technical effects thereof, reference may be made to relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment of fig. 2, and details are not repeated here.
In some optional implementations of this embodiment, the apparatus 500 may further include: an information classification model establishing unit (not shown in the figure) configured to establish an information classification model may include: an obtaining subunit (not shown in the figure), configured to obtain a sample data set and a category respectively associated with each sample data in the sample data set; an accuracy determining subunit (not shown in the figure), configured to, for each of the pre-stored candidate information classification models, use the candidate information classification model as a candidate information classification model to be evaluated, and determine an accuracy of the candidate information classification model to be evaluated based on the sample data set and a category respectively associated with each sample data in the sample data set, where the pre-stored candidate information classification model is an untrained model; a target information classification model determination subunit (not shown in the figure) configured to determine a target information classification model among the previously stored candidate information classification models based on the determined accuracy; and a training subunit (not shown in the figure) configured to train the target information classification model to obtain an information classification model based on the sample data set and a category associated with each sample data in the sample data set respectively by using a machine learning method.
In some optional implementations of the present embodiment, the accuracy determining subunit may include: and a first accuracy determining module (not shown in the figure) configured to determine the accuracy of the candidate information classification model to be evaluated by using a cross-validation method.
In some optional implementation manners of this embodiment, the accuracy determining subunit may include: a weight value determining module (not shown in the figure) configured to determine, for each category of sample data in the sample data set, a weight value associated with the category of sample data, where the weight value is a ratio of the number of the category of sample data to the total number of sample data included in the sample data set; and a second accuracy determining module (not shown in the figure) configured to determine the accuracy of the candidate information classification model to be evaluated according to the determined weight value.
In some optional implementations of this embodiment, the second accuracy determining module may include: a processing submodule (not shown in the figures) configured to cyclically execute a predetermined number of times the following processing steps: dividing the sample data set into a training set and a test set, training the candidate information classification model to be evaluated based on the training set and a class respectively associated with each training sample data in the training set by using a machine learning method to obtain a trained candidate information classification model, predicting the class of each test sample data in the test set by using the trained candidate information classification model to obtain a prediction result, determining a prediction accuracy rate associated with the class of test sample data based on the prediction result for each class of test sample data in the test set, taking a product of a weight value associated with the sample data in the same class with the class of test sample data in the sample data set and the prediction accuracy rate as a weighted prediction accuracy rate associated with the class of test sample data, and recovering the candidate information classification model to be evaluated to a state without training sample data classification, wherein the weighted prediction accuracy rate added with the weighted prediction accuracy rate respectively associated with each class of test sample data in the test set is taken as a first weighted prediction accuracy rate associated with the candidate information classification model to be evaluated, and the number of the test sample data classification after training of the class and the training sample data classification is correctly classified as a test accuracy rate of the test sample data of the training sample data; and an accuracy determining sub-module (not shown in the figure) configured to use an average of the obtained predetermined number of first weighted prediction accuracies as the accuracy of the candidate information classification model to be evaluated.
In some optional implementations of this embodiment, the determining unit of the target information classification model may include: and a target information classification model determining module (not shown in the figure) configured to use the candidate information classification model with the highest accuracy among the pre-stored candidate information classification models as the target information classification model.
Referring now to FIG. 6, shown is a block diagram of a computer system 600 suitable for use in implementing a server according to embodiments of the present application. The server shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, the processes described above with reference to the flow diagrams may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a receiving unit, an extracting unit, a classifying unit, and an outputting unit. The names of these units do not in some cases constitute a limitation to the unit itself, and for example, a receiving unit may also be described as a "unit that receives information sent by a client".
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by an apparatus, cause the apparatus to comprise: receiving information sent by a client; extracting a feature vector from the information, wherein the feature vector is used for representing the content of the information; importing the feature vectors into a pre-trained information classification model for classification to obtain the category of the information; and matching target feedback information associated with the information from a pre-stored feedback information set indicated by the category, and outputting the target feedback information to the client.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements in which any combination of the features described above or their equivalents does not depart from the spirit of the invention disclosed above. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (10)
1. An information output method, characterized in that the method comprises:
receiving information sent by a client;
extracting a feature vector from the information, wherein the feature vector is used for representing the content of the information;
importing the feature vector into a pre-trained information classification model for classification to obtain a class to which the information belongs, wherein the information classification model is obtained by utilizing the accuracy of a model determined after an acquired sample data set is trained on the basis of a plurality of pre-stored untrained models, the accuracy is determined on the basis of a determined weight value, and the weight value comprises the ratio of the number of sample data of each class in the sample data set to the total number of the sample data contained in the sample data set;
and matching target feedback information associated with the information from a pre-stored feedback information set indicated by the category, and outputting the target feedback information to the client.
2. The method of claim 1, further comprising the step of building an information classification model, the step of building an information classification model comprising:
acquiring a sample data set and a category respectively associated with each sample data in the sample data set;
for each candidate information classification model in pre-stored candidate information classification models, taking the candidate information classification model as a candidate information classification model to be evaluated, and determining the accuracy of the candidate information classification model to be evaluated based on the sample data set and the category respectively associated with each sample data in the sample data set, wherein the pre-stored candidate information classification model is an untrained model;
determining a target information classification model among the pre-stored candidate information classification models based on the determined accuracy;
and training the target information classification model to obtain an information classification model by utilizing a machine learning method based on the sample data set and the category respectively associated with each sample data in the sample data set.
3. The method of claim 2, wherein the determining the accuracy of the candidate information classification model to be evaluated comprises:
and determining the accuracy of the candidate information classification model to be evaluated by adopting a cross validation method.
4. The method of claim 2, wherein the determining the accuracy of the candidate information classification model to be evaluated comprises:
for each class of sample data in the set of sample data, determining a weight value associated with the class of sample data, wherein the weight value is a ratio of the number of the class of sample data to the total number of sample data contained in the set of sample data;
and determining the accuracy of the candidate information classification model to be evaluated according to the determined weight value.
5. The method according to claim 4, wherein the determining the accuracy of the candidate information classification model to be evaluated according to the determined weight value comprises:
the following processing steps are executed in a loop for a predetermined number of times: dividing the sample data set into a training set and a test set, training the candidate information classification model to be evaluated based on the training set and a category associated with each training sample data in the training set respectively by using a machine learning method to obtain a trained candidate information classification model, predicting the category of each test sample data in the test set by using the trained candidate information classification model to obtain a prediction result, determining the prediction accuracy rate associated with the category test sample data based on the prediction result for each category test sample data in the test set, taking the product of a weight value associated with the sample data in the same category as the category test sample data and the prediction accuracy rate as a weighted prediction accuracy rate associated with the category test sample data, and recovering the candidate information classification model to be evaluated to a state without being evaluated, wherein the sample data set and the test set comprise the same category test sample data, and the accuracy rate associated with the category test sample data in the test set is taken as a first weighted prediction accuracy rate associated with the candidate information classification model to be evaluated is taken as a first weighted prediction accuracy rate of the number of the training sample data classification model after the training sample data set and the test set comprise the same category test sample data, and the correct number of the prediction accuracy rate of the training sample data classification model;
and taking the average value of the obtained predetermined number of first weighted prediction accuracy rates as the accuracy rate of the candidate information classification model to be evaluated.
6. The method according to any one of claims 2-5, wherein said determining a target information classification model among said pre-stored candidate information classification models based on said determined accuracy comprises:
and taking the candidate information classification model with the highest accuracy in the pre-stored candidate information classification models as a target information classification model.
7. An information output apparatus, characterized in that the apparatus comprises:
the receiving unit is configured to receive information sent by the client;
the extraction unit is configured to extract a feature vector from the information, wherein the feature vector is used for representing the content of the information;
the classification unit is configured to import the feature vectors into a pre-trained information classification model for classification to obtain a class to which the information belongs, wherein the information classification model is obtained by utilizing the accuracy of a model determined after an acquired sample data set is trained on the basis of a plurality of pre-stored untrained models, the accuracy is determined on the basis of a determined weight value, and the weight value comprises the ratio of the number of sample data of each class in the sample data set to the total number of the sample data contained in the sample data set;
and the output unit is configured to match target feedback information associated with the information from a pre-stored feedback information set indicated by the category and output the target feedback information to the client.
8. The apparatus of claim 7, further comprising: the information classification model establishing unit is configured to establish an information classification model, and comprises the following steps:
the acquisition subunit is configured to acquire a sample data set and a category associated with each sample data in the sample data set;
the accuracy determining subunit is configured to, for each candidate information classification model in pre-stored candidate information classification models, use the candidate information classification model as a candidate information classification model to be evaluated, and determine the accuracy of the candidate information classification model to be evaluated based on the sample data set and a category associated with each sample data in the sample data set, wherein the pre-stored candidate information classification model is an untrained model;
a target information classification model determination subunit configured to determine a target information classification model among the pre-stored candidate information classification models based on the determined accuracy;
and the training subunit is configured to train the target information classification model to obtain an information classification model based on the sample data set and the category respectively associated with each sample data in the sample data set by using a machine learning method.
9. A server, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710063291.2A CN108388563B (en) | 2017-02-03 | 2017-02-03 | Information output method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710063291.2A CN108388563B (en) | 2017-02-03 | 2017-02-03 | Information output method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108388563A CN108388563A (en) | 2018-08-10 |
| CN108388563B true CN108388563B (en) | 2022-11-08 |
Family
ID=63075842
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710063291.2A Active CN108388563B (en) | 2017-02-03 | 2017-02-03 | Information output method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108388563B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109146076A (en) * | 2018-08-13 | 2019-01-04 | 东软集团股份有限公司 | model generating method and device, data processing method and device |
| CN109376868B (en) * | 2018-09-30 | 2021-06-25 | 北京字节跳动网络技术有限公司 | Information management system |
| CN109409419B (en) * | 2018-09-30 | 2021-05-07 | 北京字节跳动网络技术有限公司 | Method and apparatus for processing data |
| CN109559239A (en) * | 2018-11-26 | 2019-04-02 | 泰康保险集团股份有限公司 | Generation method, device, electronic equipment, storage medium are suggested in complaint handling |
| CN109670111B (en) * | 2018-12-20 | 2021-04-27 | 北京字节跳动网络技术有限公司 | Method and device for pushing information |
| CN109740156B (en) * | 2018-12-28 | 2023-08-04 | 北京金山安全软件有限公司 | Feedback information processing method and device, electronic equipment and storage medium |
| CN110909074A (en) * | 2019-10-21 | 2020-03-24 | 北京海益同展信息科技有限公司 | Method and device for processing social data, computer equipment and storage medium |
| CN111159465B (en) * | 2019-12-31 | 2023-09-29 | 杭州网易云音乐科技有限公司 | Song classification method and device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101127029A (en) * | 2007-08-24 | 2008-02-20 | 复旦大学 | Method for training SVM classifier in large scale data classification |
| CN102254193A (en) * | 2011-07-16 | 2011-11-23 | 西安电子科技大学 | Relevance vector machine-based multi-class data classifying method |
| CN102799635A (en) * | 2012-06-27 | 2012-11-28 | 天津大学 | Image set ordering method driven by user |
| CN103136377A (en) * | 2013-03-26 | 2013-06-05 | 重庆邮电大学 | Chinese text classification method based on evolution super-network |
| CN103425677A (en) * | 2012-05-18 | 2013-12-04 | 阿里巴巴集团控股有限公司 | Method for determining classified models of keywords and method and device for classifying keywords |
| CN104809434A (en) * | 2015-04-22 | 2015-07-29 | 哈尔滨工业大学 | Sleep staging method based on single-channel electroencephalogram signal ocular artifact removal |
| CN105447750A (en) * | 2015-11-17 | 2016-03-30 | 小米科技有限责任公司 | Information identification method, apparatus, terminal and server |
| CN105786969A (en) * | 2016-02-01 | 2016-07-20 | 百度在线网络技术（北京）有限公司 | Information display method and apparatus |
| CN106156809A (en) * | 2015-04-24 | 2016-11-23 | 阿里巴巴集团控股有限公司 | For updating the method and device of disaggregated model |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103914767B (en) * | 2013-01-06 | 2017-02-22 | 华为技术有限公司 | Meeting scheduling method, device and system |
| CN103258147B (en) * | 2013-05-24 | 2016-12-28 | 重庆邮电大学 | A kind of parallel evolution super-network DNA micro array gene data categorizing system based on GPU and method |
| BR102014023780B1 (en) * | 2014-09-25 | 2023-04-18 | Universidade Estadual De Campinas - Unicamp (Br/Sp) | METHOD FOR MULTICLASS CLASSIFICATION IN OPEN SCENARIOS AND USES OF THE SAME |
| CN104537252B (en) * | 2015-01-05 | 2019-09-17 | 深圳市腾讯计算机系统有限公司 | User Status list disaggregated model training method and device |
| CN105630917A (en) * | 2015-12-22 | 2016-06-01 | 成都小多科技有限公司 | Intelligent answering method and intelligent answering device |
| CN105893465B (en) * | 2016-03-28 | 2019-12-27 | 北京京东尚科信息技术有限公司 | Automatic question answering method and device |
| CN105930723A (en) * | 2016-04-20 | 2016-09-07 | 福州大学 | Intrusion detection method based on feature selection |
-
2017
- 2017-02-03 CN CN201710063291.2A patent/CN108388563B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101127029A (en) * | 2007-08-24 | 2008-02-20 | 复旦大学 | Method for training SVM classifier in large scale data classification |
| CN102254193A (en) * | 2011-07-16 | 2011-11-23 | 西安电子科技大学 | Relevance vector machine-based multi-class data classifying method |
| CN103425677A (en) * | 2012-05-18 | 2013-12-04 | 阿里巴巴集团控股有限公司 | Method for determining classified models of keywords and method and device for classifying keywords |
| CN102799635A (en) * | 2012-06-27 | 2012-11-28 | 天津大学 | Image set ordering method driven by user |
| CN103136377A (en) * | 2013-03-26 | 2013-06-05 | 重庆邮电大学 | Chinese text classification method based on evolution super-network |
| CN104809434A (en) * | 2015-04-22 | 2015-07-29 | 哈尔滨工业大学 | Sleep staging method based on single-channel electroencephalogram signal ocular artifact removal |
| CN106156809A (en) * | 2015-04-24 | 2016-11-23 | 阿里巴巴集团控股有限公司 | For updating the method and device of disaggregated model |
| CN105447750A (en) * | 2015-11-17 | 2016-03-30 | 小米科技有限责任公司 | Information identification method, apparatus, terminal and server |
| CN105786969A (en) * | 2016-02-01 | 2016-07-20 | 百度在线网络技术（北京）有限公司 | Information display method and apparatus |
Non-Patent Citations (2)
| Title |
|---|
| 基于高斯混合模型的遥感影像半监督分类;熊彪等;《武汉大学学报(信息科学版)》;20110105(第01期);第112-116页 * |
| 多核学习方法在分类中的应用研究;崔清亮等;《科学技术与工程》;20131118(第32期);第61-65页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108388563A (en) | 2018-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108388563B (en) | Information output method and device | |
| CN109460513B (en) | Method and apparatus for generating click rate prediction model | |
| CN108520470B (en) | Method and apparatus for generating user attribute information | |
| CN110543946B (en) | Method and apparatus for training a model | |
| CN108197652B (en) | Method and apparatus for generating information | |
| CN107944481B (en) | Method and apparatus for generating information | |
| CN108960316B (en) | Method and apparatus for generating a model | |
| CN108989882B (en) | Method and apparatus for outputting music pieces in video | |
| CN109976997B (en) | Test method and device | |
| CN107609506B (en) | Method and apparatus for generating image | |
| CN109740018B (en) | Method and device for generating video label model | |
| CN108520324A (en) | Method and apparatus for generating information | |
| CN106897905B (en) | Method and device for pushing information and electronic equipment | |
| CN110555451A (en) | information identification method and device | |
| CN109961032B (en) | Method and apparatus for generating classification model | |
| CN109214501B (en) | Method and apparatus for identifying information | |
| CN110070076B (en) | Method and device for selecting training samples | |
| CN111695041B (en) | Method and device for recommending information | |
| CN113743971A (en) | Data processing method and device | |
| CN111125502B (en) | Method and device for generating information | |
| CN112308648A (en) | Information processing method and device | |
| CN112149699A (en) | Method and device for generating model and method and device for recognizing image | |
| CN113742593A (en) | Method and device for pushing information | |
| CN107451860B (en) | Information pushing method and device | |
| CN111126649A (en) | Method and apparatus for generating information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |