CN115438267A - Portrait analysis processing method and system based on big data - Google Patents
Portrait analysis processing method and system based on big data Download PDFInfo
- Publication number
- CN115438267A CN115438267A CN202211183703.3A CN202211183703A CN115438267A CN 115438267 A CN115438267 A CN 115438267A CN 202211183703 A CN202211183703 A CN 202211183703A CN 115438267 A CN115438267 A CN 115438267A
- Authority
- CN
- China
- Prior art keywords
- big data
- user session
- service preference
- data example
- mining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Computational Mathematics (AREA)
- Algebra (AREA)
- Probability & Statistics with Applications (AREA)
- Operations Research (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides an image analysis processing method and system based on big data, wherein the method comprises the following steps: responding to the portrait analysis processing request, and obtaining user session big data to be processed; loading the user session big data to be processed to a service preference mining model which finishes debugging to obtain mining information of target service preference in the user session big data to be processed; the service preference mining model is obtained based on debugging of multi-class user session big data examples. Therefore, the service preference mining model is obtained based on the multi-class user session big data examples in a debugging mode, the sensitivity of the service preference mining model to the target service preference can be guaranteed, the target service preference is rapidly and accurately mined from the user session big data to be processed, and the accuracy and the efficiency of service preference mining are improved.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to an image analysis processing method and system based on big data.
Background
In the big data era, not only can ordinary users enjoy the convenience brought by the technology, but also enterprises can extract information with commercial value from the data to construct user figures, thereby analyzing and predicting user behaviors. While user portrayal is not a fresh idea, the advent of big data technology has made user portrayal more clear and objective.
The user representation can be understood as user information labeling, such as collecting various data and behaviors of the user, so as to obtain some basic information and typical characteristics of the user, and finally forming a character prototype. User profile analysis typically involves 3 dimensions, namely basic attributes (user basic information), business service circles (user preference area for business services, preference type, preference manner, etc.), and social circles (social relationship network).
Currently, it is one of the current working focuses on how to further improve and optimize the image analysis technology aiming at the endless image analysis technology.
Disclosure of Invention
In order to solve the technical problems in the related art, the invention provides an image analysis processing method and system based on big data.
In a first aspect, an embodiment of the present invention provides an image analysis processing method based on big data, which is applied to a big data processing system, and the method includes: responding to the portrait analysis processing request, and obtaining user session big data to be processed; loading the user session big data to be processed to a service preference mining model which finishes debugging to obtain mining information of target service preference in the user session big data to be processed; the service preference mining model is obtained based on debugging of multi-class user session big data examples.
Therefore, the service preference mining model is obtained based on the multi-class user session big data examples in a debugging mode, the sensitivity of the service preference mining model to the target service preference can be guaranteed, the target service preference is rapidly and accurately mined from the user session big data to be processed, and the accuracy and the efficiency of service preference mining are improved.
In some exemplary aspects, the debugging step of the service preference mining model includes: obtaining a first user session big data example and a second user session big data example; the first user session big data example is collected by a first data server and contains a historical service preference annotation, the second user session big data example is collected by a second data server and does not contain a target service preference annotation; debugging the pre-debugged service preference mining model based on the first user session big data example and the second user session big data example to obtain a debugged service preference mining model; the debugged service preference mining model is used for performing service preference mining on the user session big data to be processed to obtain mining information of target service preference in the user session big data to be processed.
It can be seen that, on the basis of obtaining the first user session big data example and the second user session big data example, the pre-debugged service preference mining model can be debugged through the first user session big data example and the second user session big data example to obtain a service preference mining model for performing service preference mining on the to-be-processed user session big data.
When the service preference mining model is debugged, a data acquisition scene of a first data server (such as a cloud server) for acquiring a first user session big data example can be used as a basic scene, a data acquisition scene of a second data server for acquiring a second user session big data example is used as a target scene, wherein the difficulty for acquiring the user session big data of the target scene is low, so that the debugged service preference mining model can show high-precision and credibility service mining performance for the basic scene and can still show high-precision and credibility service mining performance for the target scene on the basis of transforming a session interest knowledge vector which is not labeled under the related target scene into a labeled session interest knowledge vector under the basic scene, and based on the target service preference mining model, the target service preference mining model can be timely mined to carry out big data recommendation based on the target service preference, so that resources additionally consumed by the label in the model debugging process are avoided, and the usability of the whole scheme is improved.
In some exemplary schemes, the debugging the pre-debugged service preference mining model based on the first user session big data example and the second user session big data example to obtain a debugged service preference mining model, includes: on the basis that the service preference mining model loaded to the pre-debugging currently comprises a first user session big data example, model debugging is carried out on the pre-debugging service preference mining model by combining the first user session big data example and historical service preference comments contained in the first user session big data example; on the basis that the service preference mining model loaded to the pre-debugging currently comprises a second user session big data example, performing service preference mining on the second user session big data example by combining the pre-debugging service preference mining model to obtain mining information, and performing model debugging on the pre-debugging service preference mining model by combining the second user session big data example containing disturbance service preference annotation by taking the mining information as the disturbance service preference annotation of the second user session big data example; when the debugging end requirement is met, obtaining a service preference mining model for completing debugging:
it can be seen that the handling can be flexible for different user session big data examples. For the first user session big data example with the historical service preference annotations, model debugging can be carried out based on the first user session big data example and the contained historical service preference annotations; for the second user session big data example which does not contain the target service preference annotation, the disturbance service preference annotation can be determined by combining the service preference mining model, and the relevant model debugging of the non-core second user session big data example is carried out on the basis of the disturbance service preference annotation, so that richer session interest knowledge vectors of the user session big data example can be extracted and obtained during the whole model debugging, and the high-quality service preference mining model can be obtained.
In some exemplary aspects, the method further comprises: obtaining a third user session big data example; the third user session big data example is collected by a second data server and contains target service preference annotations, the number of the third user session big data examples is less than that of the second user session big data examples, and the number difference between the second user session big data examples and the third user session big data examples is greater than a set judgment value; and on the basis that the service preference mining model loaded to the pre-debugging currently is a third user session big data example, performing model debugging on the pre-debugging service preference mining model by combining the third user session big data example and a target service preference annotation contained in the third user session big data example.
Therefore, the model debugging can be performed by combining the third user conversation big data example containing the target service preference annotation, the conversation interest knowledge vector extraction performance of the debugged service preference mining model on the target scene is further improved, the service preference mining precision is guaranteed, based on the fact that the number of the third user conversation big data example containing the target service preference annotation is obviously lower than that of the second user conversation big data example not containing the target service preference annotation, for the collection of the debugging example, the collection complexity of the debugging example can be reduced, and the debugging timeliness of the whole model is further improved.
In some exemplary aspects, the service preference mining model includes a knowledge mining submodel and a multivariate regression submodel; the debugging of the pre-debugged service preference mining model based on the first user session big data example and the second user session big data example comprises: the pre-debugged service preference mining model is debugged by a first user session big data example containing historical service preference annotations, a second user session big data example containing perturbation service preference annotations, and a third user session big data example containing target service preference annotations.
In some example aspects, the debugging the pre-debugged service preference mining model by the first user session big data example containing the historical service preference annotations, the second user session big data example containing the perturbation service preference annotations, and the third user session big data example containing the target service preference annotations comprises: loading the first user session big data example, the second user session big data example and the third user session big data example to a knowledge mining sub-model included by the pre-debugged service preference mining model, and respectively mining a first session interest knowledge vector corresponding to the first user session big data example, a second session interest knowledge vector corresponding to the second user session big data example and a third user session big data example corresponding to the third user session big data example; loading the first session interest knowledge vector, the second session interest knowledge vector and the third session interest knowledge vector to a multivariate regression submodel included in the pre-debugged service preference mining model respectively to obtain mining information about the first user session big data example, mining information about the second user session big data example and mining information about the third user session big data example; determining a first comparison result between the mining information of the first user session big data example and a historical service preference annotation contained in the first user session big data example, a second comparison result between the mining information of the second user session big data example and a perturbation service preference annotation contained in the second user session big data example, and a third comparison result between the mining information of the third user session big data example and a target service preference annotation contained in the third user session big data example; debugging the pre-debugging service preference mining model based on the first comparison result, the second comparison result and the third comparison result.
In some exemplary aspects, the debugging the pre-debugged service preference mining model based on the first comparison result, the second comparison result, and the third comparison result comprises: determining a mining cost index of the service preference mining model based on the first comparison result, the second comparison result and the third comparison result; and debugging the pre-debugged service preference mining model based on the mining cost index.
In some exemplary aspects, the service preference mining model further comprises an intermediate submodel configured between the knowledge mining submodel and the multivariate regression submodel; determining a mining cost index of the service preference mining model based on the first comparison result, the second comparison result and the third comparison result, including: determining a commonality score between the first session interest knowledge vector and the third session interest knowledge vector based on the middle sub-model based on determining that the first user session big data example and the third user session big data example include a same type of service preference; determining a mining cost index of the service preference mining model based on the commonality score, the first comparison result, the second comparison result, and the third comparison result.
Therefore, in view of the relevance of session interest knowledge vectors of the same service preference in different session environments, the commonality score between the first session interest knowledge vector and the third session interest knowledge vector can be determined based on the intermediate submodel, the larger the commonality score is, the higher the probability that the service preferences corresponding to the two session interest knowledge vectors are the same service preference is, otherwise, the smaller the commonality score is, the lower the probability that the service preferences corresponding to the two session interest knowledge vectors are the same service preference is, and based on the joint limitation of the commonality score and the comparison result, the determined mining cost index can be subjected to model improvement which is as accurate and reliable as possible, and the debugging quality is improved.
In some example aspects, the knowledge mining submodel includes a knowledge refinement node and a depth residual processing node; mining a first session interest knowledge vector corresponding to the first user session big data example according to the following thought: loading the first user session big data example to a knowledge extraction node included in the knowledge mining submodel to obtain a first initial session interest knowledge vector generated by the knowledge extraction node; and loading the first initial session interest knowledge vector generated by the knowledge refining node to a deep residual error processing node included in the knowledge mining submodel to obtain a plurality of first session interest knowledge vectors of different scales generated by the deep residual error processing node, and using the first session interest knowledge vectors as the first session interest knowledge vectors corresponding to the first user session big data example.
Therefore, the mining of the first session interest knowledge vectors of different scales is realized by combining the deep residual error processing node, so that the detail content contained in the mined first session interest knowledge vectors is more sufficient, the service preference is conveniently and accurately mined, and the mining quality is guaranteed.
In some exemplary aspects, the second user session big data example or the third user session big data example is obtained according to the following idea: obtaining user session big data examples obtained in each obtaining mode by changing the obtaining mode of the second data server relative to the target service preference, wherein the user session big data examples comprise the target service preference; or, by changing the preference event state of the target service preference relative to the second data server, obtaining a corresponding user session big data example, where the user session big data example includes the target service preference in each preference event state.
In some example aspects, the debugging the pre-debugged service preference mining model by the first user session big data example containing the historical service preference annotations, the second user session big data example containing the perturbation service preference annotations, and the third user session big data example containing the target service preference annotations comprises: selecting a user session big data example comprising the same service preference from the first user session big data example, the second user session big data example and the third user session big data example; and taking the screened user session big data example as a debugging basis of the pre-debugging service preference mining model, and debugging the pre-debugging service preference mining model.
Therefore, when model debugging is performed, separate type analysis is not needed, resource overhead required when labeling is performed on the user session big data examples can be reduced, and timeliness of subsequent service preference mining is improved.
In a second aspect, the present invention also provides a big data processing system, comprising a processor and a memory; the processor is connected with the memory in communication, and the processor is used for reading the computer program from the memory and executing the computer program to realize the method.
In a third aspect, the invention also provides a readable storage medium, on which a program is stored, which program, when executed by a processor, performs the method described above.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a schematic flow chart of a big data-based portrait analysis processing method according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of a communication architecture of an application environment of a big data-based portrait analysis processing method according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided by the embodiments of the present invention may be implemented in a big data processing system, a computer device, or a similar computing apparatus. For example, when running on a big data processing system, the big data processing system 10 may include one or more processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, and optionally, the big data processing system may further include a transmission device 106 for communication function. It will be appreciated by those of ordinary skill in the art that the above-described architecture is merely illustrative and is not intended to be limiting with respect to the architecture of the above-described big data processing system. For example, big data processing system 10 may also include more or fewer components than shown above, or have a different configuration than shown above.
The memory 104 can be used to store computer programs, such as software programs and modules of application software, for example, a computer program corresponding to the image analysis processing method based on big data in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, so as to implement the above method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory located remotely from processor 102, which may be connected to big data processing system 10 over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Examples of such networks may include wireless networks provided by communication providers of big data processing system 10. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
With reference to fig. 1, fig. 1 is a schematic flow chart of an image analysis processing method based on big data according to an embodiment of the present invention, where the method is applied to a big data processing system, and may further include the following technical solutions.
The technical scheme disclosed by the embodiment of the invention can comprise the contents described in the step one and the step two.
Step one, responding to the portrait analysis processing request, and obtaining conversation big data of a user to be processed.
Loading the user session big data to be processed to a service preference mining model which finishes debugging to obtain mining information of target service preference in the user session big data to be processed; the service preference mining model is obtained by debugging a large data example based on multiple types of user sessions.
Therefore, the service preference mining model is obtained based on the multi-class user session big data examples in a debugging mode, the sensitivity of the service preference mining model to the target service preference can be guaranteed, the target service preference is quickly and accurately mined from the user session big data to be processed, and the accuracy and the efficiency of service preference mining are improved.
In the embodiment of the invention, portrait analysis can be understood as interest analysis and preference analysis, and the user session big data to be processed obtained based on the portrait analysis processing request can relate to different business fields, such as various services (e.g., e-commerce, office, game, government and enterprise, etc.), such as VR, AR, MR, etc.
In practical application, the training/debugging of the service preference mining model is a key for guaranteeing the mining accuracy and the reliability of the target service preference, and as for the training/debugging of the service preference mining model, the embodiment of the invention provides the technical scheme described in the following step 101 and step 102.
Step 101: obtaining a first user session big data example and a second user session big data example; the first user session big data example is collected by a first data server, and the first user session big data example contains historical service preference annotations, the second user session big data example is collected by a second data server, and the second user session big data example does not contain target service preference annotations.
Step 102: debugging the pre-debugged service preference mining model through the first user session big data example and the second user session big data example to obtain a debugged service preference mining model; the service preference mining model which is debugged is used for performing service preference mining on the user session big data to be processed to obtain mining information of target service preference in the user session big data to be processed.
In the embodiment of the invention, the target service preference can be understood as a potential service preference or a more personalized service preference, the traditional service preference mining focuses on mining of group-oriented service preferences, the sensitivity to some hidden and personalized service preferences is lower, and the model can be endowed with mining sensitivity and accuracy to such hidden and personalized service preferences (target service preferences) through training and debugging of the model, so that the intelligent degree of service preference mining is improved.
Further, historical service preference annotations may be understood as annotation information for group service preferences, while target service preference annotations may be understood as annotation information for hidden, personalized service preferences, and user session big data examples may be understood as user session big data samples.
In the embodiment of the present invention, a first user session big data example obtained by a first data server configured in advance may be obtained, and a second user session big data example obtained by a second data server configured in advance may also be obtained. The first data server may be a cloud server and the second data server may be a micro server.
The cloud server may obtain the first user session big data example in real time, or may obtain the first user session big data example according to an instruction, where the corresponding first user session big data example generally includes user session big data related to a plurality of service preferences in a data acquisition mode of the first data server where the first user session big data example is located.
Compared with the cloud server, the micro server is more convenient for obtaining the user session big data of the target service preference, and the corresponding second user session big data example can be the user session big data only containing the target service preference. Generally, the second user session big data example acquired in each acquisition mode can be obtained by changing the acquisition mode of the micro server relative to the target service preference; the corresponding second user session big data example may also be obtained by changing the preference event state of the target service preference with respect to the micro server.
In this way, the data acquisition scenario of the second data server may be used as a target scenario, the data acquisition scenario of the first data server is a basic scenario, and currently, there are related annotated user session big data examples (such as a first user session big data example) in the basic scenario, and the data acquisition scenario of the first data server annotates fewer user session big data examples (such as a second user session big data example). The micro server can quickly obtain the big data of the related user session, so that the flexibility and the efficiency of model training and debugging are improved.
The embodiment of the invention fuses the first user session big data example and the second user session big data example for debugging, and the historical service preference annotation contained in the first user session big data example is used as a first debugging reference (which can be a core debugging reference) during debugging, so that the mining information obtained by the service preference mining model aiming at the second user session big data example can be used as a second debugging reference (which can be a non-core debugging reference), and the service preference mining model subjected to debugging can be matched with mining in a basic scene (an initial data acquisition scene or a service interaction scene) and is also suitable for service preference mining in a target scene (a specific data acquisition scene or a service interaction scene).
The service preference mining model in the embodiment of the invention is debugged by a loaded user session big data example and a mapping list aiming at mining information about service preference in the user session big data example, and model variables/parameters of a relevant service preference mining model can be obtained on the basis of obtaining the mapping list by debugging, so that the user session big data to be processed can be directly loaded into the service preference mining model which is debugged on the basis of carrying out service preference mining on the user session big data to be processed to obtain the mining information about the service preference.
It can be understood that the service preference mining model pre-debugged in the embodiment of the present invention may be the created initial AI model, and may also be an AI model subjected to several rounds of debugging.
When the service preference mining model is debugged, if the service preference mining model loaded to the pre-debugging currently comprises a first user session big data example, the pre-debugged service preference mining model can be debugged by combining the first user session big data example and the historical service preference annotation configured for the first user session big data example, in other words, the first user session big data example can be used as a debugging basis of the pre-debugged service preference mining model, and the historical service preference annotation configured for the first user session big data example is used as a debugging reference of a generated result, so that related model debugging is realized; if the currently loaded pre-debugged service preference mining model comprises a second user session big data example which does not contain the target service preference annotation, service preference mining can be carried out on the second user session big data example by combining the pre-debugged service preference mining model to obtain mining information, the mining information is used as the disturbance service preference annotation of the second user session big data example to carry out model debugging on the pre-debugged service preference mining model by combining the second user session big data example containing the disturbance service preference annotation, in other words, estimation analysis can be carried out on the second user session big data example by combining the service preference mining model, and model debugging can be realized based on debugging supervision of the disturbance service preference annotation on the basis of determining that the second user session big data example containing the disturbance service preference annotation needs to be debugged.
It can be appreciated that the mixed debugging of the first user session big data example and the second user session big data example can be cross-debugging of the first user session big data example and the second user session big data example, for example, after a group of the first user session big data examples is debugged, a group of the second user session big data examples can be debugged, the first user session big data example can be debugged first to obtain a modified service preference mining model, then the modified service preference mining model is combined as a service preference mining model for service preference mining of the second user session big data example, the second user session big data example is configured to perturb the service preference annotation through the service preference mining, and then the model variable can be continuously modified after the second user session big data example with the perturbed service preference annotation is combined.
On the basis of carrying out mixed debugging on a related first user session big data example and a related second user session big data example, checking required debugging ending can be carried out after each round of debugging, and therefore a service preference mining model which is debugged is obtained. The debugging end requirement here may be that the debugging reaches a set number of times, and the excavation cost index (the loss function value of the model) tends to be stable.
Generally speaking, in addition to performing model debugging based on a related first user session big data example and a second user session big data example which does not contain a target service preference annotation, model debugging can be performed by combining a small part of a third user session big data example which contains a target service preference annotation, so that session interest knowledge vectors of abundant and diverse target scenes can be extracted as far as possible, and the quality of model debugging is improved. The session interest knowledge vector may be session interest features corresponding to the user session big data, such as behavior features, text features, and the like, which can reflect the user interest/demand.
Further, the number of third user session big data examples is smaller than the number of second user session big data examples, and the difference in the number between the second user session big data examples and the third user session big data examples is larger than a set decision value, in other words, the number of third user session big data examples is significantly lower than the number of second user session big data examples. Further, the third user session big data example may also be collected by the second data server.
In the debugging stage, the debugging can be carried out by combining with a sparse debugging thought, so that the debugging timeliness is improved, and the flexible application of the model is ensured.
Based on the above, in order to facilitate the multiple regression analysis of the service preference, the user session big data examples including the same service preference can be screened from the first user session big data example, the second user session big data example and the third user session big data example in advance, and on the basis of putting the user session big data examples including the same service preference into the same data set, the user session big data examples can be directly used as a debugging basis, that is, the relevant mining window and the actual result of the multiple regression analysis can be obtained by mining once.
In the embodiment of the invention, the pre-debugged service preference mining model can be debugged through the first user session big data example containing the historical service preference annotation, the second user session big data example containing the disturbance service preference annotation and the third user session big data example containing the target service preference annotation, and the model debugging can be exemplarily carried out according to the following ideas: inputting a first user session big data example, a second user session big data example and a third user session big data example into a knowledge mining submodel included in a pre-debugged service preference mining model, and respectively mining a first session interest knowledge vector corresponding to the first user session big data example, a second session interest knowledge vector corresponding to the second user session big data example and a third session interest knowledge vector corresponding to the third user session big data example; respectively loading the first session interest knowledge vector, the second session interest knowledge vector and the third session interest knowledge vector to a multivariate regression sub-model included in a pre-debugging service preference mining model to obtain mining information aiming at a first user session big data example, mining information aiming at a second user session big data example and mining information aiming at a third user session big data example; determining a first comparison result between the mining information of the first user session big data example and the historical service preference annotation contained in the first user session big data example, a second comparison result between the mining information of the second user session big data example and the disturbance service preference annotation contained in the second user session big data example, and a third comparison result between the mining information of the third user session big data example and the target service preference annotation contained in the third user session big data example; and debugging the pre-debugging service preference mining model based on the first comparison result, the second comparison result and the third comparison result.
In the embodiment of the present invention, a knowledge mining submodel included in the service preference mining model may be combined to obtain a first session interest knowledge vector corresponding to a first user session big data example, a second session interest knowledge vector corresponding to a second user session big data example, and a third session interest knowledge vector corresponding to a third user session big data example, respectively. The relevant extracted first session interest knowledge vector may be a session interest knowledge vector related to the first user session big data example, may be description information including angles of session emotion, text and the like, and may also be an area description including service preferences in the user session big data, for example, distribution characteristics of data blocks corresponding to the service preferences in the user session big data. Furthermore, the correlation extracted second and third session interest knowledge vectors are approximated to the first session interest knowledge vector.
Further, on the basis that mining information of the first user session big data example is determined based on a multivariate regression submodel included in the pre-debugged service preference mining model, a first local mining cost index of the pre-debugged service preference mining model can be determined based on a first comparison result between the mining information of the first user session big data example and historical service preference comments included in the first user session big data example, a second local mining cost index can be determined based on a second comparison result between the mining information of the second user session big data example and perturbation service preference comments included in the second user session big data example, a third local mining cost index can be determined based on a third comparison result between the mining information of the third user session big data example and target service preference comments included in the third user session big data example, and the pre-debugged service preference mining model can be debugged based on the mining cost indexes determined by the three local mining cost indexes. Further, the comparison result can be understood as a matching degree or an operation result, and is used for analyzing the difference between the mining information and the related annotation information so as to guide model debugging.
The mining cost index can reflect an error between a result generated by the service preference mining model and the annotation aiming at each user session big data example, the larger the error is, the model operation quality is not ideal, and at the moment, the model variable can be changed through the mining cost index. On the basis of obtaining the modified first AI model, the next round of AI model debugging can be carried out based on the modified service preference mining model, in other words, the user session big data example is loaded into the service preference mining model again for vector mining, and the modified mining cost index can be determined, so that the service preference mining model which is debugged can be obtained until the next round of AI model debugging meets the debugging termination end requirement.
In some design ideas, in order to implement joint mining characteristics in a target scene and a basic scene, an intermediate sub-model may be set between the knowledge mining sub-model and the multivariate regression sub-model, so as to perform vector matching (feature alignment processing) between the basic scene and the target scene through the intermediate sub-model, and exemplarily, the determination of the relevant mining cost index may be performed by combining matched vectors, and the method includes the following steps: determining a commonality score (vector similarity) between the first session interest knowledge vector and the third session interest knowledge vector based on the intermediate submodel based on determining that the first user session big data example and the third user session big data example include the same kind of service preference; and determining a mining cost index of the service preference mining model based on the commonality score, the first comparison result, the second comparison result and the third comparison result.
Therefore, the first session interest knowledge vector and the third session interest knowledge vector can be matched based on debugging and supervision of the same service preference, and the determined mining cost index can be subjected to model improvement as accurate and reliable as possible.
In the embodiment of the present invention, in order to further improve the AI model debugging quality, mining of multiple scale session interest knowledge vectors may be implemented by combining a knowledge refining node (such as a backbone network) and a deep residual processing node (such as a feature pyramid network), and the exemplary implementation may be implemented by: inputting the first user session big data example into a knowledge extraction node included in the knowledge mining submodel to obtain a first initial session interest knowledge vector generated by the knowledge extraction node; and inputting the first initial session interest knowledge vector generated by the knowledge refining node into a deep residual error processing node included in the knowledge mining submodel to obtain a plurality of first session interest knowledge vectors of different scales generated by the deep residual error processing node, and using the first session interest knowledge vectors as first session big data examples corresponding to the first user session big data examples.
By the design, the initial session interest knowledge vectors can be mined by combining the knowledge refining nodes, then a plurality of session interest knowledge vectors of different scales/dimensions are extracted by combining the depth residual error processing nodes, and the accuracy and the reliability of mining of related service preference can be remarkably improved in view of the fact that the session interest knowledge vectors of different scales are suitable for mining of different service preference.
Further, in the embodiment of the present invention, a second session interest knowledge vector corresponding to a second user session big data example and a third session interest knowledge vector corresponding to a third user session big data example may be mined based on the above related ideas.
In general, the depth residual processing node ensures that each model sub-layer can be provided with an intermediate sub-model to realize accurate mining in the feature dimension.
For example, on the basis of summarizing the first user session big data example into a text information set and summarizing the second user session big data example into a text information set, an initial session interest knowledge vector of the relevant user session big data example can be extracted through a knowledge refining node. And after the deep residual error processing node is used, third-order conversation interest knowledge vectors can be mined aiming at the first user conversation big data example and the second user conversation big data example, and a middle sub-model is arranged at each stage of output of the deep residual error processing node to realize matching of the two corresponding conversation interest knowledge vectors, so that mining information is obtained.
On the basis of carrying out comparison analysis on the mining information of the first user session big data example and a priori annotations configured in advance, a mining cost index can be determined, and model variables of a relevant service preference mining model are changed through error feedback.
It will be appreciated that for the second user session big data example, as an unindicated training sample, perturbation service preference annotations (such as pseudo-labels) may be configured by the correlation technique to assist in debugging of the above model.
Under some design ideas which can be independently implemented, after the to-be-processed user session big data is loaded to the service preference mining model which completes debugging, and mining information of the target service preference in the to-be-processed user session big data is obtained, the method can further comprise the following steps: determining a big data recommendation indication aiming at the target service preference based on mining information of the target service preference in the user session big data to be processed; and pushing the big data by using the big data recommendation indication.
For example, the pushing demand prediction can be performed based on mining information of target service preference in the user session big data to be processed, so that a big data recommendation indication is determined according to an obtained prediction result, then reasonable and intelligent big data pushing is realized, and blind pushing and harassing pushing are avoided.
Under some design ideas which can be independently implemented, determining a big data recommendation indication for a target service preference based on mining information of the target service preference in the big data of the user session to be processed may include the following contents: acquiring an event preference knowledge chain and a scene preference knowledge chain in mining information of target service preference in the user session big data to be processed; combining an event preference knowledge chain and a scene preference knowledge chain in the mining information of the target service preference in the user session big data to be processed based on the knowledge chain correlation degree between the event preference knowledge chain and the scene preference knowledge chain in the mining information of the target service preference in the user session big data to be processed to obtain a knowledge chain combination result; determining a scene preference knowledge chain with abnormal combination as a candidate scene preference knowledge chain, and determining a demand element matched with the candidate scene preference knowledge chain according to a knowledge chain commonality value between the scene preference knowledge chain and the candidate scene preference knowledge chain in the knowledge chain combination result; combining the demand elements matched with the candidate scene preference knowledge chain to obtain a demand element combination result; and determining a pushing demand prediction result corresponding to mining information of target service preference in the user session big data to be processed according to the demand element combination result and the knowledge chain combination result, and determining big data recommendation indication by matching the pushing demand prediction result and a preset relational database. Therefore, demand element prediction and knowledge chain combined analysis can be carried out by combining small-range event preference and large-range scene preference, and therefore the pushed demand prediction result is accurately and completely obtained, and the personalized difference of the obtained big data recommendation indication is guaranteed.
Under some design ideas which can be independently implemented, the acquiring an event preference knowledge chain and a scene preference knowledge chain in mining information of target service preferences in the user session big data to be processed includes: acquiring not less than two event preference data and not less than two scene preference data in mining information of target service preference in the user session big data to be processed; acquiring the matching degree of the event preference data and the difference value of the event preference data between the at least two pieces of event preference data, and acquiring the matching degree of the scene preference data and the difference value of the scene preference data between the at least two pieces of scene preference data; combining the at least two event preference data according to the event preference data matching degree and the difference value of the event preference data to obtain an event preference knowledge chain in mining information of target service preference in the user session big data to be processed; an event preference knowledge chain comprising at least one event preference data; splicing the at least two pieces of scene preference data according to the matching degree of the scene preference data and the difference value of the scene preference data to obtain a scene preference knowledge chain in mining information of target service preference in the user session big data to be processed; a scene preference knowledge chain includes at least one scene preference data. By the design, the event preference knowledge chain and the scene preference knowledge chain can be completely determined, and the event preference knowledge chain and the scene preference knowledge chain are prevented from being lost.
Based on the same or similar inventive concepts, please refer to fig. 2, which further provides a schematic structural diagram of an application environment 30 of a portrait analysis processing method based on big data, including a big data processing system 10 and a user session terminal 20 that communicate with each other, where the big data processing system 10 and the user session terminal 20 implement or partially implement the technical solution described in the above method embodiment when running.
Further, a readable storage medium is provided, on which a program is stored which, when being executed by a processor, carries out the above-mentioned method.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus and method embodiments described above are illustrative only, as the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, the functional modules in the embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention or a part thereof, which essentially contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a media service server 10, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes. It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. An image analysis processing method based on big data is applied to a big data processing system, and the method comprises the following steps:
responding to the portrait analysis processing request, and obtaining user session big data to be processed;
loading the user session big data to be processed to a service preference mining model which finishes debugging to obtain mining information of target service preference in the user session big data to be processed; the service preference mining model is obtained by debugging a large data example based on multiple types of user sessions.
2. The method of claim 1, wherein the step of debugging the service preference mining model comprises:
obtaining a first user session big data example and a second user session big data example; the first user session big data example is collected by a first data server and contains a historical service preference annotation, the second user session big data example is collected by a second data server and does not contain a target service preference annotation;
and debugging the pre-debugging service preference mining model based on the first user session big data example and the second user session big data example to obtain a debugged service preference mining model.
3. The method according to claim 2, wherein the debugging the pre-debugged service preference mining model based on the first user session big data example and the second user session big data example to obtain a debugged service preference mining model, and the debugging the service preference mining model comprises:
on the basis that the service preference mining model loaded to the pre-debugging currently comprises a first user session big data example, model debugging is carried out on the pre-debugging service preference mining model by combining the first user session big data example and historical service preference comments contained in the first user session big data example;
on the basis that a service preference mining model loaded to be debugged currently comprises a second user session big data example, performing service preference mining on the second user session big data example by combining the service preference mining model to be debugged to obtain mining information, and performing model debugging on the service preference mining model to be debugged by combining a second user session big data example containing disturbance service preference annotation by taking the mining information as the disturbance service preference annotation of the second user session big data example;
and when the debugging end requirement is met, obtaining the service preference mining model for completing debugging.
4. The method of claim 1, further comprising:
obtaining a third user session big data example; the third user session big data example is collected by a second data server and contains target service preference annotations, the number of the third user session big data examples is less than that of the second user session big data examples, and the number difference between the second user session big data examples and the third user session big data examples is greater than a set judgment value;
and on the basis that the service preference mining model loaded to the pre-debugging currently is a third user session big data example, performing model debugging on the pre-debugging service preference mining model by combining the third user session big data example and a target service preference annotation contained in the third user session big data example.
5. The method of claim 1, wherein the service preference mining model comprises a knowledge mining submodel and a multivariate regression submodel; the debugging a pre-debugged service preference mining model based on the first user session big data example and the second user session big data example comprises:
the pre-debugged service preference mining model is debugged by a first user session big data example containing historical service preference annotations, a second user session big data example containing perturbation service preference annotations, and a third user session big data example containing target service preference annotations.
6. The method of claim 5, wherein the debugging the pre-debugged service preference mining model by a first user session big data example containing historical service preference annotations, a second user session big data example containing perturbation service preference annotations, and a third user session big data example containing target service preference annotations, comprises:
loading the first user session big data example, the second user session big data example and the third user session big data example to a knowledge mining sub-model included in the pre-debugged service preference mining model, and respectively mining a first session interest knowledge vector corresponding to the first user session big data example, a second session interest knowledge vector corresponding to the second user session big data example and a third session interest knowledge vector corresponding to the third user session big data example;
loading the first session interest knowledge vector, the second session interest knowledge vector and the third session interest knowledge vector to a multivariate regression sub-model included in the pre-debugged service preference mining model respectively to obtain mining information about the first user session big data example, mining information about the second user session big data example and mining information about the third user session big data example;
determining a first comparison result between the mining information of the first user session big data example and a historical service preference annotation contained in the first user session big data example, a second comparison result between the mining information of the second user session big data example and a perturbation service preference annotation contained in the second user session big data example, and a third comparison result between the mining information of the third user session big data example and a target service preference annotation contained in the third user session big data example;
debugging the pre-debugged service preference mining model based on the first comparison result, the second comparison result and the third comparison result;
wherein the debugging the pre-debugged service preference mining model based on the first comparison result, the second comparison result, and the third comparison result comprises: determining a mining cost index of the service preference mining model based on the first comparison result, the second comparison result and the third comparison result; debugging the pre-debugged service preference mining model based on the mining cost index;
wherein the service preference mining model further comprises an intermediate sub-model configured between the knowledge mining sub-model and the multiple regression sub-model; determining a mining cost index of the service preference mining model based on the first comparison result, the second comparison result and the third comparison result, including: upon determining that the first user session big data example and the third user session big data example include a same service preference, determining a commonality score between the first session interest knowledge vector and the third session interest knowledge vector based on the intermediary submodel; determining a mining cost index of the service preference mining model based on the commonality score, the first comparison result, the second comparison result and the third comparison result.
7. The method of claim 1, wherein the knowledge mining submodel comprises a knowledge refinement node and a deep residual processing node; mining a first session interest knowledge vector corresponding to the first user session big data example according to the following thought:
loading the first user session big data example to a knowledge refining node included in the knowledge mining submodel to obtain a first initial session interest knowledge vector generated by the knowledge refining node;
and loading the first initial session interest knowledge vector generated by the knowledge refining node to a deep residual error processing node included in the knowledge mining submodel to obtain a plurality of first session interest knowledge vectors of different scales generated by the deep residual error processing node, and using the first session interest knowledge vectors as the first session interest knowledge vectors corresponding to the first user session big data example.
8. The method in accordance with claim 4, wherein the second user session big data example or the third user session big data example is obtained according to the following idea:
obtaining user session big data examples obtained in each obtaining mode by changing the obtaining mode of the second data server relative to the target service preference, wherein the user session big data examples comprise the target service preference;
or obtaining a corresponding user session big data example by changing the preference event state of the target service preference relative to the second data server, wherein the user session big data example comprises the target service preference in each preference event state.
9. The method of claim 7, wherein debugging the pre-debugged service preference mining model by a first user session big data example containing historical service preference annotations, a second user session big data example containing perturbation service preference annotations, and a third user session big data example containing target service preference annotations comprises:
selecting a user session big data example comprising the same service preference from the first user session big data example, the second user session big data example and the third user session big data example;
and taking the screened user session big data example as a debugging basis of the pre-debugging service preference mining model, and debugging the pre-debugging service preference mining model.
10. A big data processing system, comprising a processor and a memory; the processor is in communication with the memory, and the processor is configured to read the computer program from the memory and execute the computer program to implement the method of any one of claims 1 to 9.
11. A readable storage medium, having stored thereon a program which, when executed by a processor, performs the method described above.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211183703.3A CN115438267A (en) | 2022-09-27 | 2022-09-27 | Portrait analysis processing method and system based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211183703.3A CN115438267A (en) | 2022-09-27 | 2022-09-27 | Portrait analysis processing method and system based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115438267A true CN115438267A (en) | 2022-12-06 |
Family
ID=84250097
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211183703.3A Withdrawn CN115438267A (en) | 2022-09-27 | 2022-09-27 | Portrait analysis processing method and system based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115438267A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116975455A (en) * | 2023-09-24 | 2023-10-31 | 太仓市律点信息技术有限公司 | User interest identification method and device applied to artificial intelligence |
-
2022
- 2022-09-27 CN CN202211183703.3A patent/CN115438267A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116975455A (en) * | 2023-09-24 | 2023-10-31 | 太仓市律点信息技术有限公司 | User interest identification method and device applied to artificial intelligence |
| CN116975455B (en) * | 2023-09-24 | 2023-12-22 | 太仓市律点信息技术有限公司 | User interest recognition method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110020424B (en) | Contract information extraction method and device and text information extraction method | |
| CN114840486B (en) | User behavior data acquisition method and system and cloud platform | |
| CN110807085B (en) | Fault information query method and device, storage medium and electronic device | |
| CN112668586B (en) | Model training method, picture processing device, storage medium, and program product | |
| CN114357197B (en) | Event reasoning method and device | |
| CN115438267A (en) | Portrait analysis processing method and system based on big data | |
| CN112364622A (en) | Dialog text analysis method, dialog text analysis device, electronic device and storage medium | |
| CN115760216A (en) | Order data analysis method and system based on artificial intelligence | |
| CN110378739B (en) | Data traffic matching method and device | |
| CN109741739A (en) | A kind of application processing method and device | |
| CN116883035A (en) | Service matching method based on user grouping statistics | |
| CN115495656A (en) | User demand mining method and system applied to big data | |
| CN116363457A (en) | Task processing, image classification and data processing method of task processing model | |
| CN113327154B (en) | E-commerce user message pushing method and system based on big data | |
| CN115712843A (en) | Data matching detection processing method and system based on artificial intelligence | |
| CN115577037A (en) | Data visualization interactive processing method and system based on artificial intelligence | |
| CN112860759B (en) | Big data mining method based on block chain security authentication and cloud authentication service system | |
| CN115129976A (en) | Resource recall method, device, equipment and storage medium | |
| CN115658675B (en) | Noise optimization method and AI system applied to data processing | |
| CN115658454B (en) | E-commerce information processing method and artificial intelligence system applied to deep learning | |
| CN115174633B (en) | Industrial Internet service data processing method and system and cloud platform | |
| CN117910963A (en) | RPA robot recommendation method, recommendation device and computer readable medium | |
| CN116975455B (en) | User interest recognition method and device | |
| CN117667979B (en) | Data mining method, device, equipment and medium based on large language model | |
| CN114048104A (en) | Monitoring method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230419 Address after: No. 60 Yihe Road, Luolong Street Office, Chenggong District, Kunming City, Yunnan Province, 650500 Applicant after: Yunnan Feisi Technology Co.,Ltd. Address before: No. 69, Hongfeng Road, Chenggong District, Kunming, Yunnan 650500 Applicant before: Huang Hao |
|
| TA01 | Transfer of patent application right | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20221206 |
|
| WW01 | Invention patent application withdrawn after publication |