CN116402249A - Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation device, storage medium and program product - Google Patents
Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation device, storage medium and program product Download PDFInfo
- Publication number
- CN116402249A CN116402249A CN202310219162.3A CN202310219162A CN116402249A CN 116402249 A CN116402249 A CN 116402249A CN 202310219162 A CN202310219162 A CN 202310219162A CN 116402249 A CN116402249 A CN 116402249A
- Authority
- CN
- China
- Prior art keywords
- user data
- recommendation system
- overflow effect
- saturation
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000694 effects Effects 0.000 title claims abstract description 169
- 238000011156 evaluation Methods 0.000 title abstract description 90
- 238000012545 processing Methods 0.000 claims abstract description 48
- 230000006399 behavior Effects 0.000 claims description 94
- 238000000034 method Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000009286 beneficial effect Effects 0.000 abstract description 4
- 238000005259 measurement Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 11
- 238000002474 experimental method Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013488 ordinary least square regression Methods 0.000 description 2
- 238000010835 comparative analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Strategic Management (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Physics (AREA)
- Operations Research (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Entrepreneurship & Innovation (AREA)
- Computational Mathematics (AREA)
- Economics (AREA)
- Quality & Reliability (AREA)
- General Business, Economics & Management (AREA)
- Educational Administration (AREA)
- Bioinformatics & Computational Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Development Economics (AREA)
- Evolutionary Biology (AREA)
- Game Theory and Decision Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Tourism & Hospitality (AREA)
- Algebra (AREA)
- Marketing (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a recommendation system overflow effect evaluation method, a recommendation system overflow effect evaluation device, a recommendation system overflow effect evaluation storage medium and a recommendation system overflow effect evaluation program product, wherein the recommendation system overflow effect evaluation method comprises the following steps: clustering user data of a recommendation system according to preset user attributes; randomly distributing different saturations for a plurality of clusters, and dividing user data in the clusters into an experimental group and a control group according to the saturations; acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in an experimental group or a control group, whether the saturation of the cluster where the user data is located is 0 or not, and the saturation of the cluster where the user data is located; and calculating the overflow effect according to the saturation of the cluster where the user data is located from the value of 0 to the value of 1 and the change amount of the preset user behavior index. The embodiment of the invention realizes the measurement of the overflow effect in the evaluation of the recommendation system, enhances the comprehensiveness of the evaluation of the recommendation system, and is beneficial to improving the confidence coefficient of the evaluation result of the recommendation system.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation equipment, a storage medium and a program product.
Background
In order to improve service efficiency, recommendation systems are designed in many platforms, for example, recommendation systems in residential service platforms recommend house sources for users and display the house sources on pages such as a home page. If the recommended house source and the like are viewed in a large amount, the recommendation effect of the recommendation system is good. In order to gradually improve the recommending effect of the recommending system, the recommending system iterates continuously in practice so as to better realize the recommending effect. In the iteration of the recommendation system, whether the recommendation effect of the new strategy is improved or not and how much is improved need to be evaluated.
Current protocols are typically based on a/B experiments for comparative analysis and significance testing. The A/B experiment uses all users tested as a control group to test by using an old recommended strategy, uses all users tested as an experiment group to test by using a new recommended strategy, and measures the effect of the new recommended strategy by comparing test results. Saturation is defined as the duty cycle of an experimental group within a cluster. Saturation can interfere with the evaluation of the recommendation system, which is known as the overflow effect. In the existing scheme, the saturation is simply and respectively set to 0% and 100%, and no overflow effect is evaluated, so that the confidence of an evaluation result of a recommendation system is poor.
Disclosure of Invention
Aiming at the defects existing in the prior art, the embodiment of the invention provides a recommendation system overflow effect assessment method, recommendation system overflow effect assessment equipment, a storage medium and a program product.
The embodiment of the invention provides a recommendation system overflow effect evaluation method, which comprises the following steps: clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters; randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster; acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is; calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 of the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
According to the embodiment of the invention, the recommendation system overflow effect evaluation method further comprises the following steps: acquiring a full-scale processing effect according to the user data from the change of the preset user behavior index when the control group is changed to the change of the experimental group; the full processing effect represents a change value of the preset user behavior index after the recommendation system is applied.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the preset user behavior index is expressed as follows:
wherein Y is ic Representing the preset user behavior index corresponding to the ith user data in the c-th cluster; t (T) ic Indicating whether the ith user data in the c-th cluster is classified into the experimental group, if so, the experimental group T ic The value is 1, otherwise, the value is 0; s is S ic Indicating whether the saturation of the cluster where the ith user data in the c-th cluster is located is 0, if not, taking the value of 1, otherwise taking the value of 0; pi c Representing the saturation of the c-th cluster; x is X ic Representing the calculation Y ic Is a covariate of (2); beta 1 、β 2 、δ 1 、δ 2 、
Representing the coefficient, beta 0 Represent constant, E ic Representing the error term.
According to the recommendation system overflow effect assessment method provided by the embodiment of the invention, the overflow effect is expressed as delta 2 *π c 。
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the full processing effect is expressed as beta 1 +δ 1 。
According to the method for evaluating the overflow effect of the recommendation system, which is provided by the embodiment of the invention, the user data of the recommendation system are clustered according to the preset user attribute, and the method comprises the following steps: and clustering the user data of the recommendation system according to the user access time or the region where the user is located.
According to the embodiment of the invention, the recommendation system overflow effect evaluation method is used for randomly distributing different saturation degrees for the plurality of clusters and comprises the following steps: the saturation having an arithmetic relationship is randomly allocated to the plurality of clusters.
The embodiment of the invention also provides a recommendation system overflow effect evaluation device, which comprises: a clustering module for: clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters; a grouping module for: randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster; a regression module for: acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is; an evaluation module for: calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 of the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
The embodiment of the invention also provides electronic equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the steps of any one of the recommendation system overflow effect evaluation methods when executing the program.
The embodiments of the present invention also provide a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the recommendation system overflow effect assessment methods described above.
The embodiment of the invention also provides a computer program product, which comprises a computer program, wherein the computer program realizes the steps of the recommendation system overflow effect evaluation method according to any one of the above when being executed by a processor.
According to the recommendation system overflow effect evaluation method, device, storage medium and program product, the user data of the recommendation system are clustered according to the preset user attribute to obtain a plurality of clusters, different saturation levels are randomly distributed to the plurality of clusters, the user data in the clusters are divided into an experiment group and a comparison group according to the saturation levels, the expression of the preset user behavior index corresponding to the user data is obtained through regression processing, the overflow effect is calculated according to the variation of the preset user behavior index when the saturation level of the cluster where the user data is from 0 to 1, the measurement of the overflow effect in the recommendation system evaluation is realized, the comprehensiveness of the recommendation system evaluation is enhanced, and the confidence of the recommendation system evaluation result is improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a recommendation system overflow effect evaluation method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of regression processing results in a recommendation system overflow effect evaluation method according to an embodiment of the present invention;
FIG. 3 is a second schematic diagram of regression processing results in the recommendation system overflow effect evaluation method according to the embodiment of the present invention;
FIG. 4 is a schematic diagram of saturation distribution in a recommendation system overflow effect evaluation method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a recommendation system overflow effect evaluation device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flowchart of a recommendation system overflow effect evaluation method according to an embodiment of the present invention. As shown in fig. 1, the method includes:
step S1, clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters.
In the embodiment of the invention, the recommendation system can be a recommendation system for recommending the house source information through matching of the tenant or other recommendation systems.
Saturation has an impact on the performance assessment of the recommendation system. If only 0%, 100% of extreme saturation is used, reliable evaluation of the recommendation system is not possible. The user data is required to be clustered to obtain a plurality of clusters, and then saturation is distributed to the plurality of clusters, so that reliability evaluation is performed on the effect of the recommendation system based on different saturation.
When the user data is clustered, the clustering can be performed according to the preset user attribute of the user. The selection of the preset user attribute clusters needs to meet the requirement of obtaining clusters with balanced distribution as far as possible, such as obtaining independent clusters with the same distribution.
S2, randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster.
After user data of the recommendation system are clustered according to preset user attributes, different saturation degrees are randomly distributed to the multiple clusters. Saturation is defined as the duty cycle of the experimental group in the cluster. The saturation range is [0,1]. The number of saturations in the set of saturations to be allocated may be determined based on time and operating costs and randomly allocated to the respective clusters.
User data in the clusters are divided into an experimental group and a control group according to the saturation condition of each cluster. I.e. the duty cycle of the experimental group in the cluster needs to be consistent with the value of the saturation of the corresponding cluster. When user data is divided into an experimental group and a control group according to saturation, the randomness of the grouping should be noted.
S3, acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is.
The preset user behavior index can be set according to the specific situation of the recommendation system, and can reflect the effect of the recommendation system, such as average click rate, login times and the like.
The preset user behavior index is related to whether the user data is in an experimental group or a control group, whether the saturation of the cluster where the user data is 0 or not, and the saturation of the cluster where the user data is. Taking a preset user behavior index as a dependent variable, taking whether the saturation of the cluster where the user data is in an experimental group or a control group and the user data is 0 or not, taking the saturation of the cluster where the user data is in as an independent variable, and acquiring an expression of the preset user behavior index according to the influence relation between each independent variable and the preset user behavior index, such as giving coefficients to the independent variables and adding to obtain the expression of the preset user behavior index.
The preset user behavior index may be expressed as a preset user behavior index of a certain cluster corresponding to a certain user data. By carrying out regression processing on the preset user behavior indexes of all user data in all clusters, the unknown quantity in the preset user behavior indexes can be obtained, and then the expression of the preset user behavior indexes corresponding to the user data is determined.
S4, calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 according to the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
The overflow effect is reflected by the change amount of the preset user behavior index when the saturation of the cluster where the user data is located is from 0 to 1, so that the overflow effect can be calculated according to the change amount of the preset user behavior index when the saturation of the cluster where the user data is located is from 0 to 1. According to the expression of the preset user behavior index corresponding to the user data, the preset user behavior index when the saturation of the cluster where the user data is located is 0 and the preset user behavior index when the saturation of the cluster where the user data is located is 1 can be obtained, so that the variation of the preset user behavior index when the saturation of the cluster where the user data is located is from 0 to 1 can be obtained, and the overflow effect can be obtained. The overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index, namely the change condition of the preset user behavior index under the influence of the mutual interference between the experimental group and the control group in the cluster.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the user data of the recommendation system is clustered according to the preset user attribute to obtain a plurality of clusters, different saturation levels are randomly allocated to the plurality of clusters, the user data in the clusters are divided into an experimental group and a control group according to the saturation levels, the expression of the preset user behavior index corresponding to the user data is obtained through regression processing, the overflow effect is calculated according to the change amount of the preset user behavior index when the saturation level of the cluster where the user data is from 0 to 1, the measurement of the overflow effect in the recommendation system evaluation is realized, the comprehensiveness of the recommendation system evaluation is enhanced, and the confidence of the recommendation system evaluation result is improved.
According to the embodiment of the invention, the recommendation system overflow effect evaluation method further comprises the following steps: acquiring a full-scale processing effect according to the user data from the change of the preset user behavior index when the control group is changed to the change of the experimental group; the full processing effect represents a change value of the preset user behavior index after the recommendation system is applied.
The full-scale processing effect represents the change value of the preset user behavior index after the application of the recommendation system, namely the change value of the preset user behavior index after the application of the new strategy of the recommendation system to all users. Before the new strategy is applied, the user data is in a comparison group, namely, the old strategy is utilized; after the new policy is applied, the user data is in the experimental group, i.e. the new policy is utilized. Therefore, the full-scale processing effect can be obtained according to the user data from the change of the preset user behavior index when the control group is changed to the experimental group.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the effect evaluation of the recommendation system under multi-cluster and multi-saturation is realized by acquiring the full processing effect according to the change of the preset user behavior index when the user data changes from the control group to the experimental group, and the reliability of the recommendation system overflow effect evaluation is improved.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the preset user behavior index is expressed as follows:
wherein Y is ic Representing the preset user behavior index corresponding to the ith user data in the c-th cluster; t (T) ic Indicating whether the ith user data in the c-th cluster is classified into the experimental group, if so, the experimental group T ic The value is 1, otherwise, the value is 0; s is S ic Indicating whether the saturation of the cluster where the ith user data in the c-th cluster is located is 0, if not, taking the value of 1, otherwise taking the value of 0; pi c Representing the saturation of the c-th cluster; x is X ic Representing the calculation Y ic Is a covariate of (2); beta 1 、β 2 、δ 1 、δ 2 、
Representing the coefficient, beta 0 Represent constant, E ic Representing the error term.
Preset user behavior index of expression of beta 0 、β 1 、β 2 、δ 1 、δ 2 、
As an unknown quantity, beta can be determined by carrying out regression processing on preset user behavior indexes corresponding to user data in all clusters 0 、β 1 、β 2 、δ 1 、δ 2 、/>
Is a numerical value of (2). X is X ic Representing the calculation Y ic Covariates of (E) for reducing E ic As the variance of the preset user behavior index, one or more variables may be taken according to circumstances. E-shaped article uc The error term is expressed as the difference between the true value and the estimated value, and can be obtained after regression processing, and no specific calculation is generally made.
FIG. 2 is a schematic diagram of regression processing results in the recommendation system overflow effect evaluation method according to an embodiment of the present inventionOne of them. As shown in fig. 2, a schematic diagram of the result of regression processing using OLS (ordinary least squares, general least squares method). Wherein const corresponds to Y ic Beta in the expression 0 T corresponds to Y ic T in the expression ic S corresponds to Y ic S in the expression ic T_pi corresponds to Y ic T in the expression ic *π c X1 corresponds to Y ic X in the expression ic 。
It can be seen that the variable s is not significant (when P>When |t| is larger than the set threshold, the threshold is set to 0.1 in the embodiment of the invention), which indicates that S ic The coefficients of the terms being approximately equal to 0, i.e. Y ic Beta in the expression 2 *S ic Approximately 0, can be ignored.
FIG. 3 is a diagram illustrating a second regression process result in the method for evaluating the overflow effect of the recommendation system according to the embodiment of the present invention. As shown in fig. 3, in order to remove s and then re-perform regression processing, it can be seen that each variable is significant, and a result of each coefficient can be obtained, and a result of coef column is a coefficient corresponding to each variable. From FIG. 3, it can be seen that beta 0 ＝0.0405，β 1 ＝0.3118，δ 1 ＝0.4673，δ 2 ＝0.2995，
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the specific numerical values of the overflow effect and the full processing effect are favorably obtained by giving the expression of the preset user behavior index.
According to the recommendation system overflow effect assessment method provided by the embodiment of the invention, the overflow effect is expressed as delta 2 *π c 。
And calculating the overflow effect according to the saturation of the cluster where the user data is located from the value of 0 to the value of 1 and the change amount of the preset user behavior index. Therefore, when calculating the overflow effect, only the variable S can be considered ic The term involved, the remaining terms can be offset when subtraction is done. According to S ic Overflow from 0-1 changeEffect.
When the saturation is 1, S ic ＝1，π c =1 due to beta 2 *S ic Neglecting, S ic The term involved is delta 2 *(S ic *π c )＝δ 2 *S ic 。
When the saturation is 0, S ic ＝0，π c =0, due to beta 2 *S ic Neglecting, S ic The term involved is delta 2 *(S ic *π c )＝0。
Therefore, the saturation of the cluster in which the user data is located changes from 0 to 1, i.e. the overflow effect is expressed as delta 2 *S ic The overflow effect and the saturation are linear, and the larger the saturation is, the larger the overflow effect is. The overflow effect of each cluster can be obtained according to the saturation condition of each cluster.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the expression delta of the overflow effect is obtained 2 *π c The concrete evaluation of the overflow effect of the recommendation system is realized.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the full processing effect is expressed as beta 1 +δ 1 。
And acquiring the full-quantity processing effect according to the change of the preset user behavior index when the user data is changed from the control group to the experimental group. Therefore, only the variable T can be considered in calculating the full processing effect ic The term involved, the remaining terms can be offset when subtraction is done. Due to Y ic Representing the characteristics of the whole user data, when a new strategy is applied, all the user data in the cluster are in an experiment group, T ic ＝1，π c =1; when the old strategy is applied, all user data in the cluster are in a comparison group, T ic ＝0，π c =0. According to T ic The full processing effect is derived from the 0-1 variation.
Pi when user data is in experimental group c ＝1，T ic =1, variable T ic The term involved is beta 1 *T ic +δ 1 *(T ic *π c )＝β 1 +δ 1 。
Pi when the user data is in the control group c ＝0，T ic =0, variable T ic The term involved is beta 1 *T ic +δ 1 *(T ic *π c )＝0。
Thus, the user data is represented by the change of the preset user behavior index from the control group to the experimental group, i.e. the full-scale processing effect is represented as beta 1 +δ 1 。
The recommendation system overflow effect evaluation method provided by the embodiment of the invention obtains the expression beta of the full processing effect 1 +δ 1 The concrete evaluation of the effect of the recommendation system is realized.
According to the method for evaluating the overflow effect of the recommendation system, which is provided by the embodiment of the invention, the user data of the recommendation system are clustered according to the preset user attribute, and the method comprises the following steps: and clustering the user data of the recommendation system according to the user access time or the region where the user is located.
Finding independent clusters with balanced user data distribution is beneficial to improving the reliability of recommendation system evaluation, and two clustering modes are mainly adopted: clustering according to the region where the user is located and clustering according to the access time of the user. Since the recommender effect check may appear as a module click rate UCTR in terms of user behavior, i.e. number of clicks/number of exposures. The preset user behavior indexes have independence in time and region, and the preset user behavior indexes of different time periods and regions do not interfere with each other, so that the interference among clusters is greatly reduced based on time and region clustering. Therefore, the clustering can be performed according to the access time of the user or according to the area where the platform login user is located.
When the user access time is clustered, for example, the user data may be clustered according to the user access time in the morning, afternoon or evening. When the users are clustered according to the region, the users can be clustered according to the region of the logged-in users in the western region, the eastern region, the north region, the south region and the middle region.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the user data are clustered according to the user access time or the region where the user is located, so that the reliability of recommendation system overflow effect evaluation is improved.
According to the embodiment of the invention, the recommendation system overflow effect evaluation method is used for randomly distributing different saturation degrees for the plurality of clusters and comprises the following steps: the saturation having an arithmetic relationship is randomly allocated to the plurality of clusters.
Since the saturation has an influence on the evaluation result of the recommendation system. The saturation range is [0,1]. In the set of saturation sets for randomly distributing the saturation to each cluster, setting each saturation as an arithmetic relationship according to the number of the saturation is beneficial to reducing the influence of the saturation on the final evaluation result.
Fig. 4 is a schematic diagram of saturation distribution in the recommendation system overflow effect evaluation method according to the embodiment of the present invention. As shown in fig. 4, 0%, 25%, 50%, 75% were chosen as saturation sets and randomly matched to the clusters. Taking cluster 4 as an example, the saturation of the random allocation of cluster 4 is 75%, which means that 75% of all user data in cluster 4 belongs to an experimental group, a new version recommendation strategy is applied, and the other 25% belongs to a control group, and an old version recommendation strategy is applied. Note the randomness of the saturation assignment during the assignment process and the randomness of the groups of experiments per saturation assignment for each cluster.
According to the recommendation system overflow effect evaluation method provided by the embodiment of the invention, the saturation with the arithmetic relation is randomly distributed for a plurality of clusters, so that the reliability of the recommendation system evaluation is further improved.
An example is provided below to illustrate the flow of the recommendation system overflow effect evaluation method provided in the embodiment of the present invention. The recommendation system overflow effect evaluation method provided by the embodiment of the invention comprises the following steps:
(1) Determining cluster classification
User data is selected in the form of clusters, and each cluster of user data is uniformly distributed and independent from cluster to cluster, so that clusters can be distinguished according to time or ground.
(2) Determining saturation
And determining a saturation set according to the recommendation system evaluation requirement and the system support condition, and randomly distributing the saturation set to each cluster, wherein the saturation set with the arithmetic relationship can be adopted.
(3) Packet processing
The user data in the clusters are randomly grouped by the saturation allocated per cluster, i.e. the user data in the clusters are randomly divided into experimental groups and control groups according to the saturation.
(4) Regression treatment
And fitting the expression of the user behavior index by taking the user behavior index of the control group and the experimental group as dependent variables and taking whether the user behavior index is the experimental group, the saturation level and the like as independent variables to obtain each coefficient.
(5) Conclusion output
And estimating the full processing effect and the overflow effect according to the coefficient obtained by regression processing.
As shown in fig. 3, the variables in the regression model are all significant, and the overall processing effect is equal to the sum of the coefficients of t and t_pi, i.e., 0.3118+0.4673= 0.7791. Therefore, when the new policy is full of all users online, the preset user behavior index value is increased 0.7791. At 50% saturation, the overflow effect is 0.5x 0.2995 ≡ 0.1478, i.e. the cluster where the preset user behavior index is located will increase 0.1478.
The preferred embodiments of the present embodiment may be freely combined on the premise that the logic or structure does not conflict with each other, and the present invention is not limited to this.
The description of the recommendation system overflow effect evaluation device provided by the embodiment of the invention is provided below, and the recommendation system overflow effect evaluation device described below and the recommendation system overflow effect evaluation method described above can be referred to correspondingly.
Fig. 5 is a schematic structural diagram of an apparatus for evaluating an overflow effect of a recommendation system according to an embodiment of the present invention. As shown in fig. 5, the apparatus includes a clustering module 10, a grouping module 20, a regression module 30, and an evaluation module 40, wherein: the clustering module 10 is used for: clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters; the grouping module 20 is configured to: randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster; the regression module 30 is configured to: acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is; the evaluation module 40 is configured to: calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 of the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the invention, the user data of the recommendation system is clustered according to the preset user attribute to obtain a plurality of clusters, different saturation levels are randomly allocated to the plurality of clusters, the user data in the clusters are divided into an experimental group and a control group according to the saturation levels, the expression of the preset user behavior index corresponding to the user data is obtained through regression processing, the overflow effect is calculated according to the change amount of the preset user behavior index when the saturation level of the cluster where the user data is from 0 to 1, the measurement of the overflow effect in the recommendation system evaluation is realized, the comprehensiveness of the recommendation system evaluation is enhanced, and the confidence of the recommendation system evaluation result is improved.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the invention, the evaluation module 40 is further used for: acquiring a full-scale processing effect according to the user data from the change of the preset user behavior index when the control group is changed to the change of the experimental group; the full processing effect represents a change value of the preset user behavior index after the recommendation system is applied.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the invention, the effect evaluation of the recommendation system under multi-cluster and multi-saturation is realized by acquiring the full processing effect according to the change of the preset user behavior index when the user data changes from the control group to the experimental group, and the reliability of the recommendation system overflow effect evaluation is improved.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the invention, the preset user behavior index is expressed as follows:
wherein Y is ic Representing the preset user behavior index corresponding to the ith user data in the c-th cluster; t (T) ic Indicating whether the ith user data in the c-th cluster is classified into the experimental group, if so, the experimental group T ic The value is 1, otherwise, the value is 0; s is S ic Indicating whether the saturation of the cluster where the ith user data in the c-th cluster is located is 0, if not, taking the value of 1, otherwise taking the value of 0; pi c Representing the saturation of the c-th cluster; x is X cc Representing the calculation Y ic Is a covariate of (2); beta 1 、β 2 、δ 1 、δ 2 、
Representing the coefficient, beta 0 Represent constant, E ic Representing the error term.
The recommendation system overflow effect evaluation device provided by the embodiment of the invention is beneficial to realizing the acquisition of specific numerical values of the overflow effect and the full processing effect by giving the expression of the preset user behavior index.
According to the embodiment of the invention, the overflow effect evaluation device of the recommendation system is provided, wherein the overflow effect is expressed as delta 2 *π c 。
The recommendation system overflow effect evaluation device provided by the embodiment of the invention obtains the expression delta of the overflow effect 2 *π c The concrete evaluation of the overflow effect of the recommendation system is realized.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the invention, the full processing effect is expressed as beta 1 +δ 1 。
The recommendation system overflow effect evaluation device provided by the embodiment of the invention obtains the expression beta of the full processing effect 1 +δ 1 The concrete evaluation of the effect of the recommendation system is realized.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the invention, the clustering module 10 is specifically configured to: and clustering the user data of the recommendation system according to the user access time or the region where the user is located.
According to the recommendation system overflow effect assessment device provided by the embodiment of the invention, the reliability of recommendation system overflow effect assessment is improved by clustering the user data according to the user access time or the region where the user is located.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the present invention, the grouping module 20 is specifically configured to: the saturation having an arithmetic relationship is randomly allocated to the plurality of clusters.
According to the recommendation system overflow effect evaluation device provided by the embodiment of the invention, the saturation with the arithmetic relation is randomly distributed for a plurality of clusters, so that the reliability of the recommendation system evaluation is further improved.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 6, the electronic device may include: processor 610, communication interface (Communications Interface) 620, memory 630, and communication bus 640, wherein processor 610, communication interface 620, and memory 630 communicate with each other via communication bus 640. The processor 610 may invoke logic instructions in the memory 630 to perform a recommender system overflow effect assessment method comprising: clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters; randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster; acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is; calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 of the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
Further, the logic instructions in the memory 630 may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, an embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute a recommendation system overflow effect assessment method provided by the above methods, and the method includes: clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters; randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster; acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is; calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 of the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
In yet another aspect, an embodiment of the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the recommendation system overflow effect assessment method provided by the above methods, the method comprising: clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters; randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster; acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is; calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 of the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A recommendation system overflow effect assessment method, comprising:
clustering user data of a recommendation system according to preset user attributes to obtain a plurality of clusters;
randomly distributing different saturation degrees to the multiple clusters, and dividing the user data in the clusters into an experimental group and a control group according to the saturation degrees; wherein the saturation is the duty cycle of the experimental group in the cluster;
acquiring an expression of a preset user behavior index corresponding to the user data through regression processing; the preset user behavior index is related to whether the user data is in the experimental group or the control group, whether the saturation of the cluster where the user data is 0, and the saturation of the cluster where the user data is;
calculating an overflow effect according to the variation of the preset user behavior index from the value of 0 to the value of 1 of the saturation of the cluster where the user data is located; the overflow effect is used for measuring the influence of the mutual interference between the experimental group and the control group in the cluster where the user data are located on the preset user behavior index.
2. The recommendation system overflow effect assessment method according to claim 1, wherein said method further comprises:
acquiring a full-scale processing effect according to the user data from the change of the preset user behavior index when the control group is changed to the change of the experimental group; the full processing effect represents a change value of the preset user behavior index after the recommendation system is applied.
3. The recommendation system overflow effect assessment method according to claim 2, wherein the preset user behavior index is expressed as:
wherein Y is ic Representing the corresponding ith user data in the c-th clusterThe preset user behavior index; t (T) ic Indicating whether the ith user data in the c-th cluster is classified into the experimental group, if so, the experimental group T ic The value is 1, otherwise, the value is 0; s is S ic Indicating whether the saturation of the cluster where the ith user data in the c-th cluster is located is 0, if not, taking the value of 1, otherwise taking the value of 0; pi c Representing the saturation of the c-th cluster; x is X ic Representing the calculation Y ic Is a covariate of (2); beta 1 、β 2 、δ 1 、δ 2 、
Representing the coefficient, beta 0 Represent constant, E ic Representing the error term.
4. A recommender system overflow effect assessment method according to claim 3, wherein said overflow effect is denoted δ 2 *π c 。
5. A recommender system overflow effect assessment method according to claim 3, wherein said full processing effect is denoted as β 1 +δ 1 。
6. The recommendation system overflow effect assessment method according to claim 1, wherein said clustering user data of a recommendation system according to preset user attributes comprises:
and clustering the user data of the recommendation system according to the user access time or the region where the user is located.
7. The recommender system overflow effect assessment method of claim 1, wherein said randomly assigning different saturation levels to said plurality of clusters comprises:
the saturation having an arithmetic relationship is randomly allocated to the plurality of clusters.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the recommender system overflow effect assessment method according to any of claims 1 to 7 when the program is executed by the processor.
9. A non-transitory computer readable storage medium, having stored thereon a computer program, which when executed by a processor, implements the steps of the recommendation system overflow effect assessment method according to any of claims 1 to 7.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the recommender system overflow effect assessment method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310219162.3A CN116402249B (en) | 2023-03-06 | 2023-03-06 | Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310219162.3A CN116402249B (en) | 2023-03-06 | 2023-03-06 | Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116402249A true CN116402249A (en) | 2023-07-07 |
| CN116402249B CN116402249B (en) | 2024-02-23 |
Family
ID=87009339
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310219162.3A Active CN116402249B (en) | 2023-03-06 | 2023-03-06 | Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116402249B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107391692A (en) * | 2017-07-26 | 2017-11-24 | 腾讯科技（北京）有限公司 | The appraisal procedure and device of a kind of recommendation effect |
| US20190282155A1 (en) * | 2018-03-15 | 2019-09-19 | Arm Ltd. | Systems, devices, and/or processes for behavioral and/or biological state processing |
| CN110263213A (en) * | 2019-05-22 | 2019-09-20 | 腾讯科技（深圳）有限公司 | Video pushing method, device, computer equipment and storage medium |
| CN111143699A (en) * | 2020-01-03 | 2020-05-12 | 上海理工大学 | Recommendation system based on similarity and confidence clustering |
| CN112380430A (en) * | 2020-11-10 | 2021-02-19 | 腾讯科技（北京）有限公司 | Artificial intelligence based recommendation method and device and electronic equipment |
| CN113312554A (en) * | 2021-06-15 | 2021-08-27 | 北京百度网讯科技有限公司 | Method and device for evaluating recommendation system, electronic equipment and medium |
| CN113779395A (en) * | 2021-09-10 | 2021-12-10 | 粒子文化科技集团(杭州)股份有限公司 | Media asset recommendation method, device, system, electronic device and storage medium |
| CN114840767A (en) * | 2022-05-27 | 2022-08-02 | 中国平安财产保险股份有限公司 | Service recommendation method based on artificial intelligence and related equipment |
| US20220292152A1 (en) * | 2014-06-05 | 2022-09-15 | Snap Inc. | Web document enhancement |
| US20230020268A1 (en) * | 2017-06-12 | 2023-01-19 | Pure Storage, Inc. | Evaluating Recommended Changes To A Storage System |
-
2023
- 2023-03-06 CN CN202310219162.3A patent/CN116402249B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220292152A1 (en) * | 2014-06-05 | 2022-09-15 | Snap Inc. | Web document enhancement |
| US20230020268A1 (en) * | 2017-06-12 | 2023-01-19 | Pure Storage, Inc. | Evaluating Recommended Changes To A Storage System |
| CN107391692A (en) * | 2017-07-26 | 2017-11-24 | 腾讯科技（北京）有限公司 | The appraisal procedure and device of a kind of recommendation effect |
| US20190282155A1 (en) * | 2018-03-15 | 2019-09-19 | Arm Ltd. | Systems, devices, and/or processes for behavioral and/or biological state processing |
| CN110263213A (en) * | 2019-05-22 | 2019-09-20 | 腾讯科技（深圳）有限公司 | Video pushing method, device, computer equipment and storage medium |
| CN111143699A (en) * | 2020-01-03 | 2020-05-12 | 上海理工大学 | Recommendation system based on similarity and confidence clustering |
| CN112380430A (en) * | 2020-11-10 | 2021-02-19 | 腾讯科技（北京）有限公司 | Artificial intelligence based recommendation method and device and electronic equipment |
| CN113312554A (en) * | 2021-06-15 | 2021-08-27 | 北京百度网讯科技有限公司 | Method and device for evaluating recommendation system, electronic equipment and medium |
| CN113779395A (en) * | 2021-09-10 | 2021-12-10 | 粒子文化科技集团(杭州)股份有限公司 | Media asset recommendation method, device, system, electronic device and storage medium |
| CN114840767A (en) * | 2022-05-27 | 2022-08-02 | 中国平安财产保险股份有限公司 | Service recommendation method based on artificial intelligence and related equipment |
Non-Patent Citations (3)
| Title |
|---|
| R. LOGESH 等: "Enhancing recommendation stability of collaborative filtering recommender system through bio-inspired clustering ensemble method", 《NEURAL COMPUTING AND APPLICATIONS》, vol. 32, pages 2141 - 2164, XP037079134, DOI: 10.1007/s00521-018-3891-5 * |
| 杨若玉: "协同过滤算法在个性化推荐系统中的研", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 4, pages 138 - 616 * |
| 贾航: "基于相似度划分及关联规则的商品推荐模型的研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 1, pages 138 - 2334 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116402249B (en) | 2024-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bi | A review of statistical methods for determination of relative importance of correlated predictors and identification of drivers of consumer liking | |
| CN106484777B (en) | Multimedia data processing method and device | |
| CN104951465B (en) | Application recommendation method and device | |
| CN107301247B (en) | Method and device for establishing click rate estimation model, terminal and storage medium | |
| US11381635B2 (en) | Method of operating a server apparatus for delivering website content, server apparatus and device in communication with server apparatus | |
| CN110427358B (en) | Data cleaning method and device and information recommendation method and device | |
| CN109167806B (en) | Uncertain QoS (quality of service) perception Web service selection method based on foreground theory | |
| CN106708875B (en) | Feature screening method and system | |
| CN116402249B (en) | Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation equipment and storage medium | |
| CN112494935B (en) | Cloud game platform pooling method, electronic equipment and storage medium | |
| CN111414528B (en) | Method and device for determining equipment identification, storage medium and electronic equipment | |
| CN107291860B (en) | Seed user determination method | |
| CN105701207B (en) | Resource request quantity prediction method, application recommendation method and device | |
| CN112650940A (en) | Recommendation method and device of application program, computer equipment and storage medium | |
| CN111401969B (en) | Method, device, server and storage medium for improving user retention | |
| CN110083517B (en) | User image confidence optimization method and device | |
| CN111598390B (en) | Method, device, equipment and readable storage medium for evaluating high availability of server | |
| CN111859057B (en) | Data feature processing method and data feature processing device | |
| CN114090204A (en) | Resource prediction method and device | |
| CN113395336A (en) | Mode recommendation method and device, electronic equipment and storage medium | |
| CN115408617B (en) | Internet user integration method based on artificial intelligence and big data service system | |
| CN115509890B (en) | Test method and device based on reinforcement learning, electronic equipment and storage medium | |
| CN114849225B (en) | Server scheduling method and device, computer readable medium and electronic equipment | |
| CN111722912B (en) | Data center virtual machine management method oriented to balance of performance and power consumption | |
| CN112734090B (en) | Target curve acquisition method based on information data processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |