CN112346951A

CN112346951A - Service testing method and device

Info

Publication number: CN112346951A
Application number: CN201910721492.6A
Authority: CN
Inventors: 王勇
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-08-06
Filing date: 2019-08-06
Publication date: 2021-02-09

Abstract

The invention provides a method and a device for testing a service; the method comprises the following steps: determining a plurality of user groups of a target service layer in a plurality of service layers included in a corresponding service based on a preset number of homogeneous indexes; when the target service layer adopts the parameter configuration of the preset version, respectively acquiring first click data of different user groups corresponding to the target service layer; carrying out similarity matching on the first click data of different user groups corresponding to the target service layer; selecting at least two target user groups with the same service index from the plurality of user groups based on the similarity matching result; respectively acquiring the flow of each target user group corresponding to the target service layer; and executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested. The invention can improve the efficiency of the service test.

Description

Service testing method and device

Technical Field

The present invention relates to internet technologies, and in particular, to a method and an apparatus for testing a service.

Background

With the development of internet technology, users' experiences are more emphasized in service development, and multiple configuration parameters of the same service are usually tested, and the parameters most suitable for the user experiences are determined from multiple alternatives.

In the related art, each time a service is tested, a target user group with similar attributes needs to be determined to ensure the accuracy of the service test, but at the same time, the problems of long test time and low test efficiency are caused.

Disclosure of Invention

The embodiment of the invention provides a service testing method and device, which can improve the efficiency of service testing.

The technical scheme of the embodiment of the invention is realized as follows:

the embodiment of the invention provides a service testing method, which comprises the following steps:

determining a plurality of user groups of a target service layer in a plurality of service layers included in a corresponding service based on a preset number of homogeneous indexes;

when the target service layer adopts the parameter configuration of the preset version, respectively acquiring first click data of different user groups corresponding to the target service layer;

carrying out similarity matching on the first click data of different user groups corresponding to the target service layer;

selecting at least two target user groups with the same service index from the plurality of user groups based on the similarity matching result;

respectively acquiring the flow of each target user group corresponding to the target service layer;

and executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.

An embodiment of the present invention provides a service testing apparatus, including:

the group determining unit is used for determining a plurality of user groups of a target service layer in a plurality of service layers included in the corresponding service based on the homogeneity indexes of the preset quantity;

the data acquisition unit is used for respectively acquiring first click data of different user groups corresponding to the target service layer when the target service layer adopts the parameter configuration of a preset version;

the similarity matching unit is used for matching the similarity of the first click data of different user groups corresponding to the target service layer;

a group selecting unit, configured to select, based on the result of similarity matching, at least two target user groups with the same service index from the multiple user groups;

a traffic obtaining unit, configured to obtain traffic of the target service layer corresponding to each target user group;

and the service test unit is used for executing different service tests on the target service layer based on the acquired flow, and the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.

In the above scheme, the group determining unit is further configured to obtain a user portrait of a user corresponding to the target service layer;

and determining a plurality of user groups with the user information similarity corresponding to the target service layer meeting the similarity condition based on the acquired user portrait.

In the above scheme, the data obtaining unit is further configured to obtain click rates and/or click amounts of the services corresponding to different user groups respectively when the target service layer adopts a parameter configuration of a preset version.

In the above scheme, the group selecting unit is further configured to determine at least two user groups of which the similarity reaches a similarity threshold among the plurality of user groups;

and selecting at least two user groups with the similarity reaching a similarity threshold value from the plurality of user groups as the at least two target user groups.

In the above scheme, the similarity matching unit is further configured to determine a plurality of click rate intervals;

and acquiring click rate intervals where click rates of different user groups corresponding to the target service data layer are located.

In the above scheme, the group selecting unit is further configured to determine at least two user groups with click rates in the same interval as a target user group with the same service index.

In the above scheme, the similarity matching unit is further configured to obtain a difference between click rates of the services corresponding to different user groups;

and determining two user groups with the minimum difference in different user groups.

In the above scheme, the service test unit is further configured to allocate the traffic to the services configured by the configuration parameters of the different versions to be tested when the target service layer adopts configuration parameters of at least two different versions to be tested;

respectively acquiring second click data of different target user groups corresponding to the service according to the corresponding relation between the flow and the configuration parameters of the version to be tested;

and determining the configuration parameters meeting the test conditions in the configuration parameters of at least two different versions to be tested based on the second click data.

In the above scheme, the service testing unit is further configured to obtain click rates of the services corresponding to different target user groups respectively;

comparing click rate sizes corresponding to different target user groups to determine the minimum click rate value;

determining a target user group of which the difference value between the click rate corresponding to different target user groups and the minimum click rate value reaches a preset threshold value;

and determining the configuration parameters corresponding to the target user group with the difference value reaching the preset threshold value, wherein the configuration parameters meet the test conditions.

An embodiment of the present invention provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for realizing the service testing method provided by the embodiment of the invention when the executable instruction stored in the memory is executed.

The embodiment of the invention provides a storage medium, which stores executable instructions and is used for causing a processor to execute so as to realize the service testing method provided by the embodiment of the invention.

The embodiment of the invention has the following beneficial effects:

when the target service layer adopts the parameter configuration of the preset version, respectively acquiring first click data of the target service layer corresponding to different user groups; carrying out similarity matching on the first click data of different user groups corresponding to the target service layer; based on the result of similarity matching, at least two target user groups with the same service index are selected from the multiple user groups, so that different service tests can be executed on the target service layer according to the obtained target user groups, the target user groups do not need to be determined for each service test, and the efficiency of the service test is improved.

Drawings

Fig. 1 is a system for testing services provided by an embodiment of the present invention;

fig. 2 is a schematic hardware structure diagram of a server according to an embodiment of the present invention;

fig. 3 is a schematic flow chart of a service testing method provided in an embodiment of the present invention;

fig. 4 is a schematic flow chart of a service testing method provided in an embodiment of the present invention;

fig. 5 is a schematic flow chart of a service testing method provided in an embodiment of the present invention;

FIG. 6 is a schematic diagram of traffic distribution provided by an embodiment of the present invention;

fig. 7 is a schematic structural diagram of a device for testing services according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail with reference to the accompanying drawings, the described embodiments should not be construed as limiting the present invention, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

In the following description, references to the terms "first", "second", etc. merely distinguish similar objects and do not denote a particular order, but rather denote a particular order, and it is to be understood that "first", "second", etc. may be interchanged under appropriate circumstances with a particular order or sequence order, such that embodiments of the invention described herein may be practiced in other than those illustrated or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to be limiting of the invention.

Before further detailed description of the embodiments of the present invention, terms and expressions mentioned in the embodiments of the present invention are explained, and the terms and expressions mentioned in the embodiments of the present invention are applied to the following explanations.

1) A user representation comprising a user interest representation and a user base representation; wherein the content of the first and second substances,

the user interest portrait is a virtual representation of a real user, is a target user model established on a series of attribute data and is used for indicating the interest classification of the user;

the basic user figure is the overall information of the tagged user abstracted according to the basic user information such as the real sex, age, income, resident login place, etc.

2) The click rate refers to the ratio of the number of times a certain content on a website page is clicked to the number of times the content is displayed;

3) the click rate refers to the number of times that a certain content on the website page is clicked within a certain period of time on the website page;

4) the buckets are realized by performing hash calculation on the designated columns, dividing data under a column name into a group of buckets through hash values, and enabling each bucket to correspond to a storage file under the column name.

Referring to fig. 1, fig. 1 is an alternative architecture diagram of a service testing system 100 provided by an embodiment of the present invention, in order to support an exemplary application, terminals (exemplary terminals 400-1, 400-2, 400-3, and 400-4 are shown) are connected to a server 200 through a network 300, where the network 300 may be a wide area network or a local area network, or a combination of the two.

The server 200 determines a plurality of user groups corresponding to a target service layer among a plurality of service layers included in a service based on a preset number of homogeneous indexes;

the terminal is used for receiving a first operation of a user aiming at the service when the service adopts the target parameter configuration and sending first click data corresponding to the first operation to the server;

the server 200 is configured to obtain first click data of target service layers corresponding to different user groups respectively; carrying out similarity matching on the first click data of the target service layer corresponding to different user groups; selecting at least two target user groups with the same service index from the plurality of user groups based on the result of similarity matching; respectively acquiring the flow of each target user group corresponding to the target service layer; and executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.

The following describes an electronic device implementing a method for testing a service according to an embodiment of the present invention. In some embodiments, the electronic device may be implemented as various types of terminals such as a smart phone, a tablet computer, a notebook computer, and the like, and may also be a server. The embodiment of the invention takes the electronic equipment as an example of the server, and the hardware structure of the server is explained in detail.

Fig. 2 is a schematic diagram of a hardware structure of a server according to an embodiment of the present invention, and it is understood that fig. 2 only shows an exemplary structure of the server, and not a whole structure, and a part of or the whole structure shown in fig. 2 may be implemented as needed. Referring to fig. 2, a server provided in an embodiment of the present invention includes: at least one processor 201, memory 202, user interface 203, and at least one network interface 204. The various components in the server are coupled together by a bus system 205. It will be appreciated that the bus system 205 is used to enable communications among the components. The bus system 205 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 205 in fig. 2.

The user interface 203 may include, among other things, a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touch pad, or a touch screen.

It will be appreciated that the memory 202 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), a Flash Memory (Flash Memory), and the like. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM). The memory 402 described in connection with embodiments of the invention is intended to comprise these and any other suitable types of memory.

The memory 202 in the embodiments of the present invention is used to store various types of data to support the operation of the server 200. Examples of such data include: any executable instructions for operating on the server 200, such as executable instructions, may be included in the executable instructions, and the program implementing the method for testing a service of an embodiment of the present invention may be included in the executable instructions.

As an example of the service testing apparatus provided by the embodiment of the present invention implemented by combining software and hardware, the service testing apparatus provided by the embodiment of the present invention may be directly embodied as a combination of software modules executed by the processor 201, where the software modules may be located in a storage medium, the storage medium is located in the memory 202, and the processor 201 reads executable instructions included in the software modules in the memory 202 and completes the service testing method provided by the embodiment of the present invention in combination with necessary hardware (for example, including the processor 201 and other components connected to the bus 205).

By way of example, the Processor 201 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor or the like.

As an example of the hardware implementation adopted by the Device for testing the service provided by the embodiment of the present invention, the Device provided by the embodiment of the present invention may be implemented directly by using the processor 201 in the form of a hardware decoding processor, for example, the Device may be implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), or other electronic components to implement the method for testing the service provided by the embodiment of the present invention.

The service testing method disclosed by the embodiment of the invention can be realized by the processor 201. The processor 201 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the service testing method may be implemented by an integrated logic circuit of hardware in the processor 201 or instructions in the form of software. The Processor 201 may be a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 201 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the service testing method disclosed by the embodiment of the invention can be directly embodied as the execution of the hardware decoding processor, or the combination of the hardware and the software module in the decoding processor. The software module may be located in a storage medium located in the memory 202, and the processor 201 reads the information in the memory 202, and performs the steps of the service testing method provided by the embodiment of the present invention in combination with the hardware thereof.

Based on the above description of the service testing system and the electronic device according to the embodiment of the present invention, a service testing method according to the embodiment of the present invention is described next, fig. 3 is a schematic flow chart of the service testing method according to the embodiment of the present invention, in some embodiments, the service testing method may be implemented by a server, for example, by the server 200 in fig. 1, and with reference to fig. 1 and fig. 3, the service testing method according to the embodiment of the present invention includes:

step 301: the server determines a plurality of user groups of a target service layer in a plurality of service layers included in the corresponding service based on the homogeneity indexes of the preset number.

Here, the service includes services of an algorithm class, an operation class, and a user interface class. Various algorithms are the places with the most application of service tests, and whether a new algorithm or small optimization can improve the service index of the algorithm or not can be verified through the service tests, such as recommending, searching, accurate advertising and refining the products of the algorithms involved in operation. Various operation means, such as user operation (user update, member operation, and the like), content operation (programming in the video industry, and the like), activity operation (lottery, and the like), and the like, can verify which operation strategy is more effective by means of a service test. The user interface is a part which can be directly perceived by any internet product by a user, the user interacts with the internet product through the user interface, and the perception of the user on the product is also established by the user firstly; the simple and attractive user interface and the smooth user interaction can often leave a good first impression for the user. For optimization of the visual and interactive parts of the user, the experience of designers is often insufficient, and technical means are required to verify which user interface display style and which interactive mode is preferred by the user and can bring the maximum benefit.

It should be noted that the homogeneity indicator may be user information such as age, gender, academic history, income, and the like of the user, the user may configure one or more homogeneity indicators according to needs, and the server determines a plurality of user groups corresponding to a target service layer of a plurality of service layers included in the service according to the homogeneity indicator preset by the user. Here, each user group includes a plurality of users, where a plurality means two or more.

In practical implementation, a user can configure service layers according to different modules of services, taking a search service as an example, the search service includes three modules, and a user presentation module, a result sorting module and an advertisement insertion module, so that the three service layers can be configured to correspond to the three modules respectively, and a server can obtain first click data of different service layers at the same time, and the different service layers are not interfered with each other. When the homogeneous indexes preset by the user are age and gender, and the target service layer is a result sorting layer, the server can group the users according to the age and gender corresponding to the users corresponding to the result sorting layer to obtain a plurality of user groups corresponding to the result sorting layer.

In some embodiments, the server may determine a plurality of user groups corresponding to a target service layer of a plurality of service layers included in the service in the following manner: acquiring a user portrait of a user corresponding to a service; and determining a plurality of user groups of which the user information similarity of the corresponding service meets the similarity condition based on the acquired user portrait.

Here, the user representation may include at least one of: user interest representation, user base representation. In some embodiments, the server may obtain user figures of all users corresponding to the service, screen out some users from all users according to the obtained user figures, and divide the screened users into a plurality of user groups, so that the similarity of user information corresponding to any two user groups meets the similarity condition; in other embodiments, the server may divide the plurality of users corresponding to the service into a plurality of user groups, then obtain a user representation of the user corresponding to the service, and determine, from the plurality of user groups, a plurality of user groups in which the similarity of the user information of the service satisfies the similarity condition according to the obtained user representation.

Here, the condition that the similarity of the user information of the service satisfies the similarity condition indicates that the components of the users in the user group are the same, for example, the server acquires the age and gender of the user corresponding to the service, and determines a plurality of user groups according to the age and gender of the user, wherein the number ratio of male users to female users in each user group is 1:1, and the age is between 20 and 40 years.

In some embodiments, the server may determine the plurality of user groups for the corresponding service by: distributing the flow of the corresponding service to a plurality of sub-buckets; determining a user corresponding to the flow in each sub-bucket according to information carried in the access request corresponding to the flow in each sub-bucket; and determining a plurality of user groups corresponding to the service according to the user portrait of the user corresponding to the flow in each sub-bucket, wherein the user groups comprise at least one user corresponding to the flow in the sub-bucket.

It should be noted that the traffic may be a number of independent visitors (UV) or a web View volume (PV). The traffic corresponds to the number of access requests, for example, the traffic is 10000, and the corresponding number of access requests is also 10000.

In practical implementation, the flow is distributed to different sub-buckets in a random flow mode, the access flow in each sub-bucket is the same, the information carried by the access request comprises a user identifier, the server can determine a user corresponding to the flow in each sub-bucket based on the user identifier, obtain a user portrait of the user, and perform similarity matching on the user information corresponding to the flow in different sub-buckets to determine at least two user groups.

The embodiment of the invention carries out similarity matching on the user information corresponding to the flow in different sub-barrels, and improves the matching efficiency because the matching is the average value of the user information corresponding to each sub-barrel, and the similarity matching of each user information is not needed.

In some embodiments, the number of users in each user group may be configured, and the bucket number corresponding to the user group is selected according to the configured number. For example, the traffic in each sub-bucket is 1000, that is, the number of users corresponding to the traffic in each sub-bucket is 1000, and the number of users in a certain user group is configured to be 2000, then the user group includes users corresponding to the traffic in two sub-buckets. Here, the number of users in each user group may be different, for example, a first user group may include users corresponding to traffic in bucket a and bucket B, and a second user group may include users corresponding to traffic in bucket C.

Step 302: and when the target service layer adopts the parameter configuration of the preset version, respectively acquiring first click data of the target service layer corresponding to different user groups.

Here, the parameter may be a flow algorithm for the target service layer, or may be an interface display parameter for the service. Illustratively, when the target service layer is a search service result sorting layer and a sorting algorithm for sorting results is preset by a user, after different user groups send search requests, the server sorts the search results corresponding to the search requests according to the sorting algorithm for sorting preset results, generates a search result page, and obtains click data of different user groups on the search result page for the search results.

In practical implementation, the first click data includes click data of a service corresponding to each user in the user group, or may be an average value of click data of services corresponding to a plurality of users in the user group.

In some embodiments, the server may allocate users in the user group to multiple sub-buckets, and obtain an average value of the first click data of all users in each sub-bucket, so that accuracy of subsequently determining the target user group can be ensured, and meanwhile, the calculation amount is reduced.

In some embodiments, the first click data may be a click rate and/or a click rate, and when the service adopts the target parameter configuration, the click rate and/or the click rate of the service corresponding to different user groups are respectively obtained.

Step 303: and performing similarity matching on the first click data of the target service layer corresponding to different user groups.

In some embodiments, the server may perform similarity matching by: determining a plurality of click rate (click volume) intervals; and acquiring click rate intervals where click rates of different user groups corresponding to the target service data layer are located.

For example, five click rate intervals are preset, which are respectively less than 3%, 3% -8%, 8% -13%, 13% -18% and more than 18%, and the user groups are divided according to the click rate intervals.

In some embodiments, when the number of the target user groups is two, the server may perform similarity matching by: acquiring a difference value between click rates of services corresponding to different user groups; and determining two user groups with the minimum difference in different user groups.

For example, the click rates of the user group A, B, C are 5%, 7%, and 17%, respectively, where the click rate difference between the user group a and the user group B is at least 2%, and the user group a and the user group B are two user groups with the smallest difference.

Step 304: and selecting at least two target user groups with the same service index from the plurality of user groups based on the similarity matching result.

In some embodiments, when the server obtains a click rate interval where click rates of different user groups corresponding to the target service data layer are located, at least two user groups having click rates in the same interval may be determined as a target user group with the same service index.

In some embodiments, when the server acquires two user groups with the smallest difference among different user groups, the two acquired user groups with the smallest difference are used as the target user group with the same service index.

In some embodiments, the server may further determine at least two user groups of the plurality of user groups whose similarity reaches a similarity threshold; and selecting at least two user groups with the similarity reaching a similarity threshold value from the plurality of user groups as the at least two target user groups.

In some embodiments, the server may determine that the similarity of at least two user groups reaches a similarity threshold by hypothesis testing.

In practical implementation, the similarity threshold of the similarity of at least two user groups is set as zero hypothesis, and the similarity threshold of the similarity of at least two user groups is set as alternative hypothesis, then if p-value < alpha/n, the zero hypothesis is represented to be not true, otherwise, the zero hypothesis is true. Here, p-value is a probability that the original hypothesis is established calculated by a statistical method, alpha represents a detection level when the probability that the original hypothesis is rejected is smaller, n is the number of parameters included in the first click data, and n is 2 when the first click data is a click rate and a click amount. In actual implementation, alpha may be set to 0.05.

In some embodiments, when the first click data has a plurality of parameters, the p-value may be corrected by a Family of error rate (FWER) or False positive rate (FDR) algorithm.

Step 305: and respectively acquiring the flow of each target user group corresponding to the target service layer.

It should be noted that one user may trigger one flow or may trigger multiple flows. For example, if the user a swipes a microblog only once in a day, it can be recorded that the user a triggers a flow in the microblog in the day. For another example, if the user a swipes a microblog n times in one day, it can be recorded that the user a triggers n traffic flows.

Step 306: and executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.

Here, the server may execute different service tests based on the acquired traffic, and does not need to determine a target user group for each service test, so that the time for the service test can be saved, and the efficiency of the service test can be improved.

In some embodiments, the server may perform the service test on the target service layer by: when the target service layer adopts configuration parameter configuration of at least two different versions to be tested, flow is respectively allocated to the services configured by the configuration parameters of the different versions to be tested; respectively acquiring second click data of services corresponding to different target user groups according to the corresponding relation between the flow and the configuration parameters of the version to be tested; and determining the configuration parameters meeting the test conditions in the configuration parameters of at least two different versions to be tested based on the second click data.

Taking a user interface test of a page as an example, a page a is configured by using configuration parameters of a first version to be tested, a navigation bar is red, a page B is configured by using configuration parameters in a second version to be tested, and the navigation bar is black, and the flow is respectively allocated to the pages configured by the configuration parameters of different versions to be tested, so that when a target user group a accesses the page, the page a is displayed, and when the target user group B accesses the page, the page B is displayed, and click data of different pages corresponding to the target user group A, B is respectively acquired. And when the click data of the target user group A is significantly larger than that of the target user group B, determining that the page A is a page preferred by the user.

In some embodiments, the server obtains click rates (click volumes) of services corresponding to different target user groups respectively; comparing click rates (click amounts) corresponding to different target user groups to determine the minimum value of the click rates (click amounts); determining a target user group of which the difference value between the click rate (click quantity) and the minimum value of the click rate (click quantity) reaches a preset threshold value in the click rates (click quantities) corresponding to different target user groups; and determining the configuration parameters corresponding to the target user group with the difference value reaching the preset threshold value, wherein the configuration parameters meet the test conditions.

For example, the click rates corresponding to the target user group A, B, C are 2%, 3%, and 10%, respectively, the minimum value of the click rates corresponding to the target user group A, B, C is 2%, and if the preset threshold is 5%, it is determined that the configuration parameters corresponding to the target user group are the configuration parameters that satisfy the test condition.

It should be noted that, if the difference between the click rate corresponding to a certain target user group and the minimum value of the click rates corresponding to the target user groups is greater than the preset threshold, it indicates that the click rate corresponding to the target user group is much greater than the click rates corresponding to other user groups, which indicates that the configuration parameters adopted by the service corresponding to the user group are superior to other configuration parameters, and determines that the configuration parameters are the configuration parameters meeting the test conditions.

In some embodiments, configuration parameters meeting the test conditions can be determined in a hypothesis testing manner, in actual implementation, two groups of configuration parameters are respectively parameters corresponding to an original scheme and parameters corresponding to a new scheme, the click rate and/or click rate of the new scheme are set to be higher than those of the original scheme by the original hypothesis, p-value is calculated by a statistical method, and if the p-value < alpha indicates that a null hypothesis is not established, the configuration parameters meeting the test conditions are parameters corresponding to the original scheme; otherwise, the zero hypothesis is established, and the configuration parameters meeting the test conditions are the parameters corresponding to the new scheme. In actual implementation, alpha may be set to 0.05.

According to the embodiment of the invention, when the target service layer adopts the parameter configuration of the preset version, first click data of the target service layer corresponding to different user groups are respectively obtained; carrying out similarity matching on the first click data of different user groups corresponding to the target service layer; based on the result of similarity matching, at least two target user groups with the same service index are selected from the multiple user groups, so that different service tests can be executed on the target service layer according to the obtained target user groups, the target user groups do not need to be determined for each service test, and the efficiency of the service test is improved.

Next, a service testing method provided by the embodiment of the present invention is described by taking a search ranking layer in a search service as an example. Fig. 4 is a schematic flow chart of a service testing method provided in an embodiment of the present invention, as shown in fig. 4,

step 401: the server distributes the traffic of the corresponding search ranking layer to 1000 sub-buckets.

Step 402: and determining the user corresponding to the flow in each sub-bucket according to the information carried in the access request corresponding to the flow in each sub-bucket.

Step 403: and determining a plurality of user groups with the similarity of the similarity indexes of the users reaching a similarity threshold according to the similarity indexes of the users corresponding to the flow in each sub-bucket.

Here, each user group includes at least one user corresponding to the traffic in the sub-bucket.

Step 404: and acquiring a first click rate of each user group aiming at the target page.

Here, the target page displays the search results in a target ranking manner.

Step 405: and carrying out similarity matching on the click rates of the target pages corresponding to different user groups to determine two user groups with similarity reaching a similarity threshold value in the plurality of user groups.

Step 406: and taking the two user groups with the similarity reaching the similarity threshold as target user groups.

Step 407: and acquiring a second click rate of the first target user group for the page corresponding to the original sorting mode and a third click rate of the second target user group for the page corresponding to the new sorting mode.

Step 408: and when the second click rate is improved by more than thirty percent relative to the third click rate, if so, determining that the new sorting mode meets the target sorting mode.

Here, whether the second click rate is improved by more than thirty percent relative to the third click rate indicates that the new sorting method is obviously better than the original sorting method.

The description continues on the service testing method provided by the embodiment of the present invention. Fig. 5 is a schematic flow chart of a service testing method provided in an embodiment of the present invention, and referring to fig. 5, in actual implementation, the service testing method provided in the embodiment of the present invention includes the following steps.

First, the traffic corresponding to the service layer is divided into a plurality of buckets, for example, the traffic corresponding to the service layer is divided into 10000 buckets according to a random traffic manner, and the traffic of each bucket is one ten thousandth.

Here, the traffic may be UV or PV, and the user creates a corresponding service layer according to service requirements and allocates the traffic to each layer. For example, fig. 6 is a schematic flow allocation diagram provided by an embodiment of the present invention, and as shown in fig. 6, a search service includes three modules, and user presentation, result sorting, and advertisement insertion may create three service layers respectively corresponding to the three modules.

It should be noted that the traffic in the same layer can only hit one service test, and cannot hit two or more service tests at the same time. Traffic in different layers may hit different traffic tests, but the traffic is orthogonal. That is, the traffic hitting the service test in the user presentation layer-B is uniformly distributed in the result sorting layer-C, which can ensure that the service tests in the two layers do not interfere with each other.

In some embodiments, a user may create a cross-layer test according to a service, for example, as shown in fig. 6, the user has created a user presentation layer-B and a result sorting layer-C, and the user wants to simultaneously test configuration parameters of the user presentation layer-B and the result sorting layer-C, and since traffic of each layer is orthogonal, the user needs to create a merging layer-E, and the merging layer-E is used to simultaneously test configuration parameters of the user presentation layer-B and the result sorting layer-C. Here, different configuration parameters correspond to different interfaces or flows.

And then, performing an AA pre-stage experiment through an idle module, acquiring basic information of a user corresponding to the flow in each sub-bucket, and acquiring first operation information of the user corresponding to the flow in each sub-bucket when the service layer adopts parameter configuration of a preset version.

It should be noted that the idle module records the usage of the flow, including the allocated flow and the unallocated flow. Here, the idle module acquires the flow rate in the idle sub-bucket corresponding to the user information and the first operation information, the user information refers to the age, sex, academic calendar and the like of the user, and the first operation information may be the click rate, click rate and the like. And storing the acquired user information and the first operation information into an idle database. It should be noted that, the user corresponding to each sub-bucket is allocated to 50 sub-buckets, and therefore, the storage format of the idle database is 51 columns of data: the sub-bucket ID and the 50 sub-buckets correspond to user information and first operation information of a user.

And then, determining two user groups, wherein each user group comprises at least one user corresponding to the barreled flow, acquiring user information and first operation information corresponding to each user group from a database, and judging whether the user composition of the two user groups is the same or not through hypothesis testing according to the acquired user information and the first operation information. In actual implementation, the number of users corresponding to the user group may be configured by the user.

For example, if the first user group includes users corresponding to two {12, 1019} sub-buckets and the second user group includes users corresponding to two {3330, 8912} sub-buckets, the user information and the first operation information of the sub-bucket 1019 and the sub-bucket 12 are merged, the user information and the first operation information of the sub-bucket 8912 and the sub-bucket 3330 are merged, and then hypothesis testing is performed.

Here, assuming that the checked checking level alpha is 0.05, if p-value < alpha/number of indexes indicates that the user composition of two user groups is different, it is necessary to re-determine the two user groups; otherwise, the user composition of the two user groups is the same, so that the two user groups respectively correspond to two groups of different configuration parameters. Here, the index includes first operation information and basic information of the user. If there are multiple indicators, the p-value can be modified using FWER or FDR.

And finally, operating an experiment AB stage, respectively acquiring second operation information of the two user groups relative to different configuration parameters, and judging whether the second operation information corresponding to the two user groups is obviously different, thereby obtaining which of the two groups of configuration parameters is more optimal. Meanwhile, the unallocated flow of the service layer is updated and returned to the idle module.

Here, the second operation information may be a click amount, a click rate, or the like. In practical implementation, a hypothesis testing method may be used to determine whether the second operation information corresponding to the two user groups is significantly different.

It should be noted that, in the same layer, multiple configuration parameter tests can be performed simultaneously; in a test system, testing of configuration parameters may be performed in multiple layers simultaneously.

Next, a description is given of a service testing apparatus provided in an embodiment of the present invention, in some embodiments, the service testing apparatus may be implemented in a software module, fig. 7 is a schematic structural diagram of the service testing apparatus provided in the embodiment of the present invention, and referring to fig. 7, the service testing apparatus includes:

a group determining unit 701, configured to determine, based on a preset number of homogeneous indexes, a plurality of user groups of a target service layer in a plurality of service layers included in a corresponding service;

a data obtaining unit 702, configured to obtain first click data corresponding to the target service layer for different user groups when the target service layer adopts a preset version of parameter configuration;

a similarity matching unit 703, configured to perform similarity matching on the first click data corresponding to the target service layer in different user groups;

a group selecting unit 704, configured to select, based on the result of similarity matching, a target user group with at least two service indexes being the same from the multiple user groups;

a traffic obtaining unit 705, configured to obtain traffic of the target service layer corresponding to each target user group respectively;

a service testing unit 706, configured to execute different service tests on the target service layer based on the obtained traffic, where the service tests are used to select a configuration parameter of a target version from configuration parameters of at least two different versions to be tested.

An embodiment of the present invention further provides an electronic device, where the electronic device includes:

a memory for storing an executable program;

and the processor is used for realizing the service testing method provided by the embodiment of the invention when the executable program stored in the memory is executed.

The embodiment of the invention also provides a storage medium stored with executable instructions, wherein the executable instructions are stored in the storage medium and can cause a processor to execute the service testing method provided by the embodiment of the invention when being executed by the processor.

All or part of the steps of the embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as a removable Memory device, a Random Access Memory (RAM), a Read-Only Memory (ROM), a magnetic disk, and an optical disk.

Alternatively, the integrated unit of the present invention may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, a RAM, a ROM, a magnetic or optical disk, or various other media that can store program code.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims

1. A method for testing a service, the method comprising:

2. The method of claim 1, wherein the determining the plurality of user groups corresponding to the target service layer of the plurality of service layers included in the service based on the predetermined number of homogeneity indicators comprises:

acquiring a user portrait of a user corresponding to the target service layer;

3. The method of claim 1, wherein the obtaining first click data corresponding to the target service layer from different user groups respectively when the target service layer adopts a preset version of parameter configuration comprises:

and when the target service layer adopts the parameter configuration of the preset version, respectively acquiring the click rate and/or click quantity of the service corresponding to different user groups.

4. The method of claim 1, wherein the selecting a target user group with at least two service metrics being homogeneous from the plurality of user groups based on the similarity matching result comprises:

determining at least two user groups with similarity reaching a similarity threshold in the plurality of user groups;

5. The method of claim 1, wherein the similarity matching of the first click data corresponding to the target service layer for different user groups comprises:

determining a plurality of click rate intervals;

6. The method of claim 5, wherein the selecting a target user group with at least two service metrics being homogeneous from the plurality of user groups based on the similarity matching result comprises:

and determining at least two user groups with the click rate in the same interval as a target user group with the same service index.

7. The method of claim 1, wherein the similarity matching of the first click data corresponding to the target service layer for different user groups comprises:

acquiring the difference between the click rates of the services corresponding to different user groups;

8. The method of claim 1, wherein said performing different service tests on said target service layer based on said obtained traffic comprises:

when the target service layer adopts configuration parameter configuration of at least two different versions to be tested, the flow is respectively allocated to the service configured by the configuration parameters of the different versions to be tested;

9. The method of claim 8, wherein determining the configuration parameters that satisfy the test condition from among the configuration parameters of the at least two different versions under test based on the second click data comprises:

respectively acquiring click rates of the services corresponding to different target user groups;

10. An apparatus for testing a service, the apparatus comprising: