KR102309715B1 - A/B Testing Platform with Anomaly Detection - Google Patents

Abstract

The present invention relates to a method for providing an A/B testing platform, which includes the steps of: receiving input data including a testing unique number through software development kit interworking, by a processor; managing, by the processor, A/B testing matching the unique number; and providing, by the processor, an A/B testing result value derived based on the input data. In the method for providing an A/B testing platform, a step of managing an anomaly based on a preset threshold, by the processor, is further included.

Description

How to provide an AB testing platform capable of managing anomalies {A/B Testing Platform with Anomaly Detection}

The present invention relates to a method of providing an AB testing platform capable of managing abnormal symptoms.

A/B test is a method of comparing the existing version (A) with the new version (B, C, ...) to determine which version performs better. Here, good performance is based on business performance indicators that companies want to know, such as click-through rate and purchase conversion rate. The AB test can be used in various fields, such as web page design change, product layout change, and software logic change.

The AB test is divided into two versions, and during the test, except for one variant that can affect the user's behavior, the remaining conditions remain the same, so that the impact of a specific change that the company wants to analyze is data. can be grasped as Through the AB test results, companies can identify the best plan for business growth and make decisions based on data.

Registration No. 10-1784106 discloses a method and system for providing services based on AB testing. The AB test according to the prior art including registration number 10-1784106 derives the result value without managing the AB test process or managing abnormal symptoms generated during the AB test. In this case, there was a problem that the risk had to be maintained until the AB test was finished even in a situation where the risk was exposed to the application of the new version.

Registration number 10-1784106

An object of the present invention is to provide a method for providing an AB testing platform capable of safely conducting an AB test in order to solve the above problems.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, the method for providing an AB testing platform according to an embodiment of the present invention manages the AB testing process and manages abnormal symptoms generated during the AB testing.

A method for providing an AB testing platform according to an embodiment of the present invention includes: receiving, by a processor, input data including a testing unique number through SDK (Software Development Kit) interworking; managing, by the processor, AB testing matching the unique number; and providing, by the processor, an AB testing result value derived based on the input data.

The managing of the AB testing includes, by the processor, managing the AB test based on a signal received from the customer device by matching the unique number.

The managing of the AB testing may include, by a processor, setting an exposure range of the AB testing based on a first signal received from the customer device; starting, by the processor, AB testing based on the second signal received from the customer device; and stopping, by the processor, AB testing based on the third signal received from the customer device.

The method for providing an AB testing platform according to an embodiment of the present invention further includes, by the processor, managing abnormal symptoms based on a preset threshold.

The step of managing the abnormal symptom may include: providing, by the processor, information on the abnormal symptom to the customer when it is determined that the abnormal symptom has occurred; and normalizing the processor by returning to the preset version.

The details of other embodiments are included in the detailed description and drawings.

According to the present invention, there are one or more of the following effects.

First, it has the effect of providing a safer AB testing platform by managing risks that may occur during the AB testing process.

Second, it has the effect of helping companies easily, quickly, and accurately understand the impact on business performance and improve products safely and quickly with little risk.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is an AB testing system according to an embodiment of the present invention.
2 is a block diagram of an apparatus for providing an AB testing platform according to an embodiment of the present invention.
3 is a flowchart referenced to explain a method for providing an AB testing platform according to an embodiment of the present invention.
4 is a detailed flowchart of S320 of FIG. 3 .
5 is a detailed flowchart of S330 of FIG. 3 .

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as "comprises" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

1 is an AB testing system according to an embodiment of the present invention.

The AB testing system 10 may include an AB testing platform providing apparatus 100 , a customer device 20 , and a user terminal 30 .

The apparatus 100 for providing an AB testing platform is an apparatus for providing an AB testing platform, and may be described as a device including at least one processor. For example, the AB testing platform providing apparatus 100 may be implemented as a server.

The AB testing platform providing apparatus 100 may provide an AB testing service to a customer. The AB testing platform providing apparatus 100 may receive input data from the customer device 20 and provide an AB testing result value based on the input data.

The AB testing platform providing apparatus 100 may provide a result value associated with a conversion rate.

Conversion Rate (CVR) refers to the rate at which users take actions that are actually intended by the service. For example, the conversion rate may be described as a click-through rate or a purchase conversion rate.

An online advertisement was provided to 100 users as a banner advertisement at the bottom of the screen. When 2 users clicked on the online advertisement, the conversion rate was 2%. In addition, the main product was posted to 100 users who accessed online shopping through the top of the screen. If one user purchased it, the conversion rate was 1%.

The company collects the conversion rate through AB testing, and when the testing period expires, it manages to provide services with the project with the highest conversion rate. As a result of providing plan A to 50% of the users who access during the testing period and providing plan B to the remaining 50%, if the conversion rate for plan A is 1% and the conversion rate for plan B is 3%, after that, service to plan B is provided. will provide

The AB testing platform providing apparatus 100 may provide a result value in an image. For example, the AB testing platform providing apparatus 100 may provide a result value using various graph formats or animation formats. The AB testing platform apparatus 100 may provide a graphic image in which key matters among the result values are highlighted.

The customer device 20 may be a concept including at least one of a customer system, a customer server, and a customer terminal.

The user terminal 30 may be described as a terminal of an individual user using a service provided by the customer device 20 .

2 is a block diagram of an apparatus for providing an AB testing platform according to an embodiment of the present invention.

Referring to the drawings, the AB testing platform providing apparatus 100 may include an input data acquisition unit 110 , a testing management unit 120 , an abnormal symptom management unit 130 , and a result value providing unit 140 .

The input data acquisition unit 110 may receive input data from the customer device 20 through software development kit (SDK) interworking. The input data acquisition unit 110 may perform the operation of step S310, which will be described later with reference to FIGS. 3 to 5 .

The testing manager 120 may manage AB testing based on a signal received from the customer device 20 . The testing manager 120 may perform the operation of step S320, which will be described later with reference to FIGS. 3 to 5 .

The abnormal symptom management unit 130 may manage the abnormal symptom based on a result of comparing the change value according to the AB testing with a preset threshold value. The abnormal symptom management unit 130 may perform the operation of step S330, which will be described later with reference to FIGS. 3 to 5 .

The result value providing unit 140 may provide the AB testing result value derived based on the input data. The result value providing unit 140 may perform the operation of step S340, which will be described later with reference to FIGS. 3 to 5 .

Meanwhile, each of the input data acquisition unit 110 , the testing management unit 120 , the abnormal symptom management unit 130 , and the result value providing unit 140 may be implemented as a software module installed in a processor to be described later. In this case, the input data acquisition unit 110 , the testing management unit 120 , the abnormal symptom management unit 130 , and the result value providing unit 140 may be described as a sub-concept of the processor 170 .

The AB testing platform providing apparatus 100 may include at least one processor 170 as hardware, and the processor 170 is, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical devices for performing functions. It may be implemented using at least one of the units.

The processor may implement an AB testing platform providing method, which will be described later.

3 is a flowchart referenced to explain a method for providing an AB testing platform according to an embodiment of the present invention.

4 is a detailed flowchart of S320 of FIG. 3 .

5 is a detailed flowchart of S330 of FIG. 3 .

Referring to the drawing, the processor 170 may receive input data from the customer device 20 ( S310 ). The processor 170 may receive input data including a testing unique number through software development kit (SDK) interworking.

The input data may include data on at least one of an AB testing unique number, a testing target identifier value, event information necessary for calculating a testing result, and event attribute information.

The test target identifier value may be a concept including the user terminal 30 ID and the user ID.

The event information is related to the conversion rate, and may be a concept including, for example, payment success event information, item addition to shopping cart event information, and the like.

The event attribute information may include payment method information upon successful payment, user personal information (eg, gender information, age information), and the like.

Meanwhile, the input data may include information on the AB testing target. For example, the AB testing target may be any one of elements constituting a web page.

The processor 170 may manage AB testing (S320). The processor 170 may manage the AB testing that matches the testing unique number.

Step S320 may include, by the processor 170, managing the AB test based on a signal received by matching a unique number from the customer device 20 . The customer may continuously manage the AB testing in the AB testing process through the customer device 20 .

As illustrated in FIG. 4 , step S320 may include steps S410 to S460 .

The processor 170 may set the exposure range (S410). The processor 170 may set the AB testing exposure range based on the first signal received from the customer device 20 . For example, when receiving a signal for setting the exposure range to 40% from the customer device 20 , the processor 170 may include only 40% of users using the customer service as the test subject.

The processor 170 may start AB testing (S420). The processor 170 may start the AB testing based on the second signal received from the customer device 20 .

If, when receiving a signal for stopping AB testing from the customer device 20 (S430), the processor 170 may stop the AB testing (S440).

When a problem occurs (S450), the processor 170 may stop AB testing (S460). The processor 170 may stop the AB testing based on the third signal received from the customer device 20 . If the customer recognizes that a problem occurs while checking the AB testing data, the customer device 20 may request stopping the testing. In this case, the processor 170 may stop the AB testing as soon as the signal is received.

If a webpage version containing new developments is distributed without AB testing, and a problem occurs, the service can be normalized only by redistributing the modifications to the server or rolling back the distribution of the changes. However, if the AB testing according to the embodiment of the present invention is used, when a problem occurs, the problem can be solved relatively easily.

The processor 170 may manage abnormal symptoms based on a preset threshold in the AB testing process (S330). The processor 170 may manage abnormal symptoms based on a result of comparing the conversion rate of the new version (B) compared to the existing version (A) and a preset threshold in the AB testing. For example, when the conversion rate of the new version B appears below the threshold, the processor 170 may determine that it is an abnormal symptom.

As illustrated in FIG. 5 , step S330 may include steps S510 to S550 .

The processor 170 may determine the occurrence of an abnormal symptom by comparing the conversion rate with the threshold ( S510 ). For example, when the conversion rate of the existing version (A) is 90% and the conversion rate of the new version (B) falls below the threshold of 70%, the processor 170 may determine that an abnormal symptom has occurred. .

When it is determined that an abnormal symptom has occurred, the processor 170 may provide information on the abnormal symptom to the customer company (S520). For example, the processor 170 may transmit a message including information on abnormal symptoms to the customer device 20 . For example, the processor 170 may provide information on abnormal symptoms through a phone call, an e-mail, an instant message, or the like of a pre-registered customer company.

When automatic rollback is set (S530), the processor 170 may automatically rollback (S540). The processor 170 may return to a preset version (eg, the existing version (A)) to normalize the testing target.

When automatic rollback is not set (S530), the processor 170 may stop AB testing (S550).

Meanwhile, the processor 170 may accumulate and store abnormal symptom related event data. The processor 170 may apply an artificial intelligence algorithm by using the accumulated stored data as input data. The processor 170 may define criteria for determining abnormal symptoms by itself using the output data of the artificial intelligence algorithm. The processor 170 may perform step S330 based on the defined determination criterion.

The processor 170 may provide an AB testing result value derived based on the input data (S340). The processor 170 may calculate the conversion rate and provide the result. For example, the processor 170 may calculate an order amount per customer, a site stay time, a purchase success rate, and the like, and update the corresponding information in a preset time unit.

The processor 170 may provide a result value by graphic processing. For example, the processor 170 may calculate the conversion rate and provide the result in the form of a graph. For example, the processor 170 may calculate a conversion rate and provide a graph in which the result value is changed in a preset time unit in the form of a graph.

Meanwhile, the processor 170 may perform a plurality of AB testing independently of each other. For example, the processor 170 may independently perform AB testing for the first homepage, product detail page, and shopping cart of the shopping mall app. At this time, the derived result values do not affect each other between AB testing.

In this case, the method of providing an AB testing platform includes: receiving, by the processor 170, input data including a first unique number and a second unique number through SDK interworking; The processor 170 may include: managing a first AB testing matching the first unique number and a second AB testing matching a second unique number; managing, by the processor 170, an abnormal symptom of the first AB testing based on a preset first threshold; managing, by the processor 170, an abnormal symptom of the second AB testing based on a preset second threshold; and providing, by the processor 170, a result value based on the input data, wherein the managing of the testing includes managing a plurality of tests independently of each other, and providing the result value includes: a plurality of testing It is characterized in that each result value does not influence each other.

In most cases, when a new function of a service is distributed, several functions are included in one deployment. When distributed in this way, when 10 functions are reflected together in one distribution, if one function has a problem, all 10 functions must be rolled back to their original state before the system can be normalized.

However, using the method according to an embodiment of the present invention, if AB testing is set for each of the 10 functions and distributed, the 10 functions can be managed independently. In this case, by utilizing the AB testing according to the present invention, even if a problem occurs in one function out of 10 functions, it is possible to deactivate only one function in which the problem occurs without rolling back all 10 functions.

In this way, unnecessary processes are reduced, and the effort of system monitoring after multiple new function deployments is reduced.

Through this, all ten functions are not applied at the same time from the time of distribution, and the distribution time and the system application time of the functions can be managed separately through the AB testing platform according to the embodiment of the present invention.

In addition, when the platform according to the embodiment of the present invention is applied, it is possible to set the exposure range of each experiment for 10 functions and manage the state, if necessary.

The present invention described above can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is this. In addition, the computer may include a processor or a control unit. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

10 : AB Testing Platform System
100: AB testing platform providing device

Claims (5)

receiving, by the processor, input data including a testing unique number;
managing, by the processor, AB testing matching the unique number;
managing, by the processor, an abnormal symptom based on a preset threshold; and
providing, by the processor, an AB testing result value derived based on the input data;
The step of managing the AB testing includes:
managing a first AB testing for a first function among new functions of a service to be distributed; and
including; managing a second AB testing for a second function among the new functions;
managing the first AB testing and the second AB testing independently of each other;
The step of managing the abnormal symptoms is,
determining, by the processor, occurrence of an abnormal symptom by comparing the conversion rate according to the first AB testing with the threshold;
providing, by the processor, information about the anomaly to the customer via an instant message when it is determined that an abnormal symptom has occurred; and
In a state in which automatic rollback is set, the processor automatically rolls back the distribution of changes related to the first function when it is determined that an abnormal symptom has occurred;
The criteria for judging the occurrence of the above abnormal symptoms are,
A method of providing an AB testing platform defined based on accumulated and stored anomaly related event data. The method of claim 1,
The step of managing the AB testing includes:
The method of providing an AB testing platform further comprising: by the processor, managing an AB test based on a signal received by matching the unique number from the customer device. 3. The method of claim 2,
The step of managing the AB testing includes:
setting, by the processor, an exposure range of AB testing based on a first signal received from the customer device;
starting, by the processor, AB testing based on the second signal received from the customer device; and
A method of providing an AB testing platform comprising a; the processor stopping the AB testing based on the third signal received from the customer device. delete delete

Images