JP2015011690A - Effect measurement program, method, and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize effect of an event which occupies KPI after the event.SOLUTION: An effect measurement program causes a computer to execute a prediction model generation step in which a measurement value is read from log data on a network, and by using a measurement value during a period before an event, a prediction model is generated based on a predetermined statistic prediction algorithm, and an output step for outputting a difference between a prediction value after the event which the prediction model shows and a measurement value after the event.

Description

  The present invention relates to an effect measurement program, method, and apparatus.

Research is a method for measuring the effectiveness of advertising. For example, a person who has been in contact with an advertisement is compared with a person who has not been in contact with the person based on a period or region. However, for example, campaigns using internet advertisements or mass advertisements are applied for product sales or services, CVR (Conversion Rate), repeat rate (also called revisit rate or reorder rate).
, KPI (Key Performance) such as the number of mentions of the product etc. in SNS (Social Networking Service) like miniblog, the number of browsing of websites (PV (Page View) number), CTR (Click Through Rate) It is difficult to measure the effect on Indicator. That is, for example, a questionnaire survey can be conducted, but the influence on sales cannot be measured. In addition, there has been no method by which managers of a relatively small EC (Electronic Commerce) site can easily measure the advertising effect.

  In addition, a method of estimating an expected click rate when content is displayed on each user terminal and creating a list for displaying an advertisement has been proposed (for example, Patent Document 1).

JP 2011-243125 A

  Here, it can be read from KPI whether or not the results have improved. However, it is difficult to separate the cause of the change in performance into the influence of the campaign and other trends. That is, it cannot be determined whether the results have improved because the campaign has been implemented or whether the results have tended to improve without the campaign. The effectiveness of a particular campaign could not be measured.

  Therefore, an object of the present invention is to make it possible to recognize the influence of an event on the KPI after a certain event.

  The effect measurement program according to the present invention reads a measured value from log data on a network and uses a measured value in a period before a predetermined timing to generate a prediction model generation step based on a predetermined statistical prediction algorithm. And an output step of outputting a difference between a predicted value after a predetermined timing indicated by the prediction model and an actually measured value after the predetermined timing.

  In this way, the magnitude of the influence (effect) of the event performed near the predetermined timing among the measured values of the log data recorded after the predetermined timing can be output as the difference. The predetermined timing is a timing designated by the user, for example, a timing at which a predetermined event such as an advertisement campaign is started. The user can objectively evaluate the outcome of an event performed near a predetermined timing, and can be used, for example, when improving the next event.

  Further, the computer may further execute a fitness calculation step for obtaining a fitness of the prediction model using a value in a period before a predetermined timing indicated by the prediction model and an actual measurement value in a period before the predetermined timing. Also good. In this way, the user determines how much the output result (event effect) is based on the prediction model that is close to the actual measurement value (that is, how reliable the output result is), based on the degree of fitness. it can.

  Furthermore, in the prediction model generation step, a plurality of prediction models based on a plurality of statistical prediction algorithms are generated, respectively, in the fitness calculation step, the fitness is calculated for each of the prediction models, and in the effect output step, the fitness is calculated. Alternatively, a predetermined number of prediction models may be used in descending order, and a difference between a prediction value after a predetermined timing indicated by each of the prediction models and an actual measurement value after a predetermined timing may be output. In this way, a highly reliable result can be output preferentially.

  In addition, in the prediction model generation step, a plurality of prediction models based on a plurality of statistical prediction algorithms are generated, and in the fitness calculation step, the fitness is calculated for each of the plurality of prediction models, and the calculated multiple matching A plurality of weights to be applied to each of the prediction models based on the degree, and in the effect output step, a sum of values weighted using the weights for the prediction values indicated by the plurality of prediction models, and after a predetermined timing A difference from the actually measured value may be output. In this way, it is possible to generate an unbiased prediction model that suppresses the tendency specific to each prediction algorithm and output the effect of the event.

  The program may be recorded on a computer-readable recording medium. Here, the computer-readable recording medium refers to a recording medium that accumulates information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from the computer. . Among such recording media, those removable from the computer include flexible disks, magneto-optical disks, optical disks, flash memories, magnetic tapes, and the like. As recording media fixed to the computer, there are HDD (Hard Disk Drive), SSD (Solid State Drive), ROM (Read Only Memory) and the like.

  Further, a method for realizing the present invention may be executed by a computer, or an apparatus for realizing the present invention may be provided.

  According to the present invention, it becomes possible to recognize the influence of an event on the KPI after a certain event.

It is a block diagram which shows an example of a system configuration. It is a functional block diagram which shows an example of an evaluation apparatus. It is a functional block diagram which shows an example of a WEB server and a SNS server. It is a block diagram which shows an example of a computer. It is a processing flow of an effect prediction process. It is a figure which shows an example of log data. It is a graph for demonstrating a measured value and a predicted value. It is a processing flow of the effect prediction process which concerns on the modification 1. It is a processing flow of the effect prediction process which concerns on the modification 2.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In addition, the structure of the following embodiment is an illustration and this invention is not limited to the following structures.

<System configuration>
FIG. 1 is a diagram for explaining a system for measuring an advertising effect according to the embodiment. This system includes an evaluation device 1, a WEB server 2, an SNS server 3, and a user terminal 4. These are connected to each other via the network 5.

  The evaluation device 1 measures the advertising effect according to the present embodiment. The WEB server 2 holds a web page such as an EC site that sells products and provides services, and transmits the web page in response to a request from the user terminal 4. The WEB server 2 records an access log and a transaction log (also referred to as “transaction log”). The SNS server 3 provides SNS to the user. In addition, the SNS server 3 publishes, for example, a log of postings to the SNS (for example, posting contents and the number of postings, also referred to as “posting log”) to an external device. The user terminal 4 accesses the WEB server 2 or the SNS server 3 based on a user operation, and receives a web page.

  In addition, although one user terminal 4 is shown in FIG. 1, a plurality of user terminals 4 may exist. Also, the evaluation device 1, the WEB server 2, and the SNS server 3 may be configured such that a plurality of devices perform processing in parallel or perform processing in a shared manner.

<Functions of each device>
FIG. 2 is a functional block diagram illustrating an example of the evaluation device 1. The evaluation device 1 according to the present embodiment is a device that measures the effect of an event such as an advertisement campaign. The evaluation device 1 is a computer, and includes a log acquisition unit 11, a model generation unit 12, a fitness calculation unit 13, a weighting processing unit 14, an effect estimation unit 15, and a data storage unit 16.

  The log acquisition unit 11 acquires an access log or transaction log from the WEB server 2 or acquires a post log from the SNS server 3 via the network 5. In the present embodiment, the aggregation of logs such as the sales for each predetermined period, the repeat rate, the number of posts to the SNS referring to specific matters, and the number of browsing web pages (number of PV) is also referred to as KPI. The log acquisition unit 11 acquires logs before and after a predetermined timing specified in advance by the user. For example, the user designates in advance an arbitrary timing before and after a predetermined event such as an advertisement campaign. In the present embodiment, the period before the predetermined timing is a period before a predetermined event such as an advertisement campaign is performed, and the period after the predetermined timing includes a period after the event is performed.

  The model generation unit 12 generates a prediction model based on the log by using a statistical prediction method such as Multiple Holt-Winters, GARCH, or SARIMA, and generates a prediction model of a future KPI. The model generation unit 12 generates a prediction model based on a log in a period before a predetermined timing. In addition to the above-described statistical prediction methods, various variations have been proposed, and existing techniques can be appropriately employed.

  In addition, the goodness-of-fit calculation unit 13 uses the prediction model generated by the model generation unit 12 and the actual value indicated by the log to determine the goodness of fit of the prediction model (“fit rate” (“fit rate”) expressed as a percentage). Ask). The goodness of fit is obtained using, for example, a square error or χ (chi) square.

The weighting processing unit 14 weights each prediction model using the fitness, and obtains prediction values based on a plurality of prediction models. For example, when the model generation unit 12 generates a plurality of prediction models, the fitness calculation unit 13 calculates the fitness of each prediction model, and the weighting processing unit 14 calculates each prediction based on the fitness of each prediction model. Determine the rate at which the model is adopted.

  The effect estimation unit 15 indicates to the user the difference between the predicted value of KPI in the period after the event and the log in the period after the event, calculated based on the log before the execution of a certain event.

  In addition, the data storage unit 16 holds a log received from the WEB server 2 or the SNS server 3, a generated prediction model, a calculated predicted value of KPI, a fitness level, and the like. In addition, the data storage unit 16 may temporarily store intermediately generated data.

  FIG. 3 is a functional block diagram illustrating an example of a server such as the WEB server 2 and the SNS server 3. The WEB server 2 includes a control unit 21, a content storage unit 22, and a log data storage unit 23. In response to a request from the user terminal 4, the control unit 21 transmits data (also referred to as “page data”) that is stored in the content storage unit 22 and is used to display a website. In the present embodiment, it is assumed that the WEB server 2 holds page data for displaying an EC site in the content storage unit 22. Further, the control unit 21 records an access log from the user terminal 4 and a transaction log received from the user terminal 4 in the log data storage unit 23.

  The SNS server 3 includes a control unit 31, a content storage unit 32, and a log data storage unit 33. In response to a request from the user terminal 4, the control unit 31 transmits page data for providing the SNS held in the content storage unit 32. In the present embodiment, it is assumed that the SNS server 3 provides a so-called miniblog. Specifically, the SNS server 3 accepts posting of text from the user, and publishes it to some other users permitted by the posting user or opens it without restriction. In addition, each user can include an identifier (keyword such as a so-called hash tag) indicating the content of the post in the posted text so that, for example, a post related to a certain product or service can be extracted.

<Device configuration>
FIG. 4 is an example of a configuration diagram of a computer. The evaluation device 1, the WEB server 2, the SNS server 3, and the user terminal 4 are computers as shown in FIG. The computer includes a CPU (Central Processing Unit) 1001, a main storage device 1002, and an auxiliary storage device 1.
003, a communication IF (Interface) 1004, an input / output IF (Interface) 1005, a drive device 1006, and a communication bus 1007. The CPU 1001 performs processing according to the present embodiment by executing a program (also referred to as “software” or “application”). The main storage device 1002 caches programs and data read by the CPU 1001 and develops a work area of the CPU. Specifically, the main storage device is a RAM (Random Access Memory), a ROM (Read Only Memory), or the like. The auxiliary storage device 1003 stores programs executed by the CPU 1001, information used in the present embodiment, and the like. Specifically, the auxiliary storage device 1003 is an HDD (Hard-disk Drive).
And SSD (Solid State Drive), flash memory, and the like. The main storage device 1002 and the auxiliary storage device 1003 are the data storage unit 16 of the evaluation device 1, the content storage unit 22 and log data storage unit 23 of the WEB server 2, the content storage unit 32 and log data storage unit 33 of the SNS server 3, etc. Work as. The communication IF 1004 transmits / receives data to / from other computers. The communication IF 1004 is specifically a wired or wireless network card or the like. The input / output IF 1005 is connected to the input / output device and accepts input from the user or outputs information to the user. Specifically, the input / output device is a keyboard, a mouse, a display, various sensors, a touch panel, or the like. The drive device 1006 reads data recorded on a storage medium such as a magnetic disk, a magneto-optical disk, and an optical disk, and writes data to the storage medium. The above components are connected by a communication bus 1007. A plurality of these components may be provided, or some of the components (for example, the drive device 1006) may not be provided. Further, the input / output device may be integrated with the computer. In addition, the program executed in this embodiment is provided via a portable storage medium readable by the drive device 1006, a portable auxiliary storage device 1003 such as a flash memory, a communication IF 1004, and the like. It may be. Then, when the CPU 1001 executes the program, the computer as described above is caused to work as the evaluation device 1, the WEB server 2, the SNS server 3, or the user terminal 4.

<Effect measurement processing>
Next, the operation of the evaluation apparatus 1 will be described. FIG. 5 is a process flow showing an example of the effect measurement process executed by the evaluation apparatus 1. In this embodiment, it is assumed that the WEB server 2 publishes an EC site on the Internet, and an access log from the user terminal 4 and a transaction log with the user are recorded. In addition, the SNS server 3 receives a post from the user terminal 4 and publishes a post log on the Internet.

  First, the log acquisition unit 11 of the evaluation device 1 acquires settings used for processing (FIG. 5: S (step) 1). In this step, the timing (for example, date) at which an event such as a prediction target KPI, an advertisement campaign, or the like, which is set in advance by the user of the evaluation apparatus 1 (or date) is acquired. In the present embodiment, the effect of the event is measured for KPIs after the timing.

  Next, the log acquisition unit 11 acquires log data (S2). In this step, the log acquisition unit 11 accesses the WEB server 2 or the SNS server 3 via the network 5 and acquires log data based on the setting acquired in S1. In the present embodiment, it is assumed that at least one of an access log, a transaction log, and a posting log is acquired according to the prediction target KPI acquired in S1.

  Here, for example, log data as shown in FIG. 6 is acquired. In the example of FIG. 6, the KPI (that is, measured value) total values such as sales, repeat rate, number of SNS posts, and number of PV (viewing number) are recorded for each day. Note that the log data may be aggregated in advance in the WEB server 2 or the SNS server 3, or may be aggregated after the log acquisition unit 11 acquires the log data. FIG. 7 shows an example in which KPI (for example, sales) is represented by a graph. In FIG. 7, it is assumed that the solid line indicates KPI. In this step, the KPI in the period before the timing when the predetermined event is performed and the KPI in the period after the timing when the predetermined event is performed are acquired.

  Next, the model generation unit 12 of the evaluation apparatus 1 generates a KPI prediction model using the acquired log data (S3). In this step, the model generation unit 12 generates a prediction model using log data in a period before the timing at which a predetermined event is performed.

  Moreover, the model production | generation part 12 utilizes statistical prediction methods, such as Holt-Winters, GARCH, and SARIMA. Note that the model generation unit 12 can calculate a predicted value of a future KPI using a prediction model generated based on the log data. For example, a prediction model as shown by a broken line in FIG. 8 is generated. The prediction model is generated based on log data in a period before the event. Therefore, it can be said that the value in the period after the event implementation indicated by the prediction model is the predicted value of KPI when there is no event.

  As a statistical prediction method for time-series data, existing techniques can be adopted in addition to those exemplified. For example, by deriving a regression equation based on log data (actually measured values), it is possible to generate a prediction model having a tendency, circulation, and seasonality.

  In addition, the effect output unit 15 of the evaluation apparatus 1 includes a predicted value of KPI after a predetermined timing indicated by the prediction model generated by the model generation unit 12 and an actual measured value of KPI after the predetermined timing indicated by the log data. The difference is output (S4). Here, the effect output unit 15 may calculate and output the difference value, or after a predetermined timing indicated by the prediction model, for example, so that the user of the evaluation device 1 can visually recognize the difference. The predicted value of KPI and the value of KPI after a predetermined timing indicated by the log data may be output as a graph. For example, in the graph shown in FIG. 8, the actual measurement value (solid line) in the period after the event and the predicted value (dashed line) when there is no event are output. In other words, the difference between the actually measured value and the predicted value can be said to be the effect of the event estimated from the output result.

  According to the present embodiment, the user of the evaluation device 1 can know the difference between the predicted value when the event is not performed and the log KPI that is the actual measurement value after the event. That is, it becomes possible to recognize the influence (effect) of the event occupying the KPI after a predetermined timing. For example, as shown in FIG. 7, when the actual measurement value (solid line) is higher than the predicted value (broken line) in the period after the event is performed, the difference is considered to be the effect of the event. For example, when the KPI is sales, the effect of the event can be calculated as a specific numerical value of sales for a predetermined period after the event. Further, by comparing the effect of an event performed near a predetermined timing with the effect of an event performed before that, it is possible to compare which event was more effective. For example, the user of the evaluation device 1 who is an operator of the EC site can evaluate the executed advertisement campaign with reference to the output result, and can improve the next campaign plan.

<Modification 1>
Here, it can be said that the reliability of the difference output in the above embodiment depends on how close the prediction model is to the actually measured value. In the first modification, the degree of fitness indicating how close the generated prediction model is to the actually measured value is calculated and presented to the user of the evaluation device 1.

  FIG. 8 is a process flow illustrating an example of the effect measurement process according to the first modification. In addition, since the process of S11-S13 is the same as that of S1-S3 of FIG. 5, description is abbreviate | omitted. After the KPI prediction model is generated, the fitness level calculation unit 13 calculates the fitness level between the log data and the prediction model (S14). In this step, the goodness-of-fit calculation unit 13 calculates a value indicating the magnitude of the error using the KPI log that is the actual measurement value and the prediction model (for example, regression equation) generated in S3. The value indicating the magnitude of the error can be obtained, for example, by the ratio of the square error with respect to the actually measured value or the chi-square test.

ｔは期間の開始を示し、Ｔはイベントを実施又は開始したタイミングを示す。ｘ（ｔ）は、実測値であり、ｆ（ｔ）は予測モデルによる予測値である。なお、ＲＳＳの値が低いほど適合度は高いといえる。 For example, the ratio of the square error with respect to the square sum of the actually measured values (herein, also referred to as RSS (Residual Sum of Squares)) can be calculated by the following equation (1).

t indicates the start of the period, and T indicates the timing at which the event is performed or started. x (t) is an actual measurement value, and f (t) is a prediction value by a prediction model. Note that the lower the RSS value, the higher the fitness.

Ｏは実測値であり、Ｅは期待値（予測モデルによる予測値）である。なお、カイ二乗値が低いほど、適合度は高いといえる。 Further, the chi-square value (χ ² ) can be calculated by the following equation 2.

O is a measured value, and E is an expected value (a predicted value based on a prediction model). Note that the lower the chi-square value, the higher the fitness.

  The effect output unit 15 also includes a difference between a predicted value of KPI after a predetermined timing indicated by the prediction model generated by the model generation unit 12 and a measured value of KPI after a predetermined timing indicated by the log data, and a prediction. The fitness of the model is output (S15). The user of the evaluation device 1 can recognize not only the effect of the event occupying the KPI after a predetermined timing, but also the reliability of the influence of the event by referring to the output fitness.

  Note that a plurality of prediction models based on a plurality of prediction methods may be generated in S13, and the fitness may be calculated for each of the plurality of prediction models in S14. In S15, a plurality of differences and a plurality of matching degrees are output. In this way, the user can compare the effects of events based on values predicted by different methods and refer to them appropriately.

  Further, a plurality of prediction models based on a plurality of prediction methods are generated in S13, the fitness is calculated for each of the plurality of prediction models in S14, and a predetermined number of prediction models having a high fitness are adopted in S15. Also good. In this way, the effect of the event calculated using a highly reliable prediction model is output preferentially. Moreover, you may make it employ | adopt a prediction model with higher fitness than a predetermined threshold value.

<Modification 2>
A plurality of prediction models may be employed in a composite manner. FIG. 9 is a process flow illustrating an example of the effect measurement process according to the first modification. Since the process of S21-S24 is the same as that of S11-S14 of FIG. 8, description is abbreviate | omitted.

  In the modified example 2, the weighting processing unit 14 of the evaluation device 1 generates a prediction model weighted according to the fitness (FIG. 9: S25). In this step, first, the weighting processing unit 14 calculates a weight to be applied to each of the plurality of prediction models based on the fitness. For example, the weighting processing unit 14 uses a reciprocal of the square error to determine a weight that becomes higher as the prediction value of the prediction model having a higher fitness. For example, the ratio of the reciprocal of the square error of a certain prediction model to the sum of the reciprocal of the square error of the prediction model is used as a weight. Then, the weighting processing unit 14 calculates a sum of values obtained by multiplying the value indicated by each model by the weight as a predicted value in the second modification.

  Then, the effect output unit 15 outputs the difference between the predicted KPI value after the predetermined timing indicated by the prediction model generated by the weighting processing unit 14 and the measured KPI value after the predetermined timing indicated by the log data. (S26). The user of the evaluation device 1 can recognize the effect of the event occupying the KPI after a predetermined timing. According to the second modification, an unbiased prediction model that suppresses a tendency specific to each prediction method can be generated, and an event effect can be output.

<Other variations>
This embodiment and a modification are examples. For example, the algorithm for generating the prediction model and the algorithm for obtaining the fitness are not limited to the exemplified methods. The value used as the KPI is not limited to sales, repeat rate, number of SNS posts, number of PV (viewing number), and various log data such as the number of members acquired, the number of drugs, the number of document requests, and the like can be used.

  5, 8, and 9 are examples of the processing flow, and the order may be changed or executed in parallel as long as the processing result does not change.

DESCRIPTION OF SYMBOLS 1 Evaluation apparatus 11 Log acquisition part 12 Model production | generation part 13 Conformity calculation part 14 Weighting process part 15 Effect output part 16 Data storage part 2 WEB server 3 SNS server 21,31 Control part 22,32 Content storage part 23,33 Log data Memory

Claims (6)

A prediction model generation step of reading an actual measurement value from log data on the network and generating a prediction model based on a predetermined statistical prediction algorithm using an actual measurement value in a period before a predetermined timing;
An output step of outputting a difference between a predicted value after the predetermined timing indicated by the prediction model and an actually measured value after the predetermined timing;
Effect measurement program that causes a computer to execute. Using a value in the period before the predetermined timing indicated by the prediction model and an actual measurement value in the period before the predetermined timing to cause the computer to further execute a degree-of-fit calculation step for obtaining the degree of fit of the prediction model; The effect measurement program according to claim 1. In the prediction model generation step, a plurality of prediction models based on a plurality of statistical prediction algorithms are generated,
In the fitness calculation step, the fitness is calculated for each of the plurality of prediction models,
In the output step, a predetermined number of prediction models are used in descending order of the fitness, and a difference between the prediction value after the predetermined timing indicated by each of the prediction models and the actual measurement value after the predetermined timing is output. The effect measurement program according to claim 2. In the prediction model generation step, a plurality of prediction models based on a plurality of statistical prediction algorithms are generated,
In the fitness calculation step, the fitness is calculated for each of the plurality of prediction models, and a plurality of weights to be applied to each of the prediction models is determined based on the calculated fitness.
The output step outputs a difference between a sum of values weighted by using the weights with respect to prediction values indicated by the plurality of prediction models and an actual measurement value after the predetermined timing. Effect measurement program. A prediction model generation step of reading an actual measurement value from log data on the network and generating a prediction model based on a predetermined statistical prediction algorithm using an actual measurement value in a period before a predetermined timing;
An output step of outputting a difference between a predicted value after the predetermined timing indicated by the prediction model and an actually measured value after the predetermined timing;
The effect measurement method that the computer executes. A prediction model generation unit that reads an actual measurement value from log data on the network and generates a prediction model based on a predetermined statistical prediction algorithm using an actual measurement value in a period before a predetermined timing;
An output unit that outputs a difference between a predicted value after the predetermined timing indicated by the prediction model and an actual measured value after the predetermined timing;
An effect measuring apparatus.

Images